Designing Features for Mature Systems: Lessons Learned from Manta

Designing a distributed system (or any system) effectively requires careful design decisions based on tradeoffs within a set of constraints. Making changes to a mature system, whether by adding a new feature or just improving it, constrains the problem further: Changes should not violate existing invariant of the system, and they should work harmoniously in the context of tradeoffs made for the system long ago.

This talk will explore a case study of expanding a distributed system: designing a multipart upload API for Manta, Joyent’s highly-scalable distributed object store. We will talk about the design goals of Manta, then discuss the goals of the new API and why users wanted it. From there, we will design the feature from the ground up, discussing how to achieve the goals of the API -- including idempotent distributed commit and abort operations and performance goals for other operations -- within Manta’s established invariants and design goals.

Distributed systems can be especially difficult to reason about -- especially when making changes to them -- and the lessons learned from adding a multipart upload API to Manta resonate with me for all systems I work on.