Systems Architecture at Massive Scale: The Unreasonable Effectiveness of Simplicity

Building distributed systems that work is quite challenging. But systems that provide customer value will grow and evolve, and scaling those systems by multiple orders of magnitude is even harder. Citing specific failures and successes from internet-scale consumer properties like Google and eBay, and measuring customer value through the architecturalilities, this session deep-dives into the often counterintuitive architectural principles of simplicity, orthogonality, asynchrony, and feedback. Simplicity helps us turn seemingly intractable problems into discrete and solvable ones. We will discuss component responsibilities, single-threaded straight-line processing, and making changes in small steps. Orthogonality allows us to compose separate concerns and bound the complexity of each dimension of the problem. We will discuss processing abstractions like mixins and addons, as well as communication abstractions like channels and pipes. Asynchrony and eventual consistency further reduce complexity and lead us to drive the system design directly from what really matters – what is actually changing in our core business domain. We will explore events, caching, and dataflow. Finally, fast and consistent feedback help us continually maintain and improve a complex system. We will discuss quality feedback in the small, as well as systems feedback in the large. You will take away some interesting anecdotes, as well as actionable insights you can put into practice in scaling your own systems.