Patterns of Distributed Systems

Unraveling the Foundations: A Patterns-Based Exploration

James Lewis: Hello, and welcome to GOTO Book Club. Once again, my name is James Lewis. I'm a director at ThoughtWorks. And today I'm delighted to be talking to Unmesh Joshi about the release of his new book, and the patterns within it, "Patterns of Distributed Systems." Unmesh , would you like to introduce yourself to the audience?

Unmesh Joshi: Hey, folks. My name is Unmesh Joshi, and I work as a principal consultant at ThoughtWorks. I've been with ThoughtWorks for close to 16, or 17 years now. And last several years, I've been thinking and writing about distributed systems. And we'll cover more of that, I think, as we talk.

James Lewis: That's a fairly long time to be in a consulting organization. Can you tell me a bit about the background? Maybe some of the some of the experiences that led you to become interested in distributed systems, the sort of experiences you've had that led you down this path? Because, I mean, frankly, this is a lesser trodden path in, you know, the software engineering literature, certainly the sort of...the more sort of user-friendly literature, because it's regarded as a very difficult topic. So, what were the experiences that led you to this?

Unmesh Joshi: It was an interesting journey, I would say. I think any professional for the last several years working with the cloud, I mean, you see services and products you use on the cloud or even in your data centers that are distributed by nature, right? I mean, you use Kafka, you use even simple things like Amazon S3. And I had been using, like everyone else, some of that. And I was always, like, curious, I mean, why now? And why they are designed the way they are? But I never spent more time going in-depth or in detail. Until the point where, like, around, I think, 2017, I got a chance to work on a very interesting project. And so, we were involved in developing software for an optical telescope. And that's going to be one of the largest telescopes once it goes live called the Thirty Meter Telescope.

Naturally, because it was a telescopic ecosystem, the software that we were developing, we could not use off-the-shelf products. I mean, we used frameworks like Akka. But we had to do a lot of framework development ourselves. It turned out just because multiple telescopes of systems need to coordinate, and you need to keep data that's highly available and also consistent. What we were building turned out to be a distributed system. At that point, I mean, I kind of had to go in-depth of how these things are built and how they work. All the literature, I mean, the books and academic papers that are out there, I mean...

James Lewis: You're essentially going back to Lamport, aren't you?

Unmesh Joshi: Yes, we'll talk about Lamport, I mean, Paxos and other things. But a lot of that, I mean, it took a lot of...I mean, you need to stretch your imagination when you read something and then you go and, say, start writing Java code. How do you map things? And what I started doing at that point was, I mean, I think, I mean, to test if you understand something is to write code and to run that in production, I mean, that's the ultimate test always. But then I thought it might be easier if I looked at code that...I mean, there are a lot of open-source frameworks out there, like Kafka and Cassandra and stuff like that, those are open-source tools. And they are battle-tested in production. So why not go and look at their code and try to make sense of it and see what the building blocks look like? So it's also a good validation of some of what we are building if it matches with some of the decisions that are taken there as well. To start with, it was hard because you can imagine, you open a Cassandra code base, and you don't know where to start really.

So what I started doing was, I mean, I just thought if I need to build Kafka, Kafka-like thing, what are the things are needed and maybe look at some of the earlier versions of Kafka. And it's a good thing with open source that you can go through the commit history. Generally, you see that some of the earlier versions of all these complex products are simple and easier to look at. So I went through those, and I tried building my miniature versions of these products. And that naturally, because I looked at multiple of these, I could figure out that okay, similar things are happening, even if not the same, there are similar building blocks in code that you see. And I just happened to talk to Martin Fowler about it, and this is the approach that I'm taking to learning how distributed systems work. I also tried teaching using code because we were attempting to teach more developers about distributed system concepts. And we also had some college professors taking sessions for our developers. But, I mean, it never worked well because, as I said, there is a gap between the theory that you learn and the code.

So this more code-driven approach of if you need to build something that is a distributed key-value store, let's say, what building blocks you need to think of and how they look like in code. And that worked pretty well. That's when Martin thought that, I think the patterns approach is probably better because you see things that are similar but not the same. They cover a broader range of implementations out there. Yeah, so that's how it started.

James Lewis: You touched on several things I find quite interesting there. I remember a conversation I had some years ago with Brian Goetz about the "Java Concurrency" book he wrote. And I was sort of amazed at how he got into it. I was like, "Oh my God," because it became the Bible, right? If you're working in multi-threaded systems in Java, everyone has that book on their shelf. I said, "Brian, how did you...you know, how come you to write this book? You know, were you this massive expert in concurrency? He said, "No, it turns out I wanted to learn how it worked. I thought the best way of doing that was to start writing this stuff down and start writing code to work out how it worked for myself, and it turned into the 'Java Concurrency' book." It seems like a very similar thing that you've done [crosstalk 00:08:10.845] curiosity.

Unmesh Joshi: It was more like demystifying some of the words that you...I mean, you read Lamport's clock, and you want to see, I mean, how does that look in code and where it's used?

James Lewis: Yes, because jokingly you mentioned the Lamport's papers earlier, a minute or so ago. And actually, they genuinely are that, them and the Liskov papers and going back almost like two generations of software engineers now. They are. But also, they're academic. They're academic. So it is fascinating how you've taken those ideas and brought them into a form that, you know, regular Joes like myself, regular software engineers like myself can understand, right? It's a pretty staggered achievement, I have to say. First of all, thank you from everyone for doing it.

The other thing I think you talked about there, which I thought was quite amusing in a sense was this idea that I think it's typical of many of our colleagues, I think. It's one of the reasons I think I love working with ThoughtWorks, which is the idea that the way you get to understand something is by turning it into code and running it into production. It's almost like, you know, Richard Feynman, you know, the Nobel Prize-winning physicist, he used to say about, you know, the way you understand something is by teaching it to others. I guess this is like the strong version of teaching it to others, right? It's you're teaching it, right, where there is no misunderstanding, you know?

Unmesh Joshi:, I have it so many times. And maybe it's just me, but I've seen it with most of the professional developers that you understand things. And that's maybe the nature of software development itself that you...I mean, there is a limit to how much you can specify and imagine things on a whiteboard and maybe with some abstract concepts. Finally, when you write code, you get more insights and then you can reflect on those insights and then maybe go back to specifications and stuff like...I mean, that's just the very nature of software development, I think.

Recommended talk: When To Use Microservices (And When Not To!) • Sam Newman & Martin Fowler • GOTO 2020

Should Every Software Developer Understand Distributed Systems?

James Lewis: So if we were to, I guess, come back to basics, right, so when we say distributed systems, what do we mean when we say distributed system? What makes...I know maybe it's a silly question, but I mean, let's come right back, and when we say distributed system, what is that? What does that entail?

Unmesh Joshi: That's a great question, I think because people interpret distributed systems with their context. I mean, the microservices ecosystem itself is...I mean, can be called a distributed system as well. And Kafka is a distributed system as well. And cloud is a distributed system. What I primarily have focused on is, essentially, when you have data systems, I mean, you need to maintain state and you need to maintain it on multiple servers to make it available. Then you need to make sure that that state is consistent across servers and whatever systems, and obviously, you need to shard or partition that state so that you can have lots and lots of data stored and processed. So I think what I have focused on is this kind of systems and things like Kafka, Cassandra, Amazon S3, Cosmos DB, and I mean, all the data things.

James Lewis: So really, we're talking in terms of CAP theorem, we're talking...and correct me because I might be wrong here, but in terms of CAP theorem, we're talking about consistent systems as opposed to available systems.

Unmesh Joshi: Consistent. And there is always a balance. I think that all these systems, like Paxos, for example, also is available as long as the majority of the servers are available. But beyond that, it will just favor consistency.

James Lewis: Cool. Thank you. Maybe let's talk about, I mean, you mentioned briefly about the patterns. Why did you go for the patterns approach to start with? Why not just go for, you know, writing your piece of software essentially? Why not release your own Kafka, right, and say, look at this?

Unmesh Joshi: I mean, so patterns I thought was a useful approach, because let's say if you have to build something like Kafka, the concepts that you need to know or need to implement, there is a lot of similarity between, for example, when you implement something like Cassandra, even if not the same, or you are implementing something like HCD, which is a Raft implementation. There are a lot of common concepts between, say, Raft and Kafka's replication mechanism. So patterns approach, allows you to talk of these similar concepts that span across multiple implementations. And then they become more useful, I think, because once...for example, there is a concept of, let's say, high water mark that when you are replicating a log, you need to demarcate which portion of your log is, like, stable, it's not going to change, and which portion is volatile. I mean, it might change if your nodes crash and letters change and stuff like that. Now, that high watermark concept, whenever you use log replication like in Kafka, is also there in Raft, which is called a committed index. And whether you use some other replicated log algorithm like viewstamped replication or multiplex source, which no one has documented how it's implemented.

But you need some concept like that. And once you know what that concept is and how it's typically implemented, you read a Raft paper or a view-stamped replication paper or Kafka's replication mechanism, you know what's going on and you can connect the dots.

Recommended talk: The Many Meanings of Event-Driven Architecture • Martin Fowler • GOTO 2017

Comparing Patterns: Raft and Paxos

James Lewis: So you mentioned Raft and Paxos. I mean, probably these are names that people have been bandied around or different mechanisms for staying consistent, etc., etc. So I mean, just to put you on the spot, here you go, describe the differences between Raft and Paxos for us.

Unmesh Joshi: I think that's important because you see that Paxos, the Microsoft and Google folks, and all the services they have, they always mention that they implement Paxos for replica consistency and all the open-source world says that they implement Raft. And if you read Paxos paper, the original "Paxos Made Simple" or other papers, what it describes is what is called as a single-degree Paxos, like how multiple nodes achieve consensus on a single value. So in the literature, what's understood as Paxos, and I have documented that as a pattern. So I've documented Paxos as a pattern because we thought that concept that you need two phases to reach an agreement between nodes. Then you use a majority quorum to make sure that the decision taken by the previous quorum is visible whenever you try next. And you have something called as I mean, what I have documented as a different pattern called generation clock.

So all this coordination is done by a leader, which is called a proposer in Paxos. And you mark the leadership with monotonically increasing numbers so that if there are...I mean, you always tolerate multiple leaders, but you can always cut off older leaders. So only the latest leader can make progress. So this basic mechanism of how to combine these concepts of epochs or generation clock, two-phased execution, and then the recovering from, let's say, partial rights, Paxos documents that and how to do that for, let's say, a single request or a single value.

James Lewis: You mentioned something, Unmesh ...Do you mind if I just interrupt, because you mentioned something really which we should probably...maybe this is not the right time we can come back to it, but you mentioned something like the latest, we tolerate multiple, but the latest. And that comes to the heart of the problems with distributed systems, doesn't it?

Unmesh Joshi: We'll talk about it. So that latest comes in various areas of...so what is the latest value? I mean, in terms of Paxos, when I say latest leader, I mean, what is the leader that has just won the election, let's say, which has the latest epoch or the latest generation clock, and it can be the only leader that can make progress. So these concepts, and how they work together are documented in Paxos, and literature, mostly it's documented how to do that for a single value or a single request. But obviously, one of the side effects or one of the issues with this basic mechanism is this basic mechanism is what you can call initialized ones. So once you reach an agreement, you cannot have another request or another value that can be chosen. So it becomes immutable after that.

And that's...even if it's a good way to understand the basic mechanism and how to structure that basic mechanism to reach an agreement, it's not very useful in practice if you have to build a database, for example, because, I mean, you need to update values multiple times and you need to maybe execute requests like register release when you implement something like HCD. And the mechanism to do that essentially is to arrange all your requests in an ordered log. And you can imagine, again, that if you execute this basic Paxos mechanism on each entry on this ordered log, then you know that all the nodes will reach agreement on each entry. And then if they execute these requests and follow that log order, they will execute all the requests in the same order and then they will reach the same state.

This basic Paxos mechanism, like two phases and establishing epoch by quorum voting, it's not very efficient, naturally, if you do it every time you get a user request. And you need to optimize that for any production implementation. And generally, that's what happened in practice. I mean, whenever someone says that they implement Paxos for implementing, let's say, replica consistency in their database, mostly they are using something like a replicated log mechanism. And that replicated log mechanism essentially then combines this basic Paxos to elect a leader at startup. And then all the requests essentially are ordered in the log. This single leader does that. So it's not like for every request you go and try to establish a leadership. And then it also combines things like the high water mark that I talked about that you demarcate your ordered log so that you know what is the stable portion of the log and what is the not-so-stable portion. And Raft essentially details how to do that with all the nitty-gritty in the implementation.

James Lewis: So essentially, Raft is an evolution towards practicality...

Unmesh Joshi: Practicality, yes.

James Lewis: ...of the basic Paxos algorithm. When I say basic Paxos algorithm, it's not basic at all. I mean, it reminds me...I think I shared with you previously an article that James Lewis Mickens, the former Microsoft Distributed Systems researcher wrote. He used this magazine on Byzantine generals. "The Byzantine generals' algorithms are popping up all the time. And frankly, they're really simple." They're not simple.

Unmesh Joshi: Oh, yeah.

James Lewis: It brings me actually, because you talked about a couple of things there relating to...well, I mean, in my mind relating to time. And I think that's something crucially we should probably talk about because it's at the heart of the problem with distributed systems, isn't it?

Unmesh Joshi: Yes.

James Lewis: How you manage time across multiple different processes, multiple different systems.

Unmesh Joshi: Yes. Absolutely. And very interestingly, I mean, as you know, Google has this timing infrastructure called TrueTime with atomic clocks and GPS and stuff like that. And a good thing to know is that even with atomic clocks and all the infrastructure that Google has, when they have data centers across the globe and servers spanned so far, they cannot get exact clocks exactly matching. So there is always a delta. And what essentially Google does is make sure that the delta between two clocks is minimal. So it's 7 milliseconds, I think, is what they guarantee. But essentially, practically, I mean, you will never get two servers to agree at same time. And that's one of the big issues.

Lamport clock is one way. And I have documented in the book how the Lamport clock works. I mean, you essentially just maintain an integer and you pass along that integer with every request and then every server receiving the request also keeps track of what value it's getting and what the value it has. And when I'm saying, I mean, typically in databases, you can imagine if it's a key-value store or, let's say, even your document store, your document is considered, you can consider as a value or in your RDBMS, your row, you can consider as a value. So you store this version with that value or with the primary key or with the key of that value. And then comparing these values, sorry, these keys or these numbers attached to the keys, you can figure out which update happened before another one. I mean, that's a simple Lamport clock mechanism.

Now, as you can imagine, in databases that we all use, if we have to compare integers to know which values are ahead and later, it can be quite confusing. So most databases combine this technique, this basic Lamport clock technique of tracking the integer with the actual date and time that you get on your servers. And that's called a hybrid clock. So most of your databases like MongoDB and YugabyteDB and CockroachDB, I mean, you see that in practice, they use something called a hybrid clock. And that allows you to combine this system time with this integer that can track which values are greater after the other. But yeah, I mean, it's fascinating and I think it's something that people will find exciting when they read in the book how this is typically implemented. I mean, there's a simple Java code that I have in there.

James Lewis: You mentioned Spanner. I remember maybe our friends at the Book Club at GOTO could dig this out and add it to the notes because I remember seeing some of the Spanner engineers talking about it. So you talk about TrueTime and Cloud Spanner is the database implementation that makes.

Unmesh Joshi: Right. 

Unmesh Joshi: Just one thing to add there. So just an example of how this pattern approach is useful. I mean, what's done in TrueTime or with the use of TrueTime in Spanner is essentially that you get this, let's say, maximum delta between two clocks. I mean, how much do clock values differ?

Then how you can use that essentially is to, if you are writing a particular value at a particular time, let's say you are writing a value at time 5 and you know that it will take maybe 500 milliseconds to...and I'm just giving an example value, 500 milliseconds is too high to wait. But within 500 milliseconds, all the other servers in the cluster are going to get that 5 p.m. as the value. So what this writer does is it waits for 500 milliseconds just to make sure that everyone else has that clock value or time value and then it makes that value visible or then it writes that value at that timestamp.

So if you know how much that maximum skew between or across servers in a cluster can be, then you can implement it without TrueTime as well. I mean, that's what most of the open-source databases do. I mean, they assume that maybe the clock skew, the max drift, maybe 200 milliseconds within your AWS cluster. And then they can just implement that wait for 200. I mean, TrueTime just makes it more practical saying it won't be more than 7 milliseconds. So you can wait for 7 milliseconds. It's not that bad.

James Lewis: You're guaranteed, essentially. I remember the talk and it was a fascinating talk they gave. They had this quite amusing, sort of, portion of it when they were talking about the CAP theorem again, back to the CAP theorem. And they were talking about how they are consistent and available. And they said, "Oh, you know, it turns out we've broken the CAP theorem," that was the start of the talk. At the end of the talk, they said, "Actually, we haven't broken the CAP theorem at all. It's just that we never partition. Our networks are so good that we just don't partition," which I thought was quite an amusing, kind of, appendix to the end of the talk.

Unmesh Joshi: It's tricky. I mean, one of these...I mean, understanding things at the implementation level, I think how it helps is you get this kind of intuitive...and I mean, even if you don't read specifications and you don't know exactly how these things work, you get this intuitive understanding. So if something breaks in production, for example, and you need to make sense of what might have happened, you at least have some pointers. Like this...and I've given this example, and I think a lot of people give this example, that there was this Cloudflare outage, I think a couple of years back, and the outage was essentially because the HCD server that they were using, there was outage in that. And then there is this inherent issue with Raft's leadership election. And I mean, it can very well happen with any basic consensus implementation that you just keep on electing leaders, and no one can make progress. I mean, it can very well happen unless some care is taken. But if you are aware that, okay, something like this might happen, I mean, it might happen in your Kafka cluster or it might happen in your Kubernetes cluster as well. We are all fortunate that things don't fail that often.

James Lewis: I mean, just talking about failure, because I mean, failure is, I think, one of the...the edge cases in these systems are...I mean, there's lots of them, and they're pretty sharp, right? I mean, sharp edges. And I don't know if you've come across, I'm sure you have come across the Jepsen suite of tools.

Unmesh Joshi: Oh, yes. Yes.

James Lewis: Which are useful? I mean, that's provided me with hours of entertainment over the years.

Unmesh Joshi: I think, I mean, just to compare the patterns approach. So Jepsen, I mean, it's an excellent tool and it takes a black box approach. So it treats your products as black boxes, and then it runs some tests and introduces failures in between. And patterns are, I mean, you can think of it probably as a white box approach. So you know what the building blocks look like. So you can maybe think through some of the issues that you might face.

James Lewis: Well, I think, I mean, I think probably you could look at the failures that Jenkins detects, and then you could describe them in terms of failures as a particular implementation of a pattern.

Unmesh Joshi: Yes.

James Lewis: I guess that comes back to the actual implementation. So it comes back to the fact that the book is rooted in code. It's rooted in actual executable code, is unit-testable, and you can inspect it yourself. I think that's a really important part of the book, isn't it?

Unmesh Joshi: That is the reason, I think, so before writing each of these patterns, I had working Java code. Java is a language I chose, I mean, just because it's more people familiar with it, and it's a simple language. There is no other reason.

James Lewis: Even the C-Sharp people are familiar with Java. Haha.

Unmesh Joshi: I made sure that I explained the pattern around this basic Java implementation that's out there and tried not to use pseudo-code to explain the code structure.

Recommended talk: Designing A Data-Intensive Future: Expert Talk • Martin Kleppmann & Jesse Anderson • GOTO 2023

Why Are Patterns Useful?

James Lewis: Maybe we should maybe chug up a little bit now because, I mean, we're in the wheeze, we're talking about clocks, we're talking about...you know, not in the weeds, we're in the details, right?

Unmesh Joshi: Yes.

James Lewis: Which, to me, is the fascinating part of all this. But I guess we can maybe take a step back because, you know, I think many people working in our industry at the moment, right, are coming across the distributed systems more and more, whether they're using them, whether they're, like yourself, involved in implementing them. I mean, you know, whether that's for telescopes or whether that's for point-of-sale solutions or, you know, whatever it is in the always-on world that we're in. How would you sort of recommend...what would you recommend to someone interested in going a little bit more deeply into this? How would you, sort of, recommend them to proceed to go forward?

Unmesh Joshi: One of the goals for these patterns work as well is that someone reading this material should be able to navigate open-source code bases. And I think navigating, I mean, following open-source products and going through their code bases, trying out, maybe isolating some pieces of it and playing with it, I think that's a great way to learn about distribution. I mean, particularly topics like distributed systems, it's a great way to learn because if you go down the route of theoretical understanding, I mean, there are a lot of nice books, I would say, but you might get lost. So I think it helps a lot to remain closer to code while you are learning stuff.

James Lewis: One of the things I kind of, again, really like about this is it opens up a whole new set of patterns to a whole new generation of people, I think, right? So, because as you say because they are patterns, because there is code there, it gives us another set of tools, or rather it exposes the tools that have been there, but that have been used by systems engineers generally, right? And I think that is...obviously, my background is in, I would say, sort of, distributed systems in terms of microservices and that kind of thing. But not...I mean, you know, myself, I've implemented some of these patterns myself, but we are probably in the minority, right? I think what's useful about this is it's opening these patterns up to a much wider audience and making them available.

Unmesh Joshi: Absolutely, I would say. So like something as widely known, but not well understood as a two-phase commit, I think.

James Lewis: Starbucks does anyway,..

Unmesh Joshi: Yeah, so I have documented a two-phase commit in here as well and some of the nitty-gritty of what might go wrong and how it's solved in modern implementations. And knowing something like that, I mean, these scenarios come in your microservices implementations as well. I mean, at a different level of abstraction, obviously, but...

Unmesh Joshi: Or things like an idempotent receiver. I mean, I have a pattern called an idempotent receiver and it's...I think in any microservices implementation, you will need that. And getting some clues and ideas of how, let's say, Kafka and HCD implement this idempotency, opens up, I think...

James Lewis: It opens up a whole new set of approaches because, I mean, we talk about idempotency a lot, but then people sort of...it's one of those things that people forget to implement and then go, "Oh, but there was this thing I forgot, it's called idempotency."

Unmesh Joshi: Oh, yeah. Absolutely.

Recommended talk: Elixir's Impact: How Flow Works & Other Curiosities • James Lewis • YOW! 2023

Conclusion

James Lewis: Okay, so I guess we should, sort of, think about what's next. One question I had, what have you found most surprising while going on this journey?

Unmesh Joshi: Oh, what's so surprising?

James Lewis: What's the most surprising thing that you've, sort of discovered?

Unmesh Joshi: I think one of the most surprising things is, I mean, all these things are out there so many years and there are so many implementations now, at least, but they are not...I mean, I was surprised that these are not well-known things. And like Paxos, for example, taught in academia for 20-plus years and still, you required a new Ph.D. thesis to document something called Raft. Even before that, there was a viewstamped replication algorithm, which is exactly like Raft, just no one knew about it. I mean, it's so surprising that as an industry, I think, I mean, these things were not well-known for some reason.

James Lewis: Well, thank you very much. So what's next now that the book is out? Congratulations, by the way, I think it's destined to become an instant classic. So, I mean, I've enjoyed reading it. It's certainly just, like, widened my eyes, opened my eyes to certain things because it's so practical because it's so readable and so accessible. So I think it's a really important contribution you've made. I wanted to thank you. But what's next?

Unmesh Joshi: I think just going back through some of the patterns, I mean, I see a lot of scope for improvement on some of those. Maybe we'll take, like, a stab at improving some of these patterns. And also, I will, I think, continue...I did a couple of workshops teaching Paxos and Raft and stuff like that. So I'm planning on extending that, I mean, have more workshops where people can play with code and try out some of these things.

James Lewis: And are you planning on...I should mention that the original repository of these patterns was on martinfowler.com, or a lot of them. Are you planning on continuing that work as well?

Unmesh Joshi: I think I will continue on that. And I wish I get time to continue on that.

James Lewis: We're looking out for you in a workshop near you. And I guess, yeah, I mean, go get the book, everyone. It's brilliant, it genuinely is. So I just want to say thank you so much, Unmesh, for joining us at GOTO Book Club to talk about patterns of distributed systems. As I say, I think it's destined to be an instant classic and is full of, like, just fascinating detail on how the things that run our world work, actually, right, because that's really what we're talking about here. So thank you very much for joining us.

Unmesh Joshi: Thank you, James, for giving me this opportunity.

James Lewis: No, it's fantastic for me to get the opportunity to talk to you. So, thank you to GOTO and the GOTO Book Club for organizing this. You can see this. Hopefully, you'll enjoy it. Thank you very much.

Unmesh Joshi: Thank you.