Software Architecture: The Hard Parts

There are no easy decisions in software architecture. Instead, there are many hard parts--difficult problems or issues with no best practices--that force you to choose among various compromises. With this book, you'll learn how to think critically about the trade-offs involved with distributed architectures.

Architecture veterans and practicing consultants Neal Ford, Mark Richards, Pramod Sadalage, and Zhamak Dehghani discuss strategies for choosing an appropriate architecture. By interweaving a story about a fictional group of technology professionals--the Sysops Squad--they examine everything from how to determine service granularity, manage workflows and orchestration, manage and decouple contracts, and manage distributed transactions to how to optimize operational characteristics, such as scalability, elasticity, and performance.

By focusing on commonly asked questions, this book provides techniques to help you discover and weigh the trade-offs as you confront the issues you face as an architect.

• Analyze trade-offs and effectively document your decisions
• Make better decisions regarding service granularity
• Understand the complexities of breaking apart monolithic applications
• Manage and decouple contracts between services
• Handle data in a highly distributed architecture
• Learn patterns to manage workflow and transactions when breaking apart applications

* Book description: © O’Reilly

Intro

Neal Ford: Hello everyone. My name is Neal Ford, and I am joined today by my frequent collaborator and co-author, Mark Richards.

Mark Richards: Hey, everyone. My name is Mark Richards.

Neal Ford: And we are here today to talk primarily about the...but we can never restrict ourselves to one set of topics, but primarily about our most recent book, "Fundamentals of Software Architecture?" And it turns out it was not at all. When we were writing the Fundamentals book, we kept bumping into these really difficult problems, and as we started diving into them, we started saying, "No, that's too hard." And at one point, Mark sort of jokingly said, "What?"

Mark Richards: We had so much additional material and I said, "This really is all the hard parts, and we should have that as the title of our next book." And sure enough, it became the title of our next book.

Neal Ford: Well, and so "Architecture, The Hard Parts" became sort of our get-out-of-jail-free card for really complex problems. It's like, "Okay, let's put that over in the hard parts pile." As we were writing that book, that pile got bigger and bigger and bigger. When we published the Fundamentals book, we started thinking, "Well, okay, what next book should we write?" We looked at that giant pile of difficult problems and thought, "Okay, well, let's poke around and see what's making those things so difficult." And we quickly arrived at...

There were a lot of difficult things there, but it seemed like the lion's share of the difficult things had to do with distributed architectures like microservices, which is the common cool kit architecture right now. But there are a lot of challenges inherent in distributed architectures. And as we started poking around in those problems, we discovered that a lot of it came down to coupling and actually started separating the concept of coupling into two different kinds of coupling.

Mark Richards: Yes. But as a matter of fact, the interesting thing is... What would you think? You know, I bet it was almost close to seven years, all total, from the first book that we wrote. Neal and I were together, and Neal said, "You know, it's a shame that there's not a great comprehensive book about software architecture that covers all of the techniques, the technologies, and also the soft skills piece." And both Neal and I... Well, Neal, I think I've known you now for almost 16 years, I think almost 18 years.

Neal Ford: A long time.

Mark Richards: It's been a very long time. And actually, we met at a conference in the United States that was kind of like a traveling road show. We would go to cities twice a month and give all of our sessions, our material. And it turns out that Neal and I were two of the only speakers who actually touched on architecture talks as opposed to these really deep technology ones. Now, we also did a lot of deep technology talks, but we also talked about software architecture. And that started to grow. So Neal and I started to collaborate a little more.

Then finally, about seven years before the Fundamentals book was published, Neal came up with this idea. He said, "You know, we ought to do a video. We ought to put some material together and start getting this stuff out and collected." And that's what ended up being the Fundamentals book.

But my point of all that was we both had so much fun writing the "Fundamentals of Software Architecture," and it worked so well with the way we both came at architecture with a different perspective that we both knew. Okay, we gotta write another book but we weren't sure what it was. And that's how all that material that we started to say, can't put that in the Fundamentals book, started to evolve into the Hard Parts. But Neal came up with a fantastic way of organizing that material, and that is how our book ended up being structured. 

Organizing the concepts: the rubber band

Mark Richards: Do you wanna actually talk about the concepts that you have, this rubber band?

Neal Ford: Well, he actually touched on one of the things that we also sort of stumbled upon, but have taken great advantage of, which is this idea of iterating on design in both architecture but on IP as well. Because when we started talking about software architecture, there was just too much of it. 

As you teach training classes, as you do more writing about it, as you do videos, as you start refining and realize, you know what really needs to be there. And that's really this kind of iterative approach to IP. And we realized that same approach would work with the Hard Parts material, even though there was less sort of inherent structure in it. And so, as soon as we had this giant pile of examples, we thought, we need some way to unify this. And one of the first things we did was a two-day workshop at one of the O'Reilly software architecture conferences.

We needed some way to kind of organize this material. And many years ago, I wrote a book called "Presentation Patterns." And one of the patterns in there, we called the Triad pattern. Which is when you're in doubt about how to organize stuff, organize it in threes, because people, you know, think in chunks of threes and three main ideas away from a talk. And there's this famous quote, I think it's Larry Constantine about, it's basically this idea that it seems like there's this force in software architecture. We keep breaking things down into smaller and smaller pieces until we get pieces that are so small, they're not useful anymore, and then we have to start gluing them back together to make useful work. And so I thought, well, it seems like there's this constant pendulum swing in the software architecture of, "We gotta bring things down more." But then they're broken down, it's like, "We gotta put them back together."

And so this idea of pulling things apart and then putting them back together. Part one was pulling things apart. Part two was putting things back together. And then part three was with simple pieces and simple patterns. So trying to embrace that idea of simplicity. And then, at some point, after iterating on that a few times, we realized that the third part wasn't really needed. What we really needed was the first two parts. And that became actually the macrostructure of our book. That really nicely fell into, sort of, part one, pulling things apart was about how static coupling works, how things are wired together in software architecture. And part two was about dynamic coupling, which is once you've got these things, how do they talk to one another? Which turned out to be a really, really difficult, interesting problem.

But the interesting side effect of that was we started with this kind of three-part thing and then ended up with this two-part thing. Then we sort of forgot that existed. We just wrote. And we wrote the natural material for part one and part two and got it all together. And it was during the copy editing phase that I noticed that out of a 450-page book, part 2 started on the page, I think it was 221. So it was almost exactly the midway point. That was not intentional at all. But we ended up writing almost exactly equal parts around the kind of static coupling wiring versus dynamic…

Recommended talk: Is Domain-Driven Design Overrated? • Stefan Tilkov • GOTO 2021

Mark Richards: And that was amazing how that kind of happened like that. But this was something Neal originally kind of proposed about these pulling things apart, putting them back together. This was so clever because if you think about decomposing systems into a level of modularity, breaking those things apart. Neal just hit on this play of words of putting them back together. He's like, "Wait, what do you mean? We just spent all this effort tearing apart and breaking apart our systems. What do you mean, now you want us to put them back together?"

And he's like, "Well, how are you communicating? What do those contracts look like? How are you doing distributed transactions? How are you accessing data you no longer own? How are you doing distributed workflows and transactional sagas?" And all of a sudden, you realize, "Oh, right, you have to interconnect everything." Which is just such a clever way of saying you gotta put things back together.

Writing the book 

Mark Richards: But you know what, Neal? You know what would be really interesting to talk about in this kind of book club is our process of writing. I mean, how we wrote this book because we have a very unique way of actually writing a book, which actually is all of Neal's ideas. I just stole them. But it was really cool. And I think knowing this kind of little Meta background about how we went about writing this book might be kind of interesting for any of you out there who are actually reading it or kind of curious about that.

Neal Ford: Well, this goes back to... And I end up advising a lot of new writers who think they have a book idea and want to talk to someone who's written a book before. I have a lot of colleagues who are like that, and they schedule these calls with me and they think I'm going to talk about choosing publishers and arcane things about royalties and all that, which is mostly not that interesting in detail.

But what I end up talking about is something that just never really occurred to them is that writing a book can be an iterative process, just like designing software can be an iterative process. There are a couple of different ways you can write a book. One of them is to go to a quiet, beautiful nest and close yourself away from sound and noise and distractions and sit and think deep thoughts. Well...

Mark Richards: That place exists?

Neal Ford: Only in the mythical mind of people who think that's what writers do, and can afford to do that. Particularly technical writers who are not writing as a full-time job. It's always, you know, an extra job. But then your best shot is you release some first draft that goes up to a limited pool of reviewers and then you revise that and you publish it. That's really still kind of a first draft of an idea. And it's not until the second or third edition of that book, if it's successful enough, you get a refinement of those ideas. What we realized, both in training and teaching classes, but also in writing, is you can iterate on ideas. And the first book that I did this with was "Building Evolutionary Architectures," which I wrote with Rebecca Parsons and Pat Kua.

We recently just did a second edition of that book, but we started with this really small seed of an idea. Because we realized when you do a lot of public speaking and you get over the shock of, "Oh, no, I'm standing in front of all these people and they're all paying attention to me." And, you know, worrying about the nervousness of that. You can enter into this rarefied mental state. Because when you're really trying to explain something to someone, you enter into this mental state that you can't really achieve otherwise. You can't do that in your quiet study away from distractions and all that. There's an energy that shows up there and it leads you to better innovations. And so the idea is that you talk about something to get insights, which gives you more to write about, which gives you more to talk about, which gives you more insights.

And before long you've blossomed this IP, the seed, into, you know, an interesting structure. And that's exactly less with the Fundamentals book. Because that was really more of a kind of filtering process of too much material out in the world, how do you refine it? But the Hard Parts book was very much this idea of, okay, the vastness of hard problems in software architecture. Which are the things that we're going to tie together that make sense and are all uniformly difficult? And Distributed Architecture turns out to be the perfect candidate for that because just about everything you do in Distributed Architecture is difficult.

The feedback loop

Mark Richards: That's right. All right. So now, Neal, you made me go on my little soapbox. In Scrum, one of the foundational aspects of Scrum is that feedback loop. I mean, this is...whether Scrum or Agile, it's all about that feedback loop. And we have found since what, the early '90s I guess, late '90s, 2000 was the manifesto, I believe, right? But then Scrum came around. But so we have learned for over two decades that feedback loop in developing software is kind of the secret sauce. That's how we actually get effective software written. And we all see that and we all agree with it.

But then we don't apply that same concept to all other areas of what we do, which is what...like Neal, what you were just talking about with having that feedback loop. Can you imagine if Neal and I spent two weeks hidden away in some dark cave and we wrote the Hard Parts book and then published it? Well, what you are seeing is the very, very, very first rough draft that no one has seen before.

There's no feedback loop. Neal's process, which I just think...I just loved and it really worked, was to say, "No, no, no, no, let's not write the book yet. Let's find out what we want to write about." And that was just pure magic. And that's how we structured both of these books really and then, especially the Hard Parts book. But all right, so just a quick one-minute tangent about feedback because I want to give you another great example of how valuable this whole feedback loop process is.

I can't ask all of you, but I'm sure all of you get performance reviews. How often are those? Every six months, every year. Here's an effective way to change your career because it's the same kind of thing as writing a book or developing software because you get that feedback loop of six months to a year. Can you imagine once a year your performance review and it's like, "Well, I gotta tell you you need some improvement in this area." And what do you reply? "Well, gee, if you would've told me that 12 months ago, I would've been a lot more effective this year. But now it's gonna take me an entire year to be more effective because you just told me my feedback either good or bad." These are examples of how we write the books now, all of our books that we've been writing and wow, it works great. So, just a little Meta kind of story there.

Neal Ford: Well, and that's one of the sorts of benefits of pair programming because you get much faster feedback on how things are going. But a great example of the way iterative design helps books is that when we first started... One of the things that we wanted to do in our Fundamentals book was do a real apples-to-apples comparison between all these different architectural styles, these names, and topologies of architecture. Like how do you really compare a layered monolith with a microkernel or microservices or service-oriented architecture?

Recommended talk: You Must Be Crazy To Do Pair Programming • Dave Farley • GOTO 2022

So, we already come up with a way of comparing those things. And our initial stab at that was for all these different lists of architecture characteristics like performance and scalability and elasticity to deployability, we had a binary that was either a thumbs up or thumbs down. And that wasn't quite enough nuance.

Then we added a third thing which we called the Caesar thumb, which was a sideways thumb, which is neither up nor down. So, that gave us three, but that wasn't quite enough either. We graduated to something that has become a very popular feature of our book, which are these star rating charts, which are five stars. You either get one star or five stars. And that is a great example of iterating on an idea and realizing it didn't have enough depth and so, we need to expand the dimensions that we're talking about so that we have more nuance in our comparison. And we never would've really done that until the fifth or sixth or eighth edition of the book that we were writing in a different way. But getting in front of an audience and eyes and getting that immediate feedback from an audience of, you know, we need more nuance here really, really helps that process.

Trade-Off Analysis and Distributed Architectures

Mark Richards: Hey, you know, a question I get a lot Neal Ford, is we talk a lot about pulling things apart, putting them back together, which you might think is only about migration, I guess. But the question I get a lot is, "Well, can I use the principles in your book, the Hard Parts book, for greenfield applications, or even just even maintaining distributed architectures?" And of course, the answer is, "Yes, all of those things apply." But the problem is, and here's the pushback I get on this, "Yes, but you talk about so many metrics in the book and all of these different ways of analyzing trade-offs." Well, in a greenfield we don't have those metrics. Sometimes we don't even have enough information for the trade-off analysis. So, how do you address those kinds of things?

Recommended talk: When To Use Microservices (And When Not To!) • Sam Newman & Martin Fowler • GOTO 2020

Neal Ford: Well, that's a great question that comes up a lot. And one of the things that we realize is that you can't just write a book about difficult examples in software architecture. I mean, you could but it would be a very unsatisfying book because you'd get to the end of it and it's like, "Great, just a whole bunch more things that are complicated in my world that don't actually teach me anything." And so what we wanna do is do trade-off analysis between all these things. And so, really what our book is, is a whole bunch of examples of, okay, given this complex scenario, how do you trade off analysis? In fact, that's the subtitle of the book is "Trade-Off Analysis and Distributed Architectures." The last chapter of our book really talks about how to apply all the tools that we use throughout the book to do that. And Mark is talking about a very specific one that we end up using all the time, which is really powerful in that situation where you don't have existing...a scenario which is this, what we call qualitative versus quantitative analysis. So, let's say that you have a particular workflow that you need to figure out, what's the correct way to model this workflow.

You know, ultimately, ideally, what you could do is build every possible version of it. So, let's say there are eight different possibilities with what we call the three primal forces. Build all eight of them and measure them in real-world scenarios. And you'd know deterministically, exactly which one is the best one, and then move forward. Good luck going to your boss and telling them we need eight times as long to build something that we normally take...because we wanna do this deep, deep analysis. But the problem is nobody's really built this thing before so how do you know? Well, that's where we go to... That's quantitative analysis, actually building and measuring the thing. And that's the Rolls Royce because there's no doubt, there's no speculation in that. But you don't have the luxury of doing that a lot.

So, we do a lot of qualitative analysis, which is okay, I don't know the hard numbers in this scenario but if you look at, for example, synchronous versus asynchronous communication. I know that asynchronous communication will give me higher levels of scalability potentially than synchronous depending on implementation details. You can mess it up and make it worse. But, in general, from a principal standpoint, this will make it better. That allows you to do qualitative analysis and build matrices and actually look at...narrow down candidates until you find the few that you can actually implement and, you know, do some experiments with that to see what the real measurements are. And then we encourage you, once you've actually implemented it, actually measure all those things. And this is actually a little bit of a side discussion about "Building Evolution Architecture's" second edition.

We have an entire section of that book... So one of the things that we did after the first edition was soliciting all of our coworkers and say, "Give us examples of how you've used these fitness functions, these ways of validating architecture characteristics in the wild." And this whole category that came back that was fascinating to us, was using fitness functions as an experimental medium. Because we've never built this before. We're doing this intake from a cloud, from this mainframe, and nobody on earth has ever built this combination of technologies.

We don't know what the initial value should be. So let's take a stab at it, but immediately start measuring it and see what those are. That qualitative analysis gets you to that first stab. But then you immediately need to start measuring it to make sure... Because it turns out that in many cases, we underestimated by a lot because of some factor we didn't take into account. So it's really good to use this as a way to gradually get to more and more concrete engineering-like measures.

Turning abstract concepts into concrete examples

Mark Richards: Hey, you just triggered a thought in my mind, Neal. Do you know what would be kind of fun to talk about? Other than the structure of the Sysops Squad. We did a really unique thing in this book. We kind of took this idea, well, I guess borrowed it maybe from like the Phoenix project and stuff but thought this book as we were writing it was so abstract. And one of the challenges...and any kind of technical book, especially architecture is how to take something that's abstract and make it concrete. And so we both said, "Yes, we need an example." But when we started that out it was kind of boring and we thought, "Hey, wait a minute, why not tell some stories? Why not have some conversations to really set the stage for every one of these abstract concepts?" Let's talk about the Sysops Squad because that was actually a unique aspect of our book that I really love. And as a matter of fact, that was fun writing those conversations, wasn't it?

Neal Ford: Yes. And it evolved really interesting because we realized that...and one of the things that we said was the Hard Parts book could either be a 120-page book, 450-page book, or 20,000-page book. A 100-page book would be all just abstract things and it would be completely unreadable because there's just nothing concrete in it. A 20,000-page book would be every abstract idea taken to its logical conclusion with the combination of all the other abstract ideas. And that's unreadable because nobody would ever read through it...

Mark Richards: That's right.

Neal Ford: ...all those things. And so, we fell upon the 450-page version of this book, which was...It started out just like in our Fundamentals book we had...we wanted to sprinkle in some concrete examples to illustrate the things that we were talking about. But we realized, unlike the Fundamentals book, this is not an occasional thing. This has to happen a lot. Because we keep talking about abstract things and trade-offs. And with trade-offs, it's okay to have matrices, but then until you see the real-world examples of those trade-offs, it's really hard to see how they play against one another.

We started with individual examples that were sometimes related and sometimes not. But this is where the training classes came in. Because that's where the entire idea... We like to do hands-on training classes and we were inspired by this idea from now 15 years ago, besides that architecture caught us. Which are little hands-on architecture exercises, lines and boxes, and that kinda stuff for hands-on classes. For a lot of the hard course material, we wanted it to be... concrete examples we used a kata called the Sysops Squad. Do you wanna describe the Sysops Squad?

Mark Richards: Yes. So the problem kind of in all the different architecture katas, these small targeted problems, we found one that was most interesting from a distributed architecture standpoint and had some very interesting challenges and problems, was in fact this architecture kata called the Sysops Squad. It's a problem ticket system for a large electronics giant.

And we kind of started out with these little conversations that happened. But you know what evolved through the writing of this book was to say, "We should have a storyline. We should actually make this a story." And in fact, we started interconnecting all these different players, all these actors, the people in the Sysops Squad, all the IT folks. And well, it took a while to realign everything because we weren't even sure of the order of the chapters. But what evolved was this really interesting story about the Sysops Squad.

Every chapter starts with a conversation. Neal put in so many Easter eggs. Now we're not going to tell you what those Easter eggs are. Oh, I still want to but no, we want all of you to try to find them but just a little kind of nuances about different aspects of those stories. But the one thing I did want to tell everybody, please read those, it's part of the story. But we drew those conversations and scenarios actually after real client experiences.

Some of those meetings I was actually in. So no, you'll never be able to tell the client, but let's call it the Sysops Squad. Anyways, it was such a great way, a clever way, of being able to bring in a concrete example instead of the dry piece of, "Okay, now let's show you some abstract concepts. Let's apply it to this scenario, in this scenario." No, they're all interconnected to tell a story. It's a really cool way to...

Neal Ford: The concrete examples are not dog and cat and pony, they're actually a real working system. And we actually build an entire narrative around, okay, we have this monolithic application that's starting to show the exact kind of problems that, in our experience, monolithic applications tend to have. How do you work through all of the issues of how to pull things apart and then put them back together? So, it turned out to be great fodder.

The interesting thing about actually humanizing the story, is so you'll actually see a font break. The Sysops Squad starts every chapter, and then they're sometimes interspersed in the chapters to show examples. But for the first time in any of our books, Mark and I had to write dialogue, which is really, really challenging. In fact, we got a professional writer to judge our dialogue and give us some really sorely needed tips, making it more convincing and less awful. So, that...

Mark Richards: That was an appreciative piece about writing a technical book and the kind of voice that you write a technical book in. And then switching the voice and having to actually write prose, like an actual story, like a novel. And it's much different.

Neal Ford: Well, and in fact, when we were writing, we would talk about  the Sysops squad voice versus the narrator's voice. So, then I could literally, as I was reading the prose hear the narrator's voice come in after the characters are talking, like the deep voice and everything. And we did a clever thing since this is for a book club...a little interesting writer's hack that we ended up doing, which ended up really taking great advantage of the platform we wrote it on.

We actually wrote this, it was through O'Reilly and they have a GitHub or git-based, not GitHub, git-based platform for writing books that use AsciiDoc. And AsciiDoc has the ability to do includes. And so, we had all these Sysops Squad stories interspersed throughout the book, but then we also wanted them to be an actual linear narrative, something happens here and something happens later here.

It was hard because some of them are interwoven within the chapters. It was hard to look at the book and just see those. And so we came up with the idea of, let's do all of the Sysops Squad stories as imports, includes in the main text. And that allowed us to completely separate those. But the other nice thing about that was, we built a synthetic chapter, the last chapter of the book, a little bit of build code in Rake, which is my preferred build language.

I wrote this but would actually just go take all the Sysops Squad stories and put them in the order that they appeared in the book as one single chapter at the end, so you could just read the Sysops story from front to back independent of where it was within the book. That was really cool because that helped us get the narrative and make sure the story was stitched together and made sense.

Mark Richards: Well, especially since there were multiple authors of those dialogues to make sure we were consistent, and that the personality of each character was consistent throughout. But these are things novel authors do, and they spend years and years analyzing the character's personality. Well, you know, we didn't quite go that deep, but we did at least make sure they were consistent and accurate to the conversation, what they said, the type of way they said something. It was actually kind of fun. It was almost like we wrote a novel, in a way.

Neal Ford: No. And you'll definitely see that at least one of our Sysops Squad members was a fan of very, very extreme sports. And so there's a lot of little personality quotes like that interspersed to break up the... And, you know, it also serves to break up... You know, software architecture stuff is relentlessly abstract because it's a bunch of very complex subjects. So, having a little bit of conversation and dialogue, you know, breaks up the denseness of that in a way that I think works well in a book that abstract.

Mark Richards: Hey, you know what? I know it's a book club, and I know hopefully everybody's reading the book, but Neal made a really, really important discovery while writing this book. As a matter of fact, it was towards, I would say the last 80% of the book. And it was amazing. So, Neal pinged me and said, "Hey, you got some time for a quick Zoom call?" I said, "Sure."

So I put away everything and we hopped on a Zoom call and Neal said, "Take a look at this." He said, "You know how we have things like communication whether it's sync or async?" "Yep. We have a whole chapter on that." "Workflows, orchestration of choreography." "Yep. We have a whole chapter on that. And we've been writing all these things." And Neal said, "And then kind of the coordination of things, whether we're trying to do eventual consistency or simulate atomic transactions."

And I said, "Yep." He said, "You know..." He said, "I think I've made a discovery here." He said, "You know, we treat these things as an individual, but I don't think we can." And so on the Zoom call, he's sketching out this three-dimensional plane. Remember the poorly drawn three-dimensional diagram on Zoom. And he is like, "Take a look at this." And I'm like, "Oh my gosh, Neal, I think you're onto something here."

So, that call lasted, I don't know, an hour and a half. And then we hopped on another call the next day and the next day, and we kept refining this. He's like, "I think you're onto something here, that these three forces that we've always treated as individual things, individual decisions." Neal discovered, through writing the book, they're all intertwined. And it was an amazing discovery of actually how to design distributed systems. And Neal always downplays this and...you know you always do, but I think this is just such an important discovery that kind of evolved through writing the book. And I think it's a significant part of it.

Recommended talk: Balancing Choreography and Orchestration • Bernd Rücker • GOTO 2020

Neal Ford: Well, and literally the insight that I had was, we were teaching an online class and I was listening to you describe orchestration versus choreography for, I'm guessing the 48th or 49th, or 63rd or 95th time. And it's literally just something that's kind of going in the background, but it's something I've been thinking about a lot. Because we were in the middle of writing this book and we were looking for the thing that tied the second part together. Because we had all these orchestration versus choreography examples, but there was nothing that really kind of...

Mark Richards: Tied it together.

Neal Ford: ...tied it together in a way that created a narrative around why these things matter...why are these contributing to the hard parts? There was nothing that kind of tied them all together. And that's when I realized that... And I think the reason this is a useful insight is that we have so long created those things as "How do you make things faster? "Go from sync to async."

Mark Richards: Make it async.

Neal Ford: Don't think anything about the other side effects that is going to have...

Mark Richards: The other forces at play.

Transactional sagas

Neal Ford: ...but those other forces are tangled up and extract and exhibit gravity. When you make a decision like that the ball moves in places that you might not want it to in terms of some of the characteristics you're interested in. 

What that really ends up doing, and this is sort of a valuable side effect of that, is that tied with this kind of qualitative analysis allows you to do a comprehensive analysis of a space like that.

So, if you take those three dimensions that he talked about, which are communication, consistency and coordination, and treat them as binaries, you have eight possible combinations. And so we actually model all eight of them isomorphically, as generic services, and looked at which one has better scalability. And once you go through that exercise, it gives you a feel for what forces really are at play and how things interact with each other. For example, your observation about orchestration and...

Mark Richards: The fact that you say, for example, "Orchestration is bad, you shouldn't use it in microservices." Well, we have ratings for, let's see, responsiveness, scalability consistency or no, complexity and...

Neal Ford: Coupling.

Mark Richards: ...coupling. And look at how bad it is. But then you look at another transactional saga that has orchestration and it's almost all green. And there are so many of these that really...you should call this some sort of Neal's law, or you should actually do some sort of theory like, you know, like cap theorem, this is Neal Ford's theorem. But anyways, there are so many ways that it demonstrates and proves this idea. But two really funny anecdotes about these transactional sagas. First of all, as Neal Ford said, we did some isomorphic diagrams, we analyzed them, we went back and forth with all these ratings, kind of like star ratings, that these are red, green bubbles.

And Neal's like, "You know, we have to name these." Well, a saga is a story. A transactional saga is a story about how a request goes through your system. And so Neal said, "Well, these all ought to have names that have to do with stories." Wasn't that fun doing this? Okay. So, we came up with one that Neal termed "the horror story." It's the worst and the scariest possible story you can ever see. And here's the anecdotes. Okay, so anecdote number one. Neal's like, "Well, you know, Mark, we've got all these things, we analyze them but we shouldn't include all of these in the book because some of these are gonna drop off." Like, in other words, "Yes, this is a combination of all three of those forces, but no one would ever do these." He's like, "Okay." So, guess what happened? Exactly two weeks after naming the horror story, the worst possible combination of these primal forces, what happens, Neal?

Neal Ford: The project I was working on, the Arctics, suggested exactly the horror story. And it is the combination of asynchronous communication choreography, but atomic transactionality and... It's obvious how they got there because they said, "Okay, super high performance, super scale, so we need asynchronous choreography, oh, but the business says it needs to be transactional." Congratulations, you've stumbled right into the horror story.

And I explained to them how hard it was to get that right because all of the powers that you have to make that easier are gone. You don't have an orchestrator, you don't have synchronous communication. So, all these bad things are happening all at the same time, and you're trying to atomically coordinate all this stuff. It's just like, "Is there...?"

Mark Richards: Then your head explodes.

Recommended talk: Beyond Microservices: Streams, State and Scalability • Gwen Shapira • GOTO 2020&

Neal Ford: Exactly. And so, fortunately, we talked them out of that, but it really convinced us that all of these things are legitimate patterns or antipatterns that show up in the wild.

Mark Richards: And that we have seen them in the wild. So it turns out that we did include all eight of these in the book, because you may stumble upon these, but it's so, so beautiful. If you want to see the best example, go to the epic saga that we describe in the book and take a look at this combination: atomic, synchronous, and orchestrated. Well, it's really slow and you'll see why when you take a look at the error handling. But how do you make something faster?

Neal Ford: Asynchronous.

Mark Richards: Asynchronous. So, with the best of intentions the team says, "Oh, yeah, this is too slow. Make it async. That'll speed it up." And it turns out that turns into what we called the fantasy fiction saga. A very complex story of world-building that's really hard to believe all the way through it and even at the end. Well, it was the best intention of making something async but it doesn't change the responsiveness.

We analyzed this and marked it with a solid red dot. So, isn't it interesting, this really again, demonstrates this theory that none of these forces individually will make a change? It's all three of these combinations that allow you to have better responsiveness. Pure magic. It was just, in my opinion, humbly Neal, it was the greatest discovery I think we made in that book.

Conclusion

Neal Ford: You might be biased. So, let's wrap things up, and we'll give you another little bit of a writer example of one of the things that we did in the Hard Parts book. We were very careful about the prose in the Hard Parts book because we were writing in it during a pandemic, and we had full control over all of the prose. And if you look at the first 14 chapters of that book, they're all either in the third person or first person.

The last chapter of the book is the one that's called "Build your Own Trade-Off Analysis." And it's where we consolidate all the techniques that we've used to illustrate all these trade-offs throughout the book. And we purposely switched that to the second person, because we're trying to now directly address the reader to say, "Okay, now it's your turn to actually go and apply some of these things."

That really was the narrative thrust of the book and kept it from being just a collection of difficult examples...ultimately, it was about, as our subtitle says, a trade-off analysis, how to do trade-off analysis. And wrapping up that way was a great way to summarize that. Hopefully, I give you some insight into the structure of the book and why it exists.

Mark Richards: That's right. And listen, you know you don't have to be an architect to read this book. As a matter of fact, many of the decisions a developer makes almost every day, a developer team, are in this book. Having to choose what kind of contract, having to choose orchestration choreography. A lot of times it's development teams that make this. So, I think that's kinda cool that the audience for this book is actually pretty broad. Software developer, architect, and tech lead. I'd go as far as to say even product ownership because there are aspects that we talk about here of understanding why teams make certain decisions or struggle with something. 

Neal Ford: And it really illustrates... So in our Fundamentals book, we coined our two laws of software architecture. The first one is that everything in software architecture is a trade-off. And that's exactly what Mark is talking about. You don't have to be an architect to face significant trade-offs and decisions that you make. But the Hard Parts book really tries to exemplify the second law which is the why is much more important than the how. So, seeing how something's implemented is okay but understanding why you chose that and, you know, how to make those decisions is, I think, really important. And that's really kind of what we were striving for in the Hard Parts book.

Mark Richards: Yes.

Neal Ford: All right.

Mark Richards: This has been awesome,

Neal Ford: All right. Fantastic. Thanks so much for joining us. Hope you enjoyed it. And hope you get a chance to read one of our books and to come back for a future installment of the book club.

Mark Richards: Yeah, awesome.

Neal Ford: Again, my name is Neal Ford.

Mark Richards: And Mark Richards. And enjoy reading our books and have fun.

Neal Ford: Thanks.