The Value Flywheel Effect: A Serverless First Strategy at Liberty Mutual

"The Value Flywheel Effect"

Charles Humble: Hello, and welcome to this episode of the "GOTO Podcast." I'm Charles Humble. I'm a freelance techy, editor, author, and consultant. And this is the fourth episode in a mini-series of podcasts that I'm doing for GOTO, talking to software engineering leaders. I'm aiming for each episode to have actionable insights and suggestions for further research, such as books and papers to read, conference talks to watch, and so on. When I was working at Container Solutions, I hosted a podcast called "Hacking the Org," and interviewed Adrian Cockcroft, amongst others. And in the course of my conversation with Adrian, he asked if I'd read "The Value Flywheel Effect." I hadn't, but I went out, bought it, and read it on the strength of his recommendation. And I'm really glad I did, because it's honestly one of the best textbooks I've read in years. So it's a pass on that favor. If you haven't read it, I really recommend you do. And I'm thrilled to have one of the co-authors, David Anderson, on the show today.

David has been at the leading edge of the technology industry for 25 years. He formed the service edge and continues to work with clients and partners to prove out the thinking in "The Value Flywheel Effect" book. The book is largely based on his experience at Liberty IT, where he worked for 14 years, including 7 as Director of Technology. He is now an architect at G-P, Globalization Partners, and is also a member of the Wardley Mapping community. David Anderson, welcome to the show.

David Anderson: Thank you, Charles. Great to be here.

Charles Humble: It's wonderful to have you on. So can you give us a sense of the size of the team you have within Liberty IT in Belfast, and your position within that, and maybe the role your team played more widely within the company?

David Anderson: Yes, great question. Firstly, we were architects. You know, I was the Director of Technology or sort of CTO. Liberty IT in Belfast, it's a kind of a software engineering specialty, kind of Department of Liberty Mutual, the fortune 100 insurance company. So there's maybe like a 6000, 7000 person IT department within Liberty Mutual, and probably 10% of that is in Belfast and in Ireland. So my role was almost like the technical leader, like the CTO of the organization. So I had a small architecture team of maybe around 12 to 14 Architects. And really, we worked across all the leading-edge projects in the Liberty Mutual enterprise globally. So we were always at the cutting edge of whatever big advancement was kind of happening.

And probably the role we played was probably kind of cattle wrangling. You know, how do we drive change and ensure best practice. And it's very much not, I would say, an ivory tower architect, but more like an enabling architect. I base the model on Gregor Hohpe, the elevator architect, that we could go with the teams, sit shoulder to shoulder and build with them, or pop into the executive team and explain why a certain direction is important, and every floor in between. So that was really important. Not only having the sensing engine, recognize change in the organization, but having the experience to ship that change, and then the skills to make it real within the teams. So lucky enough that we were in that position, and it was a brilliant experience.

Adoption of Serverless-First Approach

Charles Humble: Liberty moved to cloud around 2013. And I think that was the catalyst for you starting to think about how you could build software differently. So can you talk about that? What were some of the things that were influencing you? And how did you settle on the serverless first approach you ended up taking?

David Anderson: We had experimented in, you know, private cloud, virtual cloud, all the various variations over back 15 years ago. And around 2013, 2014, we actually sat down with AWS, and said, okay, we're going to do this seriously now for public load. And I was lucky enough to be at that table, as one of the technical leaders, from an enterprise perspective. My peers were, like, network specialists, security specialists, and I was the only real software guy. And it struck me that we need to build differently. We can't keep doing what we've been doing, because this is a complete paradigm shift. So, one of my peers, a guy called Ed Carmody, used to always say, "Reduce undifferentiated, heavy lifting. Like, what do we not need to do?" So that was something that kind of stuck in, you know, that's how we need to think. So we started thinking about how we can offload work to the vendor, not just compute, like a whole bunch of stuff. How can we make them, you know, give us building blocks, that we can move faster.

So we started to experiment with everything that was around at the time. And we were lucky enough to be at re:Invent when Lambda was launched. And that idea of serverless was just a game changer. So it was a lot of thinking at that time, 2015, 2016, around serverless first, and starting at that technology and working your way back to containers, and back to other technologies. I mean, there's no point where a team would write a logging service, like, we're an insurance company, where are you writing the login service for? Let's just use something else. So trying to get builders not to build was really the goal. And let's be high up in the value stream, build systems for our business and not worry about building low level components that we're not good at. So that was really the catalyst. And then what we really did was start experimenting and getting quick wins. Because the technology was very early back in '14, '15. So really trying to understand how it worked. So there's probably two things going on, understanding the technology, and then convincing our peers that we can actually build software differently.

Recommended talk: Insights on Leadership & Innovation • Gene Kim & Charles Humble • GOTO 2024

Charles Humble: How did you come to write the book for the IT Revolution? I assume you had a meeting with Gene Kim at some point. How did that go?

David Anderson: That was funny. Well, serverless was still very kind of funny. And the whole idea of serverless first is still not very well understood. It's not just about functions and Lambda, it's just about the higher plane of how you write software. So we flashed out and kind of worked out this way of working, which was starting to really get results. Like, we had things where we would take a $20 piece of work and then register for a cent. So there's massive cost reductions, massive speed. So we've been doing this and driving massive whole scale changes within the organization for several years. And then it was actually Adrian Cockcroft, as you mentioned at the start of the conversation. I met him a few years ago. I was kind of sharing with him what we were doing. This was our kind of playbook, our approach. And I thought to myself... And I have massive respect for Adrian. I was looking for some constructive criticism. And he said, "Wow," He said, "That's really impressive." And I almost fell off my chair.

So we started working... He was like, "I want to understand how you did this at a 100-year-old insurance company." So we spent a lot of time working with him and became good friends. And then he said to me, when I moved on from Liberty, he said, "You have to write a book." I was like, "Oh, right. Okay." I had no intention. So I started writing. And he says, like, "This is pretty good. You should keep going." So he connected me with a few publishers. And I met Gene Kim. And I was always a massive fan of the IT Revolution for many years. I used to go to the DevOps Enterprise Summit. And then I was thinking, okay, going to the father of DevOps, Gene Kim, and saying, okay, well, this serverless thing, we don't need a bunch of, you know, the lower-level stuff. But the first thing he said was, "I don't think this is for us." And he said, "But the Wardley Mapping thing is interesting." And so he said, "I'd love to do a Wardley Map someday." And I was like, "Okay." I was like, "Why don't we do one now" We were sitting in a Zoom call.

So I mapped out his strategy for the Enterprise Summit, and gave him a couple of key learnings, and helped him understand. And he just went, "Wow." You know, Gene went, "Holy cow." He says, "You've just encapsulated everything in my brain in a single diagram in 20 minutes." He says, "That's incredible." And right there on the spot, he says, "We got to publish this book because this technique is too important, this way of thinking is too important." So then we started a brilliant partnership with IT Revolution. And it's such a good company. There's such a brilliant amount of learning. And what they kinda say is, they go to practitioners and help them write books, as opposed to going to authors and publishing books. So I thought that was perfect. And such a good body of work. I mean, it's almost like an NBA in their back history of books by themselves.

Practical Application of Wardley Mapping

Charles Humble: I think they're phenomenal. And I love that story, actually. I'm also a huge fan of Wardley Mapping. And it's something I used quite a bit when I was at C4Media, when I was InfoQ's Chief Editor. I found it incredibly helpful. But I also found it's really hard to get across to people kind of how it works and why it works. And really, I think from my experience, the only way you can really do that is to sit down and map with people. And then at some point, hopefully, they kind of get it. And actually, I've heard Adrian Cockcroft use music as an analogy. So, like, you know, most people can listen to music and enjoy it. But to actually compose music, you need to study forms and patterns, and the language of it, which I rather liked as a comparison. Do you think that's a useful analogy actually?

Recommended talk: Wardley Mapping Strategy at BigCorp – A True Story • Erik Schoen • GOTO 2023

David Anderson: I think it's really good because there's like an abstract level of thought, which is composition and music theory that you need to understand. You can appreciate music, but you need that music theory to really understand it. And I think it's the same around Wardley Mapping and even technology. You've got to understand the abstractions and the principles to really create. And I think, Simon... Like, I'm a huge fan of Simon. I speak to him regularly. I was speaking to him yesterday, actually. He's just thinking...he's completely amazing in how he thinks about it, the Wardley Mapping technique. And again, if anyone hasn't heard of it, it's basically, you draw a value chain of, like, start from the customer, and all their kind of needs with dependencies. Then there's an evolutionary axis of genesis, custom, product, and commodity. And like, every single component will evolve through that cycle. And even to distill, like, that abstract thinking so cleanly. So when you start to map out the value chain in your organization, in your tech stack, even in your industry, the actual market, you start to see, what's the thing that you should be building? And what's the thing that you should just rent?

Having that clarity, that sense making is absolutely critical. I mean, we started using that technique, again, probably around '13 and '14. And that helped us build awareness around what we should buy and what should be built. Because it wasn't clear at the time, you know, and really understanding what's the value proposition of this team or this organization. And let's double down on that. So it's such an important technique. And the way I think of it is like sense making, it's making sense of something complicated.

The Role of Architects in Driving Change

Charles Humble: Do you think the fact that you were all practicing architects was relevant to how you approached the problems you were trying to solve at Liberty?

David Anderson: Yeah, that's a great question. I think we were open to new ideas. The way I started the team, and we were all architects as peers at the start. So we all knew each other. We had a good relationship. So there were no real eagles. And we spent a lot of, like, flat teams, that we would just get on a whiteboard or get in the room and we would talk about things. So we were never afraid to start whiteboarding and exchange ideas. We were always fans of different... Frameworks are important. You shouldn't blindly follow frameworks, but to give you new prompts and new ways of thinking. So we very much embraced the openness of these things. So we spent a lot of time just thinking about Wardley Mapping and drawing very bad maps. But again, the conversation was more important. And that good atmosphere between us, we were able to get creative, and there was no fear of saying something stupid. I would often say something stupid.

So I think that sort of courage, openness, and ability to think abstractly, and put your ideas out there really helped. I think had it been in a management team, with everyone having their own organizations, we wouldn't have gotten to that same level, you know, because there's a different dynamic in those types of teams.

Charles Humble: I think it's worth saying, certainly, in my experience of mapping, the business of doing the map is much more valuable than the map itself. That's to say, the process and what you get out of the process is more valuable than the piece of paper or the whiteboard at the end of it, I think.

David Anderson: Absolutely. Because what it does is it clarifies your thinking, and it exposes maybe things that you're...like a bias, if you had a bias, where you maybe thought something wasn't important. You could actually explore that and gain understanding. And what we started doing, because it's very hard if you spend a couple of days working on a map, and then you go to an executive and you show them the map. They're like, "What are you showing me?" So we would often take the observation from the map and say...like, distilling the bullet points. So you say, "Here's three bullet points of things that we should do." They go, "Ah. Wow, that sounds good." And they would say, "Well, how did you come to that? Or, can you tell me more about this?" So you're prepared for the conversation, the executive challenge, of why should we do that. So it's like you've done your homework, but you're not showing your working out. You know, so it's a very important technique. So yeah, absolutely, the mapping is... The activity is much more important than the outcome.

Charles Humble: I mean, in terms of a sort of overall key idea of the book itself, I think it's really about how you bring business and IT strategy together in what's a very large legacy company. Is that right? Would you kind of agree with that as a one-line summary?

David Anderson: Yes, absolutely. I think that's fair. I mean, that idea of joining business technology... I was always struck by the phrase, the business, used to really annoy me, because it's like, we're employees, we are the business. Like, there is no business. Let's understand the company mission and let's make it happen. You know, IT does not have a special dispensation to be separate from the company. You know, we all have a badge. So I think of the idea of having a single strategy, because I often hear... Liberty was very good at this. When you talk to very senior business people, they will say, "I don't really care what you do under the hood. I just want to protect policyholders. That's it."

When you have a really strong mission like that, you can then think about what we can do. And then you translate that because that means maybe a faster quote or a faster response. Or, you know, we equate that in the non-functional requirements. But you're not going to get the head of the business telling you, you need to have an SLO of 200 milliseconds. You know, you need to work out yourself, or you translate the business goal to your technological [inaudible 00:15:18] picture.

Charles Humble: It sounds really obvious, but I actually think it's a really key insight. And the fact that it goes both ways, as it were. I mean, I'd have loads of conversations in my consulting jobs with, you know, IT people saying, "Oh, I can't get the business to sponsor me to pay down technical debt or something." And you're like, "Well, of course, you can't. They have no idea what that means." If you can relate that to something that actually affects a business outcome, then you're much more likely to get someone who isn't an IT person to sponsor you. And so having some understanding of the business and how it works, and your role within that, I think is absolutely key.

David Anderson: I think the CIO at the time within Liberty Mutual, James McGlennon, was very good at bringing together different parts of the organization. You know, it's like Nicole Forsgren says in "Accelerate," it's like IT is not a call center. And so it's a creation of value. That's a super important concept. Yeah, 100%.

Recommended talk: Structures Shape Results: Software Insights • Elisabeth Hendrickson & Charles Humble • GOTO 2024

The Four Phases of the Value Flywheel

Charles Humble: You have four phases in your value flywheel. The flywheel itself I think is broadly from Jim Collins' great book, right? That's certainly where I first came across it. And you have your four phases. You have, you know, clarity/purpose and so on. Can you talk about those four phases and kind of why you have them and how that works?

David Anderson: One of the things that used to always slightly frustrate me is that, you get the Gantt chart of we're going to start and we'll keep going, and then we're done. And the thing that struck me about serverless, there was a guy called Sheen Brisals from LEGO, who talked about serverless being a rocket ship. Once you get on this rocket ship, you never get off. So software is continuous. Once you invest and you invest in software, you got to keep working on it. So I always liked the idea of the flywheel effect by Jim Collins. I think it's such a good model. It's a much better way of thinking about technology. So I started to think...retrospectively, I started thinking about how we drive this change in Liberty. Such a huge change going from, you know, a huge amount of success stories. And if you're interested, I would Google Liberty Mutual serverless. Google those three words, and you'll see a whole raft of talks and articles about the things that we did.

The four phases that I distilled on to join business technology strategy was first clarity of purpose. Like, the teams must have clarity of purpose. Don't hand the ticket to the engineers and say, build this. Explain to the engineers why we're building this. Like, what is the north star metric? Every company has a...you should have a north star for a product. What's the thing that makes this successful? Make sure engineers understand that, and, you know, really help them obsess over the value, and understand what's the competitive advantage in the market. Like, what are we trying to do as a company? Don't lock them in the basement of the IT department. So that's clarity of purpose.

The second one is the challenge, right? You have to create an environment of success. And this is really about psychological safety, and the socio technical elements of software. You know, don't just draw an R chart before teams, and expect them to go. This is very much in the DevOps kind of mindset. You know, we're actually building a system here. And the system contains software, and people, and teams. So think about that in its entirety, and how your software will help that system. And, you know, have the environment where you can raise your hand and say, I'm really not happy about how this is working, or I think there's a problem here. You know, encourage that safety. So that's that idea of challenge not in an aggressive way. You should be able to challenge the thinking. And challenge the thinking is always welcome. Because, you know, you might be right, you might be wrong, it doesn't matter. But you're asking the question.

The third phase is the next best action. What can we do to create value quickly? So that's the whole idea of code is reliability. You can't say, "This is our goal. Here's the environment. And I'm gonna disappear for 18 months and write a million lines of code." That's not helping. You don't need to do that. So with code reliability and the serverless first mindset, how can we quickly pull in together some public cloud services to actually write capability very quickly. And in order to do that in an organization, you need a frictionless developer experience. You may be an internal developer portal, where people can just grab a template, scaffold something, and have an API up and running in a matter of hours or days. That's where we need to get to. We have the capability to do that now in the industry. We can build API components really quickly. Let's do that. Let's build quickly and learn. That's that idea of next best action, what can we do quickly to prove the value.

And then the fourth phase is long term value. So long term value is based on the well-architected framework. So all three cloud providers, AWS, Azure, and Google, have a very simple well architected, where they say, from all our solution architects, these are the best practices in architecture. And I'm probably biased, and I've been using the AWS one for almost 10 years. So they have six pillars, which is, security, cost, operational excellence, reliability, performance, and sustainability. You can actually have an opinion of what good look like. The architecture isn't what someone thinks is nice. There's a very strict list of nonfunctional requirements that you need to meet, decide what they are, and use the well-architected framework as a common language in your organization, to say, this is our expectation for performance. It's black and white. It's not an idea. It's very clear.

And if you do follow a serverless first mindset, you can actually create very sustainable software, and keep a very low carbon footprint. Which is a nice way to think about efficiency, and writing good solid software. And that's that kind of flywheel effect. And then you're back to, once you're starting to, you know, have that ability to write good software in a very efficient manner, then when someone says, well, we have a north star about X, you can hit that very quickly. So it's about moving at pace. And as you're...the flywheel, where it really comes in is you're spotting inertia points, where there's maybe a problem with, you know, the security process, with infrastructure, with the business aren't invested in the technology team, the engineers don't have the skills, to start to spot inertia. And then saying, okay, you know, we have a problem here with our security scanning, how do we need to improve that? And don't do it for security. You'll sit down with the security team and say, "We need to approve this." And they'll say, "Yes, we do." So it's kind of like, in a collaborative environment, you start to spot the inertia, and then try and fix it. So you break down the silos and move the pace. This is what we did in Liberty for many years and traveled at extreme pace, I would say.

Charles Humble: I want to pick that up in a second. But I also just quickly wanted to pull out...because you used the code as a liability thing. There was kind of a key light bulb sort of insight for me there, which is this idea of having something like a simple motto. So, you know, code is a liability, or improving time to value, or something like that. And you say in the book that that can be much more effective as a way of keeping engineers moving, you know, or a 55-minute town hall with a boring presentation. And I just thought that's a really... It's not something I've seen sort of articulated before, it's not sort of particularly earth shattering. But somehow... I don't know, it's just one of those things that sort of really connected for me. I just think it's such a useful insight. This idea of Maslow's I think it's really worth drawing out.

David Anderson: You sometimes find slogans in the corporate world that don't mean anything. There was a book I read many years ago about "Enterprise Architecture Principles," and the idea of principles driven development. Having principles to help people understand, it's like, can you read my white paper, or just learn these three words? And I always thought this was a brilliant way to communicate thinking. And then when I started to work more with Amazon, I understood their leadership principles and the idea of their tenants. And they actually have a...they describe how they write a tenant. So something like disagree and commit. So disagree and commit is... And then it's like, you know, the fact that when something happens, you can disagree, you have the conversation. But once you agree to progress something, then we're gonna commit to it.

Disagree and commit is such a simple principle, or think big. You know, so they will need to spill these very complicated ideas, a few words, explain them, and then actually use them. So code is a liability was something... And it was mind blowing for many engineers. And I had many conversations, people say, "My job is to write code. Are you telling me not to write code?" I'd say, "Your job is not to write code. Our job is to solve business problems." They go, "Same thing." "No, it's not the same thing." So it led to politeness... I would describe that as a challenge, a few people having a conversation, though respectfully. And you get to where you get to, but it's a way of... Like, I've had many, many conversations like this. We help people understand, like, why you're here basically, in your organization, and what's important.

Charles Humble: But it's really interesting. I mean, you said it right at the beginning. You're kind of asking builders not to build or not build as much. It's one of those things I think you're so used to as a programmer to thinking that your value is essentially in the amount of code that you write. And it's a very important mind shift, I think.

David Anderson: It's as old as the hills. I've been looking at software engineering for decades, and would often look back at old books. I think it's fascinating figuring out what were the books that were written 50 years ago. And if you hear people joining companies in the early 80s, where there were incentives for lines of code written, and then someone comes in and refactors it. Their success criteria for the week is like minus 1000 lines of code. So it's like, this was happening in the 80s. So these concepts are not new. We forget them because we think the technology changes, the principle no longer applies. The principles are evergreen. It's the language technology that changes.

Recommended talk: Team Topologies, Software Architecture & Complexity • James Lewis • GOTO 2022

Building Momentum & Enabling Teams

Charles Humble: I think the thing that really blew me away reading the book was just the speed at which your team was able to do things. It's much, much faster than anything I've seen anywhere else. And part of what I found interesting about that is because you're a very large organization, so you have so many groups. In my world, that might be, I might have Java EE people, and I might have Cloud Foundry people, and so on and so on. So even just getting everyone on board with, we're now doing AWS and Lambda, you know, how do you do that?

David Anderson: Not easy. At least in the book, you can write a paragraph that describes four years of pain. I think for me, the secret is, like, engineering excellence, or high performing engineering. You want really good engineers who understand how to write software, who have an open mind. Your identity isn't the language that you use. Your pride is engineering. So I think that really helped. And I'd be biased here, and I'm located in Belfast. And in Ireland, we have a lot of really strong engineers, and people who will think in the right way. So I think it has the value of really strong software engineering. I don't want to think that I'm saying we don't code. Code is really important. It's critically important that we write code. But the right code is more important. So having that mindset. So I think when we are working with real engineers, who will think about the business problem, and what we're trying to achieve, then you can move with pace. Because when a technological improvement comes along, they get it.

Okay, so if I do that, instead of writing, like, you know, 2000 lines, I can now write this in like 20 lines. Okay, I see the improvement there. So good engineers are important. But also, I mean, it's a show don't tell. It's very hard to preach from your ivory tower and say, we should do this. We have projects where somebody would... There was one project in particular that a team...one of the guys had taken a piece of work in January, to write a product for August, because there was a regulation target that they had to hit with software. So he said, let's do this serverless first, using that approach. And they were finished by May. So they were actually finished, like three or four months early. So the team was like, "We're done now. Like, that's all finished." And it's like, okay, now we get into well-architected. Let's secure it. Let's reduce the cost. Let's optimize it. Let's improve the design, the user experience.

So building the thing quickly gives you time to test and learn, and improve some of the nonfunctional stuff. So by the time that regulation hits in August, the team were, like, well ahead of their brief, and the business stakeholders were delighted, expectations completely blown away. So that just has happened time and time again. And then you start to get to this environment where people talk about, wow, they did this. There is this new way of doing things that's really fast. Or not really fast, but just generally better. You know, then you start to get that... The flywheel starts to turn, you get rid of the inertia. So it took a long time.

Charles Humble: I was curious as well about how you found dealing with some of the enabling teams, you know, like, the security team, the audit team. Because, again, in my consulting work, and other experiences I've had, I've found those sorts of folks can be a particular challenge to kind of get on board with a change like this.

David Anderson: Well, the first thing is, I would say to people is that, security people are actually human beings. Go and talk to them and understand what they're trying to do. And I spent a lot of time working in cybersecurity and with all the teams, and I say with purposely. You know, because you understand what the controls are, and you understand the process that you're having to deal with is an implementation of a control. So if you can go to understand what's the control, maybe you can help the team create a better process. Like, a brilliant example was, I did a lot of work... At one point I was the secure development kind of lead for all of Liberty Mutual, like, the global lead for secure development, many years ago. And we had a controller, a processor on code reviews for secure development. But we evolved out of threat modeling, because threat modeling was a bit higher quality, and it met the control much better than the code review.

So just working with the team, like, the audit team in Liberty, we actually taught them the well-architected process, and said, instead of coming up with a list of questions, why don't you ask these questions from AWS, that are just as good. So go to the team, understand their control, and help them design a process that's a better fit for development teams and engineering teams. Because no one's happy if there's a bad process. We're not meeting the control well, the team, there's friction, you know, there may still be issues with audit or security. So if you understand both sides of the fence, then you can design a better process with more automation or more quality, whatever. Certainly working with the teams to help them create a better process to meet the control. I think that's the only way to do that. If you clash with these enabling teams, forget about it, end of story.

Charles Humble: I mean, obviously, part of the point of the flywheel is to try and build up momentum. So how did you build momentum at Liberty?

David Anderson: Well, we were kind of lucky, I think, in the sense that around...I think it was around 2016, there was a big change in the organization. Again, I talked about the CIO at the time, and wrote a technology manifesto. Because there were three changes that happened at the same time, the move to the close, the move to kind of customer centric, you know, kind of design thinking, and the customer centricity. And then, like, full scale agile across the board. So they did those three changes at the same time, which if they're looking back, maybe it wasn't a good idea. It was successful, but it was an interesting year, too. So there was an appetite for change. So sometimes... I mean, I certainly will not claim that serverless first changed the entire company. There were many, many people driving a lot... But there was an era of change. You know, there was an era of transformation.

So that's always a good time too, I think... Someone asked me this, I got asked this question yesterday. Like, how do you drive change? A lot of it's about timing. So what I'll find is there are lots of different ideas of things we can improve. But sometimes you'll see that, okay, there's no appetite to make this improvement. I have an approach for that. So I would always be thinking about, you know, what are the potential changes? So we were really lucky in the sense that as we were working out this serverless first way of working, that kind of environment of transformation was there. And it was a good time to kind of, you know, I wouldn't say create the momentum, but kind of jump on with the...take advantage of the momentum.

Charles Humble: I think that's really interesting, actually. And I think it's spot on, the timing and an appetite for change is absolutely key. And being able to identify, this is one of those moments where we can make change, because you can't always, for all kinds of reasons.

Implementing the Cloud Development Kit

Charles Humble: We should talk about the cloud development kit and the building blocks you used. Why was that so impactful?

David Anderson: Well, one of the... Again, I say it was an inertia point. One of the early inertia points we had within AWS was the idea of cloud formation. We talked about infrastructure code. I often think with enabling constraints. So one of the enabling constraints that we designed in Liberty was that every single change going into the public cloud would be infrastructure code. So in order to do that, we disabled the AWS console, so you physically couldn't get access to the console. You could in a sandbox, but once you went into a dev environment, not even production, a dev environment, everything was automated. So that was an enabling constraint. Because that was...you know, that's repeatability. I would say that's a good kind of DevOps kind of practice. But then it's a massive kind of learning curve to learn cloud formation. And even reusing cloud formation is not great. It's a bit better now. But at the time, it was quite painful.

So CDK gave us a nice abstraction. We'd always thought about what are the right abstractions that we need. Give me an API, versus give me a bunch of code that I can use. So CDK was able to give us some core kinds of architectural patterns that we could give to teams. And say, here's a partner for finding an open API. More importantly, I mean, this was before backstage. We had the need to write an internal developer portal, effectively early days of platform engineering. So we had a few failed attempts. First of all, we were very opinionated. And we said this is exactly how all the software should look. Developer says, "Nope, I don't agree with that build." Then the team said, "Let's build an abstraction around the cloud." "Nope, that's not going to work. We're not fast enough." So then what we ended up doing was with CDK, it gave us a language to create reusable patterns. And then we had, like, an internal developer portal called software accelerator, that people could publish their patterns, and other people could use them. And if it didn't work, they could change them.

So it was an early form of inner source. I was always a fan of inner sources. You know, it shouldn't be a single team running all the templates. Let's open up and have the single team may be responsible for quality control, and have the engineers... Because the engineers will move much faster than any central team. So empower them to actually create these assets, create these templates, extend things. So another enabling constraint was the fact that you could actually...there was a golden pathway, there was a single path to production, which had some constraints. And the CDK development kit gives us a way to be opinionated. Like, this is how you should tag resources. There are some, like, you know, I would say cloud infrastructure standards you need to meet. But once you meet those, you pretty much do whatever you want.

It's important that they have a building block, because often we'll think of a new technology, but we don't enable the teams to actually, you know, make use of it. So it was really important... Again, as I say, it was an early form of platform engineering. Platform engineering I think is still in its infancy, because a lot of people will think of it as the build it and they will come. I will build this thing for the engineers, and they will use it. And that never works, you know. So I think through our experience, we knew that there was a nice pattern there of what's now called the internal developer portal. And CDK was kind of our language around that.

Charles Humble: To start bringing all of this together, can you talk about what the other things you need to do to enable teams to move as quickly as you're able to? Because it's not purely about technology.

David Anderson: So there's probably a nice tie into what I'm currently doing now, because currently I'm with G-P or Globalization Partners now, as an architect. I'd say we're a scale up much, much more than Liberty Mutual. It's maybe 1500 people, probably 10% the size of the teams. But what we've done is we thought, like, what are the building blocks we need to provide the teams to make them move really quickly? So again, you're right, it's not about technology. So some of the things that we've put in place, we have quite an opinionated AWS design, that we've said, like, there's going to be completely event driven and demand driven. And every workload has its own account. There are certain constraints around that, around how you get your account, how you get your environment. So it's quite an opinionated way of how you get your cloud environment.

The building blocks across the entire organization, we've kind of laid those out to be kind of pointed contexts in the domain driven design element. So the components of this big system fit together a certain way. We're using a single technology as an eventing backbone, which is an event bridge. So it's preferred with asynchronous communications and synchronous when needed. And then being kind of opinionated on the technologies that we use. So giving people a specific set of building blocks, and it was able to move very fast. So I would say, the key there is using the architecture team as an enablement team. So architecture as an enablement team. Your kind of DevOps team, your infrastructure team, your cloud engineer enablement team, security or enablement team. And then kind of our process for helping the teams is the well-architected framework, what we call it SCARP, which is a process that every two weeks, we'll sit with the teams, with individual teams, and work through nonfunctional metrics, like performance, security, etc.. And kind of help the teams.

There's a whole idea of empowerment. The teams are empowered to build and own their workloads. But we create the right environment in which they can actually have the tools and building blocks that they need. So I think the technical decisions that we've made are quite high up in the stack. You know, we don't really want teams gone too low on the stack. So we're actually encouraged in Lambda, event bridge, API gateway, as the building blocks or not going kind of too low. And that's the way you really want teams to move fast. But also think there's a piece around explaining a concept to the rest of the organization. That there's a different way that we'll build software. Because you're right, a lot of people think that serverless first is just writing everything in functions. It's actually about thinking about capabilities. You know, another way of this is thinking capabilities or features. Someone says, we need a new feature, say, well, what's the capability that we need? Or we need to bring in vendor X.

Well, it's not vendor X, it's a capability. So let's think abstractly with the capability and find the best match for that capability. So I think it's a different way of thinking. It's a hard change, because it is a mindset change within the organization. But I think you need to kind of create that... And really what we're starting to call this now is modernization. I still think we're struggling for what's the name of this different way of working. And it really is the modernization. So what I find is a lot of companies, they've done cloud migration, now they're starting to do cloud modernization.

Recommended talk: Enabling Developers in a Multi-Cloud World • Mauricio Salatino • GOTO 2023

Charles Humble: That's interesting. Because I think that, you know, the first move to cloud, a lot of that was, containers and orchestrators, one way or another, Docker and Kubernetes. It was basically Docker and Kubernetes. So do you see a move through mass approach to serverless? Is that kind of what you're driving at?

David Anderson: I think the serverless world and the container world will start to come together. I think it's really important to say containers are not bad. It's not this versus that. But it's almost like...it's what's in the container. Do you know what I mean? If you just took your monolith and just dropped it in a single container, you know, you've still got a monolith in the container. But it's much about...it's how you break that up. So what I find, a lot of companies have basically moved, lock stock, into the cloud. And now what they've got is they've got a nicer data center that they can't get into. The company's stand up like observability, and processes that now give them better eyes on their software. They can see cost, they can see security, performance. And they're starting to get eyes, they're starting to measure how the software is performing from a nonfunctional perspective. Speed of change. And then I think the next natural evolution is to start decoupling and breaking up. And that leads you to event driven or maybe even serverless things.

A lot of the early work we did was maybe take a piece of something and make that serverless. So you have maybe a serverless capability with a container capability underneath it. So you may start to break up the monoliths. Again, it's just microservices, really. Often the migration is like one move. The modernization could go on for years. Because you could reimagine your business. There's a whole Conway's Law with the structure of organization. There will be legacy code. It's a very complex kind of nut to crack. There's no sort of one and done. Which is why...going back to the idea of the book, which is why this idea of the flavor, the fact you could pick a single purpose, hit it, learn, and do something else with that. It's the evolutionary architecture nature of that.

So again, I'm not going to say containers are bad. I think they're... I mean, serverless, it's all containers under the hood. But it's the idea of how you think about kind of your system domain and your architectural approach. I think that's really what that is. And moving towards a more granular decoupled approach.

Charles Humble: That's fantastic. David, thank you very much indeed for taking the time to chat to me today.

David Anderson: Thank you. I really enjoyed the conversation.