Serverless Development on AWS: Building Enterprise-Scale Serverless Solutions

Why a Book on Serverless?

Khawaja Shams: My name is Khawaja Shams. I'm the Co-Founder and CEO at Momento. I'm very excited to be here with Sheen Brisals and Luke Hedger to talk about their exciting new book. So why don't we get started with a little bit of history and some introductions about you guys. Luke, why don't we start with you?

Luke Hedger: Yeah, thanks so much, Khawaja. Great to be here and talking about the book. My name is Luke Hedger. I like to describe myself as a software engineer, plain and simple. I like the nuts and the bolts and being deep in the details. I also dabble in writing books, and that's why we're here today. I've been working with AWS Serverless probably for about five or six years now. And I'm an AWS community builder. And that's what I love most about Serverless, is the community around it and having these thoughtful, interesting discussions, and really kind of pushing the envelope forward. So, great to be here.

Sheen Brisals: Thanks Khawaja. I'm an engineer at heart. And I started my journey long ago. I've been in the industry for over 30 years, starting as a software engineer, being through big and small enterprises, traveling pretty much from the procedural language days of C, all the way to JavaScript, hopping through, you know, object-oriented C++, Java, ecosystem, etc.. And these days, my focus is mostly on AWS and specializing in Serverless. And I also share what I learned with the community via articles and conference talks, so on and so forth. And that's kind of, you know, became the reason why we are here, and why I wrote the book with Luke. And I'm also an AWS Serverless hero. That's a recognition for people who engage with the community, spreading the technology around. So, excited to be here.

Khawaja Shams: So we've got a community builder and AWS Serverless hero in our midst. So that's exciting for everybody that's tuning in. Where are you all located? Sheen, you, and then Luke?

Sheen Brisals: I am based in London, but outside of London.

Luke Hedger: I'm in North London myself. Born in South London, so I traded the South for the North, trading my roots.

Khawaja Shams: So you're both in the AWS ecosystem for a very long time. And in that ecosystem, you can evangelize best practices and, you know, really get passionate about teaching people how to use these capabilities appropriately. There's so many forums and media outlets to do that. You can have podcasts, you can write articles. Why write a book and why write it on this topic?

Sheen Brisals: Good question. I'll make a start and hand over to Luke Hedger. I think it's partly because of my background. Because I've been through several teams and decades of, you know, building software. In the early days, I've been used to the software development life cycle, the typical STLC, and going through the different phases of engineering applications. But when I started to adopt or, you know, learn serverless and to develop serverless applications, I noticed that because of several reasons, which we will talk about later, many in the industry were considering it as a toy to develop something quickly and get it over. But I noticed that in order to be successful and sustainable with the development process, there has to be a proper, you know, process or methodology in place. That was in my mind as I started sharing the knowledge. And then as I was traveling across different places for conferences, I gathered or observed from engineers speaking to me that, oh, there is a real need for people to understand what this technology is and how we can build better solutions. Not just like, you know, throw something together and get it done. No, the enterprises don't work like that. We need kind of long lasting applications. So that kind of became the core inspiration for thinking along the lens of how we structure the book.

Luke Hedger: I think the book format lends itself to going deep on a topic, right? Some topics wouldn't fit the book format, but I think serverless is one of those things that forces you to pause and step back, and take a look at something holistically. There's plenty of great information out there in formats like you mentioned, Khawaja Shams, blog posts and podcasts. And I think if you're looking for a very specific answer about a specific service or a problem you're working on, those are great for that. Book, maybe not so much. But I think, yeah, when you start off with serverless predominantly, but also when you continue with serverless, having that foundation to build upon is super important. And I think encapsulating that in various blog posts is very, very difficult. It's kind of difficult to maintain a consistent narrative. Whereas a book, it definitely is the perfect format for that.

Khawaja Shams: It sounds like you both have this belief that serverless is not just for toy projects. It is for serious enterprise ready applications, and the right venue or the right medium for diving really deep into the enterprise grade explanations warrants a book as opposed to just a set of articles and podcasts. That makes perfect sense. 

Case Studies of Serverless in Enterprises

Khawaja Shams: We're seeing more and more enterprises go all in on serverless. It's becoming... I think it was perceived as a toy project or only for toy projects a little bit ago, but we're seeing it grow more mainstream recently. Can you talk about some of the case studies of real enterprises that are in your book?

Luke Hedger: I was going to say, just before getting to the case studies, I think what I found fascinating from my own personal case study, a team I've worked on in serverless, is this idea of the adjacent possible. So within software, we observe lots of trends, things like devops, CICD, gitops, automated testing, cost optimization, these sorts of things, available architectures, are a good one as well. These things are great and we've had a lot of success applying them to more traditional serverfull architectures. I think what I realized when serverless came into my radar, I saw that as an adjacent possible, something that we could build upon, that would enable all these other trends to become more than just trends, to become practical strategies that we could apply day to day. And I think that's what a lot of enterprises realize when they begin to understand serverless, hey, this could unlock so much more potential. It will allow us to iterate very quickly, allow us to scale spiky traffic, and handle costs a little bit more tangibly. That's definitely what we saw from the two case studies in our book. And I think they're both really, really fascinating case studies that have been discussed a lot. We have PostNL, the postal service of the Netherlands, and we have Taco Bell, obviously a fast food outlet across the world. Two very different case studies, but absolutely fascinating use cases for serverless.

Sheen Brisals: To add to what Luke mentioned, Khawaja, as you mentioned, there are several articles on how to do various things, focusing specifically on Lambda functions, or this service or that service. But as I mentioned, when it comes to enterprises, you need a real case study to show people the possibilities of technology and how to take the technology through different phases. And that's what we wanted to bring into the book. So we were looking at different options. And Taco Bell came in because I knew Robby for a long time. And every time we met at ReInvent, we used to have a long chat, sharing ideas, and practices, and better things, structuring teams, etc.. And, you know, his serverless adoption story at Taco Bell was unique and really wanted to bring that in. My initial doubt was whether he would be open to the option. But when I discussed it with Robby, he was like, "Oh, I'm in." And that's really a unique story showing what was the state before, why serverless, and how they may start, getting on the changes they applied along the way. And even at the end, giving advice, now that we've been through the journey, what can I tell the future generation of serverless adopters or enterprises? Here are a few things that you need to keep in mind. So that's how it happened. And PostNL, another global brand, and a different domain altogether, which also kind of brings in a different flavor for readers to understand how a different kind of set of logistics companies, postal companies, cannot be serverless. And I know from the days when I started to talk about EventBridge and Event Driven Architecture. And that's how I came to understand his use case and their use case. So, you know, we are honored to have both as part of the book and have several constructive conversations, how to structure things or how to improve the content. Absolutely happy to have both as our case studies.

Recommended talk: Operating EDA with Quality in Mind • Sheen Brisals & Sarah Hamilton • GOTO 2024

Khawaja Shams: These are fairly traditional enterprises that are using serverless for more than just toy projects. And I remember Robby telling me that he used to keep getting in trouble with his finance team because every time he would transition a project into full serverless, the cost savings would make it hard for them to meet their spend commitment with Amazon. It's a very spiky business, right? So it fits serverless really, really well. And that was a really good awakening to just see... I think he was a good leading indicator in terms of what others are going to see because people have lots and lots of spiky businesses.

Sheen Brisals: That's an important point you mentioned about cost. Because recently I wrote an article about serverless expense here. How do you take cost into account? One of the aspects I touched on on that article is, like, if you had a non-serverless ecosystem, containers and things, you know the kind of monthly expense because you pay X amount for this particular instant time for this duration. Whereas when it comes to serverless, as you said, the spiky traffic situation can vary the price. So, one week could be low, the next week is an important event that will increase the price. The finance team should have that sort of understanding to know why the variation when we adopt serverless. Yeah, that's an important aspect of serverless.

Khawaja Shams: I personally like the consumption-based model because usually, if you're paying more, you're probably using more. And you can write poorly performing or more expensive applications, whatever framework you want. So, the serverless aspect is one way to do it. You can write it really well in the serverless world as well. 

Recommended talk: Expert Talk: Are We Post-Serverless? • Julian Wood & James Beswick • GOTO 2024

Barriers of Serverless Adoption

Khawaja Shams: What's still missing in the serverless world in you guys' opinion? What are the biggest barriers for every enterprise to just start with serverless?

Luke Hedger: Well, yeah, that's a good question. I think there's different levels on which you can look at that. I think at a high level, there's definitely still some big barriers, particularly around enterprise adoption. Let's face it, it's a departure from traditional engineering techniques and familiarity is always going to be a safe bet, especially when you've got lots of critical workloads. I think part of the barrier therefore is the maximalist approach, one technology or another technology, looking at things in a binary way. And so I think serverless perhaps needs to be repositioned as a tool rather than a particular methodology or a dogma. It's a means to an end. It's something to get you from a problem to a solution. And it can do that really, really well because it allows you to iterate and evolve your architecture. It is so suitable for that. And you can do that whilst keeping costs low and visibility high. But I think there's also a barrier to entry for engineers. Sometimes you'll have a group of engineers who have never heard of the term serverless. You might even extend that. They've never worked with the cloud before. They've never worked with AWS before. There's lots of levels or barriers to jump over. And that is a big part of why we wrote this book. I remember when I took a new job in the summer of 2019, went through the interview, seemed like a perfect fit, got in there, and they said, "Right, we're just working out which team you're going to join. Okay, you're going to join this team now, Luke Hedger. We want to implement everything on AWS serverless." And I said, "I have no idea what any of what you've just said means. I have never worked serverless. I've never built anything, certainly not a production workflow on AWS." I had to frantically search around the internet to try and understand what all these things meant. And so I always say this, the book that we've written is the book I wish I had on my desk in that summer of 2019 because it would have been that reference manual. How do I get started? How do I massively reduce that barrier to entry? So hopefully, our book does contribute to the adoption of serverless, at least at that high level.

Sheen Brisals: So interesting that Luke mentioned about his wish he had this book a few years ago. I was at the AWS Heroes Summit last week and speaking to one of our AWS heroes, he echoed the same thing, because he almost finished reading the book. And he said, "Sheen, I wish I had this book four years ago. Because having worked in the industry for these many years, heroes, I mean, people who are in the serverless and AWS ecosystem, they know how to do most of the things." So this is again proving the point that this book is out there helping people who are just onboarding onto serverless or going through the process.

Addressing Misconceptions About Serverless

Khawaja Shams: I think the points that are worth underscoring in the answers I heard so far are, we're not zealots that are trying to tell everybody that you should just go serverless blindly. You're trying to make sure it's... Rather than serverless adoption, you're trying to eliminate the serverless aversion to combat the familiarity bias. And you're trying to do that. One of the reasons you see the barrier is there isn't enough educational material out there to dive deep in the book. The book that people wish they had when they started serverless can help eliminate some of that cognitive dissonance that people have, which stops them from... And I totally empathize with this. If I take a backend server-side programming job and I show up for my job the first day and they're like, do serverless, like, wait, server-side job, what do you mean serverless? So it could be quite confusing.

Sheen Brisals: That's a valid point. We have a chapter exactly for assessing the readiness for serverless and the middle price of it. In there, we talk about the use cases that are fit for serverless and use cases that may not be fit for serverless, and how do you identify and what can you do about it. And also I mentioned somewhere that the serverless first does not mean serverless must. So there are principles, starting from domain driven design, all the way down, that enterprises need to be aware of in order to make it profitable or enabling technology. And that's where, as Luke mentioned, the knowledge or the education should come. People should understand the characteristics and capabilities of serverless before they dive too deep in and start developing solutions.

Luke Hedger: I think that's one of the things we noticed as we were sort of deciding to write the book and even drafting out the rough outline and the proposal, and the content, was that there was a trend towards a lot of myths and uncertainty around serverless, particularly in some of the more unexplored topics. So, security being one. There would seem to be a constant question around, you know, is serverless secure? Do I need to do anything to secure it? Does it actually benefit my application security? Other ones being, you know, can I develop serverless locally? Can I test serverless? Is it really, really expensive? And I think what we tried to do was not just demystify those points, but to turn them into opportunities, because I think that's where serverless really comes into its own. It's not necessarily just about saying, hey, no, like security is a thing with serverless. It's about saying, there's a lot of potential here. You can kind of adapt your security processes around serverless. Same with local development and testing. Okay, you can't do things the way you used to, but maybe that's a good thing. Maybe you can actually take out... Sheen mentioned the software development life cycle. Perhaps you can take out a few extraneous steps from that and make that a lot quicker. So now your iteration loops are actually quicker than they were before serverless. It was about demystifying a lot of those things and about education. Also saying, hey, look, if you look at it from this slightly different angle, maybe you'll stop thinking about serverless versus how things were and start thinking about serverless and how things can be.

Khawaja Shams: I think that fits really well with your notion that serverless is a tool rather than just a methodology or philosophy. If you see it as a philosophy, you'll litigate whether it's secure or whether it's performant or not. If you see it as a tool, you'll check fit again, the specific use cases that you have in mind. And that's, I think, where you're getting to, Sheen, with the criteria that you have to assess which workloads are best fit for serverless. Which chapter is that? Do you remember off the top of your head?

Sheen Brisals: Chapter two.

Khawaja Shams: Chapter two. Okay.

Sheen Brisals: Enterprise readiness. Just to add to that, I often give a different view when we look at serverless, because we've been through all sorts of definitions for serverless as a word or a doctrine, spectrum, etc.. The thing is, services like SQS, S3, managed services, existed before the term serverless was even a thing. Right? No one had any issues because they were considered as applications, invoked via APIs, etc.. When Lambda functions came along, function as a service, that's when people realized, oh, to execute my code, I don't need to own a box of my own or a container that I own. I can just upload and the magic happens. So that's how the notion of the serverless started to take shape. And of course, there are a bunch of other things like scale to zero, etc.. Even with SQS and S3, we had those things, except you pay for the storage. Right? So the way I see it, I kind of often convey is, like, when you think of serverless as technology, you bring in all the managed services. Basically, you are collaborating with these managed services and stitching with the infrastructure as code. So that means, already you have managed services, infrastructure as code, and you as the engineer in the picture, and who you are building the solution for, for your stakeholders. So they are in the picture. So that's why I often convey it as a technology ecosystem. Because everyone is part of it, including all the core, primitive services, and the tools, and frameworks, and all sorts of things that we talk about. That's a kind of different take, just to think as a whole piece, not just a serverless word or something like that.

Recommended talk: The Road To Event-Driven Architecture At LEGO.com • Sheen Brisals • GOTO 2022

Khawaja Shams: My favorite interaction with some of the enterprises that say, I am not serverless, we don't do serverless here, we're fully on EC2. And I always ask them, where do you think your EMIs come from? They're coming from S3. What about IAM? You use IAM, right? How many units of IAM did you configure? And what's the minimum for IAM? The best thing about serverless technologies is they do blend into the background. And again, they're not the right fit in every single application. And that's where the tools like the one you mentioned come into play.

Advice for Teams Evaluating and Adopting Serverless

Khawaja Shams: We'll do a couple of really rapid fire questions before we end. And we'll switch off, like, each one of you can pick whichever one you want. But three part question, rapid fire answers. What advice do you have for a team that is evaluating going into serverless for the first time? They're a hardcore shop that believes they've never touched serverless. What advice do you have for them?

Luke Hedger: I'd say just go ahead and try it out. Identify something that you're going to run in production, but is fairly low criticality. And just try it out. Forget the marketing terms, forget everything you've heard before. Just try it out. See how it feels. And if you evaluate it and it's great for you, then brilliant. And if not, you've not wasted too much time. So, yeah, I'd say just go ahead, try it out. It'll be good.

Sheen Brisals: I will add one extra bit to Luke's answer. I mentioned the capabilities and characteristics. I think I understand some of those, you know, what it is capable of, then go experiment, learn.

Khawaja Shams: I like that. I think it's like, take it for a test drive and be clinical about your evaluation criteria. Just what being a good engineer is about, right? All right. What about...

Luke Hedger: What we should have said was buy the book and read it. That should be the first thing that you do. We're obviously not good sales people, so we didn't think about that.

Khawaja Shams: That's why you're good engineers. It's hard to be good at too many things at once.

Luke Hedger: Yes, it's very difficult.

Khawaja Shams: What about if you're in the middle of adopting serverless? What would you do there? What advice would you have for an enterprise that's in the middle of adopting serverless?

Luke Hedger: I think I'll probably stick with it. Maybe you're starting to climb that slightly steep learning curve. Maybe you're starting to have a bit of regret. Maybe you want to go back to the familiar tools and services you've used before. But I would say stick with it. And if things aren't working, if you're trying to map your existing processes onto a new technology, that's probably where you're getting a lot of friction. So look at those. And there's loads of stuff about that in the book. If it's security, if it's testing, if it's development, operation, that's what we've tried to do in the book, is really try to help you understand how to build the best processes and workflows around serverless. You know your use cases, you get in there and implement those as you see fit. But we'll help you kind of look at all the things around that so that you get a nice efficient workflow all the way through. Stick with it and look at potentially where you're getting a lot of friction in terms of the way you're building and the way you're approaching your serverless application.

Sheen Brisals: Whatever Luke said, plus as you iterate, improve. Okay? Keep improving. So I'm a great believer in continuous refactoring, especially when it comes to cloud and serverless technologies, because things are changing all the time. Not just technology, you know, serverless or AWS, the technology around us is changing. So that's why you need to keep that open mind of refactoring. So we talk about that in the final chapter, the ways we can sort of bring that mindset and keep refactoring. So that's important. And as enterprises, you know, going through the adoption, put the sort of guardrails, and principles, and practices in place so that, you know, they are improving all the time, not going back. So that's, you know, critical for the adoption.

Khawaja Shams: Seems like, you know, when you go from, you start with serverless aversion, and you take it for a test drive and you start mastering the tool, and you get clinical about the criteria with which you evaluate its efficacy. Then you have to kind of keep going in the middle stage where you keep mastering the tool because as you master the tool, you'll learn more about whether or not it's a fit. You have to make your criteria even more clinical than before, because as you become master at it, you don't want to throw it at everything either. And that's where that checklist comes into play as well.

Sheen Brisals: Now that you mentioned test drive, I can't stop thinking, like, you know, once you own the car, you know what's the optimum speed that gives you the maximum mileage, you know what tire pressures you need to maintain. So, you know, thinking shifts to those sorts of ways. Like, you know, how often you need to change your oil and all sorts of things, especially with the traditional gas powered cars.

Khawaja Shams: No, absolutely. I think, you know, mastery is an interesting thing. You can always keep getting better. You know, once you take your car out on the racetrack... We'll push this analogy further, we'll call this the production, you get to learn the limits of that technology. And sometimes, you know, when I'm on a racetrack, I learn that my car could do a lot more than I thought it could do, especially when I'm not the one driving it. If it's an instructor driving it, I say, holy cow. So taking inspiration from the experts like you, reading your book, and, you know, I think belonging to a community so you can address, not just how do I use service for this, but does it actually fit or not? And having that discourse can also help people. Where do people go to find this book? How do I get one? And can I get a digital copy? I mean, I already know the answer, but I think for the audience, you know, how do they get this book?

Luke Hedger:  I think all the usual outlets, so you can either get a digital copy or a hard copy. All the usual outlets, Amazon, your favorite bookstore. And also what's amazing is if you do have access to the O'Reilly Learning Platform, you'll have access to this book and all the other amazing books, some of which we do actually allude to and reference in the book. So, yeah, that's a really great platform if you can get access to that.

Sheen Brisals: And also, the case studies we talked about. So there is a link in the book that you can download and then you can share with your team, you know, study it, learn it.

Khawaja Shams: So my confession, I made potentially a mistake, but I bought the digital copy and then I realized I can't get you guys to sign it for me.

Luke Hedger: It's a common mistake. 

Khawaja Shams: But I'm sure we'll drop the links, some of the options, some of the ways to get this. Sheen, Luke, it's been great talking to you guys about this. It's such an honor to meet the two authors of a great book. Thank you for writing this. Thank you for, you know, having a not so overzealous approach, a clinical approach for evaluating serverless and just blindly doing it. It's a great book. I've personally read it. I really enjoyed it. And I'd highly recommend people to grab a copy and read it, regardless of what stage your enterprise is in, because it gives you a clinical way to assess how to make the most use of this tool. Thank you.

Sheen Brisals: I mean, thank you, Khawaja, for taking your time and chatting with us. And yes, you know, go out, explore the book. Something we haven't talked about much is every chapter has an expert and their opinions shared, unedited. It's great insights into various topics that we discuss in the book. I love your Memento's theme, believe in serverless. That is, you know, critical when it comes to serverless adoption, because seeing is believing and, when you believe, magic will happen. 

Luke Hedger: If you do go out and buy the book, which we obviously really hope you do, we love hearing what you thought about it, good or bad. We love to hear what you're thinking, what you've learned from it. We love having those discussions. Reach out to us on Twitter or LinkedIn, or even better, if you see us in person at a conference, we'd absolutely love to chat with you. So yeah, that would be absolutely brilliant. Thank you so much, Khawaja Shams, for taking the time to do this interview. I've really enjoyed it. I could go on for another hour, I think. But we'll leave it there.