Unlock the Secrets of Great, Lasting APIs

Listen to this episode on:

Apple Podcasts | Google Podcasts | Spotify | Overcast | Pocket Casts

---

Have you been tasked with creating web APIs for your organization? Mike Amundsen, author of the book "Design and Build Great Web APIs", goes through some of the secrets of building great APIs and what the best process and tools are for doing that with Casey Rosenthal.

This episode of the GOTO Book Club was made possible thanks to the support of SimCorp. SimCorp provides front to back, integrated investment management solutions to the world’s leading asset managers, asset servicers, pension and insurance funds, wealth managers, central banks, and sovereign wealth funds. Learn more at Simcorp.com.

---

Should I read Design and Build Great Web APIs: Robust, Reliable, and Resilient if I am not a web developer?

Preben Thorø: I've been going through your book and the first thing that's come to my mind is that this is about designing and building great web APIs. But I think in a connected world as it looks now, an API is way bigger than just from a web world. So should I read this book even if I'm not a web developer?

Mike Amundsen: Actually, we had some debates on titling the book as to whether or not to include the web as part of the book title and we decided "yes" because it's really the most common and most accessible way to start thinking about these APIs. But as you said, APIs, application interfaces, they're connecting everything. They're connecting our watches, our shoes, our doorways, our heating systems, our computers, our cars, everything! So there are lots and lots of APIs that we encounter every single day. And there are lots of things in this book that talk about how to design, how to focus on implementation, how to test, how to secure, how to deploy, and those were well beyond the idea of just APIs on the web. So there's lots of material in there even if you're not web-focused at this present time. So there's lots of good stuff there.

Mike Amundsen: Yes. I do. As you pointed out, this book is pretty tight. It's the product of years of me working on APIs in various formats and coming up with a kind of a generic or abstract, or generalized version of what that experience can be like; plus I used this material to teach. I've taught public classes and private classes. So this material is kind of a reflection of the years of experience, the years of teaching, feedback from students. I've boiled it into, as you probably noticed, into these chunks and I tried to arrange the book in a way that said, "Look, I really just need to focus on testing. I understand about design and deploy, but this testing thing needs time." So to make sure that there's a nice tight testing chapter that you can work in or a deployment chapter, or a sketching chapter, or a design chapter. So it really was a big effort.

As an author... I don't know. You can probably relate to this as well. I read a lot. I write every single day. There's always something that's got an itch for me.

This book took a lot longer than I had expected it to. Having done this material for years, I thought, "Oh, this...," you know, "I can put this together. I can use the material I have." It was a lot of effort. So it's nice to hear that it looks like it's tight and it fits well. It's been a project that I've been working on for more than a year which is kind of atypical for me.

Casey Rosenthal: Well, certainly different authors have different time scales too, right?

It is very polished. So I appreciate that about it. Did you imagine that the primary audience, the person who primarily would be attracted to this book, is somebody who has a lot of software engineering experience, has written a couple of APIs before, and maybe that project didn't quite go the way that they want it to, or somebody who is coming more from like a project management side, maybe isn't a software engineer, and, you know, a project like this falls in their lap, and they're, you know, now they have to figure out how to develop an API without having any background on maybe even what that means and what the benefits of an API are? 

Illustrations by Dana Amundsen.

Mike Amundsen: Yes. The primary audience that I envisioned when I wrote the book is the individual developer, the person who has been tasked with actually creating an API. And in small organizations, a lot of times that person needs to be the veritable full stack developer, right? They've got to be able to do design and modeling, and definitions, and then sketching, and prototyping, and then actually coding, and then testing, and then securing it. They have to do that whole thing, right? So originally that's the primary audience, but I wrote it in a way that if you're primarily a project manager or an architect -maybe you're a software architect - and you don't have a lot of experience with APIs, but you know a lot about distributed systems and stuff like that, there's still something in there for you as well. So it was originally designed with a hands-on developer in mind, but it also offers some value in lots of other categories. That's really sort of the aim that I had.

Key takeaway no. 1

The main audience for the book is the individual developer, but project managers and software architects can definitely benefit from it too.

How do we handle the friction if you've got multiple teams and iteration cycles don't line up?

Casey Rosenthal: That's such an interesting point. So even if your responsibility, you know, working in an organization, is only one stage of the API development, the book kind of puts in context the whole, you know, the whole process. So maybe, you know, if you're just doing the design phase, maybe you can do that a little bit better in concert with the other components. I know at some companies, the API development isn't just one person. So have you run into cases in, you know, in your experience where some of the cycles that you talk about are, you know, is there friction if you've got a distributed team and those cycles of iteration don't quite line up? You know, how does that work?

Mike Amundsen: Yes. That comes up in the book a couple of times. Because of the nature of books --books are sequential, right? -- it had to be kind of written in a sort of a single arc or a single storyline. For example, testing doesn't show up until, I think, chapter 9 or something which is really a bad...sort of a bad idea, but there are definitely cases where there are lots of overlapping circles. 

I've worked in this in a few distributed teams and you may recall, it's actually in a couple of key sections. There's a design section, a build section, and a release section. So a lot of times that sort of encapsulates communities. The design community, at some point, has to produce enough assets that the build community can start working in and the build community has to have enough assets to somewhere along the line the release, the testing, the DevOps has to start up.

 So while it's written in a sort of a straight line, it's really kind of overlapping circles again and it acknowledges this notion of iteration. It brings up iteration and cycles a lot. So everybody has to adjust a little bit to figure out how they're doing this. If you're a single person, this is a nice adventure. If it's a team, you have to kind of coordinate the hand-offs or coordinate the sharing part of it and in this case I use GitHub. And since I'm doing Node and NPM, it's sort of the central place. So there's an asset section for dealing with stuff from design and there's a security, and a testing folder in the system, and stuff like that. 

Illustrations by Dana Amundsen.

So, you know, it's kind of adjusted as to how you can mix and match to match your own style. So whether you're a single person who's doing the full stack, a distributed team where you're working alone, but you're online, or whether you're actually all in person, there are still lots of things that you can hand-off and share together.

Key takeaway no. 2

Though the book describes the process of building and using APIs as a straight line, this is not how an iterative development works. Depending on the nature of your development teams, you will probably have to find your own best way to read the book.

Did you struggle with the technical choices?

Casey Rosenthal: The technical choices, did you struggle with those at all or did Node.js, you know, the dark framework, was that just naturally the best fit for this kind of...

Mike Amundsen: I definitely struggled. I've taught this kind of class for years and originally, the very first time I taught, it was in C# a long time ago, more than 10 years ago. I actually settled on Node several years ago because it's adjacent to enough other languages that it makes sense. Even if you're a Java developer or a Python developer, or even a PHP, or a Ruby developer, it makes enough sense. I have to say the rise of Go and Rust have sort of challenged that notion because they're so different in their construction. So it definitely comes up and I sometimes get requests to teach the material using a different language.

So it was a bit of a struggle, but so far I think it's still the language most of us can still look at it and sort of getting the point even if it's not quite what we're into. So that's sort of the decision I made, and that's just on the language side. 

As you mentioned, there's a sort of technology in every single chapter, whether it's the testing tools or the authentication tools, or the sketching, or the definition tools. And I tried to get an arc of tools that you need to have in your kit, how you do diagramming, how you would do definition languages, how you would do sketchings and prototyping, and how you do mocks, and how you do test platforms. So I tried to touch on all those bases.

The biggest thing that worries me in a book like this is when I say things like use Node and use Postman, and use Auth0, and use web sequence diagrams, and so on, so forth, 5 years, 6 years, 10 years, these things are all going to change. So this is one of the books that breaks a rule I have which is it's tied directly to products that are on the market today. Who knows if they're going to be around? Hopefully, there's enough theory in here to help people say, "Yeah. Postman is not around anymore, but I'm going to do this instead." You know, and the book is still valuable. But it definitely is a challenge.

Key takeaway no. 3

The book provides you with a toolset that you can use. It focuses on Node, Application-Level Profile Semantics and YAML, but it can be adapted to different technology stacks.

Helping your clients

Casey Rosenthal: Yes. And I mean not to get too religious here, but in the book you used XML, YAML, and JSON. So which of those is best?

Mike Amundsen: You know, there's a great story. One of the key elements in the book is this language called Application-Level Profile Semantics. It's kind of a generic description language that can be converted into other things like OpenAPI and Protobuf, and things like that. And originally that is a format that I and Leonard Richardson, and Mark Foster, designed, and we designed it only in XML because Leonard Richardson and I go back a long way and we're sort of XML heads. Mark Foster who is a co-author with us said, "It's the first time we ever had to deal with XML," and that was sort of a marker. We both went, "Really?" So we started using JSON.

Recently a new project started using the ALPS specification, and they were totally annoyed by the JSON specification. They said, "Maybe you would like this. We just created a YAML format for the ALPS specification." So I actually see this travel through time and what kind of reflects some of that as well. 

I didn't expunge all of the XML because some people are still going to be using XML at some point along the way. I have lots of customers that still do XML, but it's become a dimmer and dimmer bulb in the series. And YAML has become much more common in a lot of things because of the way Kubernetes and some of the other languages use it now. It's really quite amazing.

Casey Rosenthal: You mentioned some of your customers still use XML. Was this book inspired by a need that you saw among your customers that, you know, I'm helping them do these seven things and API development and execution is one of the things that they seem to need particular assistance with? 

Mike Amundsen: A little bit. As I said, I still have customers who use SOAP services. They've invested heavily, they've got a lot of intellectual property built up, and it doesn't make a lot of sense for them to actually convert it just so that they have curly braces, right? So there's still a lot of that there, but one of the things I've noticed over the years is the idea of actual API construction is really different than constructing class modules or constructing code, or even constructing services. APIs are often just the external interface and often those APIs span multiple services. They're sort of aggregators or orchestrators. There's this kind of different mindset.

So I definitely wrote the book to help people get past the idea of singular service. I won’t belabor it a lot, but a lot of times in my class I talk a lot about the notion that we're programming the network. We're not actually programming machines anymore. We're programming the actual network. I touched on it a little bit in the getting started section, but helping people figure that out is sometimes a big ask or it's a big shift. And I will say, sort of going back to your original question, I recognize that conversation pretty clearly when I'm dealing with folks who have built a lot of services maybe 10 or 15 years ago. They were really optimizing for that machine, optimizing a service itself, but when you realize you're actually optimizing for the network -for lots of services - things start to change a little bit. And that's kind of the theme in the book.

Key takeaway no. 4

I wrote the book to help people get past the idea of singular service.

Sketching

Casey Rosenthal: That's cool. You have a vignette in there about Frank Gehry's sketching. Can you share a little bit about that? It struck me as something that you had come across personally and then incorporated it into your way of thinking about development.

Mike Amundsen: Definitely. Frank Gehry is a physical architect, does large museums, auditoriums, things like that. One of the things that he's famous for is his beautiful Disney auditorium complex in Southern California. And Frank Gehry is a real nut about sketching. There is actually a published book of Frank Gehry's architectural sketches and he does them on like napkins and little bits of paper. They almost look like scribbles. It's really quite amazing. I find them really interesting. I first got exposed to Frank Gehry through a colleague of mine, Ronnie Mitra, and Ronnie had done a talk about design. He actually even talked about this notion of disposable design and how design really starts with these simple sketch ideas.

So I included that story in the book, and it really does have an effect. It sort of added a new tool in my kit. When somebody says, "We need an interface, we need an API," I can just simply sketch out some curly braces and a couple of properties, or an angled bracket, or whatever they need, and say, "Is this it? Is this what you want?" And I don't have to worry about writing a lot of code, I don't have to worry about writing a lot of other things. 

Illustrations by Dana Amundsen.

So in my kind of discussion of it, you can do a sketch in 5 or 10 minutes, and show it to somebody. Just like Gehry would sketch the doorway or he'd sketch the pinnacle, or he'd sketch the building against a mountain. He wouldn't sketch the whole thing. He would just explore one element. So Gehry's ideas of sketching brought to me by Ronnie Mitra, really opened up a new door and I think speeds along developing APIs. Because now what I say in the book is sketches are made to be thrown away. You keep writing them until you find the ones you like, keep the ones you like, and then you a go on to the next step, which is that heavy prototyping where you're actually kind of mocking up what it's really going to look like. But I love sketches. I do lots.

Casey Rosenthal: Yes. That resonates with me. I really love that notion of the throwaway design and you and I touched on this previously, this notion of reversibility as a tool that you can explicitly optimize for so that you can speed up the way that you can navigate a complex situation like an API. This is a very reversible decision because there's not a lot of effort invested into a sketch and it's very easy to put that in front of somebody, get the feedback, go back and change it fairly cleanly.

Key takeaway no. 5

Sketches can save a lot of time and are easy to use to align with clients or stakeholders before putting too much effort into building something.

Reversibility’s role in building an API

Casey Rosenthal: You have that section in there on the deployment of the API and some of the CI/CD elements. Do you see reversibility playing a role in either the advantages of building things with an API versus previously that application spoke to each other or is that even one of the founding principles for the APIs just in the first place?

Mike Amundsen: Yes. The reversibility is super important like you just said. So a low-cost attempt, right? Sketches are a low-cost attempt. Even prototypes are lower cost than the real thing. The example I used in the prototyping section of the book is creating a toile or a muslin when you are sewing a complicated garment. Say, you want a suit or a gown for a movie premiere or maybe the initial listing of your startup on the stock exchange. You want to celebrate! You want an expensive and dazzling outfit! So you design an outfit but you don’t apply that pattern directly to expensive materials first. Instead, you use a muslin or cheap cloth as sort of your prototype. You do the cuts and the sewing and you try it on a mannequin. Now, you can reverse your mistakes quite easily when you are trying it out on this cheap cloth. When you finally get it all right, then you can commit to production.

So that really is a theme to the book and definitely covered in the deploy section. And there's a section after the deployment which is actually called modifying; what happens to the API after it goes into production. And that's a great case where you really need reversibility. So I make some changes to the API, run tests locally and it looks fine. I start to bring it into production -- maybe do some A/B testing or a small dark release -- and it causes problems. I need to reverse that out right away. 

Illustrations by Dana Amundsen.

So reversibility is, as you say, sort of part and parcel the DNA of a kind of an API space. I need to be able to move things, reverse things, bring things back to where they were before, on and off all over the place. Multiple versions need to coexist side by side. So there's a lot of that that goes on all the time and reversibility itself is a kind of a characteristic or a feature, I think, of the way the sites work.

Key takeaway no. 6

Reversibility is part and parcel the DNA of an API space. Make everything reversible until you finally have to commit.

Versioning APIs gone bad

Casey Rosenthal: Do you see organizations deal with versioning of APIs? I'm sure you have some stories of things going sideways. Do you have any good stories about where that didn't work out as planned?

Mike Amundsen: One of my favorite stories on versioning... And we do talk about versioning in the book as well. But one of my favorite stories on versioning is I was working with a company... I tend to teach, use versions only when you have to. Try to do backward-compatible changes so that people don't even really realize what's going on. That way you don't break anything else and so on, and so forth. But anyway, I was working with the customer and I noticed they had a version 2.3 or something in their URLs. And I said, "Well, how is that working?" Like, "What's going on?" And they said, "Well, actually that's not the current version. We're really on version 7 right now." And I said, "Well, how come all the URLs say 2.3?" And they said, "Well, we've got a bunch of customers that installed these remote devices and they can't go update those devices anymore. If we changed the URL we would break all the remote devices in the field. So we decided to just leave the URL the way it is." 

There are all sorts of things you just can't imagine. They never thought anybody would install their API on a bunch of remote field devices. It was actually stuff in lots of small convenience stores, bodegas, and things like that that they couldn't get back to anymore. They were sort of hardwired in.

The idea of versioning is an endless space of storytelling. I think the biggest thing that people need to keep in mind is if your API is popular, if your API is effective, you're going to have people using it who you've never met, they're going to be using it in ways you've never thought of to solve problems that you can't imagine. And any small change anywhere could just throw all of that out the window. 

Hyrm Wright works at Google and has this thing that's now called Hyrum's. He said something to the effect of once your API is deployed and it's used long enough, there will be some fatal dependency on every single aspect of that API, whether you had promised that in the documentation or not. So you have to be prepared for that in all kinds of ways. Change happens all the time. It doesn't always have to be through explicit naming or labeling, but you've got to have that in your plan as well. 

Casey Rosenthal: We ran into similar things at Netflix. I believe their streaming service still supports BluRay players from 2011 and so yes, a lot of those things are hard-coded in there.

Key takeaway no. 7

If your API is popular if your API is effective, you're going to have people using it who you've never met, they're going to be using it in ways you've never thought of to solve problems that you can imagine. Use versions only when you must.

APIs Security

Casey Rosenthal: You do have a section on security. One of my colleagues, Aaron Rinehart, says that in his opinion, security isn't a separate thing from engineering. Really it's just an outcome from good engineering. It still feels like from an API point of view, identity management and those other things are things that we put on top of the API after the fact. How do you think about the interplay between security, security solutions, engineering the security if you're building API in-house and API development? Does it come up earlier in the development cycle for you? Does it impact the shape of the API at all?

Mike Amundsen: Yeah. So the answer to that is yes to everything! It definitely comes up early and often, and it definitely affects the shape of the API. Security on API is a weird beast. Again, when you think of APIs as a sort of glue across lots of services, there's another level of complexity in the way you would think about security. 

So glue across a lot of services behind a firewall is very different than glue across a lot of services outside the firewall. Inside the firewall, I have a lot of customers that basically have some gatekeeping. Once you get inside there, inside the "inner sanctum", they let you wander around and mess with just about everything, which can really turn out to be scary as you build smaller and smaller services, or as you're adding services left and right.

I've got other customers that do a lot of external APIs and they have to drag around credentials for every single service if they're going to use a mapping service, a credit card service, if they're going to be using all these other things. So it's a real mess, it's a tangle, and this goes back to some of the original design of the internet itself and then the web specifically. There wasn't a lot of really thought on security, there was sort of an assumption. It was mostly research tooling and things like that. So we don't have good things builtin. We've got some good things bolted on, especially in the HTTP space, we have the ability to add all sorts of security schemes or things like that. But still, in design and implementation, it's a real challenge.

I have to admit, in the book, I really treat security as a kind of a simplified space. I tell people early on, assume that you're going to need to be focusing on security and keep it straightforward, and simple. So the rules I talk about are shared identities. So get some kind of identity provider service, whether using Auth0 or something like that, but then localize access control. Make sure every single service has its own access control rules that could be either localized or possibly distributed in some other way. And that's a super simplistic version of the story. 

But in the long run there are whole careers there. I've got lots of colleagues who spend their whole time on security. I love the comment you made which is this notion that security is sort of an emergent property or an outcome. Is that what you said? An outcome of good engineering. Is that what it was?

Casey Rosenthal: Yes. That's Aaron Rinehart's encapsulation of security, that it's not a separate thing. Security isn't a separate thing from engineering. It's just a byproduct of good engineering.

Mike Amundsen:I like it a lot. So getting back to the book. In the book, I focus on three just core elements, message encryption through TLS, shared identity through things like OAuth or providers, and then localized access control. And those are really the only things that I talk about there. There is a lot more. There's a real challenge if you're dealing with browsers. Again, browsers are part of this old, sort of like original school technology, keeping secrets in browsers. It's just a nightmare experience. So I have some sort of hints, some ways that people kind of work around it, but I find it a really challenging space.

Key takeaway no. 8

API security shouldn't be neglected. The book focuses on three core elements.

Casey Rosenthal: Well, Mike Amundsen, thanks again for joining us. It was a pleasure talking to you as always, and I hope we have the opportunity to do this again sometime soon. 

Mike Amundsen: Yes, I really enjoyed it, Casey Rosenthal. Every time you and I get together, I get to learn something new. I love discussing these things with you, and I'm really looking forward to seeing some of the other things in the book club. So hopefully we're going to get a chance to talk again real soon

About the author and interviewer

Mike Amundsen is an internationally known author and speaker, Mike Amundsen travels the world consulting and talking about network architecture, Web development, and the intersection of technology and society. He works with companies large and small to help them capitalize on the opportunities APIs and microservices present for both consumers and the enterprise.

Amundsen has authored numerous books and papers. He contributed to the O'Reilly Media book Continuous API Management (2018). His RESTful Web Clients was published by O'Reilly in February 2017, and he co-authored Microservice Architecture (June 2016). Amundsen's 2013 collaboration with Leonard Richardson RESTful Web APIs and his 2011 book Building Hypermedia APIs with HTML5 and Node are common references for building adaptable Web applications. His latest book Design and Build Great APIs for Pragmatic Publishing.

Casey Rosenthal is CEO and co-founder of Verica; formerly the Engineering Manager of the Chaos Engineering Team at Netflix. He has experience with distributed systems, artificial intelligence, translating novel algorithms and academia into working models, and selling a vision of the possible to clients and colleagues alike. His super‐power is transforming misaligned teams into high-performance teams, and his personal mission is to help people see that something different, something better, is possible. For fun, he models human behavior using personality profiles in Ruby, Erlang, Elixir, and Prolog.

He co-wrote the book on Chaos Engineering.