The Blind Spots of Platform Engineering

Blind Spots of Platform Engineering

Erik Wilde: Hello and welcome to a new episode of  GOTO Unscripted. This is going to be an unscripted conversation. My guest today is Matt McLarty, CTO of Boomi. Hey, Matt. Thanks for joining.

Matt McLarty: Erik, great to see you.

Erik Wilde: You're back after a very short while. Recently we talked about your new book, "Unbundling the Enterprise API's optionality and the Science of Happy Accidents." Great book. Watch the episode. Read the book. But we only very briefly touched upon platform engineering. I kind of mentioned it in passing when we were getting to the end of the interview, and I sensed that you found the topic interesting. I find it interesting myself. So today we're back and we're doing an unscripted episode on platform engineering, and we're calling it "Platform Engineering Blind Spots" to understand a little bit better where those might be.

My question to you was whether platform engineering, as it is mostly portrayed today, might have a little bit of a blind spot because there is not so much a focus on which APIs to provide throughout the organization to provide these options that new, interesting things can happen. Let's discuss this a little bit as a starting point, because then we can really kind of start from the premise of the book and then dive a little bit deeper into platform engineering topics.

Matt McLarty: There's a whole wide range of areas to discuss. I think platform engineering is the discipline of designing and building toolchains and workflows that enable self-service capabilities for software engineering organizations in the cloud native era. That's one way to define it. When you read that definition, it's very code-centric, very engineering-centric, very IoT-centric.

I think the first opportunity or missed opportunity is that business people actually understand the value of composable architectures and high optionality software environments. They picture a business as a platform with digital building blocks that are business functionality, able to be used in different ways to solve new problems, open up new channels, and exploit new markets.

One offshoot of DevOps has been this platform engineering movement. We bridged the development and operations divide with the DevOps movement. That was about these two functions working in isolation, and by getting better communication and optimization of the relationship between development and operations, we would be much better off. But the next big chasm is between business and DevOps - between the people providing the funding who have the ultimate outcomes and those building the technology.

Recommended talk: Effective Platform Engineering • Chankramath, Cheneweth, Oliver, Alvarez & Reisz • GOTO 2024

Erik Wilde: I completely agree. A while ago I gave a presentation where I talked about five different notions of platforms: IT platforms, internal developer platforms, API platforms, business platforms, and platform businesses. I did this to explain to people that if you start talking about platforms, take a minute to make sure that you talk about the same kind of platform because they are very different ideas.

In the platform engineering space, the word "platform" implicitly always refers to the internal developer platform (IDP), which is this efficiency thing. It's good that you can build stuff faster, but if you want to build valuable stuff faster, you probably need the right building blocks to build those valuable things. You have to carefully think about those building blocks that give you this optionality. Otherwise, you can build stuff faster, but it's not necessarily stuff that will provide value.

Matt McLarty: I think an overarching thesis of our book is around optionality. Optionality is an unmeasured property most of the time in our enterprise landscape. In the financial world, we have things like stock options, and optionality is understood as a valuable thing in markets. There have been studies by Dr. Carliss Baldwin, the preeminent expert in the field at Harvard.

What if we were to apply that and measure the value of optionality in this digital business context? If we accept that optionality is a really valuable property, then you want to have platforms that allow you to build that optionality in as well. That means environments where you're doing a lot of bespoke code development for one monolithic application might not harvest the optionality that you want.

The right way to orient the discussion for people deep into platform engineering concepts is to step outside that bubble, think about business terms, and especially think about the conversation you probably have to have with your business partners. If you took that technical definition of platform engineering to your business people and said, "We need money for this," I'm not sure they're going to sign the check. But if you tell them, "We're going to create a platform of business capabilities that's going to get you to your business outcomes quicker, more reliably, with less investment," then you have a basis for discussion.

Recommended talk: Unbundling the Enterprise • Stephen Fishman, Matt McLarty & Erik Wilde • GOTO 2025

Building for Optionality: Platform Engineering Lessons from APIs, Amazon, and Avoiding the Hype Trap

Erik Wilde: You said you have to build in optionality. Is that something where you would refer to things like the Bezos mandate? I remember during the book interview, Steven Fishman, your coauthor, gave this interesting idea that maybe you don't have to really issue the Bezos mandate that there must be APIs, but it should be the default. It should be the default assumption that if you build something, you build it so that it can be reused and provides options. And if you don't do that, there has to be a reason for that.

Matt McLarty: Yes, it really is a style of building and a default. If you're providing the platform to developers, what a great way to provide a forcing function for people to build things. You're providing an opinionated platform on how they can build.

We in the API world are very familiar with developer portals with APIs. Probably many developers in that code-centric world know about APIs and think about external services and utilizing those. They may not be thinking about things the same way within their own enterprise, but it is really a set of principles and design approaches.

Even across agile methodologies, at what stage are you actually looking through a catalog of existing building blocks to start your development? It can start as early as when you're doing business ideation. But even if you've already decided what you're going to build, at what point are you looking through the catalog of assets that you already have? If that's not a facility of your platform, then you've already had a missed opportunity.

Building with optionality versus without optionality is more in the approach that you're taking than a set investment. It's more of a design consideration.

Erik Wilde: It sounds very similar sometimes to the microservices discussion where it became this mechanical principle where people just divided everything up into very small individual services. Sometimes that was a good idea, and sometimes maybe it was not really necessary, but it just became this mechanical thing to do because everybody else was doing it. Is platform engineering kind of following in those footsteps?

Matt McLarty: This is not platform engineering's fault as a movement. I think this is the pattern of tech movements in general. Service-oriented architecture was a big deal 20 years ago - a set of principles around software architecture. People wanted to be really pedantic about it or they focused on implementation versus principles, like "I'm going to make web services everywhere, so I have a service-oriented architecture."

Then you go down Gartner's hype cycle into the trough when everyone realizes the benefits tied to the original principles didn't materialize because we abandoned the principles and just focused on technical aspects. So this must be a bad movement, right? Then those principles may come back in a different form - like microservices, which had the same sort of idea with new technology and angles.

This happens again and again. We run to one side of the boat and it tips this way, then we run to the other side and it tips that way. We observe success like Netflix and Amazon doing this thing with small, modular services they're reusing, and we call it microservices, when Netflix and Amazon were still calling it service-oriented architecture and actually abiding by the principles.

Amazon has platforms that they bootstrap for their developers so they can get up and running quickly. They have a platform that abstracts away complexity for developers so they can push code through easily. But there's so much more than that - they have platform teams and a very flat organizational structure with very autonomous platform teams. You can't just extract one piece of it and say that's going to work everywhere. You have to consider the whole context.

I think if you can go back to those case studies and say, okay, this is where platform engineering worked, what else was working, what were the other conditions that happened? I think Amazon is a good example of how they had a tight affinity with the business teams.

Matt McLarty They had very strong principles around this away team's concept of how they reuse pieces and APIs from other teams in the organization. So maybe that is the answer to closing some of the blindspots - go to the archetypal examples of platform engineering and see what else is going on that led to the outcomes, because I think people extract a piece of the mechanisms that are leading to the outcomes, but they just take all the outcomes.

They're like, Amazon did this when it came to platform engineering and they got all these benefits. But all those benefits came from a lot more than just the platform engineering piece.

Erik Wilde For sure. One of the things that I like to say is that every organization has a platform. There's always some starting point that developers have to live with. It's not like they're buying new servers and writing their own programming languages.There always is a platform that is just given to developers as the environment that they develop in.

Oftentimes it's not as refined as more advanced ideas of platform engineering, but it's there. People oftentimes - it is maybe, let's say, there's potential, which is the nice way of saying things that are fluid. Then people ask themselves, Amazon does all of this. Because of Amazon's Amazon, of course they have a very sophisticated model.

I think sometimes the easy assumption is to say, let's just crank up everything to 11 and it's going to be great. And I think even in theory that might be true, it's just a very bad way of spending resources and money if you try to do the most sophisticated way of doing that.

Plus you might be over-constraining things in a way that just doesn't make sense for smaller organizations that don't have the scale of Amazon. Stepping back and trying to better understand what the different knobs are that you can turn - coming back to that, I think if you would look at that as like different knobs you can adjust, then I would say there's this knob of making business building blocks available that is a knob that very often doesn't get turned up enough.

I think that's really the interesting part when coming back to the hype cycle, at some point we might actually see organizations looking, being in the trough and saying, oh, it's not nice here. What can we do? They might realize that if we actually do a little bit more of that, then actually that gets us to this nice plateau of productivity.

So is that something that you would say maybe can help as a general idea to create better business outcomes from platform engineering, or make sure that we get something that is not just technically really impressive, but also something valuable from the business point of view.

Recommended talk: Platform Engineering: A Deep Dive Conversation • Russ Miles & Kevlin Henney • GOTO 2024

Focusing on Business Outcomes Over Tech Hype

Matt McLarty: The focus has to be on the outcomes rather than on the approach. As technologists, we get excited by technology and solutions. We like to reinvent wheels. But you have to focus on the outcomes, which is very grounding.

The ironic part of platform engineering is that the principles are about starting small, measuring, iterating, following intuition, being ready for disruption and quick changes. Yet we want to implement these things in a grandiose way - we want to build a cathedral.

If you're an organization with legacy systems and a complex business model with many stakeholders, you don't want to just say, "That's the holy ground over at Amazon, and we're going to build all that." You have to ensure you're moving in the right direction based on business outcomes.

If your velocity is terrible and you want to focus on getting release sizes down and cycle time reduced because you keep missing market opportunities, do that - but recognize that platform engineering is just one approach to help you. I've worked with organizations obsessing over putting in CI/CD pipelines when they have a big manual testing cycle. The pipeline is not going to solve that problem - you've got to do the test automation first.

Erik Wilde: There's this great article by Richard Seroter about "shifting down" - taking the shift left to shift down. I really like that as the superpower in platform engineering - to really understand what needs to be shifted down and what may even be harmful if you do it, and identify those things where from a business point of view, these things should happen, but from an efficiency point of view, they should not happen in the teams. They should be something that the teams can just get done, supported by a platform.

I think that that would be the perfect platform engineer in my mind.

Matt McLarty:  I just had to grok the shift down concept. But that means that if you've got your product team who's building the sort of the, the or the streamline team and, and Team Topologies terminology, this would be what you want to shift down into the platform to, to drive more efficiency. Is that right?

Erik Wilde: Exactly. Just picking up on the shift left, topic right from DevOps that you try to do things early on so that they don't bite you later. People are really trying to shift left things, which is a good thing. But if 100 people shift things left, then maybe we can also shift them down so that they don't have to do them at all, because the platform will do it for that.

Matt McLarty: I like that idea of shift down. It gives us a good purpose for platform engineering and brings it back to the bigger picture of where platform engineering fits into a complex organization. We've been talking about business capabilities and APIs as an opportunity for platform engineering to factor in, but the organizational dimension is extremely important as well.

We have to think about this in an organizational context - what does your organization look like that's working with this methodology around how you ship software, with this platform that you built, with these capabilities that you have? It still comes back to business outcomes.

Matt McLarty: For those tuning in to learn platform engineering 101, this is not that - we've assumed some prior knowledge. There's so much goodness in the platform engineering movement and literature, but always keep your eyes on the prize. What are the business outcomes you're after? Don't do technology for technology's sake. Don't argue pedantically about terminology. What are you hoping to accomplish?

Sometimes platform engineering literature shifts down too much, assuming business outcomes are taken care of. Having the big business picture outcomes always in mind is the number one grounding thing. It helps you address blind spots and come up with the platform approach that works for your organization given all the context. APIs factor into it, and it's definitely something we talk about in the book - maintaining that focus on the business, because that's who's going to write the check for the platform anyway.

Erik Wilde: I agree with everything you say. It's all about why we're even doing this, and you better figure out where the biggest gains can be made. But sprinkling a little bit of that API goodness over the platform engineering movement would help with proving business value, making sure valuable things get built, and making it more clear why this whole thing exists - not just to be fast, but to create value fast.

Matt McLarty: It is a fair litmus test - if the only APIs in your platform are APIs for the platform services, how are you supposed to get to business value quickly? We've been building API portals and API catalogs in organizations with business functions for years, and what have we been building? We've been building platforms.

Erik Wilde: When you have platform conversations, spend a minute just level-setting everybody on what kind of platform you're even talking about. Otherwise, you will have very confusing conversations.

Matt McLarty: Really appreciate the conversation. It's always fun to talk with you, Eric.
Erik Wilde: Thanks so much for being here, Matt, providing us a little bit of insight into optionality and the science of happy accidents and how all of that connects to platform engineering. Thanks, everybody, for listening. And check out the book interview in the GOTO Book Club and also, of course, other GOTO Unscripted episodes.