Expert Talk: Managing Complexity in Software

Complexity of software systems sometimes grows beyond control. Left unchecked, it can leave behind bloated applications.
Kevlin Henney talks to Hadi Hariri, developer advocate at JetBrains, about how some of the key traits of developers like creativity and problem solving make them prone to innovate more but also over-engineer their code and not choose solutions based on context.

Complexity in software

Kevlin Henney: Good morning. Good afternoon. Good evening, wherever you are, wherever you are. Welcome to another GOTO Unscripted session with me, Kevlin Henney, and today I'm joined by Hadi Hariri from JetBrains. So, Hadi, please introduce yourself, tell us a bit about what it is that you do at JetBrains and, the things that interest you at the moment.

Hadi Hariri: My name is Hadi Hariri. I work at JetBrains on developer advocacy, which is, for folks that still don't know, it's helping developers get a grip on technology and advocating for our tools, etc. And what interests me at the moment? Guitars. I've been around the industry for, I can't even remember. I'm old. I got more gray hair. Every time I go to the hairdresser... I remember when I first went to the hairdresser, and they were cutting my hair and I'm like, "Oh, look, there's a gray hair." And now, it's like, "Oh, look, there's a black hair."

I've been in the industry for a very, very long time. I think my latest passion has been seeing how we constantly go around in circles. We're just kind of like entangled ourselves in a web of complexity that we can't seem to get out of. We just keep increasing.

Kevlin Henney: I think it's interesting. The complexity thing is...I don't think I ever had to talk about complexity until I entered software development. I mean that in the kind of like when it comes up in conversations or even stuff at a university, and I didn't do computer science the first time around at university. But I remember the first job I had after university. One of the greatest insights I had from my project manager who was technical is he said, "Kevin, actually, all of this, it's about managing complexity." I saw the summit of the mountain and understood, but it really reframed the whole software development.

So, I said, "Okay, that's what we're trying to do is doing that." And then there's kind of the follow-on question was like, "How good are we at it?" It seems to be that we are very good at manufacturing or creating complexity.

Hadi Hariri: Oh, yes. We're the best.

Recommended talk: Don't Walk Away from Complexity, Run • Venkat Subramaniam • GOTO 2018

Kevlin Henney: It's just like, "Do you see that simple problem? We can take that and make it more complex." So, when you think about complexity, when talking about developers because as a developer advocate, that means you're in contact with a lot of people, and you see a lot of different things. And people are going to ask lots of different questions or come at you with a different angle. I guess sometimes there's a hidden agenda that they don't realize, and it's, you know, how does this tooling or this language or whatever solve this. And you kind of scratch a bit further and you're thinking about, "Well, what are they actually asking?"

You scratch a little further, and they're probably asking something that has to do with a hugely creative complexity of theirs. It's just like, "Oh, okay, right. I would not want to solve that problem because I wouldn't want to be there. But now you've got it. That's kind of interesting."Brooks called it accidental complexity as opposed to essential complexity. Do you think that's endemic in the industry? Is that just part of what we do? Or is it part of who we are?

Hadi Hariri: I think it's part of who we are with help from the industry. The majority of us that, got into software development, I guess...I mean, I speak for myself. I got into it when I started to realize how much I love automating things. "This is brilliant," you know, "I can automate stuff." And then at some point, it clicked like, "Oh, look, I can automate stuff and help people be more efficient. And on top of that, I'm gettng paid for it."

Kevlin Henney: It becomes a profession. It becomes a career.

Recommended talk: How to Read Complex Code? • Felienne Hermans • GOTO 2021

Hadi Hariri: It's brilliant, right? But that idea of tackling a problem and trying to find a solution for it, I think that's part of us, right? And we all enjoy that. The problem is that often I think we get lost in forgetting what the problem is and just trying to play with the solutions that the industry offers us.

Kevlin Henney: That's an interesting one because that's one of the observations that I've had. Because occasionally I get asked, "What is it that makes a good developer?" You know, "If you're gonna be a programmer, what is it that you...what are the things...?" You know, there's historically been, "Oh, yeah, you need to have good mathematical skills." Not so much.

There are all kinds of different things. You eventually meet so many people and you go, "Oh, okay, actually, it's not any of the standard things." But there do seem to be a couple of common threads, and one of them exactly as you said is that problem-solving. There has to be something within you that probably drives that. Tat's regardless of any kind of other backgrounds you have.

I think one of the other things that make the perfect storm is that perhaps there's also an element of creativity. And you put those two together, and you're able to create some amazing problems. It's a perfect storm. 

Therefore, we love that kind of tinkering, that possibility of like, "Oh, I wonder what this tool gives me? I wonder what this API is? And what if? And if I do this and if I have this call back to..." It's very easy to escalate. You forget the original problem, you get into the flow state, but you get swept away by the flow.

Hadi Hariri: As the saying goes, you know, "A solution looking for a problem,". I've got this brilliant thing. I don't know what it's solving, but it's brilliant. It's fantastic. So, I think it is a combination of both. I don't think it's ill-intentioned. I don't think any of us go about saying, "Oh, you know what? I'm going to come up with the most overly complex architecture I could possibly come up with." I mean, imagine that in any other field, you know, like, you know, building bridges, "Oh, I'm not just going to build a bridge to cross this pond., I'm going to over-engineer it in case tomorrow we need something else."

We do that all the time, all the time. Yesterday, actually, we were doing a recording of my podcast, and this section was brought to you by Talking Kotlin podcast.

Kevlin Henney : Check it out.

Design patterns in software

Hadi Hariri: We were talking to the guest that had written a book on design patterns. We were talking about how the whole Gang of Four and what is the worst thing that you've encountered over your lifetime in terms of overuse of design patterns. I clearly remember, when I was doing consultancy, going to a customer that had actually a code base that had organized their code based on folder names of design patterns, right? No, I'm not kidding.

Kevlin Henney: Okay, I'm gonna just roll my jaw straight back up there. That trumps the story I have on that one.

Hadi Hariri: The guest was saying about how someone had said, "Oh, I've actually managed to use most of the design patterns in the Gang of Four book. It reminded me of that time when I had... You could structure your code based on features, on MVC, whatever. No, this was on design patterns. Like I have all my decorators here. I have all my strategy patterns here.

Kevlin Henney: Wow. It's an interesting one because I think this kind of ties into sometimes the way that we create...we mistake one thing for another. So, I've been into design patterns for a while. In fact, I remember the time before design patterns. It has always distressed me that people have misunderstood them. Some people treat them as a checklist. There are always a set of principles I need to check off. And it's just like, "You do know there are more than 23 design patterns?" And actually, the 23 in that book are really very contemporary or well-aligned. It's a good first step. It's that book is 30 years old as of next year.

It’s really impressive. But they did not intend...that was never their goal. When you understand design patterns you're supposed to be solving problems. Then people are sort of, "Oh, it's a checklist." I remember my own kind of horror story, but, honestly, having folders organized by design patterns, beats...

Hadi Hariri: Yeah, that was...

Kevlin Henney: We talk about that in "Pattern-Oriented Software Architecture, Volume 5" that I wrote with Frank Buschmann and Doug Schmidt. We kind of talk about how people misunderstand it and they start thinking of these things as like checklist items, or, "Oh, they're like components, and therefore you add them to one another." We actually demonstrate you cannot have exactly the architecture you described. That's actually not possible if you are writing sensible software.

But I remember just one company. I remember looking at the code and I just couldn't understand what was going on, and I thought, "Oh, I've just been in the car for two and a half hours. Maybe I'm just staring at the road. And maybe I'm just missing something here." Eventually, it was a very slow penny that dropped, and it was a notification relationship based on observation and it was built up to be something like MVC.

Recommended talk: Applying the Saga Pattern • Caitie McCaffrey • GOTO 2015

I said, "You got the callbacks going the wrong way." And the guy said, "Oh, oh, right. Oh, okay." I said, "But, you know, it's actually fairly clear. You got to really understand the motivation of this and read through that." He said, "I'm a senior developer. I don't need to read the words. I just look at the code in the book and then..." And it's just like, "Well, clearly, that's not how this works because it's all a discussion about what you're supposed to be thinking, and then here's some code to demonstrate what we're thinking. It's not a copy and paste exercise."

They eventually adapt. I remember discovering that that team had actually pretty much did a pattern a day. There are 23 design patterns officially in that book. And it's about 23 working days in the month. It was pretty much, "What day is it today? You know what? I think today is singleton day. Didn't we have that last Tuesday? We always fill the extra days with singleton."

Hadi Hariri: And everybody had a printed-out calendar.

Kevlin Henney: I remember mentoring some developers a few months later from this team. And when I hit the section design patterns, everybody went like this. And I thought what has happened? How's the earth been so poisoned with these people? And then they told me some of the designers, "Yeah, there's this bit where there should be a flag. So we use the state pattern." I was just like, "Actually, what you want is a flag, use an if-statement. That's it." Sometimes an if statement is just...

Hadi Hariri: It's an if-statement. Exactly.

Kevlin Henney: And that's just fine.

Hadi Hariri: Exactly, yeah.

Kevlin Henney: They had created this amazing thing. And it was very enterprisy in the worst sense of their word.

Over-engineering your code

Hadi Hariri: I talk about seeing this in other codebases. I've done it myself, right? I mean, when I started out, and not even when I started out, for many years, I was over-engineering everything. I was creating these abstractions and I was creating interfaces that only ever had a single implementation. I wrote the other day on this social media, it's called Twitter. You've heard of it?

Kevlin Henney: I've heard of that one. 

Hadi Hariri: said, "We build in these flexibilities and clean architectures to try and use them at some point in the lifespan and end up creating these complex code bases that don't make use of everything that we've foreseen, right?" And I look at my own codebases and it was exactly that. I create these monstrous code bases of abstractions and flexibility and throw in a DLL or a JAR and it'll instantly recognize it. Did I ever use that? No. Did I ever need it? No.

Kevlin Henney: It's the whole what if. And this actually, interestingly enough, goes with what I mentioned about creativity. It turns out, that the more imagination you have, the worst this gets because you're imagining all the possible futures. And so, your skill at imagination...there's absolutely nothing wrong with imagination. It's an immensely powerful tool. But the danger is we kind of take speculation and embed it as a commitment in the code. 

This is going to be a confessional. I've definitely done that and I've kind of looked at those things. Sometimes you kind of look at them a few years later, and it's always one of those things. It's always good to go back to things that you wrote, or, you know, worked on in the past. I kind of get that kind of sense of, like, "And how do I feel about that now?" And it was...we often don't do that, but it can be quite healthy.

Hadi Hariri: I wasted my life.

Kevlin Henney: Well, okay. At that point maybe I should have just sold ice creams and made more people happy. Who doesn't like an ice cream vendor? But there's a humility that comes with that. Sometimes get that sense of I was wrong about that. You talked about helping other people, and sometimes the thing is...maybe if we can't save ourselves, we can save somebody else.

I had this opportunity with a team a number of years ago. I visited them over a number of years. One of the most interesting things is we got to the end of the sunset period of that, you know, It's been over a decade. It wasn't a full retrospective and one of the guys said, "You know all those times you said we were over-engineering and that...yeah, you were right."

Because we also said to any of those decisions...did any of the shortcuts take them to deadlines, and any of the overengineering, or the extensions and generalizations that you put in that we discussed and debated, did any of those work out? And it's just like, "Oh." That was a hugely important thing, but they became one of the best teams I've ever worked with. But they found that actually just keeping things kind of small and simple and comprehensible and tested and all of this kind of stuff was actually. It took an immense amount of discipline. It turns out it's harder to do that than it is to create complexity. But that was the thing that actually they said was the biggest game-changer in the long term.

Hadi Hariri: We have this at so many levels. Before, I was listening to some of the interviews, and every time you talk about software now, you hear Kubernetes, you hear Docker, you hear microservices, you hear nanoservices, picoservices, you hear whatever services you want, right? And it just sometimes makes me wonder why are we doing all of these things? 

Recommended talk: When To Use Microservices (And When Not To!) • Sam Newman & Martin Fowler • GOTO 2020

Kevlin Henney: Why are we doing this to ourselves?

Hadi Hariri: Back in my day, back in the old days... I mean, even the web development stuff, right? We used to do this thing called...there was an application, there was a CGI and it used to spit out some HTML combined with data. And it is used to render a webpage, right? And it used to work. Then someone said, "No, users need instance, instant updates. So we're gonna invent this whole thing called single page application, right?"

Twenty years later, you're coming back to articles that are saying, "Oh, to get better usability and performance, we're gonna do this thing called server-side rendering." It's like, right, that's what we used to do 25 years ago. It was called CGI. CGI is serverless now, right? Do you remember CGIs?

Kevlin Henney: Yes/

Hadi Hariri: So you used to call it an executable, right? Now you call it a function and they call it serverless. And now, we're driven to this whole cloud-native, which for me is fascinating. Because, if you take a look at any of the cloud providers... I'm not single pointing out anyone, but you take a look at Google Cloud providers, you take a look at Azure, you take a look at AWS. AWS, the other day, I was scrolling through the services they provide, it's like 20 pages of services that they provide that you don't even know...

You have to take a course to figure out what it is that you need to learn to then take a course on that to then figure out what you need to do. And then by the time you finish all of that, it's already obsolete, or legacy, and they've come up with a new service. Take that, because when I was saying to you that the industry kind of help motivates us to do this, right? Take that, combine it with our eagerness to solve problems and to automate stuff, and you're like, "Oh, look shiny little toy. So, I can play with all of this and put all this together and create this massively overcomplex, scalable architecture that no one's ever gonna use. For what?"

Kevlin Henney: I think that that's, that kind of idea of trying to understand there's a feedback loop that we're sometimes missing. Exactly as you said, the scalable thing I think it's absolutely fascinating, because many of us, "Oh, well, we do this to be more scalable." Well, what is it that you're doing? You know, do you have a scalability problem?" "No." And it turns out that they are borrowing, the standard thing, go to a conference, you see the big names, make their recommendations or talk about their architecture, and these people are, you know, the Googles and so on. They are doing planet-wide engineering at this level.

And the point is that what is it that you're doing? You know, I know somebody who wrote software for a funeral company. 

Hadi Hariri: How scalable is that gonna be?

Kevlin Henney: How scalable do you expect that to be? By the time you start hitting that kind of curve, you have...trust me, software is the least of your worries. That was the whole thing. He developed one generation of the product. And now a larger team, two people developed the first generation. Now a larger team was in and they were rewriting and it was all microservices this, microservice that. And he said, "Well, they've spent an awful lot longer apparently doing a lot less, but there appears to be a lot of code. Most of the code is as it was feeding the frameworks." The framework says, "I want to do..." The framework that was supposed to make things easy, you're now bowing before, "Oh, you want this, you want this, I need to feed you that, and I need to respond to that."

And suddenly, it's not that there is no case where these may be reasonable. But the result for the developers is that they are constantly feeding this, and somewhere in the heart of that is the idea of the business and the user. It's sort of lost in there. There's all of this stuff. And that's a really common experience. I see it in the blog space as well. I see lots of practices that are being recommended as good, that are reversals of the simplifications that we've been encouraging people for a long time.

And it's kind of like, what drives this? And it's again that idea that sometimes...you know, you say the industry, but also, sometimes when I make something look intricate, it makes it look like it's a solution. And again, going back to what you said, it's a solution in search of a problem. I've seen so many recommendations there. And you say, "You know, why you don't just create an object here?" Oh, but you might want to change this and that, so we're going to parameterize it and do this. And before you know it, you know, you're shooting up and injecting everything. And it's just like, but you could just create an object just right there, right there in that line of code. And that would be it. And that, you know...

Recommended talk: Modular Monoliths • Simon Brown • GOTO 2018

Hadi Hariri: Yes, I know. 

Kevlin Henney: It's addictive.

Hadi Hariri: Yes, it is. And it's conferences... I mean, I remember clearly...I remember when this happened. It was kind of like the first time I had heard of Uber, right? Because, I mean, I lived in Spain. Well, I still do. Uber only came to Spain a couple of months ago. We got a strong union, taxis. And there was this blog post about how Uber went from Postgres to MySQL for performance. It was back in 2013. I just saw a whole bunch of folks saying, "Oh, yeah, I have this problem." "No, you don't." "No, I do. I gotta move to MySQL as well."

And then another one's like, "So why are you using MySQL?" "Well, because Postgres is crap." "Why?" "Well, look at this blog post." Right, okay. 2016, Uber comes out with another blog post moving from MySQL to Postgres. Now, it could have been the other way around, or whatever, but in the interval of three years I'm like, "You could have just not read that blog post and saved yourself a lot of money." And it's, like, we just copy everyone, right? Because it looks cool. It looks interesting. We're going to do chaos, monkey engineering. Do you ever need that for your funeral home? No, but it's brilliant, isn't it?

Choosing a software solution in context

Kevlin Henney: And I think that's the interesting thing. It goes back to the kind of like the not getting the point of design patterns type of thing. It's another one of these ideas of not understanding the full idea is that a lot of these people, you know, the Ubers, they are dealing with a particular scale. Well, these people are solving problems that they have, but it doesn't mean it's your problem.

Hadi Hariri: Exactly.

Kevlin Henney: We should take these as data points in the large landscape of that person in this situation who had this particular problem. They identified it this way. This is the pain it caused them. This is where they started. This is where they ended up. And their journey continues, and they may do an about-turn, which is exactly what you've described. But that's the point. Now, the question is, do you have that problem? Or are you just looking at the surface waves, the surface ripples, the easy thing, which is, "They made this migration? We must make that migration?"

We like to copy things. That is one of humanity's greatest skills and failures. You know, copying, it's one of the things we're really good at. It gives us stuff like civilization, written language, and all of this kind of stuff. It's just, like, yeah, copying, absolutely amazing. But it also gives us fashion and overuse of microservices and stuff like that.

Hadi Hariri: And we have the same with microservices. You see now articles about going back from microservices to monoliths, right? I mean, you know Simon.

Kevlin Henney: Yeah, I know Simon.

Hadi Hariri: And he said it clear. He's like, "If you don't know how to write proper monolith, microservices aren't going to help you. You're just gonna end up with a lot of little..."

Kevlin Henney: Really bad.

Hadi Hariri: ...bad monoliths." That's where we are.

Kevlin Henney: I think that's a really interesting observation. Because, again, it goes back to one of the most important things for me that I learned originally in trying to learn about patterns, not just this Gang-of-Four stuff, and actually going back and trying to understand is that, for me, the greatest insight I had was trying to understand, "Oh, this is about context." What you're trying to do is say, "I've got this problem in this context, I don't just have this problem, I have this problem in this context."

And if that context changes, then the nature of my problem changes. If my context is different from your context, that context can be anything as details of a programming language. You know, I'm in a particular programming language. I am following these practices. So if I'm in a programming language, where I have to manage the memory myself, there are going to be a set of problems that I have. And there are also probably a set of practices that I could follow that keep me on the straight and narrow.

But I can't simply transplant that to a managed language and say, "Oh, look, I'm using the same techniques because it's just like..." Well, no, you have some memory problems, but they're different. The tools available to you are also different. In other words, you are not in the same context. With this idea, we like to have the one rule. You know, one rule and we're going to solve it that way. And we've ended up with exactly this.

So, sometimes people are moving to microservices for the right reasons. But whenever anybody says we're moving to microservices because we've made a mess of engineering something that's a monolith. This makes you think that you're going to not make a mess of this? If you're moving to microservices for, the elasticity, the flexibility. All the other kinds of performance characteristics that hits the mark, microservices will always make a piece of code more complex, or they will make the architecture more complex.

But there's the equivalent, put it all in one place. But then if you're doing it for reasons of structuring, maybe you get lucky. Maybe you do end up with better architecture. But the chances are you're going to end up with the same problems because it's probably developed by the same people.

Hadi Hariri: It was our code all along. It wasn't the platform. It wasn't the architecture. It wasn't any of the databases. It was our code.

Kevlin Henney: We have met the enemy, and he is us.

Hadi Hariri: Yes.

Kevlin Henney: I think that's the interesting thing that many people are not confronting this. And that's, you know, one of the lessons of history, given that we both have gray hair. One of the lessons of history is watching it being repeated. There are a couple of observations, the classic one being, that those who do not understand history are doomed to repeat it. But there's the counterpoint to that, which is those who do understand history are doomed to watch it being repeated. We can't always extract all of the right lessons, but there is this thing. And you kind of threw in the picoservices, nanoervices thing earlier. I thought that was an interesting one.

Hadi Hariri: When the whole microservices came out, there were times that you had to facepalm when you would hear conversations of, like, how many classes constitute a microservice? And how many lines of code? Is a class with a single method a microservice or is it a nanoservice? Why are we having these types of conversations? 

Then, on top of this, you talk about patterns, and we talk about architectural patterns, and you talk about microservices. And then when you, like, the CQS, right? Command and query separate. Many years ago, I'm pretty sure you were familiar with CQRS that came about, right?

Kevlin Henney: Yes.

Hadi Hariri: Which if you were around back then, which you were, we were, if you would actually Google CQRS, Google will respond, "Did you mean CARS?" Like it used to say, "Did you mean CARS?" There's actually a blog, which is cqrs.blogspot or something like that, and the tagline is, "Did you mean CARS?" Because Google didn't know what CQRS was, right? And so, Greg Young, who was talking about CQRS, was also doing event sourcing. Then people thought that what if I'm doing CQRS, I have to do event sourcing.

They started to like add this complexity of event sourcing to situations and contexts that didn't apply. And it's just like, "No, we got to do CQRS with event sourcing, and it's not working out well all because you're lacking domain-driven design." Now, you got to throw in the domain-driven design. Now, you're gonna get all of that. We keep seeing how we keep adding these levels of complexity to things instead of just thinking about, "Hey, you know what CQRS is? It's just about having your reads separate from your writes. It's about not needing the ORM at times. Right?

We do the same with microservices. "Are you using microservices?" "Yes." "Then you need to use, you know, Kubernetes." "Why?" "Well, because you can't just, deploy a container." "Why?" "Well, because you need Kubernetes for scale." "Okay, but I don't have scale." "It doesn't matter. Just in case you need it. And then you need all of these monitoring services, and you know, you need to..." Yeah.

Kevlin Henney: Exactly as you say, there's this kind of you go to pick one off the shelf, but then a bunch of other stuff comes with it. Have you thought about this, sir? How about one of these?

Hadi Hariri: It's like a car showroom and it's like, "Oh, you're getting a car? You might as well get this outfit."

Kevlin Henney: You walk in. I'm after a motorbike, and you walk out with just a campervan. And that's the point every single one of the ideas that we're talking about is individually useful. It has an appropriate place, an appropriate time, and a place. There's no criticism of these things at that level.

It's that kind of like shiny hammer syndrome. You eventually end up asking the question: is this the right hammer for this screw? It's a complete mismatch. But more importantly, you just then start dragging all these other things by association. Oh, if you're doing this, you do this. It's worth considering them, but you don't necessarily need them. And that's where I think going back to the heart of this, the complexity, where does it come from. Is this aggregate kind of quality. It's, like, it's got this gravitational field that pulls in all these other things.

As you say, CQRS, in principle, is a profoundly simple idea that can benefit a whole load of architectures. "Hey, you have a different read stream from the write stream. You can optimize them and have different technologies, as long as you're happy with the data quality differences." 

This is brilliant, absolutely fantastic. And suddenly, you've got something else there, which, you know, the big break there is breaking from, you know, kind of acid transactional tyranny, let's call it, where we have to lock everything and lose all kinds of scalability.

And this is great for a large class of things. That's fantastic. And suddenly, it's just like...are we doing this with microservice because that sounds like microservices, and then you draw in that domain as well. But I think that the funny thing for me with the microservices stuff is watching again that historical aspect being repeated, because when people coined that term around...the first time I saw anybody talking about microservices was Fred George, and that was about 10 years ago.

Recommended talk: It is not just Microservices • Fred George • GOTO 2016

He had a really clear idea. He was very clear about how he was doing it and what he was doing. And it was genuine to do it. Like we want more pieces. We want something that an individual developer can write easily and could be quite happy to pretty much throw away and write again. The word micro was justified in what he was advocating. And these days, when people talk microservices, you kind of look at it and go, like, "I remember the whole kind of services the first time around, and you've now created stuff that's now bigger than what everybody was accusing, you know, and so, now we need to scale down. We're at pico and it goes down to...

Hadi Hariri: Yeah, and it's so tightly coupled that if one service goes down, the whole system goes down. But that defeated the purpose of the thing now.

Software engineering cycles

Kevlin Henney: It's interesting because there are a couple of phrases that... Frank Buschmann and I did a talk where we actually took an excerpt from Clemens Szyperski's book on component software in the late '90s, a really important book, really talking about the thinking and so on, and the rationale behind it. A lot of that stuff also seemed almost old because he was very good at giving a historical context and a narrative there.

And it's funny. We took a paragraph out of it, and just what is a component, and it was identical to what is a service just a few years later, the SOA-style service, and is identical to what we would now call microservices. It was so funny, it was just this particular paragraph. You only had to change no words or one or two words, and it was just like, "We've been here before." And it's not that none of these are any...they all make progress in some way. They're all helpful to a large class of systems. But when people can't, they say, "This is going to help us." "In what way?" "We're not sure. Because other people are doing it, we have faith, we will be blessed by the right outcomes."

Hadi Hariri: It reminds me of when you said, the definition of a service I remember back in the WCF days. Do you remember WCF? And so, they had come up with the four tenants of services by Microsoft. It was WCF that complied with the four tenants of services. But of course, Thomas Earl had 11, right? But you didn't need 11. The four were enough, right?

Kevlin Henney: Actually, I have forgotten that distinction. Yes. There's an interesting kind of cycle there. It's almost like there's some merry-go-round. And what we want to do is each time round, we want to be aware that we're going round and "Hey, this time around, we're doing it like this and we've called it this, and we're totally aware of that. Let's not fall into the same traps regarding silver bullets." And, you know, "Hey, guess what? You know, this is the silver bullet that will do is." Haven't we fallen for a bunch of silver bullets before? I'm pretty sure we have, you know.

I've got the kind of the wounds to share it. Let's just understand the thing that this does give us, and then there are all these other things, and be really clear in our minds that these are not necessarily going to happen by magic. This is why we want this technology because this is the difference it makes to us, you know, as opposed to all of these other things, which genuinely get a bit cargo culty.

Hadi Hariri: There are many people that argue unfairly that we're coming up with new solutions, we're trying to find new ways to solve the same problems, and so on and so forth. But one of the things that I've found a lot again with us is that, you know, let's say that I have a framework, okay? And that framework works for 80% of the cases. And now I need a special case that doesn't cover it. So, I won't, you know, make a special case. I will create a new framework that covers my cases and a little bit more.

\I do that at the expense of complexity. I do that at the expense of I'm not taking something that we know, building on it, and abstracting the complexity away. I'm making the developer have to deal with that complexity. I'm making them have to deal with the complexity of learning the framework, learning everything underneath it. And I don't understand that part. You know, it feels like we are trying to every time we talk about applications and user experiences, we always try and say, "Let's make the experience more pleasant for the user. Let's not make the user have to understand the application and, you know, know that they've got to go to three menu items to accomplish what they want to do."

And yet, we don't seem to apply that to ourselves. Like, as developers, we don't say let's try and abstract away things to the point that it makes it simpler for people to use, as opposed to...no. You know, I mean, if you take a look at any of the frameworks. We were just joking earlier about the I gave a talk on AngularJS. Yes, it was a horrendous mistake I made, right. And then React JS came about, and then there was this flux architecture of React JS, which is this massive diagram of all of these parts. And you're like, "Why? Like, why do I need to understand all of this, right? Why?" I mean, it was easier to understand machine code. Seriously, the 68000 Motorola suddenly was easier than programming with the React JS. I don't get it. I really don't. I think we just...like we enjoy complexity.

Kevlin Henney: Yeah, I think it comes as a reward. It's like a puzzle. You know, again, that's the other aspect is like. These aspects are creativity, and problem-solving. And then in that we find puzzle solving. Complexity has its own reward, its own intricacy. And they can be a lot of fun when you created something there. And that and software development is one of the most creative professions that we have. You literally can create something from nothing. You can summon things from the universe, you know.

In times past developers would have been burnt at the stake for casting magic. Some people may have the opinion that that should continue to be the case. I'm definitely not going to advocate that. But we do have this invitation to complexity. Because there's so much to know. I mean, that's the thing is that there's so much to possibly know. But perhaps sometimes we don't make it easier. Although, going back to your point, earlier on, nobody ever sets out to over-engineer things. Nobody has the meeting and said, "So we're going to develop this. Any suggestions for how we should do it?" "I think we should overengineer it beyond comprehension, but we should do it under the veil, and mandate of simplicity."

Nobody's going to say that. And even if somebody does say that and you're in that meeting, you know what to do. You stop. And actually, if it's a Zoom meeting, and you happen to be the host, you can just kind of accidentally kick them out. Oops, your connection was dropped. But that opportunity doesn't happen. That's not how it happens. It happens over time. And people go in with such good intentions, every single framework is always an attempt to simplify, but then things start leaking out leaky abstraction. And you sort of end up having to say, "I now need to know the universe in order to use this." There was a common cycle there.

Recommended talk: DevOps & Software Architecture • Simon Brown, Dave Farley & Hannes Lowette • GOTO 2021

Hadi Hariri: Then even like with presentation patterns. We talked about MVC, simple concept, okay, MVC. Now I'm doing desktop application, MVP. Okay, now, MVP doesn't quite work. I'm going to do MVVM. Okay, what's MVVM? Well, there's this long blog post of 17 pages that tells you the subtle differences between MVP and MVVM. And then, you know, six months later now MVVM sucks. I'm gonna do MVI. And when you look at it, it's basically the same thing. Just someone decided to call it differently.

Kevlin Henney: Yes, I must confess that that's been my...the funny thing is that a lot of it is like, "Wait a minute, isn't that just a variation of that?" Should we not acknowledge that? But in other cases, there's a lot of MVC stuff you look at and go, "Well, I haven't been familiar with the original MVC. It's just like, "It's not quite what we had in mind." But, you know, there's calling different things the same name and then there's calling things different names that are actually essentially variations of the same idea.

Naming is one of the hardest problems we know. But there is that idea. We like to explore the space. As you say, we're all trying to solve different things. But it's a sense of, like, we are repeating some of the same mistakes rather than finding new mistakes to make.

Hadi Hariri: And that's a problem. That's a problem.

Kevlin Henney: That's the thing we should be doing there's a lot of innovation to be had, but we spent a lot of time... I think it's necessary. We always repeat certain mistakes and certainly as an individual in the profession, you have to go through certain mistakes that you're going to make. Nobody can think for you. Nobody can understand your code, yet you have to go through... But it does seem collectively we are capable of making some of the same mistakes. It's kind of like, "Shouldn't be marked that one as done so we can make new and exciting mistakes that actually push us into different spaces rather than hovering in the same areas?"

Hadi Hariri: Then when you, you know, you've been in the industry as long as we have, and you start saying, "But that... We kind of..." And you're like, "Oh, shut up." You know, an old man screaming at the cloud." Oh, I'm not really. I'm just...yeah.

Kevlin Henney: I actually did have that one experience, to be honest. I thought when somebody showed me something and said, "Oh, you're doing that. Have you had this problem?" Rather than say, "Oh, yeah, we did this back in the day." Shouts at the cloud. You know, it's just like, "Have you had this...". It's just like you've seen our code. Yes, it is, 40 years ago. So, there is an element there. Yeah, it's interesting because we're a knowledge-based profession, and yet, surprisingly...and we are very good at individually learning certain things, just like developers are always tracking moving targets. So, their skill at learning is great.

But there's the kind of the bigger learning that we're not necessarily so good at, collectively. There seems to be a kind of a paradox there almost. 

The Ktor project

Kevlin Henney: You mentioned something earlier on when we were talking about building web services and talking about microservices, but really all of this kind of like stuff of like, "I want to put something out there so that other things can communicate with it." You've been working on something recently in that kind of space.

Hadi Hariri:, I'm on the Ktor Project, which is an open-source framework that we've developed at JetBrains for, I'm sorry, creating microservices. It's for creating connected systems. So essentially, it's a multi-platform, because it's written in Kotlin. So, it's multi-platform, meaning that you can actually like now build...with the 2.0 release that we did, you can build servers using Kotlin native, which means no JVM, nothing, just targeting binary, like in the old days, compilers and linkers and get your EXEs. And then it's got a client, which is also multiplatform. So you can use it on iOS or mobile and all of these different things. Yeah.

Kevlin Henney: Because I had quick look at that. And I thought, "Oh, this is quite interesting because it seems that...curiously enough, the thought went through my head, and it's kind of validated by what you've just said. It's reclaiming some lost territory here. If somebody wants to look at this interview in five years' time go, "Yes, he was wrong." That turned out to be terrible.

One was the Kotlin native aspect. And I thought actually interesting thing in terms of Kotlin does seem to have stuck to. It's now just over 10 years old. And it seems to have stuck fairly closely to its original vision of basically, "Yeah, we want to program something like Java, but there's a lot of extra noise in that. And there are so many things that we do that are common that should be brief. Those things which are common should be the shorter type of thing," Whereas, justifiably, Java gets that as a criticism that even doing the most trivial thing...it's not that it's not possible. It's just that you've got to spend a lot of time doing it.

If you're getting paid by the line of code, this is a magnificent solution. You ignore every recommendation we make in this talk. But there is that idea of actually it seems to start fairly close to that mission statement and seems to have optimized in that space. And it goes back to that idea of the developer user experience. You know, the developers are users. What's their experience like?

Recommended talk: Programming Kotlin: Why, How & Its Future • Venkat Subramaniam & Hadi Hariri • GOTO 2021

So Kotlin seems to kind of fit comfortably in that space, but then also gives you that actually solving the portability problem. And so there is a return to native because, you know, the people are interested in that. But actually, we also need to run things on VMs, and so on. And it seems to sort of, say, "This is kind of a solved problem that we can take care of that. This is something we can take care of."

So, I was quite impressed by just looking, just sifting through some of the examples on Ktor going like, "Actually, these are fairly simple." And as a non-Kotlin programmer who's not in the space either, I'm looking through this like going, "This makes sense." I'm coming at it as not shaking my fist at the cloud. But, as I say, you know, this is all subject to change. In five years' time, it could turn into a monster. Who knows?

Hadi Hariri: No, that's actually one of our goals.

Kevlin Henney: I'm just like going around there. But, at the moment, what I was seeing, it's just like, "Oh, actually, this does look fairly sane, but maybe I'm getting lured in by the examples. But..."

Hadi Hariri: To give you some background around Ktor. What happened is that I used to work, you know, I've gone through many platforms, right. And I used to do a lot of . NET. And at the time I was doing ASP.NET, and MVC, and then I got kind of interested in Node.js. And I started to do a bunch of JavaScript. And then I got hooked on to Express.js. And I really, really liked Express.js because it was so...it was like back to my, you know, roots of ISAPI DLLs where you just basically had an entry point, and you would just execute something, and I loved it.

So when I started to play with Kotlin, there wasn't anything like that, right? There was the Spring framework, and that was the Spring and Spring. And so, I created this framework called Wasabi, which was basically trying to port Express.js over to Kotlin. And then one of my colleagues that was working on the Kotlin team started to experiment as well with Kotlin as a DSL and started to create this other thing called Ktor.

At some point, Coroutines came out, and of course, it made sense to move to asynchronous, and he had already kind of like started building it on the asynchronous model. Wasabi wasn't, and I'm like, "You know, why are we doing two things?" Right. So I said, "You know, like, I'm going to drop Wasabi, and I'll help you with Ktor." And then eventually, he stepped aside, and then I took over the team. But, like, our essence is to try and keep it really, really simple.

It's funny, because folks come to us, and they're like, "Oh, you know, if I use Spring, or if I use x, or if I use Y, I could just, like, take a class and add an annotation to it, and poof, I've got it all done." I'm like, "Yeah, you do. But that annotation, you pile up seven other annotations, and then you don't know what that little line, you know, the class definition is doing. And it starts to become..." And we want to be more, like, explicit, we want you to exactly know what is happening, but at the same time, not be overly verbose. So, that's kind of like the balance that we're trying to keep with Ktor.

Kevlin Henney: I kind of like the way you've expressed that. And actually, we can step back from Ktor and actually kind of look at that as a sort of general principle and a general approach, and the idea of, you should know what is going on and being done on your behalf, you should know what's going on, you know, the fact that something's been done on your behalf should not be a matter of magic, and mystery, you know, the annotation stacking.

In fact, there was an interesting one with dependency injection framework a few years back where, you know, members of a team, a couple of members had kind of led the charge, and had ended up with so many subtle edge cases and complexity that they were very subtle uses and dependencies on subtleties in this thing. And it was interesting because they were kind of like, obviously. This is often the case when people lead an architectural effort. You always got somebody who does know everything because they were present at all of the key design decisions. They have a deep history there.

Then you got all the newer developers. It was really interesting that one of the developers said, "Yeah, I don't really know what's going on. When I have to add in and I extend the code, I basically copy and paste. So, it's already working and do that." Because it's just like, well, that's working, and I want something similar and a variation. And copying is one of the most successful human traits. So that's what they do. They do that as a perk...you know, that's not...they're not being... We need to get away, I think, from the moral judgment that somebody who duplicates code is bad. No, they're actually doing what is human and safe.

I genuinely don't understand that, its complexity. The way I can manage that complexity is by copying it. And I've taken a thing that already works and now I'm going to adapt it, but I'm gonna be really cautious so I don't break it. But I don't genuinely understand how it works, so that's why I'm being cautious. But I am taking a thing that works. And that is a value to me. They're being incredibly human. And they've done the right thing in that sense. The thing that is not right is perhaps the origin story of this.

We want things to be...we want things to do things on our behalf because we don't actually want to have to write the whole stack, you know, although there are a couple of developers out there, "Oh, I'm going to do this. True craft. I'm going to craft all the bits all the way down to the 68000. I'm going to build the emulator for that." Yeah. So.

Hadi Hariri: In Javascript.

Kevlin Henney: Absolutely. So, yeah, you've got that. But the point is, you want things done on your behalf. You want to have abstraction. You want to have benefits. But at the same time, it shouldn't leave you with a sense of mystery of I don't know what I'm doing. You know, because that is effectively what we end up doing. So, there's that balance of, like, I've got enough control and I've got enough comprehension. I can see what I'm doing. It doesn't mean I'm gonna do it perfectly, but then somebody else can see what I'm doing. But we still get the benefits of not having to do it all ourselves.

It's very careful, and I think there's a careful balance there. You used the word balance very specifically. I think a lot of design is that. It's just like, you know, there's something that pulls you this way, but there's also something that pulls you this way. We want things done for us, but at the same time, we want to know what it is that we are creating and have that kind of control. Otherwise, it's not our code. We are merely observers who poke it. We want that it should be our code collectively.

Hadi Hariri: And I think that balance is the hardest thing. I feel like, even myself, and many, I guess, that it's so much easier to, instead of trying to keep that balance, fall into the trap of complexity, and say...look, it's hard to understand. So you need to be a...you know, understand all of this. So it's hard to keep simple, right? We don't...I think that sometimes we don't put enough effort into that balance, right? Which I think is where we are now, kind of.

Kevlin Henney: I think it's our idea of like, "This stuff is hard. We acknowledge that it's hard. But it shouldn't be at the situation of like, "Well, it was hard to build. It was hard to understand it. And it shouldn't mean..." You know, the idea of like, "It is hard. We're not going to say it's simple, but we will try and simplify it." But to simplify takes enormous effort. It takes discipline, doesn't it?

Hadi Hariri: Exactly.

Kevlin Henney: You gotta kind of really focus on that. Right. All right. Well, in that case, thank you very much, Hadi Hariri.

Hadi Hariri: Thank you.

Kevlin Henney: And good day and good night. I hope this has been useful to you, and that you will join us for more at another point, in five years' time.