How to Continuously Deliver Quality Software Against All Odds

20 Years of GOTO: From Agile Beginnings to Sustainable Software and Beyond

Julian Wood: Welcome to another session of GOTO Unscripted. We're here in the beautiful Copenhagen. I live in London, and people in Denmark say it is often rainy and it is often grey, but the weather is absolutely beautiful. But that is of no concern, sort of, except for our enjoyment of the conference. But today, I am joined by the one and only Daniel Terhorst-North. Daniel, how's your conference going?

Daniel Terhorst-North: It's amazing. I've been to a lot of GOTOs. I was thinking about this. This is...GOTO Copenhagen 2024 is 20 years.

Julian Wood: Wow, 20 years.

Daniel Terhorst-North: My first GOTO wasn't actually GOTO. It was called JAOO back then. In fact.

Julian Wood: Very nice. Nice and retro.

Daniel Terhorst-North: I have several souvenirs. I've actually brought six shirts to this conference. I'm switching as I go through. Twenty years ago, my first joint talk with Martin Fowler. And at the end of the day, in 2004, we were on a panel together. And at the end of today, in 2024, we're on a panel together. So I feel like this is a wonderful kind of circularity to the whole thing.

Julian Wood: Has anything changed in 20 years?

Daniel Terhorst-North: No. Yes. It is interesting. I think, because...and again, I'm going to speak a little bit about GOTO and I think maybe a bit broader, the GOTO, kind of the JAOO, and then the GOTO series of conferences have always been where I learned stuff and where I learned stuff from the horse's mouth. So I first came across Clojure from Rich Hickey and Stu Halloway talking about it at GOTO. HTML5, when that came out, the folks who were actually building it were here. And so on. And so on. Sophie, who invented the AMD chip layout, is standing on the stage, talking about the history of these chips. It's just bonkers.

Twenty years, what I think the biggest shift is if you look at how software delivery works, I think there was, 20 years ago, this agile movement thing was very new, it was very nascent. The world it was in was very structured, and was very gated. It was really weird. It was not what the rest of the kids were doing. And it was seen as very maverick and irresponsible and all that kind of stuff. And in fact, the panel I mentioned, there was, I remember, a chap on there from the U.S. Department of Defense, and he was busy telling us how we should run projects. And Martin and I were sitting on this panel going, "No. I'm afraid we're going to find no middle ground at this point."

And you play it forward, and certainly, in most of the organizations I see now, it is really unusual not to have cross-functional software teams, people working together solving a thing. They're not kind of, "Take this data and convert it and shove it into that thing over there." It's, "Build this feature that's going to have this capability, and if there's some database stuff, do the database stuff. If there's some UI stuff, do the UI stuff." The full-ish stack, it's not a real full stack but a full-ish stack, is normal, and that's exciting. And you see a lot on the socials these days of how agile is dead and agile is well. I look at it as, like, it's just become so normalized that it's not that interesting a conversation anymore in a good way. Why would you keep talking about the fact that we have clean running water when we've had clean running water for decades? Oh, yeah. But the water, the clean running water people, they're like a stuck record. When they started, we didn't have clean running water. Guess what?

Julian Wood: I think that's entirely applicable, and that sort of keeps happening. In a way, as you said, it's a good thing when something becomes that ubiquitous and that useful and that everyone cottons on to it. And of course, not to denigrate marketing in any way, but when something becomes successful and good, it also becomes a marketing term and everyone wants to be agile. Everyone wants to be "something else." And so it seems that agile then becomes everywhere, and then people use agile in the wrong way. And then that's sort of agile's fault rather than the implementation of agile. But yeah, I mean, in 20 years, nobody would think about deploying software in a non-agile manner. But I still think there's still a lot to be done.

I think even CI/CD pipelines, the sort of backbone of being able to deliver software, is I think sort of in early days, and people are sort of still trying to figure out and building new novel ways to actually deliver software even though it's been sort of 20 years of the ethos and the manifesto. So yeah, 20 years. You also mentioned seeing the sort of future unfold here at GOTO. Are there any talks of any things which sort of sparked your interest to go, "Oh, there's the beginning of something interesting?"

Daniel Terhorst-North: I think beginning is probably a bit generous, but certainly, the growth and prominence of something that's interesting is, I'm here for this, right, is folks like Holly Cummins talking about, I guess, the economics of software from a sustainability perspective. So, how do you write green software? How do you write in a greener code? And it's not generally when the people have had these conversations in the past about kind of environmental impact and sustainability and blah, it's kind of worthy but dull. Oh, come on, all this climate stuff. But then what she's doing, which is brilliant, and what a bunch of other folks are doing, is they're saying, "No, there's a strong economic model, there's a strong economic argument that better architected software is cheaper to run, has less environmental impact." Excuse me, frog in my throat.

Architecting, understanding of mechanical sympathy, understanding the machines you're running on, understanding the impact your code has when it runs, it turns out, is as much about economics and as much about performance as it is about running leaner. And I love that a conference with this kind of profile is running a talk like that and pushing a message like that because it's absolutely a thing that this time has come. So that's really exciting.

Julian Wood: And I think a lot of software has always been about efficiency, but that efficiency has often been about because it needs to run in a constrained environment. Now, with even cloud and sort of unlimited scale, sometimes that message is lost, that it's not that interesting. And I think her example was looking at Java and looking at the ways you can use GraalVM and Micronaut and things like that to make your Java program far more efficient so that it's saving less money. And electricity prices have shot through the roof recently. I mean, there are colo providers who literally have gone bankrupt and companies have been evicted out of colo facilities literally because the electricity bill has gone up. And if you can be more efficient with your code, you're using less server resources, you're using less electricity. Yeah, 100% think that's super important. I mean, I happen to work for AWS and I happen to work in the serverless product team, and we build by the millisecond. And if you are paying by the millisecond to write something, writing efficient code, which is efficient architecture, literally saves you money and is also good architectural practices.

Daniel Terhorst-North: Well, actually, the serverless thing is interesting. There's a chap that you and I both know, Gojko Adzic. He has a mind mapping web tool called MindMup. Again, I love, he builds products to scratch his own itches, right? He wanted a mind mapping tool that he could use at a client site, and he wasn't allowed to install anything, so he shoved it on a website. And the backend of that is all AWS serverless. It's all lambdas. And he said he was the primary user of all of the lambda stuff, because basically, all of the mind mapping is just in the browser. It's all JavaScript in the browser. The serverless stuff was if you want to get the mind map out, so if you want to render like PNGs or PDFs. And the entire user base of people who wanted PNGs and PDFs was Gojko. So he's like, "Well, I'm going to write the thing I want."

And then this thing went viral, it turns out. It's really popular in U.S. schools. School teachers love it because there's some...I don't remember exactly the story, but it's some particular style of teaching in the U.S. where MindMup really lends itself, and there's certain templates that teachers started building. And he's got his Lizard Optimization thing now where you look at, like, odd behavior. And we're all thinking, "That's odd behavior." "Well, I'm going to make that odd behavior easier." And suddenly everyone's doing the odd behavior. So he did this thing, he built these templates for schools, and suddenly, all these schools are using it.

Recommended talk: Lizard Optimization • Gojko Adzic & Dave Farley • GOTO 2024

What does that mean? That means he now goes from, like, a few dozen people using MindMup to millions of people using MindMup. And they're starting to hit his serverless lambdas. And he tells the same story. Basically, he looked at the architecture, like the cool kind of path of each thing. And he said that his big insight, his big epiphany, he said, people think of lambdas, of serverless, as stateless. He said, it's not stateless, it's shared nothing, which is very different. So it can have state, right? It's not allowed to talk to any of the other things or not allowed to assume anything else is there. So you can, for instance, if you're rendering PNGs or PDFs, there's probably a bunch of stuff, static stuff that is useful to have already. So you make that all part of your serverless startup, then you can service these things as quickly as you like, and then you have your wind-down.

So he architected very deliberately like this, and his bill went down to, like, double digits of dollars a month, with millions of users. Insanely tiny, tiny bills. Because the first few million hits or whatever are free. And then after that is 0.00-tiny per whatever. And, you know, and it was the...so there's a strong economic drive to clean up your architecture. And that got me thinking, the last time this probably happened was the 1960s, on the mainframe when you were billed per CPU second, and so you wanted to write efficient code, because, otherwise, it was going to cost your company a ton of money to IBM or to Amdahl or whoever is selling you your mainframe.

We had this middle bloated sort of three or four decades where, you know, Sophie, this morning, she was saying like, "The compute is so far ahead of what you can do with programming languages. You idiots with your sequential programming languages throwing away your gigaflops or whatever if you started writing much more parallelizable things. We've already got whatever we're waiting for." And that world starts to happen.

Julian Wood: I suppose that is the future of seeing what microprocessors can do. You know, never mind, you know, AI chips and all that kind of thing, which is going to be a whole new sort of paradigm. We're talking a lot about people. You're obviously, you know, well famous for behavior-driven design. That's been around for many years, and it's very much also about the, you know, people and communication and the sort of three different stages of that. 

Evolving TDD and BDD: Insights, Innovations, and Open-Source Aspirations

Julian Wood: I know you said before that behavior-driven design was sort of based on test-driven development. How have the sort of two evolved? And we've got obviously Kent Beck talking at the conference here today, and he had this interesting concept which he was talking about, which I loved, about test, commit, and release. So how, if test...

Daniel Terhorst-North: No, test, commit, and rollback. Revert, yeah.

Julian Wood: Got the wrong R, thank you.

Daniel Terhorst-North: If it doesn't pass, you just lose your work.

Julian Wood: Exactly. So if behavior-driven development was built on test-driven development, and this is a new way of thinking of the test size, the test part of it, how does a BDD evolve or, you know, move and change in a modern time? What should people think about it?

Daniel Terhorst-North: So I have to say...so I just spoke to Kent recently. Obviously, he's at the conference, we just caught up. And I was saying that, you know, one of the things I've always found as a challenge introducing TDD to people is the kind of ego-driven developer saying, "I'm not going to write tests. I'd actually stop calling them tests." And he leant across and very generously said, "You need to get over that." So on the very principled point that I'm not getting, I refuse to get over it. You know, they are, when you stand back and squint, when you're doing them both as intended, they're basically the same thing. I've always maintained they're the same thing. So it's not TDDs that are down in the detail and BDDs that feature up. They're both both. And they've always both been both.

You know, I started JBehave back in 2004, 20 years ago. In fact, on Christmas Eve, 2003, in Cape Town, I registered the domain jbehave.org, much to my wife's dismay. She said, "What are you doing on Christmas Eve?" I just said, "I've got to register this domain, what is in my head."

Julian Wood: How many domains do you actually own? Separate question, I'm going to derail you.

Daniel Terhorst-North: So I think of my domain renewal emails, right? It's God's way of telling me how bad a completer I am.

Julian Wood: I agree.

Daniel Terhorst-North: Several.

Julian Wood: I completely understand.

Daniel Terhorst-North: Actually, well, I know. So I have a platform right now. So this is an appeal, okay? There's a number of projects that I have in flight at various stages of either not starting or starting or whatever. It would be great to get some community stuff behind those. So one of them I thought of is howtotdd.in.

Julian Wood: Okay.

Daniel Terhorst-North: All right. So I've got the domain, howtotdd.in, and the idea is you go to howtotdd.in/python, and it's just a one pager, right? I know how to do TDD. I don't know Python. I know how to do TDD. I've never seen Rust, right? I've never seen anything like it. And typically, as an experienced TDD-er, as an experienced, you know, XP-er or whatever, the first thing I want to do when I get any language is write my first code, and the way I write my first code is I want to write, start with a code example, start with a test. Rust has...you have this thing called crates and you have cargo, and cargo is this, like, Uber tool that brings everything together. And then in your Rust code, you put your tests in the same actual source file as, you know, the convention is you put it in the same code. I didn't know that. With Go, it has a really cool built-in test framework called testing module, and the testing module allows...and when you see this, you can't unsee it, it allows you to fail a test, but that doesn't stop the test.

Let me just let that settle for a minute. We are used to it, so all of the xUnits school of TDD frameworks start with SUnit and Smalltalk. And then it goes to JUnit, which was the huge one. That's the one that caught everyone's attention. And then you get, you know, C, C#, and all the other languages we have, but they're all based on the same thing, which is you make an assertion, if it fails, you raise an exception. That's it, game over. What Go does is they say, "Well, that seems strange." Also, Go doesn't really use exceptions like that. Okay. So instead, what Go does is when a test fails, you just got your test failed, and you carry on, which means you can have a whole bunch of different things that you're checking for in the same test. And at the end of it, it goes, "All these things are wrong." It's more like an MOT for your car. You take your car in for a roadworthiness test. They don't go, "Oh, your front lights on, bye." And you go and fix your front light, come back, and they go, "Your brakes are..." "You could have told me that when I was..." "No, we stopped at the lights."

Julian Wood: And I suppose it's efficient as well because...

Daniel Terhorst-North: Right, they do the whole...they test the whole car, and they give you a list of things that need fixing. That's how Go does testing. And I was like, "Right, now I want to do that in all the languages." So it's not just the technology bit, it's the paradigm bit. Somehow it's different.

So anyway, howtotdd.in/languages, right? It's there. It's a repo. I've done nothing with it, right? And so what I love is a bunch of pull requests of people going, "I just did Rust. I just did Erlang," right? Because I don't know how to set up Erlang. Anyway, that's one. I have others.

Julian Wood: Excellent. That's sort of, it's funny. You sort of think of the tests, all that, and you think of the sort of procedural programming where it's a step by step step by step. And when it fails, it errors out. And you think, no, you want to complete as much as you can and then understand, "Oh, that part may have failed."

Daniel Terhorst-North: But it also has, if it knows it can't go further. It has FailNow. So fail is registering a failure and cracking on. FileNow is trying to do some database testing. I can't even connect.

Julian Wood: Can't complete.

Daniel Terhorst-North: I'm out. Or I tried to look, that row doesn't or that table doesn't exist. What was the next thing you wanted to check? I wanted to check these things. Yeah, they're fine. So your report is the test failed, the test failed, and for these three reasons. And you're like, "I want that in Java now." So we have a thing called JGoTesting. And JGoTesting was more like JUnit 4 style. It was exactly that. It was a thing in...so you could use JUnit but with Go-style testing. And no one used it. Two people adopted JGoTesting, "This is really helpful." And then, yeah. So I'm really good at starting open-source projects. I'm not particularly good at growing them.

Julian Wood: I'm good at buying the domain names with great ideas, and they're not even getting to...

Daniel Terhorst-North: So I'll say we've got a dozen of these that are rolling around each year and going, "Oh, I still haven't done that."

Bridging the Gap: The Case for Domain Language in Code

Julian Wood: You also talk about things every programmer should know. And what are some of the things that every programmer should know? And what are the things that programmers shouldn't maybe know?

Daniel Terhorst-North: Right. So this is wonderful, wonderful, Kevlin Henney curated a series of books, 97 Things...

Julian Wood: I was wondering, why was it 97? Couldn't they roll it up to 100, or was there a...?

Daniel Terhorst-North: I don't know. I don't know the origin of this. I don't know the origin of this. But I do know that Martin Luther nailed a scroll to a church door, which was 95 things every cardinal should know. He's got 95 criticisms against the church. So I don't know if…

Recommended talk: 97 Things Every Java Programmer Should Know • Trisha Gee & Kevlin Henney • GOTO 2020

Julian Wood: So programmers are improving on cardinal nature.

Daniel Terhorst-North: ...if it's, like, "What's the first prime number above 90?" I don't know if it was that or if it was supposed to be 100 things, but they only got 97 volunteers.

Julian Wood: Maybe it's more memorable.

Daniel Terhorst-North: But I submitted one. Again, I'm not a finisher. I had about four. I've probably got the other three in a file somewhere. And the one that I talked about was coding in the language of the domain. And it was really front of mind for me. I'd just been...I was working on trading. I do a lot of software in banks and trading systems. I've just been working on a trading system. And this is around the time Java's got generics, so there's angle brackets everywhere, but it's relatively new to Java. So if you're a Java programmer, they were really unfamiliar and really jarring, and you're still trying to get the hang of your brain parsing them and stuff. And somewhere amongst this code, there was a map of ints to a map of ints to ints. And there was a ton of logic around this. We're trying to figure out what this clearly important code did.

It turns out that the map of ints to ints was something like, and again, I'm going to get the language of it, but like a portfolio, which is basically a book of trades, and you have a portfolio that has an ID, and then there's, I think it was a country code or something, or a region, which was also an ID. There's two IDs really, so, like, a country ID and a region ID, but they're just ints to ints. And the int in the outer map was a trader ID. Anyway, we've been reaching out in this code for literally days, trying to figure out what's going on. What's going on is this, certain traders are not allowed to see certain other books, right? They call it separation. And there's a whole load of, like, legislation around it. It's, like, a hardcore regulator. If, as a trader, you can see the other side of a book you're trading, that's illegal. You can get massive fines for this. So it's a big deal. And this really critical business rule was locked up in this map of ints to maps of ints to ints, right?

We kind of just refactored it, you know, Kent style, small change at a time, change it, write the test, change the code, change test, refactor it out. And we had portfolios by trader. Then we had a trader, an object, and we had a method, CanSee. And CanSee took a portfolio ID. So this whole map of ints to ints thing, which is if trader.CanSee portfolio ID, right, boom, whole business rule, one line of code. So it was kind of, if you bring the domain language into the code, it's not just that it makes it easier to see what's going on. It means that actual business rules jump out. So that was what I was trying to get across.

Julian Wood: And it's fascinating because that obviously sort of brings together even the other talks here, microservices and domain-driven design and all that kind of connecting the business much closer to the code.

Daniel Terhorst-North: Well, it's the closest. Lovely, lovely integrates in, like, domain-driven design, mind mapping, and team topologies. And it's just, yeah...

Julian Wood: Once again, that's the exact example.

Daniel Terhorst-North: ...really into the socioeconomic studies she's on.

Julian Wood: And it's an example that also, you know, a technical person, like a coder, can understand the business intent from that far simpler because, you know, they can read it. It's even in English that is done, you know, exactly the same way.

Daniel Terhorst-North: So let me say, the thing that...sold is probably the wrong word, but the thing that sold domain-driven design to me, I'd come across the blue book and whatever, we're doing stuff. I was working, again, and it might've been the same bank, but there was a new project and the team was very deliberately, very actively domain-driven, you know, domain modeling and knowledge crunching and all that good Eric Evans stuff. And we were, like, really kind of focusing on that. And there was a moment, I was sitting here, I remember so vividly, I was sitting here, there was a pair of developers there, and they were talking about how you price an order. And it turns out the domain of pricing orders can get quite complicated. And they're talking about how you're going to price an order, and they're getting quite animated because there's some gnarly logic going on.

Someone else walking behind the desk behind them leans in and says, "That's not how you price an order." Like, "How do you price an order?" And so this person is chatting away, explaining how to price an order. And while he's saying it, these two folks are sitting here. These two folks were discussing the code. This trader, was a trader, was talking about how you price. None of them knew that the other one wasn't talking about the thing. These folks thought that this person was talking about the code. They thought these two folks were... And I just went, "Wow." That's how it should be. You shouldn't know whether you're talking about the code when you're discussing how it works. I was like, "I want that. I want that every day on my projects." It's a thing of joy. Yeah.

Julian Wood: Because, I mean, ultimately, all of IT and all the programming is about solving business problems. So when there is, you know, a language discrepancy or a difference in understanding between the two, I mean, things are just not going to work.

Daniel Terhorst-North: You're having to do translating in real-time in your head, and it's just nuts.

The ThoughtWorks Era: Innovation Through Necessity and Collaboration

Julian Wood: You worked at a time also at ThoughtWorks, and there were some amazing luminaries, I'll include you in it as well.

Daniel Terhorst-North: Baby luminaries back then. 

Julian Wood: Not at all. Who worked at ThoughtWorks at the time and were, you know, the unbelievable thought leaders of the time. In your opinion, what caused that to happen? It was in London. Was it a random group of people who collected together? Was it the collaboration? Was it the time? Was it the city? Was it the beers in the pub? Yeah. How did this sort of collection of genius all seem to happen?

Daniel Terhorst-North: You're very kind. I think one of the things, and obviously, you look back on that time, there's a, what's the word, a truism. If you ask any ThoughtWorks, any ex-ThoughtWorks worker at what was the golden era of ThoughtWorks, it was them arriving plus a year to them leaving minus a year. Like, that was the golden age. Everything else is rubbish. So for me, I can say, that was the best time. Now, it was...

Julian Wood: Even to name names, because I mean, you know, people who don't know who work at ThoughtWorks, that's sort of assuming.

Daniel Terhorst-North: So ThoughtWorks was a global software consultancy. So when I joined, my technical interview was with a chap called Matt Foemmel. I didn't know any of these people by then. I didn't know any of these people. There's a chap called Matt Foemmel. Matt Foemmel invented continuous integration. He wrote the first...he wrote CruiseControl. He wrote the first continuous integration server. That would probably have intimidated me if I'd known that. I'm quite glad I didn't know that. My first project manager was a lady called Rebecca Parsons, Dr. Rebecca Parsons, who's now, like, CTO emerita from...yeah, like, just insanely smart people. And it was...

Julian Wood: And Martin Fowler, James Lewis, Sam Newman, I mean.

Daniel Terhorst-North: So hang on, I haven't got there yet.

Julian Wood: There are even more.

Daniel Terhorst-North: So I was tech hire number one in London. So when I joined, there was an MD, there was a salesperson, there was an office manager, and there was me. There was a business analyst who kind of arrived and left shortly after. But, like, so I was the first tech hire. And we grew that...

Julian Wood: So you paved the way?

Daniel Terhorst-North: We grew that office. Well, so they had, like, a sort of an advanced party from Chicago. So we had this idea of when you're a consultancy moving into a new territory, you land a couple of big gigs, just like a couple of big oil firms, energy firms, and so then you staff that from your pool of smart people. And you know, you send over the best and the brightest because you want to get a good impression, and then you start hiring and backfilling, is the idea. And then people can go home and some of them stay, and whatever else. So I was the first local hire. And then we started hiring. So I hired Sam Newman, James Lewis. We hired Julian Simpson, the Build Doctor. I was working at a client, and he was their sys admin.

And I convinced ThoughtWorks that we needed to hire ops people. And they were like, "Well, no, we don't really, because all of our developers kind of do sysadmin." I was like, "No, that's true, but they do sysadmin like developers. A dedicated sys admin is a different creature." And they said, "Well, no, we don't need to." And eventually, the hiring manager in the UK, our recruiter in the UK, was brilliant. And she and I cooked up this thing where we invented a job called an environments manager. And we basically wrote down Julian Wood's profile, and we put that out. And he arrived, and he went out on his first project. And, like, the feedback after a day was, "We need another six of these." And so then Julian and I hired Chris Read. Chris and I hired Jez Humble.

Julian Wood: More continuous delivery.

Daniel Terhorst-North: So, like, the great and the good of the build world then. But yeah, where does this come from? Where this comes from is necessity, necessity being the mother of invention. So we would be on client projects. They were typically very lean-staffed. They didn't send loads of people in because clients don't want to pay for loads of people. And some of it was quite political or quite technically challenging or whatever. And for a lot of this stuff, certainly in the mid-2000s, you kind of had to invent your way out of it. And as we were inventing our way out of it, we started writing stuff down. So Chris and Julian Wood and myself, no, sorry, Chris, Jez, and myself wrote the build production line, which was the first time anyone had written down automated build pipelines and single deployable artifacts and promoting an artifact alone.

Suddenly you're the build guy. But just, you know, other people were doing it. We just wrote it down. And then the BDD was I was trying to get client developers to do TDD because, for me, it was, like, obvious and brilliant. And they were just going, "We don't do tests. We're not going to write tests for programmers." What if I just don't mention tests? What if I try to teach TDD without using the word? That was this challenge I set myself. And I started talking about examples and blind specification. I'm like, "Of course you do that. That's fine." And suddenly they were doing it. And I was like, "This is brilliant." And so I started talking about it at conferences. So BDD was accidental. I didn't mean to go out and invent the thing. It was all just trying to get work done.

Around that time, there were two big projects. There was AOL, a UK ISP or was American ISP in the UK. They were pushing out. They were just starting to push out broadband. And it was helping them automate all of that. And at the same time, Dixons, which was a UK high-street retailer, had four major high-street brands. They were explosive growth, running on creaky, old mainframes, and they needed to basically rebuild the whole company from a technical perspective. And they had an incumbent. They made a three-year plan. It probably was a three-year plan. They were over two years into this three-year plan. They had zero lines of code. Zero. They had shelves and shelves of documents, about 10 months left, and they knew that their systems weren't going to get through Christmas at their peak. At which point, they phoned ThoughtWorks, and they said, "We're stuck." And then we sent a couple of folks in, or they sent a couple of folks in. And they were like, "Yeah, you really are. We're not touching that."

Then what happened was they said, "Look, we can do this. We can build." And this was, like, EPOS in the stores, supply chain, stock, order processing, basically, building a company from public static void main, outwards. "We can do this. It will be ugly. We will cut corners. We will hack and slash our way through this deliberately. We're going to leave a load of signless markers along the way. We will only do this if you promise we will then get to do it properly after Christmas." And they did. They were super grown up about it. It was a brilliant collaboration. And so they did. They built the Dixons' whole thing. And really, at the time, it was one of the biggest software projects in the world, let alone agile. You had teams in the UK, U.S., India, multiple teams, multiple offices, follow-the-sun development. Rebecca Parsons started as the tech lead, handed it over to Dave Farley. A single build. Matt Foemmel built the build, single build, single source repository, bunch of teams all around the world pounding on this thing. Yeah. Dave, he became CLO. He was the delivery lead on it, basically ran it like a military operation. It was incredible, the stuff they did.

We came out of it, like, days like this, we said, "Right, some really interesting things happened there. We should capture those." And so a bunch of us started writing down things like blue-green deployment came out of AOL, the single deployable artifact came out of AOL, the DBdeployer, the version database changes. Sam Newman had been kind of pioneering this stuff in Dixons. And so we had all of the elements of a book, and we're like, "This is really cool. We'll build a book." And then, one by one, we all fell off because we're rubbish. And eventually, Jez Humble and Dave Farley kind of took this book over and wrote a brilliant book. They won awards for it. They totally deserved it.

That's where continuous delivery came from. Whereas it was just we had to, right? It was the only way we were going to get the thing done. And I think, you know, I've always said this, you don't learn anything on the easy gigs. It can be really, really tough and, actually, you know, quite exhausting. You've got to be very careful about burnout. But yeah, basically, go into somewhere that's completely insane and completely impossible, and fix stuff, and then write it down.

Julian Wood: The whole CI/CD process, software delivery, you know, parts of agile, and everything is a British high-street retailer which was not going to be able to sell computer games for Christmas is the reason we have all of this.

Daniel Terhorst-North: But again, there's all of this, and there's a lovely phrase that I can't think of right now, but this idea of, like, concurrent discovery, when you look at, what's it called, calculus, where you've got Newton? No. Who came up with calculus originally? Yes, Sir Newton.

Julian Wood: I think so.

Daniel Terhorst-North: Principia Mathematica. And Leibniz, in Germany, Leibniz, Leibniz, in Germany. Both had the same idea at the same time, slightly different vocabulary, and they both went, actually, that's really cool that we both came up with this, their relative, their respective minions were all attacking each other for, like, plagiarism and theft and blood. And they were both going, "No, no, this happens." And so we didn't invent continuous delivery. We just happened...we built our own version of it and wrote it down. Other people were doing this, like DevOps. The term DevOps came out of a bunch of other folks looking at exactly the same thing.

Recommended talk:Interaction Protocols: It's All About Good Manners • Martin Thompson • YOW! 2019/h3>

DevOps, Constraints, and Organizational Flow

Julian Wood: You mentioned hiring the operational folks. Often, I think DevOps, it's a subject for probably another chapter, how DevOps is coming together and getting the dev folks to do all more operational tasks and the operational folks to do the operations in more sort of dev kind of mode. But yeah, having software delivery, people who code this, understanding the operations, and now we've got to build it and run it and everything, is a radical idea. But I mean, now you look at it and think, "There's no way you could build software without having an understanding of the operational aspect of it."

Daniel Terhorst-North: Well, I described it slightly differently. So I look at it in terms of constraints. So when we started doing this stuff differently, software development was the constraint. You would pour millions, hundreds of millions of dollars into a void.

Julian Wood: Your three years of design before you write any code.

Daniel Terhorst-North: Right. A two-year, three-year release of the wrong thing that we needed two and a half years ago anyway. That was how people saw IT, because that was how a lot of IT was. So that was the constraint. So now, you broaden that constraint, you elevate that constraint, and you start doing these lightweight, these iterative adaptive agile methods. And now, suddenly, it's not two years. You know, it's not even two months. Sometimes it's two weeks, or you know, much, much, much more fine-grain delivery. That's no longer the bottleneck. Where does the bottleneck go when we can actually ship stuff? It goes downstream to release. If you're releasing every two years, it doesn't matter that it takes six weeks and involves tears and tantrums, right? If you're trying to release every week, suddenly, it's under the spotlight.

So the constraint moves downstream to operations. And then the smart operations folks get together with the smart devs folks, and they go, "Well, what did you do? Let's do what you did. So we want to take some of your ideas, like the automation, like working in small chunks, like the feedback stuff, but we're going to overlay it with our rigor." And I see DevOps is the rigor of operations with all of the tricks, if you like, of XP and agile kind of stuff. When it's done right is it then becomes this kind of, like, we're not going to be a constraint either. And this is where they fall in, and Jez did brilliantly, is they articulated, like, continuous delivery, is it becomes a pipe. It becomes just a flow. And now it becomes a flow, where does the bottleneck go?

It turns out the bottleneck at that point is now upstream, right? Your business folks, your product folks, can't get ideas into the pipe fast enough because they're used to working in big batches. They used to like BRDs and PRDs and, you know, big wireframes and all this kind of stuff. They go, "Just tell us what you want. Sketch out what you want. We'll iterate. Honestly, by the time you've told us what you want as a sketch, we've probably written half of it, show it to you, and then you fit." And they're like, "Wow. Can we do that?" And then you start getting into, like, lean UX and lean startup. And so now you're moving it upstream to them. And so it's just a series of theories of constraints.

Julian Wood: Having visibility and being cognizant of the constraints. And funny enough, you have no idea what the next question I was going to ask was. Tell me about organizational flow. And you've just...you literally went straight off the chain. I was like, "Excellent."

Daniel Terhorst-North: It was a nice segue. And I guess this is more like my arc in the last kind of 10, 15 years. So I'm 30 years into this, 30-something years into this. And it kind of splits fairly neatly into decades. The first decade, developer, and then just various adjectives in front of developer, becoming more senior, doing stuff. Then joined ThoughtWorks, 2002. And then it's much more working with a variety of organizations, helping them get stuff done, and a lot of that getting stuff done is sitting there coding or pairing with people, coding shoulder to shoulder.

Julian Wood: So in a way, it's taking a developer-focused task and scaling it out to an organization so they can be more effective individual developers in a pipeline organization.

Daniel Terhorst-North: Right, exactly. And then, so around sort of mid-2000s, I've always been a bit of a...like, I love the glue, much more than particularly... I've been in, like, trading environments, telcos, retail. I'm not that. I don't get, like, particularly excited about a vertical, about a particular domain. What I want to do is glue stuff together. So I'm a build nerd, a release engineering nerd. I love all that stuff. So at one point, with AOL, we had a build team of eight people. And the eight people were, like, you know, Chris Read, Jez Humble, Julian Wood Simpson, insane, just some brilliant, brilliant people.

Julian Wood: The Avengers build pro people.

Daniel Terhorst-North: It really was. And I still describe it as eight too many people for a build team. Your build team should be no people. But out of that came, as I say, a lot of the patterns that ended up in the book, the blue-green deployments came out of that single deployable artifact, actually, the paper, the build production line paper. And where was I going with that? So they're kind of the release engineering, the build engineering piece.

I was kind of starting to pan back a bit, and it wasn't like writing software is okay, but written software is useless unless it's live. Live software is useless unless it's solving a business problem. Live software that's running and solving a business problem is useless unless you get the feedback to figure out, and so on. So I started panning back to how the teams work, how the teams of teams work. And what I wanted to do and still my mission, if I have a mission, I'm trying to figure out how to help organizations work effectively in a way that I can step away and it carries on being true. And I'll tell you why that is. When I was at ThoughtWorks, and it happened again and again, and it wasn't their fault, it wasn't ThoughtWorks' fault, it wasn't anyone's fault, but this kept happening, is you'd go in, you'd write great software, because they had very great, brilliant engineers and they would typically work with a team, and what you saw... And James Lewis is also at this conference. I'm so fond of James. James was one of my very first kind of folks that I met in ThoughtWorks and thought, "Man, you're going to be special. There's something about you." And I got to work in a team with James in about 2007. And the energy in the team changed, right? People were bouncing into work. They were having fun. And you're like, "We did this. We caused these people to have fun. Isn't that cool?"

Recommended talk: How Flow Works • James Lewis • GOTO 2024

And then so you'd step away, and we go off to the next thing. I remember, I went back to a particular bank about a year and a half later, with the same team, and everyone was just checked out. Half of them were gone. Half of them quit because it's never going to be like that again. The ones who were there were just, like, dead behind the eyes. And it broke my heart. I was like, "Right, how do we do this? So that when we go in there and make a change, it's sustainable, it's sticky, it's systemic." And that's still my kind of kick now, about 15 years later, is this what I do. I go into organizations, I work with organizations, "I'm available. Hit me up on LinkedIn. Love to hear from you." That is, I tend to go to a fairly senior level. I get hit up by COOs and CIOs and folks, or CXO minus one, or that kind of level. I don't know why, but they come across my talks, they come across work, or they hear about me, or whatever. And they're saying, "I've got this organization, and it's a bit broken. Can you help?" And my default position is always, "I have no idea. I have no idea why I come in and meet you and meet some people and do some listening and see what happens." But that is, for me, that's the big win. Being able to write software well, being able to ship it, deliver it, get it out into the world, get feedback on it, that is a necessary but not sufficient part of the puzzle.

Tips on Product Management and Effective Delivery

Julian Wood: Is there something in your engagements that comes top of mind, that sort of circle of a business idea into all the technical implementation, gets out to prod, is used by users, and that sort of missing gap?

Daniel Terhorst-North: And the feedback. 

Julian Wood: The feedback circle. Is there something you can distill in? What is missing with that? Are you still figuring that out?

Daniel Terhorst-North: That's easy. I can tell you that right now. It's called product management. The bit that's missing is product management. And I've been very lucky, I've gotten to work with some brilliant, brilliant product folks over the years. I mean, at ThoughtWorks, I worked with Jeff Patton, the guy that came up with story mapping, user story mapping, utterly, utterly brilliant, Marc McNeill, fantastic UX designer, product manager, just wonderful, wonderful people. Lastly, Jeff Gothelf, I've been very lucky to work with Jeff Gothelf, who literally wrote the book on Lean UX. And the way I describe it is baking the right cake and baking the cake right. I'm really, really good at baking the cake right. I do not have a commercial bone in my body. I can talk about a good game. I can talk a good game about theory of jobs to be done, about Clay Christensen, you know. I'm fascinated by this stuff. But you know, hands down, I don't do it. And when I see it done well, I'm like, "Yeah, you need that."

So I tend to now buddy up with...so there's, like, product management, which is squaring the circle piece that you're talking about, and product development, which is, how do you make the thing? So what I'm really good at, I like to think I'm good at, what I specialize in at least, is making the thing. Whether it's the right thing, what the thing should be, that's a whole product strategy piece. And I'll find someone who's really good at understanding, you know, user research, looking at the market, figuring out what's going on there, and that whole feedback, that full circle, that works as a really good complement.

Julian Wood: Is that still some of the failure of big IT organizations, even small IT organizations, having the project thinking rather than the product thinking? And you know, there's a book out there, you know, thinking about products instead of projects. When you have a project, we're going to do this kind of thing and it finishes, while the sort of product is you are running something that runs forever or for a period of time, and then that implicitly requires that feedback loop because you need to continue to evolve your product rather than just, you know, you've got a two-year project, then that's finished.

Daniel Terhorst-North: So I hear you. Now, so all that is a yes. And what you're describing is still baking the cake right. So the project-product thing is I look at it as a funding and a governance piece. So I'm going to do the whole of governance. I'm going to say, the whole of governance, right? It's two questions. What are you spending my money on? How's it going? That's the whole of governance. What are you spending my money on is investment. How's it going is a risk. The problem is in any non-trivial organization, I've got no idea what I'm spending my money on and I've got no idea how it's going. And so this is where all the governance theater comes in and the steering groups and the rags and all of that stuff is, "Oh, we can't answer these questions." And so rather than, "Let's jump through hoops trying to answer these questions," this is where I want to flip the whole thing around and say, "Well, let's change the organization such that these questions become easy to answer."

That's when you shift from, like, cost accounting to throughput accounting or when you shift from activity-based metrics to flow-based metrics, is all of that makes these questions easy. I can see right on my Wardley map of which things are genesis versus which things are commodity, right? All of that is, where am I spending my money? How's it going? It's all the same stuff. So the project-product thing is you can think of a project on a bigger-scale program of work as a way of batching money. That's the whole thing. And the reason you batch money is to manage risk. At worst, it will spend, you know, I'll lose that money. If I'm going to run a £10-million pound program, the most I'm going to lose is £10 million, which is a complete nonsense because the most you're going to lose is all the opportunity cost of all the other things you could have done, and there's probably hundreds of millions you're leaving on the table, but you're looking at the cost and not the opportunity. So we get the economics wrong in the first place.

But project versus product really is an operational thing about how you fund work. So product funding is going to basically treat you like a venture investment, like an angel investment. I'm going to keep on funding you as long as I'm seeing interesting stuff. As soon as I stop seeing interesting stuff, I turn the tap off. That's pretty much the whole difference. It means that I have much less, what's called VaR, value at risk. If you're continually delivering stuff and that stuff is starting to make money, I've got much less value at risk. If I'm just pouring money in and I'm not getting anything out yet because there's a big batch release coming up, that's a ton of value at risk. So it's a very different governance model. But none of this is still, am I building the right cake? Am I baking the right cake?

I will have trailing indicators of whether I'm baking the right cake. But what if this thing is making money? I'm not going to know that a different thing could be making six times that money because I'm not thinking in hypotheses, right? I'm still thinking of, like, "Well, it's kind of, it's paying its own way, it's probably fine," right? So we don't even have the thinking around that. So product-project is a funding-governance question. There's still, and I'm just as guilty as anyone, and probably at this conference, certainly, that we are the "baking the cake right" people, really, really good at this.

What my appeal is for folks watching this now as well, hang out with product people, go to product conferences, go to places outside of the how the machine works, and start talking to people about what the machine is even there for. I think, when we do that, when you bring those things together, it's really, really powerful. I've seen brilliant product people who cannot execute. They've got all the ideas, and they're the right ideas, and nothing happens. I've seen brilliant delivery people, like Peter Drucker's lovely quote, there's nothing more wasteful than very efficiently doing exactly the wrong thing. And we do that again and again. So, yeah.

Recommended talk: Serverless Compute at the Heart of Your EDA • Julian Wood • GOTO 2024

Programming and Socio-Technical Systems

Julian Wood: So your 15-year challenge is still not finished. It's still working all this out.

Daniel Terhorst-North: No, no. It's still exciting. It's still exciting. I've got some brilliant times.

Julian Wood: But I've spoken to a few other speakers who said, when are you writing a book then?

Daniel Terhorst-North: So let me see about my undiagnosed ADHD. Let me see about my inability to finish anything.

Julian Wood: You've bought the domain name.

Daniel Terhorst-North: Well, no, I have. I'm 12 years into writing a book that has been called Software Faster, which is patterns of effective delivery. I started collating these patterns, curating these patterns about 2011, 2012. There were 10 of them. A couple of years later, I was teaching classes, and it was iterative and generative and blah. And like Kevlin Henney talks about stone soup, and you have this kind of thing where, certainly, all these ideas are there. And now there are, like, 40-something of these patterns. And I'm like, "Well, 10 patterns is a class, 40 patterns is a book." So I'm going to write down these 40 patterns. And so I wrote, on Leanpub, I wrote 40 chapter titles with little one-paragraph things. I see, you just have to go and fill in 40 chapters, and I'm done. So that's been languishing for a while. So then I got working with a brilliant, brilliant co-author, and she said, "Right, make it manageable." Because these 40 patterns are about 8 sort of chunks.

Julian Wood: Just like software delivery. Make it manageable and in smaller chunks.

Daniel Terhorst-North: Make it manageable. And I'm like, "On, yeah." So we took out just the team patterns. So these are, actually, a GOTO Chicago talk from a few years ago, Patterns of Effective Teams, and basically, it's those and a couple of others. Let's just take the team patterns. They're well-formed. They're well-articulated. We can get that that's tractable, right? Let's get those done. So we are dangerously close to shipping that now. There's, like, probably one or two chapters left to write about that. And we're going to get some pictures, some illustrations in there, and then we're going to ship it. So it may even be this year, certainly early next year. So I'm very excited about that. That'll be the first actual...but I've written chapters of things and, like, a ton of forewords and stuff. I've never actually published a book.

Julian Wood: Well, many people are keen and waiting for that as well. Is there anything we haven't spoken about today that you think is top of mind?

Daniel Terhorst-North: Good heavens. Almost certainly. Almost certainly. What's that mean? I guess it's worth mentioning, the thing I'm talking about here at this conference is a talk I've given a couple of times now. It's called The Best Programmer I Know. And it started as a stream of consciousness thing. Like, on Twitter, back when it was Twitter, before the bin fire, and before... I was going to say before the Nazis, but that's not true, before the much more prevalent Nazis. And someone was...I'll tell you how it started. It started with someone saying, "Oh, you know, you need a CS degree to be a programmer." And I just went, "No, you don't. No, you don't." And I said, like, the best programmer I know doesn't have...never went to university. It was actually a lie. He went for six weeks, decided it wasn't for him, and quit. The other best programmer I know has got a First Class Honors degree from Imperial College.

Recommended talk: The Best Programmer I Know • Daniel Terhorst-North • GOTO 2024 Dan North

Julian Wood: So both.

Daniel Terhorst-North: And neither necessary nor sufficient. Then that got into, what do you mean by best programmer? So I just went, bleh. And it's this big stream of consciousness Twitter thread. I should have done something with that, four years ago.

Julian Wood: It is a good thing.

Daniel Terhorst-North: Finally, finally, got it into a talk. I've been talking about this, and sort of I took it out the first time, it was earlier this year. Honestly ,I was so nervous about this talk, because, A, it's very personal and, B, I didn't know, is this going to be useful for people or is this just a massive self-indulgent? And anyway, so I gave this talk, and the feedback I got was just lovely, like, really lovely. So I'm talking about that, and I think it's important that we remember that programming is, you hear this word, and I'm glad this word is becoming much more prevalent, I'm worried it's going to become a cliché or already has jumped the shark, is how they're socio-technical systems. Programming is a socio-technical activity. It involves technology and involves human beings. And unless we are very, very cognizant of the human-beingness in it as much as we are with the technology bit in it, we're going to just keep getting it wrong.

Julian Wood: Well, that's from your stories earlier of, you know, doing the circle of the whole product design phase, feeding back that in. I mean, that's communication between people. And sure, code is written and delivered, but you know, articulating a business need to a business problem to do all of that is people.

Daniel Terhorst-North: Basically the same thing. If you think of your delivery world as a product, the people involved in that delivery are basically the customers of that product, the users of that product. We want to, you know...what do we want to do with that product? We want to affect behavior change. We want to make people have more fun. All right. So, yeah.

Julian Wood: Excellent. Well, it's been fascinating to chat to you. I think we've got a book to look forward to, more talks to come. Daniel, it's awesome to talk to you. Thanks so much for joining us here today for another session of GOTO Unscripted. Until next time, thank you very much.

Daniel Terhorst-North: Thank you.