Living with Broken Systems: How AI Coding Tools Change Our Relationship with Complexity

Intro

Intro

Diana Montalion: Hello, I'm Diana Montalion. I'm a systems architect and author of Learning Systems Thinking, and I get the incredible privilege and joy to spend another 30 minutes of my life talking to Jessica Kerr, who's Jessitron online. If you don't know her, she's very passionate about the intersection of code and people and semanthesy—the learning system part of learning, meaning how we both embody a system and evolve it at the same time.

Thank you for spending time today. I know we're going to talk about AI because we're talking about AI.

Jessica Kerr: It's spring 2025. You have to talk about AI.

Diana Montalion: AI again. I have so many mixed feelings at the moment that I am glad that's what we're going to talk about, because I think I will be a bit untangled by the time we finish. One of the things I was thinking about today is that the thing I value most about your work, your talks, and having coffee with you is how I can focus on whatever you're focusing on in terms of the system.

Systems Are Always Broken – And That's Okay

Diana Montalion: There are so many things we can talk about—the tech, the people, the learning, the patterns, AI, events. If I follow you into a particular part of the discussion, it's always really fruitful. So I was thinking today we could follow you into thinking about AI as part of the system that we have now.

Jessica Kerr: As part of software and software development.

Diana Montalion: Part of software and software development. Awesome. So where do we begin? What's the context for us to start thinking about this?

Jessica Kerr: Well, it is spring 2025, and these days it's really easy to recognize that the system is broken—the much wider world system is quavering and clearly not working. You can't pretend that our system of government, across the world, works the way I grew up thinking it worked—as smoothly as I thought it worked, where there were rules and if you followed them, you'd be fine.

So the system is broken, and every year I realize it more, and it sinks in more. But at the same time, as I notice that the system is broken, I can also notice that life goes on. It's not that the system is suddenly broken. It's always been broken. I mean, it has to be. If it can deteriorate, that means we've always been living in a broken system.

But it's just like a distributed system in software. Some piece of it is failing somewhere. It's never perfectly working smoothly. And it's not that we wait until it perfectly works smoothly and then call it done, and then the software runs perfectly smoothly forever. No, we work within the brokenness and among it, and we compensate. We have redundancies, and we deal with surprises and look for surprises.

In software, sometimes we get to fix the brokenness, and other times we work around it. Not all of it is ours anyway, because our software is always working with other software.

So once again, I find that working in software, distributed systems in particular, gives me a handle on complexity—not a handle like making it work perfectly, but a handle like moving forward anyway. And that works in life too.

Diana Montalion: Right. As you're talking, I think about how we grew up thinking these rules and structures were solid, real. The paradigm was "what is truth?" I wonder if that's a constantly evolving thing. I agree with what you're saying about thinking it's in our lifetime that we discover this thing we thought was true doesn't quite work the way we thought.

But I think maybe everyone throughout time is always experiencing that shift, because we're always figuring something out. As you said, something's always broken, something's always working, and something's always broken. When we talk about leverage points in systems—meaning the place where you can make a difference, where you can change something that really will have an impact rather than rearranging deck chairs on the Titanic—I love how you're describing that.

In a system that we know is always broken and always working and always in relationship and always complex and never certain, there are places we can intervene.

Jessica Kerr: Yes. And this is very abstract, so I want to give concrete examples. It's safe to give concrete examples from software. When your distributed system is having a bunch of failures, sometimes you get to fix the bug. That's a great day.

Diana Montalion: Exactly. It's in the code.

Jessica Kerr: It's right here. My for loop isn't working—that makes sense.

But other times we don't get to make that kind of detailed rational "I can reason this out" sense. We have to observe that there were a lot of errors. I can't reproduce it, but I can detect that a downstream service was timing out. It was timing out after 60 seconds, and my system was slowed down by holding all those connections open for 60 seconds.

I can't fix that downstream system—it's outside of my domain—but I can shorten my timeout. I can ask, "When that system fails, what can we do?"

Here's something I've learned recently: the difference between morality universals that ask "what should you do" versus ethics, which are situational and ask "what can we do?" Given the people we are, the relationships and response options that we have, what can we do?

What we can do is shorten the timeout, do fallbacks, or give the user an informative message that says "it's not working right now, it might work in a few minutes, or I can tell you when it's back up."

Jessica Kerr: If you're being industrious, we get to decide how to respond to these failures, and that's really important. You can't decide ahead of time how to respond to every possible failure, which gets us into AI-assisted coding. The agent writes code and checks every error six different times, which is really messy. Some of those errors are never going to happen, and even when they do, you didn't actually do anything useful. You just fell back and hid the error. It's so good at hiding its own failures and the failures that happen at runtime.

One response option is to hide the failure. Sometimes that's great if it gives the user a smooth enough experience, but much more often, it's hiding the failure from us as developers. I want failures to fail loudly in the telemetry at least, and then tell the user as much as they need to know. How we respond to failures matters so much more than preventing them.

Recommended talk: Learning Systems Thinking • Diana Montalion & Charles Humble • GOTO 2024

From Error Handling to AI-Assisted Typing

Diana Montalion: In software, like in life, by reframing boundaries, we can improve our relationships with what we think of as the broken parts. Maybe they're not broken. Maybe in their context, slow is perfectly fine, but in our relationship to them...

Jessica Kerr: Maybe it's not them, maybe they're fine and it's the network.

Diana Montalion: Or something in between. It's all within the connections. This is what happens in my brain with rumination. If I have a problem, my brain wants to fix it. Keep thinking about fixing it. We're getting that from AI too - it just keeps going back to the problem.

Jessica Kerr: It wiggles and wiggles and wiggles.

Diana Montalion: The ability to respond rather than react shifts us toward improving the interaction, improving where the boundaries are in which we can improve that relationship. I'm surprised we're going to AI coding because one of the challenges for me is inference. I would have assumed AI is actually bad at the gaps between knowing. I can know this part and how fast it goes, and I can know this part and what's going on in this part of the system. But these problems are about the inference between them, the relationship between them. It sounds like you're having experiences that give you insight into how AI coding actually helps us do better with that.

Jessica Kerr: Sometimes we don't handle all the errors that our code could possibly throw just because it's more typing and it clutters up the code. More typing is no longer a problem - it's very fast at typing. I should clarify that with AI-assisted coding.

I was trying to write something yesterday in Java, which I haven't used recently, and I wasn't signed in to my AI assistant correctly. It wasn't autocompleting, and I was like, "What is wrong? I can't type this." I wrote Java for 12 years, but I don't have that expectation of understanding it at the top of mind anymore because something is going to type it for me. I write the comment "if this is null, set an attribute" and the syntax is immediately handled for me.

It's good at typing. It can do more error handling when our restriction is the time of typing it out, but it can't do the thinking. All it can do is what other people on the internet in open source code have done. You know what I do in my public repositories? My public repositories are all demo apps. I don't do good robust error handling in my demo apps. I send a good error message because that's one of my obsessions and because I work in observability now, so I'm careful with my telemetry. But I don't think through all the timeouts and stuff. If my demo apps fail, great - observability gets to shine.

Observability is a great field to be in when speaking about live demos, because every failure is an opportunity. Which is a lovely way to feel anyway.

Diana Montalion: The reason I remain dubious, yet I'm having a really positive experience of learning which tools help and how to use them and how to apply them. I wasn't ready to admit that. Apparently I'm ready to admit that now. Only with you.

Jessica Kerr: You only know when it comes out of your mouth.

Diana Montalion: The typing really has made me more effective and efficient because of the typing and also the synthesizing. I'm looking for patterns in a solution I don't know how to solve. The fact that it goes to repositories and answers and discussions in a way that can synthesize that - it takes me a lot longer to do that synthesis. The debugging and error handling recommendations are not always correct. Maybe they're not even often correct, but when they are correct, it has saved me a lot of time.

Jessica Kerr: Sometimes they're wrong in ways that lead me to the correct answer. Sometimes getting the AI to code something is inspiration to code it well.

Diana Montalion: Right, because if something is wrong on the internet, I will now do it right.

Jessica Kerr: Exactly. I know it's wrong so quickly in my IDE. Now pretty often I'm like, "No, go back to the beginning." Sometimes I'm like, "Well, except actually that part is useful, but completely differently."

Diana Montalion: We're talking about this, but we're often looking at things that we've looked at for years and years. When we look at bad code, we find it bad. I can't always define exactly why, but we've developed that taste over decades. How is that for newer developers using it as a tool who might not have that "I can sort of smell it when it's wrong" sense? How do they learn to discern what to use and what not to use in the answers?

Jessica Kerr: I'm not a new coder, so I'm probably not the person to ask, but a friend of mine who is new to coding uses AI, and one thing I observe is that she's able to just do more. There are things that weren't approachable that are now approachable. So what if the code is bad? Does it matter sometimes? We need to notice that in production and respond when it is.

I think observability is more important because there's just more code and we don't yet know which errors we should be responding to and how. We don't yet know which bad code matters and what's fine. So we need to be able to see what it's doing. Also, we don't have to do the typing anymore, so it's easier.

Diana Montalion: So you want to have more of a safety net before it goes into production.

Jessica Kerr: Or during, depending on the risk profile of your app. Everyone should be testing in production. Your customers are testing in production. Try looking at the results.

Recommended talk: Gamification, Sys. Thinking & the Power of Observability • Jessica Kerr & Jessica Cregg • GOTO 2023

The Art of Scaffolding: Scripts, Tests, and Automation

Diana Montalion: Engaging with AI accelerates learning because we can practice more and do more practice in a shorter amount of time. Practice is key.

Jessica Kerr: We can build scaffolding. A lot of what we code is actually scaffolding - tests, deploy scripts, automation, integration tests. When I need to test something by clicking in the browser a couple times to see that it looks okay, that was hard to automate. It depends on how much I've loaded in my brain, but I don't want to load up a JavaScript testing framework when I'm writing a Python app.

I can have the AI write a script to do that manual test for me, and then have it run that script every time it makes a change. I have to care about whether that script is a decent expression of what the app needs to do, but I don't have to care what that code looks like because it is scaffolding.

So I can get way more scaffolding way faster. That's not going to production. If in its Playwright script it's trying three different things to pull the trace ID out of the browser data, and two of those are complete garbage and will never work, but one of them does and the rest is just cruft - it's cruft. It runs on my computer and I don't care.

Scaffolding is now more available to us, and it's also way cheaper because I need to look at all the code that's going into production. I need to find out when it's cheating and making the tests pass without actually doing the thing. It's good at that.

Diana Montalion: Our brains are good at not seeing problems we create ourselves. I can see problems if I didn't generate the code, but if I generated it, it's completely invisible to me. That's why we have code reviews.

Jessica Kerr: It's like the stuff I left on the kitchen counter - I know why it's there, it's fine. But the stuff my family members leave on the counter? What a mess, so sloppy, what is this doing here?

Diana Montalion: I go have a slice of pizza, take a shower, come back and look at the code, and suddenly I can see the problems. You're describing this at a systems level - knowing there's always going to be blind spots or gaps.

Jessica Kerr: The AI does the sycophant thing of telling you what you want to see - passing tests or deleting tests if you let it get away with that, making everything look green, falling back on a failure to something that might have been returned by the downstream system. You can't see the failure until you get to production or load test and wonder why it's returning the same value every time. It was good enough to get past your sniff test, but you really have to watch it.

Diana Montalion: There's always delusion in systems, whether it's a human system or technical system. The delusion is willful.

Jessica Kerr: Sometimes that's what lets us work, because we can't understand the whole system. Part of how the system works when parts are broken is that we don't understand the whole system and we don't understand what we're trying to achieve. We can't imagine utopia - whatever you imagine is bad. We don't need to imagine the perfect future we're trying to get to. We need to imagine the next better state from here. Not what should we do, but what can we do, and how do we get there so that next time we have more options and more ways to respond.

Diana Montalion: And more insight. We don't want just a pile of information. We want the right information at the right time that helps us make the right decision when there is no such thing as "right" - we have to discern.

Jessica Kerr: What I want the system to do first is tell me what it did. I don't know exactly what the right thing to do is every time, but I know I'm going to need to know what you did. Then we can come back with more information and make it a little better, a little smoother, and move forward with better responses.

Diana Montalion: Using AI for scaffolding and especially testing scripts almost makes me emotional because that's the typing I most want to avoid. My entire career I've wanted to back out of writing tests.

Jessica Kerr: I can have all the tests now! The other day I was making a toy agent with AI help. I got it to create software that was close enough that I could start shaping it. I've got it outputting a conversation, and my code calls into a fake LLM named Frank because otherwise I'd spend money on stuff that doesn't work.

I had it write the conversation history, then Frank replays the conversation between the prompt, function call, and response. I thought it would be easier to read if it was formatted like cowsay - the cow played the role of the prompt and maybe a dragon played the role of the LLM. I asked it to make me a script that reads the JSON file with the cow printing the prompts and a dragon on the right printing what Frank said. Done.

Now I can read my conversations and enjoy it. It's really good at drawing cows in ASCII, but has no idea how to draw a dragon - it's just some lines that look like little pigs. That's the saddest dragon and I love it.

Diana Montalion: I've given up on visual rendering because I'm not good at it, so I need real support there. But AI often fails at that too.

Jessica Kerr: But it's great at Mermaid diagrams, which help in your README on GitHub. It's also great at HTML and CSS because it can run Playwright. I've configured Playwright MCP in the agent - it can run the browser, look at the output, take a screenshot. I can ask it to modify the appearance of a website, and it gets farther than I would in a short amount of time, unless I already know how to do it.

Diana Montalion: I'm trying to move components I've built in a completely different tool into another frontend tool. That's not my strong point, but AI is much faster at that than I am. If it's translating HTML and CSS grid structure into Tailwind syntax, that takes me a long time.

Jessica Kerr: It can do that, and then we can fix the problems. It gets us there. We have the responsibility to respond to the problems after it gives it a try.

Diana Montalion: If you don't have to do all the heavy lifting, you get to discern which heavy lifting matters. I'm not sure I've yet learned to trust that we're good at that discernment.

Jessica Kerr: That's the skill we need to develop - not just coding, but knowing when to do the coding, when to delegate the coding, and when to accept or reject what AI produces.

What I loved about programming in college was that when I started as a physics major, it was fun until we got to quantum mechanics. Then I thought, "What do you mean you can't know for certain where the electron is? What do you mean you can't predict its trajectory? It's all statistics. I'm going to programming where it's deterministic."

Diana Montalion: I love our shared history because I also studied physics. Unlike you, I thought the linear mechanistic approach was boring - I could diagram and memorize it. But when I got to quantum mechanics, I thought, "This is real."

Jessica Kerr: You weren't twenty!

Diana Montalion: That's true. When I was twenty, I wasn't interested in any of it unless it was fun. My definition of fun has evolved though.

I think we've come full circle. Originally you were talking about relationships between software parts - controlling and changing what you can, making better interfaces, better boundaries, better ability to respond. I'm hearing you say the same thing about our relationship to AI as a tool for our work. It's the quality of the relationship we develop - what's being communicated, what we're asking, how we use what we're hearing, discerning what to do. The quality of the relationship between me and the AI responses is like learning to work with any downstream system.

Jessica Kerr: Exactly. The AI tools are non-deterministic, unlike perfectly correct auto-completions. We don't get full control anymore, but it turns out relinquishing control gives us the opportunity for a lot more capability and response. The AI tools are powerful in a way that perfectly correct auto-completions were not.

AI as a Thinking Partner

Diana Montalion: I'm going to ask a question and I apologize ahead of time because it's not really fully formed in my head. So maybe it doesn't lead us anywhere fruitful. I'll start with a statement. One of the things I'm discovering is that as a person always thinking about understanding and describing systems, I have found that my AI tools understand me better than anyone has before.

Jessica Kerr: Oh wow.

Diana Montalion: There's a synchronicity in what it means to architect these relationships. I have usual rants and frustrations about asynchronously designing in a linear thinking world. What I'm finding is that the responses, the help in thinking about this is actually good. And I'm surprised by this. I don't know what to make of it.

I'm sort of still in the closet - I secretly use it a lot more than I'm confessing because it's helping me design a front end monorepo using NPM workspaces, which I hadn't done before. There's not a lot of documentation on that. I think my question is: it's a tool that improves your thinking because it makes it faster and gives you information faster. But is it also helping you as a systems designer in finding leverage points and knowing these more nuanced things that are really hard for us? How is AI improving that skill for you?

Jessica Kerr: I'm not there yet. The systems that I'm working on are still demo apps. I'm not working on our production Honeycomb systems.

Diana Montalion: This is beautiful because when I tried it on my laptop, I was away and downloaded the repo I worked on and tried to spin it up on my laptop. It didn't work because it's not robust - there were packages missing that I had on my machine. As a demo app, it helped me. But as a production something, it was very far away.

Jessica Kerr: I will use it on existing fairly large demo codebases. I find it interesting that you talk about how it helps you think. If we use it well, it can help us think. If we try to get it to do the thinking - total failure.

Diana Montalion: That's the thing. I was hearing a lot about how you get it to do the thinking, and that brought a lot of my doubt.

Jessica Kerr: You get it to do the typing and then separately, in chat, especially when it has a good context engine and can dig around in the codebase and point you to things that might matter. When I hook it up with the Honeycomb MCP, it can also do queries about what's happening in the software when it's live, which is helpful.

It can gather context and run a lot of queries and dig around and bring stuff to me that I wouldn't otherwise find because it's just too much work. It can totally contribute to the thinking, but the thinking is still yours.

Diana Montalion: If we come back to concrete examples for people that listen to this conversation and think, "All right, I'm going to experiment with my thinking relationship"...

Jessica Kerr: I will point out concretely, one thing that you can do to improve the relationship: unlike a person, it's not going to magically learn your style over time. Your interactions with it are not persistent, but you can put stuff in its memories. You can put stuff in its guidelines file. Different agents have different file names that they use for this, but you can change its instructions.

Whenever it makes me really mad, I'm like, "How do I make that not happen again?" And I either need to change my prompts or more often change my instructions.

Diana Montalion: So that it knows the scope of what to consider or the why.

Jessica Kerr: I'm like, "Don't write tests unless I ask you to write tests," because it loves to write a bunch of crappy tests. Does it still write tests that I didn't ask it to? Yes. But fewer.

Diana Montalion: Mine is like when we're solving a bug: stop giving me the answer you've given me eight times. I've already shown you this is not correct.

Jessica Kerr: Even with the AI we can work on the relationship.

Diana Montalion: I guess what I'm thinking is for people that have listened to this and been like, "Yeah, okay, so this is good. I want to put my hands into something." Maybe the more concrete start place is: what is the typing or what is the investigation that I'm doing that I could deploy AI to do some of that typing?

Jessica Kerr: What is scaffolding? What is the yak I would love to shave? What is this script that I wish I had, but it's way too much work? It's clearly not worth it. Well, it is when it's not you doing the work and it's just spinning for five minutes and it's done.

Diana Montalion: "This script I wish I had."

Jessica Kerr: Yeah, let's try it.

Diana Montalion: Perfect. Thank you as always - delightful. I feel more friendly and less doubtful. I hope that's true for everyone. Thank you so much.

Jessica Kerr: Thanks.