The Debugging Book

Intro

Clare Sudbery: Hello, my name is Clare. I'm a technical coach, and I am here today with Andreas Zeller, who's going to tell us about the debugging book. Andreas, would you like to introduce yourself?

Andreas Zeller: I am Andreas Zeller, I'm a software researcher at the CISPA Helmholtz Center for Information Security. What you would call a senior researcher in the US would be a full professor with plenty of experience in testing and debugging software, both testing and debugging software myself, as well as building tools and techniques that help testing and debugging software.

Clare Sudbery: Marvelous. And how long have you worked there?

Andreas Zeller: I think the first time I was debugging a program was at the age of 18, and I spent a long night debugging it until it worked. And that's now something like 41 years ago.

Clare Sudbery: Wow. What language was that were you using at the age of 18?

Andreas Zeller: That was a programmable Casio calculator, of all things.

Clare Sudbery: Fantastic. What were you getting it to do? What was the program?

Andreas Zeller: I think I wanted to get it to play some game, something like Monopoly, and there was something wrong with the alignment of the individual fields. I was programming games for that calculator and selling them to make a bit of extra money.

Clare Sudbery: Fantastic. I love that. And then you ended up being a software engineer. Was that a quick journey?

Andreas Zeller: Well, it was a bit complicated first. I found that I had a passion for programming and for developing. So I studied computer science, and I was pretty good at that. After I finished my studies, I was offered a position to do a PhD, and I thought that researching software is fun. And at the time I thought there are so many things that could still be improved, so let's do some research in the area.

I did some work in improving version control systems, which at the time were very new. Today, of course, common practice thanks to my great community who's been working on that. And while I was building systems, I also found that we still need to do much more when it comes to helping developers in testing and debugging the software. So after I did my PhD, I continued an academic career, which would give me all the freedom I wanted for this particular purpose.

Revolutionizing Tech Education: The Executable Book Format

Clare Sudbery: Amazing. And here we are now. You've written this book, the Debugging Book. But it's not a book in the traditional sense. It's an unusual format. How did you come up with the idea, and want to describe a bit how the book is packaged? And how did you come up with that idea?

Andreas Zeller: So, traditionally in academics and in publishing, a book is a collection of pages. Whether these would be printed or come as a PDF these days for download, or whether you look at them on your reader. It's a text that is to be written like a novel would be written from beginning to end. Or maybe it's a reference where you can look things up, but this is also the standard in academic publishing. We publish papers, and we call them papers because they come in paper form.

But when we are doing our research, when we are building things, we typically build something that is not just made to be described in words, but is actually something that is to be used, that is to be dealt with, that is to be executed. So most of our artifacts in research in my field actually come in the form of programs that you actually write, and then you run them and then you see that they work or they don't work.

And the same goes if you're building a new tool. For instance, say I build a new debugging tool because I have this new idea for what could help you in debugging. So I implement something, and then I see that it works. But then magic happens, the tool disappears, and all I do is write a paper about it. Of course I can make the tool downloadable somewhere, but the two things are very separated, classically.

The idea is actually to bring these two concepts, the concept of a book and the concept of an executable program, together in one single format. While you already have the book on your screen, because most of us, particularly in the academic world, are reading things on screen anyway, when you have these things on your screen already, why not just go and interact with them directly, trying out whether something works, trying out what the effect of something is.

When I, as an author give an example, the idea is to make this available immediately to the reader, such that the reader can directly toy with it and experiment themselves. Such that figuring out how a particular thing might work. This is the key of the books I'm doing these days, notably the debugging book. The idea is that you're reading things on the screen. You're reading about a particular technique, but you also follow the steps of how to build this, what comes out at the end. And you can toy with it directly, interactively on your screen and try out how a particular technique works. Try out other parameters, variants. You can even add your own code to the whole thing and make your own experiments.

So it's much more than a book in the traditional sense. It is actually a playground where you can do your own experiments, change things at your liking. You can even go and change the writing if you like, except that your changes stay with yourself and don't propagate to the book themselves.

Clare Sudbery: What users get is a combination of short video clips and printed words, which is navigated via a website. So you're clicking through to get from one chapter to the next. And also these Jupyter notebooks. So you're making the code available. There's a GitHub repo where you can browse the code as you would any other code base. But there are also these Jupyter notebooks that allow people to interact with the code. You want to talk a little bit about that technology?

Andreas Zeller: Jupyter notebooks is a technology that allows you to combine code and text to your liking, as well as the results of the code that you execute. It is mostly used in data science where you go and analyze some big chunk of data and you want to see the results of your analysis, such as plots or drawings. You want to see these immediately. So what you do there is you write a piece of code that goes, okay, I want to load this data set and I want to plot this particular data set. I want to do a prediction. This allows you to interact with the data as you like.

The nice thing is it also allows you to add text such that you can explain what you're doing. So this is helpful because if you want to share your notebook later on, then you can actually have text that explains the individual steps that you're doing. And the text actually comes in a very rich format. It's not just what you would have in a program like typewriter text. You can add formulas to it. You can include images, you can do anything that a professionally typeset book would allow you to do. So you can also have tables and formulas and images and figures and whatnot.

These Jupyter notebooks, as I found, are also great tools for authoring in the style that I do, because there I can take the text fields and then put in what you would normally put in a regular book. So you explain a particular technique, you give the references. This is how it works, and so on, and then come short pieces of code that implement these individual techniques.

So step by step I can then show the reader, okay, first general idea. This is what we plan to do today. Here's the general idea. And then let's go step by step. While I'm formulating my thoughts, let's first do this step, let's do that step. I can illustrate this immediately with code that implements the whole thing.

So this means that in contrast to a regular book that shows how general technique works, I can illustrate each of these steps with actually executable code that readers can try out themselves. And on the other hand, compared to just regular program code, which is normally just a long bunch of instructions that are typically not very well documented, but they work, in contrast to that, I do have plenty of documentation that illustrates the individual steps.

So readers can use the program code to better understand the meaning of the words that preceded. And they can use the words to better understand what the program is doing. From a pedagogical perspective, this is, of course, a nice combination because I can be both casual in my natural language text, but I can also be very precise and formal. That's what the program code is doing, because the program code is the ultimate authority on what will be happening in the program.

What the Debugging Book Teaches

Clare Sudbery: We've been talking about the format of the book, and yet we haven't talked about what the book is about. I think people can guess because it's called the debugging Book. But you want to talk a little bit about what the book is about, what it is that you want to learn by using and interacting with the book?

Andreas Zeller: Debugging is the act of figuring out why your program fails and fixing the program accordingly. So you have a program, and the program crashes or it does the wrong thing. And then you realize you must fix the whole thing. And for a programmer, these can be very challenging situations, because you first have to find out where the bug is before you can fix it.

And if you think about it, programs can easily have thousands to even millions of lines of code. Somewhere, something is wrong in there. It might be your code. It might be the code of someone else. And you have to figure out where something could be wrong in here?

So, all of a sudden, you change the perspective. First, as a programmer, you're the creator of things. You build things, and they run. And now all of a sudden, you're a detective. You have to figure out what went wrong. And often it is even your own code that is wrong, which means that you are in this situation where you are both the detective as well as the murderer, as one puts it. So you have done something wrong, but you don't know what and why or where. You just know that it was you, okay? But now you have to figure out what actually happened, what did you actually do wrong?

And for many programmers, this is difficult because they have to carefully then revisit their assumptions on how the code should work, because one of these assumptions must be wrong because otherwise it would work. On the other hand, you also cannot simply assume that everything is wrong because something must be right. Otherwise, you would be searching forever.

So figuring out how to best choose and how to best search in your code where to go for this is a hard task. And since searching for the cause of the bug is something that can take hours, days, weeks, sometimes even... If you have somebody who is sitting in front of their code until well after midnight, that is typically because they are still debugging something. And they are looking into it. They have a number of hypotheses on what might go wrong. They've made a number of checks. Is this hypothesis true? Is it this? Is it that? Like a detective working on a murder case.

Many programmers then simply keep things in their head, and they're super concentrated, and you cannot disturb them under any circumstances, because otherwise they would lose everything they have built up in their head. And for the same reason, they cannot stop. That's something I also experienced as a younger programmer. I would debug things until 3 a.m., 4 a.m. in the morning until it would finally work. And then I found it.

But the nice thing is that there are a number of techniques that can very much help you in guiding your search and in figuring out where the bug is. These techniques that help you debugging are called automatic debugging techniques. That is, they help you finding out where the bug is and what the causes are. These are the techniques that I illustrate in this book. And as I said, I don't just illustrate them, I actually implement them. So with the techniques that are built in the book, you can actually debug your own program just as well.

Recommended talk: Five Lines of Code • Christian Clausen & Kevlin Henney • GOTO 2023

Interactive Debuggers and Advanced Techniques

Clare Sudbery: So let's talk a little bit about the details of that. How about we start by talking about interactive debuggers, which I know is a topic that's very central to the book. You want to talk a little bit about that?

Andreas Zeller: Interactive debuggers are the standard technique that programmers use these days in order to figure out what is going on in the program. When you're just reading the code, reading the code is how the program tells you how the program should work in the abstract. That is, how it generally works. Many people are able to spot bugs in there just by reasoning about the program.

But if your program interacts with lots of interfaces, which you cannot fully trust, and where you cannot read the code. So whatever you're doing, some call into some library that is coming from a third party. You typically don't want to debug this as well. You don't want to know what's going on. But still, the bug might be in there, or it might be you doing something wrong, and then the library returning something wrong.

In order to figure out what is going on in the program, that's where interactive debugging tools come along. These tools allow you to execute a program in small parts, step by step, or also in larger chunks. And then they allow you to inspect what the program is doing. For instance, programs operate by assigning values to individual memory contents called variables. So you can look up: is this program variable, does it have the right value? Is the program in the current state that I expect it to be? And what if I make use of some third-party function? Does the function give me the result that I expected it to be?

So I execute the program under a microscope, so I can actually look into what's going on at each individual step. And then I can also halt the program and resume execution just as I like. So that's already a very helpful tool for programmers. The downside is that it's not very automated. So if you have a long execution of thousands and millions of lines, figuring out the exact place where to look at and going through things can take a long time.

Clare Sudbery: And depending on the language and the framework that somebody is working in, most programmers will have access and will already be using debugging tools. But I think one of the things that often happens, and certainly for a long time, when I was new to software engineering, I was really only minimally using interactive debugging tools. So there were lots of capabilities of interactive debugging tools that I was barely aware of. For instance, something like a conditional breakpoint where the code will stop its execution at a time, or in a circumstance that has been determined by you. And I think that may be one of the first things that you help people with is how to make the most of those interactive debuggers. Do you want to talk a little bit about the maybe underused features of interactive debuggers that people maybe don't make best use of?

Andreas Zeller: One of the least used features of debuggers, yet one of the most powerful, the so-called watch points. So you can specify in a debugger that the program should halt under certain conditions. You can set a breakpoint in program code. And whenever execution reaches that location in the code, then the execution will stop and will allow you to inspect what's going on.

However, you can also set a condition on these breakpoints. And these conditions would be things that you're particularly interested in. Maybe you want to stop the program only when a particular variable has a particular value.

But there is something that people rarely see, and that's also the ability to check a program continuously, whether some variable ever gets a particular value. And these are called watch points. These are things that conceptually get executed with every line in the program. So you can set a watch point on a particular variable, and then you can say, okay, now I'm going to run the program. And whenever this variable, if it ever gets a particular value, then I want my program to stop.

I can also make it stop in case the value of this variable changes. And if this variable, for instance, is an error indicator, then I can set the watch point on this variable, waiting for it to actually get flagged as an error just occurred. And if I interrupt the program at this very moment, at the moment the variable gets assigned, then I jump directly into the piece of code that actually makes this happen. And this is something that is extremely valuable because it gives me a first idea: where did this value actually come from?

Python as the Language of Choice

Clare Sudbery: You are using one particular language for all of your examples. So obviously you're going to particularly focus on debuggers that people can use with Python, which is your language of choice. How did you land on Python? Why is Python the chosen language for this book and all of the examples?

Andreas Zeller: In my book, I wanted to get the central mechanisms of automatic debugging and implement them so that readers can understand how they're working and can either find them in their own tools or can implement them in tools that they build themselves. And for this, in order to show off how all these tools work, Python turned out to be a most excellent choice for a number of reasons.

First, Python is a language that is very popular. Plenty of people know Python and if they do not know Python, they can learn Python in an afternoon. Even if you come from another programming language, it's not exactly the language that is known to have plenty of huge hurdles for beginners. In particular, if you already have an idea of what programming is. So this is purely pragmatic.

The more important thing, however, for my purposes, is that Python is an interpreted language, which means that you don't compile it with extra compilation steps, but you can execute things directly as they are, which of course is helpful if you want to have a book where readers can actually execute the code when they see it. Because if you tell them in order to execute this piece of code, you first have to go and put all this extra setup code around it and then compile it, and then invoke it as follows, oh that's horrible.

Think of a Java program where in order to do a hello world, you need to do what is it: public class, and within it a static public... public string main string args System.out.println. Hello, world. There you have it. Something like that. I need to look this up every time I want to write a demo example in Java because I can't remember the right order of things. But if for every single example, I would have to come up with such a scaffold, this would make things far more complex. And in Python, of course, this is just print.

The most important reason, however, is that in Python, it is possible for programs to actually inspect other programs and themselves very easily. So a program can actually, at runtime, check what the individual values of its variables are, and it can also step through its own execution as well as through the execution of other Python programs.

Which means that if I want to build an interactive debugger in Python, that is going to take me something like 30, 50 lines of code. And then after 50 lines of code, I have everything I need. I can show off: here's how you step through a program. And here's how you look at individual variables. If I do this in any other language, notably in a language that comes with a compiler, it wouldn't take me 50 lines of code, it takes me two months.

Because then I would have to go and expand a Java compiler or C compiler with debugging features, or I would have to show how all these things are implemented in a C compiler or Java compiler. All of this is horribly complicated because in a compiled language, you have to carry all of this information, this runtime information: where are the individual variable locations, how are they translated into machine code, how can you reverse the machine code to the original execution order. To make all of this possible is a huge engineering effort.

But in order to illustrate how this works, this is much better in an interpreted language. And Python is absolutely great language if you want to build debuggers.

Recommended talk: The Quick Python Book • Naomi Ceder & Luciano Ramalho • GOTO 2022

Tracking Dependencies and Delta Debugging

Clare Sudbery: Let's talk a little bit about the topics that are covered in the book. So one of the things that you talk about is, in the context of debugging, tracking flows and dependencies. So tell me about that and how that relates to debugging.

Andreas Zeller: The key thing is that when you are debugging a program and you find that at the end your program fails, that's where you typically start. Your program fails because it gets into some state that it is not expected to be, or returns some value that it was not supposed to produce. So the result of your computation is wrong.

So the question is, where does that result come from? So we have to trace back the steps that particular values come from, which in traditional debugging, notably in debugging with an interactive debugger, is actually fairly difficult because your program executes forwards, whereas your investigation on where individual values come from has to go backwards.

So I have to go back in time in order to figure out where this value comes from. Does it come from some other value? And every time, also, you have to check whether the value it comes from is actually valid or not. But this is a classic way of debugging. So you're reasoning backwards and you're following the origins of the value.

First, you have a value that is wrong. That's what you start with. And that's the very end of the computation. And then you check out its origins, and you also find an origin that is wrong. And then you check the origins of that origin, and you find another origin that is wrong. And you continue to do so until you find a situation where some inputs are all correct, but the output is wrong. And that's the function, the one function or the one line of code that did it wrong. That's a typical way of progressing.

As I said, in interactive debugging, that's difficult because your program executes forward. But there are nice tools that allow you to track these dependencies. And the idea here is you go, and every time a variable is assigned a value, you track where the value came from. So you check where the value came from. And you keep this history of all variables until the end of the computation.

When you're at the end of the computation, it's much easier because you can simply follow the origins. It's like navigating a map. You can see, okay, this value came from here. You go to that location. What is that value? Oh, it came from there. Then you go to that location, and so you can progress much quicker in your search than you would normally be able to.

And tracking such information, of course, is tricky. Such information of course, is somewhat expensive. So it slows down the program. But it's a great help for debugging and that's something I illustrate in the book: how to augment the program execution such that those dependencies and origins are automatically tracked. When your program fails, you can be directly pointed to a number of origins where this particular value that is now wrong could possibly have come from.

Clare Sudbery: In that context you also talk about delta debugging and failure-inducing inputs. You want to talk a little bit about that.

Andreas Zeller: That's another technique. Delta Debugging is actually a debugging technique that I am famous for. So I think if there were a Wikipedia entry on me, it would certainly mention me as the inventor of delta debugging. By the way, if there's any readers around, I don't have a Wikipedia entry. So maybe it's time to build one. I cannot do this myself, but delta debugging has one.

Delta debugging is another technique for automated debugging. It's also a very powerful one. The idea is you have a very long and complex input to your program. And if you feed that long and complex input to your program, it fails. And since a long and complex input also means plenty of steps to look at and plenty of memory to examine, this is difficult for programmers to handle. So programmers typically ask to simplify an input to the largest amount possible such that it becomes easier for them to debug what's going on.

And delta debugging is a technique that does the simplification automatically. The way it works is pretty straightforward actually. It's a wonder that such a simple technique that fits into something like 20 lines of code actually deserves a name on its own, but I invented it. It works like a charm.

The way it works is as follows. You take the entire input, and the program crashes. So far, so good. And then it cuts out parts of the input and sees whether the crash still occurs. So it takes out the first quarter. Crash still occurs. Great. Now you just have simplified your input by one quarter. It takes out the next quarter, takes out the next quarter and so on.

And the key thing about delta debugging is that it starts by taking out huge chunks of input. Actually it starts with half of the input first. And then when it doesn't see any progress, because you can't simply take away half of an input and expect it would still work, in many cases, you can't. And if that doesn't work, then it starts taking on smaller chunks. So it takes quarters and then eights and then 16th. At the very end, it takes out individual characters and see whether this helps.

It seems that every time, with every iteration, it checks whether the crash occurs again and again, and it keeps on doing that until your input is small. Examples of megabytes being reduced to just two individual characters. So it turns out, these ten characters already suffice to bring your program down. And with this already, programmers, simply by looking at what remains of the input they already see: hey, it's related to this, because this is the one single feature that is left in the end, and that's already a great help.

And then it also helps with figuring out whether there are other bugs that actually get reduced to the same input. This can be extremely helpful if you find out that your payment processing fails simply because somebody has a loud character or some eight bit character in the name or some weird UTF-8 encoding or, how about that, I have a payment to a person who has an emoji in their name, something like that, because somebody thought would be funny to put in an emoji in there and everything fails. If you find out that your program fails whenever there is an emoji and just this emoji in your input, you immediately get an idea of where an error came from.

Clare Sudbery: Would one example of that be, it if, for instance, you deliberately created a string that contained all of the characters in a particular character set because you wanted to see whether or not there were any failures, whether there were any problematic characters, then you might find, and so in many contexts, you might find that many of those characters were problematic.

So would this technique allow you to identify all of the problematic characters? Rather than just one of them, if you see what I mean.

Andreas Zeller: The debugging part comes after you have found a failure. So what you're describing right now is actually a testing technique. But yes, there are testing techniques not known as fuzzing that produce plenty of random inputs to the program. And these would then also include, well, what we call nasty inputs such as well emoji or zero bytes or super long strings or whatsoever.

The typical combination is you first run a fuzzer that produces all these nasty strings, then hopefully finds a couple of situations where your program fails, and then when you have these nasty strings that you know they fail, then you apply delta debugging to figure out what it is in these nasty strings. What is the particular property of these strings that causes a program to fail?

And in the book, Delta debugging actually comes with a full-fledged implementation. So this is something that you can actually deploy every day in your program because it does not depend on a particular program code. This is something that you can apply to any program.

Recommended talk: Learning Test-Driven Development • Saleem Siddiqui & Dave Farley • GOTO 2022

Automated Software Repair and Testing

Clare Sudbery: So that's one of the tools that you share. And you share the code for it and you explain how it works. You also share the code for automated software repair. Do you want to talk a little bit about that, your tool that will automatically repair people's code for them?

Andreas Zeller: This is a very hot topic in research. In research and debugging, we are typically not just content with figuring out where the program has a bug or how the bug manifests itself, or, for that matter, how we can reproduce a bug with minimal input. But what we actually want is we want to suggest a repair to that very code.

This is mostly still a research topic, although there are companies like Facebook, notably, that have already deployed automated repair techniques. The way this works in practice is you have an automated test that runs every night. And whenever there's a failure, these bug reports then tell you first, here is a minimal number of steps to reproduce that failure or a minimal input. This comes from delta debugging. And the,n second, here is a possible repair for that very bug, a repair that we already found to satisfy all tests.

The way this works is first there is debugging to reduce inputs to a minimum. Second, there are techniques of so-called fault localization that try to locate potential locations in the code where the bug is. This can, for instance, come from this origin tracking that we found earlier. Because if a line of code is not part in the origin, then by construction it cannot have contributed to the failure.

Then what automated repair does is it tries to apply random changes to the code, random changes, which at first sight, sound horrific. But it also checks these changes on their impact on the failure. So what we do is we try to find changes that fix as many failing bugs as possible, but without touching the passing tests. So we assume there's a test suite with plenty of passing and failing tests. And every single change that we can make that causes one less test to fail without touching the passing test is a potential candidate.

This process can actually take lots of time. Because you have to recompile the code and try again and again. Another advantage of Python: you can just run things without having to recompile them. But in small examples and in specialized environments, this actually works like a charm, and it produces good and sensible repairs.

It, of course, assumes that you do have a very good test suite that actually is able to catch plenty of bugs. If you do not have that, then the chances are that repairs are going to fix symptoms rather than causes. In practice, many of the fixes look like: if the pointer equals null, then return. That's the fix. And this fixes all sorts of pointer errors. But this fixes things with air quotes only, because at the same time, it also removes functionality. And you want your tests to make sure that this functionality is still in there.

Clare Sudbery: That brings us to two topics that are very close to my heart. Two things that I'm very interested in: writing well-factored, easy-to-understand, and easy-to-maintain code. And another thing that I'm very interested in is test-driven development. But let's start with well factored code, because I think the first thing that occurs to me, if I'm going to use an automated tool to fix my code for me, is that it may well make the tests pass, but will the code still be readable and easy to maintain, and understand? So I'm interested in that side of things.

Andreas Zeller: Fortunately, the tools that we had so far in automated repair produce repairs that are fairly small. So we're talking about adding 1 or 2 lines of code or coming up with a fix for an off-by-one error, which introduces a minus one plus one, all that stuff. However, now with the advent of large language models in coding, where you have plenty of AI-generated code, the sky's the limit for whatever repairs can be produced.

And there all of a sudden you have a new risk coming in because your repair or the repair suggested by the language model is no longer small, but it actually suggests why don't you just replace these 100 lines of code by this other 200 lines of code, which I just hallucinated.

Just two days ago, I had to port a rather large, some 200 lines of Python code to C++. And I haven't programmed in C++ for something like 15 years, which is why I said GPT, please do that for me. Translate that. And that worked. And within a second, I had C++ code. Great. And then I used it and it compiled. It just didn't work. It just didn't work properly. And then I spent working for hours just debugging. And so let me say it produced code. But if I had built this code from the very start myself, I would have spent less time than if I had asked them to do CLI.

So that's just anecdotal evidence, of course. But the problem here is, the future of debugging may be more artificial intelligent as it is right now. And that may be a good thing, but I think that for debugging, we have to beef up the abilities of AI to understand code much better than it is today. AI is great in spotting errors in the code that everybody makes because this has been fixed in other places and this is discussed in tutorials, and that's where AI is great.

But understanding what's going on in the program, which is what the debugger does, stepping through individual things and figuring out what's going on, that's knowledge you can only have if you really execute the program and is not something that you can get by just examining the program code.

So if I had been able to not just give my little 200 lines of code, but also give it the 20,000 lines of context, and if the LLM had been able to actually execute this and test this, and then finally, maybe a few hours later, come back victorious, including having debugged it itself. Sure, I would have been very happy. But we're not there yet. So debugging will remain a manual activity for the time being, and our attempts at automating things will make things better. And notably, you can go to sleep earlier in the night and come back the next morning with a fresh mind. But for the time being we'll have to debug things ourselves, I'm afraid.

Clare Sudbery: I am interested in the relationship with tests. You talked about the automated software repair tool that requires tests. It needs a good test suite in order to work. And of course, if you've been using the test-driven development paradigm in order to develop your software, then you can be reasonably confident that you will have a good suite of tests. Do you have ideas about the relationship between test-driven development and the kind of debugging techniques that you recommend?

Andreas Zeller: Clearly, the more you test, and notably, the more automated your tests are, the better for debugging. Absolutely. Because for one thing, your tests of course, can help you detect failures in the first place. If you have a good test suite that comes with plenty of assertions or that simply checks for all sorts of things that might go wrong, that's absolutely great.

And the more such checks, internal checks, you have in your code, the easier it is for debugging. If you come into a function and everything is all right, and at the end of the function, something is wrong, then you already know that something has gone bad in that function, or at least during its execution, which makes life much easier for any programmer.

And second, many of the techniques that we discuss in the debugging book they depend on a good test suite: automated repair wants a good test suite, Delta debugging wants automated tests. There are very nice techniques that do statistical correlation between failing tests and the specific locations being executed. So called statistical debugging. This is very common. These also depend on a comprehensive test suite.

The better you test, the easier it is to debug in the end. And since the effort in testing is much more personable than the effort in debugging, testing is you just add one more test and yet another test and another test. That's clear progress all along the way. Debugging, you never know how long it's going to take. That's the risk.

I ask the folks in game development when they're being told, well, our big new release is in four weeks. And until then we have to fix all the bugs. And then they're working 20-hour shifts in order to try to squash all of them. Very obviously, that's something you can avoid. But that's where testing and of course test-driven development play a major role.

Clare Sudbery: I'm very happy to hear you say that. So I have one more question, which I think is something that's particularly interesting about this book, because the goal of the book is primarily to help people to debug their software. And you describe a lot of techniques that will help people to debug their software, but you also describe tools such as the automated software repair and the automatic delta debugging. And you also give the code for those tools so that people could build them themselves or edit them themselves, which in a sense, that's not just about debugging their code, that's building tools to help them debug their code.

So rather than just creating a tool yourself and sharing it with people, you're helping them to build those tools and get into how the tools do the debugging, as well as how they will do the debugging. And so I'm really interested in the motivation behind that and why you wanted to actually share the building of the tools as well as the using of the tools.

Andreas Zeller: It is very simple. In our education and training of programmers, we spend a lot of time on how to build code and program. However, we spend much less time on how to test programs. We spent a lot of time on how to build code and how to program. However, we spend much, much less time, we spend much, much less time on how to test programs. Already, we spend extremely little time on how to debug programs, which is a pain because about half of software development goes into debugging, and it will be even more with our Elm-driven vibe coding and whatnot.

You need to understand how to systematically find bugs in your software, and something that is hardly taught at all is actually the usage of modern techniques that help you find bugs automatically. That is, automatic debugging tools and that's the main reason why I wanted to collect these and some executable form, such that students get an idea about what is possible and oh, maybe at some point they can invest into open source programs for platforms of their liking.

I mean, it doesn't have to be Python. Maybe you want to build a nice, delta debugging tool for C programs or whatnot so they can invest in that and then make their own, debugging easier in the long run. But yes, that's an investment, and that's going to take time. And, yep. Unfortunately, I always felt that debugging is like the ugly stepchild of a program.

Everybody talks about how they built these beautiful things, but nobody wants to talk about what effort it was to debug them. And, people invest a lot into, driven code, into an Elm-driven code development, and whatnot. But, the real the real thing that eats up time and that is hardly that is that is very, very hard to plan.

That's actually the business risk is the debugging of code. And that's where, well, I hope, I hope to give you some ideas on what we could invest in in order to make debugging easier, printable and more enjoyable.

Clare Sudbery: Wonderful. And that is a fantastic place to stop. Thank you so much for talking to me, Andreas. It's been wonderful to meet you.

Andreas Zeller: Pleasure's mine. Thank you.