Gamification, Systems Thinking and the Power of Observability in Software

Software's evolving landscape goes beyond mere programming, creating far-reaching consequences. Dive into software development's intricate world in this #GOTOUnscripted talk where Jessica Kerr explores observability, gamification, and systems thinking. Speaking with Jessica Cregg, Kerr highlights the distinct satisfaction derived from collaborative tasks and the pros and cons of integrating gamification as well as competition-driven incentives in software engineering. The talk underscores the interplay of observability and feature flags for agile deployment and issue resolution. Gain deeper insights into systems thinking, a framework that has evolved to embrace software's constant evolution, and uncover how observability extends beyond monitoring to provide actionable insights. 

---

Going deeper into gamification

Jessica Cregg.: Hello, and welcome to GOTO Unscripted. I'm here at YOW! London. I'm joined by the brilliant Jessica Kerr, Engineering Manager, Developer Relations at Honeycomb.

Jessica Kerr: Right. Right.

Jessica Cregg.: Hello.

Jessica Kerr.: It's good to meet you, Jessica.

Jessica Cregg.: It's great to meet you. And you're joining me fresh from your keynote.

Jessica Kerr.: Right. It went really well. It's so much fun. I love going first.

Jessica Cregg.: Yes. Get it out of the way early.

Jessica Kerr: Yes. Yes.

Jessica Cregg: So, your keynote was talking about going deeper on gamification. Can you tell us a little bit about what that means?

Jessica Kerr: Sure, sure. First of all, we look at gamification of our work specifically, and go, "eh." Because gamification takes the superficial characteristics of games. Scores, achievements, competition. They're like, "Well, games have these, and games are fun. So if we just slap these on something else, maybe it'll be fun." No. There's a different kind of satisfaction that we get from our work than from a game, because, at work, everything is very interdependent. That's why we have teams. That's why we have companies. And our work has a richness to it.

Software development, it's not just programming anymore. It used to be just solving puzzles, and something you could do alone. But now, everything that we do has consequences, that are just broader and more interesting. There are some consequences...if we implement a feature, there are some consequences for the customers. There are also consequences for every other feature, every other feature we might implement, and the data and the database.Then all the other teams whose software interacts with ours. And what we do affects our future. So, reducing it to a number is really destructive.

Jessica Cregg.: It's interesting because we've seen a real growing around, sorry, real growth of these support systems design and systems thinking, and growing recognition of the interdependence of everything. Is that kind of what we're talking about here?

Recommended talk: Going Deep on Gamification • Jessica Kerr • YOW! 2022

Jessica Kerr: I read a lot about systems thinking. Systems thinking, as a field, has grown from being, well, you can't say "reductionist" with systems thinking, because the point is it is the relationships between the parts. But at the same time, systems thinking has been like, "Oh. Well, now that we've recognized that it's the relationships, we can model those. We have computers." You know, and so, it's, there's still this attempt to get control. Hard systems thinking, this is, is the general category, of just recognizing that it's a system, and also trying to master it.

But the more modern forms of systems thinking get into soft systems thinking and socio-technical systems, and Nora Bateson's term, symmathesy, is my favorite because it means learning systems of learning parts. You cannot hope to master that. You cannot hope to model it, because the system is becoming a new system every day. Just like us. Our software is becoming something new, and we are learning more and becoming different and gaining skills, or vengefulness, as we go. Right? So, you can't control the sociotechnical system that is a software team.

Pitfalls of gamifying software

Jessica Cregg: You're talking about the illusion of control. It just took me back to a couple of my favorite song lyrics there. Just, that came off my head, so, going full unscripted with this. But you were hinting at some of the pitfalls that people can face when they try to implement gamification and don't necessarily take into account the differences between developing software and completing a level.

Jessica Kerr: And playing a game. Or even sales. Someone tweeted, in response to my keynote, that a manager was like, "Competition makes people better, so I try to introduce friction in my team." And if you come from a sales background, I don't understand it, because I don't enjoy competition that way. But salespeople thrive on it. They have a different value system. There are different value systems for different, like, what does Luhman... There's a philosopher named Luhman, I think, who divides us into functional systems, as opposed to classes.

Doctors care about "Does it help?" And lawyers care about "Is it legal?" Could I defend it in court? And salespeople care about "Did it close?" Did I make money on it? And software developers care about "Does it run?" It always boils down to that, right? We can always take it to the computer and we can test it, we can run it, we can get yes or no. And that's our value system. It turns out that competition doesn't help with that. Because we can make more interesting things run, and run in more interesting ways, collaboratively. And competition seems to work for sales. Let them use that. But imposing that on software engineering doesn't work.

Jessica Cregg: Interesting. That's spot on. Everything is, when we do step back and look at things, the amount of importance that incentive plays into how we operate can't really be underestimated, right? 

Jessica Kerr: And yet, in software development, where, where we're looking at things richly and widely, and trying to think about more consequences, I think incentives are mostly a distraction. Because our job is partly to figure out what we need to do. Our job is constantly making decisions. What to type, what to test next, what to look at next, how to add clues for ourselves and when, and what to prioritize. Any particular incentive is usually a distraction. If we have an incentive to, I don't know, close more tickets, then we're gonna close them in ways that don't help in the future, that leads to more tickets. If we have an incentive to make more commits, then maybe we'll make smaller commits. I'm okay with that. I don't see any negative there. But I think most of what you can do to help software development is remove obstacles, smooth things out, is making it easier to find out what's happening in production. Making it easier to respond to a page. Making it easier to talk to each other. Making it easier to find out whether the thing runs.

Enhancing software with observability

Jessica Cregg: And that's where observability comes into the mix, right?

Jessica Kerr: Oh, true. Yes. Yes. I work at Honeycomb, in part because I believe very deeply in observability. Part of the magic of software development is we can inject those clues for ourselves. Not just does it run, but how does it run? What is it doing? Because there are so many different things it could do. Honeycomb helps with that. Honeycomb helps us surface those clues once we've stuck them in there for ourselves.

Jessica Cregg: And it takes, from what I understand, it takes the process of discovery when it comes to an outage from just, "Okay, it stopped running," to, "Okay, why did it stop running? When did it stop running? Who did the thing to create the issue?"

Jessica Kerr: Yes. And traditional monitoring can often tell you when it stopped running. Because it's counting things. But, "Why?" I mean, in a lot of places, you have to go to logs. And going to logs works just fine if you're the developer who wrote those logs. Or you've spent years reading those logs, and you know what they say, and you know how to notice what's missing, for instance. But to a new developer coming to the team, logs are not helpful. Whereas distributed traces are. Distributed traces have a structure to them that can teach people what's happening in the system. So, it sets people on a path to understanding. It gives people, like, a concrete reference to talk about together. We don't have as many whiteboards anymore, but you know what's better than a whiteboard? A literal description of what just happened in production.

Jessica Cregg: Sounds so helpful. I'm just looking back on all of these Hypnotoad moments I had looking at logs, thinking, "Hmm. I'm sure I'll just understand this. It'll just become very obvious, right?" 

Jessica Kerr: Oh, right. The weird thing is, eventually it does. Years later, if you look at it enough, you do develop a feel for it. But traces are right there, and Honeycomb even has...one of our killers features that nobody else has is bubble up, where you can say, "Okay, some requests are failing. Some requests are not. What's different?" And it'll be like, well, a lot more of the failing ones are in China. And you're like, "Huh. Bet that's not my fault." Or a lot more of the failing ones are for a very large customer. Or for a particular customer. Or for this one user on this one device. You can narrow it down like that if that's the failure condition because Honeycomb goes and it compares all the fields. The other reason we can do that is because we store all the fields. We don't index any of them. We indexed on timestamp, and a little bit of special handling for trace ID. But that's it. All your fields are created equal. And so we can look at all of them as easily as any other, and surface these really obscure, "Wow. I had no idea that discount code could make a difference here," kind of thing.

Jessica Cregg: And one of, I think, someone I met recently was, we were chatting about what they loved about Honeycomb, and they said that there's no, like, endpoint. It's a very open world...

Jessica Kerr: That's true.

Jessica Cregg: ...when it comes to observability. You can kind of zoom in as far as you need to.

Jessica Kerr: That's a good phrase. What specifically do our product people talk about, they don't want any dead ends. There's always, from a trace, you can get to graphs, and you can get to new queries. And from a graph, you can always get to a trace, or to other graphs.

Jessica Cregg: Going full, like Breath of the Wild with it. You can explore as many avenues as you like.

Jessica Kerr: Yes This is fun, because Breath of the Wild is one of the bases for Genshin Impact, which is the game I talk about the most in my keynote that I just did.

Jessica Cregg: Oh. Oh, incredible. Is that your favorite...yeah?

Jessica Kerr: Yes, because I love the open worldness of it.

Jessica Cregg: Yeah. Nice.

Jessica Kerr: I love that every mountain you see, you can also climb.

Jessica Cregg: This is the exact reason why I have not finished a game in the last three years. Because, I mean, what is finishing, when it comes to video games?

Jessica Kerr: The best games are infinite games. Right? They just keep going.

Observability and feature flags

Jessica Cregg: One of the reasons why I was really excited to speak to you as well is because Honeycomb usually makes an appearance when we're at LaunchDarkly. We are a Honeycomb customer, and we are...I think we're also just big fans of understanding why the thing went wrong and then using, hopefully, feature flags to be able to quickly enact on that, and...

Jessica Kerr: Exactly, exactly. Observability and feature flags totally go well together. Because, well, for one thing, feature flags let you deploy faster. They take away the restriction of, "Well, you can't deploy this because marketing's not ready," or because this other piece doesn't hook up yet. Or because the data hasn't been migrated, or, or, or... So, feature flags let you work in much smaller steps. Including adding observability, for instance. Although I don't know why you'd put that behind a feature flag. But also, you can see which feature flags are active, and Honeycomb can help you find the problem.

Jessica Cregg: Yes.

Jessica Kerr. But it has nothing to say about fixing it. But feature flags can help with that.

Jessica Cregg.: And if you are able to use one of the integrations, then you can be in that situation where if you fall below a certain metric, then you can trigger a flag to go off, or...

Jessica Kerr: Oh, that's true.

Jessica Cregg: ...into safe mode, which is quite nice.

Jessica Kerr: That's true. That would make me really nervous. But safe mode, yes, something like that.

Jessica Cregg: Safe mode. 

Jessica Kerr. You could be like, "Okay, the load on our database is excessive. We're gonna turn off recommendations." Or something.

Jessica Cregg: Everyone is asleep, and we need to let them sleep.

Jessica Kerr: On the observer, on Honeycomb's side, LaunchDarkly sends notifications that we can display as markers on the graph. So you can see when a feature flag was flipped.

Jessica Cregg: Hey. Exactly. Yeah, that's the slide that I always point to when I'm like.

Jessica Kerr: So, it goes both ways.

Jessica Cregg: I got quite excited when you were speaking about your favorite video game, your favorite game.

Jessica Kerr: Yes.

What video game made you a better engineer?

Jessica Cregg: Is there a video game that you've played that's made you a better engineer?

Jessica Kerr: Oh, a better engineer. Whatever my teammates are playing. Honestly, because if just, like, building a rapport with people. I don't play Magic: The Gathering, but if that's what people are doing, somebody's gonna loan me a deck, just happily. The best was when he had a team that played Hearts together at lunch, every day, for, like, five years.

Jessica Cregg: That sounds amazing.

Recommended talk: Observability Engineering • Charity Majors, Liz Fong-Jones & George Miranda • GOTO 2022

Jessica Kerr: Oh, it was amazing. Because we had all kinds of conventions and strategies and silly phrases, and things we made fun of each other for. That kind of rapport just smooths all kinds of communication. It changes asking a question from, "Oh, do I wanna bother this person? I see they're the last person to change this." It changes trepidation into anticipation, like, "Oh, I get to bug so and so. I'm gonna tease him about this variable name."

Jessica Cregg: Hey. How are you doing? Remember that thing that you... I feel like it also is really, playing games as a team also is great at uniting people around the one issue, and reminding us that we are on the same path to solving the same thing.

Jessica Kerr: Even if it's a competitive game. Mostly I play co-op games these days. Ever since COVID, I just wanna play co-op games. But even playing competitively with a team, when you're playing to play well, and not really caring whether you win, yeah, it's team-building. But also, even if you don't play together if you play the same games, then you can talk about them. That's why I play Genshin. It's because my kids talk about it all the time, and I wanted to be part of those conversations.

Jessica Cregg: Oh, that's so wonderful. That's great.

Jessica Kerr: Now I am. My teenager texts me, "Mom, there's all these Hú Táo-Albedo ships. What do you think of that?" And I'm like, "No, I don't think they would work together at all. Those characters would just be at each other's throats all the time, mate."

Jessica Cregg: Nice. At least you'd have better conversations around game mechanics with family members. I've got to say, I don't think I play any of the same games as my family yet. But it's probably because I've been in this really lovely lull, where I'm playing a lot of, like, this game called Coffee Talk, where you …

Jessica Kerr: Coffee Talk. I don't know that one.

Jessica Cregg: People come into your cafe, tell you little stories and you make them drinks, and they leave.

Jessica Kerr: Well, are they real people or NPCs?

Jessica Cregg: They are NPCs, unfortunately. 

Jessica Kerr: Low stress.

Jessica Cregg.: I think so, you know. Not like real users. Gosh. God forbid.

Principles of game design

Jessica Kerr: Not like people you have to, like, have manners with. That's funny. I do wanna leave people with more hope, because, while on my talk about gamification, I'm super down on gamification, I'm very bullish on principles of game design.

Jessica Cregg: Interesting.

Jessica Kerr: Yes.

Jessica Cregg: Okay. Tell me more.

Jessica Kerr: That's the thing. It's like, gamification is these superficial characteristics, but games do have something to teach us. But we have to step back and look not at what tools do designers use, in the case of board games and card games and sports, but what questions do they ask. What are they trying to achieve? And they're trying to achieve an experience. That's what I want to achieve in my team, an experience that also makes us better at our jobs every day.

Jessica Cregg: I thought that sounds like a rallying cry to join your team. That's excellent. What do you think people can take from, like, what did you want people to take from your discussion about game design to their teams? How would you wanna leave people inspired from this morning's session?

Jessica Kerr: Some of it is recognizing that we have the ability to design the agency that our team adopts, with goals, rules, and abilities. Set goals that align with the company, instead of numeric objectives that are small. And then, if we don't have the abilities we need to fulfill those, if we don't have the knowledge, if we don't have the tooling, if we don't have the permissions to push to production, or to look at production logs, work on that, or else de-scope the goals, or get the people that do have those permissions on the team, something, to make the abilities match the goals. Then, the rules that we follow, like the things we choose not to do, such as logging directly into production, we need to agree on those, mostly as norms. Some of them we enforce in software, like linting. I mean, if you're gonna be an asshole, at least let a robot be the asshole. Mostly let the robot just commit, and fix the stupid problems, and don't bother me. But...

Jessica Cregg: Choose bad robocop instead of bad cop. Sorry, that was a horrible comparison

Jessica Kerr: Yes, but for things like this what tools do we use? What variable naming conventions? What testing framework? When are we gonna update what? Social norms are much more flexible for that than standards imposed from the ivory tower.

Jessica Cregg: It's interesting because it sounds like implementing those measures will help people to believe that their efforts matter, and to see evidence of that in the work that they produce. So, is it...

Jessica Kerr: Which measures?

Jessica Cregg: Believing that you're... Tying your metrics to things that matter. Believing that you matter on your team. It seems like that's kind of what we're enforcing.

Jessica Kerr: Yes. And a lot of things, you can't measure, but you can notice. I don't want people to have individual performance goals, period, at all. But also, I don't even wanna have team goals of closing tickets, features being used, or something like that. Because the outcomes that matter are much bigger than our team. The outcomes that matter are that people are happy, people are recommending our software, and people are giving us money so we can continue to work. There's so much more to that than anything our team can do. If we focus only on a goal that our team can directly influence, is too small. But do notice these things. Notice how often you deploy. 

Someone told me, after the keynote, that they count the number of commits across the 300-developer org, per day. Notice when that goes down. "Well, maybe we should just take off the week between Christmas and New Year's because nobody's doing anything." Notice things like our trend of being able to deliver meaningful features is downward. Can we stop, and work on architecture, organization, or whatever it's gonna take to remove obstacles, and increase feedback loops, to help us maintain some level of developer speed? Because the natural trajectory of software is ossification. Add features, add features, add features until it's this big wad that you can't change any more. Also, as more people use it, it gets harder to change too. That's kind of a plus. I mean, the part where people use it is a plus, so I'm willing to live with that.

Jessica Cregg: It's like, it's better to have something that people use, right?

Jessica Kerr: Yes. But measure things. Inject the clues, the observability in there, so you notice who's using this feature. Then you can go back to products, or the product can do the query themselves, and talk together about whether is it because it's too slow. Are they actually not interested? Do they not find the button? Do they hit the button all the time, but mostly it's accidental?

Jessica Cregg: And then use those insights to actually create an action that...

Jessica Kerr: Iterate.

Jessica Cregg: Iterate. Yeah. In a way that matters.

Jessica Kerr: Because we're never done. We don't want the game to end. We want to continue having a useful product.

Jessica Cregg: While the game might never be done, I think eventually YOW! will unfortunately end. So, thank you so much for joining me today. I really appreciate it.

Jessica Kerr: Thank you.

Jessica Cregg: Thank you.

Jessica Kerr: It was great.

Jessica Cregg: Cheers.