Dev Harmony: Communication and Proven SRE Practices

Developer Experience and Happiness

Liz Fong-Jones: Hi there. Welcome to "GOTO Unscripted." I'm Liz Fong-Jones and I'm the field CTO at Honeycomb, and I'm joined by Marit van Dijk, and we're both here at GOTO Copenhagen 2023. Marit, would you like to introduce yourself?

Marit van Dijk: Yeah, I'm Marit van Dijk van Dijk. I'm from the Netherlands. I'm currently working as a developer advocate at JetBrains. And before that, I was a backend developer using Java and Kotlin mostly.

Liz Fong-Jones: Ah, JetBrains, Java, Kotlin. Yep. That sounds like a very compatible combination of things.

Marit van Dijk: Yes.

Liz Fong-Jones: Awesome. Well, today we wanted to maybe spend some time talking about developer experience and making for happy developers. So what's to you kind of the most important thing for a developer to kind of feel happy?

Marit van Dijk: Working for a tool company, I wanna say tools can help. Tools that, you know, don't get in your way. Tools that let you do your job, maybe even offer helpful suggestions. I know that you mentioned in your talk earlier today a bunch of tips on, you know, how to get better as a team in developing software. So maybe you wanna share or reiterate some of those tips.

Liz Fong-Jones: I think there's no shame in saying you work for a developer tooling company. Certainly, one of the reasons that I come to work is I feel like our industry deserves so much better, that so many people wind up wasting a lot of their time on doing break-fix work or having their state of flow interrupted. There's the opportunity to empower a lot more people to get a lot more done. Yeah. So kind of in that vein, you know, when I was talking earlier today, I spoke about platform engineering and kind of the important role that developers can play in not just figuring out best practices for their teams, but also how we think about ensuring that those practices make it out to other teams and that we're not reinventing the wheel a whole bunch of times.

Marit van Dijk: Sharing the knowledge that you have or what worked for you and might work for other teams?

Liz Fong-Jones: The knowledge and also the kind of know-how, the practical kind of implications for what's our process, what's our tooling, as we were just discussing.

Recommended talk: Modern Platform Engineering: 9 Secrets of Generative Teams • Liz Fong-Jones • GOTO 2023

"You Build It, You Run It" Philosophy

Marit van Dijk: So one of the things that you mentioned in your talk was you build it, you run it, which I found very interesting. In the company I worked at before I joined JetBrains, our slogan was, "You build it, you run it, you love it." So, you know, in the DevOps way, you build the code, you deploy it to production, but you also own it in production. If you deploy it and something goes wrong, then it's the team's responsibility primarily to fix that, which is very different from what was happening maybe 10 years ago or more where there was a project organization that would build features and then throw it over the wall to ops who would be like, "We don't wanna deploy anything. It's sort of running right now." So, do you have any thoughts on that? You know, is that progress in the industry? How does that help developers if they're also responsible for production?

Liz Fong-Jones: I loved the addition of "you love it" to the end of it because I think, you know, for too many software developers, they've kind of been forced into on-call just as part of this, you know, you must own your code. And I think one of the things that I said was, you know, it doesn't make sense to make people do something and, like, to give them the wrong tools and to poorly equip them and have them continue to be afraid of production and afraid of what happens if they break things. Like, it's much better to empower people, and then over time they will feel a sense of ownership for it and they'll develop operational excellence. But as one of the people who came from the ops side, you know I don't want to go back to the old days of having my pager be on fire and not having the kind of control to do anything about it, aside from saying, you know, "No, you can't roll out new features." No one likes saying no, that's really what it boils down to. No one likes blocking or gatekeeping things from going out to production.

Marit van Dijk: Okay. But most people also don't like, you know, having their pager wake them up in the middle of the night.

Liz Fong-Jones: The solution is to have your pager go off less often, right? And, to do that, you kind of have to embrace, you know, making sure that your software is well tested, is making progress towards reliability, is controllable, is predictable, and, you know, yes, insert obligatory plug here, right? Like, you have to have adequate observability. If you can't like, figure out what's going wrong, then it's going to take you a lot longer to figure it out, and it's going to be at 3 a.m. as opposed to, kind of, more seamlessly looking after your software, being able to debug it if it breaks.

One of the interesting things though is my boss has caught a lot of flack over the years for saying, you know, "The role of no Friday deploys is bullshit." Am I allowed to say bullshit here? Yes. Okay. Right. Like, so people have been like, "You know, oh, but my organization, you know, doesn't deploy on Fridays because we respect our developers' weekends or whatever," right? Like, the point that Charity is making, and that I agree with, is that it doesn't matter what day of the week it is. You don't push code at times unless you're able to follow it to production, right? Regardless of what day of the week it is, I do not push code at 5 p.m. and then walk away and leave the pager on fire for my colleagues.

Marit van Dijk: Exactly. So, we also had discussions...at my previous job, we had discussions on whether we should or shouldn't deploy on production, and Charity was quoted on that. So the way I interpret it is you have to make sure that you are confident enough that we can deploy this to production. And if you are confident enough that you can deploy this to production and that you know none of the known things will go wrong, and that you're able to deal with what you said, the unknown unknowns, because you have observability in production and you'll know if something went wrong and you have a way of dealing with that, you know, ultimately maybe a revert if you have to. So, it means making sure that you know what you're deploying to production and that you feel in control of that and you feel like you've adequately identified and mitigated risks to be able to confidently deploy any day of the week. Of course, we've had issues where we did a postmortem and one of the questions was, why did you deploy this on a Friday? So, this is still a mindset that is in progress, I think.

Liz Fong-Jones: I think one other piece to this is about keeping context, right? That it's not just about can any one developer, you know, understand what another developer wrote, but is there kind of this continuity of when it rolls out that the person who wrote it has a say on what's going on so that they are the first one who's going to look after things if the change did not have the intended effect.

Marit van Dijk: I've been on teams where the code ownership was very much shared. And even though we didn't do a lot of pair programming or ensemble programming, we did do a lot of collaboration on code, or sometimes...Every person on the team brought their own, I don't know, maybe niche interest or knowledge or background, and we would have conversations like, "Oh, hey, can you help me with this topic that you're good at?" Or, you know, they would come to me with, "Hey, do you wanna spar with me on how we're gonna test this thing and make sure that it's okay?" So very much the whole is better than the sum of the parts. A kind of situation where we would do code reviews and make sure that everybody at least understood in general lines the code that's going to production so that we are also able to support it if it breaks in the future. Of course, unfortunately, sometimes bugs occur when the person who wrote it is on holiday. So we were very mindful of making sure that everybody had sufficient knowledge of the code that we were deploying and was able to support that code building, running, and loving it. Or sometimes when we weren't feeling the love, taking a step back and needed to show some love to this code base so that we could confidently run it again.

Liz Fong-Jones: I love that. That metaphor is gonna stick with me for a while. I think the other useful thing to consider there is when you're doing those code reviews, right, like, to ask those questions, right? Like, did you add sufficient testing or did you add sufficient...? Right? Like, how am I going to observe this? And how is it that someone who has no previous exposure to this could figure it out from first principles by looking at the output of the data?

Marit van Dijk: So some code comments that have happened over time is we had a new developer who joined our team who wasn't a big fan of testing. So the first code review, I was like, "Maybe give your tests names that cover what they're supposed to do just in case they break in the future." You don't wanna on top of, "Hey, something broke," figure also out, or also have to figure out what it's supposed to be doing. If you can encapsulate that in the name, like, you know, in this case, this should happen you can use the given-when-then structure, or actor-range-assert, which is the same thing. But describe the intended behavior in the name of a test that's gonna help you down the road if it ever breaks in the future. Things like that. I would often just put code comments like, can you please explain what's going on here so that I know if anything breaks here what it's even about? And some of that would go, like, I don't know, offline, we would just have a conversation about what's going on rather than go through code comments. That's what happens if you have a good functioning team where you can just talk to each other.

Liz Fong-Jones: Although, I imagine that that's kind of changed for a lot of companies over the past several years, given that you can't necessarily sit over each other's shoulders.

Marit van Dijk: Not anymore. So when the pandemic hit, I was on a team, on this specific team that I'm thinking of now, where we had the relationship where we would just ping someone and be like, "Hey, do you wanna spar with me on this topic?" And, because we already had that dynamic, we kind of continued in a similar vein, of course, it wasn't sitting at someone's physical desk anymore, it was just going over Zoom, or whatever G-meets, or whatever tool you talked...

Liz Fong-Jones: You had kind of this reservoir of psychological trust and safety built up with each other to be able to do that.

Marit van Dijk: We had the whole team feeling and knowing each other well and, you know, kind of making jokes and...So, when you already have that basis, it becomes much easier to work remotely then. And of course, if you're onboarding two teams that are new with people that you don't know, getting there online is more difficult. But I've also worked on open source projects where that also happened where I didn't meet the people that I was working with until maybe a year later when there was an event where we finally met in person, which felt very '90s, "Oh, meeting my online friends in real life."

Liz Fong-Jones: We did one of those about eight or nine months ago at Honeycomb, and it was great to meet colleagues that I hadn't had to meet in like two years of working at the company together. Yeah. I think one of the other terms that I think I picked out from what you said earlier that I might have to adopt is the term ensemble programming rather than mob programming. Mob programming always just struck me as a little bit weird.

Marit van Dijk:  It's not my term, I'm not sure who came up with it, but yes, it's because the mob sort of triggers the idea of people with pitchforks and an angry mob, which isn't at all what Woody Zul intended, but someone came up with ensemble, which sounds so much nicer.

Recommended talk: Mob Programming and the Power of Flow • Woody Zuill • GOTO 2019

Liz Fong-Jones: And collaborative, right? It is like an ensemble.

Marit van Dijk: Yes. Which I don't make music with anyone, but I kind of...That is a community effort where everybody has to do their part to make it sound good, which I think is a good concept to keep in mind when building software, which, you know, building software is a team sport, I think.

Liz Fong-Jones: And it has to be, otherwise, your code winds up being not looked after if you go on vacation.

Marit van Dijk: Or people,  save tasks for you when you go on your honeymoon and be like, "Yeah, we saved this job for you, even though we said we would all do it. But, you know..."

Liz Fong-Jones: Knowledge sharing also seems to be this thing that, you know, different teams and different practices on. Some teams wind up writing kind of exhaustive documents down into a wiki. Other teams, you know, rely on, kind of, this shared state that exists in people's heads. But I think, either way, right, like, the important bit is the communication of that state.

Marit van Dijk: Ye I like to write stuff down because, you know, I'm getting older, and remembering things is getting harder. So, you know, for example, if I join a new team, I see if their code base has READMEs that explain, you know, here's how to run this project, here's how to build it, how to run it locally, how to run the tests. And either there'll be like a standard way. So, for Java projects, if you're using Maven, there'll be Maven Clean Verify, if that works, great, I don't need to write it down. If there's additional stuff that I need to do, I write it down in the README because then next time I don't have to go looking for it, I know where to find that information. And we had similar things for...So we had an engineer of the day, so, one person on the team, which was rotating every day, who was responsible for answering questions from other teams about our applications or answering questions from business people about our applications, but also, like, doing daily checks of the health of the system.

We also wrote that down on a wiki page or a Confluence page just, you know, "Look here if this goes wrong." You know, there were standard things like maybe...you know, sometimes it goes wrong, we haven't quite had the time to figure out how to properly fix it. Or you know, sometimes this just happens, you know, just press this button, give it a little push, and now it'll work again. Just writing that down helps because then you don't have to remember in case it, you know, happens again too far in the future where you don't remember the details. I found that very helpful.

Efficient Team Communication and Streamlining Workflows

Liz Fong-Jones: I love that your team did that because so often I've come on to teams where everything is getting...we call it kind of the "spraying the fire hose," right? Like, every single developer gets wet, right? Like, you wind up with, you know, all these interrupts that come in, which mean that people can't do them...you know, focus on their day job. Whereas if you, you know, have the on-duty engineer, it kind of streamlines that. And, you know, two teams can have the same amount of toil, you know, whether it be [inaudible 00:15:25] work, whether it be interrupts from other teams, but they can be in vastly different states of productivity depending upon how that incoming workload is allocated.

Marit van Dijk: So instead of every question interrupting all of the developers, it's one person's responsibility to monitor the channel for incoming questions. Of course, if that person happens to be in a meeting and something urgent comes in, like production is literally on fire, someone else can jump on it, but we have one person who's primary responsibility for the day or week, other teams would do it by week, it is to monitor that channel and limiting interruptions, which is, you know, also gets in the way of developer flow, as you mentioned earlier.

Liz Fong-Jones: I'm excited about your talk on managing software dependencies. This is something that, at least in my book, is a category of toil, right? Like, it's a category of a thing that doesn't advance the system.

Recommended talk: Keep Your Dependencies in Check • Marit van Dijk • GOTO 2023

Marit van Dijk: But it's necessary to not degrade the system as well. So it's one of those things where we know we have to do it. We know we'd rather be doing something else. So if we can automate the pain away, it makes our lives easier. I'll be talking about different things that you can do to make that easier for developers and different tools that you can use. It'll be geared more towards the Java ecosystem because that's where I live. However, some of the categories of tools are also available for other languages and ecosystems. So hopefully, even if you're not a Java or Kotlin developer, you'll get something out of it.

Liz Fong-Jones: There's this kind of interesting paradox of automation situation going on, I think, in that topic in that, you know, you see these, like, scary warnings being like, you know, "Hey, like, if you automatically apply these updates, you might pull in malware. Oh my goodness."

Marit van Dijk: Oh, well, that's the other side. So, do I always update to, like, the latest version of everything? No, because then you run the risk of, you know, automatically pulling in malicious software, but also you shouldn't be on, like, the oldest, oldest, oldest version, and you should at least update if there is a known vulnerability in your particular version of a dependency that you have, and there is a new version available where they fix that security vulnerability, then absolutely upgrade. Because, you know, once they are known, then they become known attack factors, and if you're working for any company that's even a remotely interesting target...So I was working in an online retail platform in the Netherlands, which makes you...Like, a well-known...In the Netherlands, a well-known platform. 

Liz Fong-Jones: No one wants to get ransomed and have their website held to hostage.

Marit van Dijk: Exactly. So, not how you wanna end up in the news, which is...you know. Which is a question I learned from a tester friend. Like, you know, if you're thinking about what could go wrong when building a new feature, one of the questions to ask yourself is like, what is the worst possible headline that could happen if we build this feature or don't build this feature? And I think that's something good to consider.

Liz Fong-Jones: In the observability space, we call that a premortem, where you're kind of enumerating your known and knowns, right? Like, you're enumerating, "Here are the risks that I think could go wrong. Now how am I going to design and defend against that?"

Marit van Dijk: I haven't heard that term before, so I'm gonna take that and...

Liz Fong-Jones: Awesome. This is part of the fun of exchanging these ideas from radically different spheres. So, going back to the software update discussion, I think one of the things that I was realizing as I was, you know, digesting this question of, you know, how much time and energy do I spend on manually validating updates versus automatically applying them? What are the acceptable risks? And then I realized like, if I'm manually clicking through and approving these things, am I actually doing any meaningful oversight or am I just saying yes, right?

Marit van Dijk: Do you know what this particular update entails to properly evaluate it? But, I mean, the same can go for code reviews as well. If you are not the one actively developing this feature and thinking about, like, the best way to implement it, and should I use this pattern or that pattern, or this implementation or that implementation, how much of that are you gonna get from reading the code during a code review? And I guess the same for doing the updates. Like, if you don't know what the updates entail, how are you gonna properly evaluate whether we can merge this code or perform this update? So then again, maybe pairing and ensembling can be a way to both share the knowledge of how we're implementing this feature, but also ask those questions upfront rather than, "Hey, I've written this code, I've done my very best..." Because as you said in your talk as well, nobody wakes up and thinks, "Okay, what crappy code can I write today?" Honestly, ever. So, you know, if you do that upfront or even just think about like, how are we gonna implement this, and get all of the stupid questions out of the way, then the solution they end up with is gonna be better as well.

Liz Fong-Jones: I think there's also kind of this element of if two people have worked together very closely for a while, right, like, they might wind up rubber-stamping each other's changes because they both kind of have the shared model. But you can use that sometimes as an opportunity to bring in someone newer and have them do a code review of something that...you know, the person who wrote the code knows what they're doing, right, this is fairly routine, but it's an opportunity to bring someone new up to speed as to, like, you know, how does this work? What is this doing? Even if it's...You know, the example that I like to use here is infrastructure as code changes, right? Like, if I'm changing from, you know, trying to saturate our CPs at 80% to 85%, right, like, that doesn't necessarily require a lot of...You know, someone could easily, like, look at a graph and agree that the same graph that I'm looking at says what it says, but it's an opportunity to bring in someone new and teach them and show them how it works.

Marit van Dijk: Yeah. So the whole infrastructure as code is where I'm maybe also lacking in knowledge. So those would be very hard for me to evaluate because sometimes, you know, you have this big YAML file, you have all of these properties. I as a developer don't necessarily understand the impact of, you know, particular settings or tweaking them or whatnot. And that's where maybe having dedicated teams or dedicated people who have that knowledge who can help the developers understand that or what works for their particular situation. 

Recommended talk: Observability Engineering • Charity Majors, Liz Fong-Jones & George Miranda • GOTO 2022

SRE Models and Collaboration between Teams

Marit van Dijk: You mentioned some of that regarding SREs. So, do you have some examples of how SRE teams can help development teams?

Liz Fong-Jones: This is an evolving topic because there are many different models for practicing SRE. The site reliability workbook by Google, the kind of second Google SRE book goes into some detail as to models that people have tried. So the original Google model was an SRE team, essentially an operations team that uses software and is comprised of at least half of software developers, but it still very much is the team that owns production. And there's a minimal degree of kind of shared responsibility with the software development team. So, it's kind of how that started. But there are other models like embedded SRE where you say, you know what, you have a collection of SREs, you just spread the SREs out. You put, like, one or two on each software development team. And then there's kind of the consultative SRE model where every team is responsible for its production, but you have SREs who are available to answer questions, and that's kind of the genesis of where platform teams come from.

I think it's kind of what can you learn from consulting to kind of give people better practices so that you have to answer fewer questions about what the right way to do things is. But yeah, I think the, you know, SREs running the system soup to nuts is not a sustainable model, and it kind of recreates a lot of those pathologies of, like, you know, knowledge being divorced, of teams not having sufficient kinda visibility into the blast radius of their changes. The reason why I don't particularly love SRE is that you can very easily wind up having those one or two people be kind of the focal point for all of the operations-focused work of that team. It doesn't percolate as well to the rest of the team, and then they're...

Marit van Dijk: Because you have one dedicated person who is doing it, so they might as well be doing it somewhere else, maybe because they're not necessarily sharing their knowledge with the team, right?

Liz Fong-Jones: Or at least it's spreading out more slowly. So, you know, those teams do work in that, you know, the SRE is not the only person on call at all time, so they certainly are the person, you know, writing most of the automation. But yeah, I think it tends to work a little bit better when it's just, you know, better spread out. The other thing about that also is the career development. So, this is kind of the thing where...Facebook was the company to kind of practice embedded SRE first, and they call it production engineering for various historical reasons. However one of the challenges that they had was that they wanted to have the production engineers all report to one set of managers. So you would be working for one functional lead, but you would be reporting to someone in the reliability engineer organization, right? Like, so that kind of creates all of these misaligned and weird incentives, where your manager is not the person that you're directly working with every day.

Marit van Dijk: I've been in those situations and it is no fun. We had several different types of platform teams where we had one team that was responsible for, kind of, our internal framework, which was spring or spring boot wrapped with other dependencies that you'd need to build applications in our landscape who were...you know, they spent their time doing that. They wrote great documentation also on how to migrate to newer versions what to look for and what the impact was. And they were very available for questions. And then we had maybe platform teams that helped with moving to the cloud and running applications on the cloud. And we had, like, a custom in-between kind of system. You asked about who's using the console to configure their cloud stuff or who's keeping it in their code base.

So, we had it in our code base, but there was a way to also run the configuration elsewhere. And that was the part where I was always confused. I'm not too sure about the changes that I'm making and what their impact is. And then if the people who do know have limited availability and don't have the time to help all of the developers properly understand that, that's where you get a bit of friction. Do you have tips maybe for how to deal with that from a developer side?

Liz Fong-Jones: I think that it's really important for people to make sure that there is a sensible default way of doing things. And one of the interesting points of feedback that comes back is when those defaults don't work for people, right, like, then you find people who want to, you know, break the glass and press the button. And that's an opportunity to say, you know, either we need to change the tooling to accommodate this use case or, you know, this is sufficiently off the beaten path that, you know, that team should go off and do things their way.

It's when you have that situation that you're describing you need to go off the beaten path and you're not sure, like, what to do, right, like, that's a sign that the platform is probably not providing sufficient guidance or is not capturing a key use case where people who have an idea of something that they have a good reason to do and don't have kind of a imagine they want to do it. I think that's a sign of, okay, we need to design for that use case. You know, assuming it comes up, like, a couple of times. If it's just a one-off, then, you know, that's just a one-off. But more of the time I tend to see...Oh, here's an example for you.

We migrated most of our applications to run on top of Kubernetes. We just finished that migration a couple of months ago, but one of the applications we knew could not run in the traditional Kubernetes deployment model because it was stateful, it had pairs of workers, one of which had to be up at any time. So, that was a situation of, "Okay, the standard patterns of, you know, just create a Helm chart, you know, just use a deployment," that was just not going to work. So that team knew that they had to build out their custom operator, that was what was going to be needed to manage the individual pods.

Marit van Dijk: I'm also...I'm sitting here giggling because you hit the nail on the head. I think I ended up in several situations where, you know, the standard way of doing things that worked for 80%, 90% of the cases and 80%, 90% of the teams and code bases, which is definitely what they were aiming for, like, automate the basic standard use cases. And then I ended up in teams where we had a slightly different use case that wasn't part of that 80%, 90%. And then you end up having to figure out how to deal with that and with, you know, the options provided by the cloud platform that we were using, and also the ones that were allowed by the IT department or architects, or whatnot. But yeah, then definitely you need more help from the SRE teams or platform teams on how to cover that particular use case. And it could be like, we are not gonna include it in our basic or our standard configuration, and we need to go look for something else, so.

Liz Fong-Jones: From the perspective of the platform or SRE owner, right, like, that is a useful signal as to which teams I should spend the most time with, right? Like, it's those that are kind of pushing the limits of what's possible on the platform.

Marit van Dijk: I understand why, you know, okay, so we have maybe 100 teams, this solution works for 90% of them, and then we have, like, these 3 teams and they need something else, do you incorporate that into your platform and make it X amount times to more complex or do you figure out something for those teams who have a different use case?

Building Internal Platforms and Product Management Strategies

Liz Fong-Jones: This is such a wonderful question to ask, and it's one that I wish I had gotten to in my talk because, like, it turns out running on a platform and writing the software for our platform, congratulations, you are creating a product for your internal developers. You need product managers, and developer advocates, right? Like, it turns out you cannot just set loose about a bunch of backend engineers and, you know, tell them, "You know, you're building a platform, have fun," and then they'll build something that's completely not end-usable. No one's going to use it, right? Like, you have to do those...You have to have user research to kind of understand what are the tradeoffs. What are we building? Who are we building it for? What's gonna make sense? Because if you don't have a coherent product management strategy and kind of the input gathering and the evangelism, like, your only lever there then is, kind of, fear of, you know, you're going up against the architects rather than building the best thing for your developers.

Marit van Dijk: To do that, you have to understand their needs and what their use cases are, etc. And it means, again, talking to people because software development is a team sport. So, yeah.

Liz Fong-Jones: It's really interesting how software developers have this reputation, you know, stereotypically as people who don't like to talk to other people and want to sit in their cubicle and write code all day. And like, I think that is the antithesis of, like, what makes for someone who's a great software developer. A great software developer is someone who communicates well about what they're doing.

Marit van Dijk: Yes. But it's almost as if, you know, we cultivated this stereotype and then that's what we ended up with. So, maybe we need to cultivate a new stereotype of developers who like talking to other developers and occasionally business people and other people and can explain what we're doing.

Liz Fong-Jones: So yes, kind of less of an arcane priesthood as it were of people who understand the code magic and those who don't, right? Like, no, like, we're here to share it.

Marit van Dijk: I would much rather work with people who are also able to communicate about what they're doing than...You know, sometimes you need someone to fix the really hard problems, but most of what we're doing isn't rocket science. We're building, well, in my case, web services. So, you know, it's been done before. It helps if you can explain what you're doing, talk to the other teams who are using your application, talk to the business about what they need, and make sure that you're building the right thing.

Liz Fong-Jones: Excellent. Do you have any parting thoughts before we wrap up?

Marit van Dijk: It was a great conversation. Thanks so much.

Liz Fong-Jones: Yeah. Thank you for joining us and please watch the talks from GOTO Copenhagen