Rust in Action

Rust in Action introduces the Rust programming language by exploring numerous systems programming concepts and techniques. You'll be learning Rust by delving into how computers work under the hood. You'll find yourself playing with persistent storage, memory, networking and even tinkering with CPU instructions. The book takes you through using Rust to extend other applications and teaches you tricks to write blindingly fast code. You'll also discover parallel and concurrent programming. Filled to the brim with real-life use cases and scenarios, you'll go beyond the Rust syntax and see what Rust has to offer in real-world use cases.

© manning.com

Richard Feldman, author of Elm in Action, goes over the benefits of this multi-paradigm, high-level, general-purpose language together with Tim McNamara, author of Rust in Action.

Intro

Richard Feldman: All right. I am Richard Feldman. Welcome to this Book Club interview. So, I am the author of "Elm in Action," which is not what we're discussing here. We did a separate book club on that, but, from the same publisher, Manning Publications. Tim McNamara is the author of "Rust in Action." You'll meet him in a second. And this is a different book and a much different audience. So, we're gonna talk all about the book, what motivated him to write it, and all sorts of things about the process. I'm excited to talk about it because I also am a big fan of Rust. I teach a course for Frontend Masters called "Intro to Rust." I've written a compiler in Rust. Big fan of the language, and am excited to talk about it today. So, Tim, do you wanna introduce yourself?

Tim McNamara: Thanks, Richard. So, my name is Tim. I am a developer based in New Zealand, but, before Rust, I was a Python developer, primarily in, like, the data science space. And a few years ago I was sort of, like, scratching around for trying to figure out, like, how computers work. And then I kind of accidentally got asked to write a book. That's kind of the pitch for "Rust in Action." If you are from sort of a dynamic programming background, or even a, say, a Java, but you've never really encountered these terms, "systems programming," or any of the concepts around that, that's really what "Rust in Action" is trying to provide a little bit of a glimpse in, to not so much teaching you systems programming. It's primarily there to teach you Rust, but it uses concepts drawn from that, from, sort of, that subfield, to really kind of enable you to feel welcome inside the Rust community.

Why should you learn Rust?

Richard Feldman: Nice. Well, so, let's talk about that a little bit more. So, you said your background is kind of a Python developer. I came to Rust from a similar perspective of not really having done any kind of systems programming. Like, I knew C and C++ from a long time ago, and kind of had a bad experience with them, to be honest. Didn't have any interest in getting back into that world, but there was this kind of curiosity about..for my whole career up to that point, I'd always been on top of some sort of virtual machine or interpreter, something like that. I never really knew what was going on under the hood, and that was always something intriguing to me. It sounds like it was for you too. So, how did you go about, like, learning Rust yourself? You know, this book didn't exist, right? There was no...and we've got it right here, "Rust in Action," right?

Tim McNamara: Nice. I've got my copies down the back. 

Richard Feldman: But, you know, it didn't exist because you hadn't written it yet. But there are all these resources out there. I mean, there's, like, the official, like, free Rust book that you can read. Other people have written Rust books. So, I'm kind of curious about, you know, what was it about, that sort of motivated you to write the book in this way? Like, you know, what did you wanna sort of accomplish to present people with something different than what was already out there?

Tim McNamara: Right. So, the very last, the very end of your question kind of hinted at the answer. So, I, intentionally, I'm, like, one of the last people to learn Rust without the book, the official, "The Programming Language," the free one online. I intentionally didn't refer to that at all because I was overly concerned about, like, copyright protection and things like that...and I didn't want to plagiarize on the material.

Richard Feldman: Oh, interesting.

Tim McNamara: And the same with the O'Reilly book, "Programming Rust." So, it turns out there's, like, accidental plagiarism. It turns out that the "Rust in Action," as well as "Programming Rust," the O'Reilly book, have a renderer for the Mandelbrot set. My one happens to be inside the terminal. And I was like, after the publication, I, after my book was published, I was reading through the others, and I was like, "Oh, wait a minute. That was maybe a mistake."

But to answer your question, I wanted to produce a book that was different than every other book I'd ever seen. From a marketing perspective, I didn't want to write, like, a me-too book. I couldn't compete with the official reference, or, like, the official guide. So I can't write a book that just takes you, let's say, start with integers, and then we get to floats and then we get to strings, and then we, like, have structs and we combine them, and then we might talk about traits. I can't write that book again. It's impossible to compete with free, essentially.

Richard Feldman: Sure. Unless you're doing something different.

Tim McNamara: Right, right. Exactly. And so, "Programming Rust" from O'Reilly was the other one that was coming out, and Manning was concerned. We really gotta get on in this Rust space. And that was focused on C++ developers who were kind of, like, Rust-curious, let's say. Like, "Ah. Shared pointer." So, if you're not familiar with Rust, Rust is protective of data access. C++, in a very broad way, is less protective of you, the programmer, for when you want to access variables, particularly across threads, and you end up they clobber each other. It turns out you don't even need multi-threading to cause these kinds of problems because we have multiple CPUs that might also want to access the same variables at the same or roughly the same time.

Anyway, ignoring all that, Rust has this kind of reality and strongly-enforced guardrails that you need to opt out of intentionally. Whereas in C++, you need to opt into the guardrails, essentially. And some types exist in C++ that are ubiquitous in Rust. Now, the reason why you're a C++ programmer, and you might look to, say, a Rust is that C++ is notoriously difficult to get right. And maybe this is what you're referring to when you were saying... 

Richard Feldman: I have a concrete example of that is that I remember when I was doing, many years ago, like, a little bit of C and C++, is I would all the time get a program crash that would just say "segmentation fault." Meaning that I did some stuff with some memory somewhere that I wasn't allowed to, or I made a mistake somewhere. And then figuring out what the actual underlying problem was behind the segmentation fault was often very time-consuming and painful. But when I moved to higher-level languages that have automatic memory management, if I ever saw a segmentation fault, it was never like, "Oh, I made a mistake." It's like somebody who implemented something that I'm building on made a mistake. Somebody else made a mistake, I don't know who, but...and it was super rare.

That's also been my experience with Rust, is that I mean, there is the whole, like, the unsafe section where you can cause segmentation faults, but if you're staying away from that, and at first, I just avoided unsafe keyword altogether, I found that I can just feel that I will be able to write my code, and maybe the compiler will complain at me and say, "you can't do what you just tried to do," but as long as I can get everything to compile, it's not going seg fault. It's not going to catastrophically blow up. And if it does, it's because somebody else made a mistake, rather than me making a mistake.

Tim McNamara: And the same with me. I was writing Python, and I had these jobs that were running for, like, three or four hours, and then they would, like, crash with a name error or, like, a tight error. I'm like…

Recommended talk: Ready for Rust • Erik Doernenburg • GOTO 2021

Richard Feldman: Oh, no. 

Tim McNamara: ...how did this happen? And it just blows up. And the exception, and then, is there. What I wanted from a language was something like Elm, that was, like, developer happiness is paramount. And, error messages are part of the language, and the error, like, debugging experience is part of the design of the language. And what I kind of got from Rust was much closer to an Elm experience, versus, like, a C or a C++ experience, where things would just blow up, even without it, because there are...and in C, where I tried to play around...oh, actually I tried to write Python extensions in this C macro language called Cython, which essentially is C with Python syntax.

Richard Feldman: I've heard of that. Yeah. Right.

Tim McNamara: But again, it blows up, spectacularly, and you, like, literally don't know what to do. It's like, okay, segmentation fault. What do I do now? Like, is that the end? And so...

Richard Feldman: What you're supposed to know is it's like, well, now you need to attach a debugger and go back through and step back into it, and then, like, you know...but it's this whole complicated thing. And then even once you know how to do it, it's still pretty difficult to track down, like, what was the actual thing that caused the memory to get in this bad state? It's really a whole different level of pain.

What does segmentation fault really mean?

Tim McNamara: Absolutely. So, one of the things that I was really curious about is, like, what does this term segmentation fault even mean? And that's one of the things that I explained in "Rust in Action." I think it's Chapter 5, in data, we talk about the fact that memory needs to be translated in your program. We talk about a memory address as an integer. So, some integer between 0 and 2 to 64 minus 1, like this is what we call our address space. But somehow that needs to be translated into something that's physical electrons on a chip somewhere. Now, that translation process is complex and involves software, which is kind of like a bit of a weird mind bend. But it turns out that the operating system segments memory, or, like, into 4-kilobyte chunks, and the segmentation fault is a problem caused by when you're attempting to access a memory address that sort of doesn't actually...the operating system doesn't know how to translate into the physical chip. Like, it's something on-RAM. It can be more complex than that, but...

Richard Feldman: Right. Or you're not allowed to...

Tim McNamara: Right, right. The operating system will also enforce that there are some parts of the memory address space that are inaccessible to you. So, it turns out that the operating system can mark some sections of memory as executable, some as data, and some as code. All of this background knowledge that you're supposed to somehow acquire is kind of what I wanted to inject into the book. 

Part of the problem that I faced as a writer, or as an author, was that I was always late. Because I was essentially trying to write two books in one. And by the way, the other thing that I tried to do differently than most other books is, like, pack in big, like, relatively chunky examples. But it turns out that that's hard because you kind of need to debug them, and you can only really introduce...

Richard Feldman: Right. I remember this.

Tim McNamara: ...one new language feature at a time. Because if you try to peck in two new parts of the language into a new example, well, suddenly, you've got five extra pages of writing you need to do to explain to people. And people, like, they have this kind of mental stack, and you're only trying to add one thing at a time, and kind of give some... Anyway, that's why I thought providing this sort of extra context around Rust would be helpful.

Intro to the Rust world

Richard Feldman: I think that's cool. You have this cool example here. I mean, there are awesome examples throughout the book, but, like, here you have, the title of this section "All of Networking in Seven Paragraphs." And you've got just a diagram. It's like, look, here's how your network stack works. So, it's not just a Rust book. It's a combination of, while you're learning Rust, you're also learning about low-level parts of the computer, which I love, because that, I think, for a lot of people who are in the position that you and I were in, coming from, like, high-level languages, and being like, "I wanna learn about this low-level stuff, but I don't wanna deal with segmentation faults, as errors to my program," or maybe I just, like, wanna understand them better, you know, while also not having to debug them. That's great. That certainly was not a resource that existed when I was getting into Rust.

Tim McNamara: Yeah. And also, because... Well, no. I wanna also just kind to segue slightly into talking about the cultural differences between programming language communities.

Richard Feldman: Sure.

Tim McNamara: Because I, as a Python developer, was always told, "Native extensions are difficult. This is for experts."

Richard Feldman: Sure.

Tim McNamara: I had some encounters with, actually, a couple of other language communities, where I would say things, and I would kind of immediately get pounced upon, because. In retrospect the whole... Because I didn't have the mental model, I didn't use the language in the right way, and so therefore people didn't address the question, they addressed the inaccuracies because I used the words wrong. Whereas I feel like, and I'm very clouted now. Like, I'm very deeply embedded in the Rust space. And I'm very hopeful that new entrants to the community still feel this way.

But my experience was that the Rust community was particularly welcoming and particularly interested in preventing people from feeling excluded. That's kind of baked in with the slogan. Like, empowering everyone is how it starts. And you, as someone watching this podcast, as someone curious but a little bit fearful, potentially, or maybe fearful are too strong, just a little bit hesitant should feel as though you're welcome. And part of the book is to say, or at least part of my rationale for creating something new and interesting and different was essentially to say that...is to give people the confidence to learn more. Like, I can't teach you all of Rust. There are some parts of the language that I exclude from the book. People have asked...

Richard Feldman: But it's a beginner book. I mean, you can't simultaneously give a good introduction and also cover absolutely everything there is, right?

Tim McNamara: Right. So, there are typically two complaints. Like, Of about 90%, 95% of people are happy, but there are complaints. And the two themes primarily are A, why isn't your book like every other book? Like, "this is not something that I'm familiar with. I'm used to being able to, like, pick things apart." It's like, "well, I wanted to look up what an enum is in Rust." Or, like, "So, where is that? Like, I can't find that on the contents page." I sort of feel like that's slightly misguided. I wanna say that "Well, you kind of didn't read the first introductory thing," where it was like, who is this book for? And the other complaint, I guess, would be that it's too introductory. Like, it's like, "Ah, this isn't a guide to writing idiomatic, high-performance, zero-copy Rust." And I also feel like that's slightly misguided. I'm pretty happy with the reception to the text in general.

Richard Feldman: I mean, if one of the biggest complaints is "I wanted you to do even more,", that kind of tells you that, like, well, you did a good job with what was there. They wanted more of it, right? It almost sounds like what they're asking for is, like, "Hey, can you do another book," you know, advanced for us? How about it, Tim? Just kidding.

Tim McNamara: Well, like, I should say, the decision between do you do a second edition... So, there are things that have changed very slightly, but primarily in the libraries. The ecosystem has matured. Essentially, I can sand off some edges and create a second edition. Or do you essentially do, like, a second volume? Or, like, another book, like a follow-on. I told Manning I don't... I've got three little girls. I don't think I can get you a book on time, let's say in the space of a year, 18 months. That's a new book. But I can do a different edition. And so, very, very, like, within a couple of weeks, you should see an announcement of "Rust in Action" second edition appear...

Richard Feldman: Nice.

Recommended talk: The Ideal Programming Language • Richard Feldman & Erik Doernenburg • GOTO 2021

Tim McNamara: ...on Manning's website. But I'm also interested in potentially fleshing out some of these next ideas via say, my YouTube channel, or relatively thorough blog posts that I'm hoping to somehow combine at some stage, or kind of distill into a book over time.

Richard Feldman: That makes a lot of sense. I think I got a little bit lucky with the workshop that I did on the intro to Rust because it's very time-limited. It's like you have an eight-hour workshop and that's it. There's no, "Hey, I wish you could have had it be longer," because it's like, well, now it's a multi-day workshop. You know this is one day. I had way less scope in that than you have here in "Rust in Action." First of all, I didn't have practical examples so much. It was more just, like, okay, we gotta get through. And, I mean, I'm kind of curious to talk to you a little bit about your sort of thoughts on, you know, how to teach Rust effectively to beginners, because what I found challenging...

So, this is Frontend Masters. It's a primarily JavaScript audience, and a lot of people are interested in Rust because a lot of JavaScript tooling is moving to Rust, and there's, you know, applications of web assembly and things like that. But what I found, after doing some practice runs of that, was that it was just hard enough to get... I basically pared it down all the way to, like, okay, I have eight hours to get to lifetime annotations. If you can understand lifetime annotations at the end of eight hours, starting from no Rust knowledge, then I succeeded and I was like, traits, out the window. No time for 'em, you know. Everything else is just, all the way building up to that, and now, if you're wanting to understand lifetime annotations, you can at least read the documentation, and now you can kind of go off on your own. But you've managed to pack a lot more scope into here. 

The hardest things to teach in Rust

Richard Feldman: But I'm curious what you thought was...you know, for me, that was the hardest thing to build up to, because it requires all this background understanding. What did you think were the hardest things to teach in Rust?

Tim McNamara: Strings.

Richard Feldman: Yes.

Tim McNamara: Right, right. So, I wanna provide a little... So, I fought really hard with this idea. So, I was convinced that I don't wanna teach strings after numbers. Normally, you see a programming language tutorial, and they introduce numbers first, because they're easy, addition, and so forth, and we have a little toy calculator. And then the next thing we do is text. It turns out that within Rust, knowing that A, you need a distinction between a string literal as a different type than a string, which is kind of super weird. And, by the way...

Richard Feldman: If you had a Rust REPL and you put a string literal into it, you type, "Hello, world" in quotes, it's not gonna come back and say, oh, this is a string. Nope.

Tim McNamara: That's right. Yeah. And then there are sounds...this is gonna sound ridiculous. There are sound technical reasons why this is the case. Essentially...

Richard Feldman: Oh, yes. Absolutely.

Tim McNamara: ...it bubbles up a lot of the implementation details about how programming languages and how programs are built. So, into the programming language. Rust always wants to offer you the choice to kind of boil away the abstraction and go right into the depth of what's happening. So, just to kind of flesh that out. A string, conceptually, is a list of characters. But it turns out our conception, as someone, we're probably from a Western, Latin script, or at least we're used to having this conception of written language as, like, one character after another. It turns out that there are many scripts in which this doesn't work.

Therefore we need character encoding, which is a way of translating numbers to sort of symbols, which we'll call glyphs. And that particular system also needs to be backward compatible with the way that people have been using computers for decades. So we have this way of encoding what's known as Unicode, where we can kind of encapsulate all programming languages, ideally, inside the computer. And we also need to kind of... So, we got this thing called UTF-8, which can kind of expand out. But again, it's one character after another. But where does that live? If you are a programming language, you need to decide where to put this in your memory. And it turns out that a function is a special kind of section of data. Your programming... I'm trying to get the semantics right.

The compiler, let's say, is trying to decide, I need to put, for technical reasons that relate to the CPU... Your CPU, when it calls a function, what's happening is that it's jumping to a memory address, and then the return is kind of popping back off and going to the last place. For that to happen, the function needs to be exactly the right size, and it can't expand or contract. That conflicts with our notion of text. Now, text, we can append to the string, and we can concatenate with other strings. We're used to kind of manipulating them. So, that, we need to put the string in a different part of memory. And what we keep inside the function is a reference to that memory. This is a kind of malleable string. So, we've got a reference in there. But...

Richard Feldman: But then you get into the performance issues, right?

Tim McNamara: Right, right, right, right, right.

Richard Feldman: And you can put it on the heap, but...

Tim McNamara: It turns out that there's a third spot. So, we think, as programmers, there's the heap and the stack, which is explained, again, in "Rust in Action," or, like, you can look up any introductory systems book on what these terms mean. It turns out there's, like, a third ….

Richard Feldman: There's even more than that. There are more than three. 

Tim McNamara: So, shared libraries, for example, live in their part of the address space, but we also...

Richard Feldman: Global variables, yes.

Tim McNamara: Yes. And we have a region of memory called static, for kind of weird, esoteric reasons...

Richard Feldman: That's what Rust calls it.

Tim McNamara: It's, like, the read-only section of memory, right?

Richard Feldman: Right. And it's part of the compiled executable. When it's on disk, right? Like, For that string, if you say, "Hello, world," and you put that into your program, the string "Hello, world," the bytes for that, are actually in your compiled Rust program, on your disk, and then when your operating system loads the contents of that file into memory, it's like, you know, it's right there in memory now. It's, it took it from the disk, moved it into RAM, and that's where it is now. Not the sack, not the heap. It's just, it was on the disk, and now it's in memory. That's it.

Recommended talk: Next-Generation Programming: Rust & Elm • Richard Feldman • GOTO 2020

Tim McNamara: So, to pull this back to a string literal, your string literal lives in a different place entirely than most other strings, or at least what we think of as, like, the string type. It turns out that they can sort of kind of be unified through a reference to some array. Let's use that term. Some list of characters, of bytes, that happen to be encoded as UTF-8, which is this encoding for translating numbers to what we would say is a, like, a writing script. Now, all of these kinds of... So, we've got, like, four or five different forms of translation happening. And this sucks up people's working memory when they're learning something new.

Richard Feldman: Well, and I think especially because, like, "Hello, world" is such a canonical, you know, way to start a program. Because if you're trying to explain the types of things, it's not just that the type is not a string. The type is &'static str, all lowercase. And then, the whole library uses things called String, S-T-R-I-N-G, capital S, no static, no ampersand, and no apostrophe. It's like...

Tim McNamara: What's going on there? This is your point of, like, well, maybe lifetime annotations are enough. Then the question's, like, what on Earth is a lifetime annotation? It turns out, like, I know kind of maybe conceptually, that variables have a type, but a lifetime? Like, that's kinda weird. And, like, the syntax is ugly and weird. By the way, this term static is kind of this weird, magical term for, by the way, lives forever, or at least from this point on until the rest of the... The Rust compiler has, like, a sidecar part of it, or essentially, like, a really strong linter, called the borrow checker.

Richard Feldman: That's a good way to think of it. 

Tim McNamara: And the borrow checker's job is to analyze your program and say, for all invocations, or every single time, for all possible invocations of this program, is every data access valid? And can it ever possibly be invalidated? And it turns out that if I, in my function... Well, let's say I create a variable ref, and I'm creating a reference to some other data. Now, the borrow checker, and by the, for kind of weird, esoteric reasons as well, which happen to be slightly historical, we call this a "borrow," I have a reference to some other data, that lives outside of my function. The compiler guarantees that the data that I'm referring to will sort of never disappear. I will never get a segmentation fault in Rust, because I can't accidentally, do what's known as "free" the memory. So I can't return it to the operating system and tell the operation system, "I don't need this anymore, and so invalidate my reference." This is impossible to do in Rust. And all of this kind of boils down to a lot of complexity that is sort of front-loaded, as a learner. My job, by the way... So, this book, "Rust In Action," has been really good for my career. I've scored a couple of really neat jobs out of it. And...

Richard Feldman: Nice.

Tim McNamara: ...not exclusively, but I am a relatively competent developer as well. I work at AWS. And AWS is kind of doubling down, or Amazon, broadly, is doubling down on Rust. And my job, this year, is to create, like, an education program for Rust, primarily focused on people who are learning or have been used to Java, JavaScript, TypeScript, and Python are the kind of other primary languages. And so this question of, like, what people need to know, and, like, where they get tripped up, is, like, really front of mind at the moment for me. So, I think that you're right about lifetimes. I think that the other part of... The thing that I'm struggling with is sort of sequencing. So, there are...

Richard Feldman: That's always hard.

The good parts of Rust

Tim McNamara: Because there are so many parts to Rust, and there's also some really, really good parts. So, one of the things that you forget when you hear, "Oh, Rust is really hard to learn," is that actually, it has a lot of niceties as well.

Richard Feldman: True.

Tim McNamara: Like enums and pattern matching, for example, is something that I just think is incredible, and I miss when I go back to other languages. So, for example, let's think about how... Conceptually, if I've ever used a language of an exception construct, Rust doesn't have exceptions, and so you kinda get this cultural mismatch, like, well, what do I do if I might get an error? It turns out you can just return those as values.

Richard Feldman: Right. Well, but importantly... Because I think if I were coming from, like, a Go background or a C background, where errors are returned as values, I might think, "Uh-oh, uh-oh. This is not gonna be good, because what if I forget to handle it," right? But, yeah. But the way that Rust does it, it's not like that at all. I mean, you cannot forget to handle it. The compiler will be like, "You forgot to handle this error here. You need to do something about that."

Tim McNamara: Hmm. And what's amazing to me... So, yeah. For every function that might create an error, and so this, primarily, you'll encounter this when you're learning, when you deal with the filesystem. So, essentially, you consider anything saved to disk as flaky. It could potentially fail. For many reasons. You might not have permission to open files. There are a whole bunch of reasons why things could fail.

Richard Feldman: Totally.

Tim McNamara: And so, there might be not enough disk space. Who knows? I mean, disks are pretty big these days, but...

Richard Feldman: Lots of reasons, though. Or you could be on a virtual machine, and the virtual machine's disk is very small and limited.

Tim McNamara: Interesting. No, that's right. It's just virtualized away. They might not even be a disk, right? So, now, Rust will guarantee...it'll say, "Look, by the way, that IO function, like, the function dealing with input-output, returns something that might fail. Dear programmer, you've only addressed the case where everything is on the happy path. I'm gonna refuse to compile now." What I love about this is that this is known in Rust as the result type. So, the result is normally gonna be okay, and everything's gonna be fine, or it might be an error. And on the error side, you might think, well, this is... It's so fundamental to the Rust way of thinking now that you kind of think of it as being part of Rust, but it turns out that the result is defined in the standard library. It's not part of the Rust language. It just falls out of the enums.

So, part of, I think, the magic of Rust, the language, is that there are these really powerful constructs that are consistent and embedded in the way, like, the practice of writing Rust. And primarily, Rust results, but also an option, which is a similar but different type, that encodes the possibility that there is a missing value. I think it's just wonderful that instead of just, like, the Python, none, or, like, JavaScript, null, or undefined, it could be essential throughout your entire program. Like, every...

Richard Feldman: Right. And I think it's interesting that some modern languages are starting to adopt some of these things. Like, Java now has an optional type, which works, essentially, as I understand it, the same as Rust's option type. The difference is that Rust uses this exclusively. It's not like Rust has an option, and then they also have null, and they also have undefined, and they also have another way of... It's like, no, there's just the one. There's an option. Whereas in Java, you do have optional, and you also have null, which, you know, for legacy reasons, if you want to argue it that way, it's like, well, you still have to be aware that nulls might happen. In Rust, it's like, no, you don't. You don't have to be aware that nulls might happen. That's not a thing in normal Rust code. Now, granted, if you do specifically have a Rust pointer, that can be null. Okay, fair enough. But now we're already in unsafe territory. So...

Tim McNamara: Essentially what you're saying there is, well, you've kind of, yeah, if... There are ways to kind of unlock Pandora's box, but if you're doing that, it's intentional, right?

Safe Rust

Richard Feldman: It's interesting like you were talking about earlier, like, in Python, it's sort of culturally a thing that if you're gonna do interop with another language and drop down to the lower-level stuff, it's like, that's kind of for experts, and "here there be dragons." But to me, it, at least when I was getting into Rust, I felt the same way about unsafe code, where it's like, well, I'm doing low-level stuff in Rust, but there's still this other level, that's deeper and scarier, where I use the unsafe keyword, and now all bets are off. We're back in segmentation fault land. So, it almost feels to me like somewhere in between, like, a Python or a Java or something like that, and the lowest level, like, assembly language, machine code, you know, C, which is just portable assembly, as they colloquially say, Rust is somewhere in between. Like, it is definitely lower level... Safe Rust, I should say. When you're not using the unsafe keyword. It feels a lot more like Python or Java, with, in a lot of ways, better ergonomics, and a higher learning curve. But at the same time, I get almost all the power of that lowest-level stuff. There's just this little extra bit that requires going into unsafe if you really wanna get into those things.

Tim McNamara: This is one of the things that I intentionally included in the book. I got quite a lot of pushback from reviewers. So, Manning has this process where they send out your manuscript to, say, 20 people that anonymously provide feedback.

Richard Feldman: I remember this.

Tim McNamara: And one other of the reviewers... It's, like, the opposite of fun, right? No. It's a really important part of why Manning's books are very high-quality. And one of the things that I stayed quite firm on was exposing readers to unsafe. I was like, this is a valid part of the language, and we shouldn't be kind of intimidated by it. Or at least there are some really powerful things that this unlocks. And one of them, or at least if we ever...like, for example, one of the examples is a memory scanner.

So actually, we wanna invoke segmentation faults, and kind of find out where things are... And we kinda need to do silly things, like converting integers, and raw integers into memory addresses, for example. And this means we need to kind of lie to the type system, or, like, lie to the compiler.

Richard Feldman: Right. Well, and I think...

Tim McNamara: No, from, like, a teaching point of view, I did also wanna say that you shouldn't feel as though this isn't learnable. I just had a slight flash of a thought, which was, one of the reasons why you might want to do unsafe. Well, you probably don't want to do it, but one of the things Rust does badly, by design, essentially, is cyclic data structures, or, like, a graph. So, normally, things in Rust land are in a tree. As soon as you have references from children back to parents, you create a cycle. And those sorts of cycles turn out to be quite nasty in languages that don't have a garbage collector, like Rust.

Richard Feldman: Yes. I think that's interesting... I'm surprised, first of all, that reviewers would push back on that. I mean, to me, it seems like there's an open question of, like, should a beginner book get into this or not? But it seems worth mentioning at the very least, because, essentially because I think there are a lot of misconceptions out there around unsafe. Steve Klabnik wrote that whole blog post about, you can't "turn off the borrow checker" in Rust, which is not what unsafe does. It's like, the borrow checker is still very much there. There are just certain operations that are unsafe for potentially other reasons, or potentially a lot of reasons, that require the use of the keyword. I think it's helpful, if you have time to do it, to explain to readers, here's what unsafe is. Even if you're not going to go on a whole long tangent about here are all the different things you can do with unsafe. Or especially giving people the wrong cultural impression. Because certainly, from what I've seen, in Rust, there's a very strong cultural aversion to using unsafe, and really, people wanna see that, you know, a library that they're depending on is using it as little as possible.

Recommended talk: Elm in Action • Richard Feldman & Thomas Anagrius • GOTO 2020

Tim McNamara: I don't wanna give people the impression that being unsafe is going to ruin your program. And I think what you're trying to say there, Richard is that it's very easy for tooling to detect the use of unsafe. And by the way, if libraries are known as using this construct, it becomes, like, a very strong target for, like, refactoring. And this happened with, like, a very prominent web framework called Actix, which had extremely good benchmarks. But, the immune system of the Rust community kind of came back, in some ways, and said, "Whoa, whoa, whoa. You're faking it." Well, not quite. It turns out that there were good reasons and so forth. And it seemed like this big cultural war around what is a...

Richard Feldman: Right. A big controversy.

Tim McNamara: It was quite big. I mean, there are programming language dramas. Like...

Richard Feldman: Of course.

Tim McNamara: I feel like explaining a little bit about what's safe. So, there are two uses of the word unsafe. So, the keyword itself says not "this is inherently unsafe." What's more, I, as the programmer, take responsibility for verifying everything that the compiler would by itself verify. So, remembered why we had, like, this very strong linter. They're called the borrow checker. Now, essentially, you as the programmer take on responsibility for being the borrow checker. That's essentially what unsafe is used for in the primary use of that keyword. It doesn't provide you, like, a free-for-all to do everything. That's what Steve's post was about. And it's intentionally ugly. So it's hard to just kind of accidentally insert. The other use of...

Richard Feldman: For sure.

Tim McNamara: ...unsafe is you can define a trait. So, this is kind of an interface or an abstract base class, that is unsafe. And this is a subtly different use of the keyword. What this is saying is that people that implement it need to guarantee that their implementation sort of complies with a contract that is probably written in the documentation because the type system can't really encode it. That's kind of my understanding of the two big reasons why you might want to use the keyword.

Timekeeping in Rust in Action

Richard Feldman: One thing I want to make sure to ask you about is, in the timekeeping chapter, you have a photo of a physical device that keeps time. What's the story behind that?

Tim McNamara: You're just getting me to geek out on, like, another one of my big passion projects, I suppose. So, I'm quite into, I guess, nature protection broadly. So, it's a box. And inside... it's a device that takes infrared cameras, from a project called Cacophony. It turns out that when European settlers arrived in the 18th century into New Zealand, over about a century, they introduced a lot of species that were harmful to a lot of our native birds. And things like stoats and weasels and rats like to eat either birds or eggs. And turns out that if you're a flightless bird, this is bad for you.

Richard Feldman: I bet.

Tim McNamara: So, there were some diaries of explorers. One of the phrases was that at night, the forest was just this cacophony, and the whole forest was just alive with birds. We didn't have any predators at all, so a lot of our birds are nocturnal also. So, famously, the kiwi, the bird with the long beak that can't fly, is a nocturnal species.

Richard Feldman: I didn't know that.

Tim McNamara: And there are a lot of adaptions that are quite strange, by the way. So, we have birds that pretend to be rodents. We have birds that can hop around trees, so they're kind of similar to monkeys, or provide the same...they live in the same ecological niche, or provide the same function as what mammals would in other biospheres. Anyway, the Cacophony project is attempting to use machine learning to identify, is this a cat? Is this a rat? Is this a weasel? And it does it with infrared vision. It takes photos, or takes videos, actually, and timestamps them. But it turns out that if you're a tiny little device, it's really hard for you to tell time. You need a clock. And then I was like, wait a minute. So, part of this book was this amazing research, and I was like, how on Earth does any computer tell time? And it turns out, one of the properties of quartz crystals is that when they're electrified, they oscillate. So, they go backward and forward at a very specific frequency. That frequency is counted. And therefore we know the voltage that is being sent into the crystal. We know the rate at which it will oscillate. And therefore we can tell time because we say we start at zero and we just start clicking across.

Eventually, we will hit some number. Let's say we have, like, 16 bits. We'll get to 65,000, and we'll need to wrap around to zero, but we might have some way of dealing with that. It turns out they do. But it turns out this is essentially a primitive way of keeping time that is used by essentially every digital computer. But, then, how do they synchronize? And this is one of the problems of distributed computing. So, how does everyone say, okay, I'm currently at oscillation 25,000 of my current cycle? How does that translate to the 6th of March, 2023? And it's all distributed... I'm gonna get into a big flame war. But essentially, what we have is canonical devices. So, much more accurate timekeeping devices, which are what are known as atomic clocks, use even more specific... And they all kind of agree.

It turns out that, I think there's about half a dozen, maybe a dozen of these atomic clocks. They're called tier one devices, or tier zero devices, inside the network time protocol. They agree on what the time is. Then they just tell other people. And then just like, let's say, a DNS system, it kind of propagates through. It turns out that if you are inside a data center, you can get clocks to agree on what the time is without the... Because at some point, it kind of becomes irrelevant what the atomic clock thinks the time is. What matters is that the clocks that are really close together agree. Because you want, say, for example, logs or messages to be roughly in order. It turns out you can't do this…

Recommended talk: Why is Rust Successful? • Florian Gilcher • GOTO 2017

Richard Feldman: Yes. Right. Monotonic ordering, as they call it. 

Tim McNamara: Like, in a distributed system. And it turns out, by the way, that a single multi-core laptop, or a single multi-core phone, becomes a distributed system because each CPU operates independently. This is kind of, like, one of the mind-bending things about technology. Like, it's just so incredible that, like, this stuff even works. And, like, I just think about it, like, at every single level, you have these examples where at every single level... Like, let's think about the text. We started with, like, plain bytes, and then we needed... So, we started with, like, Morse code and other coding signals for interpreting, like, binary digits and translating them to letters and numbers. And then we kind of, like, wrapped around more and more on top. So, in ASCII, which is, like, the encoding system from the '60s, there are control characters, and we still managed to kind of hold onto those, even though, like, the bell doesn't mean much today. And famously, operating systems still can't agree on what a new line means. But, do we need a carriage return, and, likely there's no physical carriage?

Richard Feldman: Depends on which operating system you ask.

Tim McNamara: That's right. And somehow we've just added on more and more layers. Again and again and again and again. I love it. And that is kind of, I was like, you know what? I am gonna spend essentially years of my life writing a book. So, I did some calculations as I was in the final stages of processing, or publishing, about how long this horrible project took. I spent, on an average week, between 10 to 15 hours per week on this thing. So, three or four hours a night, three or four nights a week. Plus I try to do, like, a chunk on the weekend. From the initial conversations with Manning through to publication, it was essentially a four-year project, because I needed to, like, do extensive rewrites of both the prose as well as the code, and answer lovely questions from, like, people reading it as well. That was a bit of a time suck. And, like, doing diagrams took weeks and weeks and weeks in some of them.

Richard Feldman: I remember all the diagrams. It's like they don't take up that much space on the page relative to how much time they take up to make.

Tim McNamara: That's right.

Richard Feldman: In a lot of cases.

Tim McNamara: So, I was like, I've probably dropped, like, 2000 hours into this thing.

The writing process

Richard Feldman: Well, that's a great segue into my final question for you, which is, looking back on how much time you spent on it, you know, was it worth it? Are you happy with the result? You know, you said, you mentioned it was good for your career, but just as an author, like, you know, what do you think of the whole process?

Tim McNamara: What I picked up from this interview is that I learned a tremendous amount, through the research process, and through, the writing process, I became significantly more skilled at communicating. I always had the goal would speed up people's learning journey by 100 hours. At this point, I know that potentially hundreds of thousands of people have read the book. And, like, if you were to multiply my input versus the output that it's generated, I think it's been, like, a huge win for the industry. It's created a whole wave of Rust material inside Manning and other publishers, who have said that Rust is a viable product, or, like, is a viable ecosystem. So, the only thing is that my health suffered quite badly. I used to run marathons. I used to lift weights. I used to be able to pull, like, 200 kilograms off the ground.

I've put all of that aside, and I think that my kids are the people that suffered the most. Because there were times when we could have gone to the park, or we could have gone for a walk, and I was, like, writing the book. So, my message to readers is, like, I hope that you find the material that's useful for you. It might be "Rust in Action." It might be something else but find it. Like, this is a community that is welcoming and accepting of you, and there are now, like, a very rich set of resources to draw from. And I think my message to my kids is that, like, I love you lots, and I hope that you forgive me.

Richard Feldman: I'm glad that if you're going to make a big sacrifice like that, in so many different aspects of your life and your kids, I mean, it's great to hear that, you know, like, at least you got a really good book out of it. And it's not just, you know, something you look back on as like, "Oh, I'm never doing that again," you know.

Tim McNamara: People keep asking me for the second edition and the second volume, or, like, the next book. I'm like, well, actually, I might say yes, but there are consequences, right?

Richard Feldman: Well, I think that's a good spot to wrap things up. Tim, thanks so much for talking to me about the book, and thanks for writing it. It's an excellent book, and I'm just glad we've had the chance to talk to you about it.

Tim McNamara: Nice. Hey, thanks so much, Richard.

Richard Feldman: All right.