C++ Memory Management

Introduction and Background

Kevin Carpenter: Welcome, everyone. Kevin Carpenter and I'm with Patrice Roy and we're with GOTO Book Club. We're here to talk about Patrice's new book, Memory Management and C++. I have to say, Patrice, I love chatting with you, but I always love picking on your experience. I'm not great with memory management, which is why I'm here with you.

Patrice Roy: Well, I don't believe you, but you're very sweet. I am hostinghref="https://www.eventbrite.com/e/c-memory-management-masterclass-tickets-1580340644409?aff=packt"> an event about the book too

Kevin Carpenter: Let's just say when I got offered the opportunity to edit your book, I'd already done one book with Packt, and they're like, well, we have this other project. I'm like, I don't want to do that project. They're like, but it's Patrice Roy and he's got... oh, it's Patrice. Okay, well, you know, it's your baby. Thanks. So it was an honor.

Patrice Roy: I was granted very good technical reviewers, I was lucky.

Kevin Carpenter: Introduce yourself because then you can give your credentials, which I definitely want.

Patrice Roy: I've been programming since the early 1990s. I've been doing C++ for way over 30 years now. I used to do military flight simulators. I have been teaching since 1998, to both undergrad and grads. I work a lot with video game programmers, and I've been a member of the C++ Standards Committee since late 2014. My first meeting was the May 2015 meeting in Lenexa, Kansas.

Kevin Carpenter: I know you from all the CppCon conferences, all of your classes. When I think of things that I've seen you do before, this to me was new memory management. I mean, I know obviously with all your experience, you've got a deep knowledge on it, but am I right? This is your first book on memory management? I don't remember you doing any classes on memory management.

Patrice Roy: In fact, I have done a few. That book was interesting in the sense that after doing the memory management class two or three times, I think I was getting deep in thinking, and that's where Packt approached me and said, well, would you like to turn that class into a book? It's an interesting challenge. It's a very different experience. I like doing it, but it's not the same thing.

Recommended talk: What Every Programmer Should Know about How CPUs Work • Matt Godbolt • GOTO 2024

Challenges for New C++ Developers

Kevin Carpenter: What is one of the hardest concepts for someone who's new to C++ memory management? What's one of the harder concepts for new C++ developers to pick up?

Patrice Roy: People do too much of it. We get people who come from an earlier C++ background from way back or from other languages where they have garbage collectors. C and D tend to try to allocate memory dynamically themselves all the time, but C++ works best with value semantics. It works best with objects. The book is not to tell you never do it, but if you need to do it, here are some ways so you should not get yourself into trouble and to get success out of it. Most of the time you shouldn't use it at the point of C++. The best angle for C++ is to avoid this thing unless there's a need. But there are needs and I think there was a need for the book.

Kevin Carpenter: Do you think a lot of that need, you know, with all the talk that we went through in the past couple of years about security and everything else, knowing how to manage C++ memory keeps security solid? Would that be a fair statement?

Patrice Roy: Well, we have all the tools to do things right these days. The metaphors that we're using with iterators and algorithms, they reduce the number of bugs tremendously. We have good containers that manage memory for us: vectors, strings and such lists. They're good at what they do. When we do need to write them ourselves and get our hands dirty, we have to know what we're doing. All the tools are there to do something safe and clean. There's still some things with lifetime and C++ that are hard to solve, like binding to a reference to something that's not there anymore. We can get in trouble with that still today. But we are getting better. The book is not about safety, but it does emphasize proper ways of doing these things without getting too much in trouble.

Kevin Carpenter: I agree, it is definitely not about safety, but I think of the way that you've written things and the way that you pull comments where it's like, okay, you can do this, but if you're doing it, you're probably doing it wrong. Or if you're doing this, remember, it's your own foot you're shooting off.

Patrice Roy: I did take a risk. You remember reading the book, one of the early chapters is things you shouldn't do. But when we go to a low level, we have to do them sometimes. If you have to do them, please do them right. Please understand that some things might work on your compiler, but they're not really working. You're just lucky. It's just an accident. Or it's something non-portable and you're going to get into trouble. But there are tools. They exist. This is C++, you can do what you need to do. Do it right.

I thought it risky to put a chapter early on things you shouldn't do. It felt weird but it seems to have worked, so I'm kind of happy about that.

Code Examples and Teaching Approach

Kevin Carpenter: The thing that I really liked is your code examples and the way you wrote the code examples. Some of them are really concise. What I mean by that, it's not like I'm going to necessarily copy out the particular piece of code, drop it in the compiler and make it run. That doesn't mean you can't, because I tested every piece of code, but the way that you wrote the comments in a particular piece of code, showing how if I assert this value, it should be this. If I start this, it should be that. But then right there, clearly in the code you're like this assertion is going to fail because we shouldn't have done it this way.

Patrice Roy: Maybe it's a tick or something like that. I've been teaching for a long while. I like examples that convey information in a few words without being too obscure. I'm trying to do that. That's not always possible. You might remember the later chapters where I deal with containers is when you do get your hands dirty and do the very efficient memory management stuff, it's hard to understand. It's something you need to test, but I try to keep most of them small and understandable. I'm glad if you tell me it shines through. I'm very happy about that.

Kevin Carpenter: You're absolutely right. As you get into those later chapters when you're building your own container and such, I was definitely putting that code in the compiler, stepping through it. But still the way that you would document the code, we talk about writing self-documenting code, and that's one thing. But when you're writing it as an explanation, the way you would do the assertions in the comments just made it flow. You'd read the concept in the paragraph, and then it was very clear in the code example exactly what you meant.

Performance Considerations and Modern C++ Features

Kevin Carpenter: When I came into reading your book, yes, I know pointers. I made the point about an old MFC app, which I do want to hit on, because in that app we actually do pointer arithmetic over an array of doubles that's probably 400 or 500MB in memory. It's incredibly fast. When you consider something like std::array, which would give you your finite size and make it so you can't run off the end and give you all those safety items, do we give up a lot in speed if you go from an old C-style array of doubles to using std::array?

Patrice Roy: It's not the same use case. In your case you probably, if you don't know the size in advance, you're going to need to allocate something. And if you want the pointer to an array, a C array, you will have to specify the size there. So it's not going to match what you want to do. You could use a vector and it'd be very fast. As long as you reserve once and then fill the thing, it's going to be blazingly fast too. If you look at what pointer arithmetic does, it's the same thing as the array operations with the square brackets. One translates directly into another. You won't be losing speed with this, but I don't think std::array would be your use case if you're using that immediately, then you can reallocate stuff unless it's always the same size.

Kevin Carpenter: And that is the benefit. Yeah, I was going to laugh and say, well, of course we know the size. It's a macro because it's MFC and we have this many lengths and this many widths across this many accounts, and there's probably four or five other macros. I've been trying desperately to move to constexpr, but the code is old and brittle.

Patrice Roy: Mind you, if MFC still compiles given the idioms of way back then with today's compilers, which is something I would have to check. I haven't used that library in about 20 years though. But I did use a pre-98 compiler 10 years back for real-time systems. It was an experience to get my old bag of tricks out and just rewrite everything.

But what you're referring to, you might be tempted to use the span. This new span stuff from C++23. Proper indexing very efficiently without having to go low level and do it yourself.

Kevin Carpenter: I will have to take a look at that. Because we are building with C++17. To that point of other chapters you have in your book about shared_ptr versus unique_ptr and weak_ptr. Even with your examples, we probably use shared_ptr more than we should because in some cases as they adapted the app, too many things are being accessed everywhere. But at least you're getting the extra protection of having that reference count versus a raw pointer.

Patrice Roy: Yeah, but the idea of shared_ptr is not having many users. Raw pointers do that. It's having many responsible bits of code over the lifetime of the object. If you have a single owner of your object, you don't need a shared_ptr. In most cases, unique_ptr does the job. It's a matter of responsibility. For sharing data, raw pointers work fine.

Recommended talk: Functional Programming in 40 Minutes • Russ Olsen • GOTO 2024

Smart Pointers and Modern C++ Practices

Kevin Carpenter: That's the part where I have been kind of torn, you know, because this is not like being a micro application. It is a Windows app that is double the complexity of an Excel because it manages financial data. It manages some of the largest financial institutions' balance sheets in the United States here. With that code's age, you try to bring it in and you bring in more modern parts, it's that little bit of a double-edged sword. There's parts that when you write something new, you're starting to write things more modern, but then you still have that raw pointer access that's going on, which is very efficient. When I think of the chapter in your book early on where it's like you really don't want to use raw pointers unless you have to, as you point out. That's the part where I like how in the book throughout it, you point out the things to consider. Don't use it. Are you considering these three, four or five items if you are using it.

Patrice Roy: The kind of message that we're trying from Standards Committee members these days, what we're trying to convey to our users is that if you get a raw pointer as an argument to a function, it's telling you you're not responsible for it. Use the thing, to be an observer of it. Consume stuff, but don't take it as your own or try to delete it or reallocate it. This is the mission of your function. Of course, if you're writing something else, then yes, it turns out fine for that. What you often see, especially with legacy apps like yours, is the owner is a unique_ptr unless there's co-ownership and then if you use the get member function, you get a raw pointer. And you provide this to non-responsible-for-the-pointy functions. And it all works fine. There's no overhead at all.

Key Takeaways for New C++ Developers

Kevin Carpenter: Someone new to C++ coming into this book, can you think of three things out of your book that you would really want someone new to C++ to take away from it?

Patrice Roy: I did make an effort to promote best practices through writing new code. Try not to wrap your memory management into objects that you can test and do it right. Use the standard library provided ones. They're very good. Vector and stuff, they're awesome. If you do that, that's cool.

C++ is easier to use tools, I'm sure there are. But it's that tool that will get you where you need to go no matter what you have to do. It's very good.

I'm hoping that someone will understand reading that book that yes, there's tricky stuff you can do with it, but there are reasons and you can measure a need and that's fine. Now you know how to do it if you're stuck. I'm hoping that newcomers will try to write that, okay. This Ruby container that they use.

The standard ones are very good. But if you have a specific need, here's how to think about it. Here's a way at least to approach this. I tried to offer alternatives to these. We have new kinds of containers and we have PMR stuff these days. There are upsides to these, downsides to these. I'm hoping that people will be able to weigh these things and make informed decisions.

If someone's newer, maybe this is not a beginner's book, but someone who's a junior in C++, right? C++ from another language. From this book, I'm hoping they'll learn some practices and the reflex that they will sit down and think before they act and when they need to act and use the tricky stuff. Well, they'll do that in an informed way. Help responsible programmers.

Kevin Carpenter: You did say something there where it's like, you hope somebody who's new to C++ after reading the book doesn't come in and write their own custom allocators.

Patrice Roy: Unless there's a real need. But when you begin, there's normally not the first thing you do, right.

Custom Allocators and Performance Optimization

Kevin Carpenter: That's the part where I'm like, if there's a real need. So it makes me think of a talk I gave last year called "Almost Always Vector”.

Patrice Roy: I was in the room.

Kevin Carpenter: Yes. You were, okay. So you get my point then. Because that's that part where similarly you don't need to write a custom allocator. But if you did, here are the reasons why. I think of some of the memory patterns like the arena pattern and the ways that you would allocate memory, because depending upon that particular type of object that you're creating, if you're creating multiples of them, then you can gain that kind of efficiency. That might be the way or the reason you would write a custom allocator, right?

Patrice Roy: Maybe you can measure an actual gain. The point normally with these things is that your standard tools are very good on average. They're very good. But when you know the context and you know things about the data and the way it's going to be used, consumed and handled, you can always do better. Because the average case, it's something you know stuff about.

But write to measure a need, of course, because you're going to spend time on this thing, you're going to have to debug it and test it. So there has to be a real reward for this. But if it does bring rewards, I mean, the things I do in the book, they're funny and weird, but it's blazingly fast. It's very, very fast. But the conditions make it possible.

Another thing that I don't touch as much in the book, I could have, it's a matter of size and number of pages at some point, it's instrumentation. You want to know where the data is. I've worked for game people. You want to know if there's cold regions in your game, hot regions where lots of people are coming in. Well, you can track these things and you can know where the blocks are allocated. So there's ways there to optimize your code a posteriori, afterwards, just because now you know how the thing is being used because you instrumented the thing, maybe not as people are playing, but maybe if it's fast enough, maybe as the game is going on, maybe in your laboratory when you're testing stuff.

But you can write allocators because you want to know where stuff is. In both cases, arena style stuff and instrumentation stuff, it makes it easier than the old allocators. But there's PMR. You're incurring a virtual function call every time. It's a matter of simplicity with respect to efficiency. You have to know where your needs are. But this is C++. We give you tools. You build with them.

Kevin Carpenter: But to your point, measuring and knowing where and why, because what does that saying go, premature optimization is the root of all evil or something.

Patrice Roy: Be careful, that quote is always incomplete. 97% of you don't really need it. But on the 3% we need it, you better be able to do it.

Kevin Carpenter: That was where I was going to say your book to me really addressed when I need that 3%. The examples and the knowledge that you detail in there, when I find the spot that is definitely eating resources, whether it be from allocations being inefficient or such, what I got from the book was exactly how to fix that.

Patrice Roy: Well, there's other things that we ignore when we do day-to-day programming. When you need to access exotic memory, something that is different from your normal RAM. You want to write non-volatile stuff, you want shared memory. C++ doesn't delve into that as a language. We don't talk about processes in C++, we talk about threads. We don't talk about the operating system all that much because it's outside of our purview. But we do have tools in C++ to let you interface with those other things efficiently. This is another reason why the book might be interesting to some people. Sometimes you are doing something operating system specific and you want to map it to your C++ code in a nicer way, in a more agreeable way maybe, than doing it the C style.

So there's a whole chapter on that. I hope that these exotic cases, well when you face them sometimes you're kind of stuck. How do I do this properly? Yeah. It helps you.

Kevin Carpenter: You were talking about the way that you mapped data structures and stuff. It's something small, but it kind of made me laugh at myself because it made me think of the part where inside the book, you're talking about padding. It was interesting because I never really thought about it too much beforehand. But then when I would go back and look at some of the data structures, simply changing the orders suddenly saved 16 bytes of memory on an object that we were creating 100,000 of because it was a particular type of credit card transaction. And just over and over, it's like some things that I would call little gems, once I got it, it was hard to unsee.

Patrice Roy: That's insane. Alignment. Alignment in other languages, they don't know about this. But since this is one of the peculiarities of C++, we let user codes see the actual addresses of things so that we can better change the order in which the objects are, because the order in which we put them has meaning and is observable elsewhere in the code. There's something that's different. In other languages we see these things. If we put things without taking care of alignment, we could well, considering it, yes, we can make the object significantly bigger than they should be. Maybe. But you have to be careful. There's other considerations. When you have multi-threading, sometimes you want to keep things apart, sometimes you want to keep them close by. Depends on the kind of code you're writing, but you have to be aware of that to be efficient. And yes, indeed, if you want to use your cache efficiently, well, alignment has a big effect on your size and thus on your speed.

Kevin Carpenter: That makes me think about cache alignment and such. That takes me back to the part that you have in there about shared_ptr. When we're looking at that, and about whether you make_shared or shared. I'm curious, in your experience, what is that difference of having the counter ending up being on a separate cache line from the rest of the allocation? Is that like, does that end up becoming a big performance hit depending upon the type of objects that you're creating, the speed of them?

Patrice Roy: Well, the short answer is yes. The longer answer is, it's kind of fun. When I teach that thing in the classroom, I make students different from different tables and one of them's the counter, one is the pointer to the object. But the thing is, you have to, with each access of your shared_ptr, let's say you pass it to a function or return it from a function, something like that. Well, you hit two cache lines at once. So there's a limit to the number of cache things that you can access efficiently in the machine. You're taking them away for no reason. So there's less left to do other interesting stuff. So yes, it will have a measurable impact on your program and on the quality of your life.

Kevin Carpenter: It's fair to say in our C++ community, we definitely have the high frequency trading, which I think of where these things are a big performance type item. Then we have games, of course, embedded where you're really constrained on your resources. But what was interesting, what I kind of laugh at myself about. So the day job I work, credit card transactions, we're processing billions of dollars a day and generally speaking, I am not worried about cache line hits and everything because of the size.

Patrice Roy: It's a matter of size. Your problem space is so big with arrays of transactions.

Kevin Carpenter: In a way, yes. I mean, a lot of our transactions, we do a bunch of JSON, which ends up, of course, representing as a map, as a std::map. And then, actually, as they are passed down through the system, they finally end up becoming raw arrays of data, whether it's arrays of unions. Once you break down, it's like the actual credit card transaction itself comes down to an integer, another integer and another integer, because the card number represents as an integer, the date represents as an integer. And what people don't realize is actually the dollar amount represents as an integer in the specifications. That part ends up getting passed around really quick. It's interesting because as the data comes in and we're actually working within, it trims down before it actually gets processed because you actually go to charge your hotel and they're actually sending you had a queen bed in your hotel room and everything else in the transaction because of fraud controls. But then, of course, it does break down the data. But to your point, the problem space is just huge. That's the part where it's like there's a lot of tips that I got going through the book that makes me look at the code that I write now differently and with more care.

Patrice Roy: I'm touched.

Patrice Roy: Can I ask you a question? Do you know when you're processing your credit cards like this? Do you do mostly reads, or do you also do some writes to the array? Because that will make a big difference in your code. If you have two cache lines accessed concurrently by two different threads in your code, one of them does a write, it's going to wreak havoc in your performance. But if you're doing reads concurrently, nobody cares. It's going to be very fast.

Kevin Carpenter: Most of it comes, so there's two sides of credit card transactions. One is the authorization. And that's a pretty concurrent situation because they're all coming in at different times. You're handling one at a time. But that part happens, that's when you swipe your card at the restaurant. But at the end of the night when they go to process, that's when we're turning around and reading 20,000 transactions in one line, processing them through and then writing them out somewhere else. The typical programmer, we read from databases to process, to write back to databases.

Patrice Roy: Awesome. So this is very linear stuff that you can do without concurrency at all that you don't need to. The cache was efficiently handled. In this case, it's probably different from the shared_ptr case you were referring to, where you're allocating the counter on one side, the pointy on the other side. So you get two pointers one size the other. And if you can close, since you're using them together most of the time, it's going to be more efficient.

Template Computing

Kevin Carpenter: Gotcha. So real quick, one last question, because I know we got to wrap up here at some point. Your book, the little bit of template stuff that is in there was, because I am not a template programmer by nature either, just as I learned memory management from you. But when it came to template programming, it's the kind of stuff that I think we all expect just using C++ in a way. It's like you're using std::vector. We know it's a template. We don't really think of it because we're not writing it as a template, same with std::shared_ptr. Now in some of your examples, when we start writing our own container, even that little bit of template stuff was, I'm going to say template easy. I didn't need to know a lot of templates in order to go through your memory management book.

Patrice Roy: You're sweet.

Kevin Carpenter: But is that generally the case with the memory management that it's template stuff, but it's not like getting big into generic programming, right?

Patrice Roy: No. I'm sure you can find use cases for complicated stuff. You should look for it when I tend to find it useful. Do you remember there's a chapter on deferred reclamation, kind of garbage collector-ish. If you want to finalize the objects you're pointing to, it's kind of nice to somehow remember what you're pointing to. So being able to store this in a function that itself remembers the type of the thing you have to destroy. It's kind of a neat trick that I stole from the standard when they added it. So there's these tricks there that are templates that this is not too hard, but mixed with function pointers, you can go a long way with them.

Kevin Carpenter: Yes, I do remember that. That's very cool. And I have to say some of the other parts that I liked, I really liked when you broke into new and delete and the details of how new is actually a two-step operation, and then you're customizing new so that you can turn it into a tool to watch your allocations and deallocations.

What’s Next?

Kevin Carpenter: But you know, back to my generic part. So next we're going to work on a generic book, right.

Patrice Roy: We started working on that. My next book, which the first chapter is written now, is one that I think is going to be nice. I'm trying, the way of approaching this is again, turning one of my classes into a book. But of course, it's never the same thing as you can see though.

I gave a class a few years back called "Generic Programming From Strange to Normal." So it starts looking weird and wondering why you would use that thing and what are the benefits. And then you're progressing and learning stuff gradually. At some point it just becomes normal programming and that's the beauty of it. And I can get you there with C++ because it's such a cool language.

First, right now I'm working on developing an intuition of what's the point with this thing? And if you have questions about deeper template programming, I'm going to get there. But I'm playing this one as an expert book, as a book for someone who wants to understand what he or she is going through. So there will be SFINAE stuff, but I'll build all the way to make it understandable for people.

I'm hoping that they will be as happy with this book as you were with the memory management book. And they will think that they learned something and understood something from it.

Kevin Carpenter: Well, I do look forward to that. I really do, because generic programming is also not something that's my forte. I do want to end with, anybody watching if you, I think Patrice has made clear this is not, if you're new to C++, this is not the first book you want to pick up. But I want to say from my experience, going into this, even with ten years of programming, I never really saw outside of some raw pointers and such. I didn't do a lot of memory management. I certainly didn't do things to the level that Patrice has in his book. Whereas there were parts that, there were chapters that I definitely pored over more than once, the book was really accessible. The examples, as we kind of talked about, really were easy to follow, even if it took me once or twice reading through it because I think there's a benefit to that, right? You're doing the mental CPU work, which is where we really learn, at least for me, it's like trying to lift a weight up the first time. It's a bit of a struggle, but then once you get it, you're like, oh, oh yeah, that. If you haven't read the book yet, you want to pick up a copy and your memory management skill will definitely improve. That is for sure.

Patrice Roy: You're very kind. And if you find bugs or things that you want to talk about, write to me. I'm going to be happy to speak with you.

Kevin Carpenter: That's excellent. Patrice, I really appreciate your time today and talking about your book, and I look forward to when we can do this again about a generic book.

Patrice Roy: It's always a pleasure, my friend. Thank you so much.

Kevin Carpenter: Thank you, Patrice.