Problem Solving With Erlang and the BEAM

The early days of Erlang

Francesco Cesarini: Hi, everyone. I'm Francesco Cesarini. I'm the founder and technical director of Erlang Solutions. I've been working with Erlang, initially a programming language, which later became an ecosystem of languages in 1995, and we are here today to talk about the whole theme of problem-solving, problem-solving with both technology and technological issues, but also political and managerial issues.

Robert Virding: Hi, I'm Robert Virding. I used to work at Ericsson and was one of the initial developers of Erlang together with Joe Armstrong and Mike Williams. This then became the Erlang language and, well, as Francesco mentioned, part of an existing ecosystem.

Francesco Cesarini: Can you tell us a little bit more about how you came about, who came up with the idea of inventing a programming language? Was it a particular problem you were trying to solve?

Robert Virding: It was a particular problem. We were looking at telecoms applications, and we'd done several...We've worked in a computer science lab, we'd done several experiments and found that no language we thought was well suited to solving that problem. And that evolved then into looking at how we would like to solve it ourselves, understanding more properties of the problem, and also developing a language that suited attacking these types of problems. We were very lucky to have an external group who came in with a lot of feedback who were very knowledgeable in the systems themselves, what they needed to be done, and they could give feedback on our ideas of what should go in the language and the properties and how you would use these things to try and solve the problem.

Francesco Cesarini: How was it perceived internally within Ericsson at the time? I mean, there's Erlang the movie, which has kind of leaked its way and made its way out the door but what was management's view?

Robert Virding: Well, there were a lot of projects going, internal Ericsson projects going on trying to develop the new systems and architectures for new systems, so in that sense, we were one of them. And well, like all of them, some of the people supported us and some people didn't like us, but we managed to keep on going. We managed to get the first group who were using Erlang in a commercial product inside, and I think that did very much to ensure that it would keep going on.

Francesco Cesarini: What year was that? When did they start working on this product?

Robert Virding: The product came out in '94. I think they started sometime in '92 to work on the product. But this was a group who had given us all our feedback for it, so they were very into the system and how we intended things to be used and things like this.

Francesco Cesarini: And so it came out in '94, I assume that '94, that's when it went into maintenance mode, new features started being developed. How did that influence the language?

Robert Virding: The base part of the language was still the same. We'd got that right, or right of the way you could use that to build whatever you need. But new things had to be added to the language that came along. New requirements came, a few new data types and things like this that were required, or that the people wanted to do that to build their systems more efficiently.

Recommended talk: Erlang, the Hidden Gem: Solving Problems at Scale for 30+ Years • Francesco Cesarini • GOTO 2021

From just a project to mainstream at Ericsson 

Francesco Cesarini: What got Erlang mainstream into Ericsson? I mean, when I joined, I joined Ericsson in '95, and then as an intern and permanent in 1996. In 1996, it was becoming mainstream and all new projects were using it. What happened between, say, '94 and '96, which...?

Robert Virding: There was another, a large internal project inside Ericsson looking at developing, how to develop the architecture for the new system. And that project failed. It just didn't work. Their system didn't work. Then they had new products that came along, had to start looking for something else, and they started using our system.

Francesco Cesarini: So basically there was a void all of a sudden. The project failed, and there was a huge void.

Robert Virding: Yes.

Francesco Cesarini: They could either do nothing or bet on something new, which might save their bacon. Is that...?

Robert Virding: But there was a number, we weren't the only partner in Ericsson trying to develop new things as well, so, but our system, well, worked and there was a product to show that it actually could be used.

Francesco Cesarini: It's very cool. At the time, there were lots of products that were being started, all of Ericsson's broadband solutions, so ADSL, fiber optics, HFC. So, you know, broadband over cable TV networks, the predecessor of YMAX, GPRS, the HD 301 switch, they were all based on Erlang.

Robert Virding: It was a time of major change inside Ericsson what the type of products Ericsson was looking at. Before that had been the AXC telephone exchange, which was an extremely successful product, no doubt about that. But then new requirements came, new technology, new ideas that just had to get into the system.

Francesco Cesarini: It was a whole migration to packet-based switching from kind of fixed. You no longer needed a copper wire from one end to another. 

Robert Virding: No. That was a major, it was a major change. It was a major change in the system and what was required. And I think it was just because of the major rethinking it, almost everything, that was the time the new ideas could get in.

Finding Erlang developers at Ericsson

Francesco Cesarini: So, two questions of the time before when Erlang was only available within Ericsson. I mean, this project had a hundred-plus developers. Ericsson just got out on the street and found 100 Erlang developers for each project, you know? So, I think at the time they had something like 15,000 Erlang developers or something like...No, not 15,000, sorry, 1500 to 2000 Erlang developers, and then we have today companies complaining, "Oh, it's so hard to find Erlang developers." How did Ericsson do it?

Robert Virding: There were internal people. So they said, "Yes, this project is now going to...this product is now going to use Erlang," so now they've got people do that and learn Erlang so they could work on this as well.

Francesco Cesarini: How long did it take them?

Robert Virding: That could vary a lot depending on what they were doing. It could sometimes go very quickly, sometimes take a year or so, very much depending on the product and what...the pressure or the push on it to do these types of things.

Francesco Cesarini: Because I recall studies from the GPRS project, which has today become the 5G, which unfortunately were never published externally, but internally within Ericsson, they came to the conclusion that a student or a graduate straight out of university was productive within a month of having attended two courses and have dabbled in a bit of self-learning, someone who'd been working with a particular technology for a long time, it took them about three months before they became productive. And these were pretty impressive numbers at the time. Do you recall?

Robert Virding: I don't recall directly. I know that we took part, in this work and were still working in the lab at this time, but we took part in doing training of internal training for our Erlang people, or Ericsson people then teaching them Erlang. And, there were quite a few. We did that as quite a lot of that. And I think it very much depends on your background. If you'd worked with other systems, you might have become more locked in that technology and might take more time to migrate from it and to rethink because it was a large rethink for many people, still is actually.

Recommended talk: The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019

The growth of the Erlang ecosystem

Francesco Cesarini: What was the first company outside of Ericsson you learned about that was using Erlang? So, not do university academics because I knew there was very strong support back then.

Robert Virding: No, no. So what happened was, well, in 1998, Erlang became open-source, which if I remember correctly, was Ericsson's first open-source software. And a group of us left the lab in '99 and formed a company called Bluetail. And I think we were the first company outside Ericsson to use Erlang in a product for it as well. This was in the middle of the IT boom, so it was quite an experience. And then I know you left later to form what is now Erlang Solutions.

Francesco Cesarini: Yes. So, what got Erlang to be released as open source? Because, I mean, the whole..." The Cathedral & The Bazaar" paper had just been written in '98. I mean, it was the year that the term open source was coined, so it was something very scary and very novel for higher management.

Robert Virding: Yes, it was. We were very fortunate there, but in the group, not inside the lab, in the group working on spreading Erlang inside Ericsson, there was a woman called Jane Walerud. She was working with that group. And she was fun. She was the base of getting the Erlang language open-sourced. She knew how to talk to people and get them to understand that this would benefit Ericsson, and it should be much better than having some form of secret internal technology, which you didn't want to spread.

Francesco Cesarini: But what were the benefits she sold to management?

Robert Virding: You are getting more people using it. You're getting better feedback, the right things for tools, applications, and things like this for that, and you give people a different view looking at the language and coming up with comments and criticisms and suggestions, etc., for it.

Francesco Cesarini: Okay. And what are some of the success stories outside of Ericsson...? This was the 25th anniversary this year of Erlang being released as open source, so.

Robert Virding: Our company, Bluetail, was a success. We were finally bought up by Nortel Networks around a few steps along that, which was fine.

Francesco Cesarini: Two steps forward, one step back?

Robert Virding: Well, two steps. First bought up by the company on the West Coast of the U.S. we were then bought up by Nortel Networks and so we ended up working there, but then the IT boom ended and things became very different.

Francesco Cesarini: We were expecting Nortel to be acquired by Ericsson so we'd have the full circle.

Robert Virding: Ericsson was buying stuff there as well, too. There were a lot of changes going on. About that time. I left the Erlang community for a while. So, again, until I came back and started working for Erlang Solutions later. So, apart from...I don't know that many. I think you probably have much better contact with what was going on in those first couple of years, right?

Francesco Cesarini: I think Nortel was shipping a lot of products, which were based on...I mean, Erlang was spreading like wildfire and it started around 2005, 2006, but Nortel, many of the telcos, you know, it was a way for them to save money more than... So, they'd invest and save by investing, both for building their infrastructure and also saving in terms of increased productivity.

Robert Virding: Well, it was, still is a very good language for building that type of system, right? And I also, I think about then it also started spreading to other types of applications, not just telecom.

Francesco Cesarini: That's correct. I remember working my first banking system in 2004, you know, and that was a real experience. Databases, NoSQL databases at one point at the hype of NoSQL, I think 7 of the 10 top key-value stores were all implemented in Erlang.

Robert Virding: All that type of stuff surprised me because as I mentioned, we were very focused on developing a language and a system for looking at telecom applications. And hearing it was being used for other things came as a surprise. I mean, the first time I heard that Erlang was used for a web server, I thought, "Why would anyone want to write a web server in Erlang?" But when you think about it, it's quite natural because the type of problems you run into, and the type of things you want to solve are very close to each other actually, so it's not strange.

Francesco Cesarini: I was gonna mention web servers, and the fact is we had probably the fastest and most powerful web servers if you look at the benchmarks from 2002 to 2003. But no one ever thought of packaging it as a standalone web server, and shipping it as you'd ship Apache or many of the other web servers.

Robert Virding: Well, we were very bad at selling it, so sort of selling the product. We weren't very good at doing that. Others came along later that could do that much better.

Francesco Cesarini: Mnesia was another, I mean, it was the first distributed database that I came across in 1995 and it was..., if you didn't use it for transactions, which we weren't aware of the cap fair back then, but if you didn't use it for transaction across distributed systems, it was incredibly powerful. It is a distributed key-value store, which didn't necessarily provide strong consistency of your data, but that was taken care of in your business logic, so it was an amazing cache.

Robert Virding: Again, that's getting back to things we hadn't expected that used the type of properties that Erlang and the system around it had that provided for it. And it's been spread to a lot of different things as well.

From a lab language to a language ecosystem

Francesco Cesarini: Again, the question is, how did one pitch to the manager? How did anyone get the budget to go in and write your database? Today, I think, I doubt many companies would get that budget. You know, there's the database you can just go out and use, but back then, I mean, it must have been huge.

Robert Virding: From what I can recall, it was needed in some of the products, the internal products, they needed a database for doing these types of things, and they needed the distributed fault-tolerant database. So, they were paying quite...in that sense, became quite natural just to implement it in Erlang.

Francesco Cesarini: There was nothing on the market back then.

Robert Virding: No.

Francesco Cesarini: There was absolutely nothing when it came to distribution. I remember demoing it in the '90s and people were just like, you'd write in one node and the data would appear on another node, and people thought it was black magic. They loved it.

Robert Virding: Well, it's not black magic, it was just how...

Francesco Cesarini: I know. But they'd never seen anything like that and it was just like, "Oh, wow, this is super cool."

Robert Virding: I think, well, from my point of view, the fun part is that it was built on the basic concepts inside Erlang. They weren't developed for doing Mnesia, Mnesia could use them in a good way.

Francesco Cesarini: Then the only API you could have at the time was through narrowing API...

Robert Virding: Yes.

Francesco Cesarini: ...to do that core. But yeah, it's not something we speak about these days.

Robert Virding: In that sense, it was restricted, yes, to Erlang Systems.

Francesco Cesarini: When did you realize...So Erlang is a programming language. I think the first... was running on a virtual machine called the BEAM, initially the JVM which then became the BEAM. The first programming language I think I became aware of which was in Erlang on the BEAM was Raya, which was a Ruby-inspired Erlang which goes to show it wasn't a bad idea if we look at what happened 10 years later. We then came across Efene, which was a C-inspired language written by Mariano Guerra. You wrote more or less at the same time, so 2008, 2009, Lisp Flavored Erlang. I think that was the first...

Robert Virding: That was me because as a functional programmer, I started with Lisp and I've always been fascinated by Lisp.

Francesco Cesarini: So, these are just three or four languages we mentioned in 2008, 2009, and when did you realize that this programming language you built at the lab had become an ecosystem of languages, you know, together with tooling, virtual machines, frameworks, and different languages all suited to solve different problems?

Robert Virding: I think that was a bit later when Elixir started appearing on it. That was a major change in that direction or going towards an ecosystem because that came, I think, in 2011, 2012 it was officially for it. That became a very significant language used on the ecosystem, and it originally came from a lot of influence from the Ruby on Rails environment for building web servers, they expected things like tools and development systems and things like this just to exist, right?

Francesco Cesarini: Well, and they had a completely different development approach or what you would have in kind of server-side backend systems, which was top-down, we were used to bottom-up because, you'd like to solve the complex problems first because if you fail to solve the hard problems, you'd fall in the project, so there's no point in gluing everything together. They were doing UI/UX, you needed the top-down approach because you need to see what you are starting with.

Robert Virding: Definitely. They had different targets and different requirements as well. I mean, a lot of the systems we've been developing, we expected them to run for years, and if it took two months to develop the system, that wasn't much...that wasn't very bad at all. That was quite good. But if you're coming from the other side, you might want to get your system up in a week, right? Then you just have to have a different set of tools that you can use to build these types of things for them. That's why one of the things that came with Elixir was these tools, was these build systems and things like this, which came in from a very early time and helped the language spread a lot, which it's still doing.

Frameworks in the BEAM

Francesco Cesarini: What I like about Elixir is also the frameworks that come with it where you're productive just by knowing a subset of Elixir. Have you looked at any of these frameworks?

Robert Virding: A bit.

Francesco Cesarini: And I'm not just talking Elixir, I mean, we've got frameworks in the Erlang world as well and I'm sure, you know, whatever new languages come along.

Robert Virding: Several new languages are coming along on top of the Erlang ecosystem, and I would be very surprised if they didn't come with frameworks because the language themself might be very interesting like the type languages coming, but without the framework, I think they wouldn't be that interesting for people to actuality? Not seriously.

Francesco Cesarini: One of the questions that I always ask programming language inventors is what problem are you trying to solve? And I love Jose Valim's answer to that question. For those of you who don't know Jose Valim, you know, is the creator of Elixir and he said, you know, "I wanted to bring the power of Erlang to a wide range of communities, starting with the web but not stopping at the web." So, he's created Phoenix, the Phoenix framework, which is kind of the new Ruby on Rails. So there's a huge influx of UI/UX, Ruby and web developers coming into the community. In parallel, I think what happened is they took the work we were doing on Erlang embedded and started building on it, and I think we were out too early with embedded frameworks. I don't think the world was ready when we were working with it. And yet we had a lot of users, mainly hobbyists, but with Nerves, it's finally making it out into production and the real world, and that's fantastic and brilliant.

Robert Virding: But he had the same idea. He was looking at a specific type of problem, how can we use Erlang as the base for this? Well, he'd built a new language on top with specific properties. You have the tools, you have the applications and packages that were there, so people would just use them very quickly and get going. Again, just very problem-oriented in that sense. As you mentioned, Phoenix is for building web servers, and Ecto is for doing the database backend for them. They were just, they came very early on in the system

Francesco Cesarini: I'm learning all of the components, which are slowly coming out. It will take a while, but which around machine learning, and bit by bit, brick by brick. If they're able to reduce the barrier to entry and accessibility of machine learning in the same way they did it for web development, again, that's gonna be another huge influx of…

Robert Virding: I'm starting to look a little bit at those systems and they seem quite interesting, quite straightforward in many ways as well too, to using them. Then we'll see how it goes.

Recommended talk: Concurrency Oriented Programming in a Modern World • Robert Virding & Francesco Cesarini • GOTO 2023

The future of the BEAM languages

Francesco Cesarini: Where do you think the future will take us?

Robert Virding: I don't know. I think one thing is going to happen, there are going to be more languages in the ecosystem.

Francesco Cesarini: Well, there's something like 35, 36 at the least.

Robert Virding: There's a lot there, but many aren't used, of course, but I think it's just the fact that Elixir came in and is being extensively used shows that there's a lot there we can build on, or that can be built on. And so what's coming now, for example, are statically typed languages because the original Erlang language and Elixir on top of it are dynamically typed, and people require static typing, and very much would like it, so they're coming several statically typed languages built on top of the Erlang ecosystem.

Francesco Cesarini: I think the first statically typed language that kind of came on my radar was Alpaca.

Robert Virding: That was one.

Francesco Cesarini: But then, you know, Fez and I think the one which is gaining the most traction of them all is Gleam. 

Robert Virding: It's become a big requirement for systems, for what people want for building their systems and...

Francesco Cesarini: It's security, as I see it. With the BEAM Erlang and Elixir and other languages making their way to devices, some of these devices have to be secure. And we're seeing right now, Kry10, which is a seL4, kind of, proven operating system, the kernel of the operating system has been proven to be safe and secure. But as soon as you've got a dynamically typed...and the BEAM is a first-class citizen. Security is provided by using concepts of, kind of, isolation, and immutability. But as soon as you've got a dynamically typed language, you're opening yourself up for a whole new can of worms.

Robert Virding: That I know. Well, unfortunately, depending on how you wanna see it, one of the requirements we had when we were initially developing the language in the first 10 years it was in use required, or rather could not handle static typing because we wanted to be able to upgrade the system while it was running new code and things like that. And to do that properly, you would require dynamic typing.

Francesco Cesarini: Yes.. Because you don't link modules together and...

Robert Virding: But now, well, that's changed quite a lot now, and the static typing requirements are quite strong now, I think. It will come. There's no doubt about that. I think it'll come. The languages, the latest one...well, the latest one I know about is Gleam, which is being seriously developed, and I think that will start being used or there might come something else. I don't know, but I think that that is something that is being attacked today. And when that comes, that will open up Erlang, the Erlang ecosystem to more uses for it, because I think once the static typing comes in, they'll probably migrate it to other systems as well, do things for it. So yeah, it's a big change, but I think the system is dynamic and versatile enough to be able to handle things like this. It's not static in that sense, right?

Francesco Cesarini: So, yes. To the next 25 years then.

Robert Virding: To the next 25 years, yes, hopefully. We'll see what happens with it, and how it goes. And it is continually growing. The number of companies using it and the number of systems that have written in Erlang, languages in the Erlang ecosystem being extensively used, which people just don't realize is Erlang at the bottom.

Francesco Cesarini: You know, Joe Armstrong in '96 told me, "Oh, Erlang won't be around forever. Something new, something better will come along." And I said, "Okay," you know, and never would I believed that over 25 years on, I'd still be working with it.

Robert Virding: Well, I think it was dynamic. It was simple enough in the right way so you could use it to build It was very versatile for building things on top so it made it very adaptable. So as we've talked about, a lot of new ideas have come up, but they've been able to implement it on top of the Erlang ecosystem, and at the same time use the benefits of the Erlang ecosystem and add new features to it, which have been required.

Francesco Cesarini: And just to wrap up, what would you say to those claiming, "Oh, it's a 25-year-old technology?"

Robert Virding: It was started 25 years ago, but it has continually evolved. Even the Erlang language, which looks like very much it did before, new things come in, and we've talked about new languages, and new features. The virtual machine, the BEAM is continually evolving, that is, every new version that comes along is much better than the previous versions. Not saying the old versions were bad, but just new features come in. And again, they're backward compatible as well, which is a requirement we've also had and things like this. And they'll just keep going.

Francesco Cesarini: All of the changes are happening under the hood, and I think one of the major changes in the recent year is the JIT compiler, just in-time compiler, for example.

Robert Virding: That was a big change, yes.

Francesco Cesarini: You know, when WhatsApp went in and upgraded their servers, they were able to reduce the server needs by 30%, just thanks to this one optimization. And those are the types of things that are happening under the hood, so languages exist and new languages on top of it even if they're not evolving, you can take advantage of it.

Robert Virding: No way is the system static at any level really for it, and so work's going on everywhere for it. New packages are coming in, new languages, new implementations in the base, and things like this for it. And it just does a lot of things that you want for you without having to worry about it, like concurrency and parallelism and it's just there, and I don't have to think about it. I know the system does it, I'm happy with that. It just works. And it works. That's the most important thing, it just works right for it.

Francesco Cesarini: Brilliant. Well, we'll have to meet up in 10 years and see...

Robert Virding: Most definitely. Most definitely.

Francesco Cesarini: ...if our predictions are correct, and see what happens.

Robert Virding: I think it'll still be there, but it won't be the same.

Francesco Cesarini: Everything changes.

Robert Virding: Everything changes.

Francesco Cesarini: You have to adapt to change.

Robert Virding: It'll just get better. Just get more and get better, I think. Most definitely. Okay.

Francesco Cesarini: Thank you, Robert.

Robert Virding: Thank you, Francesco.