This text was generated using AI and might contain mistakes. Found a mistake? Edit at GitHub

Francesco Cesarini: Evolution of the Erlang Eco System

Hi, I am Alaa Reuschenbach.

Do you organize any user groups, conferences or other tech events?

Then feel free to add them to treff.tech, an uncommercial platform for tech events in the German-speaking community.

It’s free, without any advertising or tracking.

Just visit treff.tech or scan the QR code.

You will also find that link in the video description.

And by the way, you can find all Software-Architecture stream events also on treff.tech.

And if you have any questions, feel free to reach out to me.

Okay, so welcome to another episode of Software-Architecture im Stream.

This time with Francesco Cesarini.

And we will talk about the Erlang ecosystem.

So first of all, Francesco, welcome to the show.

Thanks for doing this with me.

Can you say a few words about yourself?

So hi, hi to everyone.

Thank you so much for having me.

So my name is Francesco, half Italian, half Swedish.

I’ve been working with the Erlang ecosystem since its early days.

So lucky enough from seeing a programming language become an ecosystem of languages.

I started at Ericsson’s training consulting arm, working with teams, bringing Erlang to fix the mobile broadband products.

And in 99, founded Erlang Solutions, helping companies adopt the lessons learned from telecoms in other verticals.

So be it digital health, messaging, fintech, and others.

And it’s a company I grew to about 160 people.

We have offices in Europe and the US.

And we’ve worked with startups such as WhatsApp, you know, from the days they didn’t even have an office to enterprise giants like MasterCard.

I wrote two of the Erlang O’Reilly books, lecture concurrency, concurrent programming at Oxford University.

And more recently, I’m the co-host of the Beam.

They’ve done that podcast.

Yeah.

And we will have links to the podcast.

So if you want to dive deeper into the Erlang ecosystem, you can definitely do that.

Sort of, I mean, there are two things that I want to highlight.

So first of all, you sort of already gave it away.

So Erlang is related to Ericsson.

It’s not related to the city of Erlang in Germany, which some people believe.

And the other thing that I find interesting is, well, do you want to talk about the role that Erlang played in at least the early days of WhatsApp?

Because I think that story is quite fascinating.

So, yeah, I mean, back in the early days, if you look at WhatsApp, they picked up an open source messaging solution called eJabberD.

And they started modifying it.

You know, they had one focus.

They basically, you know, Jan and Brian had no external investors.

They would be strapping it themselves.

And they had one goal, to reduce the overhead per user and try to get as many users on a single machine.

You know, getting as many users on a single machine reduces your hardware costs, but it also reduces your operational costs.

And so that’s what the focus was.

And, you know, indeed, I think 2011, they had one million TCP IP connections on a single machine.

Come 2012, early 2012, they published a blog post announcing that they had hit, you know, two million users.

And they’d done that by, you know, changing FreeBSD and changing, you know, and optimizing the beam.

So the prevalent Erlang virtual machine being used at the time.

And so that’s the impact, you know, they had on the community.

The changes they did, the changes they did were made it mainstream.

So, you know, three years on, four years on, you were able to get your two million TCP IP connections on an AWS instance.

So the changes to FreeBSD made it into mainstream operating systems and the changes they did to the beam made it to the beam.

And so they had Phoenix, which I think we’ll probably talk about a little bit later, which is a modern day Ruby on Rails.

Within Elixir, you know, one of the other languages in the Erlang ecosystem.

They were able to get two million WebSockets simultaneously open on one single machine.

And so this is what allowed them to really reduce the costs.

And I think when they got acquired in 2014, the server side team.

So that was the team who wrote all the new features and all the new features, who managed all of the existing features.

And they were also the ones who managed operations, who were woken up in the middle of the night when there was an outage consistent of about 10 people.

So, you know, it was a really small core team.

And at the time, they had 400 million users and they were adding a million users a day.

So that kind of speaks to the power of, you know, having clever people using the right tools and doing things right.

Yeah.

And I still find that story very impressive with these, as you said, 400 million users and just 10 people working on that.

Because, I mean, the sort of average team that I see is already larger and they are certainly not having these kinds of numbers.

So when we originally discussed this episode, you suggested that we shouldn’t talk about Erlang, but rather the ecosystem.

Why?

Well, I mean, Erlang is a programming language, but it’s one of the many programming languages which today make part of the whole ecosystem.

And, you know, if you look at it, you know, you can have an analogy with the Java ecosystem or the .NET ecosystem.

The Java ecosystem includes Scala, Clojure, Groovy, Kotlin, JRuby, and many more.

And the same with .NET, which will include C Sharp, F Sharp, Visual Basic, PowerShell.

I would estimate that there are over 40 languages which today run on the Beam virtual machine.

And all of these languages will inherit, you know, the Erlang semantics and properties of the Beam.

So you go back to 2008, 2009, the first ones were Rayya, which was Erlang with a Ruby-like syntax, and Efenna, which had a C-like, which was inspired by C.

The most popular ones today, which I’m aware have been used in production, include Luerl, which is a whole port of Lua on the Beam.

It’s used for scripting, for test cases, you know, giving it to other developers.

Elixir, which is probably going to become 10 times larger than Erlang, if it’s not already 10 times larger than Erlang, which was created by José Balin to bring the power of Erlang to a wider range of communities.

I think we were very much focused on high-scale distributed systems.

We thought, OK, how do we bring it to the web?

How do we bring it to the embedded space?

Gleam is a new and upcoming language by Louis Piliford, bringing static typing to the Beam.

So these are the ones which are really beginning to take off.

And just to give you an idea, new languages which I came across recently, and I’m talking the last couple of months, include Bondi, which is focusing on multi-agent, multi-party systems in the IoT space.

So there’s a lot happening in that space.

And Cure, which has been around for a while, but it just came on my radar, a dependency-type language targeting the Beam.

And the creators claim that in the next release, it will have the same functional parity as with Idris.

And these are languages which can use each other’s tools, package managers, libraries, and at the same time rely on all of the optimizations which Ericsson is doing on the Beam virtual machine, the compiler and the tool chain as well.

So it’s not the single language which is the right tool for the job.

It’s pick the right language, which is basically what’s helping expand the whole user base.

Because I have to admit, when you look at it from the outside, Erlang is a little bit special in the syntax, I would say.

I’m not sure what you think about it, but it’s unlike other languages, I think.

The syntax is there because the first virtual machine was written in Prolog, and that heavily influenced it.

And that’s, I think, one of the reasons why I guess it’s not as popular today as it should be.

But if you’re wearing your computer science hat, syntax is something you get over in a day or two and don’t think about it anymore because it’s just something your language inventors created.

If you don’t have a computer science hat, if you weren’t exposed to half a dozen languages in your first two to three years in university, then it becomes much harder.

And that’s understandable.

And this is where Elixir excels.

It’s got a Ruby-like syntax.

Users get lulled into this false sense of familiarity.

They feel, oh, this looks familiar, not realizing that it’s a completely different language from Ruby and behaves in a completely different way.

But at least the first obstacle, the first hurdle is crossed because there is no need to get used to a new syntax.

And that’s something that Jose-Berlin did really, really well.

And it’s something that we in the Erlang world didn’t quite get.

So maybe a possible homework would be to take a look at Erlang, the language, and also at Prolog and see what you can do with it.

And I had pretty good experience with spending a weekend with Emacs Lisp and having Chuchibiti talk me through how to get started with it.

So maybe that’s an option.

Okay, so, oh, interesting.

So Fabauti said that he learned Erlang in an ISLQB training.

So that’s the International Software Architecture Qualification Board.

And he has to admit that it was quite challenging coming from Java.

And, yeah, maybe that’s the point, right, with the curly braces and that you’re used to with the C languages.

And then Erlang just looks very different.

I think the real challenging thing more than it looking different is that, you know, a programming language worth learning is a programming language which changes the way you think and reason about programming.

If you’re using Java, you need to think in terms of OO, inheritance, objects, you know, hierarchies, in terms of threads, mutexes, semaphores, shared memory.

And the hardest part about learning Erlang, which is unlearning everything else you’ve learned about other programming languages, that’s the hardest part because it’s a different approach.

It’s a different way of structuring your programs, and it’s a different way of thinking.

It’s a different way of handling your errors.

And when you start thinking of it that way, you actually stop overengineering because that’s what we see a lot of C programmers do, for example.

They think in terms of performance.

No, let the VM handle the performance.

Just make sure you don’t have bottlenecks in your system.

Stop trying to be the fastest program out there.

You need to be fast enough.

It doesn’t matter if your request takes 200 milliseconds or 400 milliseconds.

What’s important is that it doesn’t take more than 400 milliseconds when you’ve got a spike of a million users hitting your system at the same time.

Those are the things you need to think about.

And that’s, I think, the hard part right there.

So it’s unlearning more than learning.

Yeah, I think it’s a good point that you’re making.

It’s the difference between performance and scalability.

And oftentimes, scalability is really the issue, the problem that if you have a lot of users, as you said, that you want to have the system scale, so add more resources to the system and then hit the right performance.

So you said that – sorry.

What you just said is really important because that’s the difference between the JVM and the Beam.

The JVM was built for speed.

If you look at floating point operations, when Airlang was ported to the JVM, floating point operations in Airlang on the JVM were 2,000 times faster than on the Beam.

But at the same time, try creating 2 million threads on the JVM.

It won’t work.

You’ll create 2 million processes on the Beam in no time without any issues at all.

And that’s what we mean by scale.

You can handle your peaks of Discord messages coming in or your tweets or your WhatsApp messages and handle them in a very predictable way, optimally using your resources versus the JVM, which will be incredibly fast at managing a single request on a thread.

But you’re going to struggle when you start hitting scale.

You just said the sort of magic word, process, because that’s the alternative to lemmaforce and the concurrency principles in the other languages.

So can you explain what a process is and how it’s important?

So the process is the main building block of your Airlang programs, and it’s a concept.

I don’t think they realized it, but it’s actually a concept they took from Smalltalk, which I think was an object-oriented language.

In Smalltalk, you’ve got objects.

Objects don’t share memory, and they communicate with each other through message passing.

And if you look at processes, processes, it’s your main building block.

They do not share memory in Airlang, and they communicate with each other through message passing.

And to quote Alan Kay in his definition of object-oriented programming, Smalltalk said, this we’ve inherited.

So that all came after.

That came when C++ came about.

And when processes don’t share memory, you’ve got the property of immutability.

You’ve got the ability.

You avoid, I think, a lot of the issues you have with, say, a model based on threads where they do share memory.

So that means that when threads need to access a critical section, they put in a lock.

Your biggest problem to scaling a multi-core architecture is memory lock contention.

And this is solved at the cost of copying the data from one process to another.

So this is the key part.

And message passing is asynchronous, so it’s non-blocking.

So you can actually send a message and then continue executing.

And it’s asynchronous message passing, which you need for speed.

And so as soon as you’ve got processes which don’t share memory, you’ve got distribution built into the language semantic.

Distribution built in.

Because what you can do is you can place processes on separate machines.

And you can basically place processes on separate machines.

And when you do that, you have distribution.

And Erlang actually has distribution built into the semantics of the language.

And so you can send a message from one process to another, not knowing if they’re on a different machine or not.

It could be on the same virtual machine or on a separate virtual machine or elsewhere.

And so location becomes transparent.

Now that you have distribution, you automatically get the scalability replication.

Because you get redundant instances.

So you can copy the content of the process on two separate machines.

So as to quote Joe Armstrong, one machine might be struck by lightning.

That was his, Armstrong is one of the co-inventors of Erlang.

And so all of a sudden you’ve got a copy of the data in another process and it works.

And that gives you full tolerance.

And you can also go in and spread your workload across nodes.

And as you spread your workload across nodes, all of a sudden you’ve got two machines.

You double your computing capacity.

But you do that with no changes to your actual program.

Because again, location is transparent.

You don’t know that you actually have two separate machines.

Because you address the processes in a cluster of nodes.

And so the message might end up on a network, it might not.

And the beauty of this concurrency model, I mean these are the side decisions they made very, very early on.

And they predate multicore.

But when you have distribution and multicore architectures, by default you also get parallelism.

And parallelism in the early days was we had two nodes on the same machine.

Which then communicated to each other by I.O.

But now, since 2006, we have each VM now will run on multicore architectures.

You will have a scheduler for each core.

And then you place cores on each, you place processes.

You share them evenly across the different schedulers.

So automatically you get parallelism.

And you get it.

As I mentioned earlier, the biggest problem to scaling on multicore architectures is memory lock contention.

Well, there is no memory lock contention here.

Because there is no shared memory.

So assuming you’ve got no bottlenecks, a program which was made to run on a quad core machine.

If you run it on a 16 core machine, it will automatically run four times faster without having to fine tune it or recompile it.

And you think these are very separate problems.

But in fact, they’re all connected.

Because for distribution, you get scalability, you get reliability.

But you also get parallelism.

And I don’t know if we’re interested in the history of computer science.

But I think 2018, I managed to get Tony Hoare.

So the creator of CSP, so communicating sequential processes.

He also did Hoare logic and invented Quicksort.

We got Carl Hewitt, so the creator of the actor model.

And John Strong, the co-creator of Ericsson concurrency in the same room in a panel.

And these are the three people who, you know. who set the foundation to modern concurrent programming and influenced it the most.

And I asked each and one of them, you know, what problem were you trying to solve?

And the answers just blew me away because, you know, Tony Hoare, oh, I was trying to figure out how to program transputers, you know, transputers, you know, the predecessor of multicore.

Carl Jude, oh, I was trying to figure out how to program distributed systems.

And Joe Armstrong, oh, I was trying to figure out how to build full tolerance systems.

So these are, yeah, and they all came up with a very, very similar solution.

Subtle differences, but the solution was very, very similar.

And so three visionaries who, on the surface, appeared to be working on, you know, very different problems, but actually, no.

They were all working on problems which were very, very tightly connected, all based on basically a concurrency model with notion memory, which gives you distribution, distribution gives you scalability, reliability, and parallelism, you know.

We realized today they were connected.

They definitely didn’t back then.

Yeah, which sort of makes you wonder why not everybody’s using this these days.

And also there is the comment by Christian Trutz, and he said a super fast Erlang tutorial.

So based on what you just said, how does, so is there garbage collection on Erlang in Erlang systems or in the Erlang VM or the Beam, I should say?

Yes, so the garbage collector is highly optimized to not have any impacts on your software and properties of the language and of the system.

So it’s a generational garbage collector which gets triggered on a per process basis.

So there is no stop the world garbage collector.

And, you know, that means that, you know, when a process has run out of memory and needs more memory, the garbage collector is triggered in, and it starts going through its stack and its heap, freeing up memory, you know, which is no longer used and stops every time, as soon as it has freed up enough memory.

So it’s very small, very short bursts of the garbage collector.

Only once every, I think, you know, and again, this is not documented for a reason.

You need to go in and read the source code, but I think once every few thousand garbage collections, it does what we call a full sweep.

And so, and frees up all of the memory, just to make sure that there are no memory leaks and reduces the risk of fragmentation.

And by doing it this way, you know, you get, you know, almost, you know, you get a really, really good guarantee of your throughput.

And you don’t have, you know, the one percentile of users who, you know, happen to be running the request right when the stop the world garbage collection tickers in.

So it’s a big, big difference from say the JVM and the .NET.

And it’s possible.

You can’t do this on the JVM or the .NET because of shared memory.

You’ve got a very, very complex, you know, the garbage collector.

I’ve met the person who, yeah, who wrote the garbage collector for the JVM.

He works on the Go team today, incredibly bright.

And yeah, and he’d read all of the papers on the airline garbage collector, knew them inside out and said, oh, I only wish, you know, memory management was this simple in the JVM.

It’s not, it’s, you know, shared memory is complicated.

And, you know, versus, yeah, versus the simple approach, you know, they’ve taken on the beam.

The other thing, I mean, we sort of, we didn’t really mention it explicitly or you didn’t really mention it explicitly, but Erlang is obviously from a high availability background.

So how is high availability done in Erlang?

So basically ensuring that there’s no single point of failure, you know, you need two of everything.

You need, you know, and that’s a simplistic view, but you need two networks.

You need, you know, you need two redundant power supplies.

You need two virtual machines.

And, you know, and making sure that, you know, if anything along the line fails, you’re able to step in and, you know, and there’s an alternative branch, which you can step in and kick in.

And this comes from the requirements you had on telecom networks.

By law, you know, telecom networks could not fail, be it the software or the hardware.

It was, you know, even, you know, during an emergency, even during an earthquake, you had to be able to pick up your phone and dial the emergency services.

And, you know, back in the days, you know, if your system wasn’t working, if your phone system wasn’t working, it would make the front pages of the newspapers because it was, yeah, there were huge penalties associated with it.

And often your legal repercussions would happen.

And so that meant, yeah, you wrote systems, you know, you wrote systems which didn’t fail.

And this is exactly the problem, you know, when they set out to invent Airline, they were trying to solve, you know, how do we make sure that our system doesn’t fail when everyone in New Year’s Eve picks up the phone and calls each other and wishes each other Happy New Year?

How do we make sure that when we go in and dig up a cable, you know, we can still switch phone calls?

You know, you know, what happens if, again, you know, the example we use comes from Gerard Strong, you know, if a computer gets struck by lightning, you know, do we have a redundant computer to take over?

And that’s how we achieve high availability.

And through the concurrency model, we can, it removes a lot of the pain of actually copying the state.

You just have a replica of the state on a separate machine ready to kick in as soon as something happens.

Yeah, and that’s basically what you said, where we have the processes and they have, they don’t share any memory.

So you would just do message passing and then you could have a replica of each process.

I have to admit that I was, how should I put it, a little bit surprised because one of the things that, when I took a look at Erlang, I mean, from just very, not in detail, one of the things that comes to mind is that this let it crash idea.

And I thought that was the answer to high availability.

So what is let it crash?

And how is it related to high availability?

Why didn’t you mention that one as the main point?

That’s a good question.

So when we say let it crash, we mean, what we do is we’ve got a concurrency model, which allows you to isolate failure.

And software will have bugs, and bugs are unpredictable.

If you knew they would happen, you would guard yourself against them.

So what is it we did?

So the approach you take, and I think this goes back to the earlier question, I think this is one of the hard things one needs to start thinking of is you don’t encourage defensive programming.

What you do is instead you try to isolate your error, your fault in a process.

So if you come across something unexpected, and it could be a bug in your code, or it could be corrupt data, let the process executing that code crash.

And we’re not saying the system, the whole system crashes.

It’s just one single process.

So, I mean, Discord runs on the Beam.

You’ll have hundreds of thousands of messages and phone calls being set up, going through the system at any one point in time.

A crash in Discord means you might lose one phone call or one message, because you’ve isolated your fault to that single phone call or that single call setup or that single message, and not everything else going through that system.

And you can do that because processes don’t share memory.

So by letting the process crash, you hopefully clear the corrupt state.

And so you focus not on preventing it from crashing, but you actually focus on recovery once it has crashed.

So you go in and you create a new process.

Oh, can I pick up?

Can I set up this phone call again?

So for example, oh yeah, the user’s still online.

Let me reconnect them, for example.

So you focus very much on recovery.

And on the recovery, hopefully you get rid of that corrupt state which existed.

So it doesn’t mean you ignore errors.

It just means that you take care of them and you handle them in a different way than what you might be used to.

And this is how they’ve managed to achieve downtimes of, and this includes software upgrades, in major systems of a few minutes and sometimes even a few seconds per year.

I mean, there’ve been cases where systems have been up and running for three, four years without the need to reboot them, even in the presence of errors, because yeah, you sometimes don’t even notice those errors.

And I think that’s, to me, when I had that, you know, I had that rough look at Erlang, that was something that I found quite surprising because if you come from the sort of traditional school of programming, to me, it seems like those experienced programmers will tell you, well, you have to think about all the errors that could happen and you have to deal with them.

And then your code becomes really complex because you have to sketch all the exceptions.

And then there is this crash idea, which basically says, well, just, well, it crashes.

And then we recover.

And the point that I find interesting is that this is the way that those switches, as you said, that have really these high availability requirements, how these are structured.

So that was something that I found very interesting.

Correct.

And, you know, oh, you need to trap all these exceptions, but if you knew what these exceptions were gonna be, you know, they wouldn’t be happening.

And so, you know, and that is the big problem, you know, by being defensive, in my experience, you actually end up introducing even more bugs in your system because you’re trying to handle the unknown.

And, you know, how do you handle the unknown?

You know, it’s a mystery to me.

Yeah, yeah.

They’re called exceptions for a reason, right?

Because they are the exception.

Exactly, they should, exactly, exactly.

Okay, so the other thing that you just mentioned, and I think that’s also one of these very interesting points, if you really want to have high availability, you have the problem that you have to deploy software without taking down the software.

And as you just mentioned, there is a solution in Erlang for that.

So what’s that?

So how does that work?

What you do is, it’s in Erlang, it’s in Elixir, you know, and you pretty much, you know, can do it very easily in all of the Beam languages.

And it’s that, what you’re able to do is, at any one point in time, you’re able to upload a new version of the module.

So into the virtual machine.

And any new process which is started will automatically start running the new version of the module.

All processes running the old version of the module will continue running the old version of that module.

Until they do what we call a fully qualified function call, where we add, we prefix the module to the function name.

And at that point, a check is done.

Am I running the new version of the code?

No, you’re not.

So automatically, it goes in, and it switches to the new version of the code.

So it sounds very, very simple on the surface, and it is simple.

And it’s extremely powerful, because even just from a developer point of view, it might take a few minutes to start up your system.

You notice a bug when you’re coding.

What you do is, you fix that bug, you upload that new version of that module, you rerun the request, and you can do so without having to restart the system.

So it really, the whole developed test cycle, it really speeds it up.

And in production, it was a requirement.

Every dollar you send on, the way you used to manage it in the old days was redundant hardware.

You had a hot standby machine, which basically did nothing and waited for the first machine to fail.

But instead, using both machines, or just using both machines, would double the capacity you could handle.

Of course, you’re assuming no machine would fail.

And that would help you manage your peak loads.

And what you did is, you were able now to go in and upgrade.

You double your capacity and you were able to do an upgrade of your machine without taking down your system.

And again, that was critical to, or without even rebooting your system, which was critical with systems for high availability.

So again, this is an idea they took from Smalltalk.

I asked Dan Ingles, one of the creators of Smalltalk, why did you create software upgrade?

He says, oh, rebooting those machines took so damn long that this was just a good way for us to continue developing and not waste any time waiting for the machines to reboot.

In this particular case, they took that idea and they used it to ensure that you could still keep on switching phone calls whilst you were doing a major upgrade.

And yeah, these upgrades in complex telephone switches is not for the faint of heart, but because you need to think of backward compatibility, you need to, you know, you might need to do schema migrations.

And so, you know, it does have its cost, you know, when your system’s complex.

If it’s a website, it’s probably easy, but when you have a telco system, it’s much more complex.

But it’s been done with code bases with millions of lines of code.

And it’s been done without the users and sometimes even the customers knowing about it.

One thing that I would like to add, so I find it quite fascinating because as you said, Smalltalk has the same idea and Smalltalk introduced a virtual machine in particular to not recompile the whole system, but instead just patch the part of the code that would actually be changed by you, which means that turnaround times on Smalltalk machines in the 70s are probably faster than what we have these days in a lot of cases.

And I find that quite fascinating in a way, because you refer so much to Smalltalk.

I think it’s an interesting thought that I never had, like it’s the same idea, message passing between independent entities.

However, in Smalltalk, the entities that would exchange those messages are objects, so instances of classes.

And I would assume that those are more fine-grained than processes, am I wrong?

Or is that actually true?

To me, processes are more like microservices, like something that is- That is correct.

That is correct, that is correct, that is correct.

So, I mean, processes are always active, they’re always executing.

It’s not just when you go in and invoke them.

But it’s this separation of your logic, I think, which is critical.

And the difference is an object will execute when you invoke it.

A process will continue executing, but it will automatically suspend itself when it has no more code to execute, and it’s waiting for an incoming message.

I really recommend all of the listeners.

There is Joe Armstrong, so the creator of airline interviews, interviews Alan Kay.

He did it at a conference called CodeMesh.

And we’ll post a link, it will post a link here, so you can actually go in and have a look at it.

Alan Kay, obviously the inventor of Smalltalk, and also the Dynabook, if I remember correctly, right?

Was it- Sorry, the?

The Dynabook, was that his idea?

Yeah, that was, yeah, a laptop for every child.

Yeah, that’s correct, it was his.

I mean, yeah, he is one of the visionaries of computer science.

He set the foundations, which we work on today on a daily basis with.

Okay.

So we have a comment by Mauro Gabriel Titimoli.

I hope I said that correctly, pronounced that correctly.

It would be really interesting to hear something about the different deployment strategies, CICD and the difference with other traditional systems.

I’m not sure what to make of that, to be honest.

I don’t think so.

I mean, airline’s been around for such a long time.

And I mean, the most common deployment strategies today are based on Docker and Kubernetes.

And I say unfortunately, because there’s a huge overlap with what the Beam already does, and what Docker does, and what Kubernetes does.

So, and it’s looking at the monitoring, looking at the isolation, and all of that.

And so it’s almost like, people don’t do software upgrades during runtime anymore, because they just take down the Docker container and deploy a new one.

And it’s something you need to do because, well, of the mutability you’ve got in the containers.

And that is, in a way, it’s a pity, but oh, it’s fine.

If there are 500 requests going through a Docker container at that point in time, it doesn’t really matter.

They can press refresh and they’ll just blame it on the network.

And that’s, I come from a world where you need to serve every request and try to optimize the whole customer experience.

My favorite deployment approach is on the bare metal, basically, running the virtual machine straight on the operating system.

And it’s, and that’s, the less layers you have between the hardware, between the chips and your virtual machine, the less memory you use and the faster your system runs.

And we’re really trying to get as much computing capacity as possible from your machine to reduce your AWS bills and to reduce the impact on the environment, if you want, but also, yeah, also to be as fast as possible.

And that’s where, yeah, that’s where it comes in.

And if you think of it, my vision for the future is a project we worked on over a decade ago, which is no longer being developed because the world wasn’t really ready for it.

We set out to build an entire network as an airline application.

So the switch firmware, the controller, the whole stack was completely automated.

So it would go in and deploy and optimize based on latency and throughput all across the different nodes.

In a way, the developer wasn’t even aware of it.

So you could basically spin off an instance and it would spin it off close to the data because it would have figured out that moving the data would take too long, would be too expensive to the compute.

So it would move the compute to the data.

And, you know, it was, yeah, the world wasn’t ready for it.

And, you know, and that’s usually when I talk about, you know, taking two steps forward, one step back, that’s usually what I mean right there.

Yeah.

And Mario Gabriel Tittimodi just added that overlapping with Docker communities is the one that I was interested to hear and the hot code reload things.

And I think that’s- For Mauro, yeah.

I mean, I would really recommend and we’ll post a link.

I took, I had the keynote at the Reactive Summit together with one of the key maintainers of Kubernetes.

So the person who managed all of the releases and our, you know, our conclusion right there was in the future, you know, there will be a Kubernetes and it will be there and running, but the ideal scenario is that the developer is not aware of it.

And that was, you know, the conclusion we came to a few years ago.

I just want to stress a point that I think you just made, which is, okay, I just have the bare metal.

I have Linux on top of that bare metal, for example, then I have the Beam on it, and then I’m done.

So I don’t have virtualization, I don’t have Kubernetes.

So it appears to be a much simpler stack, and you get rid of quite a lot of things.

And that’s also, I think, what Mauro just added.

These hot reloads, updates, this is something that would otherwise probably require Kubernetes, as you mentioned.

So you actually get a lot of, a lot less complexity.

There is one point, I mean, for like, I don’t know, 20 minutes at least, I’m wondering if you listen to these kinds of technological advantages that the system has.

The question becomes, why is not everyone using that?

And before we did this episode, Ray Garcia on LinkedIn said, one question I have to the session is a bit of the challenge.

So, Francesco just dropped off for some reason, and well, I can still read the question.

There we go.

This is what you get when you have no single points of failure.

You’ve gone from fiber over to 5G, but it had to be done manually, unfortunately.

Yeah.

So, I was just referring to the question that Ray Garcia was asking before we did the episode.

So he said, one question I have to the session is a bit of the challenge.

While Beam and its languages have an active global community that is slowly growing, and has its research followers, it remains an outlier in popularity contests with mainstream corporate buyers.

And I would like to add, I mean, we had this discussion, it’s like 35 years old, the technology, and it has been used in large production environments, so it’s really mature and well-established.

So and he goes on and says, this is even when we can find clear examples in big tech, where it has been adopted for specific high performance and stress-oriented components.

Is it a naming issue, the history, the lack of marketing, education, the forces that move entire developers like job opportunities?

What is that seems to resist that explosive adoption moment?

Which is basically the question, why is not everyone using Erlang and its ecosystem?

I think, you know, and Ray raises a very, very good point.

I mean, it starts off with the lack of marketing.

You know, when Erlang was, you know, probably the first programming language released as open source, the year the term open source was coined.

But at the time, open source meant, here’s the source code, don’t bug us.

The only public announcement, there was no press release, the only public announcement was one of the developers posted on Slashdot, and that was it.

You know, so it’s something which, you know, it’s, you know, so that was one of the areas where, you know, all the marketing, you know, there was no, Ericsson was in, you know, was building telecom switches, they weren’t in the business of selling programming languages.

You can’t sell programming languages either.

They didn’t understand the value of having a large community and having open source and how much they could reduce costs, you know, and help increase adoption.

The second was the lack of people, the lack of, you know, people, you know, capable of speaking to budget holders and decision makers, you know, that held us back.

The community often took a bottom-up approach, and we knew how to speak to technical people.

We were very good at it, and the technical people got it.

I mean, the adoption by universities of Erlang, it was being taught in a lot of universities, was much faster than the adoption, you know, within the enterprise space.

And so, you know, whilst we were taking a bottom-up approach, you know, there was no strategy tackling the top-down approach, you know, so, you know, I think we need more people like Ray who can speak to stakeholders in a language that stakeholders understand, you know.

I’ve also seen a big problem is also, you know, technical decisions not being made based on technical facts, but often by non-technical decision makers, you know.

That is where, you know, a pretty giveaway or a nice, colorful brochure or a sexy website, you know, makes all of the difference, and, you know, the real problem there hasn’t been kind of the lack of Erlang or Ericsson programmers or their career paths.

Long before Erlang was recently open source, we were, you know, that was my first job at Ericsson.

We were setting up teams with 100 plus Erlang developers, you know, and, you know, what it is is, you know, good developers will pick up both Erlang and Elixir in no time.

You know, what matters is the experience in the verticals they’re in.

So, at Ericsson, we got people who knew telecom, and in five days, we taught them Erlang, you know, and you can get people productive in Elixir in even less than that.

And so, you know, and it’s the same analogy I think we’ve got today around AI, you know, you need senior developers and an approach which allows you to be in control, you know, you start with design decisions, test cases, and then iterate on the code base, but you need someone, you know, who knows what they’re doing, and then, you know, the technology comes second.

I have to admit, I mean, if I may, I would like to add, I would argue that Erlang is in no way special.

I mean, we have been discussing Smalltalk in this episode for quite a few times, and I would argue that the popular OO programming languages that we have now that are considered mainstream are actually a step back from Smalltalk, certainly for Java, certainly for C++, very, very, I mean, obviously for JavaScript.

I mean, JavaScript has been developed in, what was it, like a week?

Yeah, I agree.

So I would actually argue that this is something that’s a phenomenon that we have in the industry, that the popular technologies are probably most often the ones that are not really that great, and then there are some spots, like we mentioned, WhatsApp, for example, where one team that bets on that technology is able to really do something that is really, really impressive and that no one else can, but that’s not what most of the industry does.

For impressive projects, you need three things.

You need good tools, good people, and cleverness.

And so Erlang is just one of them, you know, it’s just one of these tools.

It’s not, yeah, you need good people, and that’s what WhatsApp had, and you need cleverness, which showcases the fact that a team of 10 were able to manage all of the server side things.

And I would argue that you need the freedom to use those tools, because as you mentioned, management often interferes with those decisions.

And I would also argue niche technologies attract probably better people because they really want, they might be eager to do something like using Erlang.

I think it attracts, I don’t want to say better people, it attracts people who understand what using the right tool for the job means rather than the tool at hand.

And so, yeah, it’s, and that’s a big difference.

And, you know, going back to Java, I mean, Java is very successful thanks to excellent marketing.

You know, I mean, they rode on the antitrust waves against Microsoft back in the days, and that’s how, you know, but by, you know, good marketing managed to become very popular, you know, and Oracle is making a lot of money off of Java today.

You know, Ericsson is not making any money off Erlang, and it’s, you know, they’re getting a lot of help, you know, from the open source community, fixing bugs, finding vulnerabilities, et cetera.

That’s huge.

But, you know, money wise, very, very little.

Yeah, thanks for pointing out that it’s not like better people, it’s just different people and it’s always a trade off.

And yeah, so that’s a good point.

So there is one, no, sorry.

So there is a question by Ramiro Mateo Uda, and he says, it would be great to hear about the ETS, the power of it, and how you can avoid external dependencies like Redis in most cases.

What’s ETS?

So let’s step back a little bit before, yeah, so ETS is Erlang Term Storage, and it’s an in-memory key value store, which runs in the same memory space as your business logic, as your code.

So what you do is you use it to cache local data you need, you know, when you’re running.

So you could cache information on a phone call of an instant message which still hasn’t been persisted and pushed out to all of its destinations or whatnot.

Now, so we actually call it the Redis of the beam.

There’s one big difference.

It runs in the beam.

So that means there’s no encoding or decoding.

So you’re not reaching, there’s no IO to access the data, and that means in sub microseconds, you can actually extract that data and start using it versus having to, you know, JSON encode the request, send the request, get it back, decode it back into a beam format.

And so just to showcase the power of this, you know, we used to have something called the LimeStack, Linux, yours, Mesia and Erlang, you know, back in the mid 2000s.

So we were able and so we had the web server running in the same memory space.

So it was written in Erlang and yours was a web server, one of the many web servers written in Erlang.

It would hook straight into Erlang business logic.

So, again, there was no IO between the web server and your PHP or Python code or, well, Perl, if you want to let people know your age.

And then there were no external calls to external databases because you stored everything into a database called Mesia based on ETS tables.

And so we were serving, you know, dynamic, we could scale to hundreds of thousands of HTTP requests happening at the same time and serve them at a constant throughput because every HTTP request was a separate process.

So it would run in parallel and get its own time slot to execute.

And so when Apache would stall at around, again, this is mid 2000s, at around 7,000, 8,000 simultaneous connections, the beam and then you started getting a degradation in throughput because of the context switching.

The beam was able to manage about 80,000, 90,000 TCP IP connections at the time on a single node and with no degradation of service.

You know, the only thing which hit is when it ran out of memory, the whole virtual machine would crash.

And that’s when you increase the number of requests.

But the throughput in, you know, kilobytes per second of data, which it pushed out, was the same.

And this is generating dynamic web pages in milliseconds.

It was actually, and you have that today with Phoenix and Elixir, where the web server is written in, Bandit is written in Elixir, so it runs in the beam.

Phoenix, the whole web framework, so the modern day Rubin Rails with all of its channels and all of its features is written in Elixir, runs in the same virtual machine.

And you can then run Nesia or ETS tables in, again, in storing session data in the beam.

You can serve dynamic pages, you know, and they discover it faster than what it takes to actually cache static web pages in Redis and then serve them that way.

There is another question by Mauro.

So he says, what are your thoughts about Erlang, the Erlang programming language?

Is it nowadays, is the nowadays Beam main language choice Elixir?

So that is correct.

The main language of choice is Elixir.

I think Elixir is probably about 10 times larger than Erlang.

But it’s also because it’s being used in a much wider range of domains and areas.

You know, Erlang, in my view, should still be used in the low end infrastructure when you need to have full control.

And then you interface and you write all the microservices on top of it in Elixir.

And the two coexist in the same virtual machine very, very easily.

So, you know, that would be my, that’s my modus operandi.

And the reason is you write your whole control layer, you know, and it’s because there’s a different, in Elixir, you use a lot of dependencies, you use the package manager, and by doing that, you let go of the control.

In Elixir instead, by doing, in Erlang, you tend to write, oh, I need to write a connector, you know, it’s 10 lines of code, I’ll do it myself.

And that gives you full control of what you’re doing.

And you have no macros.

Well, you have macros, but they’re very different to use.

So with that, as a result, you then, with that, you know, it becomes much harder to go off in tangents and make mistakes.

So, you know, I think it’s not one or the other, but it’s, again, using the best tool for the job.

But there’s no doubt all new Greenfield projects today or Elixir, Gleam is really, really picking up as well.

And, you know, Gleam will compile both to the browser as well as the back end.

But again, keeping the two separate, it’s very much so you still you need separate mindset if you’re running in the browser versus the back end versus the server.

So there are differences, but you still use the same language for both.

And Gleam is related, I mean, as you said, Elixir is related to Ruby or Ruby-like, is Gleam more than being related?

The approach they took, the approach Louis took was he missed static typing.

He was an Erlang programmer and he missed static typing.

So, oh, OK, how can I, yeah, bring static typing to Erlang?

That was the problem he was trying to solve.

And so what that does is, you know, he still uses, you still have to use a let it crash approach, but you’re reducing the number of crashes even further because you start catching a lot of the errors at compile, which you might not do for dynamic timing at compile time.

So, you know, you ship much more solid code.

And the same is happening, you know, in Elixir, you know, where you’re retrofitting, you know, using set-heuretic types and gradual typing a type system on Elixir.

But it becomes much, much harder because, you know, it wasn’t included, you know, from day one.

And so, you know, you need to retrofit it and that involves a lot more work.

OK, so Yusuf Buketi just said, when would you not recommend Elixir?

You just said it when Erlang is the better language, if you want the control.

Control system, but also I wouldn’t recommend Erlang or Elixir when you need high throughput.

So as an example, you need to do numerical analysis, you need to do number crunching.

You know, that’s where I would, you know, go for Zig or I’d go for Rust without any doubts.

And then use Elixir for the orchestration.

So you use, you know, so you do your actual, you know, mapping in Elixir and then that will reduce where you do the heavy number computation.

You do that in a language which compiles to native and is much, much faster.

And there is a way to use those native languages as to sort of have them as processes too, right?

That is correct.

So you see them, you think you’re speaking to an Erlang or Elixir program, but there are libraries which allow you to interface and also isolate the failure, which can happen when you’re dealing with native language, with natively compiled languages.

So we are already slightly over time.

Is there anything that you want to add?

Anything that I forgot to ask you?

Any questions?

Well, I think, you know, what is the one idea from the Erlang ecosystem, you know, software developers should understand, even if they don’t write a single line of Erlang.

And I think that’s really, really important.

And if there’s one thing you take home with you from today, it’s the whole lesson about concurrency and failure handling.

So you do need lightweight concurrency in modern architectures and modern hardware stacks, you know, hardware stacks with heterogeneous cores, which are distributed where you might have multiple chips on the same machine or even on different machines.

Because what it does is it gives you scalability and it gives you reliability.

And you also need to have a way to handle the fact that failure will occur, even if your language is statically typed, you know, static typing will find some bugs, but not all.

So, you know, think about isolating your failure in a simple way.

And instead of trying to handle the unexpected, focus on recovery.

What can you actually do to recover from failure so that you don’t impact the user experience?

That is the one thing I think everyone should take away, you know, even if you don’t use Erlang on a day to day basis, you take these lessons we’ve learned and include them in the programming languages you invented.

I mean, Erlang is the Ramon of the programming languages.

It never made it, you know, it never made it mainstream, but it was back there influencing languages, you know, which became much more popular than it, you know, setting the beat in the background.

OK, so thanks a lot.

Thanks a lot for spending the time.

Thanks a lot for answering all the questions.

Thanks a lot also to the audience for asking all those questions.

So and have a great weekend and thanks a lot.

Thank you so much for having me.

It was a pleasure.

Yeah, thank you.

Hi, I am Alaa Reuschenbach.

Do you organize any user groups, conferences or other tech events?

Then feel free to add them to treff.tech, an uncommercial platform for tech events in the German speaking community.

It’s free without any advertising or tracking.

Just visit treff.tech or scan the QR code.

You will also find that link in the video description.

And by the way, you can find all software architecture stream events also on treff.tech.

And if you have any questions, feel free to reach out to me.