hckrnws
Here is the link to the git repo - https://github.com/happi/theBeamBook
> I kept going because I wanted to understand the BEAM properly. There’s value in following the real logic, not just the surface explanations.
This resonates with me. That's the kind of drive that results in great output. Buying it just for that.
I've been approached by publishers several times throughout my career. Each time the process was similar: they had an idea, I had an idea, we tried to come to common ground, and then the deal fell through because we couldn't find any. E.g. I didn't want to write a Java book aimed at 14 year olds. They didn't want me to write about classloaders (or whatever niche subject I was diving into at the time).
Would love to learn how people find (non-empty) intersections of their passions & what readers want.
Having written three books, what I found was that you either self publish or you write books that publishers want. Some of that is choosing publishers that specialize in certain types of books. Some publishers want “Learn AI in 21 Minutes with Python,” while others want “Deep Dark Secrets of Java Class Loaders.” O’Reilly is the best for niche technical stuff. Most of the rest of the industry wants beginner stuff because that’s where the volume is. Fortunately, self-publishing is easier than ever and various sites make it easy to sell copies online, so you don’t necessarily have to just give it away for free. But yea, there’s no magic formula here. If you really want to write something niche, don’t expect that a publisher will be interested in. Expect that you’ll have to self publish and promote it yourself.
That’s my experience too. My most recent book (my fifth) was closer to one publisher’s stated mission than 95% of its catalog, but they all but admitted that they had given up on it. After that rejection I didn’t have the energy to approach anyone else and did it myself instead. LeanPub first, then Amazon (print and kindle), Apple, Kobo, and Google.
Yea, good call. My books were all published back in the 1990s, back when paper was still king and you really couldn’t self-publish. Today, I’d probably look much more heavily at the self-publishing route, unless I wanted to write something uber-commercial. What was your experience with the order volume you could drive via self-publishing? Did you do any of your own marketing, or did you just list it in LeanPub and Amazon and let search engines lead people to it organically. If your subject was niche, there isn’t much volume in any case, but you want to get all you can of that small market.
More niche than mainstream I guess, but I do have what could be called a community, a large following on LinkedIn, a mailing list in the low thousands, paid subscribers, and a partner programme. Not enough to generate the kind of sales that anyone could live on, but not nothing either. Much of my blogging since publication has been in support of the book.
Two of my five did have publishers. I’m grateful for that experience. Learned a lot!
Nice! Thanks for the follow up.
Comment was deleted :(
> I kept going because I wanted to understand the BEAM properly. There’s value in following the real logic, not just the surface explanations.
Teaching is the best way to learn. I found that out when I started tutoring classmates for math in high school.
Same thing with writing a book. Something about learning a subject and turning around to speak/write out about it that really crystallizes the in-depth understanding beneath the surface.
It is easy to fool oneself into thinking one knows how something works, when in reality one only has a surface level understanding without really knowing about the internals. The serialization process exposes all the dangling pointers.
Your average person "knows" how a toilet works; water is pumped in to fill the cistern, released into the bowl when you pull the plunger, and flushes out the drain. Ask them to explain in detail how that happens, and most realize that they don't actually know how the cistern doesn't just keep filling until it overflows, or how it's not constantly leaking water into the bowl, or how the bowl can be flushed while neither overflowing nor draining completely.
This is also one of the most undersold benefits to hiring interns and entry-level developers.
Shameless self promotion but this is exactly how I ended up writing a book on strength training for climbing, just pursuing every rabbit hole I could.
I was ready to self publish but found a publisher who was interested. I had to make some changes to make it more readable, but you might have luck approaching publishers yourself
To justify your self promotion, can you tell me the name of your book? I've been climbing for over 30 years and seen the entire progression of training practices and would love to read a comprehensive book on the subject.
The Physiology of Climbing. I can mail you a free copy once I get back home this weekend - I’d love any feedback!
To be clear, I aimed to avoid prescribing certain routines through most of the book. I wanted to basically provide a knowledge foundation for readers to evaluate routines or create their own. So instead of saying eg you should campus board, I try to explain that power has to be trained separately from max strength if you care about increasing power
Happy to provide feedback! My email is my user name at Gmail. Thanks!
Can you share your book (by dm or replying here) plz? Thank you!
The Physiology of Climbing. If it looks interesting to you let me know and I can send you a copy!
Do we need to depend on Publishers? Can’t we write books independently? Or is it because of the “Brand” and other perks that come with Publishers?
Personally my favorite part of working with real publishers is editors. Having a development editor, technical editor, and a team of copy editors really helps making sure the book comes out polished. Additionally, I don't want to deal with all the layout work and the details of printing.
And, depending on the book, yes the distributor the publisher has can be very helpful for sales. It's nice to be able to grab your book off the shelf at Barnes and Noble (and, does lend a bit more credibility to your work).
All that said, if you're writing for purely economic reasons (which I would caution against regardless), you're probably going to make roughly the same if you self-publish for a small audience vs go with a traditional publisher for a larger audience, and if you can get a larger audience self-publishing then there will be no comparison.
Self publishing is easier than ever. But even shitty publishers like Packt still have a huge networking/marketing machine. My skillset does not include marketing, or checking whether a target audience exists.
If I want to write for just myself, I can just journal or blog. A book is a significant undertaking, writing one which no one reads would just be depressing.
someone should build a cooperative/self serve after stripe press books
The problem with marketing just anything is that if you market too many things that people don't actually want, your brand starts to lose its power as people stop trusting that what you're advertising is something they might want.
Indeed curation remains the important function of any publication/brand/company
the gist of the idea: >
yeah idk but something like curation is done by commitee (to try and maintain a minimum of quality overall)
Cooperative/self-serve? What does that mean?
yeah idk but something like curation is done by commitee (to try and maintain a minimum of quality overall) but as an author is super easy to appply/get editors/feedback/publish
(that’d be the self-serve part i guess)
tbh hardest is still marketing. good books are not only text but also covers and the like
> yeah idk but something like curation is done by commitee (to try and maintain a minimum of quality overall) but as an author is super easy to appply/get editors/feedback/publish
This is very much what pragprog.com is meant to be. I'm only on the volunteer curation committee so have less insight into the feedback cycle for authors post-acceptance, but every author who's published on the platform I've talked to has been pretty positive about the experience.
The OP didn't go into nearly as many (indeed, any) details as to why their second publishing attempt with them in particular did not work out, I'd be curious to learn more.
Yeah Pragmatic Programmers seems one of the best publishers on our industry
So basically a business, where day to day is run by people focused on the day to day and everyone else can pick and choose to use their stuff with low friction, not having to focus on the day to day
The hardest part still being networking into others lives to distribute the message
You just re-invented our economy
I don't know if publishers manage to get better deals from Amazon, but Amazon's Kindle Direct Publishing has an annoying aspect that the ebook royalty rate falls from 70% to 30% after $9.99, which gets very awkward for a tech book you'd like to sell for, say, $19.99.
Also, piracy is rampant and Amazon doesn't do much about it. Publishers have more resources to stay on top of it, I suspect.
Publishers add the following:
1. marketing and reach
2. financial risk on paper copies
3. Production services (e.g. editing and artwork)
If you don’t need those or can get them some other way (e.g. hire an independent editor), then you are better off self publishing and not giving the publisher a cut.
> 1. marketing and reach
In theory, yes, but they have less expertise than you might imagine. For technical writing, keep in mind that editors at publishing companies aren't actually tech people. They may have been at one point, years ago, but they don't really know what matters to programmers today in that way that an active working engineer does.
> 2. financial risk on paper copies
That was much more of a thing before print-on-demand. You don't have to take the risk of a several thousand copy offset press run anymore.
There is maybe an argument that offset printing is higher quality, but I have textbooks from major academic publishers whose print quality is clearly worse than the POD stuff I get from Amazon for my book.
> 3. Production services (e.g. editing and artwork)
This is absolutely critical, agreed. Though they are often contracting out for this and if you're comfortable finding and vetting freelancers yourself, then they don't add a ton of value.
I mean in the blog post the author literally open sourced and then self-published his book after being dropped by multiple publishers for taking too long.
Don Knuth finally sells out
https://nerocam.com/DrFun/Dave/Dr-Fun/df200002/df20000210.jp...
If you have something interesting to say then people will figure a way to understand it. Early in my career I remember coming across “Essential .Net” by Don Box and I don’t think Don Box had a particular audience in mind. He just unwrapped what’s under the hood in Common Language Runtime (CLR). It took me multiple times to really understand “essential .net”.
Comment was deleted :(
Learning to work on the BEAM with Erlang/OTP has been one of the best experiences I had in the past year, I used Joe Armstrong's "Programming Erlang: Software for a Concurrent World" book (and highly recommend it to beginners), I also heard good things about "Designing for Scalability with Erlang/OTP" (but didn't start this one yet), but nothing comes close to this in terms of depth.
The BEAM really feels like alien tech left from a highly advanced civilization and this book dropped in a such a great timing! Bought it right away, kudos to Dr. Erik Stenman for keeping it up after two cancellations!
What kind of software applications do you develop with BEAM/Erlang/OTP?
Right now they've been mostly toy projects, the more complex one being the back-end of an (WIP) MMO game (https://github.com/Dr-Nekoma/lyceum) and a couple handy libraries that I wrote myself (like https://github.com/dont-rely-on-nulls/migraterl).
The BEAM might be the most under-rated tech in all of open source.
Example -- whatsapp.
Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me.
>Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me.
I work at an Erlang shop.
For Erlang to be useful you need to have massive scale, millions of DAU. Yes Elixir might run your website serving thousands of DAU but Erlang and the BEAM was not invented for you. Few companies have the scale that makes Erlang a reasonable choice.
More pressing these days I believe is that the Erlang ecosystem is all or nothing, the BEAM is like its own little operating system and has extremely (IMHO) complicated setup and configuration with many moving parts: ERTS, epmd, Rebar,... You don't need or shouldn't use container tech and k8s with Erlang because it's already doing that stuff in its own Erlang way - but it's the Erlang way it's extremely unique and unfamiliar to most.
When you have the right use case for Erlang and you see it in action it is truly like black magic, it's been a highlight of my career to work with it.
When we tried to use Elixir Liveview we found that containerizing it (docker compose) was necessary just because of how demanding the runtime environment is regarding keeping modules up to date and synced, and what that requires of the host operating system.
We'll run prod on one server and dev on 3-4 workstations and nothing will match between any of them without a docker container to give this Elixir app a cleanroom environment to work from.
The project we were trying this on eventually ran out of funding before it got off the ground, and we lost access to our main guy who understood Elixir setup really well, so nowadays the rest of us can't even manage to stand up the service to demo it because of all of the moving parts that get angry about not having their environment set up just right.
I've basically found it the only language more difficult than python to set up an environment for. (Well.. the more I think about it, Gradle and every other mobile development stack I have yet seen is literally Satan's armpit..)
With python (though I rarely code in that either) I can stand up almost anything on almost any machine with miniconda and venv and not have to drag Docker into things.
Node/NPM is a prima donna and a PITA but IME as long as you whack it with a wrench long enough following up on each of its complaints then you'll eventually get things running.
My experience still revolves around PHP or Perl or C on the backend, Raw Javascript or sometimes Elm on the front end, and those all seem to be a lot easier to maintain over a timescale of decades as servers update and browsers gobble up new and different features.
---
What I can say in favor of Elixir Liveview is that we built a smooth as hell SPA using that. It was incredibly user friendly to work with and aesthetically amazing, but the maintenance right down at the foundation was the biggest challenge.
What environment issues were you running into? We use mise/asdf to manage which version of erlang/elixir our project is using, and only needed to containerize our database to ease its maintenance.
For the dev experience, I'd also recommend NextLS/Lexical over Elixir LS until the official one is out. It should compile much faster for nontrivial Elixir applications.
https://elixir-lang.org/blog/2024/08/15/welcome-elixir-langu...
Would something like Nix (optionally, NixOS) help in getting all the parts documented and working together?
Elixir has been quite easy to set up, configure and deploy, at least for the last 5-6 years. I can't speak for Erlang though.
> Elixir has been quite easy to set up, configure and deploy, at least for the last 5-6 years.
One of your sibling commenters noted...
> I've basically found it [Elixir] the only language more difficult than python to set up an environment for.
Quite the difference.
I'm not saying either one of you is wrong; I'm sure you both experienced what you say you did, it's just interesting to see the dichotomy, literally (as of this writing) next to each other.
We found Erlang to be the right choice at small scale. Namely, we used it in iot applications, where the self-healing property of proper supervision trees resulted in a very stable implementation. We also saw the same benefits in the cloud part of our application. No complicated k8s setup, just simple supervisor trees.
> Few companies have the scale that makes Erlang a reasonable choice.
This is a reality that I wish more people would embrace, for so many things.
elixir softens a whole ton of the sharp edges around getting started and there are only a handful of "gotchas" coming from other hlls (like all terms being immutable when you pass to a function -- but thats honestly almost a problem with the "mainstream" languages).
So you'd do:
x = doubler(x)
Instead of treating the function like a "subroutine"? Like:
x = 10
doubler(x)
print x
I'm unsure why, but I use the former. Probably due to most sources saying global variables are bad and if I do it the latter way I get errors, whereas with the former I only get errors unrelated to scope.
I'm not a professional or even amatuer programmer, that's why I ask. My only exposure to Erlang unfortunately is from coding contests like advent of code and Codegolf stack; and those are "clever" and my brain doesn't abide clever. The inverse of "why use many words when few words will do"
In my own programs, if I go back and read the code I can tell what I copy-pasted vs what I understood as I wrote it, if that makes sense. Because clever stuff that solves the problem I've been unable to solve in the time I alloted will be in my code. You see anything pretending to be, or being a lambda in my code, I lifted it.
I would like to learn Erlang and there's been a few book recommendations aside from TFA. This comment also serves as my reminder to get some books.
well probably part if the reason for example in python, you cannot use the latter for x an integer but you could if x were a dict or a map!!
that's probably why!!
but don't forget, there are core routines that must be used to set certain data that are of the latter form, so at some level you have no choice.
d = {'foo': 1}
d.pop('foo') # <== mutating function callI believe it's purely marketing
Other languages (Java, C#, Go) are supported by massive corporate backers, who have vested interest in seeing them succeed. Erlang's own corporate owner tried to strangle it in the crib, and even since then they've been standoffishness towards anything apart from technical resources.
We didn't really see much marketing like material arise until things like Elixir came about, and even that is more following the ruby model, which is very dev oriented. And the world to which Elixir emerged in 2014 is very different than the world rails sprung into in 2004.
Devs can usually be convinced about how good BEAMlangs are, but without the glossy, you have a harder time convincing the MBA set
Erlang wasn't open sourced until the late 90s. Erlang was never really marketed. Elixir arose fairly quickly.
Erlang and Elixir are nice because concurrency isn't tacked onto the language like those other languages.
Investment gap is what I'd say too. While Rust, Go, Python, etc... have had massive backers that have managed to invest a ton more into things like static analysis, type checking, and developer ergonomics, the Erlang ecosystem hasn't necessarily had the same love, and instead the major users have typically chosen to pivot, or build something outside of the BEAM.
> Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me.
1) Because the amount of "high concurrency" you can handle with a single machine with "standard" languages keeps moving up every year.
2) Because you had a single, nigh undocumented, very impenetrable to port implementation of BEAM for the longest time.
3) Because "Erlang" isn't the magic. OTP is the magic. And that requires that you shift your mindset dramatically. And you can do the OTP abstractions in other languages!
4) I think Scala sucked up a lot of Erlang's oxygen for a long time. Erlang is helping with that.
1. Fair, but at the same time - OTP is the only sane concurrency framework. To the point I still struggle to comprehend why it wasn't copied. 2. Fair. 3. I am not aware of a single language doing OTP abstractions, especially implementing preemptive scheduler for green threads. 4. Possibly, but I haven't seen a single instance of a company choosing between the two.
I think marketing is a way better explanation to be honest. Jose lives in Poland, was very active and a lot of Ruby shops moved to Elixir.
There have been multiple attempts in various languages to implement Erlang-like processes and claims of parity, Erlang processes remain unbeaten at what they do.
Go has a preemptive scheduler for green threads.
Oh, indeed, since 1.14. Nice to see them moving this direction.
OTP is the magic but the ability of the Beam to handle that many processes gracefully is part of the magic
C++ and its libraries can do everything erlang does with concurrency, and faster. erlang remains niche mainly for that reason.
You can build anything with anything. Not always a good idea. There are a reason few web apps are built using C++ directly.
everyone is trying to hire the best, but caters to the lowest common denominator.
"But what if we hire an engineer to stupid to learn a new language in a few weeks???" and on and on and on
Because it's a single stack that leaks too many things in my opinion, it's not just code but it's also handling infra, process restart, cache, queue, db ect ... And it only works with Erlang, so it's an all or nothing setup which imo is very restrictive.
Kubernetes does a lot of things that Beam does but better and completely language neutral. Then you have builtin queues and db in Erlang but they are inferior to industry standard and again only work with Erlang.
Honestly, that mixing of concerns is the biggest benefit, at least for me.
While Erlang clearly lost the popularity war, using it (well, Elixir in my case) feels like going to an alternate timeline where it wasn’t a given that a good application used separately developed tools for frontend, backend, database, cache, service discovery, load balancing, containerization, orchestration, and so on. We really could have been on a platform that incorporated all of those things into a single runtime.
The way I explain Erlang to folks is: “it’s not a competitor to Python running Django in Gunicorn; it’s a competitor for Python+Django+Gunicorn in Docker on Kubernetes on Linux, talking via a service mesh and storing data in disk-persisted Redis.” In many ways, swapping out only the top of that stack with Erlang is going to leave you worse off than you were before.
Sadly, the Erlang renaissance was years late to the boom of containerization+orchestration+distributed systems, so we’ll never know how good (or bad) things could have been. Ah well, c’est la vie.
We swapped our Squarespace website out for a Phoenix application yesterday and it was such a delightful experience.
And now with Gleam moving past version 1.0 last year, the BEAM also boasts a decent amount of "flavors" of functional programming. You can go with Erlang/OTP (the alien tech that has been battle tested for almost 30 years), Elixir for it's great webdev ecosystem, Gleam if you like an ML-like programming language on the BEAM. LFE is also another great option if need a BEAM-native LISP.
How I wish that Gleam had OTP!
Elixir is also getting some tooling love in the form of a new language server and some very much appreciated type checking
Forgive my ignorance, I was looking at this the other day, isn't this otp for gleam?
yes but i believe it is still in the ”beta” stage, as gleam is implementing the OTP by itself and not wrapping the erlang implementation.
you have to import the otp package in gleam to use the famous actor architecture
At the same time, the least secret "secret sauce" out there. BBC moved to Elixir (quite recent news), so I guess traction is there.
"We will try to use the term Erlang Runtime System to refer to the general idea of any Erlang Runtime System as opposed to the specific implementation by Ericsson which we call the Erlang RunTime System or usually just ERTS."
Love it.
That is... not dumb?
Elixir and BEAM are my all-time favorite choice for building networked or pipeline-heavy systems. I used it every day in production for several years, and while it’s a hard sell for recent projects (and generally the wrong call), I enjoy keeping up with it as much as I can.
Thank you for writing this book! I really wanted this a few years ago as I was debugging production Elixir, but existing learning sources were pretty dense and dry (or too simple and shallow).
I guess I've never worked on something of Klarna's scale, but 15ms seems like a very small amount of time to cause a post-mortem-worthy event!
Yes, it does seem small, but the BEAM often has response times in microseconds (μs). If you are used to that and something 'blows out' to milliseconds, then I can see why alarms might get triggered.
If it's millions of calls and it's 15 additional milliseconds per call, that's going to spectacularly gum up a system that's tuned in microseconds. Even if there's no additional load, the queues all just got several thousand times longer.
This exchange made me think about how I've always been a fan of using the "natural" units of a system when reporting metrics.
I do this because it helps those unfamiliar with the system more intuitively understand relative sizes, those unfamiliar with converting between units in their head to understand numbers, and even for those familiar, it can avoid confusion or misattributed units.
this can easily happen in a BEAM system. say you have some shared state you want to access. you create a gen_server to protect this shared state. a gen_server is basically a huge mutex. the gen_server is just a normal beam process that handles requests sent to its message queue and then sends a reply message back. lets say it can process a request normally in 20us. so a 15ms pause would stack up 750 messages in its message queue. now maybe this is not enough to generate a huge outage on its own but maybe as part of your handling you are using the message queue in an unsafe way. so when you check the message queue for a message the BEAM will just search the whole message queue for a message that matches. there are certain patterns the BEAM is able to optimize to prevent the whole message queue being searched (i think almost every pattern is unsafe and the BEAM only optimizes the gen rpc style message patterns) . but if you are using an unsafe pattern when you have a message queue backlog it will destroy the throughput in the system because the time taken to process a message is a function of the message queue length and the message queue length becomes a function of how long it takes to process a message.
Also, the great thing is you might not even have an explicit `receive` statement in your gen_server code. You might just be using a library that is using a `receive` somewhere that is unsafe with a large message queue and now you are burned. The BEAM also added some alternative message queue thing so you are able to use this instead of the main message queue of a process which should be a lot safer but I think a lot of libraries still do not use this. This alternative is 'alias' (https://www.erlang.org/doc/system/ref_man_processes.html#pro...) which does something slightly different from what I thought which is to protect the queue from 'lost' messages. Without aliases 'timeouts' can end up causing the process message queue to be polluted with messages that are no longer being waited on. This can lead to the same problems with large message queues causing throughput of a process to drop. However, usually long lived processes will have a loop that handles messages in the queue.
> lets say it can process a request normally in 20us.
Then what if the OS/thread hangs? Or maybe a hardware issue even. Seems a bit weird to have critical path be blocked by a single mutex. That's a recipe for problems or am I missing something?
Hardware issues happen, but if you're lucky it's a simple failure and the box stops dead. Not much fun, but recovery can be quick and automated.
What's real trouble is when the hardware fault is like one of the 16 nic queues stopped, so most connections work, but not all (depends on the hash of the 4-tuple) or some bit in the ram failed and now you're hitting thousands of ECC correctable errors per second and your effective cpu capacity is down to 10% ... the system is now too slow to work properly, but manages to stay connected to dist and still attracts traffic it can't reasonably serve.
But OS/thread hangs are avoidable in my experience. If you run your beam system with very few OS processes, there's no reason for the OS to cause trouble.
But on the topic of a 15ms pause... it's likely that that pause is causally related to cascading pauses, it might be the beginning or the end or the middle... But when one thing slows down, others do too, and some processes can't recover when the backlog gets over a critical threshold which is kind of unknowable without experiencing it. WhatsApp had a couple of hacks to deal with this. A) Our gen_server aggregation framework used our hacky version of priority messages to let the worker determine the age of requests and drop them if they're too old. B) we had a hack to drop all messages in a process's mailbox through the introspection facilities and sometimes we automated that with cron... Very few processes can work through a mailbox with 1 million messages, dropping them all gets to recovery faster. C) we tweaked garbage collection to run less often when the mailbox was very large --- i think this is addressed by off-heap mailboxes now, but when GC looks through the mailbox every so many iterations and the mailbox is very large, it can drive an unrecoverable cycle as eventually GC time limits throughput below accumulation and you'll never catch up. D) we added process stats so we could see accumulation and drain rates and estimate time to drain / or if the process won't drain and built monitoring around that.
> we had a hack to drop all messages in a process's mailbox through the introspection facilities and sometimes we automated that with cron...
What happens to the messages? Do they get processed at a slower rate or on a subsystem that works in the background without having more messages being constantly added? Or do you just nuke them out of orbit and not care? That doesn't seem like a good idea to me since loss of information. Would love to know more about this!
Nuked; it's the only way to be sure. It's not that we didn't care about the messages in the queue, it's just there's too many of them, they can't be processed, and so into the bin they go. This strategy is more viable for reads and less viable for writes, and you shouldn't nuke the mnesia processes's queues, even when they're very backlogged ... you've got to find a way to put backpressure on those things --- maybe a flag to error out on writes before they're sent into the overlarge queue.
Mostly this is happening in the context of request/response. If you're a client and connect to the frontend you send a auth blob, and the frontend sends it to the auth daemon to check it out. If the auth daemon can't respond to the frontend in a reasonable time, the frontend will drop the client; so there's no point in the auth daemon looking at old messages. If it's developed a backlog so high it can't get it back, we failed and clients are having trouble connecting, but the fastest path to recovery is dropping all the current requests in progress and starting fresh.
In some scenarios even if the process knew it was backlogged and wanted to just accept messages one at a time and drop them, that's not fast enough to catch up to the backlog. The longer you're in unrecoverable backlog, the worse the backlog gets, because in addition to the regular load from clients waking up, you've also got all those clients that tried and failed going to retry. If the outage is long enough, you do get a bit of a drop off, because clients that can't connect don't send messages that require waking up other clients, but that effect isn't so big when you've only got a large backlog a few shards.
If the user client is well implemented either it or the user notices that an action didn't take effect and tries again, similar to what you would do if a phone call was disconnected unexpectedly or what most people would do if a clicked button didn't have the desired effect, i.e. click it repeatedly.
In many cases it's not a big problem if some traffic is wasted, compared to desperately trying to process exactly all of it in the correct order, which at times might degrade service for every user or bring the system down entirely.
Depending on what you want to do, there are ways to change where the blocking occurs, like https://blog.sequinstream.com/genserver-reply-dont-call-us-w...
Part of the problem with BEAM is it doesn't have great ways of dealing with concurrency beyond gen_server (effectively a mutex) and ETS tables (https://www.erlang.org/doc/apps/stdlib/ets). So I think usually the solution would be to use ETS if its possible which is kind of like a ConcurrentHashMap in other languages or to shard or replicate the shared state so it can be accessed in parallel. For read only data that does not change very often the BEAM also has persistent term (https://www.erlang.org/doc/apps/erts/persistent_term.html).
> this can easily happen in a BEAM system
Wait... I thought all you had to do is write it in Erlang and it scales magically!
Erlang makes a lot of hard things possible, some tricky things easy, and some easy things tricky.
There's no magic fairy dust. Just a lot of things that fit together in nice ways if you use them well, and blow up in predictable ways if you have learned how to predict the system.
I know that, having built several myself. Sometimes people get a bit ahead of themselves with the marketing though.
I am also curious about the mentioned incident, does anyone have a link to the postmortem the post talks about? Couldn't find anything online.
I would assume it was an internal post-mortem. Far from all are public affairs.
Also I think the incident is over 10 years old as well, if it's the problem I think it is.
Bought it instantly, even if it’s available online for free I guess this way supports the author a little bit
> Issue #113 - “Please continue being awesome.” That emoji-laced drive-by encouragement (August 2018) still pops into my head whenever motivation dips.
This warms my heart. While the internet is infamous for its negativity and how it makes people miserable, even small positive moments like this can make a lasting difference and remain memorable years later.
It's something you can apply in your own life. If you think someone did something nice or good, just tell them. Don't make a big thing out of it; but just make sure to let them know when you get the opportunity. Most people will really appreciate that and might get motivated by that. This sounds simple but a lot of people don't do this or are a bit shy about it. Of course the value of your compliments decreases with the frequency you give them so it helps being a bit genuine about it and to not overdue it. Something simple as using the star button in github, adding a little thumbs up to an issue in an issue tracker, or upvoting a comment on HN is what keeps the good stuff going.
A lot of stuff that people do for free is rather thankless work except for the occasional appreciation. I've been maintaining some only modestly popular OSS projects on Github. They won't change the world and I mostly do it because it serves my own needs. But getting the occasional outreach from people that used some of my code is always a highlight in my day.
Are there any other VM's like the BEAM? I never heard of any (admittedly I know little of this subject), and I wondered, is it because there is no need for another one because the BEAM is just so good, or is it because the amount of work and skill required to get another BEAM-like runtime with comparable quality is too demanding?
The reality is modern Kubernetes infrastructure looks a lot like BEAM, at least in the capabilities it offers. That's the far more common way of deploying highly scalable, self-healing systems in the current year. Plus, with k8s you're not constricted to a single language (there are a few more than Erlang/Elixir, but nothing popular) with limited developer resources and interest.
So now you need several servers, an orchestrator, tons of YAML, arcane and terrible error messages and a devops team to kind of match the BEAM? That's... not a good look
Yes. This is a big part of what initially drew me to Elixir. It's more than feasible to run a server on a cheap VPS, get great, though not quite Golang or low-level language performance and have a much easier scaling story when you need multiple machines.
More importantly, you generally don't need an external queue service, in-memory KV store, task scheduler or many of the other things that JS/Ruby/Python stacks need. By consolidating just about everything but the DB in a single, well designed system, it's possible for a very small team to take on relatively large challenges on a smaller budget.
Which VPS providers do you recommend? Which relational databases do you use with Elixir?
I use a dedicated hetzner server (12 cores 64gb ram) for 60 euros with postgres
Not an Elixir expert, but my impression is that Postgres is a common choice. It’s well supported by both Ecto (ORM) and Oban (job queue).
Where does your Erlang code runs in the first place? ( maybe kubernetes already )
Kubernetes does all of that in a standard and easy way but also is completely language agnostic. So your Python code without modidication can benefit from it.
in my case it runs on a good old server.
> So your Python code without modidication can benefit from it.
that's not completely true though, say you have two python processes, you need to solve yourself how they communicate, HTTP? a message broker? through the DB? You need to handle errors, stateful deployments.
You can deploy python code without modification if the python code does very simple things. My point is that the BEAM gives you a lot of this mostly presolved for you without having to add more infrastructure.
is that a fair comparison? i dont love k8s but you can deploy anything to it, not just erlang or elixir
You can use a BEAM system to orchestrate other code, too. As ports, port drivers, nifs, c-nodes, just other OS processes spawned and using IPC/sockets. Lots of options. Using Erlang to supervise an OS process doing work perhaps enhances the isolation principles of BEAM.
oh interesting wasnt really aware
so you could use beam to orchestrate go or rust services communicating over IPC? Nice
See also CloudI a Cloud System built using Erlang - https://cloudi.org/index.html
TY! Very interesting!
To add to that, the BEAM has binaries as a data type so when you're talking to a foreign program you can quickly prototype or design binary protocols once you get tired of parsing/encoding JSON or XML or something at the edges. Depending on the facilities of the foreign language it might be more or less feasible, of course.
The binary interface makes interop easier with other langauges. A note of incompatibility in versions after OTP-24 with those before.
It's snarky but I think it is a fair comparison. You do have extra capabilities over what BEAM offers, in exchange for having to manually handle a lot of things BEAM either handles invisibly or at least comes with usable defaults for.
At the time that BEAM was invented in the late 1990s, and in the early 2000s, it was a fairly unique proposition.
Nowadays there isn't anywhere near as much stuff that it does uniquely. That's probably why there isn't another one. All of the compiled languages off-the-shelf can solve the same problems that BEAM does now, and often with other advantages to boot.
There's something about the Erlang community that convinces people in it that if it isn't solve the exact same way that BEAM does, then it must ipso facto not be as good as BEAM, but that's not true. If you ask the question can I solve the same problems, even if it's in a different way?, you've got a zoo of options in 2025, whereas your options in 2000 were much much weaker.
And yes, being BEAM-compatible is harder than meets the eye. There are projects like https://github.com/ergo-services/ergo that can do it, and I believe there are some in other languages. It's a pretty niche need in my opinion, though. If you don't need to hook up to some existing BEAM infrastructure, I don't consider it a very good solution for a greenfield project. You're better off with more modern tooling and solutions that are more native to your chosen dev environment.
Correct me if I'm wrong but a big part of the value proposition these days for Erlang/Elixir would be that it comes build in with a lot of the tools that other runtimes would require. For example a message bus is kinda integrated in the BEAM runtime or Erlang, while most other runtimes would have to use something like Kafka or AWS SQS etc. Or caching and some other things. Of course I'm not sure how good all the inbuilt solutions are compared to dedicated tools, overall or usually IMHO I prefer dedicated tools as opposed to all in one as the dedicated tools/services usually are more flexible. On the other hand of course it is also much overhead.
> a message bus is kinda integrated in the BEAM runtime or Erlang
You have that on a single node. If you need to run more than one node, you will end up inventing your own on top of mnesia and usually the results are not spectacular or/and you will end up paying happihacking to do it for you. Or one of the other Erlang oldtimers who you can count on the fingers of your hands.
This is really suboptimal compared to what you can achieve by using any normal language + any message bus. You are actually much better using a proper message bus even if you use Erlang.
I agree with what I think is your overall point, that there are other solutions that adequately solve the problems BEAM does but in different ways.
I really strongly disagree with the idea that there's no modern use for BEAM because of these other solutions. It's not simply that we've convinced ourselves that "if it isn't solve the exact same way that BEAM does, then it must ipso facto not be as good as BEAM" though I understand how you could see it that way.
Frankly what it is is that BEAM has an exceptionally well chosen position among the possible tradeoffs of solving these problems, which you are otherwise on your own to figure out and which are in themselves some of the most difficult practical considerations of systems design.
So again it's not that only BEAM can possibly do it right, but it's that BEAM does do it right. Having seen so many other attempts to do it better fail, at tremendous expense, it's an easy choice for me for systems that are expected to encounter these problems.
"I understand how you could see it that way."
It is less that "I see it that way" then that "I encounter plenty of people who speak that way.", and that the Erlang community still definitely indoctrinates people to think that way.
See my other post on the topic: https://news.ycombinator.com/item?id=44181668 Which echos some of your points.
I'm not sure if it's about the implementation rather than the vast reduction of complexity/boilerplate to attain fault-tolerant distributed systems compared to other solutions.
Can you map the BEAM features against the zoo of options you mentioned so readers understand what exactly are these other options and how they compare to BEAM?
BEAM is basically the combination of a clustering system, a message bus for communicating among its nodes with a defined protocol, and a shared data model that can be thought of as basically "JSON, and only JSON, is the data model"; not JSON mapped into data structures or classes, but just JSON.
The biggest thing you need to have to have BEAM-like reliability is a message bus. Build a message-bus based system and use it properly and you're already 80% of the way there. In 1998, who knew what a "message bus" was? Today, it's a field so stuffed with options I won't even try to summarize them here. The one thing I will point out is that BEAM is 0-or-1 and 1-or-n seems to have won the race; this is one of the things I don't like about BEAM's message bus.
BEAM is based on a JSON-like data scheme because it wasn't clear how to have what you'd consider "classes" and such on a cluster with multiple versions of things any other way. Since then, there are now multiple technologies to solve this problem, like gRPC, Cap'n Proto, and again, what was "who's heard of that?" in 1998 is now an entire field of options I can barely summarize. It is no longer necessary to sacrifice everything we get with "methods" and "user data types" to have cross-cluster compatibility.
Bringing up clusters is now just Tuesday for a devops team. Kubernetes, Docker Cloud, all the cloud-specific orchestrations like CloudFormation, bringing up clusters of things now has many technologies you can deploy. Moreover, these technologies do not impose the requirement that your nodes be essentially homogeneous, all running BEAM. You can bring up any tech stack you like doing any specialized tech stuff you need, all communicating over shared message busses.
Reliability options vary widely, from running internal services in OS processes to capture and restart things, to things like lambda functions that are their own solution to that sort of reliability, to isolated micro-VMs, to threading-based solutions... while there is a certain appeal to "just crash and take the thread down" it's not the only option anymore. Now that every language community that can is building huge servers with millions of threads in them, many solutions to this problem have developed, with different cost/benefit tradeoffs for each. The "crash out the whole thread" is not the only option, merely one interesting one.
As to how they compare to BEAM, that does slant the question a little bit, as if BEAM is the golden standard that everyone else is still desperately aspiring to. How they compare to BEAM is that there is now a zoo of options of every sort, with a huge variety of tradeoffs in performance and cost and ease-of-use and ease-of-deployment, and BEAM is merely a particular point in the huge field now, which I wouldn't even characterize as particularly standing out on any front. For every thing it does like "have a nice crashing story" there's a tradeoff where it did things like "give up user-defined data types because they couldn't figure out how do to them in a cluster in the late 1990s". BEAM's feature set, on a feature-by-feature basis, isn't particularly special any more. There's faster options, there's easier options, there's "works with my current language" options, there's more reliable options, there's cheaper options, there's all kinds of things... but not all of these at once necessarily.
So, what is BEAM's unique value proposition in 2025? It is the integration of the solutions, and picking a set of solutions that IMHO may not be "the best" on any particular front any more but has proved to be "adequate to many tasks" for decades now. You can assemble all those technologies I mentioned in an incredible number of combinations, but you have to do the assembly yourself, and burn time asking yourself, which message bus? Which clustering? Which orchestration? It's overwhelming, and made even harder by the fact that even with all these choices there's still a lot of developers who don't even know these things exist, let alone how to evaluate them and start integrating them into a system successfully. BEAM offers a one-stop shop, with prepared and opinionated answers with a proved track record, and a community that knows how to use them, libraries that integrate with the opinionated answers. I.e. who can write a library that "works with Kafka, Amazon SQS, NATS, Redis streams, Postgres events, and a dozen other messaging libraries" without having to work with a lowest common denominator that is almost useless? But someone can definitely provide "an Erlang library" that meaningfully integrates with all of the BEAM infrastructure without an issue. I don't think BEAM is best-of-breed on any particular front but it is arguably best-of-breed in providing them an answer to you for all these problems under a single roof, and then being able to build on the advantage that picking a set of solutions brings you.
I wish the BEAM/Erlang/Elixir community would lean more into this as their pitch, and stop running around and acting like they've got the only solution to the problems and as if they are the gold standard. The advantage of BEAM is not that they have the only reliability solution, the only message bus, the only clustering solution, the only etc. etc. anymore... the advantage is in the integration. The individual components of the solutions are not where the awesomeness lies; in 2025 most of them have been handily exceeded on many fronts now, but that integration and the subsequent things like libraries built on that integration is a unique proposition, even today. The very explosion of choice in the solutions to the problems BEAM addresses and the resulting diaspora in all the code bases in the world make it difficult to build very much on top of these things because there's so many differences in so many places.
I’m actually kind of a fan, I’ve read your past comments on BEAM and Elixir while digging through HN to get a sense of how people view it. I appreciate your takes, but I’m honestly confused by this one and wanted to push back a bit, I'm not as knowledgeable as you on this topic, but bear with me
I get your point that BEAM’s individual components might not be the best in 2025 and you get the point of uniformity. so whats the point of saying there's a better BEAM like system, but then fail to point out one? Elixir/BEAM community promoting themselves as the only solution to said problems isn't a bad thing imo, because what other system can give me the guarantee without forcing me to learn a bunch of new DSLs or scripting languages, and deal with the idiosyncrasies of a bunch of different systems? With Elixir or Erlang, I can stick to one coherent environment and get all that.
Again you state all this in your post, yet say elixir/beam isn't the gold standard, then what is? As i am having a blast working with phoenix, liveview and nerves and the BEAMS guarantee of a ms soft real time fault tolerant system hasn't failed me yet and there doesnt seem to be anything like it in the market. The only thing I hate about elixir is types and would switch to rust/go if there was a similar offering
"but then fail to point out one?"
I didn't point out one, I pointed out thousands. All the combinations of message busses and serializations and schedulers and monitors you can imagine. Systemd monitoring a selection of processes that read and write from kafka queues in a cluster of VMs. Lambda functions that read and write Amazon SQS/SNS queues written in Go. Azure Functions using JSON to communicate over Azure Service Bus with Terraform configuring redundancy. A microservices architecture orchestrated with etcd based on gRPC services communicating with each other on a NATS message bus. Arbitrary mixes and matches of all of these things, probably the most common architecture of all at a corporate level.
Many of these beat BEAM on some metric or other that may be important at some point. For instance it's not hard to beat BEAM on performance (though it is also not hard to lose to it on performance; it's a very middle-of-the-road system on that front, which I mean completely literally). Another thing that you get very naturally is heterogeneity; if you want everything on a BEAM system you really have to stick to BEAM's languages, which is a pretty big issue in most cases.
The reason I say BEAM is not the gold standard is that there's still people running around who speak as if BEAM is the only way to write reliable software, that it is still the only way to have lots of independent services that communicate by passing messages, that if you don't implement every single one of gen_server's features and work exactly like OTP's supervision trees and if you don't handle errors by crashing processes, then you're not operating at Erlang's Golden Standard.
And that's not true anymore. There's plenty of alternatives, many better than BEAM on many fronts. BEAM is not the natural obvious leader standing above the rest of the crowd, secure in the knowledge that nothing is as good as it is and never will be. It's now down in the scrum, and as long as it is metaphorically running around claiming it's somehow unique I'm going to grumble about it. It's not even good for BEAM itself, which really ought to be pitching integration rather than "Look! We solve the reliability problem uniquely!"
To the extent that people are reading my post and basically backpedaling and insisting "Oh, no, it's the integration all along that we've been pitching"... well, I don't particularly enjoy the rewriting of history, but other than that... yes! Good! Do that! Pitch the integration! It's a legitimate advantage! But it's not 2005 anymore. Every major compiled language can handle tens or hundreds of thousands of connections on a reasonably sized server. Every language has solutions for running this robustly and with decoupled architecture. Every language has solutions to the problems now. BEAM's in a crowd now, whether it likes it or not.
There is no gold standard for these technologies any more. One might as well ask "well, what's the best computer?" There's no answer to that question. Narrow it down to "what's the best gaming computer" and you can still ask "what sort of games" and it'll be a crowded field. There's more options that anyone can even analyze anymore.
Do note that many things mentioned just happen to be incentivized or default in the BEAM, but are not required at all. Your processes don't have to crash if you don't want to let them crash, exception handling is there. Messages don't have to be 0-or-1, Manifold exists. You are absolutely not forced to use some language, you have NIFs, dirty NIFs, ports, rustler, zigler, pythonx, etc, many offering impeccable ergonomics. Some things may not be as optimized as alternatives are? Sure, but the BEAM works for Whatsapp and Discord and of all things they just use raw ETS for their most important data structure. You almost surely don't need something "better".
There is a separate implementation for really constrained device use, called AtomVM.
There are many dead efforts to implement something like the BEAM or OTP within other ecosystems. Usually not as a VM.
I have seen attempts at actor systems like erlang/beam's. Scala/Akka seems to have been a success?
Share nothing, green-thread/coroutines also seem popular now a days.
C++ Actor framework is also considerably popular, and Scylladb has their own framework which I believe does something similar.
Foundationdb also has it's own C++ actor framework.
I think the closest in wide use is probably Go.
The BEAM is set up for "Erlang like languages" (or maybe it's the other way around). Writing Elixir, still feels a lot like Erlang because of the underlying semantics of how it operates. Even Gleam is closer to Erlang than anything else once you get past the types.
Go also has goroutines/green threads/Erlang-like processes as a core primitive for its parallelism. it doesn't have the same "opinion" about how to structure a concurrent application that you get from OTP though.
Dis is probably the closest, followed by the smalltalk VMs. It depends on what you mean by "like the BEAM". The VM itself isn't that special, it's really the whole OTP on top of BEAM that makes Erlang good at what it does.
[dead]
I keep hearing my marvelous promises about the BEAM, but the adoption is still so low. Why?
I understand it’s poorly promoted by the biggest users. But more of the web and world infrastructure runs on it than you realise.
Here’s an old post with some examples - https://www.erlang-solutions.com/blog/which-companies-are-us...
It's not being promoted much. There are no BEAM commercials during the Super Bowl. That is to say that there is no big company promoting it. Sun pushed Java hard, Google pushed Go, Mozilla pushed Rust and so on.
But those who know, know. I mean it's on the front page of HN pretty often.
This looks amazing purely from an engineer's perspective that wants to level up on something they have worked on for a while with enough understanding to build working systems with it. Just confirming, this is more useful for developers that have worked with Erlang/Beam correct? Not so useful for a Beam newcomer?
Plenty of documentation can be found about the JVM, but BEAM has always seemed like a bit of a mystery to me. This is great!
The BEAM isn't that big. When you have questions the documentation doesn't answer, the source might. (I've not read this book, but I did order it, I'd expect it to have some answers too)
The official documentation explains things a little, but only a little.
Books ask for a lot of organization in general. And books on an evolving subject never stop asking.
I don't use Erlang, but for 13 years in the making, I'm getting a copy.
Thank you.
p.s if the author sees this: Kindle edition too, please.
OK. It is currently under review by Amazon. Remember, a PDF is free on GitHub; the Kindle thingy will cost some money. - Happi
It is now available. https://www.amazon.com/dp/B0FBX7YWB1
Bought a copy!
I’ve been watching erlang from afar for a few years. Here’s to hoping to actually get to work with it one day.
Erlang is one of those languages that will make you a better programmer in other languages.
Yes, but after using Erlang, I found it very painful to switch. Easy to get spoiled by immutability & pattern matching.
Fullstops.
Elixir is a pretty great alternative
Gleam also seems to be a bit more approachable.
Comment was deleted :(
Yes, that's explained in the book's title.
The book's title, "Understanding the Erlang Runtime System", contains 25% of the words represented in the BEAM acronym, "Bogdan's Erlang Abstract Machine".
I've not used Erlang before (and naturally have not read the BEAM book); I mostly live in the world of C++ (and CUDA/OpenCL) these days. Would this be of interest to me?
Oh I thought this was gonna be about BEAM as in https://smfr.org/robots/ ha ha.
TIL that Klarna runs on Erlang.
Thank you very much!
[dead]
Crafted by Rajat
Source Code