Erlang and Elixir

Deep dive

Decision table

Memory and performance model

• Immutable data; each process has its own heap and generational copying GC, run when that process needs it.
• Messages are copied between process heaps; binaries over 64 bytes are reference-counted and shared off-heap.
• ETS: in-memory tables outside process heaps with concurrent reads; the tool for shared hot data (caches, counters).
• BEAM JIT (BeamAsm) compiles to native code at load; still slower than static languages for tight loops.

Concurrency model

• spawn, send, receive; links (crash propagates) and monitors (get a :DOWN message).
• OTP behaviours: GenServer (stateful server), Supervisor (restart strategies one_for_one, rest_for_one, one_for_all, with restart intensity limits), Application, GenStateMachine.
• Distribution: nodes connect in a full mesh; send to a remote pid looks the same as local. Practical clusters are order of tens to low hundreds of nodes.
• Hot code loading: replace modules in a running system (mostly used in telecom; most teams do rolling deploys).

Code

Elixir: a GenServer under a supervisor, plus bounded parallel fan-out.

defmodule Presence do
  use GenServer

  # Client API: runs in the caller's process
  def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  def join(room, user), do: GenServer.cast(__MODULE__, {:join, room, user, self()})
  def who(room), do: GenServer.call(__MODULE__, {:who, room})

  # Server: state lives in one process and changes only through its messages
  @impl true
  def init(_opts), do: {:ok, %{rooms: %{}, refs: %{}}}

  @impl true
  def handle_cast({:join, room, user, pid}, state) do
    ref = Process.monitor(pid) # get :DOWN when the user's process dies
    rooms = Map.update(state.rooms, room, MapSet.new([user]), &MapSet.put(&1, user))
    {:noreply, %{state | rooms: rooms, refs: Map.put(state.refs, ref, {room, user})}}
  end

  @impl true
  def handle_call({:who, room}, _from, state) do
    {:reply, state.rooms |> Map.get(room, MapSet.new()) |> MapSet.to_list(), state}
  end

  @impl true
  def handle_info({:DOWN, ref, :process, _pid, _reason}, state) do
    {{room, user}, refs} = Map.pop(state.refs, ref)
    rooms = Map.update!(state.rooms, room, &MapSet.delete(&1, user))
    {:noreply, %{state | rooms: rooms, refs: refs}}
  end
end

defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      Presence,
      {Task.Supervisor, name: MyApp.TaskSup}
    ]
    # one_for_one: a crash restarts only that child, with fresh state
    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
  end
end

# Bounded fan-out: each item in its own process, at most 4 per core at once.
defmodule Fanout do
  def total_sizes(urls, fetch) do
    urls
    |> Task.async_stream(fetch, max_concurrency: System.schedulers_online() * 4, timeout: 5_000,
                         on_timeout: :kill_task)
    |> Enum.reduce(0, fn
      {:ok, size}, acc -> acc + size
      {:exit, _reason}, acc -> acc # a failed or timed-out task doesn't take down the caller
    end)
  end
end

Efficiency

• Cost: a newly spawned process uses 327 words (233 of them heap), about 2.6 KB on a 64-bit VM (Erlang Efficiency Guide). All message data is copied between processes except refc binaries and literals on the same node (Efficiency Guide). The atom table is capped at 1,048,576 (system limits). Pure CPU work is order of 3-10x slower than Go or Java. Bound: message copying and mailbox scans; scheduler time for CPU-bound code.
• Levers:
  • spawn with a larger min_heap_size — fewer GCs and heap growth steps, for systems with comparatively few processes (Efficiency Guide) — long-lived busy processes
  • ETS tables for shared hot data — concurrent reads without copying through one process (order of) — caches and counters
  • keep big payloads as binaries over 64 bytes — shared by reference instead of copied (order of) — large messages
  • a Rust NIF on a dirty scheduler — native speed without blocking a normal scheduler (order of) — CPU hot paths
  • shard one GenServer by key — removes the single-mailbox bottleneck (order of) — hot global state
• Naive → standard → state of the art:
  • one GenServer for all state · every call serializes through one mailbox
  • process per connection or session, ETS for shared reads · flat latency, millions of processes per node
  • BEAM for the many-connection part, Rust NIF or port for the CPU kernel · each part on the runtime it suits

Build and deploy

• mix release builds a self-contained release (ERTS included) per target OS/arch; run in a container or with systemd.
• Clustering in Kubernetes via libcluster (DNS or K8s API strategies); set a strong distribution cookie and use TLS distribution across untrusted networks.
• Observability: :observer, recon, Telemetry + OpenTelemetry, LiveDashboard (Phoenix).
• Tooling: Mix, Hex packages, ExUnit, Credo, Dialyzer (success typing).

Pitfalls

• A single GenServer as a global bottleneck: every call serializes through one mailbox. Shard by key, or use ETS for reads.
• Unbounded mailboxes: producers faster than a consumer grow memory until the node dies; use call for backpressure or GenStage/Broadway.
• Atoms are never garbage-collected (default limit 1,048,576): never String.to_atom on user input.
• Long-running NIFs (> about 1 ms) block a scheduler; use dirty schedulers or ports.
• Holding small slices of large binaries keeps the large binary alive; :binary.copy/1 the slice.
• Distributed Erlang trusts all connected nodes fully (the cookie is not real security); never expose the distribution port.
• Expecting "let it crash" to fix bad external state: restarts loop until the supervisor gives up; add backoff.
• Selective receive scanning a huge mailbox (O(n) per receive).

Implementations

Sources

• Armstrong, Making reliable distributed systems in the presence of software errors, PhD thesis, KTH, 2003. thesis
• Erlang/OTP, Efficiency Guide docs and OTP Design Principles docs; System Limits docs.
• Cesarini, Vinoski, Designing for Scalability with Erlang/OTP, O'Reilly, 2016.
• Elixir team, Elixir documentation (GenServer, Supervisor, Task). docs
• Juric, Elixir in Action, Manning (3rd ed. 2024).
• Erlang/OTP, Erlang (what it is used for). site