# combd **Self-hosted Durable Objects, built from scratch in Rust.** Tiny sealed cells, packed tight. Light, but very strong. combd runs apps made of small WebAssembly programs, called cells. A cell can be a stateless HTTP handler, a durable object, or one process per WebSocket. A durable object is one per id, each with its own SQLite database. You say which in the app's manifest, `combd.toml`, and combd routes requests to them. It is one binary. It runs on one machine, or on a fleet that shares a bucket you own on S3 or R2. The bucket decides who owns each object. There is no control plane or consensus cluster to run. What it does: - **Every cell is sandboxed.** A cell has no system calls. It can't read files or open sockets. It gets only what the host hands it: its own SQLite, a clock, calls to other processes in its app, and Postgres through a pool the host runs. - **Cells take turns.** combd runs each cell a short slice at a time. A cell stuck in a loop can't starve its neighbors. - **Let it crash.** A cell that crashes takes only its own request down. For a durable object, that request's writes roll back, and the next request gets a fresh cell. - **An answer means it's saved.** A durable object answers only once its writes are safe. That is on disk on one node, and with its peers or the bucket in a fleet. See [Storage and durability](storage.md). - **Idle things are cheap.** Idle objects leave memory, and their state stays in SQLite. An idle WebSocket session sleeps, and its socket stays open. - **Two engines.** combd has its own WebAssembly interpreter, and a compiler that turns a module into safe Rust, then machine code. Each module says which it runs on. See [Compiled modules and builders](builders.md). Apps are written in Rust today, with [`combd-sdk`](rust-sdk.md). Other languages that compile to Wasm work at the host's ABI level, without an SDK. See [Other languages](languages.md). Start with the [Quickstart](quickstart.md), a page and a counter on your machine, then on a server. How combd is built and tested, and the numbers behind it, are in the repo: [`plan.md`](https://github.com/jtwebman/combd/blob/main/plan.md) and [`combd/RESULTS.md`](https://github.com/jtwebman/combd/blob/main/combd/RESULTS.md). combd is open source, under Apache-2.0. The code is on GitHub at [github.com/jtwebman/combd](https://github.com/jtwebman/combd). Every page is also plain markdown, at its name with `.md`. [`llms.txt`](/llms.txt) lists them, and [`llms-full.txt`](/llms-full.txt) has them all in one file, for tools that read docs. # combd in ten minutes You'll make an app with a page and a counter, run it on your machine, then run it on a server. The counter is a durable object, one per id, each with its own SQLite database. Each request to it is one transaction. ## 1. Install You need Rust, from rustup.rs, with its WebAssembly target: ```sh rustup target add wasm32-unknown-unknown ``` And combd. Once a release is out, take one for your machine from https://github.com/jtwebman/combd/releases: Linux x86_64 and arm64, or macOS arm64 and x86_64. Unpack it and put `combd` on your PATH. Or build it, with Rust 1.99 or newer: ```sh cargo install --locked --git https://github.com/jtwebman/combd combd \ --no-default-features --features aot-lite ``` `combd help` lists what it does. ## 2. Make an app, and run it ```sh combd new hello cd hello combd dev ``` `combd dev` builds the app's Rust for wasm32 into `modules/`, and serves it on port 8080. Open http://hello.localhost:8080 and click. Or: ```sh curl -X POST http://hello.localhost:8080/counter/demo/incr # {"count":1} curl -X POST http://hello.localhost:8080/counter/demo/incr # {"count":2} curl http://hello.localhost:8080/counter/other/ # {"count":0} ``` Stop it with Ctrl-C and start it again. The counts are still there, in `data/`. ## 3. What you made `combd.toml` says what runs and where requests go: ```toml [app] name = "hello" domains = ["hello.localhost"] [http.web] # a stateless process: the page module = "hello" routes = ["/"] [durable.counter] # a durable object per :id, with its own SQLite module = "hello" routes = ["/counter/:id/*"] ``` `src/lib.rs` is one function. A request comes in, a response goes out: ```rust fn handle(req: Request) -> Response { match (req.kind, req.method, req.route()) { ("api", "GET", "/") => Response::html(PAGE), ("do", "POST", "/incr") => { sql::run("CREATE TABLE IF NOT EXISTS counter (n INTEGER NOT NULL)", &[]).unwrap(); sql::run("INSERT INTO counter (rowid, n) VALUES (1, 1) ON CONFLICT (rowid) DO UPDATE SET n = n + 1", &[]).unwrap(); count() } ("do", "GET", "" | "/") => count(), _ => Response::text(404, "not found\n"), } } ``` `kind` is `api` for the page and `do` for the counter. A counter's `route()` is what comes after its id. Its SQL runs in its own database, `counter/demo` apart from `counter/other`. The request is one transaction. If `handle` panics, its writes are rolled back, and the next request finds the counter as it was. combd answers only once the writes are safe on disk. Each counter is a cell, a small sandboxed process. combd pauses any cell that runs too long, so a busy one can't starve the others. One that crashes starts again clean. ## 4. Change it Add a reset to `handle`: ```rust ("do", "POST", "/reset") => { sql::run("DELETE FROM counter", &[]).unwrap(); count() } ``` Run `combd dev` again, then: ```sh curl -X POST http://hello.localhost:8080/counter/demo/reset # {"count":0} ``` `combd-sdk` also has alarms (`alarm::set`), the clock (`now_ms`), and calls to other processes (`call`, `send`). An alarm comes back as an `ALARM` request. See [The Rust SDK](rust-sdk.md). Files that need no code, like images, styles, or a whole site, go in a `static/` folder beside `combd.toml`. See [Static files](static-files.md). ## 5. Run it on a server A node serves every app in a folder, laid out as `//`. On a Linux server with combd on it, and a domain pointing there: ```sh mkdir -p /srv/combd/apps/me/hello # from your machine: the manifest and the built module are all it needs # (and static/, if it has one) scp -r combd.toml modules server:/srv/combd/apps/me/hello/ ``` Put your domain in `combd.toml`'s `domains`, then start it with your certificate: ```sh COMBD_DATA=/srv/combd/data \ COMBD_TLS_CERT=/etc/letsencrypt/live/hello.example.com/fullchain.pem \ COMBD_TLS_KEY=/etc/letsencrypt/live/hello.example.com/privkey.pem \ COMBD_REDIRECT_PORT=80 \ combd serve-apps interp /srv/combd/apps 443 ``` `interp` runs the modules interpreted. `serve-apps` takes vars and secrets from a `combd.dev.toml` beside the manifest, if there is one. See [Apps and the manifest](apps.md#the-dev-environment). Compiled modules (`[modules.] engine = "compiled"`) are built by builders, which work through a fleet. On one node, keep them interpreted. See [Compiled modules and builders](builders.md). ## 6. More than one machine A fleet keeps objects in a bucket you own on S3 or R2, moves them to where their users are, and survives losing a node. Apps reach it with `combd deploy`, over a node's admin API. See [Deploying](deploying.md) and [Running a fleet](fleet.md). `examples/arena/cloud/` sets up three nodes on DigitalOcean with Terraform, with builders for compiled modules and systemd units. `examples/arena/README.md` walks through it. `plan.md` has the design and how it's tested, with a formal spec and a simulator that crashes everything. # Apps and the manifest An app is a folder: ```text hello/ ├── combd.toml # the manifest: what runs, and what reaches it ├── combd.dev.toml # your machine's vars and secrets (never in git) ├── modules/ │ └── hello.wasm # one file per module └── static/ # optional: files served as they are ``` The manifest is exactly one of `combd.toml`, `combd.yaml`, or `combd.json`, at most 1 MB. All three read into the same tree and get the same checks. combd reports every mistake at once. With any mistake, nothing starts and no deploy goes through. A whole one, with every section: ```toml [app] name = "shop" domains = ["shop.example.com"] [vars] LOG_LEVEL = "info" # a default; its type is string PG_MAX = { type = "int", min = 1, default = 8 } REGION = { type = "string" } # no default: every environment sets it "MY_FLAG?" = { type = "bool" } # ? = optional [secrets] PG_MAIN_URL = { secret = "pg-url", type = "url" } # env var -> secret name "SIGNING_KEY?" = "jwt-key" # secrets never have defaults [modules.shop] engine = "interpreted" # A process's type is its table: http, ws, durable, or pool. [http.api] module = "shop" # modules/shop.wasm routes = ["shop.example.com/api/*"] [durable.cart] module = "shop" routes = ["/cart/:id/*"] [pool.main] # code names it: {"db": "main"} module = "combd:postgres" url = "$PG_MAIN_URL" connection.timeout_ms = 3000 max = "$PG_MAX" callers = ["api", "cart"] ``` The top level takes only `app`, `vars`, `secrets`, `modules`, `http`, `ws`, `durable`, and `pool`. ## `[app]` | Key | What it is | |---|---| | `name` | The app's name. Lowercase letters, digits, `-` and `_`, at most 63. Required. | | `domains` | The hosts its routes answer on, like `["shop.example.com"]`. | ### Domains A route with no host matches only the app's own domains. An app with no domains answers on any host, but only while it's alone on the node. With more than one app, each names its domains, and no two apps may share one. A route that names a host must use one of the app's domains. The org isn't in the manifest. It comes from who deployed the app with `combd deploy `, or on a single node from the app's folder, `///`. Everything an app owns is scoped to `/`: its objects, its pools, its calls. ## `[vars]` Vars are typed settings, given to every process as environment variables. Each one is a default, or a table that says its type: ```toml [vars] LOG_LEVEL = "info" # string, from the default RETRIES = 3 # int RATIO = 0.5 # float VERBOSE = false # bool PG_MAX = { type = "int", min = 1, max = 64, default = 8 } TIER = { type = "enum", values = ["free", "pro"], default = "free" } LAUNCH = { type = "datetime", min = "2026-01-01T00:00:00Z" } CALLBACK = { type = "url" } SLUG = { type = "string", pattern = "^[a-z-]+$", max = 40 } "BETA?" = { type = "bool", description = "turns on the beta page" } ``` Types: - `string`, with an optional regex `pattern` - `int`, `float`, `bool` - `datetime`, RFC 3339 with an offset, like `2026-10-15T08:00:00Z` - `url`, `scheme://...` - `enum`, with `values` `min` and `max` go with `int`, `float`, `datetime`, and `string`, where they bound its length in characters. A table may have a `description`. A var with no default must be set in every environment, unless its name ends in `?`, which makes it optional. An optional var can't have a default, since it would never be missing. Names are `A-Z`, `0-9` and `_`, not starting with a digit. `COMBD_` names are the host's. Every process also gets `COMBD_ORG`, `COMBD_APP`, `COMBD_PROCESS` (its own name), and `COMBD_NODE`. Set them on a fleet with `combd vars`. See [Deploying](deploying.md). While a version is live, a value is checked against its type before it's taken. ## `[secrets]` A secret is an env var whose value comes from the secret store, never from the manifest. Each one names the secret it comes from: ```toml [secrets] PG_URL = { secret = "pg-url", type = "url" } # PG_URL gets secret "pg-url" "SIGNING_KEY?" = "jwt-key" # a string, optional ``` A secret's type is `string` unless it says otherwise. It never has a default, which would be in git and in every environment. Secret names are `a-z`, `0-9`, `-`, `_` and `.`, up to 128. A name can't be in both `[vars]` and `[secrets]`. Errors about a secret never show its value. Set them on a fleet with `combd secret put`. See [Deploying](deploying.md). ## Using vars and secrets in settings A process's setting can name a var or a secret with `$`: ```toml [pool.main] module = "combd:postgres" url = "$PG_URL" max = "$PG_MAX" callers = ["api"] ``` The var must be declared, and of exactly the type the setting wants. No conversions. A required setting can't take an optional var. `$$` is a literal `$`. The value is checked again when the app starts, so a var whose own limits are wider than the setting's can't sneak past it. ## Processes A process is a table under its type, named by you: `[http.web]`, `[ws.viewer]`, `[durable.counter]`, `[pool.main]`. A name is one process, whatever its type. Names are lowercase letters, digits, `-` and `_`. | Type | One instance per... | See | |---|---|---| | `http` | request (a warm cell from a pool) | [Processes](processes.md#http-workers) | | `durable` | id, with its own SQLite | [Processes](processes.md#durable-objects) | | `ws` | WebSocket, in memory | [Processes](processes.md#websocket-sessions) | | `pool` | a Postgres pool the host runs | [Postgres pools](postgres.md) | Every process but a pool runs a module of the app. `module = "shop"` is `modules/shop.wasm`. Many processes can share one module, loaded once. Each process needs `routes` from the internet, `callers` from other processes of the app, or both. One with neither is a mistake. ### `routes` A route is `/path` for the app's domains, or `host/path` for that host: ```toml routes = ["/", "/api/*", "shop.example.com/admin/*"] ``` - `:name` captures a segment. A durable object's route must name it `:id`, like `/cart/:id/*`. That segment is the object's id. - `*` at the end takes the rest of the path. - Segments use letters, digits, `-`, `.`, `_` and `~`. At most 256 bytes. - Without a `*`, a route matches only that many segments. - The query string isn't part of matching. - Two routes of the same shape in one app are a mistake. When more than one matches, the most specific wins. A route with a host beats one without. Then, segment by segment, a literal beats a capture, which beats `*`. A `ws` process starts when a socket connects, so it must have a route. A pool has no routes. Its callers reach it. ### `callers` Which of the app's processes may `call` or `send` to this one: ```toml [durable.player] module = "game" callers = ["lobby", "match"] ``` A call from anyone else is denied. For a pool, `callers` is required and lists which processes may use it. ### Settings Each type takes these, all optional, checked against these ranges. **`http`** | Setting | Default | What it does | |---|---|---| | `timeout_ms` | 30000 (1 to 900000) | The longest a `call` to it waits for an answer. | | `max_concurrent` | 32 (1 to 1000) | For an event cell: requests one instance takes at once. | | `fresh` | `false` | With `true`, every request gets a new cell. Nothing one leaves reaches the next, and nothing stays warm. | | `memory_kb` | none (64 to 4194304) | The most memory a cell may grow to. | **`durable`** | Setting | Default | What it does | |---|---|---| | `timeout_ms` | 30000 (1 to 900000) | The longest a `call` to it waits for an answer. | | `mailbox` | 1000 (1 to 1000000) | Requests that may wait for an object. Past it, new ones get `503 {"error":"overloaded"}`. | | `batch` | 100 (1 to 10000) | The most waiting requests an object takes at once, if its module exports `handle_batch`. | | `placement` | `"nearest"` | In a fleet: `nearest`, `first`, `spread`, or `pin`. See [Running a fleet](fleet.md#placement). | | `pin` | none | Required for `placement = "pin"`. Regions or areas, like `"fra1,ams3"` or `"eu"`. | | `fallback` | `"nearest"` | For `pin`. `nearest` runs elsewhere while none of its regions is up. `none` waits. | | `scope` | `"region"` | For `spread`: `zone`, `region`, or `fleet`. | | `memory_kb` | none (64 to 4194304) | The most memory a cell may grow to. | **`ws`** | Setting | Default | What it does | |---|---|---| | `timeout_ms` | 30000 (1 to 900000) | The longest a `call` to it waits for an answer. | | `mailbox` | 1000 (1 to 1000000) | Events that may wait for a session. Past it, `503 overloaded`. | | `hibernate_ms` | 5000 (0 to 86400000) | A session idle this long sleeps. 0 means after every event. | | `hibernate` | `"auto"` | `auto` has the host keep what the cell changed. `app` has the app hand back its state. `off` stays awake. | | `on_close` | `"wake"` | `wake` lets the session hear `WSCLOSE`. `drop` ends it without waking. | | `state_kb` | 4 (1 to 1024) | For `hibernate = "app"`, the most state it may keep. | | `memory_kb` | none (64 to 4194304) | The most memory a cell may grow to. | **`pool`** is in [Postgres pools](postgres.md#the-manifest). ## `[modules.]` How a module runs: ```toml [modules.shop] engine = "compiled" # or "interpreted" ``` A module not listed runs interpreted. `engine` is the only key, and the module must exist in `modules/`. A compiled module is built by builders, and a deploy waits for its builds. See [Compiled modules and builders](builders.md). ## The dev environment On your machine, vars and secrets come from `combd.dev.toml`, `.yaml`, or `.json` beside the manifest. It takes only `[vars]`, by env var name, and `[secrets]`, by secret name: ```toml [vars] PG_MAX = 4 [secrets] pg-url = "postgres://me:pw@localhost/shop" ``` `combd dev` and `combd serve-apps` read it. It holds secrets, so combd refuses it if git tracks it. `git rm --cached` it and put it in `.gitignore`. It never goes up with a deploy. ## Limits - A var or secret value: at most 16 KB. - All of a process's env vars together: at most 256 KB. - At most 512 vars and secrets. - A `pattern`: at most 1024 bytes. # Static files An app can serve files as they are: pages, scripts, styles, images. Put them in a `static/` folder next to `combd.toml` and `modules/`. There is no manifest setting for it. If `static/` is there, combd serves it. ```text site/ ├── combd.toml ├── modules/ │ └── site.wasm # optional: a static-only app has no modules └── static/ ├── index.html ├── 404.html ├── app.js ├── app.js.br # optional: a precompressed copy ├── app.js.gz └── guide/ └── index.html ``` ## What gets served Routes in the manifest come first. A `GET` or `HEAD` that no route matches is looked up in `static/`: | Request | What it gets | |---|---| | `/app.js` | `static/app.js` | | `/` or `/guide/` | that folder's `index.html` | | `/guide` (a folder with an `index.html`) | a `301` to `/guide/`, so the page's relative links work | | anything else | `static/404.html` with status 404 if it's there, else the plain 404 | The 404 page is the one at the top of `static/`. Other methods, like `POST` and `PUT`, never reach the static files. With no route, they get the plain 404. Static files follow the app's [domains](apps.md#domains) like its routes do. A request reaches an app's files only on its domains, or on any host for an app alone on a node with no domains. ## Headers Every file gets: - `content-type`, by its extension, such as `.html`, `.css`, `.js`, `.json`, `.svg`, `.png`, `.woff2`, `.wasm`, `.pdf`, and more. Anything unknown is `application/octet-stream`. - An `etag`. A browser that sends a matching `if-none-match` gets a `304` with no body. - `cache-control: no-cache`. Browsers keep the file but ask again each time, and the answer is usually that cheap `304`. - `x-content-type-options: nosniff`. ## Compressed copies combd doesn't compress files. To send a compressed copy, put it beside the file: `app.js.br`, `app.js.gz`, or both. A browser whose `accept-encoding` takes `br` gets the `.br` copy. Otherwise, one that takes `gzip` gets the `.gz`. Either way, combd sets `content-encoding` and `vary: accept-encoding`. Each copy has its own ETag. Anyone else gets the plain file. The `.br` and `.gz` files aren't served under their own names. ```sh brotli -k static/app.js # static/app.js.br gzip -k static/app.js # static/app.js.gz ``` ## In memory A version's files are read once, when it loads, and held in memory. A request never waits on a disk read. A new deploy brings its own files. ## Names and limits The files go up with the app in its [deploy bundle](deploying.md). Their paths must be plain names: - letters, digits, `.`, `-` and `_` - at most 16 folders deep, and 255 characters in all A deploy refuses any file that breaks these rules, and names it. Hidden files, whose names start with `.` (`.DS_Store`, `.env`), are left out. A bundle holds at most 4096 files and 64 MB in all, modules and manifest included. ## A site with no code An app can be only static files: a manifest with no processes, and a `static/` folder. This site is one. ```toml [app] name = "docs" domains = ["docs.example.com"] ``` ```sh combd deploy ./docs acme https://node.example.com:9443 ``` `combd dev` builds the app's Rust crate first, so it needs one. To try a static-only app on your machine, serve its folder's parent with `serve-apps`, laid out as `//`: ```sh mkdir -p apps/local && cp -r docs apps/local/docs combd serve-apps interp apps 8080 curl -H 'host: docs.example.com' http://127.0.0.1:8080/ ``` ## A page and an API Files and routes mix. The routes take what they match, and the files the rest. ```toml [app] name = "shop" domains = ["shop.example.com"] [http.api] module = "shop" routes = ["/api/*"] ``` Here `/api/cart` goes to the `api` process, and `/`, `/app.js`, and `/about/` come from `static/`. # Processes Every process in an app runs cells, small sandboxed instances of one of its modules. What differs is how many there are, how long they live, and what they keep. ## Requests and answers A cell gets one line of text, and gives back one: ```text [ ] ``` - `kind` is `api` for an `http` process and `do` for a durable object or a WebSocket session. - `METHOD` is the HTTP method or an event. combd takes `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, and refuses others. The events are `ALARM`, `TICK`, `WSOPEN`, `WSMSG`, `WSBIN`, `WSCLOSE`, `HIBERNATE`, `WAKE`. - For an `http` process, `path` is the whole path, with its query string. For a durable object, it's what comes after the id in its route, with the query string. `/counter/:id/*` and `/counter/demo/incr?n=2` give `/incr?n=2`. - `body` is the request's body, if any. The cell doesn't see request headers. The answer is ` `, or with headers: ```text 200 content-type: text/html; charset=utf-8 cache-control: no-store

hello

``` combd passes on only these headers: `content-type`, `cache-control`, `location`, `set-cookie`, `etag`, `last-modified`, `vary`, and `access-control-*`. [`combd-sdk`](rust-sdk.md) reads and writes all of this. ## HTTP workers An `http` process is stateless. Each request takes a warm cell from a pool on its event loop, runs, and gives the cell back. A burst grows the pool, which then shrinks to what's been needed lately. A cell that crashes is dropped, and the next request gets a fresh one. With `fresh = true`, every request gets a new cell, so nothing one leaves in memory reaches the next. ### Event cells An `http` process whose module exports `on_request` is an event cell. One instance takes many requests at once, like a Node process, and runs them while they wait on timers, Postgres, or calls. Thirty requests that each wait 400 ms finish in about 400 ms on one instance. - `max_concurrent` (default 32) is how many requests one instance takes before combd starts another. - A request may be answered early with `respond`, and work go on after it, like `waitUntil`. - A request gets 30 s to be answered, then a `504`, and 60 s in all, work after the answer included. - The cost: a crash fails every unanswered request on that instance, and work after an answer is best effort. Event cells are an ABI-level feature today. The Rust SDK doesn't wrap them. See [Other languages](languages.md#event-cells) and `guest-ev/` in the repo. ## Durable objects A `durable` process has one object per id. The id comes from the route's `:id` segment, or from a `call` that names it. Each object: - runs **one request at a time**, in a warm cell, with its own SQLite database. - makes each request **one transaction**. If the cell crashes from a panic or a trap, the request's writes roll back and the cell is dropped. The next request gets a fresh one that loads from storage. An answer with an error status is still an answer, so its writes commit. - answers only once its writes are safe. See [Storage and durability](storage.md). - leaves memory when idle, and comes back from storage when needed. Requests that arrive while an object is busy wait in its mailbox, up to `mailbox` (default 1000). Past it, new ones get `503 {"error":"overloaded"}` at once. A busy object sheds load instead of piling it up. ### Batches A module that exports `handle_batch` gets the requests waiting behind the first one all at once, up to `batch` (default 100). That is one run, one transaction, and an answer each. Each request is framed as its length, a newline, then itself. The answers come back framed the same way. Batches form only from a queue, so they grow with the load. WebSocket events, alarms, and ticks always go one at a time. ### Alarms An object can set one alarm, a time in ms since the epoch. Setting it changes the object's storage, so it commits with the request. A request that crashes doesn't set it. When it's due, the object gets an `ALARM` request, `do ALARM / `. The alarm is cleared as it fires, and the handler can set the next one. An alarm whose handler crashes is tried again after 500 ms, then twice as long each time, up to 6 tries. Then it's dropped. In a fleet, an index in the bucket finds due alarms of objects that aren't running anywhere. The fleet fires them as ordinary requests, routed to the object's owner. A fire from the index that keeps failing backs off, up to a minute between tries. ### Ticks `tick_every(ms)` gives an object a beat, a `TICK` request every `ms`, from 10 ms to 60 s. 0 stops it. It's in memory only. A crash, or a move to another node, stops it until the object asks again. It's for game loops. Use an alarm for work that must happen. ### WebSockets on an object A route to a durable object can take a WebSocket handshake. The object gets `WSOPEN`, with the path and the new socket's id, and accepts it by answering `101`. Then: | Event | The request | |---|---| | a text message | `do WSMSG / ` | | a binary message | `do WSBIN / `, the bytes by `ws_msg` | | the socket closed | `do WSCLOSE / ` | The object holds many sockets, like a chat room. `ws_list` gives their ids, `ws_send` and `ws_send_binary` send to one, and `ws_close` closes one. What a request sends goes out only if the request succeeds, and only once its writes are safe. The object's sockets are kept in its storage. When it goes quiet and leaves memory, its sockets stay open, and the next message wakes it. Pings never wake it. The Rust SDK doesn't wrap the WebSocket calls yet. See [Other languages](languages.md#the-host-abi). ## WebSocket sessions A `ws` process has one session per WebSocket. It starts when a socket connects on its route, lives in memory on the event loop that holds the socket, and ends when the socket does. It has no storage. It gets the same events: `WSOPEN`, `WSMSG`, `WSBIN`, `WSCLOSE`. Answering `WSOPEN` with `101` accepts it. A crash ends the session and closes its socket. **Sleeping.** A session idle for `hibernate_ms` (default 5 s) sleeps: - `hibernate = "auto"`, the default. The host keeps what its cell changed, and wakes the same cell where it was. - `hibernate = "app"`. The session gets `do HIBERNATE /` and answers `200 `, at most `state_kb`. Its cell goes. The next event first brings `do WAKE / `, with the state by `ws_msg`. - `hibernate = "off"`. It stays awake. With `on_close = "drop"`, a closed socket's session ends without waking. That's cheap when everyone leaves at once, as in a deploy or a network blip. **Reaching a session.** Another process reaches a session with `call` or `send`, naming its socket id as `"id"`. With `"ws": ""` instead of a request, the text goes straight to the session's socket without waking it. `"ids": [...]` sends one text to many sessions' sockets at once. That's how One Million Checkboxes' chunks tell their watchers about a change ([`examples/million`](https://github.com/jtwebman/combd/tree/main/examples/million)). ## Calls between processes A cell can ask another process of its app and wait, with `call`, or tell it and go on, with `send`. The message is JSON: ```json {"to": "player", "id": "ann", "method": "POST", "path": "/won", "body": "{\"by\":3}"} ``` | Field | What it is | |---|---| | `to` | The process. It must list the caller in its `callers`. | | `id` | The object's id (durable), or the session's socket id (`ws`). | | `method` | Default `POST`. | | `path` | Default `/`. | | `body` | Text. | | `timeout_ms` | How long to wait, at most the target's own `timeout_ms`. | | `node` | For a durable object in a fleet, start it on that node, or reach it there. | | `ws`, `ids` | For a `ws` process, text straight to sockets. | `call` gets back `{"status", "body"}`, or `{"error"}` if denied or timed out. A waiting cell is paused and costs no CPU. `send` goes out only if the request that sends it succeeds. For an object, that's once its writes are safe. It's refused when too much is queued, past 1 MB a message or 16 MB a request in all. A process may list itself in its own `callers`, to send itself a message after it answers. # Storage and durability Each durable object has its own SQLite database. A request is one transaction, and the object answers only once its writes are safe. What "safe" means depends on whether combd runs on one node or a fleet. ## SQL from a cell A cell sends one statement at a time, as JSON, and gets JSON back: ```json {"q": "INSERT INTO posts (title) VALUES (?)", "p": ["hello"]} ``` ```json {"rows": [[1, "hello"]], "changes": 1} ``` An error comes back as `{"error": "..."}`. In Rust, [`sql::run`](rust-sdk.md#sql) does this. SQLite runs in the host, so a cell's SQL is fenced in: - No `ATTACH`, no `PRAGMA`, and no `BEGIN`, `COMMIT`, or `SAVEPOINT` of its own. The host owns the transaction. - No tables named `_combd...`. Those are the host's own, like the alarm. - A string or blob value is at most 2 MB. An object's database is at most 1 GB, and a statement or its answer at most 16 MB. - One statement runs at most 100 ms, or about 10 million SQLite steps. The cell's own budget doesn't count SQLite's CPU. Only durable objects have storage. An `http` process or a WebSocket session that calls `sql` gets `{"error":"this cell has no storage"}`. ## One request, one transaction The host opens a transaction before each request and commits it after. - If the cell answers, with any status, its writes commit. - If the cell crashes, its writes roll back and the cell is dropped. The next request starts a fresh one from storage. A crash never leaves half a request behind. - If SQLite itself rolls the transaction back, say when out of space, combd crashes the request, and it fails as a whole. An alarm set in a request commits with it. WebSocket messages and `send`s from a request go out only after it commits and its writes are safe. ## Durability, in steps combd builds durability in steps, each its own safe system with its own spec and simulator. | Step | When | An answer leaves once... | |---|---|---| | **D1** | one node (`combd dev`, `serve-apps`) | an fsync has put its writes on the node's disk | | **D2** | a fleet with `COMBD_REPLICAS=0`, the default | the bucket has its writes, and a fresh read of the object's ownership record shows this node still owns it | | **D3** | a fleet with `COMBD_REPLICAS=2`, say | every one of the object's peers has its writes on disk. With a peer down, the D2 way instead. | ### D1: one node SQLite runs in WAL mode, and commits don't fsync. A pool of threads fsyncs each object's WAL off the event loop. Every answer is held until an fsync has covered its commit. One fsync covers every commit made before it started. So a busy object pays one fsync for many requests, and many objects sync at once. A read waits too, if it may have seen a write that isn't on disk yet. A new object's folders are fsynced, so a power cut can't lose its file. On one node, the data lives only on that node's disk. ### D2: a fleet, proven by the bucket Several nodes share one bucket. The bucket decides who owns each object, by conditional writes. A request for an object another node owns is forwarded there. An object's changed SQLite pages, read from the WAL, go to the bucket as a segment. An answer leaves only once its segment is uploaded and a fresh read of the ownership record shows this node still owns the object. Nothing acknowledged is lost if a node dies. The next owner restores from the bucket. Many writes share one upload, but each answer waits on the bucket. ### D3: a fleet, proven by peers With `COMBD_REPLICAS` set, each object has that many peer nodes, picked by hash. A write is acknowledged once every peer has it on disk, and the bucket gets it in batches, in the background. If a peer is down, a write is acknowledged the D2 way. While every peer is up, uploads wait up to `COMBD_UPLOAD_MS` (10 s), or until `COMBD_UPLOAD_BYTES` (1 MB, at most 16 MB) build up. So an object's newest writes live only on its peers for up to that long. Losing every peer's disk in that window loses them. The arena's fleet runs D3 with `COMBD_REPLICAS=2`. ### Reads In a fleet, a read waits for a round to the object's peers, as proof that no newer owner has taken over. With `COMBD_READ_LEASE=1`, owners answer reads with no round while their peers' promises hold. Every node promises. Turn the lease on only once every node runs a version that does. ### Taking over An owner that goes silent keeps its objects for `COMBD_LEASE_MS` (10 s by default). Then another node can claim them and restore them from the bucket, and from the peers with D3, with every acknowledged write. Before it serves, every node tests its bucket. 16 writers race one conditional write, and exactly one may win. A node refuses to run on a bucket that fails. Not every "S3-compatible" server passes. One popular server let several writers win. `combd bucket-check` runs the same test by itself. ### How it's checked Each part is checked four ways: - a TLA+ spec, in `spec/` - a deterministic simulator, `combd sim`, with crashes, lost messages, network splits, and lost disks - a real cluster with kill tests - known-bad versions that must fail every one of them The details are in `plan.md` and `combd/RESULTS.md`. ## What isn't durable Only what an object writes to its SQLite is kept. These are not: - **WebSocket sessions** in `ws` processes live in memory only. If their node dies, they're gone with their sockets. - **Ticks** from `tick_every`, and anything an object keeps only in memory, stop with a crash or a move. - **An event cell's work after its answer** is best effort. The arena keeps a match's game state in memory on purpose. If its node dies mid-match, that match is lost. The players' records are in SQLite. ## Disk and memory - An idle object leaves memory after `COMBD_IDLE_MS` (60 s), or when a loop holds more than `COMBD_MAX_OBJECTS` (10,000). Its data stays. - In a fleet, an object's local file goes with the object, since the bucket has it. - With less free disk than a tenth of the data disk or 2 GB, whichever is less, a node takes no new objects. A request that would make one gets `507 {"error":"no room for new objects"}`. - Data lives under `COMBD_DATA` (default `data`), one file per object, named by a hash of its name, in a folder per app, `//`. # Postgres pools Cells reach Postgres through a pool the host runs. A cell never holds a socket or a password. The node keeps the connections, and the URL lives in a secret. A cell waiting on the database is paused and costs no CPU. ## The manifest A pool is a process of type `pool`, with `module = "combd:postgres"`: ```toml [secrets] PG_URL = { secret = "pg-url", type = "url" } [vars] POOL_MAX = { type = "int", min = 1, default = 8 } [http.api] module = "x" routes = ["/api/*"] [pool.main] module = "combd:postgres" url = "$PG_URL" max = "$POOL_MAX" callers = ["api"] ``` Code names the pool `main`. Only processes in the required `callers` list may use it. A pool has no routes. | Setting | Default | What it does | |---|---|---| | `url` | required | The database URL, of type `url`, usually `$SECRET`. Add `sslmode=require` for TLS. | | `max` | 8 (1 to 10000) | The most connections, for the whole fleet. | | `wait_ms` | 5000 (1 to 600000) | At `max`, how long an ask waits in line before it fails. | | `lease_ms` | 60000 (1 to 86400000) | The longest a session may hold a connection before it's taken back. | | `idle_ms` | 60000 (0 to 86400000) | A connection free this long closes. | | `session_idle_ms` | 60000 (1 to 86400000) | An object's session unused this long is released. | | `statement_timeout_ms` | 5000 (1 to 3600000) | Every statement's timeout, lock waits included. | | `connection.timeout_ms` | 5000 (1 to 600000) | How long to connect. | | `connection.retries` | 3 (0 to 10) | Connect attempts after the first, with backoff. | | `on_manager_loss` | `"finish"` | When the pool's manager changes, older connections `finish` their lease or `stop` now. | ## Pooled statements A pooled statement runs on whichever connection is free, as its own transaction. In Rust: ```rust use combd_sdk::{json, pg, Response}; match pg::query("main", "SELECT id, body FROM posts WHERE author = $1", &[json!("ann")]) { Ok(rows) => Response::json(200, &json!(rows)), Err(e) => Response::text(500, &e), } ``` Parameters are `$1`, `$2`, .... The answer is the same JSON as an object's own SQLite, `{"rows": [[...]], "changes": n}`, or `{"error": "...", "code": ""}`. In Rust an error is the SQLSTATE, then the message, like `23505 duplicate key value ...`. A pooled statement's connection is not reset after it, as with PgBouncer's transaction mode and the pools apps keep themselves, like Postgrex, Rails, and Prisma. What a statement changes in the session, like a `SET`, a session advisory lock, or a temp table, stays for whichever statement uses that connection next. Undo it in the app, or use a session. ## Sessions A session leases one connection, so everything session-scoped works on it: transactions over many statements, row locks, advisory locks, `SET`. The host calls are `pg_open` for a handle, `pg_query` on it, and `pg_close`. The Rust SDK doesn't wrap them yet. See [Other languages](languages.md#the-host-abi). - An `http` request's sessions end with the request. - A durable object's sessions last across its requests, until it closes them, it crashes, it leaves memory, or a session sits unused for `session_idle_ms`. A durable object can hold a transaction open across many messages. - A WebSocket session (`ws`) can't open one. It uses pooled statements. - An event cell uses `pg_open_async` and friends. When a session is closed, crashes, or passes `lease_ms`, its connection is reset with a rollback, then what `DISCARD ALL` does. Every lock is released. The holder's next call after `lease_ms` gets an error. ## How the pool behaves - It starts a connection when it's asked and none is free, up to `max`. - At `max`, an ask waits in line, and fails after `wait_ms`. A session waiting for a whole connection goes ahead of pooled statements. - A connection free for `idle_ms` closes. - "Too many clients" from the database makes an ask wait, not fail. ## One pool for the fleet A pool is one for the whole fleet, not one per node. Run 1000 nodes against a Postgres that takes 100 connections, set `max = 100`, and it never sees more than 100. - Every node takes part in every pool. One node is the pool's manager, which keeps the books. The catalog names it, and a new one takes over if its node goes quiet. - A connection starts on the node of the cell that asked. At `max`, a free connection on the nearest node is leased instead, and its statements go over the wire. - Once a second, each node tells the manager what its connections are doing, so a lost message can't leak one. - When the manager changes, older connections finish their lease or stop, as `on_manager_loss` says. Until they're gone, the database may see more than `max` for a moment. ## TLS With `sslmode=require` in the URL, a connection is TLS or nothing. The server's certificate is checked against the public roots. libpq's `require` doesn't check that. A database with a private CA can't be checked yet. Without `sslmode=require`, connections are plain. ## Example [`examples/x`](https://github.com/jtwebman/combd/tree/main/examples/x) is an X clone on Postgres: users, posts, follows, likes, a home timeline. It has one `http` process and one pool. # The Rust SDK `combd-sdk` is a small crate for writing a combd app in Rust: a request in, a response out, plus SQLite, Postgres, alarms, the clock, env vars, and calls between processes. It lives in the repo at `sdk/rust/`. `combd new` puts it in your `Cargo.toml`: ```toml [lib] crate-type = ["cdylib"] [dependencies] combd-sdk = { git = "https://github.com/jtwebman/combd" } [profile.release] opt-level = "s" lto = true panic = "abort" strip = true ``` Build for `wasm32-unknown-unknown`, added with `rustup target add wasm32-unknown-unknown`. `combd build` runs `cargo build --release --target wasm32-unknown-unknown` and copies the result to `modules/.wasm`, where `module = ""` finds it. ## A handler ```rust use combd_sdk::{cell, json, sql, Request, Response}; fn handle(req: Request) -> Response { match (req.kind, req.method, req.route()) { ("api", "GET", "/") => Response::html("

hi

"), ("do", "POST", "/incr") => { sql::run("CREATE TABLE IF NOT EXISTS n (v INTEGER)", &[]).unwrap(); sql::run("INSERT INTO n VALUES (1)", &[]).unwrap(); let rows = sql::run("SELECT count(*) FROM n", &[]).unwrap(); Response::json(200, &json!({ "count": rows[0][0] })) } _ => Response::text(404, "not found"), } } cell!(handle); ``` `cell!(handle)` wraps your function in the exports the host calls, `alloc`, `release`, and `handle`. Any name works. It also sends a panic's message to stderr, which combd logs beside the crash. With `panic = "abort"`, a cell would otherwise die without a word. One module can serve many processes. Match on `req.kind`, or read `COMBD_PROCESS` with `env`. ## `Request` | Field or method | What it is | |---|---| | `kind` | `"api"` (an `http` process) or `"do"` (a durable object or a WebSocket session). | | `method` | `"GET"`, `"POST"`, ..., or an event: `"ALARM"`, `"TICK"`, `"WSOPEN"`, `"WSMSG"`, `"WSCLOSE"`, ... | | `path` | The path, with its query string, if any. | | `body` | The body, as text. | | `route()` | The path without its query string. | | `query(name)` | A query parameter, as written, not decoded. | | `json()` | The body as JSON, or `Value::Null` if it isn't. | See [Processes](processes.md#requests-and-answers) for what each kind of process gets. ## `Response` ```rust Response::text(404, "not found") // text/plain; charset=utf-8 Response::json(200, &json!({ "ok": true })) Response::html("

hi

") // 200, text/html; charset=utf-8 Response::json(201, &v).header("location", "/posts/7") ``` The host passes on only `content-type`, `cache-control`, `location`, `set-cookie`, `etag`, `last-modified`, `vary`, and `access-control-*`. ## `sql` The durable object's own SQLite. Each request is one transaction. ```rust let rows = sql::run("SELECT title FROM posts WHERE id = ?", &[json!(7)]); // Result>, String> ``` `run` takes one statement, with `?` for each parameter, and returns the rows (`Vec>`) or the error. Only durable objects have storage. See [Storage and durability](storage.md). ## `pg` A statement on a pooled Postgres connection, with `$1`, `$2`... for parameters: ```rust let rows = pg::query("main", "SELECT body FROM posts WHERE author = $1", &[json!("ann")]); ``` `"main"` is the pool's name in the manifest, and this process must be in its `callers`. An error starts with its SQLSTATE. Each statement is its own transaction, and its connection isn't reset after it. See [Postgres pools](postgres.md). ## `alarm` ```rust alarm::set(combd_sdk::now_ms() + 60_000); // in a minute alarm::get(); // Some(ms), or None alarm::clear(); ``` One alarm per object, in ms since the epoch. It commits with the request, and fires as an `ALARM` request. ## `call` and `send` ```rust let answer = combd_sdk::call(&json!({ "to": "player", "id": "ann", "path": "/stats" })); // {"status": 200, "body": "..."} or {"error": "..."} combd_sdk::send(&json!({ "to": "player", "id": "ann", "path": "/won", "body": "{}" })); ``` `call` waits for the answer. `send` doesn't, and goes out only if this request succeeds. It returns `false` if too much is queued. The target must list this process in its `callers`. The fields are in [Processes](processes.md#calls-between-processes). ## `env` and `now_ms` ```rust let level = combd_sdk::env("LOG_LEVEL").unwrap_or("info".into()); let me = combd_sdk::env("COMBD_PROCESS"); let now = combd_sdk::now_ms(); // ms since the epoch ``` `env` reads the app's vars and secrets, and the host's `COMBD_ORG`, `COMBD_APP`, `COMBD_PROCESS`, and `COMBD_NODE`. The clock is coarse on purpose, to 1 ms. `json!`, `Value`, and `to_json` come from `serde_json`. ## Not in the SDK yet These work at the host ABI level, covered in [Other languages](languages.md#the-host-abi), and the repo's examples use them directly: - WebSockets: `ws_send`, `ws_send_binary`, `ws_close`, `ws_list`, `ws_msg` - `tick_every` - Postgres sessions: `pg_open`, `pg_query` with a handle, `pg_close` - event cells: `on_request`, `respond`, ... - `handle_batch` # Other languages A combd module is a WebAssembly module that speaks the host's small ABI. Rust has an SDK, [`combd-sdk`](rust-sdk.md). Any other language that compiles to Wasm can write a module against the ABI below. ## What's been run | Language | Toolchain | Status | |---|---|---| | Rust | `wasm32-unknown-unknown` | Apps, with `combd-sdk`. | | C | `zig cc -target wasm32-wasi -mexec-model=reactor` | The test cell. | | Zig | `zig build-exe -target wasm32-wasi -fno-entry -rdynamic` | The test cell. | | Go | TinyGo, `-target=wasip1 -buildmode=c-shared` | The test cell. | | AssemblyScript | `asc --exportStart _initialize` | The test cell. | | JavaScript | | Not yet. A Workers-compatible library over combd's host, compiled to Wasm, is next. | [`langs/`](https://github.com/jtwebman/combd/tree/main/langs) has the same small cell in each language, built in Docker with `docker build -f langs/Dockerfile -o langs/out langs`. Each exports `alloc` and `handle`, calls one host function, prints, spins forever, and panics. `cargo run --release -- langs`, in `combd/`, checks each on combd's engines. ## The host ABI ### What a module exports | Export | Signature | What it does | |---|---|---| | `alloc` | `(len: u32) -> ptr` | Memory for a request of `len` bytes. The host writes the request there. | | `handle` | `(ptr, len: u32) -> u64` | Handle one request. Returns the answer as `(ptr << 32) \| len`. | | `release` | `(ptr, len: u32)` | Optional. Frees an answer once the host has read it. | | `handle_batch` | `(ptr, len: u32) -> u64` | Optional, for durable objects. Many requests at once. | | `_initialize` | `()` | Optional. A reactor's start, run once first. | | `on_request`, `on_event` | | An event cell, in place of `handle`. | The request is the line ` [ ]`. The answer is ` `, or `\n: \n...\n\n` with headers. See [Processes](processes.md#requests-and-answers). `handle_batch` gets each request as its length in decimal, a newline, then the request, one after another. It answers each the same way, in order, all in one transaction. ### What the host gives it Imports from the `combd` module. A call that waits, like Postgres or `call`, pauses the cell and costs no CPU meanwhile. | Import | What it does | |---|---| | `sql(ptr, len) -> u32` | Run one statement, as `{"q", "p"}` JSON, on the object's SQLite. Returns the answer's length. | | `sql_read(ptr)` | Copy that answer into the cell. | | `now_ms() -> i64` | The wall clock, in ms since the epoch. | | `alarm_set(at_ms: i64)` | Set the object's alarm. 0 or less clears it. | | `alarm_get() -> i64` | The alarm, or -1 for none. | | `tick_every(ms: u64)` | A `TICK` every `ms`, 10 to 60000. 0 stops it. | | `ws_send(id: u64, ptr, len) -> u32` | Queue a text message to a socket. 0 OK, 1 no such socket, 2 too big or not UTF-8. | | `ws_send_binary(id: u64, ptr, len) -> u32` | The same, for bytes. | | `ws_close(id: u64, code) -> u32` | Close a socket with code 1000, or 3000 to 4999. 0 OK, 1 no such socket. | | `ws_list(ptr, cap) -> u32` | Write the object's socket ids, at most `cap`, as little-endian u64s. Returns how many there are. | | `ws_msg(ptr)` | Copy this event's bytes in, a `WSBIN` message or a `WAKE`'s state. | | `call(ptr, len) -> u32` | Ask another process, in JSON, and wait. Returns the answer's length. `pg_read` copies it. | | `send(ptr, len) -> u32` | Tell another process, if this request succeeds. 1 queued, 0 too much queued. | | `pg_open(ptr, len) -> u64` | Lease a Postgres connection with `{"db": ""}`. Returns a handle, or 0 with why in `pg_read`. | | `pg_query(handle: u64, ptr, len) -> u32` | One statement, `{"db", "q", "p"}`, on a handle, or pooled with handle 0. Returns the answer's length. | | `pg_close(handle: u64) -> u32` | Done with a handle. It rolls back, releases its locks, and goes back. | | `pg_read(ptr, cap) -> u32` | Copy the last answer of `pg_*` or `call`, at most `cap` bytes. Returns how many. | `env.abort` crashes the cell, as does a trap. A small slice of WASI preview 1, from `wasi_snapshot_preview1`: - `fd_write` to stdout and stderr only, into a 64 KB buffer that combd logs when a cell crashes. - `fd_read`. Stdin is always empty. - `clock_time_get` and `clock_res_get`, rounded to 1 ms on purpose. - `random_get`. - `environ_sizes_get` and `environ_get`, with the app's vars and secrets, and `COMBD_ORG`, `COMBD_APP`, `COMBD_PROCESS`, `COMBD_NODE`. - `args_sizes_get`, with no args, and `sched_yield`. - `proc_exit`, which crashes the cell. Anything else answers "not supported", or "not capable" for files. There are no files, no sockets, and no sleeping. ### Event cells An `http` process whose module exports `on_request` is an event cell. One instance takes many requests at once. - `on_request(id: u64, ptr, len)` is a new request, its text at `ptr` from `alloc`. - `respond(req: u64, ptr, len) -> u32` answers request `req`, now or later. Work may go on after it. - `timer_async(req, ms)`, `pg_open_async(req, ptr, len)`, `pg_query_async(req, handle, ptr, len)`, and `call_async(req, ptr, len)` each return a ticket at once, 0 for an unknown request. Timers max out at 30 s. - `on_event(ticket: u64, len)` says a ticket's wait is done. `ev_read(ptr, cap)` reads its data. - `pg_close_async(req, handle)` and `send_async(req, ptr, len)` don't wait. [`guest-ev/`](https://github.com/jtwebman/combd/tree/main/guest-ev) is a whole one in Rust. ## Hand-written modules in the repo The examples use the ABI directly where the SDK doesn't reach yet: - [`examples/million`](https://github.com/jtwebman/combd/tree/main/examples/million), a `ws` process with `hibernate = "app"`, `handle_batch`, `tick_every` - [`examples/arena`](https://github.com/jtwebman/combd/tree/main/examples/arena), WebSockets on durable objects, ticks, `call` with `"node"` # Deploying On a fleet, apps go up through a node's admin API. `combd deploy` sends the app, and every node of the fleet picks it up through the catalog in the bucket. Vars, secrets, rollbacks, and removals go the same way. A single node with `serve-apps` has no admin API. Copy the app's folder to the node and restart it, as in the [Quickstart](quickstart.md#5-run-it-on-a-server). ## From the command line ```sh export COMBD_TOKEN=combd_... # an admin token combd deploy ./shop acme https://node.example.com:9443 # {"version":"3f9a..."} combd status acme/shop https://node.example.com:9443 # {"version":"3f9a..."} ``` | Command | What it does | |---|---| | `combd deploy ` | Bundle the app in `` and make it live as `/`, named by its manifest. | | `combd status ` | The live version. | | `combd rollback ` | Make an earlier version live. | | `combd remove ` | Take the app down. | | `combd vars [NAME=value]...` | Set vars. `NAME=` unsets one. | | `combd secret put ` | Set a secret, its value from stdin. | | `combd secret rm ` | Delete a secret. | Each prints the API's JSON answer, or the error and exits 1. Org and app names are lowercase letters, digits, `-` and `_`. ### Reaching the node - **A node with a public certificate**, ending TLS itself with `COMBD_TLS_CERT`, needs only `COMBD_TOKEN`. combd checks the node's certificate against the public roots and the URL's host. So a CI anywhere can deploy with a token and a combd binary. - **A fleet with its own CA**, the wire's certificates from `combd wire-certs`, also needs `COMBD_ADMIN_CA` set to its `ca.pem`, and `COMBD_ADMIN_NAME` to the node's name in it, default `node0.combd`. - **On the node itself**, use `http://127.0.0.1:`. A URL with no port uses its scheme's, 443 for `https` and 80 for `http`. ```sh # the arena's deploy, through node 0, with the fleet's CA COMBD_TOKEN=$(cat .secrets/token) COMBD_ADMIN_CA=.secrets/certs/ca.pem COMBD_ADMIN_NAME=node0.combd \ combd deploy ../app acme "https://n0.arena.combd.dev:9443" ``` ### Secrets ```sh printf %s "$DATABASE_URL" | combd secret put acme/shop pg-url https://node.example.com:9443 ``` The value comes from stdin, never the command line, which other users and shell history can see. One trailing newline is dropped. A secret is text, at most 64 KB. Secrets are write-only. Nothing reads one back. The nodes need the fleet's master key, `COMBD_MASTER_KEY_FILE`, to take them ([Security model](security.md#secrets)). While a version is live, a secret must be one its manifest declares, and valid for its type. To delete a secret it requires, first deploy a version that doesn't. Vars are checked against their types the same way. Text from the command line is read as the var's type, so `8` for an `int` is 8, and `lots` is an error. ## What a deploy does 1. **The bundle.** `combd deploy` packs the manifest, `modules/*.wasm`, and `static/` ([Static files](static-files.md)). Nothing else in the folder goes up, so `combd.dev.toml` stays home. A bundle holds at most 4096 files and 64 MB. Its version is the SHA-256 of its bytes, so deploying the same one twice is free. 2. **Checks.** The node unpacks it, reads the manifest, whose name must be the app's, resolves its vars and secrets for the fleet, and loads its modules. On any mistake nothing changes, and the error comes back. So set a new app's required vars and secrets first, then deploy. 3. **Live.** The version goes to the bucket, and the catalog makes it live. Every node swaps it in within moments. The catalog checks, against its newest state, that no two apps hold one domain. A node that can't load a version keeps serving the one it has. **Old code finishes.** Requests already running finish on the old version. A durable object moves to the new code once it has no request running, no open sockets, and no tick. So a deploy never ends a game in play. An object that keeps sockets open stays on the old version until they close. **Compiled modules.** If the manifest asks for a compiled module whose builds aren't ready, the deploy answers `{"version": "...", "building": true}` and waits. The node asks builders for the builds and tries again every couple of seconds. It makes the version live once every kind of node in the fleet has one, and gives up after an hour. A waiting deploy lives in that node's memory. If the node restarts, deploy again. `GET /v1/deploys` shows what's waiting. See [Compiled modules and builders](builders.md). **Rollback** checks the old version again, since vars may have changed. If its compiled builds are gone, deploy it again instead. ## Admin tokens combd knows nothing about people, only tokens and what each may do. A token's scopes look like `/:`, and any part may be `*`: | Action | Lets a token | |---|---| | `deploy` | deploy, roll back, and remove | | `vars` | set vars | | `secrets` | set and delete secrets | | `read` | read an app's live version | `acme/shop:deploy`, `acme/*:vars`, and `*/*:*` are all scopes. The fleet-wide endpoints, `/v1/nodes` and `/v1/deploys`, take a `*/*:read` or `*/*:*` token. Make one: ```sh combd admin-token 'acme/*:deploy' 'acme/*:vars' # token (give it to your CI; it's shown once): combd_6b1f... # line for the tokens file: 9c2e... acme/*:deploy acme/*:vars ``` The token is shown once. The line goes in the nodes' tokens file, `COMBD_ADMIN_TOKENS`, one token a line, its SHA-256 then its scopes. `#` starts a comment. Nodes keep only the hash, and compare in constant time. A bad token and one that's too weak get the same answer. ## The admin API A fleet node serves it on `COMBD_ADMIN_PORT`, with the tokens in `COMBD_ADMIN_TOKENS`. With the wire's certificates in `COMBD_WIRE_CERTS`, it speaks TLS and listens on every address. It shows the fleet's certificate to a client that asks for a `*.combd` name, and the node's public certificate, `COMBD_TLS_CERT`, to any other. Without TLS it listens on 127.0.0.1 only, so a token never crosses a network in the clear. The arena uses port 9443. Every request but the panel carries `Authorization: Bearer `. The token is checked on the headers, before any of the body is read. ```text PUT /v1/apps// a bundle: check it, upload it, make it live GET /v1/apps// the live version POST /v1/apps///rollback {"version"}: make an earlier one live DELETE /v1/apps// take it down PUT /v1/apps///vars {"NAME": value, ...}: set vars (null unsets) PUT /v1/apps///secrets/ the value (the body): set it DELETE /v1/apps///secrets/ delete it GET /v1/nodes every node's stats (a */*:read token) GET /v1/deploys deploys waiting for builds, and the builders (*/*:read) POST /v1/builds/next a builder's next job (signed by a builder key) POST /v1/builds/done what a builder built (signed by a builder key) GET / the panel ``` Answers are JSON, the result with `200`, or `{"error": "..."}` with `400` for a bad request or a failed deploy, `401` for no token, `403` for a token that may not, or `404`. **The panel**, `GET /` on the admin port, is a page that asks `/v1/nodes` and `/v1/deploys` with a token you give it. It shows the nodes, their stats, where each process's CPU goes, and deploys waiting for builds. The page itself holds nothing. `/v1/builds/*` are for builders, signed with a builder key instead of a token. See [Compiled modules and builders](builders.md). # Running a fleet A fleet is several combd nodes that share one bucket. The bucket decides who owns each object. It holds every app's versions and the catalog. The catalog says which version of each app is live, with its vars and its sealed secrets. Nodes talk to each other over the wire, with mutual TLS. Requests for an object go to its owner, and D3 writes go to its peers. ## Set up a fleet, step by step These steps make a three-node fleet on Linux machines you already have. They follow the arena's real setup ([below](#a-real-fleet-the-arena)). Replace each `<...>` with your own value. 1. **Name the machines and open their ports.** Point `n0.`, `n1.`, and `n2.` at the three machines. Open 80 and 443 to everyone, 7100 only between the nodes, and 9443 to wherever you deploy from. The nodes reach each other on 7100, and the admin API is on 9443. 2. **Get combd.** Take a release or build it, as in the [Quickstart](quickstart.md#1-install). You need it on your own machine to make keys and deploy, and the Linux binary on each node. 3. **Make a bucket and check it.** Make an S3 or R2 bucket, and a key that can read and write it. Then run the test every node runs at start: ```sh COMBD_S3=https://.r2.cloudflarestorage.com COMBD_S3_BUCKET= \ COMBD_S3_KEY= COMBD_S3_SECRET= COMBD_S3_REGION=auto \ combd bucket-check ``` A node refuses a bucket that fails it. For S3, use its endpoint and region in place of R2's. 4. **Make the fleet's keys** on your own machine, once, in a folder you keep private: ```sh mkdir -m 700 fleet && cd fleet && umask 077 combd wire-certs certs 3 combd master-key master.key OUT=$(combd admin-token '/*:*' '*/*:read') echo "$OUT" | sed -n 's/.*: \(combd_[0-9a-f]*\)$/\1/p' > token echo "$OUT" | sed -n 's/^line for the tokens file: *//p' > tokens ``` `certs` is the wire's CA and a certificate per node. `master.key` seals the apps' secrets. `token` is for you to deploy with, and the nodes get only its hash, in `tokens`. The `*/*:read` scope lets it read `/v1/nodes`. See [What you need](#what-you-need). 5. **Make a builder key, only if an app runs compiled modules.** On the builder, `combd builder-key builder.key` prints its public half. Put that key in a file named `builder-keys`. Nodes load only builds signed by a key in it. Then run the builder, as in [Compiled modules and builders](builders.md). 6. **Get a TLS certificate** for the three node names and your apps' domains, from any CA. The arena uses Let's Encrypt. Save the chain as `cert.pem` and its key as `key.pem`. combd ends HTTPS itself with them ([TLS in combd](#tls-in-combd)). 7. **On each node, add a user and install combd:** ```sh sudo useradd --system --home-dir /var/lib/combd --shell /usr/sbin/nologin combd sudo mkdir -p /var/lib/combd /etc/combd/certs /etc/combd/tls /opt/combd/current/combd/target/release sudo chown combd:combd /var/lib/combd sudo install -m 755 combd /opt/combd/current/combd/target/release/combd ``` The unit in step 10 runs combd from that path, as that user, and lets it write only `/var/lib/combd`. 8. **Copy node N's files to it:** - `certs/ca.pem`, `certs/node.pem`, and `certs/node.key` to `/etc/combd/certs/` - `master.key` and `tokens` to `/etc/combd/` - `cert.pem` and `key.pem` to `/etc/combd/tls/` - `builder-keys` to `/etc/combd/`, if you made one Then make its secrets combd's own: ```sh cd /etc/combd sudo chown combd:combd tokens master.key certs/node.key tls/key.pem sudo chmod 600 tokens master.key certs/node.key tls/key.pem ``` combd refuses a master key that others may read. Never copy `certs/ca.key` to a node. 9. **Write `/etc/combd/env`** on node N, mode 600: ```sh COMBD_NODE= COMBD_PEERS=0=n0.:7100,1=n1.:7100,2=n2.:7100 COMBD_PUBLIC_URLS=0=https://n0.,1=https://n1.,2=https://n2. COMBD_WIRE_CERTS=/etc/combd/certs COMBD_DATA=/var/lib/combd COMBD_REPLICAS=2 COMBD_S3=https://.r2.cloudflarestorage.com COMBD_S3_BUCKET= COMBD_S3_KEY= COMBD_S3_SECRET= COMBD_S3_REGION=auto COMBD_ADMIN_PORT=9443 COMBD_ADMIN_TOKENS=/etc/combd/tokens COMBD_MASTER_KEY_FILE=/etc/combd/master.key COMBD_TLS_CERT=/etc/combd/tls/cert.pem COMBD_TLS_KEY=/etc/combd/tls/key.pem COMBD_REDIRECT_PORT=80 COMBD_BUILDER_KEYS=/etc/combd/builder-keys ``` Only `COMBD_NODE` differs between nodes. Leave out `COMBD_BUILDER_KEYS` if you made no builder key. `COMBD_DATA` must be `/var/lib/combd`, the one folder the unit lets combd write. The arena's whole file is in [Starting a node](#starting-a-node), and every setting is in [Settings](#settings). 10. **Install the systemd unit on each node.** Copy [`examples/arena/cloud/combd.service`](https://github.com/jtwebman/combd/blob/main/examples/arena/cloud/combd.service) to `/etc/systemd/system/combd.service`. It runs `combd serve-fleet both 443` as the `combd` user, with the settings in `/etc/combd/env`, and restarts it if it stops. 11. **Start each node:** ```sh sudo systemctl daemon-reload sudo systemctl enable --now combd journalctl -u combd -f ``` The log shows the bucket test and any setting combd refuses. A node won't start with a node number another running node uses. 12. **Check the fleet** from your own machine: ```sh curl -s https://n0./_combd/ping curl -s https://n0./_combd/stats curl -s -H "authorization: Bearer $(cat token)" https://n0.:9443/v1/nodes ``` Ping answers `{}`. `/v1/nodes` lists every node's stats, and a node that doesn't answer says why. See [Stats and monitoring](stats.md). 13. **Deploy an app:** ```sh COMBD_TOKEN=$(cat token) combd deploy ./ https://n0.:9443 ``` The nodes have a public certificate, so the token is all you need. Every node picks the app up from the bucket. See [Deploying](deploying.md). ## What you need - **A bucket** that speaks S3 and honors conditional writes, such as S3 or R2. Before it serves, every node races 16 writers on one conditional write, and refuses the bucket unless exactly one wins. `combd bucket-check` runs that test on its own, with the same `COMBD_S3...` settings. - **The wire's certificates**, a CA and one certificate per node. - **A master key** for the apps' secrets, the same on every node. - **Admin tokens**, for `combd deploy`. - Builders, if any app runs compiled modules ([Compiled modules and builders](builders.md)). Make the keys once, somewhere safe: ```sh combd wire-certs certs 3 # certs/ca.pem, ca.key, node0.pem, node0.key, ... node2 combd master-key master.key # 64 hex characters; give every node a copy combd admin-token 'acme/*:*' # a token for your CI, and the line for the nodes' tokens file ``` Each node gets `ca.pem` and its own `node.pem` and `node.key`. Keep `ca.key` off the nodes. combd refuses a master key file that others may read, so `chmod 600` it. ## Starting a node ```sh combd serve-fleet both 443 ``` The engine is `interp`, which interprets every module, or `both`, which runs each as its manifest says. `both` needs a combd built with the compiled tier, as the releases are. Everything else comes from the environment. A node of the arena's fleet has this, with your values in `<...>`: ```sh COMBD_NODE=0 COMBD_PEERS=0=n0.arena.combd.dev:7100,1=n1.arena.combd.dev:7100,2=n2.arena.combd.dev:7100 COMBD_REGIONS=0=nyc3,1=fra1,2=sgp1 COMBD_PUBLIC_URLS=0=https://n0.arena.combd.dev,1=https://n1.arena.combd.dev,2=https://n2.arena.combd.dev COMBD_WIRE_CERTS=/etc/combd/certs COMBD_REPLICAS=2 COMBD_THREADS=1 COMBD_DATA=/var/lib/combd COMBD_LEASE_MS=3000 COMBD_S3=https://.r2.cloudflarestorage.com COMBD_S3_BUCKET=combd-arena COMBD_S3_KEY= COMBD_S3_SECRET= COMBD_S3_REGION=auto COMBD_ADMIN_PORT=9443 COMBD_ADMIN_TOKENS=/etc/combd/tokens COMBD_MASTER_KEY_FILE=/etc/combd/master.key COMBD_READ_LEASE=1 COMBD_TLS_CERT=/etc/combd/tls/cert.pem COMBD_TLS_KEY=/etc/combd/tls/key.pem COMBD_REDIRECT_PORT=80 COMBD_MEMORY_BUDGET=550M COMBD_BUILDER_KEYS=/etc/combd/builder-keys ``` On start, a node tests the bucket and catches up on the catalog. It gives up after 60 s if it can't reach the bucket. It loads every live app, and refuses to start with a node number another running node is using. Then it serves, and swaps in whatever the catalog says changed. A node refuses a wire with no certificates, unless `COMBD_WIRE_INSECURE=1` asks for plain TCP, with a warning in the log. ## TLS in combd A node can end TLS itself, with no proxy in front. `COMBD_TLS_CERT` is the chain and `COMBD_TLS_KEY` the key, both PEM files. `COMBD_REDIRECT_PORT=80` opens a port that only redirects to HTTPS. combd checks the certificate's file every few seconds and loads a new one for new connections, so a renewal needs no restart. Replace the key first, then the certificate. With a public certificate on the node, `combd deploy` needs only a token ([Deploying](deploying.md#reaching-the-node)). ## Placement In a fleet, each durable process says where its objects live, with `placement` in the manifest: | `placement` | An object's home | |---|---| | `nearest` (default) | The node most of its requests come in through. | | `first` | Wherever it started. It doesn't move. | | `spread` | Its hash among the nodes in `scope` of the node that asked. `scope` is `zone`, `region`, or `fleet`. | | `pin` | A node in the regions it names, such as `pin = "fra1,ams3"`, or an area. | - `COMBD_REGIONS` says where each node is, as a region and, after a slash, a zone: `0=nyc3/a,1=fra1/a`. Every node not listed shares one region. - `COMBD_AREAS` names sets of regions for `pin`, such as `eu=fra1+ams3,us=nyc3`. - If none of a pinned object's regions is up, `fallback = "nearest"` runs it elsewhere until one is back. `fallback = "none"` refuses it with 503 until then, so its data never leaves them. Once a window, each owner works out each object's home and offers the object to it. The window is `COMBD_PLACE_MS`, 30 minutes, and 0 means objects never move. `nearest` moves an object only when one node sent it at least 20 requests and two thirds of them, two windows running, so nothing bounces. The target may say no. It takes at most 16 objects a window, and none while it's under memory pressure, refusing connections, or has less than 30% of its machine's memory free. A refused object stays and asks again next window. A fork can put its own rules in `home` in `combd/src/policy.rs`, which has the last word. ## Clients picking a node Every node answers, on its main port: - `GET /_combd/nodes` gives `{"me", "nodes", "version"}`, with every node's public URL from `COMBD_PUBLIC_URLS`, so a page can time each one. - `GET /_combd/ping` gives `{}`, to time a round trip. Both allow any origin. The arena's page times every node this way, and its matchmaker starts a match on the node best for both players. `GET /v1/nodes` also uses `COMBD_PUBLIC_URLS`, to gather every node's stats ([Stats and monitoring](stats.md)). ## A real fleet: the arena [`examples/arena/cloud/`](https://github.com/jtwebman/combd/tree/main/examples/arena/cloud) runs the arena and One Million Checkboxes on three 1 vCPU, 1 GB droplets in New York, Frankfurt, and Singapore, over an R2 bucket, set up with Terraform. [`examples/arena/README.md`](https://github.com/jtwebman/combd/blob/main/examples/arena/README.md) walks through it. The parts worth copying: - **`up.sh`** makes the fleet's secrets once, with `wire-certs`, `admin-token`, and `master-key`. It runs Terraform, waits for each node's `/_combd/ping`, and deploys. **`deploy.sh`** deploys again after a change. - **`cloud-init.tftpl`** puts each node's files in `/etc/combd`, with its secrets 600 and owned by the `combd` user. It adds a `combd` system user, 1 GB of swap, and a cron job for updates, and starts combd. - **`combd.service`** runs `combd serve-fleet both 443` under systemd as the `combd` user, with no Docker, `Restart=always`, and `LimitNOFILE=1048576`. It locks combd down, as in [Security model](security.md#the-process-around-it). It sets `MALLOC_MMAP_THRESHOLD_` and `MALLOC_MMAP_MAX_`, so glibc keeps cells' buffers as lazy zero pages. - **`combd-tune.sh`** runs at every boot, before combd. It turns off connection tracking for ports 80 and 443, and the Ubuntu services a node doesn't need. - **`combd-update.sh`** runs every minute from cron and moves the node to a new combd, as below. - **`tls.sh`** gives every node a renewed certificate, which combd picks up within seconds. - **`roll.sh`** moves every node now, one at a time, each only once the last answers again. combd raises its open-files limit to the ceiling. It keeps connections under three quarters of it, and under `COMBD_MAX_CONNS`, so a full node answers instead of running out of files. ### Signed releases The arena's nodes fetch combd from a GitHub release that CI replaces on every push to `main`. Nothing unsigned runs: 1. CI writes a note, `combd-arena.txt`, with the source hash, the tarball's SHA-256, and the commit, and signs it with the ed25519 release key. The private key is a CI secret. Each node holds the public half, `release.pub`. 2. `combd-update` checks the note's signature, then the tarball's SHA-256 against the note. It ignores a note that doesn't check out. 3. Before it moves, it runs the new combd's `combd builds-ready`, which asks for its builds of every compiled module, and exits 0 only once they're all in the bucket. Moving sooner would take those apps down on this node. 4. Node `n` moves once the release has been out `n` x 2 minutes, so the nodes restart one at a time. It waits while the node has open WebSockets, up to 15 minutes past its turn, so it doesn't cut off a game in play. 5. It keeps this version and the one before. While a node restarts, for a few seconds, it refuses new connections, and browsers try another of the name's addresses. **A new combd reads bundles an older one deployed.** It meets them in the bucket through every update. The other way round, nodes still on an old combd refuse a bundle it can't read. So a new format ships in two steps, a combd that reads it, then one that writes it. ## Settings Every setting is an environment variable. Times are in ms unless they say otherwise. ### Any node | Setting | Default | What it does | |---|---|---| | `COMBD_DATA` | `data` | Where objects, the catalog's copy, and builds live. | | `COMBD_THREADS` | the CPU count | Event loops. | | `COMBD_IDLE_MS` | 60000 | An object idle this long leaves memory. | | `COMBD_MAX_OBJECTS` | 10000 | Objects in memory, per loop. | | `COMBD_MAX_CONNS` | 200000 | Connections open at once, and at most three quarters of the open-files limit. | | `COMBD_MAX_AWAKE` | 1000 | Cells at work at once, per loop ([Overload](overload.md)). | | `COMBD_MEMORY_BUDGET` | the container's limit, else the RAM | What the node may use, in bytes, or `512M`, `4G`. | | `COMBD_TLS_CERT`, `COMBD_TLS_KEY` | none | End TLS on the main port. PEM files. | | `COMBD_REDIRECT_PORT` | none | A port that only redirects to HTTPS. | | `COMBD_MAX_BODY` | 1 MiB | A request's body. Past it, 413. | | `COMBD_MAX_HEADER_BYTES` | 16 KiB | The request line and headers. Past it, 431. | | `COMBD_MAX_HEADERS` | 64 | Headers in a request. Past it, 431. | | `COMBD_MAX_WS_MESSAGE` | 1 MiB | A WebSocket message. Past it, close 1009. | | `COMBD_READ_TIMEOUT_MS` | 30000 | A request must arrive within this, and a new connection start one. Past it, 408. | | `COMBD_KEEPALIVE_MS` | 75000 | A kept connection with nothing to do. | | `COMBD_TLS_HANDSHAKE_MS` | 10000 | A TLS handshake. | | `COMBD_CPU_LIMIT_TICKS` | 100000000 | Loop iterations one call of a compiled cell may run before it's killed, about 100 ms of tight loop. | | `COMBD_EV_MAX` | 32 | Requests per event-cell instance, for a node with no manifest. With a manifest, `max_concurrent` sets it. | | `COMBD_AOT_DIR` | none | A folder of compiled builds, `.so`, that a node loads beside those built in. | ### A fleet node | Setting | Default | What it does | |---|---|---| | `COMBD_S3` | none | The bucket's endpoint, `https://...`, or `http://` for a local one. Setting it makes a fleet node. | | `COMBD_S3_BUCKET` | `combd` | The bucket. | | `COMBD_S3_KEY`, `COMBD_S3_SECRET` | none | Its credentials. | | `COMBD_S3_REGION` | `us-east-1` | Its region. R2 uses `auto`. | | `COMBD_NODE` | none | This node's number. | | `COMBD_PEERS` | none | Every node's wire address: `0=host:7100,1=host:7100,...`. | | `COMBD_WIRE_PORT` | 7100 | Where this node listens for peers. | | `COMBD_WIRE_CERTS` | none | The folder from `combd wire-certs`. | | `COMBD_WIRE_INSECURE` | off | `1` for plain TCP between nodes. | | `COMBD_LEASE_MS` | 10000 | How long a silent node keeps its objects, renewed every third of it. | | `COMBD_REPLICAS` | 0 | Peers per object, for D3. 0 is D2 ([Storage](storage.md)). | | `COMBD_UPLOAD_MS` | 10000 | With peers up, how long an object's uploads may wait. It warns above 5 minutes. | | `COMBD_UPLOAD_BYTES` | 1 MB | ...or until this much builds up, at most 16 MB. | | `COMBD_READ_LEASE` | off | `1`: owners answer reads with no round, while their peers' promises hold. Only once every node runs a version that promises. | | `COMBD_RESTORES_AT_ONCE` | 4 | Objects restored from the bucket at once, per loop. | | `COMBD_REGIONS` | none | Where each node is: `0=nyc3/a,1=fra1/a`. | | `COMBD_AREAS` | none | Named sets of regions: `eu=fra1+ams3,us=nyc3`. | | `COMBD_PLACE_MS` | 1800000 | The placement window. 0 means objects never move. | | `COMBD_PUBLIC_URLS` | none | Every node's public URL: `0=https://a,1=https://b`. | | `COMBD_MASTER_KEY_FILE` | none | The master key, to open and seal secrets. | | `COMBD_ADMIN_PORT` | none | Serve the admin API on this port. | | `COMBD_ADMIN_TOKENS` | none | The tokens file. Required with `COMBD_ADMIN_PORT`. | | `COMBD_BUILDER_KEYS` | none | A file of builder public keys this node trusts. | | `COMBD_BUILDS_DIR` | `/builds` | Where fetched builds are kept. | | `COMBD_BUILDER_PUSH` | none | A push builder's URL. | | `COMBD_PUSH_KEY` | none | The key file pushes are signed with. | ### The command line and builders | Setting | Used by | What it does | |---|---|---| | `COMBD_TOKEN` | `deploy`, `status`, ... | The admin token. | | `COMBD_ADMIN_CA` | `deploy`, `status`, ... | The fleet's `ca.pem`, for a fleet with its own CA. | | `COMBD_ADMIN_NAME` | `deploy`, `status`, ... | The node's name in that CA, default `node0.combd`. | | `COMBD_FLEET` | `builder` | Any node's admin API: `https://:9443`. | | `COMBD_BUILDER_KEY` | `builder` | Its key file, default `builder.key`. | | `COMBD_BUILDER_ID` | `builder` | Its name, default `$HOSTNAME`, else `builder`. | | `COMBD_AOT_SRC` | `builder` | The `combd/aot` sources it builds against, default the ones it was built from. | | `COMBD_BUILD_WORK` | `builder` | Its cargo folder, kept between builds. | | `COMBD_PUSH_KEYS` | `builder serve` | A file of the nodes' push public keys. Required. | | `PORT` | `builder serve` | Where it listens, default 8080. | The other `COMBD_` names in the source are for combd's own tests and benchmarks. # Compiled modules and builders combd runs a module one of two ways, and the manifest says which: ```toml [modules.game] engine = "compiled" # or "interpreted" (the default) ``` - **Interpreted** runs on combd's own WebAssembly interpreter. It starts cells fast and runs anywhere, but it's slow at raw computing. - **Compiled** turns the module into safe Rust, then into machine code for the node's CPU. The generated code has no `unsafe`, so the Rust compiler checks the sandbox holds. A module runs only the way it asks, never switching. The same bytes, compiled and interpreted, are two programs. ## Builds come from builders Nodes don't compile. Builders, machines you run, make each build and sign it. A node loads a build only if a builder in `COMBD_BUILDER_KEYS` signed it. The signature covers the build, its module, its pin, and its CPU, so no build can pass for another. - A build is for a **pin**, a short hash of what it must match: the Rust compiler, combd's compiled runtime, and the target. A new combd with a new pin needs new builds. - A build is for a **CPU**: `x86-64`, `x86-64-v2`, `-v3`, or `-v4`, by the features the node has, or `generic` elsewhere. A build for a level runs on that level and above. - A module is named by the SHA-256 of its bytes. Builds go through the bucket, under `_combd/builds///`, where each node notes its pin and CPU, so a deploy knows what to build. **A deploy waits for its builds.** When a deploy has a compiled module that isn't built for every kind of node in the fleet, it asks for the builds and answers `"building": true`. It goes live once they're all there, and fails if a build fails or after an hour. `GET /v1/deploys` and the panel show what's waiting and which builders were seen lately. **An update waits for its builds too.** Before a node moves to a new combd, `combd builds-ready`, run with the node's own settings, exits 0 only once the new combd's builds of every compiled module the fleet runs are in the bucket. Until then it asks for the missing ones and exits 1, or 2 if it can't tell. The arena's `combd-update.sh` runs it, while the old combd serves. **A builder builds for its own combd only.** Move it before the nodes. Until then, nodes on the new combd wait for builds. A late builder delays an update but doesn't take an app down. Builders serve a fleet. On a single node with `serve-apps`, keep modules interpreted. ## A builder's key ```sh combd builder-key builder.key # a builder key is in builder.key; for COMBD_BUILDER_KEYS: 3b6a... ``` The private half stays on the builder and signs every build, and every request of a pull builder. The public half goes in a file on every node, named by `COMBD_BUILDER_KEYS`, one hex key a line, with `#` comments. A builder holds no bucket credentials and never sees the bucket. ## Pull: `combd builder` A pull builder asks any node's admin API for work, builds it, and sends it back. It signs each request's path and body with its builder key, within 5 minutes of the node's clock. A node refuses a key it doesn't list before reading the body. ```sh COMBD_FLEET=https://arena.combd.dev:9443 \ COMBD_BUILDER_ID=box-1 \ COMBD_BUILDER_KEY=/keys/builder.key \ COMBD_BUILD_WORK=/work \ combd builder ``` | Setting | What it is | |---|---| | `COMBD_FLEET` | Any node's admin API, or the fleet's name: `https://:9443`. Required. | | `COMBD_BUILDER_KEY` | Its key file, default `builder.key`. | | `COMBD_BUILDER_ID` | Its name in `/v1/deploys`, default `$HOSTNAME`. | | `COMBD_BUILD_WORK` | Its cargo folder, kept between builds, so combd's runtime compiles once. | | `COMBD_AOT_SRC` | The `combd/aot` sources to build against, default the ones it was built from. | An idle builder asks every 2 s at first, doubling to every 30 s, since each ask costs a bucket listing. A builder that goes quiet for 15 minutes loses its claim on a job, and another takes it. A builder runs the Rust compiler, so it needs the toolchain combd was built with, in `rust-toolchain.toml`, and the matching `combd/aot` sources. The builder image has both. The arena's builder is any Linux machine with Docker: ```sh docker run -d --name combd-builder --restart unless-stopped --env-file ~/combd-builder/env \ -v ~/combd-builder/builder.key:/keys/builder.key:ro -v combd-build-work:/work \ ghcr.io/jtwebman/combd-builder:latest /combd/combd/target/release/combd builder ``` `~/combd-builder/env` holds `COMBD_FLEET`, `COMBD_BUILDER_ID`, `COMBD_BUILDER_KEY=/keys/builder.key`, and `COMBD_BUILD_WORK=/work`. Watchtower moves it to a new image when one is out. ## Push: `combd builder serve` A push builder waits for work instead of asking. The node that claims a job sends it, and the builder builds it while the request waits, one build a request. So on a host like Cloud Run, with one request an instance, it builds many at once and scales to zero. The nodes sign each push with a push key the builder lists. Nothing is shared. The nodes hold the push key, and the builder its builder key. ```sh combd builder-key push.key # its public half goes in the builder's COMBD_PUSH_KEYS ``` On every node: | Setting | What it is | |---|---| | `COMBD_BUILDER_PUSH` | The builder's URL, `https://...`. | | `COMBD_PUSH_KEY` | The push key file. | On the builder: | Setting | What it is | |---|---| | `COMBD_PUSH_KEYS` | A file of the push keys' public halves. Required. | | `COMBD_BUILDER_KEY` | Its builder key, which signs the builds. | | `COMBD_BUILD_WORK`, `COMBD_AOT_SRC` | As for a pull builder. | | `PORT` | Where it listens, default 8080, in plain HTTP. The host in front ends TLS. | It takes `POST /build`, signed by a push key it lists. A push for another pin gets `409` and is sent again later, not failed. A push that fails or times out after 14 minutes lets its claim go, and the next ask sends it again. A node that dies mid-push leaves a claim that goes stale in 15 minutes. One node pushes at most 4 builds at once, and the same job no sooner than 30 s after its last push. On Cloud Run: ```sh gcloud run deploy combd-builder --image \ --command /combd/combd/target/release/combd --args builder,serve \ --concurrency 1 --timeout 15m --min-instances 0 --max-instances 4 --cpu 4 --memory 8Gi \ --allow-unauthenticated \ --set-env-vars COMBD_BUILDER_KEY=/keys/builder/key,COMBD_PUSH_KEYS=/keys/push/keys,COMBD_BUILD_WORK=/tmp/work \ --set-secrets /keys/builder/key=combd-builder-key:latest,/keys/push/keys=combd-push-keys:latest ``` `--allow-unauthenticated` lets the nodes reach it. The push signature lets them in. Each cold instance compiles combd's runtime once before its first build, so the first builds after a scale-up are slow. Pull and push can run at once, since they claim jobs the same way. # Overload and limits When a node gets more than it can do, it says so instead of falling over. It keeps serving what it has, and turns the rest away early, where a client can go somewhere else. ## The memory guard A node watches two things: - **Its own memory**, against its budget: `COMBD_MEMORY_BUDGET`, in bytes or as `512M` or `4G`, else the container's cgroup v2 limit, else the machine's RAM. - **The whole machine's free memory.** combd's own count misses what the kernel holds for its connections, and a node that swaps stops answering. A tenth of the machine is kept back. | Pressure | Own memory | Machine | What the node does | |---|---|---|---| | 0 | under 80% | under 85% | Normal. | | 1 | 80% of the budget | 85% | Reclaims. Idle sessions sleep at once, idle objects go after 10 s, not a minute, and warm cells sooner. | | 2 | 90% | 95% | Refuses new connections and sockets until back under 85%, or 90% for the machine. | The machine's percentages are of the machine less its tenth kept back. The node measures every 64 connections as they pour in, not only once a second, since a burst outran the once-a-second check. ## Refusing at TCP A full node stops listening, so the kernel refuses a new connection, and a browser tries the name's next address by itself. An accepted connection could only be hung up on. Full means any of these: - memory pressure 2 - `COMBD_MAX_CONNS` connections open, 200,000 by default - every event loop behind, as below Each state holds for at least a second, so it doesn't flap. `tcp_refusing` in the stats is 1 meanwhile. A listener on 127.0.0.1 stays, and the node never refuses its own machine, so its own checks can see how full it is. A connection that gets in anyway is closed if TLS, or gets `503` with `retry-after: 1` if plain HTTP. On the live fleet, a node at about 29,500 clicking pages refused the rest at TCP and stayed up. 439 of 441 requests to the name went to another node by themselves (`combd/RESULTS.md`, "On the fleet"). ## At most so many cells at work `COMBD_MAX_AWAKE`, 1,000 a loop, caps the cells at work at once, running or waiting on a host call. A sleeping session with work waits its turn asleep, so not everyone wakes at once. While that line is as long as the cap, the loop is behind. The node takes no new connections, and each session's mailbox is 8, not 1,000. `sessions_ready` and `sessions_waited` in the stats show it working. Without it, 40,000 users ran a 1 GB node out of memory. Every message to a sleeping viewer woke it, and once the loop fell behind, more and more were awake at once. With it, the node held about 30,000, refused the rest, and lost 0.5% of their clicks (`combd/RESULTS.md`, "Overload, without Docker"). ## Mailboxes and batches A durable object takes one request at a time. The ones behind it wait in its mailbox, up to `mailbox`, 1000. Past that, a request is refused at once with `503 {"error":"overloaded"}`, not queued forever. `mailbox_refused` counts them. A module that exports `handle_batch` takes what's waiting at once, up to `batch`, 100, in one transaction. Batches form only from a queue, so they grow with the load ([Processes](processes.md#batches)). ## Cold starts In a fleet, `COMBD_RESTORES_AT_ONCE` caps the objects restored from the bucket at once, 4 a loop, since each holds its segments in memory. Under pressure, an idle object goes after 10 s, not 1 s, so it isn't let go while its neighbors are still coming back. ## Limits per request What a client may send, node-wide: | Limit | Default | Past it | |---|---|---| | Body (`COMBD_MAX_BODY`) | 1 MiB | 413 | | Request line and headers (`COMBD_MAX_HEADER_BYTES`) | 16 KiB | 431 | | Headers (`COMBD_MAX_HEADERS`) | 64 | 431 | | WebSocket message (`COMBD_MAX_WS_MESSAGE`) | 1 MiB | close 1009 | | Time to send a request (`COMBD_READ_TIMEOUT_MS`) | 30 s | 408 | | An idle kept connection (`COMBD_KEEPALIVE_MS`) | 75 s | closed | | A TLS handshake (`COMBD_TLS_HANDSHAKE_MS`) | 10 s | closed | A client that reads its WebSocket too slowly is closed once 8 MB is waiting for it. Bodies come only with a `content-length`. A request with chunked `Transfer-Encoding` gets a 400. What a cell may use: - **CPU.** Cells run a slice at a time and take turns, so a busy cell can't hold its loop. One call of a compiled cell is killed after `COMBD_CPU_LIMIT_TICKS`, 100,000,000 loop iterations, about 100 ms of tight loop. - **Host time.** A request's host calls, like SQL and sends, may take 1 s in all. Past that, they crash the cell. - **Memory.** `memory_kb` per process, if set. - **SQL.** At most 100 ms a statement, 2 MB a value, and 1 GB an object's database. - **Output.** stdout and stderr are kept up to 64 KB a request. - **Sends.** A request may queue 16 MB of WebSocket messages and `send`s in all. ## Disk A node takes no new objects, and answers `507`, when its free disk is less than a tenth of the data disk or 2 GB, whichever is less. `disk_low` in the stats is 1 meanwhile. ## What a $6 node does One 1 vCPU, 1 GB droplet, with combd under systemd and million compiled, measured 2026-10-22, from the README: - **About 58,000 idle WebSockets.** Then it turns new ones away and stays fast, pings 4 to 20 ms, no swapping. - **10,000 people clicking at once.** Nothing lost, p50 echo under 420 ms, 62% CPU. At 20,000, 0.04% of clicks lost. - **Past about 30,000 active users**, it turns the rest away and keeps serving the ones it has. The full runs, and what they don't cover, are in [`combd/RESULTS.md`](https://github.com/jtwebman/combd/blob/main/combd/RESULTS.md). # Stats and monitoring ## `/_combd/stats` Every node answers `GET /_combd/stats` on its main port with what it's doing, as JSON. It isn't behind a token, so anyone who can reach the port can read it. Counters count up from the node's start. The rest are now. ```sh curl -s https://n0.arena.combd.dev/_combd/stats ``` **Objects** | Field | What it is | |---|---| | `objects_live` | Durable objects in memory now. | | `objects_opened` | Objects started or brought back from storage. | | `evictions` | Objects that left memory, idle or squeezed. | | `alarms_fired`, `alarms_failed` | Alarms that ran, and ones whose handler crashed. | | `forwarded` | Requests sent to another loop or node that owns their object. | | `mailbox_refused` | Requests refused because a mailbox was full, `503 overloaded`. | | `batched` | Requests that ran in a `handle_batch` batch. | **Writes (D1)** | Field | What it is | |---|---| | `commits` | Commits that wrote something. | | `fsyncs` | The fsyncs that made them durable. Far fewer than `commits` means the group commit is working. | | `checkpoints` | WAL checkpoints. | | `fsyncs_slow`, `fsync_max_us` | fsyncs of 10 ms or more, and the slowest, in µs. | **A fleet** | Field | What it is | |---|---| | `peer_acks` | Writes acknowledged by peers, for D3. | | `bucket_acks` | Writes acknowledged by the bucket, for D2. | | `uploads` | Segments sent to the bucket, retries too. | | `fences` | Takeovers that fenced an older owner. | | `fence_waits` | Fences told to wait out a peer's promise, when a read lease held off a takeover. | | `tail_entries` | Entries recovered from peers in takeovers. | | `lease_reads` | Reads answered under a read lease, with no round. | | `gc_deleted` | Bucket keys deleted by compaction. | | `moves` | Objects handed to another node by placement. | **WebSockets and sessions** | Field | What it is | |---|---| | `sockets_open` | WebSockets open now. | | `ws_messages` | Messages in from them. | | `ws_frames`, `ws_flushes` | Frames sent, and the writes that sent them. A socket's frames can share one write. | | `ws_handoffs` | Handshakes one loop took and handed to the loop of their object. | | `sessions_live` | `ws` sessions now. | | `sessions_hibernated` | Of those, asleep. | | `sessions_woken` | Times a session woke. | | `sessions_ready` | Sleeping sessions with work, waiting for room under `COMBD_MAX_AWAKE`. | | `sessions_waited` | Times one had to wait. | **Connections and memory** | Field | What it is | |---|---| | `conns_open` | Connections open now. | | `conns_refused` | Connections accepted, then refused for being full or over `COMBD_MAX_CONNS`. | | `tcp_refusing` | 1 while the node is full and not listening. | | `accept_failed` | Connections `accept` failed on, out of files. | | `memory_used` | The node's memory, by its own count. With a cgroup, what the cgroup is charged, less file cache it would give back. | | `memory_budget` | Its budget, `COMBD_MEMORY_BUDGET` or what it found. | | `machine_available` | What the machine can still hand out without swapping. | | `heap_in_use` | Bytes the allocator has handed out. glibc only, 0 elsewhere. | | `pressure` | 0 fine, 1 reclaiming, 2 refusing. See [Overload](overload.md). | | `disk_low` | 1 while the data disk is too full for new objects. | **Event cells, and the rest** | Field | What it is | |---|---| | `ev_instances`, `ev_requests` | Event-cell instances started, and the requests they took. | | `procs` | Per `org/app/process`: `cpu_us`, its cells' run time with host calls, `slices`, and `engine`, `compiled` or `interpreted`. | | `node` | This node's number in a fleet, `null` on one node. | | `version` | The combd version, `dev` for a build with none set. | Watch these first: `pressure`, `tcp_refusing`, `sessions_ready`, `mailbox_refused`, `conns_refused`, and `disk_low`. Any of them above 0 means the node is turning work away. ## `/v1/nodes` On a fleet's admin API, `GET /v1/nodes`, with a `*/*:read` token, asks every node in `COMBD_PUBLIC_URLS` for its `/_combd/stats` at once, 3 s each at most. ```sh curl -s -H "authorization: Bearer $COMBD_TOKEN" https://n0.arena.combd.dev:9443/v1/nodes ``` ```json {"nodes": [ {"node": 0, "url": "https://n0.arena.combd.dev", "stats": {"objects_live": 65, "...": "..."}}, {"node": 1, "url": "https://n1.arena.combd.dev", "error": "..."} ]} ``` A node that doesn't answer says why, and the rest still come. ## The panel `GET /` on the admin port is the panel. Given a token, it shows `/v1/nodes` and `/v1/deploys` as a page: every node, where each process's CPU goes, and deploys waiting for builds. ## Logs combd logs to stdout and stderr, the journal under systemd. A cell that crashes is logged with the reason and the tail of what it printed, so a Rust panic's message shows up beside it: ```text object "acme/shop/cart/42" trapped: unreachable (panicked at src/lib.rs:12: ...) ``` ## From outside `GET /_combd/ping` answers `{}` and is cheap. The arena's scripts wait on it to know a node is up. # Security model Security in combd is walls inside walls: a sandbox around every cell, a host that charges each cell for what it asks, signed code, sealed secrets, and a locked-down process around all of it. ## The sandbox Every cell runs in a WebAssembly sandbox: - **No system calls.** A cell has no files, no processes, and no network. It gets only the host's calls ([the host ABI](languages.md#the-host-abi)) and a small slice of WASI. Anything else answers "not supported". - **Its own memory only.** Every load and store is checked against the cell's memory. - **No code of its own.** Wasm has no instruction that runs data, and a cell's memory is never executable. combd has no JIT. The interpreter runs modules as data, and builders build compiled modules ahead of time, as safe Rust with `forbid(unsafe_code)`. A compiler bug gives a wrong answer or a failed build, not memory unsafety. - **A coarse clock**, rounded to 1 ms, against timing attacks. - **Only its own app.** A cell can `call` only processes of its own app that list it in `callers`, and use only pools that list it. Objects, pools, env vars, and calls are all scoped to `/`. The Bytecode Alliance's `wasmparser` checks every module before it runs. ## The host doesn't work for free The real danger was the host doing work a cell doesn't pay for. Fuzzing the host API found one cell could panic the host, or eat seconds of CPU and gigabytes of memory and disk. Every host-side cost has a cap now: - Host calls get 1 s a request in all. - SQLite runs fenced, with no `ATTACH`, `PRAGMA`, `VACUUM INTO`, extensions, or the host's own tables. A statement gets 100 ms. Values, answers, and each database are capped ([Storage](storage.md#sql-from-a-cell)). - Output, sends, and an event cell's work in flight are capped. - A node takes no new objects when its disk is nearly full. ## HTTP - Bodies come only with a `content-length`. `Transfer-Encoding` and two lengths that disagree are refused, so there's no request smuggling. - A cell can set only the safe headers `content-type`, `cache-control`, `location`, `set-cookie`, `etag`, `last-modified`, `vary`, and `access-control-*`, with no line breaks, so there's no response splitting. - Only standard methods reach a cell, so no client can fire an `ALARM`. - Slow clients time out with `COMBD_READ_TIMEOUT_MS`, and sizes are capped ([Overload](overload.md#limits-per-request)). ## Admin tokens - Tokens are stored as SHA-256 hashes, never as themselves, and compared in constant time. A bad token and a weak one get the same answer. - The token is checked on the headers, before any of a body is read. - Without TLS, the admin API listens on 127.0.0.1 only, so a token never crosses a network in the clear. - Scopes limit each token to orgs, apps, and actions ([Deploying](deploying.md#admin-tokens)). - The panel holds nothing. It asks with the token you give it, and builds the page with `textContent`. ## Secrets Secrets are sealed in the catalog. A node opens them only in memory, for an app's instances as they start. - Each app has its own data key, made with its first secret and sealed by the fleet's master key. `combd master-key` makes the master key, and combd refuses a `COMBD_MASTER_KEY_FILE` others may read. - Each secret is sealed by its app's data key with AES-256-GCM, with the app and the secret's name bound in. A sealed value copied to another name or app won't open. - The bucket, and every node's copy of the catalog on disk, hold only sealed values. - Secrets are write-only. Values come in on stdin, never the command line, and errors never show them. - A deploy carries only the manifest, modules, and `static/`, so the local `combd.dev.toml` can't ride along. combd refuses it if git tracks it. ## Signed builds A compiled module's build is native code a node will run. So a node loads one only with an ed25519 signature from a builder in `COMBD_BUILDER_KEYS`, over the build's hash, its module, its pin, and its CPU. A build can't pass for another module, another combd, or another CPU. A node checks a build on its disk again at each load, so anything that could write that folder can't plant one for the next restart. Builders hold no bucket credentials. A pull builder signs its requests, and a push builder takes only jobs signed by a push key it lists. See [Compiled modules and builders](builders.md). ## Signed releases The arena's nodes run only a combd whose release note is signed by CI's ed25519 release key, whose public half each node holds. The tarball must match the SHA-256 in that note ([Running a fleet](fleet.md#signed-releases)). ## Between nodes The wire is mutual TLS. Each node has a certificate for `node.combd` from the fleet's CA. A node takes a connection only from a client with a certificate from that CA, and checks the node it dials against its exact name. Without certificates, a node refuses to join a fleet, unless `COMBD_WIRE_INSECURE=1` asks for plain TCP. A peer's catalog push is only news. A node reads the bucket for the truth. ## The process around it On the arena's nodes, combd runs as its own user under systemd, with no Docker. The unit, [`combd.service`](https://github.com/jtwebman/combd/blob/main/examples/arena/cloud/combd.service), takes away what combd doesn't need. It gets ports 80 and 443 and no other capability, a read-only system but its data, and no memory both writable and executable. It makes only the system calls a service makes, and can run nothing but its own binary, its libraries, and its builds. ## How it's been checked There were three rounds of security review. Each finding was proven with a failing test before its fix, and the next round attacked the fixes. They covered the compiled path against ten years of Wasm runtime exploits, the compiler and builder, the interpreter and host calls with SQLite, the public edge and admin, and the fleet's wire and bucket. Alongside them ran the official Wasm spec test suite, differential fuzzing against Wasmtime, and fuzzing of the host API, the HTTP parser, and the wire. What each round found and fixed is in `plan.md`, "Security review, in rounds". ## What it doesn't do (yet) - **One compromised node is the whole fleet.** Every node holds the bucket's credentials, and mTLS proves "a fleet member", not which one. - **No rules between tenants.** combd runs your own apps. Any org with a deploy token can claim any domain not taken, and there are no quotas per tenant. A service built on combd puts its own rules in the hooks in `combd/src/policy.rs`. `catalog_write` checks every deploy, var, secret, and removal, and `new_object` every new object. Both allow everything as shipped. - **No bucket-write budget.** Each object made costs bucket writes, and storage forever. An app that lets anyone name objects pays for that. - `/_combd/stats` on the main port isn't behind a token. # CLI reference `combd help` prints this: ```text combd: Durable Objects on your own machines. An app: combd new a new app in ./ (Rust; combd.toml says what runs) combd dev [dir] [port] build it and serve it here (default ., 8080) combd build [dir] build it: its crate for wasm32, into modules/ A node, or a fleet (see examples/arena/cloud for a whole one): combd serve-apps every app under /// combd serve-fleet a fleet node (COMBD_* settings) combd builder [serve] make compiled modules' builds for a fleet (serve: pushed to it) engine: interp, aot (compiled), or both (each module as its manifest says) Deploying (to a node's admin port, with COMBD_TOKEN): combd deploy combd status combd rollback combd remove combd vars [name=value] combd secret put|rm Keys and certificates: combd master-key combd builder-key combd admin-token ... combd wire-certs combd version combd also has its own benchmarks, simulators and fuzzers. ``` An unknown command prints the help and exits 2. A command that fails prints why and exits 1. ## An app ### `combd new ` Makes `.//` with: - `combd.toml`, with a page on `/` in `[http.web]`, and a counter on `/counter/:id/*` in `[durable.counter]` - `Cargo.toml`, using `combd-sdk` - `src/lib.rs` - `.gitignore`, for `target/`, `modules/`, and `data/` The name, in lowercase letters, digits, `-` and `_`, names the package, the module, and the folder. ### `combd build [dir]` Runs `cargo build --release --target wasm32-unknown-unknown` in `dir`, default `.`, and copies the result to `modules/.wasm`. ### `combd dev [dir] [port]` Builds the app, then serves it interpreted on `port`, default 8080. Data goes in `./data`, or `COMBD_DATA`. Vars and secrets come from `combd.dev.toml`. It doesn't rebuild on its own. ## A node ### `combd serve-apps ` Serves every app under `///` on one node, with D1 durability. The org is the folder an app is in. Each app's vars and secrets come from its `combd.dev.toml`. One mistake in any app, and nothing starts. The engine is `interp`, `aot`, or `both`, as the help says. There's no admin API. To change an app, change its folder and restart. ### `combd serve-fleet ` A fleet node. It serves what's deployed to its bucket, and the admin API if `COMBD_ADMIN_PORT` is set. It needs at least `COMBD_S3`, `COMBD_NODE`, and `COMBD_PEERS`. The engine is `interp` or `both`. See [Running a fleet](fleet.md). ### `combd builder` and `combd builder serve` A builder for a fleet's compiled modules. It pulls work from `COMBD_FLEET`, or with `serve` takes pushed work on `$PORT`. See [Compiled modules and builders](builders.md). ### `combd builds-ready` Run it, with the node's own `COMBD_S3...` and `COMBD_DATA`, before a node moves to this combd. It exits 0 if every compiled module the fleet runs is built for it, 1 if not yet, asking for the missing builds, and 2 if it can't tell. ### `combd bucket-check` Runs the startup storage test on the bucket in `COMBD_S3`, `COMBD_S3_BUCKET`, `COMBD_S3_KEY`, `COMBD_S3_SECRET`, and `COMBD_S3_REGION`. It checks conditional writes hold, under a 16-way race too. ## Deploying These need `COMBD_TOKEN`, and for a fleet with its own CA, `COMBD_ADMIN_CA` and `COMBD_ADMIN_NAME`. Each prints the admin API's JSON answer. See [Deploying](deploying.md). | Command | | |---|---| | `combd deploy ` | Bundle the app in `dir` and make it live. | | `combd status ` | Its live version. | | `combd rollback ` | Make an earlier version live. | | `combd remove ` | Take it down. | | `combd vars [NAME=value]...` | Set vars. `NAME=` unsets. | | `combd secret put ` | Set a secret, its value from stdin. | | `combd secret rm ` | Delete a secret. | ## Keys and certificates | Command | What it makes | |---|---| | `combd master-key ` | A new master key for a fleet's secrets, 64 hex characters, mode 600. It won't overwrite a file. Every node needs a copy, in `COMBD_MASTER_KEY_FILE`. | | `combd builder-key ` | A new ed25519 key in `file`, mode 600. Prints the public half for `COMBD_BUILDER_KEYS`. Also makes push keys. | | `combd admin-token ...` | A new admin token, printed once, and its line for `COMBD_ADMIN_TOKENS`. | | `combd wire-certs ` | A fleet CA, `ca.pem` and `ca.key`, and a certificate per node, `node.pem` and `node.key`, in `dir`. | ## `combd version` Prints the version. So does `--version` or `-V`. ## The rest combd also has benchmarks, simulators, and fuzzers, like `sim`, `catsim`, and `fuzz`. They're for working on combd itself, and some need its source tree.