@loess/committer

A committer reads a circle's waiting transactions in storage's order, checks each one, runs the circle's program on it, and commits the result as the circle's next head. A server, an agent, and a device run the same code with their own layers, and the package imports no device code, so a server that only commits needs nothing else. It opens and reads the circle with the circle package, which an owner also uses to create a circle and commit a new circle record.

import { commitCircle } from '@loess/committer';
 
// A server that commits one circle for as long as the scope is open.
yield * Effect.forkScoped(commitCircle(circle, server));

Architecture shows where a committer sits, and protocol holds the records it reads and writes.

What it promises

A committer settles each waiting transaction once, in storage's order. It skips none: every transaction after the head's inboxCounter is accepted or rejected in the next commit, and a transaction whose writer, device, and deviceCounter a commit already settled is dropped, so a transaction sent twice is committed once, and a writer can't settle another writer's transaction by reusing its device and counter. It commits a transaction only if the program accepts it, and commits its changes as the writer wrote them.

Interface

function commitCircle(
  circle: CircleId,
  member: MemberKeys,
  options?: { sqlite?: WebAssembly.Module; now?: () => number },
): Effect<never, CommitFailure, Storage | Relay | Programs>;
 
function openCommitter(
  circle: CircleId,
  member: MemberKeys,
  options?: { sqlite?: WebAssembly.Module; now?: () => number },
): Effect<
  { commitOnce: Effect<void, CommitFailure>; headNumber: () => number },
  NotListed | CommitFailure,
  Storage | Programs | Scope
>;

member is the MemberKeys of a committer the circle record lists, and the circle README describes the type. It opens the circle's key box, and signs each head beside the commit key.

openCommitter reads the head, checks the circle records back to the first, opens the member's key box, and loads the circle's database and program, which it keeps for the life of the scope. It fails with NotListed if the circle record doesn't list the member as a committer. Each commitOnce:

1. reads the head and the inbox in one storage read, and opens the circle again if another commit stored a newer head;
2. takes each transaction after the head's inboxCounter, checks it, runs the program, and writes it to the database or rejects it;
3. seals the changed pages, the page map leaves they fall in, and a new root with the hashes of the first pages, and stores them with the commit and the next head;
4. on LostRace, opens the circle again from the newer head and repeats from step 2, so it returns once nothing is waiting.

If a transaction is missing from storage, step 2 stops before it, and commitOnce commits the transactions before it and reads the inbox again every 250 ms. Storage swaps the inbox while it stores the transaction, so the inbox entry can be stored one put before its transaction. The committer rejects the transaction with seal once it has waited 5 seconds, so a transaction whose put failed holds up the circle for 5 seconds at most. A transaction that refers to a missing blob is rejected with blob at once, because a device sends a write only after storage stores its blobs.

If a commitOnce fails, such as when storage can't be reached, the database may hold transactions that no stored commit holds, so the next commitOnce opens the circle again from the stored head first. The same committer goes on, and a caller doesn't open a new one.

commitCircle runs commitOnce once, then after each head notice from the relay, and each transaction notice with an inboxCounter. A round reads the inbox again instead of trusting the notices, so a lost notice can't stall the loop. A server that a request wakes, such as the host Worker's committer service, runs commitOnce once for each request.

sqlite is a compiled sqlite3.wasm for a runtime that can't compile WebAssembly from bytes, such as a Cloudflare Worker. now is the clock a commit records and a program reads, Date.now by default.

Rejections

A rejected transaction stays on record in the commit with its reason:

A writer holds a role if the member list lists it, or if a committed join request admitted it from a link the record still lists. From a link's key, a committer accepts only a join request, and commits it with a key box for the joiner, so from that head the joiner is a member with the link's role.

Checks

The sdk also runs these on another device's waiting transaction, so a device checks it the same way a committer does:

Tests

pnpm vitest run --project committer runs the commit loop in process over memory storage and a memory relay: it settles the waiting transactions in storage's order, rejects a transaction that the program rejects or that references a missing blob, and stores the next head. The host Worker's committer service runs it in miniflare.