Architecture

This page shows the parts of Loess, the interface of each, and how they connect. Each part's README holds the full interface and how it works inside. Goals defines a circle, a member, a transaction, a committer, and a program.

The parts

A device writes a transaction to storage, a committer checks it and builds the circle's pages, and the relay tells the other devices. Storage keeps the data sealed, which means encrypted with the circle key, so it can't read it.

Goals starts with Ann's post and Ben's like. Here is that like, start to finish, with only Ann's phone committing her circle:

// 1. Ben's phone seals this transaction and writes it to Ann's circle.
const like: Transaction = {
  circle: 'ann-circle',
  writer: 'ben', // Ben's identity
  device: benPhone, // the phone's public signing key
  deviceCounter: 41,
  changes: [
    { id: 'like:20261010-a1', field: 'like/post', value: { id: 'post:20261010-k3' } },
    { id: 'like:20261010-a1', field: 'like/by', value: { id: 'person:ben' } },
  ],
};
 
// 2. Storage stores it under its hash, e4d9…, and adds it to the inbox as number 118.
//    The relay passes it on, and every open device shows the like as pending.
 
// 3. Ann opens the app. Her phone runs the program on the like, writes it to its database, and makes a commit.
const commit: Commit = {
  previousHash: '7c1e…',
  programHash: 'b03a…',
  committedAt: 1791662400000,
  accepted: ['e4d9…'],
  rejected: [],
};
 
// 4. Her phone sends the commit, the pages it changed, and the page map leaf and root over them, with a new head.
//    Storage swaps the head.
const head: Head = {
  circle: 'ann-circle',
  number: 57, // the stored head was 56
  commitHash: '51aa…',
  committer: 'ann',
  circleHash: '90fe…',
  pageMap: '2b6d…', // the hash of the sealed page map root
  inboxCounter: 118, // everything up to Ben's like is settled, so storage drops it from the inbox at the next write
  keyVersion: 3,
  keys: { write: '3d07…', box: 'c1e4…' },
};
 
// 5. The relay passes the head on with the objects it added, so an open device reads nothing more,
//    and the like is no longer pending.

A commit replaces one record, the head. A transaction is settled once a commit has accepted or rejected it. Devices, servers, and agents commit with the same code, and a new circle lists its owner and its storage's committer service, if any.

The protocol

Everything two parts must agree on is in one package, protocol, which has no I/O and no clock. Every record is deterministic CBOR checked by an Effect schema, and its format never changes once stored data uses it.

type Value = string | number | { id: string } | Uint8Array | null; // a reference is { id }, and null is no fact
type Change = { id: string; field: string; value: Value; replaces?: ValueHash };
type ValueHash = Uint8Array; // the first 4 bytes of the SHA-256 of a value's CBOR, or of `null` for no fact
 
/** One member's write. Sealed with the writer's signature inside, signed with the write key, and stored under its hash. */
type Transaction = {
  circle: CircleId;
  writer: MemberId; // an identity, or for a join request the link's key
  device: PublicKey;
  deviceCounter: number; // 1, 2, 3 for each device in each circle; a committer settles each one at most once
  changes: ReadonlyArray<Change>;
};
 
/** What a committer made from the transactions it took. Sealed. */
type Commit = {
  previousHash: Hash | null; // the commit before this one
  programHash: Hash | null; // the program it ran
  committedAt: number; // the committer's clock
  accepted: ReadonlyArray<Hash>; // transaction hashes; the changes stay in the transactions
  rejected: ReadonlyArray<{ transactionHash: Hash; rejection: Rejection }>;
};
 
type Rejection = {
  reason: string;
  conflict?: { id: string; field: string; value: Value }; // the fact's current value
};
 
/** The one record a commit replaces. Signed with the commit key and by the committer, and not sealed. */
type Head = {
  circle: CircleId;
  number: number; // 1 for the first head, then one more each commit
  commitHash: Hash;
  committer: MemberId; // an owner, or an identity or key the circle record lists as a committer
  circleHash: Hash; // the current circle record
  pageMap: Hash | null; // the sealed root of the page map, or null while the circle has no pages
  inboxCounter: number; // every transaction in the inbox up to this one is settled
  keyVersion: number; // one more each time new keys are made after a removal
  keys: { write: PublicKey; box: PublicKey }; // this version's: storage checks a write against `write`, and another circle encrypts its keys to `box`
};
 
/** Who is in the circle, who commits, and its program. Signed with the own key. */
type CircleRecord = {
  // not sealed, so storage can check a join request, a head, and the next record
  history: ReadonlyArray<Hash>; // every earlier record, oldest first; empty in the first, whose hash is the CircleId
  keyVersion: number; // the key version that seals `sealed`
  keys: { commit: PublicKey; own: PublicKey }; // the owners' keys; the head holds the rest
  links: ReadonlyArray<Link>; // open invite links
  sealed: Uint8Array; // the CircleMembers below, sealed
};
 
type CircleMembers = {
  members: ReadonlyArray<{ member: MemberId; role: Role; box?: PublicKey }>; // `box` for a key member; an identity's is in its head
  committers: ReadonlyArray<{ committer: MemberId; box?: PublicKey }>; // who commits transactions; a new circle lists its owner and the storage's committer service, if any
  program: Hash | null; // the sealed object that holds the WebAssembly
};
 
type Link = { key: PublicKey; count: number; role: 'read' | 'write' }; // the join requests storage takes from the key, and the joiner's role
 
/** The one object storage changes itself. Not sealed. */
type Inbox = {
  counter: number; // the newest inboxCounter storage gave
  waiting: ReadonlyArray<{ transactionHash: Hash; inboxCounter: number }>;
  links: ReadonlyArray<{ key: PublicKey; sent: ReadonlyArray<Hash> }>; // what each link key has sent, against its count
};
 
type Signed = { body: Uint8Array; shared: SignedBy; member?: SignedBy }; // a record's bytes with its two signatures
type Role = 'read' | 'write' | 'own';
type MemberId = CircleId | PublicKey; // an identity, as text, or the signing key of a device, a passkey, a phrase, a server, or a link, as bytes

A change sets a fact, and null deletes it. With no replaces, the last write in storage's order is kept. With replaces, the committer rejects the transaction if the fact's value no longer has that hash. A hash keeps the change small: an edit of a 4,000-character message carries 4 bytes of the old text, not all of it. The hash is compared with one value, so a chance match is 1 in 4 billion, and a writer gains nothing by forcing one, because leaving replaces out does the same.

// Dan renames a task he saw as "Buy milk". Cara's rename to "Buy oat milk" was committed first.
const change: Change = {
  id: 'task:a0V',
  field: 'task/name',
  value: 'Buy 2 milks',
  replaces: valueHashOf('Buy milk'), // b9b2507c
};
 
// The committer rejects Dan's transaction, and his app shows him Cara's name.
const rejection: Rejection = {
  reason: 'conflict',
  conflict: { id: 'task:a0V', field: 'task/name', value: 'Buy oat milk' },
};

A list is one id per item, with a sort key such as item:a0V in it, so two people adding items never touch the same fact. A fact refers to a photo or a file with a reference whose id is blob: and the hex of the sealed blob's hash, such as { id: 'blob:9f2c…' }. blobHashesOf finds these references in a transaction, so a committer can check that storage holds each blob.

Each transaction, head, and circle record carries two signatures:

A head isn't sealed, so storage sees which committer made it. It never sees who wrote a transaction.

/** The check storage makes on every call that changes it. Pure: storage passes the bytes of the head and the circle record it holds, and the inbox. */
declare const validate: (
  circle: CircleId,
  stored: { head?: Uint8Array; circle?: Uint8Array; inbox?: Inbox },
  request: StorageRequest, // a write, an upload, or a commit, as bytes
) => Effect<Validated, Refused>; // each transaction's hash, the upload, or the new head with its record
 
/** The part of `validate` that checks a head. A device runs it on every head it receives, against the one it holds. */
declare const validateHead: (
  circle: CircleId,
  held: HeldHead | undefined,
  next: { head: Uint8Array; circles: ReadonlyArray<Uint8Array> }, // each record after the held one, oldest first
) => Effect<HeldHead, Refused>;

Storage and devices run the same code on a head, so they can't disagree about which heads are valid. A commit carries at most one new circle record, so a device that missed several commits can lack several records. It finds their hashes in the newest record's history, reads them in one request, and passes them all to validateHead. Neither validate nor validateHead opens a seal. A seal binds what it holds to its circle, its kind, and its place, such as a page number, and pads it with Padmé, so its size shows only roughly how much it holds. A circle record's sealed part is bound to its place in history and the hash of the record before it, because the circle id is the first record's own hash.

Storage

Storage keeps bytes under the keys that protocol's storageKeyOf builds, and replaces the bytes under a key only with a compare-and-swap. Each kind of storage is a layer of one Effect service, and testStorage runs against every layer.

class Storage extends Context.Service<
  Storage,
  {
    // every device and committer
    read(keys: ReadonlyArray<string>): Effect<ReadonlyArray<Uint8Array | undefined>, StorageError>;
    has(keys: ReadonlyArray<string>): Effect<ReadonlyArray<boolean>, StorageError>; // a committer asks whether a blob is stored
 
    // writers, signed with the write key
    write(
      circle: CircleId, // storage can't read a sealed transaction's circle
      transactions: ReadonlyArray<Uint8Array>, // a device back online sends all of them in one request
    ): Effect<ReadonlyArray<number>, Refused | StorageError>; // each one's inboxCounter
    upload(
      upload: Uint8Array, // the signed BlobUpload: the blob's circle, hash, and length
      blob: Uint8Array, // the sealed bytes
    ): Effect<void, Refused | StorageError>;
 
    // committers and owners, signed with the commit key
    commit(
      head: Uint8Array,
      objects: ReadonlyArray<Uint8Array>, // the commit, and the pages, page map nodes, records, and programs it adds
      boxes: ReadonlyArray<{ id: string; box: Uint8Array }>, // the key boxes it adds
    ): Effect<void, LostRace | Refused | StorageError>;
  }
>() {}

Storage accepts a head when:

1. It references the circle record of the stored head, or a new one.
2. The commit key in that record signed it.
3. Its number is one more than the stored head's. A circle is created by head 1, whose circle record hashes to the circle id.
4. The own key in the stored record signed any new circle record, and the new record's history is the stored record's with the stored record's hash added.
5. Its key version is the stored head's with the same keys, or one more.
6. Its inboxCounter is no lower than the stored head's and no higher than the inbox's counter. An owner's commit of a circle record copies the stored one, and carries a commit that accepts nothing.

write publishes each transaction with no inboxCounter as it arrives, then stores it as it adds its hash to the inbox with the next inboxCounter. Storage retries that swap itself without limit, so a writer never gets a conflict. Each swap drops the entries the head has settled, and a reader of the inbox skips every entry up to the head's inboxCounter. If the transaction's hash is still in the inbox, storage adds nothing and answers with the counter it has, and if it was settled already, the committer drops it by its device counter. commit stores the objects and the boxes, then swaps the head. Only committers race in a circle, and only for the head: when two send the same head number, the second gets LostRace. Sending a stored head again succeeds, and everything else in a circle is written once under its hash, so a write cut off halfway can be made again.

The r2 layer runs in a Worker in front of an R2 bucket. It runs validate, keeps only a cache of circle records, and publishes each new transaction and head to the relay through StorageNotices. The http entry serves any storage over HTTP, one POST for each call (/read, /has, /write, /upload, and /commit) with a CBOR body, and its client, layerHttpStorage, is the layer a device uses. A refusal answers 422 and a lost race 409. Reads go through the Worker, and an upload carries the blob in its body, so storage checks the bytes against the hash and the length. Tests use the memory layer.

The relay

The relay passes notices and checks nothing, so it holds no rules and no state. A device checks a notice the way it checks what it reads from storage, so a false notice does nothing.

class Relay extends Context.Service<
  Relay,
  {
    publish(notice: Notice): Effect<void>;
    follow(
      circles: ReadonlyArray<{ circle: CircleId; head: number }>, // the head number the device holds
    ): Effect<Follow, never, Scope>; // open until the scope closes
  }
>() {}
 
type Follow = {
  notices: Stream<Notice, RelayError>;
  add(circles: ReadonlyArray<{ circle: CircleId; head: number }>): Effect<void>;
  remove(circles: ReadonlyArray<CircleId>): Effect<void>;
};
 
type Notice =
  | { _tag: 'Transaction'; circle: CircleId; inboxCounter?: number; transaction: Uint8Array } // the signed, sealed transaction, with no counter when storage publishes it as it arrives
  | { _tag: 'Head'; head: Uint8Array; objects: ReadonlyArray<Uint8Array> } // with the objects its commit added, up to 512 KiB, so an open device usually reads nothing
  | { _tag: 'Signal'; circle: CircleId; signal: Sealed }; // presence, passed on and never stored

For each circle it follows or adds, a follow starts with what the device missed: the circle's head if it is newer, and the transactions waiting in its inbox that the head hasn't settled, in storage's order. Then it sends each notice as it is published. A device holds one follow and adds each circle it opens, and reads only the pages of a commit it missed.

// What a device sends on its one socket, as CBOR. The relay sends each Notice.
type DeviceMessage =
  | { _tag: 'Follow'; circles: ReadonlyArray<{ circle: CircleId; head: number }> } // at most 500
  | { _tag: 'Unfollow'; circles: ReadonlyArray<CircleId> } // at most 500
  | { _tag: 'Signal'; circle: CircleId; signal: Sealed }; // to the circle's other follows

The device pings every 30 seconds. After a reconnect it sends Follow messages with every circle, each from the newest head the follow has passed on.

A circle's database

A circle is one SQLite database of 16 KiB pages. A committer compresses and seals each page it changes and stores it under its hash, and the signed head covers every page through the page map. A device fetches only the pages a query touches, and keeps them, because none ever changes.

The page map is a root, which holds the page count, the hash of each leaf, and the hashes of pages 1 to 9, and leaves that each hold the hashes of 64 pages. The root and the leaves are sealed objects, and the head holds the root's hash. A commit seals the leaves its pages fall in and a new root. A cold device reads the root with its key box, then pages 1 to 9, which hold the B-tree roots, and up to 64 leaves in one read.

class CircleDatabase extends Context.Service<
  CircleDatabase,
  {
    query<row>(statement: Statement<row>): Effect<ReadonlyArray<row>, PagesMissing | DatabaseError>;
    facts: Facts; // the program package's interface, read by a program
    fetch(pages: ReadonlyArray<PageNumber>): Effect<void, PagesMissing | PageError>; // the pages `facts` threw as missing
    write(written: {
      transaction: Transaction;
      hash: Hash;
      inboxCounter?: number; // absent for a pending transaction a device writes on top
      rejection?: Rejection; // stores the row and no facts
    }): Effect<void, Conflict | AlreadySettled | PagesMissing | DatabaseError>;
    commit(
      commit: Commit,
      hash: Hash, // of the sealed commit
    ): Effect<
      { pages: ReadonlyMap<PageNumber, Uint8Array>; pageCount: number },
      PagesMissing | DatabaseError
    >; // plain pages, which the caller seals
    moveTo(
      pages: PageSource['Service'], // a newer head's pages
    ): Effect<
      {
        settled: ReadonlyArray<Hash>;
        dropped: ReadonlyArray<{ hash: Hash; error: Conflict | PagesMissing }>;
      },
      PagesMissing | DatabaseError
    >;
  }
>() {}
 
// The database's only way to read a page, so it knows nothing about storage, keys, or the network.
// A source leaves out a page it can't give, and the database then fails with PagesMissing.
class PageSource extends Context.Service<
  PageSource,
  { read(pages: ReadonlyArray<PageNumber>): Effect<ReadonlyMap<PageNumber, Uint8Array>, PageError> }
>() {}

CREATE TABLE facts (
  id, field, value, is_reference, transaction_id, writer, committed_at, current,
  PRIMARY KEY (id, field, transaction_id)) WITHOUT ROWID;
CREATE INDEX facts_value ON facts (field, value, id, is_reference, transaction_id, writer, committed_at, current)
  WHERE current AND (typeof(value) IN ('integer', 'real') OR is_reference IS 1 OR length(CAST(value AS BLOB)) <= 64); -- sorts and matches by value
CREATE INDEX facts_field ON facts (field, id, value, is_reference, transaction_id, writer, committed_at, current) WHERE current; -- newest-first views
CREATE INDEX facts_long ON facts (field, id, value, is_reference, transaction_id, writer, committed_at, current)
  WHERE current AND typeof(value) NOT IN ('integer', 'real') AND is_reference IS NOT 1 AND length(CAST(value AS BLOB)) > 64; -- the values facts_value leaves out
CREATE TABLE transactions (id INTEGER PRIMARY KEY, hash, commit_id, inbox_counter, writer, device, device_counter, rejection);
CREATE INDEX transactions_rejected ON transactions (writer, device, device_counter) WHERE rejection IS NOT NULL;
CREATE TABLE devices (writer, device, settled, PRIMARY KEY (writer, device)) WITHOUT ROWID; -- settled counters as ranges, such as [[1,41],[43,90]]
CREATE TABLE commits (id INTEGER PRIMARY KEY, hash, committed_at);
CREATE VIEW stored_facts AS SELECT id, field, value, is_reference, transaction_id, current, writer, committed_at FROM facts;

A fact row holds its writer and its commit time, and each index holds every column, so a newest-first view walks one B-tree: the newest 20 of 30,000 messages take 3 round trips after the key box and 12 KiB. Compressed, the database is a third larger. Apps and programs read stored_facts, and SQLite uses a partial index only for a query whose WHERE repeats its condition. A reference is stored as the id it refers to, with is_reference set to 1.

A rejected transaction keeps its rejection and adds no facts, so a device back after a week finds its rejected writes by device_counter in one query. Each device has one row in devices with its settled counters, so a commit from 30 devices changes one page of that table, and write fails with AlreadySettled for a counter already in the row. A pending transaction is held in memory on top of the settled ones, and moveTo writes it again on a newer head.

A query that needs pages the database doesn't hold runs again over stand-in pages to find the other missing pages of the same B-tree level, and the database reads them in one batch. A device without storage gets PagesMissing. A query that would read more than its page limit, 128 by default, fails with TooManyPages, so a scan shows up in a test. A program can't wait for a read, so facts throws PagesMissing, and the caller fetches the pages and runs the program again.

The program

A program is one WebAssembly export, validateTransaction, that accepts or rejects a transaction. It can't change one, so a device shows a pending write exactly as it will be committed. A circle with no program accepts every write the writer's role allows.

type ValidateTransaction = (input: {
  transaction: Transaction;
  role: Role; // the writer's role, from the member list or its link
  now: number; // the committer's clock, or the device's before it writes
  facts: Facts;
}) => Validation;
 
type Validation = { status: 'accepted' } | { status: 'rejected'; reason: string };
 
/** What a program reads. The database provides it, and a test provides it over an array. */
type Facts = {
  get(id: string, field: string): StoredFact | undefined; // the current value
  first(id: string): StoredFact | undefined; // the oldest fact of the id, so its writer created the id
  history(id: string, field: string): Cursor; // every value it held, oldest first
  find(field: string, value: Value): Cursor; // the current facts with this value
};
type StoredFact = {
  id: string;
  field: string;
  value: Value;
  writer: MemberId;
  committedAt: number;
};
type Cursor = { next(): StoredFact | undefined };
 
class Programs extends Context.Service<
  Programs,
  {
    // `wasm` gives the opened program object, and is read only the first time a layer sees the hash
    load(
      hash: Hash,
      wasm: Effect<Uint8Array, ProgramError>,
    ): Effect<ValidateTransaction, ProgramError>;
  }
>() {}

The feed's program lets only a post's author change it. This is its logic as TypeScript. The program itself is AssemblyScript compiled to WebAssembly, written with the program package's library, and testProgram runs that WebAssembly in the runner a committer uses, on facts in an array:

const validateTransaction: ValidateTransaction = ({ transaction, facts }) => {
  for (const change of transaction.changes) {
    if (!change.field.startsWith('post/')) continue;
    const author = facts.first(change.id)?.writer;
    if (author !== undefined && author !== transaction.writer) {
      return { status: 'rejected', reason: 'only the author edits it' };
    }
  }
  return { status: 'accepted' };
};

The first line of the loop is a rule for every program: it accepts a change to a field it doesn't know, so a map app can add place/rating to this circle with no change to the feed's program.

A program gets no clock but now, no network, and no randomness, and each run gets a new instance and memory, so two committers with the same commit, transactions, and time reach the same result. A runaway program is stopped by three limits for one transaction: the facts it reads, the time it runs, and its memory. The runner adds a call to a tick at the start of each function and each loop before it compiles a program, and the tick stops the run by the clock or by a count of ticks, because a Worker's clock doesn't move during a synchronous run. If a program crashes or passes a limit, the committer rejects the transaction with the reason program failed.

A program is stored like a page: sealed, as an object of its circle, in the commit whose circle record references it. The object holds the WebAssembly and the program's limits. Storage can't read it, and only an owner can add one. A Worker can't compile WebAssembly from bytes, so a committer there runs only the programs its bundle imported, compiled ahead with the tick.

Committers

The committer package exports the commit loop, which a server, an agent, and a device each run with their own layers:

/** What a member holds to act in a circle. For an identity, `sign` is its write key and `box` its box key. */
type MemberKeys = { member: MemberId; sign: Signer; box: HeldSealer };
 
/** Commits one circle for as long as it runs: it follows the circle on the relay, and runs `commitOnce` after each notice. */
declare const commitCircle: (
  circle: CircleId,
  member: MemberKeys, // it opens the circle's key box with these, and signs each head with them and the commit key
  options?: { sqlite?: WebAssembly.Module; now?: () => number },
) => Effect<never, CommitFailure, Storage | Relay | Programs>;
 
/** Opens one circle for committing, and keeps its database and program open for the scope. */
declare const openCommitter: (
  circle: CircleId,
  member: MemberKeys,
  options?: { sqlite?: WebAssembly.Module; now?: () => number },
) => Effect<{ commitOnce: Effect<void, CommitFailure>; headNumber: () => number }, NotListed | CommitFailure, Storage | Programs | Scope>;
 
type CommitFailure = StorageError | Refused | DatabaseError | ProgramError | FormatError | CryptoError | …; // the union the code infers

openCommitter reads the head, checks the circle records back to the first, and opens the member's key box. It fails with NotListed if the record doesn't list the member as a committer, because storage can't check that: the member list is sealed. Each commitOnce then:

1. Reads the head and the inbox in one storage read. If another commit stored a newer head, it opens the circle again from that head.
2. Takes every transaction after the head's inboxCounter, in order, and skips none. It rejects one if its seal doesn't open (seal), if its writer's signature is wrong or was made with an identity's earlier write key (signature), if its writer holds no role that writes (role), if a link's key signed anything but a join request, or a member changes circle/join (join request), if missing after 5 seconds (seal), if a blob it references is missing (blob), if the program rejects it (the program's reason), or if write gives a Conflict (conflict). It drops one whose writer, device, and deviceCounter are already settled. Until a missing one is rejected, it commits what precedes it and retries.
3. Seals the changed pages, the page map leaves they fall in, and a new root, and sends them through storage's commit, with a head whose inboxCounter is the last one taken.
4. On LostRace, opens the circle again from the newer head and repeats from step 2, and returns once nothing is waiting.

commitCircle runs commitOnce after each transaction or head notice, and each round reads the inbox again instead of trusting the notices, so a lost notice can't stall the loop. The order of a circle's writes is the order of inboxCounter, which storage sets, and a committer can't change it. A device applies pending transactions in the same order, so it shows what the commit will hold. A rejection settles a transaction too, so a head's inboxCounter says exactly which transactions are settled. Thirty people send at once follows this loop.

Changing the circle record

An owner changes the members, the committers, the links, or the program by committing a new circle record with commitRecord. The record is not a transaction and never passes through the inbox: the owner's device signs it with the own key and sends it as an object of a commit that accepts nothing. Its keys choice says what changes: same keeps the key version; data makes the next one, as after removing a member; and owners also makes new commit and own keys, as after removing an owner or a committer. An owner's device makes the commit and own keys, so every owner holds the commit key and no committer can hold a record back. Holding the commit key doesn't give a committer the own key, so a committer that isn't an owner can't sign a circle record.

The committer service

The host Worker runs a committer service beside storage and the relay. It is a key member whose keys come from the Worker's COMMITTER_SECRET, so every Worker and object derives the same keys and stores none.

Each circle has its own Durable Object, which keeps the circle's database and program open between rounds and runs one commitOnce a round. It keeps the objects and key boxes it reads or commits. After each accepted write, the Worker sends a request to that circle's object without waiting for it, because the Worker can't tell from the sealed member list whether the service commits the circle. A write is committed within about a second with no device online. The object commits only if the record lists the service, so the request needs no signature. If the record stops listing the service, the object stops serving the circle, and a round that fails tries again after 10 seconds.

The device

A device is the sdk on top of the device-store and keys packages. An unsent write exists only on its device, and a browser can evict a site's data, so the sdk sends each write as soon as storage answers. A browser tab, an agent, and a test run the same code with different platforms.

// device-store: a cache that may drop anything, and state that is never lost
class DeviceStore extends Context.Service<
  DeviceStore,
  {
    objects: {
      read(key: string): Effect<Uint8Array | undefined>;
      keep(key: string, bytes: Uint8Array): Effect<void>;
    };
    state: {
      read(key: string): Effect<Uint8Array | undefined, StoreError>;
      update<failure>(
        key: string,
        change: (stored: Uint8Array | undefined) => Effect<Uint8Array | undefined, failure>,
      ): Effect<void, StoreError | failure>;
    };
    replaced: Effect<void>; // completes once another tab upgraded or deleted the store
  }
>() {}
 
// sdk: what an app holds, once Loess.open({ storage }) has signed a device in
type Device = {
  identity: CircleId;
  circles: Stream<ReadonlyArray<Circle>>;
  circle(label: string): Effect<Circle, LoessError>; // the identity's circle with this label, or a new one it owns
  join(invite: string, label: string): Effect<Circle, LoessError>;
  status: Stream<ReadonlyArray<WriteStatus>>; // this device's unsettled and rejected writes
  dismiss(write: WriteId): Effect<void, LoessError>; // forget a rejected write
};
 
type Circle = {
  id: CircleId;
  write(changes: ReadonlyArray<Change>): Effect<WriteId, LoessError>; // a rejection shows in `status`
  query<row>(statement: Statement<row>): Effect<ReadonlyArray<row>, LoessError>;
  live<row>(statement: Statement<row>): Stream<ReadonlyArray<row>>;
  invite(role: 'read' | 'write', options?: { count?: number }): Effect<string, LoessError>; // as an owner, a link for `count` people, 1 by default
  remove(member: MemberId): Effect<void, LoessError>; // as an owner, with a new key version
  putBlob(bytes: Uint8Array): Effect<string, LoessError>; // seals a file and gives its reference, `blob:` and its hash
  // and label, members, readBlob, leave, signal, and signals
};
 
/** A write on its own device. It stays the same when the transaction is sealed again under new keys. */
type WriteId = { circle: CircleId; deviceCounter: number };
 
type WriteStatus = { write: WriteId } & (
  | { status: 'unchecked' } // the program needs pages this device doesn't hold, so no view shows the write yet
  | { status: 'unsent' } // checked, and kept on the device until storage accepts it
  | { status: 'sent' } // in the inbox; a committer accepts or rejects it even if this device never opens again
  | { status: 'rejected'; rejection: Rejection; changes: readonly Change[] } // by the program here or by a committer
);

A write is pending while it is unchecked, unsent, or sent. A committed write leaves status, and a rejected one stays until the app dismisses it. This is status for Eve in Write offline on a big channel, after her signal returns and before the next commit:

[
  { write: { circle: 'channel', deviceCounter: 7 }, status: 'sent' }, // her reaction
  { write: { circle: 'channel', deviceCounter: 8 }, status: 'sent' }, // the edit of her own message
  {
    write: { circle: 'channel', deviceCounter: 9 }, // the edit of someone else's
    status: 'rejected',
    rejection: { reason: 'only the author edits it' },
    changes: [{ id: 'message:2~bob', field: 'message/text', value: 'fixed' }],
  },
];

Each open circle has a session: the database over the newest head the device checked, with pending transactions written on top in memory. These are the device's own writes, along with any it heard from the relay. The outbox keeps the device's unsettled writes in the device store's state, and a cache keeps every sealed object, transaction, box, and blob it read in the device store's objects. When a new head arrives, the session checks it with validateHead, moves the database to that head's pages, and applies again each pending transaction the commit didn't settle. Rows maps typed rows onto facts, one id per row and one fact field per field.

The sdk runs the program on a write before write returns, so a rejected write is never sent and shows as rejected in status, even offline. It runs the program on another member's pending transaction too, after checking the writer's signature and role, and shows it only if the program accepts it. If the device lacks the pages to check a transaction, its own is unchecked, and someone else's isn't shown. A transaction is sent once storage stores its blobs. A transaction without blobs is one request, and its size depends on the transaction and not on the pages it changes.

If the device or its identity is listed as a committer, the session also commits the circle once a transaction waits.

Identity and keys

An identity is a circle whose members are keys, so it has no record type, storage key, or rule of its own. This is Ann's:

const ann: CircleRecord = {
  history: ['a41c…', '4be0…'], // the first record's hash is Ann's identity id, and the second added her passkey and her phone
  keyVersion: 2,
  keys: { commit: '0c3b…', own: 'e902…' },
  links: [],
  sealed: '…', // these CircleMembers:
};
 
const annMembers: CircleMembers = {
  members: [
    // each also lists its `box` key, left out here
    { member: 'passkey 1f…', role: 'own' }, // an owner adds and removes devices
    { member: 'phrase 8c…', role: 'own' },
    { member: 'phone d2…', role: 'write' }, // a device signs as Ann, and can't change her devices
    { member: 'laptop 6e…', role: 'write' },
  ],
  committers: [
    { committer: 'phrase 8c…' },
    { committer: 'phone d2…' },
    { committer: 'laptop 6e…' },
  ],
  program: null,
};

Ann's devices sign as Ann with her circle's write key, and another circle that lists Ann as a member encrypts its keys to the box key in her head. Her circle's facts list the circles she joined, each circle:<id> with a circle/label, and circle/link for one she joined by link. An identity's members are always keys and never another identity, so checking a member never takes a device past that member's own circle.

The first record lists only her phrase, and every byte of it is derived from the phrase, the seal's nonce included, so the phrase alone gives her id. Head 1 holds no pages. Sign-up commits a second record that adds the passkey as an owner and the device as a writer and a committer, with new commit and own keys in place of the ones derived from the phrase. An app comes with a storage it suggests, and a new identity starts there with no account or payment. The host of that storage decides who may start one, and pays for it. No service maps an identity to its storage, because it would see who signs in.

// keys: a device's own keys, which the platform can't export, kept in IndexedDB in a browser
function createDeviceKeys(store: DeviceKeyStore): Effect<DeviceKeys, KeyError>;

A user handle holds at most 64 bytes, so a passkey keeps only the 32-byte id, and the device finds the storage at the address the app suggests. Adding a device is one commit of the identity's record by an owner, and touches no other circle.

An identity's write key and box key are these keys of its circle, held only by its devices, its passkeys, and its phrase.

A key box holds one key version's secrets for its member's role, and every earlier version's circle key, box key, and public write key, so a new member opens older pages from one box, and a committer checks a transaction that waited through a key change. The own key is in an owner box of its own. Each box is sealed to its member's box key, and its id comes from keyBoxIdOf, which the member and any holder of the circle's box key work out the same, and storage can't. A member that is also a committer gets one box, which holds the commit key too.

Only an owner changes the owners' set, when an owner or a committer is removed. Any committer can make the next version of the data set, because a committer already reads everything the circle key opens. An owner's device makes that version in the commit that drops a member, and the new keys go in a key box for each member and committer that stays, in the same commit as the head. A removed member's device finds no box for the new version, and a committer that finds no box for itself stops committing.

Removing a device starts a new key version of the identity, which the removed device never gets. That device still holds the keys of every circle Ann is in, so each of those circles needs a new key version too. The diagram shows the design. The sdk removes the device and gives new keys to each circle Ann owns that is open on her device. The committer's new key version for the other circles is not built yet.

Joining by link

A link is a member with a single key: the circle record lists the link's public key with the read role, and the link's secret opens that member's keys. An invite link's key also signs, and the unsealed part of the circle record lists it with a count and a role. The owner approved the joiner when its device signed the link into the record, so a committer admits them with no owner online. Circle.invite(role, { count }) commits a link for count people, 1 by default, and gives its text, the circle id, the secret, and the storage's address, which Device.join opens.

• Storage accepts count transactions of at most 1 KiB signed by a link's key, so a one-use link admits one joiner and is then spent.
• If a stranger uses a leaked invite link first, they are admitted with the link's role until an owner removes them, and the person the link was for finds it spent.
• A writer holds a role if the circle record lists it, or if a committed join request admitted it from a link the record still lists.
• An owner closes a link by removing it, which starts a new key version like any removal. Until then the holder of a spent link can still read the circle.
• The joining device writes the circle id and the link's secret to its identity's circle, so the person's other devices read the circle too.

// Bob's join request: one change, in a transaction whose writer is the link's key.
const request: JoinRequest = {
  identity: bob,
  signature:
    yield * signBytes('joinRequest', joinRequestBody({ circle, link: link.key }), bobsWriteKey),
};
const join: Change = {
  id: `join:${bob}`,
  field: joinField,
  value: encodeCbor(request),
}; // joinField is 'circle/join'

The signature covers the circle and the link's key, so a leaked link can't admit someone else's identity. A committer accepts only a join request from a link's key, and commits it with a key box for the joiner. From that head the joiner is a member with the link's role.

Who checks what

The last row is the trade this design makes. A device opens a circle from the pages a committer built, so a committer is trusted: one that cheats can commit a write the program rejects, change a write, or write as a member. Storage keeps every transaction with its writer's signature, and every commit lists what it accepted and rejected, so storage holds everything needed to check a committer.

A device can't tell a withheld head from no new head, so storage can keep a device on an old head. It can't move the device back, because a device refuses a head with a lower number than the one it holds.

What storage sees

POST /read takes no signature, so anyone who knows a circle id can download its head and sealed objects and learns the same sizes and times. They can't open any of it.

Any storage

A device talks to whichever storage each circle lives on, and checks a circle on a stranger's storage exactly as on its own, because it verifies the head's signatures and each page's hash, and relies on nothing else the storage says.

Storage, the relay, and a committer are each a few lines of setup, and the same code runs on every set of layers:

// The reference storage: a Worker over an R2 bucket, with a Durable Object relay.
const reference = layerR2Storage(env.CIRCLES).pipe(
  Layer.provide(Layer.effect(StorageNotices, Relay)),
  Layer.provideMerge(layerDurableObjectRelay(env.RELAY)),
);
 
// The same two interfaces in process, for a test. The relay reads the same map to answer a follow.
const bucket = memoryBucket();
const inProcess = layerMemoryStorage(bucket).pipe(
  Layer.provide(Layer.effect(StorageNotices, Relay)),
  Layer.provideMerge(layerMemoryRelay(bucket)),
);
 
// A committer runs on either one, and so does a device.
const committer = commitCircle(circle, server).pipe(
  Effect.provide(Layer.mergeAll(inProcess, layerWasmPrograms())),
);

A committer is separate from storage, though one host often does both jobs. An owner changes a circle's committers in one commit. Moving a circle to other storage is not built, so the third row is where the design is going:

Package boundaries

An arrow points from a package to one it imports. Every package but device-store also imports protocol, and sdk imports them all. A format change means stored data can't be read and the preview data is cleared. A lint test fails when a public entry has no row, and ui, test-utils, and lint have none.

Use the table to tell where a problem is. If a guarantee isn't what the importers need, the architecture is wrong. If the guarantee is right and the names make it hard to keep, the interface is wrong. If both are right and the suite fails, the code is wrong.

To run only commits on a server, import the committer package with the http or r2 storage layer, and no device code, as the host Worker does.

Limits

Each limit is a default with a reason, and whoever the limit protects can change it. The decision log keeps when each number was set and what caused it.

Every storage accepts at least these defaults, so a write that fits one storage fits any other. A host that accepts more says so in its config, and the sdk takes the same values for that host. Storage doesn't refuse a transaction over 64 KiB yet.

Principles gives the price of each choice, and Goals gives the limits of trust. Two costs show only here:

• A lost race costs a committer one more try, so the cost grows with the committers committing at once, not with the writers.
• Anyone who holds the write key can fill the inbox.

Not built

These are in the design and not in the code yet:

• New keys for a circle the identity is in but doesn't own, after it removes a device, because a committer doesn't follow its members' identities.
• A device link, which adds a device before it exists.
• Sealing an unsent transaction again after a key change, and moving an admitted joiner into the member list.
• Checking a head's committer against the sealed member list.
• Sign-up offline.
• Asking the browser for persistent storage.
• A head notice that carries the circle records a device missed.
• Reads from the bucket's public domain, and an upload URL straight to the bucket.
• A transaction's 64 KiB limit in validate.
• A sign-off, where a member's device replays a commit and signs that it agrees, and a commit signed by more than one committer.
• Gated storage, which serves objects only to holders of the newest circle key.
• Deleting settled transactions and objects no head references, and checkpoints that drop old history.
• A circle on other storage than its identity's, and moving a circle or an identity to other storage.
• Changing a circle's committers or program from the sdk.
• An export of a full copy in one step.
• Public pages that anyone reads without Loess.
• Push, and presence while the app is closed.