storage

Storage keeps each circle's bytes under the keys that protocol's storageKeyOf builds. It replaces the bytes under a key only with a compare-and-swap, and only when protocol's validate accepts the request.

Interface

Storage is one Effect service with five calls:

Call	What it does
`read(keys)`	the bytes under each key, or undefined for a missing key
`has(keys)`	whether each key holds bytes
`write(circle, transactions)`	stores each transaction under its hash, adds it to the inbox, and gives each one's `inboxCounter`
`upload(upload, blob)`	stores a sealed blob under its hash, after checking it against its signed `BlobUpload`
`commit(head, objects, boxes)`	stores the objects and the key boxes at once, then swaps the head

Each call reads the circle's head and inbox at once. A circle record is stored under its hash, so the process keeps up to 8 MiB of records it read and doesn't read them again. A call fails with StorageError when storage can't be reached, with protocol's Refused when validate refuses it, and commit fails with LostRace when another commit stored the same head number first.

Entry	Gives
`@loess/storage/memory`	`layerMemoryStorage`, over a map that `memoryBucket` makes, for tests
`@loess/storage/r2`	`layerR2Storage`, a stateless layer over an R2 binding for a Worker
`@loess/storage/http`	`handleStorageRequest`, which serves any storage over HTTP; `layerHttpStorage`, its client; and `storageRoutes`, the route table both read
`@loess/storage/test`	`testStorage`, the contract suite

The memory and R2 layers publish each new transaction and head through StorageNotices, a service of this package, so storage doesn't import the relay package and its own tests run without a relay. The relay package's Relay has the same publish, so the host Worker provides StorageNotices from the Durable Object relay. To run storage and a relay in one process, provide it from a memory relay over the same map:

const bucket = memoryBucket();
const inProcess = layerMemoryStorage(bucket).pipe(
  Layer.provide(Layer.effect(StorageNotices, Relay)),
  Layer.provideMerge(layerMemoryRelay(bucket)),
);

write publishes each transaction twice. It publishes it with no inboxCounter as soon as the request arrives, before it reads or puts anything, so the relay passes it to other devices while storage stores it. It publishes it again with its inboxCounter once it is new in the inbox. A device sends each write only to storage.

What it promises

A writer never gets a conflict. write retries the inbox's swap itself, and 30 transactions sent at once all reach the inbox on their first try, each with its own counter.
A transaction still waiting in the inbox gets the counter it has, and nothing is added. Each swap drops the entries up to the head's inboxCounter.
When two commits send the same head number, the second gets LostRace. Sending a stored head again succeeds, so a commit cut off halfway can be sent again.
A refused request changes nothing.

Tests

pnpm vitest run --project storage runs testStorage against the memory layer, against the HTTP client of the memory layer, and against the R2 layer in a Worker in miniflare, called over HTTP. The memory layer's test records each notice. It also delays a write's put to the bucket and checks that storage published the write before the put finished. The host Worker's test follows the Durable Object relay over a WebSocket and checks that each write and commit reaches it.

Edit on GitHub