docs: rewrite mdBook documentation for v2 architecture

Update 25+ files and add 6 new pages to reflect the v2 migration from Cap'n Proto to Protobuf framing over QUIC. Integrates SDK and Operations docs into the mdBook, restructures SUMMARY.md, and rewrites the wire format, architecture, and protocol sections with accurate v2 content.
2026-03-04 22:02:31 +01:00
parent f7a7f672b4
commit d073f614b3
31 changed files with 4423 additions and 2379 deletions
--- a/docs/src/wire-format/delivery-schema.md
+++ b/docs/src/wire-format/delivery-schema.md
@@ -1,193 +1,377 @@
-# Delivery Schema
+# Delivery and Keys Schema

-**Schema file:** `schemas/delivery.capnp`
-**File ID:** `@0xc5d9e2b4f1a83076`
+**Proto files:** `proto/qpq/v1/delivery.proto`, `proto/qpq/v1/keys.proto`
+**Package:** `qpq.v1`
+**Method IDs:** 200-205 (delivery), 300-304 (key packages and hybrid keys), 510-520 (key transparency)

-The `DeliveryService` interface defines the RPC contract for the store-and-forward message relay. The DS is intentionally MLS-unaware: it routes opaque byte strings by recipient key and optional channel ID without parsing or inspecting the content.
+This page documents the Protobuf message definitions for the delivery service (store-and-forward message relay) and the key management service (MLS KeyPackages, hybrid post-quantum keys, and key transparency).

 ---

-## Full schema listing
+## delivery.proto

-```capnp
-# delivery.capnp -- Delivery Service RPC interface.
-#
-# The Delivery Service is a simple store-and-forward relay. It does not parse
-# MLS messages -- all payloads are opaque byte strings routed by recipient key.
-#
-# Callers are responsible for:
-#   - Routing Welcome messages to the correct new member after add_members().
-#   - Routing Commit messages to any existing group members (other than self).
-#   - Routing Application messages to the intended recipient(s).
-#
-# The DS indexes queues by the recipient's raw Ed25519 public key (32 bytes),
-# matching the indexing scheme used by the Authentication Service.
-#
-# ID generated with: capnp id
-@0xc5d9e2b4f1a83076;
+The delivery service is a store-and-forward relay. It is intentionally MLS-unaware: all payloads are opaque byte strings routed by recipient key and channel ID. The server never inspects or decrypts message content.

-interface DeliveryService {
-  # Enqueue an opaque payload for delivery to a recipient.
-  #
-  # recipientKey : Ed25519 public key of the intended recipient (exactly 32 bytes).
-  # payload      : Opaque byte string -- a TLS-encoded MlsMessageOut blob or any
-  #                other framed data the application layer wants to deliver.
-  # channelId    : Optional channel identifier (empty for legacy). A 16-byte UUID
-  #                is recommended for 1:1 channels.
-  # version      : Schema/wire version. Must be 0 (legacy) or 1 (this spec).
-  #
-  # The payload is appended to the recipient's FIFO queue. Returns immediately;
-  # the recipient retrieves it via `fetch`.
-  enqueue @0 (recipientKey :Data, payload :Data, channelId :Data, version :UInt16) -> ();
+### Full proto listing

-  # Fetch and atomically drain all queued payloads for a given recipient.
-  #
-  # recipientKey : Ed25519 public key of the caller (exactly 32 bytes).
-  # channelId    : Optional channel identifier (empty for legacy).
-  # version      : Schema/wire version. Must be 0 (legacy) or 1 (this spec).
-  #
-  # Returns the complete queue in FIFO order and clears it. Returns an empty
-  # list if there are no pending messages.
-  fetch @1 (recipientKey :Data, channelId :Data, version :UInt16) -> (payloads :List(Data));
+```protobuf
+syntax = "proto3";
+package qpq.v1;
+
+// Delivery service: enqueue, fetch, peek, ack, batch (6 methods).
+// Method IDs: 200-205.
+
+message Envelope {
+  uint64 seq = 1;
+  bytes data = 2;
+}
+
+message EnqueueRequest {
+  bytes recipient_key = 1;
+  bytes payload = 2;
+  bytes channel_id = 3;
+  uint32 ttl_secs = 4;
+  // Client-generated idempotency key (16 bytes, UUID v7).
+  // Server deduplicates enqueue requests with the same message_id within a TTL window.
+  bytes message_id = 5;
+}
+
+message EnqueueResponse {
+  uint64 seq = 1;
+  bytes delivery_proof = 2;
+  // True if this was a duplicate enqueue (message_id already seen).
+  bool duplicate = 3;
+}
+
+message FetchRequest {
+  bytes recipient_key = 1;
+  bytes channel_id = 2;
+  uint32 limit = 3;
+  // Device ID for multi-device scoping.
+  bytes device_id = 4;
+}
+
+message FetchResponse {
+  repeated Envelope payloads = 1;
+}
+
+message FetchWaitRequest {
+  bytes recipient_key = 1;
+  bytes channel_id = 2;
+  uint64 timeout_ms = 3;
+  uint32 limit = 4;
+  bytes device_id = 5;
+}
+
+message FetchWaitResponse {
+  repeated Envelope payloads = 1;
+}
+
+message PeekRequest {
+  bytes recipient_key = 1;
+  bytes channel_id = 2;
+  uint32 limit = 3;
+  bytes device_id = 4;
+}
+
+message PeekResponse {
+  repeated Envelope payloads = 1;
+}
+
+message AckRequest {
+  bytes recipient_key = 1;
+  bytes channel_id = 2;
+  uint64 seq_up_to = 3;
+  bytes device_id = 4;
+}
+
+message AckResponse {}
+
+message BatchEnqueueRequest {
+  repeated bytes recipient_keys = 1;
+  bytes payload = 2;
+  bytes channel_id = 3;
+  uint32 ttl_secs = 4;
+  bytes message_id = 5;
+}
+
+message BatchEnqueueResponse {
+  repeated uint64 seqs = 1;
 }
 ```

---
+### Envelope

-## Method-by-method analysis
-
-### `enqueue @0`
-
-```
-enqueue (recipientKey :Data, payload :Data, channelId :Data, version :UInt16) -> ()
-```
-
-**Purpose:** Append an opaque payload to a recipient's delivery queue. The DS stores the payload until the recipient fetches it. The call returns immediately after the payload is enqueued; it does not block until delivery.
-
-**Parameters:**
-
-| Parameter | Type | Size | Description |
-|---|---|---|---|
-| `recipientKey` | `Data` | Exactly 32 bytes | Ed25519 public key of the intended recipient. Used as the primary queue index. |
-| `payload` | `Data` | Variable (bounded by transport max) | Opaque byte string. Typically a TLS-encoded `MlsMessageOut` blob, but the DS does not inspect it. |
-| `channelId` | `Data` | 0 bytes (legacy) or 16 bytes (UUID) | Channel identifier for channel-aware routing. Empty `Data` is treated as the legacy default channel. |
-| `version` | `UInt16` | 2 bytes | Schema/wire version. `0` = legacy (no channel routing), `1` = current spec (channel-aware). |
-
-**Return value:** Void. The method returns `()` on success. Errors are surfaced as Cap'n Proto RPC exceptions.
-
-**Queue semantics:** Payloads are appended in FIFO order. The DS does not deduplicate, reorder, or inspect payloads. Multiple enqueue calls for the same recipient and channel ID are simply appended to the queue in the order they arrive.
-
-### `fetch @1`
-
-```
-fetch (recipientKey :Data, channelId :Data, version :UInt16) -> (payloads :List(Data))
-```
-
-**Purpose:** Fetch and atomically drain all queued payloads for a given recipient on a given channel. This is the "pull" side of the store-and-forward relay.
-
-**Parameters:**
-
-| Parameter | Type | Size | Description |
-|---|---|---|---|
-| `recipientKey` | `Data` | Exactly 32 bytes | Ed25519 public key of the caller. Must match the key used in the enqueue calls. |
-| `channelId` | `Data` | 0 bytes (legacy) or 16 bytes (UUID) | Channel identifier. Must match the `channelId` used during enqueue. |
-| `version` | `UInt16` | 2 bytes | Schema/wire version. Must match the version used during enqueue. |
-
-**Return value:**
+The `Envelope` wrapper is returned by fetch, peek, and fetch-wait operations.

 | Field | Type | Description |
-|---|---|---|
-| `payloads` | `List(Data)` | All queued payloads in FIFO order. Empty list if no messages are pending. |
+|-------|------|-------------|
+| `seq` | `uint64` | Server-assigned monotonic sequence number for ordering and acknowledgment. |
+| `data` | `bytes` | The original payload bytes submitted at enqueue time. |

-**Atomic drain:** The fetch operation returns the entire queue and clears it in a single atomic operation. There is no "peek" or partial fetch. This simplifies the concurrency model: the client processes all returned payloads and does not need to track which ones it has already seen.
+### Enqueue (ID 200)
+
+Appends an opaque payload to a recipient's queue. Returns immediately.
+
+**Request:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `recipient_key` | `bytes` | Recipient's Ed25519 identity public key (32 bytes). Primary queue index. |
+| `payload` | `bytes` | Opaque byte string. Typically a TLS-encoded MLS ciphertext blob. |
+| `channel_id` | `bytes` | Channel identifier (16-byte UUID v7 recommended). Empty = default channel. |
+| `ttl_secs` | `uint32` | Time-to-live in seconds. Server garbage-collects expired messages. 0 = server default. |
+| `message_id` | `bytes` | Client-generated idempotency key (16 bytes, UUID v7). Server deduplicates within the TTL window. |
+
+**Response:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `seq` | `uint64` | Server-assigned sequence number for this message. |
+| `delivery_proof` | `bytes` | Cryptographic proof of delivery (reserved for future use). |
+| `duplicate` | `bool` | `true` if this `message_id` was already seen within the TTL window; the payload was not stored again. |
+
+### Fetch (ID 201)
+
+Returns and retains queued messages up to `limit`. Does not remove messages from the queue; use `Ack` to advance the read cursor.
+
+**Request:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `recipient_key` | `bytes` | Recipient's Ed25519 identity public key (32 bytes). |
+| `channel_id` | `bytes` | Channel identifier. Must match the value used at enqueue time. |
+| `limit` | `uint32` | Maximum number of envelopes to return. 0 = server default. |
+| `device_id` | `bytes` | Optional device identifier for multi-device queue scoping. |
+
+**Response:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `payloads` | `repeated Envelope` | Messages in FIFO order. Empty list if no messages are pending. |
+
+### FetchWait (ID 202)
+
+Long-poll variant of `Fetch`. Blocks on the server until messages arrive or `timeout_ms` elapses.
+
+**Request:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `recipient_key` | `bytes` | Recipient's Ed25519 identity public key (32 bytes). |
+| `channel_id` | `bytes` | Channel identifier. |
+| `timeout_ms` | `uint64` | Maximum wait time in milliseconds. 0 = return immediately (equivalent to Fetch). |
+| `limit` | `uint32` | Maximum number of envelopes to return. |
+| `device_id` | `bytes` | Optional device identifier. |
+
+**Response:** Same as `FetchResponse`.
+
+FetchWait eliminates polling latency: the server holds the RPC open until a `Notify` is signalled by a concurrent `Enqueue` call, or until `timeout_ms` expires.
+
+### Peek (ID 203)
+
+Non-destructive read. Returns messages without removing them and without advancing the acknowledgment cursor.
+
+**Request / Response:** Same field layout as `FetchRequest` / `FetchResponse`.
+
+Peek is useful for inspecting pending messages without marking them as delivered.
+
+### Ack (ID 204)
+
+Advances the delivery cursor, removing all messages with `seq <= seq_up_to` from the queue.
+
+**Request:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `recipient_key` | `bytes` | Recipient's Ed25519 identity public key (32 bytes). |
+| `channel_id` | `bytes` | Channel identifier. |
+| `seq_up_to` | `uint64` | All messages with sequence number <= this value are removed. |
+| `device_id` | `bytes` | Optional device identifier. |
+
+**Response:** Empty (`AckResponse {}`).
+
+### BatchEnqueue (ID 205)
+
+Fan-out: enqueues the same payload to multiple recipients in a single RPC call.
+
+**Request:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `recipient_keys` | `repeated bytes` | List of recipient Ed25519 identity public keys. |
+| `payload` | `bytes` | Opaque payload, delivered identically to all recipients. |
+| `channel_id` | `bytes` | Channel identifier. |
+| `ttl_secs` | `uint32` | Time-to-live in seconds. |
+| `message_id` | `bytes` | Idempotency key (16 bytes). |
+
+**Response:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `seqs` | `repeated uint64` | Server-assigned sequence numbers, one per `recipient_key`, in the same order. |

 ---

-## Channel-aware routing
+## keys.proto

-The `channelId` field enables per-channel queue separation. Each unique `(recipientKey, channelId)` pair maps to an independent FIFO queue on the server.
+Key management for MLS KeyPackages, hybrid post-quantum keys, and key transparency audit.

-### Compound key structure
+### Full proto listing

-```text
-Queue Key = recipientKey (32 bytes) || channelId (0 or 16 bytes)
+```protobuf
+syntax = "proto3";
+package qpq.v1;
+
+// Key package + hybrid key CRUD (5 methods).
+// Method IDs: 300-304.
+
+message UploadKeyPackageRequest {
+  bytes identity_key = 1;
+  bytes package = 2;
+}
+
+message UploadKeyPackageResponse {
+  bytes fingerprint = 1;
+}
+
+message FetchKeyPackageRequest {
+  bytes identity_key = 1;
+}
+
+message FetchKeyPackageResponse {
+  bytes package = 1;
+}
+
+message UploadHybridKeyRequest {
+  bytes identity_key = 1;
+  bytes hybrid_public_key = 2;
+}
+
+message UploadHybridKeyResponse {}
+
+message FetchHybridKeyRequest {
+  bytes identity_key = 1;
+}
+
+message FetchHybridKeyResponse {
+  bytes hybrid_public_key = 1;
+}
+
+message FetchHybridKeysRequest {
+  repeated bytes identity_keys = 1;
+}
+
+message FetchHybridKeysResponse {
+  repeated bytes keys = 1;
+}
+
+// Key revocation (method ID 510).
+message RevokeKeyRequest {
+  bytes identity_key = 1;
+  string reason = 2; // "compromised", "superseded", "user_revoked"
+}
+
+message RevokeKeyResponse {
+  bool success = 1;
+  uint64 leaf_index = 2;
+}
+
+// Check revocation status (method ID 511).
+message CheckRevocationRequest {
+  bytes identity_key = 1;
+}
+
+message CheckRevocationResponse {
+  bool revoked = 1;
+  string reason = 2;
+  uint64 timestamp_ms = 3;
+}
+
+// KT audit log retrieval (method ID 520).
+message AuditKeyTransparencyRequest {
+  uint64 start = 1;
+  uint64 end = 2;
+}
+
+message AuditKeyTransparencyResponse {
+  repeated LogEntry entries = 1;
+  uint64 tree_size = 2;
+  bytes root = 3;
+}
+
+message LogEntry {
+  uint64 index = 1;
+  bytes leaf_hash = 2;
+}
 ```

-When `channelId` is empty (0 bytes), the queue key degenerates to just the `recipientKey`, preserving backward compatibility with legacy clients that do not use channels.
+### UploadKeyPackage (ID 300)

-### Channel ID format
+Uploads a single-use MLS KeyPackage. KeyPackages are stored in a FIFO queue per identity; each is consumed once by `FetchKeyPackage`.

-The recommended format for `channelId` is a 16-byte UUID (128-bit, typically UUID v4). The DS treats the channel ID as an opaque byte string and does not parse its structure. Using UUIDs provides:
+| Field | Type | Description |
+|-------|------|-------------|
+| `identity_key` | `bytes` | Uploader's Ed25519 identity public key (32 bytes). Index key for the queue. |
+| `package` | `bytes` | openmls-serialised KeyPackage (bincode format, as required by `DiskKeyStore`). |

-1. **Collision resistance** -- 2^122 random bits (for UUID v4) makes accidental collision negligible.
-2. **Privacy** -- The channel ID reveals no information about the channel's participants or purpose.
-3. **Fixed size** -- 16 bytes is compact and predictable for indexing.
+Response: `fingerprint` -- SHA-256 digest of the stored package (32 bytes). Callers should record this to detect tampering.

-### Use cases
+### FetchKeyPackage (ID 301)

-| Scenario | channelId | recipientKey | Result |
-|---|---|---|---|
-| Legacy client, no channels | Empty (0 bytes) | Alice's Ed25519 key | Single queue for all of Alice's messages |
-| 1:1 channel between Alice and Bob | UUID of the 1:1 channel | Alice's Ed25519 key | Separate queue for this specific channel |
-| Group channel | UUID of the group channel | Alice's Ed25519 key | Separate queue for this group's messages to Alice |
+Fetches and atomically removes one KeyPackage for the given identity. Returns empty bytes if no packages are stored. The removal is atomic; concurrent fetches will not receive the same package.

---
+### UploadHybridKey (ID 302)

-## Version field
+Uploads the client's hybrid (X25519 + ML-KEM-768) public key. Unlike KeyPackages, hybrid keys are not single-use -- each identity stores exactly one, overwriting the previous value.

-The `version` field provides a mechanism for wire-level schema evolution without breaking existing clients.
+| Field | Type | Description |
+|-------|------|-------------|
+| `identity_key` | `bytes` | Uploader's Ed25519 identity public key (32 bytes). |
+| `hybrid_public_key` | `bytes` | Concatenated X25519 public key (32 bytes) + ML-KEM-768 encapsulation key. |

-| Version | Semantics |
-|---|---|
-| `0` | Legacy mode. `channelId` is ignored (treated as empty). Behaves like the pre-channel DeliveryService. |
-| `1` | Current specification. `channelId` is used for channel-aware routing. |
+### FetchHybridKey (ID 303)

-The server validates the version field and rejects unknown versions as protocol errors. Clients must set the version field to match the schema revision they implement.
+Fetches a single peer's hybrid public key. Non-destructive.

---
+### FetchHybridKeys (ID 304)

-## FIFO queue semantics
+Batch variant of `FetchHybridKey`. Returns one key per input identity key, in the same order. Missing keys are returned as empty bytes at the corresponding index.

-The Delivery Service provides strict FIFO ordering within each `(recipientKey, channelId)` queue:
+### RevokeKey (ID 510)

-1. **Enqueue order is preserved.** Payloads are returned by `fetch` in the exact order they were enqueued.
-2. **Atomic drain.** Each `fetch` call returns all pending payloads and clears the queue. There is no risk of partial reads or interleaving.
-3. **No persistence guarantees (current implementation).** The in-memory queue is lost on server restart. Persistent storage is planned for a future milestone.
-4. **No redelivery.** Once a payload is returned by `fetch`, it is permanently removed. If the client crashes before processing it, the payload is lost. Reliable delivery with acknowledgments is a future enhancement.
+Revokes an identity key by appending a revocation entry to the key transparency Merkle log.

---
+| Field | Type | Description |
+|-------|------|-------------|
+| `identity_key` | `bytes` | Identity key to revoke (32 bytes). |
+| `reason` | `string` | One of: `"compromised"`, `"superseded"`, `"user_revoked"`. |

-## MLS-unaware design
+Response: `leaf_index` is the index of the revocation entry in the KT Merkle log.

-The DS intentionally does not parse, validate, or inspect MLS messages. All payloads are opaque `Data` blobs. This design has several consequences:
+### CheckRevocation (ID 511)

- **Security:** The server cannot extract plaintext from MLS ciphertext, even if compromised.
- **Simplicity:** The DS has no dependency on openmls or any MLS library.
- **Flexibility:** The same DS can carry non-MLS payloads (e.g., signaling, metadata) without modification.
- **No server-side optimization:** The DS cannot optimize delivery based on MLS message type (e.g., fanning out a Commit to all group members). The client must enqueue separately for each recipient.
+Checks whether an identity key has been revoked.

-For the full design rationale, see [ADR-004: MLS-Unaware Delivery Service](../design-rationale/adr-004-mls-unaware-ds.md).
+Response fields: `revoked` (bool), `reason` (string), `timestamp_ms` (uint64 unix milliseconds of the revocation event).

---
+### AuditKeyTransparency (ID 520)

-## Relationship to NodeService
+Returns a range of entries from the key transparency append-only Merkle log.

-In the current unified architecture, the Delivery Service methods are exposed as part of the [NodeService interface](node-service-schema.md) with additional methods:
+| Field | Type | Description |
+|-------|------|-------------|
+| `start` | `uint64` | First leaf index (inclusive). |
+| `end` | `uint64` | Last leaf index (exclusive). 0 = up to current tree size. |

-| DeliveryService Method | NodeService Method | Additional Parameters |
-|---|---|---|
-| `enqueue @0` | `enqueue @2` | `auth :Auth` |
-| `fetch @1` | `fetch @3` | `auth :Auth` |
-| *(none)* | `fetchWait @4` | `auth :Auth`, `timeoutMs :UInt64` |
-
-The `fetchWait` method is a NodeService extension that provides long-polling semantics: it blocks until either new payloads arrive or the timeout expires. This avoids the latency and bandwidth overhead of repeated `fetch` polling.
+Response: `entries` (list of `LogEntry`), `tree_size` (current log size), `root` (Merkle root hash).

 ---

 ## Further reading

- [Wire Format Overview](overview.md) -- serialisation pipeline context
- [NodeService Schema](node-service-schema.md) -- unified interface that subsumes DeliveryService
- [Auth Schema](auth-schema.md) -- the companion service for KeyPackage management
- [Envelope Schema](envelope-schema.md) -- legacy framing that used `mlsWelcome`/`mlsCommit`/`mlsApplication` message types
- [ADR-004: MLS-Unaware Delivery Service](../design-rationale/adr-004-mls-unaware-ds.md) -- why the DS does not inspect MLS content
+- [Wire Format Overview](overview.md) -- frame format and transport parameters
+- [Method ID Reference](envelope-schema.md) -- all 44 method IDs
+- [Auth Schema](auth-schema.md) -- OPAQUE authentication proto definitions
+- [RPC Reference](node-service-schema.md) -- all proto definitions for all 14 files
+- [Storage Backend](../internals/storage-backend.md) -- how KeyPackages and hybrid keys are persisted