# Delivery Service Internals The Delivery Service (DS) is a store-and-forward relay for opaque MLS payloads. It never inspects, decrypts, or validates MLS ciphertext -- it routes solely by recipient identity key and channel identifier. The DS exposes three operations through the `NodeService` RPC interface: `enqueue`, `fetch`, and `fetchWait`. **Sources:** - `crates/quicprochat-server/src/main.rs` (RPC handlers) - `crates/quicprochat-server/src/storage.rs` (queue storage) - `schemas/node.capnp` (wire schema) --- ## Architecture ```text NodeService (port 7000) ======================= enqueue(recipientKey, payload, channelId) | v +---------------------------------------------------------+ | FileBackedStore | | | | deliveries: Mutex>>>| | ^ ^ | | | | | | ChannelKey { FIFO queue of | | channel_id, opaque payload | | recipient_key bytes | | } | | | | Persisted to: data/deliveries.bin (bincode, V2 format) | +---------------------------------------------------------+ | v notify_waiters() --> DashMap, Arc> ^ | keyed by recipient_key wakes blocked fetchWait calls ``` The DS is intentionally MLS-unaware. This design decision is documented in [ADR-004: MLS-Unaware Delivery Service](../design-rationale/adr-004-mls-unaware-ds.md). From the server's perspective, every payload is an opaque blob -- it could be a Welcome, a Commit, an application message, or a hybrid-encrypted envelope. --- ## Queue Model ### ChannelKey Delivery queues are indexed by a compound key: ```rust #[derive(Serialize, Deserialize, Clone, Eq, PartialEq, Debug)] pub struct ChannelKey { pub channel_id: Vec, pub recipient_key: Vec, } ``` | Field | Size | Purpose | |-----------------|-------------|---------| | `channel_id` | Variable (typically 16 bytes UUID or empty) | Isolates messages by conversation. Empty for legacy/default channel. | | `recipient_key` | 32 bytes | Ed25519 public key of the intended recipient. | The `ChannelKey` implements `Hash` manually, hashing `channel_id` followed by `recipient_key`. **Channel-aware routing** ensures that messages for different conversations do not interfere with each other. A client fetching from channel A will not see messages enqueued for channel B, even if both target the same recipient. For legacy clients (or single-channel usage), `channel_id` is left empty. ### Queue Structure Each `ChannelKey` maps to a `VecDeque>`: ```text ChannelKey("chan-1", "alice-pk") -> [msg_1, msg_2, msg_3] ChannelKey("chan-1", "bob-pk") -> [msg_4] ChannelKey("chan-2", "alice-pk") -> [msg_5, msg_6] ChannelKey("", "alice-pk") -> [msg_7] (legacy/default channel) ``` Messages within a queue are ordered FIFO (first-in, first-out). This preserves MLS epoch ordering, which is critical: a recipient must process a Welcome before application messages, and Commits in the order they were produced. --- ## RPC Operations ### enqueue Appends a payload to the recipient's queue and wakes any blocked long-poll waiters. ```capnp enqueue @2 (recipientKey :Data, payload :Data, channelId :Data, version :UInt16, auth :Auth) -> (); ``` **Handler logic:** 1. **Parse parameters.** Extract `recipientKey`, `payload`, `channelId`, `version`, and `auth` from the Cap'n Proto request. 2. **Validate auth.** Call `validate_auth()` to check the `Auth` struct. See [Authentication Service Internals](authentication-service.md) for auth validation details. 3. **Validate inputs:** | Field | Constraint | Error on Violation | |----------------|-------------------------|--------------------| | `recipientKey` | Exactly 32 bytes | `"recipientKey must be exactly 32 bytes, got {n}"` | | `payload` | Non-empty | `"payload must not be empty"` | | `payload` | At most 5 MB | `"payload exceeds max size (5242880 bytes)"` | | `version` | 0 (legacy) or 1 (current) | `"unsupported wire version {v} (expected 0 or 1)"` | 4. **Store.** Call `FileBackedStore::enqueue(recipient_key, channel_id, payload)`, which constructs a `ChannelKey` from the channel ID and recipient key, then pushes the payload to the back of the corresponding `VecDeque`. The entire delivery map is flushed to disk. 5. **Notify waiters.** Look up or create a `tokio::sync::Notify` for the recipient key in `DashMap, Arc>` and call `notify_waiters()`. This wakes all `fetchWait` calls currently blocked on this recipient. ### fetch Atomically drains the entire queue for a recipient+channel and returns all payloads. ```capnp fetch @3 (recipientKey :Data, channelId :Data, version :UInt16, auth :Auth) -> (payloads :List(Data)); ``` **Handler logic:** 1. Parse and validate `recipientKey` (32 bytes), `version` (0 or 1), and `auth`. 2. Call `FileBackedStore::fetch(recipient_key, channel_id)`, which: - Constructs a `ChannelKey`. - Calls `VecDeque::drain(..)` on the matching queue, collecting all messages. - Flushes the updated (now empty) map to disk. - Returns the drained messages as `Vec>`. 3. Build a `List(Data)` response with all the payload bytes. **Important:** The drain is atomic with respect to the `Mutex` lock. No interleaving with concurrent `enqueue` calls is possible. The returned list preserves FIFO order. ### fetchWait (Long-Polling) Combines `fetch` with a blocking wait. If the queue is empty, the server waits for up to `timeoutMs` milliseconds for a new message to arrive. ```capnp fetchWait @4 (recipientKey :Data, channelId :Data, version :UInt16, timeoutMs :UInt64, auth :Auth) -> (payloads :List(Data)); ``` **Handler logic:** ```text 1. validate inputs (same as fetch) 2. messages = store.fetch(recipient_key, channel_id) 3. if messages.is_empty() AND timeout_ms > 0: a. waiter = waiters.entry(recipient_key).or_insert(Arc::new(Notify::new())) b. tokio::time::timeout(Duration::from_millis(timeout_ms), waiter.notified()).await c. messages = store.fetch(recipient_key, channel_id) // re-fetch after wake 4. return messages ``` The implementation uses `Promise::from_future(async move { ... })` because the `tokio::time::timeout` call is async. This is the only DS handler that produces an async `Promise`. **Timeout behavior:** - If `timeout_ms == 0`, `fetchWait` behaves identically to `fetch` (immediate return). - If a message arrives before the timeout, `notify_waiters()` from `enqueue` wakes the `Notify`, and the handler re-fetches immediately. - If the timeout expires without a message, the handler re-fetches (which will return empty) and returns an empty list. **Waiter model:** The `DashMap, Arc>` is keyed by recipient key (not by `ChannelKey`). This means a notification for any channel targeting the same recipient will wake all blocked `fetchWait` calls for that recipient. This is a deliberate simplification -- the re-fetch after waking will only return messages from the requested channel, so cross-channel wake-ups result in a no-op re-fetch rather than incorrect behavior. --- ## Version Validation The `version` field in `enqueue`, `fetch`, and `fetchWait` enables future schema evolution: | Version | Meaning | |---------|---------| | 0 | Legacy (pre-versioning). `channelId` is treated as empty. | | 1 | Current wire format. `channelId` is a meaningful field. | | 2+ | Rejected with `"unsupported wire version"`. | Both 0 and 1 are accepted on the server side. The constant `CURRENT_WIRE_VERSION = 1` is used in validation: ```rust if version != 0 && version != CURRENT_WIRE_VERSION { return Promise::err(/* unsupported version */); } ``` The client library always sends `version: 1` for new operations. --- ## Notification System The waiter map provides a lightweight push-notification mechanism: ```text enqueue() fetchWait() | | v v store.enqueue(key, ch, payload) messages = store.fetch(key, ch) | | v | (if empty) waiter = waiters.entry(key) v .or_insert(Notify::new()) waiter = waiters.entry(key) | .or_insert(Notify::new()) v | waiter.notify_waiters() v | timeout(duration, waiter.notified()) | | +------- wakes ----------------------->+ | v messages = store.fetch(key, ch) | v return messages ``` `tokio::sync::Notify` is a broadcast notification primitive. `notify_waiters()` wakes all tasks currently awaiting `.notified()`. If no tasks are waiting, the notification is lost (there is no stored permit in the `notify_waiters()` path). This is acceptable because `fetchWait` always performs a fetch before blocking, so messages that arrive before the wait begins are returned immediately. --- ## Data Flow Example: Two-Party Message Exchange ```text Alice Server DS Bob | | | | encrypt("hello bob") | | | -> ct_bytes | | | | | | enqueue(bob_pk, ct_bytes) | | | -------------------------> | | | | queue[("", bob_pk)] += ct | | | notify_waiters(bob_pk) | | | | | | <--- fetchWait(bob_pk, 30s) | | | (was blocked, now woken)| | | | | | drain queue[("", bob_pk)] | | | ---- [ct_bytes] -----------> | | | | | | decrypt(ct_bytes) | | | -> "hello bob" | ``` --- ## Server Constants | Constant | Value | Purpose | |-------------------------|-----------|---------| | `MAX_PAYLOAD_BYTES` | 5,242,880 (5 MB) | Maximum size of a single enqueued payload | | `MAX_KEYPACKAGE_BYTES` | 1,048,576 (1 MB) | Maximum size of a KeyPackage (AS) | | `CURRENT_WIRE_VERSION` | 1 | Current schema version; rejects > 1 | --- ## Persistence Delivery queues are persisted to `data/deliveries.bin` using bincode serialization. The V2 format uses `ChannelKey` as the map key: ```rust #[derive(Serialize, Deserialize, Default)] struct QueueMapV2 { map: HashMap>>, } ``` On load, the server attempts V2 deserialization first. If that fails, it falls back to V1 format (keyed by `Vec` recipient key only) and migrates in memory by assigning empty `channel_id` to each entry: ```rust for (recipient_key, queue) in legacy.map.into_iter() { upgraded.insert( ChannelKey { channel_id: Vec::new(), recipient_key }, queue, ); } ``` See [Storage Backend](storage-backend.md) for the full persistence model. --- ## Related Pages - [Authentication Service Internals](authentication-service.md) -- KeyPackage storage and retrieval - [GroupMember Lifecycle](group-member-lifecycle.md) -- how `send_message()` and `receive_message()` produce and consume the payloads - [Storage Backend](storage-backend.md) -- `FileBackedStore` persistence and migration - [NodeService Schema](../wire-format/node-service-schema.md) -- Cap'n Proto schema reference - [ADR-004: MLS-Unaware Delivery Service](../design-rationale/adr-004-mls-unaware-ds.md) -- design rationale - [End-to-End Data Flow](../architecture/data-flow.md) -- sequence diagrams for registration, group creation, and messaging