quicproquo/docs/operations/backup-restore.md

# Backup and Restore Procedures

This document covers backup and restore for all quicprochat server data stores.

## Data Inventory

| Data | Location | Backend | Contains |
|------|----------|---------|----------|
| SQLCipher DB | `QPC_DB_PATH` (default `data/qpc.db`) | `store_backend=sql` | Users, key packages, delivery queues, sessions, KT log, OPAQUE setup, blobs metadata, moderation |
| File store | `QPC_DATA_DIR` (default `data/`) | `store_backend=file` | Bincode-serialized key packages, delivery queues, server state |
| Blob storage | `QPC_DATA_DIR/blobs/` | Filesystem | Uploaded file transfer blobs |
| TLS certificates | `QPC_TLS_CERT`, `QPC_TLS_KEY` | DER files | Server identity |
| OPAQUE ServerSetup | Inside DB or file store | Persisted | OPAQUE credential state (critical for auth) |
| Server signing key | Inside DB or file store | Persisted | Ed25519 key for delivery proofs |
| KT Merkle log | Inside DB or file store | Persisted | Key transparency audit log |

## SQLCipher Backup

### Hot Backup (Online)

SQLCipher supports the `.backup` command while the server is running (WAL mode allows concurrent readers).

```bash
# 1. Open the encrypted database with the same key
sqlite3 data/qpc.db

# 2. At the sqlite3 prompt, set the encryption key
PRAGMA key = 'your-db-key-here';

# 3. Perform an online backup
.backup /backups/qpc-$(date +%Y%m%d-%H%M%S).db

.quit
```

### Scripted Hot Backup

```bash
#!/bin/bash
set -euo pipefail

BACKUP_DIR="/backups/qpc"
DB_PATH="${QPC_DB_PATH:-data/qpc.db}"
DB_KEY="${QPC_DB_KEY}"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_FILE="${BACKUP_DIR}/qpc-${TIMESTAMP}.db"

mkdir -p "$BACKUP_DIR"

sqlite3 "$DB_PATH" <<EOF
PRAGMA key = '${DB_KEY}';
.backup ${BACKUP_FILE}
EOF

# Verify the backup is readable
sqlite3 "$BACKUP_FILE" "PRAGMA key = '${DB_KEY}'; PRAGMA integrity_check;" \
  | grep -q "ok" && echo "Backup verified: $BACKUP_FILE" \
  || { echo "ERROR: backup verification failed"; exit 1; }

# Retain last 7 daily backups
find "$BACKUP_DIR" -name 'qpc-*.db' -mtime +7 -delete
```

### Cold Backup (Offline)

```bash
# 1. Stop the server
systemctl stop qpc-server   # or docker compose stop server

# 2. Copy the database file
cp data/qpc.db /backups/qpc-$(date +%Y%m%d).db

# 3. Copy the WAL and SHM files if they exist
cp data/qpc.db-wal /backups/ 2>/dev/null || true
cp data/qpc.db-shm /backups/ 2>/dev/null || true

# 4. Restart the server
systemctl start qpc-server
```

## File Backend Backup

When using `store_backend=file`, data is stored as bincode files under `QPC_DATA_DIR`.

```bash
# Full directory backup
tar czf /backups/qpc-data-$(date +%Y%m%d-%H%M%S).tar.gz \
  -C "$(dirname "${QPC_DATA_DIR:-data}")" \
  "$(basename "${QPC_DATA_DIR:-data}")"
```

## Blob Storage Backup

Blobs are stored in `QPC_DATA_DIR/blobs/`. These are immutable once written.

```bash
# Incremental rsync (blobs are write-once, ideal for rsync)
rsync -av --progress data/blobs/ /backups/blobs/
```

## TLS Certificate Backup

```bash
# Back up TLS certificates (store separately from DB backups)
cp data/server-cert.der /backups/tls/server-cert.der
cp data/server-key.der /backups/tls/server-key.der

# Federation certs (if federation is enabled)
cp data/federation-cert.der /backups/tls/federation-cert.der 2>/dev/null || true
cp data/federation-key.der /backups/tls/federation-key.der 2>/dev/null || true
cp data/federation-ca.der /backups/tls/federation-ca.der 2>/dev/null || true
```

## Restore Procedures

### Restore SQLCipher Database

```bash
# 1. Stop the server
systemctl stop qpc-server

# 2. Move the current (corrupt/lost) database aside
mv data/qpc.db data/qpc.db.broken 2>/dev/null || true
rm -f data/qpc.db-wal data/qpc.db-shm

# 3. Copy the backup in place
cp /backups/qpc-20260304.db data/qpc.db

# 4. Verify integrity
sqlite3 data/qpc.db "PRAGMA key = '${QPC_DB_KEY}'; PRAGMA integrity_check;"

# 5. Start the server (migrations will apply automatically if needed)
systemctl start qpc-server
```

### Restore File Backend

```bash
# 1. Stop the server
systemctl stop qpc-server

# 2. Replace the data directory
mv data data.broken 2>/dev/null || true
tar xzf /backups/qpc-data-20260304.tar.gz -C .

# 3. Restore TLS certs if not included in the data backup
cp /backups/tls/server-cert.der data/server-cert.der
cp /backups/tls/server-key.der data/server-key.der

# 4. Start the server
systemctl start qpc-server
```

### Restore Blobs Only

```bash
rsync -av /backups/blobs/ data/blobs/
```

## Backup Schedule Recommendations

| Frequency | What | Method |
|-----------|------|--------|
| Every 6 hours | SQLCipher database | Hot backup script via cron |
| Daily | File backend / full data dir | tar + offsite copy |
| Continuous | Blobs | rsync (incremental) |
| On change | TLS certificates | Manual + secret manager |

## Cron Example

```cron
# SQLCipher hot backup every 6 hours
0 */6 * * * /opt/qpc/scripts/backup-db.sh >> /var/log/qpc-backup.log 2>&1

# Full data directory daily at 02:00
0 2 * * * tar czf /backups/qpc-data-$(date +\%Y\%m\%d).tar.gz -C /var/lib quicprochat

# Blob sync every hour
0 * * * * rsync -a /var/lib/quicprochat/blobs/ /backups/blobs/

# Prune backups older than 30 days
0 3 * * 0 find /backups -name 'qpc-*' -mtime +30 -delete
```

## Verification

Always verify backups after creation:

```bash
# SQLCipher integrity check
sqlite3 /backups/qpc-latest.db \
  "PRAGMA key = '${QPC_DB_KEY}'; PRAGMA integrity_check; SELECT count(*) FROM users;"

# File backend: check the archive is valid
tar tzf /backups/qpc-data-latest.tar.gz > /dev/null

# TLS cert: check it parses and is not expired
openssl x509 -inform DER -in /backups/tls/server-cert.der -noout -dates
```