go-synapse-backupper/.omo/notepads/synapse-backupper/learnings.md

Synapse Backupper - Learnings & Conventions

Project

• Module: git.tswf.io/infra/go-synapse-backupper
• Go 1.26+
• NO CGO, NO k8s, NO systemd daemon ceremony

Skills in use

• golang-tswf-codestyle-skill (project structure)
• golang-flexible-config-skill (config loading)
• dockerfile-skill (multistage Dockerfile)

Key Decisions

• TDD throughout
• Dual KEM: ML-KEM-768 (0x0006) + X25519 (0x0007)
• Composite HKDF combiner
• v2 .pqenc artifact format
• Two-phase Sink contract (Begin/Commit/Abort)
• Pipeline orchestrator with select-based fan-in

Issues/Problems

• (As of Todo 5) Adapter packages (mlkem768, x25519) do NOT expose a LoadPriv-from-raw constructor; their Decapsulate type-asserts to its own unexported privKey type and rejects any other RecipientPriv implementation. This means KeyManager-constructed raw-bytes-backed recipientPriv (Todo 6's ProductType) CANNOT currently be passed to the adapters' Decapsulate — they would fail the type assertion. Latent bug surfaced while authoring the composite golden fixture; resolved in tests by registering deterministic FAKE KEMs under the same schemeIDs (0x0006/0x0007, ct lengths 1088/32) so reconstruction from committed raw bytes is possible without touching the adapter packages (out of scope for Todo 5). Real-key Decrypt-with-loaded-PEM must be revisited when KeyManager (Todo 6) is integrated into backup/restore (Todos 11/12) — likely requires exposing an exported LoadPriv(raw) crypto.RecipientPriv on each adapter.

2026-07-25T00:08:53+03:00 Todo 1 Complete

• Go module initialized with go 1.26
• Project skeleton created per tswf-codestyle-skill
• All directories: bin/, doc/, deploy/docker/, pkg/{domain,adapters,infrastructure}/, cmd/synapse-backupper/, resources/
• go.mod module path: git.tswf.io/infra/go-synapse-backupper
• .gitignore excludes bin/, *.tmp, *.pqenc, .pem (allows example.pem)
• Makefile, README.md, README.ru.md, config.example.yaml are empty placeholders for later todos
• Verified: go build ./... passes, skeleton binary prints correct output
• Environment: Go 1.26.5 installed via golang.org/dl

2026-07-25T00:28:00+03:00 Todo 2 Complete

• Created pkg/domain/crypto/ ports: KEM, Encryptor, Decryptor, KeyManager interfaces
• Created RecipientPub/RecipientPriv interfaces with SchemeID/KeyID/Raw methods
• Created concurrent-safe Registry (sync.RWMutex) with Register/Lookup/KEMFactory
• Defined sentinel errors: ErrUnknownScheme, ErrDuplicateScheme
• TDD: wrote registry_test.go first, then registry.go implementation
• Verified: go test -race ./pkg/domain/crypto/... passes (1.043s)
• Verified: go build ./... passes
• No concrete KEM implementations yet (blocked for Todo 3/4)
• No crypto/mlkem or crypto/ecdh imports in domain package

2026-07-25 Todo 3 Complete

• Created pkg/adapters/crypto/mlkem768/ adapter wrapping Go 1.26 stdlib crypto/mlkem
• TDD: wrote mlkem768_test.go first, then mlkem768.go implementation
• Adapter implements crypto.KEM interface with GenerateKeyPair, Encapsulate, Decapsulate
• KeyID derived as SHA-256(pub.Raw()[:8])[:8]
• Encapsulate swaps stdlib return order (ss, ct) → adapter (ct, ss) per domain contract
• Explicit return-order test: TestEncapsulateReturnOrder asserts len(ct)==1088 && len(ss)==32
• Decapsulate implements explicit rejection because crypto/mlkem.Decapsulate uses implicit rejection (docs claim it returns an error for invalid ciphertexts, but implementation always returns nil) Workaround: recompute Kout = SHAKE256(z || ciphertext) from priv seed and compare; match = ErrDecapsulationFailed
• Registered factory under suiteID 0x0006 in init() to package-level DefaultRegistry
• Verified: go test -race ./pkg/adapters/crypto/mlkem768/... passes (1.034s)
• Verified: go test -run TestRoundTrip -v ./pkg/adapters/crypto/mlkem768/... prints PASS
• Verified: grep -q "suiteID.*0x0006" pkg/adapters/crypto/mlkem768/mlkem768.go succeeds

2026-07-25 Todo 4 Complete

• Created pkg/adapters/crypto/x25519/ adapter wrapping Go 1.26 stdlib crypto/ecdh X25519
• TDD: wrote x25519_test.go first, then x25519.go implementation
• Adapter implements crypto.KEM interface with GenerateKeyPair, Encapsulate, Decapsulate
• KeyID derived as SHA-256(pub.Raw())[:8] (full raw key, not truncated like mlkem768)
• Encapsulate: ephemeral keypair via ecdh.X25519().GenerateKey(rand); ct = ephemeralPub.Bytes() (32B); ss = ephemeralPriv.ECDH(pub) (32B)
• Decapsulate: parse ct as ecdh.X25519().NewPublicKey(ct), ss = priv.ECDH(ephemeralPub)
• Added explicit ciphertext validation in Decapsulate: reject len != 32, high bit set (bit 255), and all-zero identity point
  • Go stdlib ecdh.NewPublicKey only validates length, so explicit validation is required for KEM security
• Independent ECDH verification test: reconstruct ecdh.PrivateKey from priv.Raw() and ecdh.PublicKey from ct, compute shared secret, assert matches encapsulate output
• Round-trip stress test: 1000 iterations with fresh keypairs each iteration
• Random ciphertext test: 32 random bytes with high bit set → Decapsulate returns ErrDecapsulationFailed
• Registered factory under suiteID 0x0007 in init() to package-level DefaultRegistry
• Verified: go test -race ./pkg/adapters/crypto/x25519/... passes (3.662s)
• Verified: grep -q "suiteID.*0x0007" pkg/adapters/crypto/x25519/x25519.go succeeds
• Verified: grep -q "ecdh.X25519" pkg/adapters/crypto/x25519/x25519.go succeeds

2026-07-25 Todo 5 Complete (Composite Encryptor/Decryptor + v2 Artifact)

• Created pkg/adapters/crypto/composite/ with composite.go (production) + composite_test.go (TDD) + golden_generate_test.go (//go:build golden_generate one-time generator)
• Created testdata/golden-1byte.pqenc (1274 bytes — 1231 fixed header + 22B body chunk record for 1-byte 0xAA + 21B final marker chunk record) + testdata/golden-keys.json (committed priv raws as base64 — 122 bytes)
• All 16 required tests pass under -race: (a) Golden decrypt-equality on committed fixture + header byte-offset stability [magic@0:4=0x47535051, version@4:6=0x0002, flags@6:10=0, nRecipients@10=0x02, slot0 schemeID@11:13=0x0006, slot1 schemeID@1113:1115=0x0007], (b) round-trip 0/1/64KiB-1/64KiB/64KiB+1/1MiB, (c) empty→1 final chunk (ct=16B tag-only, flags=0x01), (d) exactly-64KiB→2 chunks (full body flags=0x00 + zero-length final markers flags=0x01), (e) counter wrap to 0xFFFFFFFFFFFFFFFF → ErrNonceCounterWrapped, (f) payload tamper → ErrTamperingDetected, (g) wrappedCEK tamper → ErrTamperingDetected, (h) wrong pq priv swap → ErrWrongKeys (via keyID sanity check), (i) format conformance, (j) version==0x0001 → ErrUnsupportedVersion + countingReader assertion reader.n==11 (no GCM ops attempted), (k) nRecipients==0 → ErrMalformedHeader, (l) nRecipients>2 OR ctLen>1<<20 → ErrMalformedHeader with reader.n==25 (ctLen validation fires per-slot BEFORE any ct io.ReadFull), (m) zero-length non-final chunk → ErrMalformedChunk, (n) oversized chunk (>64KiB+16) → ErrMalformedChunk, (o) premature EOF without final marker → ErrUnexpectedEOF, (p) truncated 30-byte header → ErrMalformedHeader with reader.n<=39 (no recipient allocation)
• Production sentinel errors: ErrMalformedHeader, ErrUnsupportedVersion, ErrWrongKeys, ErrTamperingDetected, ErrNonceCounterWrapped, ErrMalformedChunk, ErrUnexpectedEOF
• Adaptive parser ordering: magic check → version check (ErrUnsupportedVersion BEFORE any flags/nRecipients validation — no GCM ops) → flags check → nRecipients check → per-slot (read meta → validate ctLen ≤ max → read ct) → tail (wrapNonce+wrappedCEK+firstPayloadNonce) → decaps → HKDF combiner → unwrap CEK → chunk stream
• HKDF combiner: VERBATIM per plan Metis B1, but the plan's pseudocode assumed hkdf.Expand returns an infinite io.Reader requiring io.ReadFull. Go 1.26 stdlib crypto/hkdf returns ([]byte, error) DIRECTLY (signatures Extract(h func() H, secret, salt []byte) ([]byte, error) and Expand(h func() H, prk []byte, info string, keyLength int) ([]byte, error)) — no reader is involved, so neither io.ReadAll nor io.ReadFull is needed; the spirit of the plan's "do not use io.ReadAll on hkdf.Expand" guidance is preserved trivially. Verified at /home/vergil/sdk/go1.26.5/src/crypto/hkdf/hkdf.go.
• Format spec pinned (verbatim from plan line 31): magic u32 BE 0x47535051 "GSPQ" + version u16 BE 0x0002 + flags u32 BE 0x00000000 + nRecipients u8 0x02 + 2 inline-ct recipient blocks (each: schemeID u16 + keyID 8B + ctLen u32 + ct; slot 0 PQ ct=1088B, slot 1 classical ct=32B) + wrapNonce 12B + wrappedCEK 48B (= 32B CEK + 16B GCM tag) + firstPayloadNonce 12B + chunked payload [len u32 BE 4B][flags 1B][AES-256-GCM ct + 16B tag]
• Chunk AEAD: 64 KiB body chunks with flags=0x00; ALWAYS-emit zero-length final marker chunk with flags=0x01 on EOF (Metis B2 — never seek backwards to mutate; the previous chunk's fullness does not affect marker emission)
• CEK wrap: aes.NewCipher(kekFinal) → cipher.NewGCM → Seal(nil, wrapNonce(12B from crypto/rand), cek(32B), AAD=[0x00,0x02]=version u16 BE)
• Counter increment (Metis N5, verbatim): counter := binary.BigEndian.Uint64(chunkNonce[4:12]); newCounter := counter + 1; if newCounter <= counter { return ErrNonceCounterWrapped }; binary.BigEndian.PutUint64(chunkNonce[4:12], newCounter) — chunkNonce[0:4] (random base) untouched
• Slot routing: POSITIONAL (slot 0 ← pq priv, slot 1 ← classical priv) per Metis N4; keyID equality enforced as a fast wrong-key reject BEFORE any AEAD operation (plan N4's "keyID is informational/audit only" preserved — keyID collision would still route correctly positionally and surface ErrTamperingDetected downstream)
• Registry wiring: NewEncryptor/NewDecryptor take a crypto.Registry; production code never imports mlkem768 OR x25519 directly — every scheme lookup goes through Registry.Lookup(schemeID) as the plan mandates. Tests use a fakeRegistry registering deterministic fakeKem(schemeID, ctLen) under the same 0x0006/0x0007 schemeIDs — fake Priv/Pub types (fakePriv/fakePub) live in composite_test.go so the committed golden fixture can reconstruct privs from raw bytes in testdata/golden-keys.json.
• Verified: go test -race ./pkg/adapters/crypto/composite/... PASSES (1.116s) ; go test -race -count=3 ./pkg/adapters/crypto/composite/... PASSES (1.318s) ; go build ./... PASSES ; go vet ./pkg/adapters/crypto/composite/... clean ; grep -q "ErrNonceCounterWrapped" composite.go succeeds ; go test -race ./... PASSES for ALL packages (no regression)