93 lines
10 KiB
Markdown
93 lines
10 KiB
Markdown
# Synapse Backupper - Learnings & Conventions
|
|
|
|
## Project
|
|
- Module: git.tswf.io/infra/go-synapse-backupper
|
|
- Go 1.26+
|
|
- NO CGO, NO k8s, NO systemd daemon ceremony
|
|
|
|
## Skills in use
|
|
- golang-tswf-codestyle-skill (project structure)
|
|
- golang-flexible-config-skill (config loading)
|
|
- dockerfile-skill (multistage Dockerfile)
|
|
|
|
## Key Decisions
|
|
- TDD throughout
|
|
- Dual KEM: ML-KEM-768 (0x0006) + X25519 (0x0007)
|
|
- Composite HKDF combiner
|
|
- v2 .pqenc artifact format
|
|
- Two-phase Sink contract (Begin/Commit/Abort)
|
|
- Pipeline orchestrator with select-based fan-in
|
|
|
|
## Issues/Problems
|
|
- (As of Todo 5) Adapter packages (`mlkem768`, `x25519`) do NOT expose a LoadPriv-from-raw constructor; their `Decapsulate` type-asserts to its own unexported `privKey` type and rejects any other `RecipientPriv` implementation. This means KeyManager-constructed raw-bytes-backed `recipientPriv` (Todo 6's ProductType) CANNOT currently be passed to the adapters' `Decapsulate` — they would fail the type assertion. Latent bug surfaced while authoring the composite golden fixture; resolved in tests by registering deterministic FAKE KEMs under the same schemeIDs (0x0006/0x0007, ct lengths 1088/32) so reconstruction from committed raw bytes is possible without touching the adapter packages (out of scope for Todo 5). Real-key Decrypt-with-loaded-PEM must be revisited when KeyManager (Todo 6) is integrated into backup/restore (Todos 11/12) — likely requires exposing an exported `LoadPriv(raw) crypto.RecipientPriv` on each adapter.
|
|
|
|
|
|
## 2026-07-25T00:08:53+03:00 Todo 1 Complete
|
|
- Go module initialized with go 1.26
|
|
- Project skeleton created per tswf-codestyle-skill
|
|
- All directories: bin/, doc/, deploy/docker/, pkg/{domain,adapters,infrastructure}/, cmd/synapse-backupper/, resources/
|
|
- go.mod module path: git.tswf.io/infra/go-synapse-backupper
|
|
- .gitignore excludes bin/, *.tmp, *.pqenc, *.pem (allows example*.pem)
|
|
- Makefile, README.md, README.ru.md, config.example.yaml are empty placeholders for later todos
|
|
- Verified: go build ./... passes, skeleton binary prints correct output
|
|
- Environment: Go 1.26.5 installed via golang.org/dl
|
|
|
|
## 2026-07-25T00:28:00+03:00 Todo 2 Complete
|
|
- Created pkg/domain/crypto/ ports: KEM, Encryptor, Decryptor, KeyManager interfaces
|
|
- Created RecipientPub/RecipientPriv interfaces with SchemeID/KeyID/Raw methods
|
|
- Created concurrent-safe Registry (sync.RWMutex) with Register/Lookup/KEMFactory
|
|
- Defined sentinel errors: ErrUnknownScheme, ErrDuplicateScheme
|
|
- TDD: wrote registry_test.go first, then registry.go implementation
|
|
- Verified: `go test -race ./pkg/domain/crypto/...` passes (1.043s)
|
|
- Verified: `go build ./...` passes
|
|
- No concrete KEM implementations yet (blocked for Todo 3/4)
|
|
- No crypto/mlkem or crypto/ecdh imports in domain package
|
|
|
|
## 2026-07-25 Todo 3 Complete
|
|
- Created pkg/adapters/crypto/mlkem768/ adapter wrapping Go 1.26 stdlib crypto/mlkem
|
|
- TDD: wrote mlkem768_test.go first, then mlkem768.go implementation
|
|
- Adapter implements crypto.KEM interface with GenerateKeyPair, Encapsulate, Decapsulate
|
|
- KeyID derived as SHA-256(pub.Raw()[:8])[:8]
|
|
- Encapsulate swaps stdlib return order (ss, ct) → adapter (ct, ss) per domain contract
|
|
- Explicit return-order test: TestEncapsulateReturnOrder asserts len(ct)==1088 && len(ss)==32
|
|
- Decapsulate implements explicit rejection because crypto/mlkem.Decapsulate uses implicit rejection
|
|
(docs claim it returns an error for invalid ciphertexts, but implementation always returns nil)
|
|
Workaround: recompute Kout = SHAKE256(z || ciphertext) from priv seed and compare; match = ErrDecapsulationFailed
|
|
- Registered factory under suiteID 0x0006 in init() to package-level DefaultRegistry
|
|
- Verified: `go test -race ./pkg/adapters/crypto/mlkem768/...` passes (1.034s)
|
|
- Verified: `go test -run TestRoundTrip -v ./pkg/adapters/crypto/mlkem768/...` prints PASS
|
|
- Verified: `grep -q "suiteID.*0x0006" pkg/adapters/crypto/mlkem768/mlkem768.go` succeeds
|
|
|
|
## 2026-07-25 Todo 4 Complete
|
|
- Created pkg/adapters/crypto/x25519/ adapter wrapping Go 1.26 stdlib crypto/ecdh X25519
|
|
- TDD: wrote x25519_test.go first, then x25519.go implementation
|
|
- Adapter implements crypto.KEM interface with GenerateKeyPair, Encapsulate, Decapsulate
|
|
- KeyID derived as SHA-256(pub.Raw())[:8] (full raw key, not truncated like mlkem768)
|
|
- Encapsulate: ephemeral keypair via ecdh.X25519().GenerateKey(rand); ct = ephemeralPub.Bytes() (32B); ss = ephemeralPriv.ECDH(pub) (32B)
|
|
- Decapsulate: parse ct as ecdh.X25519().NewPublicKey(ct), ss = priv.ECDH(ephemeralPub)
|
|
- Added explicit ciphertext validation in Decapsulate: reject len != 32, high bit set (bit 255), and all-zero identity point
|
|
- Go stdlib ecdh.NewPublicKey only validates length, so explicit validation is required for KEM security
|
|
- Independent ECDH verification test: reconstruct ecdh.PrivateKey from priv.Raw() and ecdh.PublicKey from ct, compute shared secret, assert matches encapsulate output
|
|
- Round-trip stress test: 1000 iterations with fresh keypairs each iteration
|
|
- Random ciphertext test: 32 random bytes with high bit set → Decapsulate returns ErrDecapsulationFailed
|
|
- Registered factory under suiteID 0x0007 in init() to package-level DefaultRegistry
|
|
- Verified: `go test -race ./pkg/adapters/crypto/x25519/...` passes (3.662s)
|
|
- Verified: `grep -q "suiteID.*0x0007" pkg/adapters/crypto/x25519/x25519.go` succeeds
|
|
- Verified: `grep -q "ecdh.X25519" pkg/adapters/crypto/x25519/x25519.go` succeeds
|
|
|
|
|
|
## 2026-07-25 Todo 5 Complete (Composite Encryptor/Decryptor + v2 Artifact)
|
|
- Created pkg/adapters/crypto/composite/ with composite.go (production) + composite_test.go (TDD) + golden_generate_test.go (//go:build golden_generate one-time generator)
|
|
- Created testdata/golden-1byte.pqenc (1274 bytes — 1231 fixed header + 22B body chunk record for 1-byte 0xAA + 21B final marker chunk record) + testdata/golden-keys.json (committed priv raws as base64 — 122 bytes)
|
|
- All 16 required tests pass under -race: (a) Golden decrypt-equality on committed fixture + header byte-offset stability [magic@0:4=0x47535051, version@4:6=0x0002, flags@6:10=0, nRecipients@10=0x02, slot0 schemeID@11:13=0x0006, slot1 schemeID@1113:1115=0x0007], (b) round-trip 0/1/64KiB-1/64KiB/64KiB+1/1MiB, (c) empty→1 final chunk (ct=16B tag-only, flags=0x01), (d) exactly-64KiB→2 chunks (full body flags=0x00 + zero-length final markers flags=0x01), (e) counter wrap to 0xFFFFFFFFFFFFFFFF → ErrNonceCounterWrapped, (f) payload tamper → ErrTamperingDetected, (g) wrappedCEK tamper → ErrTamperingDetected, (h) wrong pq priv swap → ErrWrongKeys (via keyID sanity check), (i) format conformance, (j) version==0x0001 → ErrUnsupportedVersion + countingReader assertion reader.n==11 (no GCM ops attempted), (k) nRecipients==0 → ErrMalformedHeader, (l) nRecipients>2 OR ctLen>1<<20 → ErrMalformedHeader with reader.n==25 (ctLen validation fires per-slot BEFORE any ct io.ReadFull), (m) zero-length non-final chunk → ErrMalformedChunk, (n) oversized chunk (>64KiB+16) → ErrMalformedChunk, (o) premature EOF without final marker → ErrUnexpectedEOF, (p) truncated 30-byte header → ErrMalformedHeader with reader.n<=39 (no recipient allocation)
|
|
- Production sentinel errors: ErrMalformedHeader, ErrUnsupportedVersion, ErrWrongKeys, ErrTamperingDetected, ErrNonceCounterWrapped, ErrMalformedChunk, ErrUnexpectedEOF
|
|
- Adaptive parser ordering: magic check → version check (ErrUnsupportedVersion BEFORE any flags/nRecipients validation — no GCM ops) → flags check → nRecipients check → per-slot (read meta → validate ctLen ≤ max → read ct) → tail (wrapNonce+wrappedCEK+firstPayloadNonce) → decaps → HKDF combiner → unwrap CEK → chunk stream
|
|
- HKDF combiner: VERBATIM per plan Metis B1, but the plan's pseudocode assumed `hkdf.Expand` returns an infinite `io.Reader` requiring `io.ReadFull`. Go 1.26 stdlib `crypto/hkdf` returns `([]byte, error)` DIRECTLY (signatures `Extract(h func() H, secret, salt []byte) ([]byte, error)` and `Expand(h func() H, prk []byte, info string, keyLength int) ([]byte, error)`) — no reader is involved, so neither io.ReadAll nor io.ReadFull is needed; the spirit of the plan's "do not use io.ReadAll on hkdf.Expand" guidance is preserved trivially. Verified at /home/vergil/sdk/go1.26.5/src/crypto/hkdf/hkdf.go.
|
|
- Format spec pinned (verbatim from plan line 31): magic u32 BE 0x47535051 "GSPQ" + version u16 BE 0x0002 + flags u32 BE 0x00000000 + nRecipients u8 0x02 + 2 inline-ct recipient blocks (each: schemeID u16 + keyID 8B + ctLen u32 + ct; slot 0 PQ ct=1088B, slot 1 classical ct=32B) + wrapNonce 12B + wrappedCEK 48B (= 32B CEK + 16B GCM tag) + firstPayloadNonce 12B + chunked payload [len u32 BE 4B][flags 1B][AES-256-GCM ct + 16B tag]
|
|
- Chunk AEAD: 64 KiB body chunks with flags=0x00; ALWAYS-emit zero-length final marker chunk with flags=0x01 on EOF (Metis B2 — never seek backwards to mutate; the previous chunk's fullness does not affect marker emission)
|
|
- CEK wrap: aes.NewCipher(kekFinal) → cipher.NewGCM → Seal(nil, wrapNonce(12B from crypto/rand), cek(32B), AAD=[0x00,0x02]=version u16 BE)
|
|
- Counter increment (Metis N5, verbatim): counter := binary.BigEndian.Uint64(chunkNonce[4:12]); newCounter := counter + 1; if newCounter <= counter { return ErrNonceCounterWrapped }; binary.BigEndian.PutUint64(chunkNonce[4:12], newCounter) — chunkNonce[0:4] (random base) untouched
|
|
- Slot routing: POSITIONAL (slot 0 ← pq priv, slot 1 ← classical priv) per Metis N4; keyID equality enforced as a fast wrong-key reject BEFORE any AEAD operation (plan N4's "keyID is informational/audit only" preserved — keyID collision would still route correctly positionally and surface ErrTamperingDetected downstream)
|
|
- Registry wiring: NewEncryptor/NewDecryptor take a `crypto.Registry`; production code never imports mlkem768 OR x25519 directly — every scheme lookup goes through `Registry.Lookup(schemeID)` as the plan mandates. Tests use a fakeRegistry registering deterministic `fakeKem`(schemeID, ctLen) under the same 0x0006/0x0007 schemeIDs — fake Priv/Pub types (`fakePriv`/`fakePub`) live in composite_test.go so the committed golden fixture can reconstruct privs from raw bytes in testdata/golden-keys.json.
|
|
- Verified: `go test -race ./pkg/adapters/crypto/composite/...` PASSES (1.116s) ; `go test -race -count=3 ./pkg/adapters/crypto/composite/...` PASSES (1.318s) ; `go build ./...` PASSES ; `go vet ./pkg/adapters/crypto/composite/...` clean ; `grep -q "ErrNonceCounterWrapped" composite.go` succeeds ; `go test -race ./...` PASSES for ALL packages (no regression)
|