deploy/replication/README.md
On this page

Ciphertext and puzzle replicas

deploy/mirror-artifacts.py repeatedly copies public relay artifacts to independent stores. It supports a local directory (on Hetzner or the operator's Mac), AWS S3, and Cloudflare R2 through its S3-compatible API. It never receives relay credentials, traffic plaintext, epoch secrets, or access to the enclave credential broker.

The production target is AWS + Hetzner + Cloudflare, with a Mac copy as an additional operator backup. The three providers are independent administrative failure domains; copying into three directories on one machine does not satisfy that requirement. Configuration labels describe the operator's placement and are not evidence of physical independence.

Current status: implementation and local HTTP/storage tests are complete. These example configuration files are not evidence of deployed replicas. The renewed Cloudflare login has not been revalidated; R2 deployment and readback remain pending. Do not report three live providers until actual readback succeeds from all three configured services.

Public archive — verified 2026-09-10

The archive's artifacts/ prefix is now publicly readable over HTTPS, with anonymous prefix-scoped discovery:

Browse public artifacts.

The listing is the S3 XML API, not the relay's JSON discovery protocol. Direct object downloads work without credentials. The applied public policy grants only HTTPS GetObject under artifacts/* and ListBucket restricted to that prefix. ACL blocking remains on; no public write or delete is granted. Root listing, other-prefix listing and an existing other-prefix object returned anonymous 403. Account-wide settings were not changed.

All ten existing intended-public objects passed anonymous full-byte SHA256 verification. A fresh diagnostic ciphertext copy uploaded by the actual Hetzner mirror service identity also passed anonymous readback, idempotence, and duplicate conditional-write rejection. The archive now contains eleven objects in this prefix; these are diagnostic artifacts/copies, not completed production epochs. The fresh production attestation still reports warming and the current origin index is empty. See launch-review evidence.

Deletion protection and upload guarantees

The current AWS bucket has versioning and a separate mirror identity limited to GetObject and PutObject on artifacts/*. The uploader cannot delete versions, create deletion markers, change retention, or modify bucket policy under its current permissions. Object Lock is enabled with 30-day COMPLIANCE retention for new versions. Existing artifacts/ versions are protected for 30 days from backfill. See the configuration evidence.

AWS Object Lock compliance mode can prevent deletion of protected versions, including by the account root user, until a selected retention date. That period cannot be shortened. AWS documents account deletion as an exception. Apply protection to puzzles, attestation bundles and ciphertext together, and retain version IDs: new versions and deletion markers can hide a retained version from ordinary unversioned reads. Governance mode and removable legal holds do not provide the same administrator protection. The selected duration is 30 days. It protects stored versions for that period, not their perpetual public accessibility or the configuration of future uploads. The mirror supplies an explicit Content-MD5 upload checksum, required by S3 when default retention applies; artifact authenticity/readback still use SHA256.

Object Lock protects only data that actually reaches S3. The existing enclave accepts a parent storage acknowledgment, while independent mirrors upload later. To promise that every returned response already has a protected S3 copy, a future measured enclave must itself verify authenticated S3 acceptance and the required retention before releasing that response. A parent-supplied success flag does not establish that guarantee.

Discovery and restart behavior

The origin must implement GET /v1/artifacts/index.json?limit=256&after=NAME:

{"protocol_version":2,"artifacts":["64hex.record.json"],
 "next_cursor":"64hex.record.json","truncated":true}

Names must be strictly increasing and must match a SHA-256 address followed by .puzzle.json, .attestation.json, .record.json, or .bundle.json. A truncated page must supply its final name as the next cursor. A terminal page must supply next_cursor:null and truncated:false. Broken cursors, duplicate names, unknown fields, duplicate JSON fields, traversal names, oversized responses and altered content are rejected. Source HTTP redirects are rejected, including cross-origin and HTTPS-to-HTTP redirects.

Every interval starts a complete scan from the first page. Sorting by content hash means new objects may appear before an older cursor; full repeated scans avoid silently losing them. Each target is independently checked against every object. One unavailable object-store target does not stop writes to other configured targets. Failed transfers are retried on the next scan. A scan is not proof that a malicious origin disclosed every historical record, and no mirror can recover artifacts that were never made available.

Conditional creation and verification

The daemon downloads exact bytes, verifies the SHA-256 address, and reads back each replica's bytes to verify the same hash. Content-hash metadata and HTTP ETags alone are never accepted as proof. Existing wrong bytes, truncated files, symlinks, and nonregular destination files are rejected and preserved for inspection.

Directory replicas first write and fsync a fresh exclusive staging file. A no-replace hardlink publishes the completed file, then the directory is synced. This means a crash cannot publish a partially written final artifact. Staging links are retained to comply with the no-deletion requirement; successful staging/final names share the same file data rather than storing duplicate bytes. Interrupted staging files remain unused. The daemon never removes old content or staging files.

S3 and R2 writes send IfNoneMatch="*". A rejected precondition triggers readback, not an overwrite. An implementation that does not support conditional creation fails explicitly; there is no unconditional-write fallback. Existing objects are downloaded and verified. This protects writes by this daemon; bucket owners or other credentials with write/delete access can still modify availability separately.

The behavior uses the documented AWS conditional PutObject API and R2 conditional request compatibility. R2 uses the account endpoint https://ACCOUNT_ID.r2.cloudflarestorage.com and region auto. The botocore checksum mode is configured to avoid unsupported optional streaming checksum trailers while maintaining local full-byte SHA-256 verification.

Configuration and operation

Directory-only copies need Python 3.10+ and the standard library. Object storage also needs pip install -r deploy/replication/requirements.txt. Copy an example configuration into an operator-controlled path and replace origin, bucket, account, and destination paths. Keep credentials in named AWS SDK profiles outside the repository; the configuration contains profile names, never API key values. The example IAM policy grants only s3:GetObject and s3:PutObject for the archive prefix. This daemon needs no delete, bucket-administration, or list-bucket action.

python3 deploy/mirror-artifacts.py --config .local/mirror.json --once \
  --receipt-dir .local/replication-receipts
python3 deploy/mirror-artifacts.py --config .local/mirror.json --once --verify-only
python3 deploy/mirror-artifacts.py --config .local/mirror.json \
  --receipt-dir .local/replication-receipts

--once exits nonzero if a replica fails or a scan is incomplete. --verify-only performs complete downloads and verification without uploading missing objects. It still requires origin discovery. Independently retrieving a known artifact requires only its name: open the directory file or call S3 GetObject on artifacts/NAME, then verify its SHA-256 against the first 64 characters of NAME.

scan_complete reports discovered objects and each destination's created, verified, and failed counts. With --receipt-dir, a completed scan also produces a durable SHA256.replication.json receipt containing the observation time, origin, and those counts. A receipt is an operational observation from this daemon, not a signed enclave attestation or a promise of indefinite retention. Its content address permits checking whether the report itself changed. SDK error internals are kept out of logs so credentialed URLs cannot leak through error messages.

The systemd unit assumes code and a Python virtual environment under /opt/attested-relay, a relay-mirror service account, config under /etc/attested-relay/mirror.json, and private SDK profiles in /etc/attested-relay/mirror-credentials. The service writes only under /var/lib/relay-mirror. Install it only after a successful --once readback run. The macOS LaunchAgent template runs the directory-only operator copy; adjust paths and create its .local configuration before loading it.

If a public HTTPS archive is not available, ssh-tunnel.example.json accepts explicit loopback HTTP from a separately authenticated SSH tunnel. Nonloopback HTTP is always rejected. Tunnel provisioning and security-group changes are separate deployment operations; no reusable SSH private key belongs in config.

Tests

python3 -m unittest discover -s deploy/replication/tests -v

Tests include real local HTTP pagination and file retrieval, idempotent restarts, newly inserted artifacts, incomplete staging, existing corrupt files, path attacks, symlinks, malformed/truncated index pages, bounded reads, conditional S3 writes, and corrupted object-store readback. The S3 client tests are simulated; successful AWS/R2 production reads must be recorded separately before deployment is called complete.