Architecture

For operators and anyone self-hosting. If you just want to use the platform, start with How it works.

Repository layout

The platform lives in a monorepo with two independent packages — no shared build, no shared dependencies, no shared deploy pipeline:

cavs-hub/
├── cavs-node/     the platform → Google Cloud Run
└── cavs-docs/     this documentation site → Vercel (docs.cavsnode.com)

Everything on this page describes cavs-node/, and every path is relative to it.

Services

Binary	Package	Responsibility
cavs-api	cmd/api	Control plane and data plane HTTP API under /api/v1
cavs-worker	cmd/worker	Pub/Sub consumers, scheduled maintenance, GC/repack/index coordination
cavs-web	web/	React + Vite + TypeScript dashboard
cavs-admin	cmd/admin	Operational CLI
cavs-r2exporter	cmd/r2exporter	Scrapes Cloudflare R2 analytics into Prometheus metrics

Why metadata and transfer aren't separate services

The original specification listed cavs-metadata and cavs-transfer as separate deployables. They live inside cavs-api as distinct route groups (/uploads, /downloads, /lfs) — a clean seam that can be extracted later without touching domain code. Premature distribution buys operational complexity, not scale; the data plane already avoids the API for bytes.

Data stores

Store	Holds	Notes
MongoDB	all metadata	Indexes and plan seeds created idempotently on boot. Production uses a replica set (Atlas) for transactions and read scaling.
S3-compatible object store	all bytes	Cloudflare R2 in production; MinIO for local dev; any S3-compatible endpoint for BYOS.
Google Cloud Pub/Sub	events	One topic, repository id as the ordering key. Emulator locally.
Redis	optional cache	Not required.

The control/data plane split

Control plane (internal/store, internal/api/handlers_*) — MongoDB is the source of truth for users, organizations, membership, repositories, tokens, plans, subscriptions, usage, jobs and audit.

Data plane (internal/storage, handlers_dataplane.go) — never runs complex control-plane queries per byte. It resolves a repository and permissions once, then hands out presigned S3 URLs. Objects are content-addressed:

org/{orgID}/repo/{repoID}/objects/<oid[0:2]>/<oid[2:4]>/<oid>
org/{orgID}/repo/{repoID}/cas/<tree relative path>

The one exception is the CAVS chunk tree (GET/PUT …/lfs/*), which is proxied because the agent needs authorized ranged reads over a content-addressed tree. Those are small cacheable reads, not whole-object transfers.

Middleware chain

Every request passes, in order:

1. RealIP — trust the proxy’s forwarded address
2. Correlation — attach/propagate X-Correlation-Id and X-Request-Id
3. OpenTelemetry — server spans; a no-op when tracing is disabled
4. Metrics — golden-signal HTTP metrics with a normalized route label
5. Access log — structured, with status and byte counts
6. Security headers — CSP, HSTS, no-store, and the rest
7. Recoverer — a panic becomes a 500, not a dropped connection
8. Timeout — 60 seconds
9. CORS — explicit origin, method and header allowlists
10. Authenticate (under /api/v1) — Firebase ID token or CAVS API token
11. Rate limit — 30 req/s, burst 60, keyed per token → user → IP
12. loadOrg / loadRepo — resolve the path resource and the caller’s membership
13. guard — evaluate the effective permission

Authentication and authorization

Two credential types on the same Authorization: Bearer header:

1. Firebase ID token — RS256, verified against Google’s JWKS with issuer and audience checks. In dev-bypass mode claims are decoded without verification (forced off in production).
2. CAVS API token — cavs_pat_, cavs_repo_, cavs_ci_; SHA-256 hashed and looked up. Scope is a hard cap: a PAT’s effective permission is user_role ∩ token_scope; repo and CI tokens are authorized by scope alone.

Effective permission composes organization role + repository override + token scope + account status + organization status. See Roles & permissions.

Repository access is independent of org membership

The repository guard grants access via the override path even when the caller has no organization membership. That’s what makes external repository collaborators work without joining the organization. Organization routes, by contrast, hard-require membership.

Events

One topic, cavs.events; consumers route on event_type. Delivery is at-least-once and a processed_events collection makes consumers idempotent. Repository id is the ordering key, so per-repository events stay ordered.

Client setup is non-blocking and topic/subscription creation is lazy, so services boot fast even if the broker is still coming up. With no emulator in development the API falls back to a no-op bus — events aren’t delivered, everything else works.

A subset of internal events maps to public webhook events; only those leave the platform. See the Event catalog.

Worker stages

Stage	Trigger	Status
Notifications	invitation, quota, billing events	live
Usage reconciliation	upload.session.finalized, repository.usage.changed	live
Cleanup	every 5 minutes	live
Storage health	~every 10 minutes	live
Webhooks	events with a public mapping	live
Snapshots	snapshot.requested	live
Health & recommendations	scheduled	live
Quota enforcement	repository.quota.exceeded	live
GC	gc.requested	coordinator only
Repack	repack.requested	coordinator only
Index compaction	index.compaction.requested	coordinator only

The Rust maintenance seam

onGC, onRepack and onIndexCompaction are coordinators: they record the activity and emit *.completed, but the byte-level scan-mark-quarantine-delete work is not implemented. The design puts it in a separate Rust cavs-maintenance binary driven by the same contract — Pub/Sub in, object storage and MongoDB out, no FFI — so it can be added with no change to the Go control plane.

Operationally: unreferenced chunks are not reclaimed today. Plan capacity on what has been written. See Lifecycle & maintenance.

Every stage is idempotent and stateless beyond MongoDB and Pub/Sub, so cavs-worker scales horizontally. To split it into separate deployables (-notifications, -usage, -maintenance), give each its own subscription id and filter by event type.

Caution

The worker also serves /healthz and /readyz on $PORT. This is required on platforms with HTTP startup probes — Cloud Run among them. A worker without it never becomes Ready, and the platform keeps serving the stale revision, which means invitations, usage reconciliation and GC silently run old code.

The Git index layer

A read-only, Git-semantic layer over the oid-only data plane, populated by cav repo index and the pre-push hook.

Collection	Holds
repo_commits	Append-only upsert by (repo, sha); artifacts array capped at 1000; text index on the message
repo_refs	Branches and tags, with a pointer to the current tree generation
repo_tree_entries	Replace-on-push snapshots via atomic generation swap
repo_releases	Derived from tags; name and notes user-editable
repo_events	The activity feed
index_sessions	In-flight ingest sessions

Ingest is paged — 500 commits or 5000 tree entries per request — with retries on 429 and 5xx. Old generations are swept on finalize and on session expiry. Every read endpoint degrades with "indexed": false.

Storage resolution

internal/resolver maps a repository to its backing object store: the process-wide managed store, or a per-organization BYOS connection. BYOS credentials are encrypted at rest with AES-256-GCM (internal/cryptobox) under a master key held outside the database, decrypted only in memory when a connection is used.

Observability

• Metrics — Prometheus on a dedicated internal listener (METRICS_ADDR, default :9464). Never expose it publicly.
• Tracing — OpenTelemetry over OTLP/gRPC, sampled, off by default.
• Logs — structured, with the correlation id on every line.
• R2 exporter — a separate binary scraping Cloudflare analytics for storage and operation counts, reconciled against CAVS’s own figures.

Dashboards, alert rules and runbooks ship in cavs-node/observability/. See Observability.

Deliberate decisions

Decision	Why
Content addressing everywhere	Dedup, integrity and idempotency all fall out of it for free
Presigned URLs, not proxied bytes	Transfer throughput doesn’t scale with API capacity
Token scopes as a hard cap	A leaked CI token can never exceed what its scope allows
Billing, audit and token management unreachable by tokens	Contains the blast radius of a leaked credential
Unknown JSON fields rejected	A typo’d field fails loudly instead of being silently dropped
default-src 'none' CSP	The API serves no HTML, so an echoed payload can never execute
Sessions publish only at finalize	No half-published state, ever
Index ingest is best-effort	A metadata failure must never fail a push
GC in a separate Rust binary	Byte-level work belongs where the format code lives, not in the Go control plane

Next

• Run it locally — the dev stack.
• Configuration reference — every environment variable.
• Deploying — Cloud Run and Terraform.
• Scaling — where the limits are.