Observability

Everything below ships in the platform package’s cavs-node/observability/ folder: Prometheus config and rules, 13 Grafana dashboards, Alertmanager routing, Loki and Tempo configuration, Grafana Alloy, a Grafana Cloud Terraform module, and 12 runbooks.

Metrics

Prometheus format on a dedicated internal listener:

METRICS_ENABLED=true
METRICS_ADDR=:9464
CAVS_REGION=us-central1     # a low-cardinality label on every metric

Danger

Never expose METRICS_ADDR publicly. It carries operational detail about your platform and tenants and has no authentication.

What’s instrumented

Area	Metrics
HTTP golden signals	cavs_http_requests_total, cavs_http_request_duration_seconds, cavs_http_in_flight_requests, cavs_http_request_size_bytes, cavs_http_response_size_bytes, cavs_http_timeouts_total, cavs_http_panics_total, cavs_http_rate_limited_total
Auth & security	cavs_auth_attempts_total, cavs_auth_failures_total, cavs_permission_denied_total, cavs_token_revocations_total, cavs_api_key_expired_total, cavs_api_key_rotations_total, cavs_cross_tenant_access_blocked_total
Uploads	session created/completed/failed, active sessions, parts, bytes, duration, throughput
Downloads	cavs_download_requests_total (by result), cavs_download_bytes_total, cavs_download_duration_seconds, cavs_download_failures_total, cavs_download_signed_urls_total, cavs_download_signed_url_expired_total, cavs_download_checksum_failures_total
Storage & dedup	cavs_storage_logical_bytes, cavs_storage_physical_bytes, cavs_storage_saved_bytes, cavs_deduplication_ratio, cavs_deduplicated_bytes_total, cavs_chunks_total, cavs_chunks_created_total, cavs_chunks_reused_total, cavs_chunk_lookup_duration_seconds
Economics	cavs_storage_cost_estimated_usd, cavs_storage_cost_avoided_usd, cavs_storage_cost_per_logical_tb, cavs_storage_cost_per_physical_tb, cavs_storage_ledger_discrepancy_bytes
Workers & queues	cavs_jobs_{enqueued,started,completed,failed,retried,dead_letter}_total, cavs_job_duration_seconds, cavs_queue_depth, cavs_queue_oldest_job_age_seconds
Webhooks	deliveries by result, retries, failures, delivery duration
Billing	cavs_billing_mrr_usd, cavs_billing_arr_usd, active subscriptions, trials, upgrades, downgrades, cancellations, payment failures and recoveries, webhook failures
Signed URLs	cavs_signed_url_failures_total by operation

Tip

cavs_storage_ledger_discrepancy_bytes is the one to watch closely. It compares the usage ledger against the object store’s actual footprint. A growing discrepancy means metering and reality have diverged — which is a billing-correctness problem, not just a monitoring one. There’s a runbook for it.

Route labels are normalized (path parameters replaced by patterns), so cardinality stays bounded no matter how many repositories you have.

Tracing

OpenTelemetry server spans over OTLP/gRPC:

TRACING_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4317
OTEL_EXPORTER_OTLP_INSECURE=true
OTEL_TRACES_SAMPLER_RATIO=0.1     # 10%

Off by default; when disabled the middleware is a no-op, so it’s always safe to leave installed. Sampling guidance is in observability/SAMPLING.md.

Logging

Structured logs with the correlation id on every line. Since the id is also returned in X-Request-Id and embedded in error bodies as request_id, a customer can quote a single value that finds the exact request in your logs.

The access log records method, normalized route, status and byte counts. Loki configuration is included.

Cloudflare R2 exporter

A separate binary that scrapes provider-side analytics — storage bytes, class A/B operations — and reconciles them against CAVS’s own figures:

CF_ACCOUNT_ID=<account>
CF_API_TOKEN=<token>          # Account Analytics:Read + Workers R2 Storage:Read
R2_BUCKETS=cavs-prod
R2_SCRAPE_INTERVAL=5m
R2_RECONCILE=true

This is how you catch the case where your bill and your metering disagree.

Dashboards

Thirteen Grafana dashboards in observability/grafana/dashboards/:

Dashboard	Answers
cavs-executive-overview	Is the business healthy?
cavs-platform-health	Is the platform up?
cavs-api-performance	Latency, throughput, error rates by route
cavs-workers-queues	Are events being consumed? Is the queue backing up?
cavs-artifact-operations	Upload and download volume and success
cavs-deduplication	Is dedup performing as expected?
cavs-storage-economics	Cost per TB, cost avoided
cavs-customer-usage	Per-tenant usage
cavs-billing	MRR, ARR, churn, payment failures
cavs-security	Auth failures, permission denials, cross-tenant blocks
cavs-mongodb	Database health
cavs-cloudflare-r2	Provider-side storage and operations
cavs-infrastructure	CPU, memory, instances

Grafana Cloud: observability/grafanacloud/ has a Terraform module, import-dashboards.sh and verify-metrics.sh — run the last one after wiring a new deployment to confirm cavs_* series are actually arriving.

SLOs

SLO	Target	Window
API availability	99.9%	30d
API read latency p95	< 500 ms	30d
Metadata write latency p95	< 800 ms	30d
Signed URL generation p95	< 500 ms	30d
Artifact finalize p95	< 2 s	30d
Upload success	99.5% of sessions	30d
Download success	99.9%	30d
Jobs on time	99%	30d

Error budget at 99.9% availability: ~43 minutes/month.

Availability SLI (30d)

1 - (
  sum(increase(cavs_http_requests_total{status_code=~"5.."}[30d]))
  / clamp_min(sum(increase(cavs_http_requests_total[30d])), 1)
)

Error budget consumed

(
  sum(increase(cavs_http_requests_total{status_code=~"5.."}[30d]))
  / clamp_min(sum(increase(cavs_http_requests_total[30d])), 1)
) / 0.001

Recording rules pre-compute cavs:availability:ratio5m and cavs:http_latency:p95, so dashboards and alerts don’t recompute expensive queries. Full definitions in observability/SLOS.md.

Alerts

Rules in observability/prometheus/rules/alerts.rules.yml, routing in observability/alertmanager/. Each alert links to a runbook.

Runbooks

Twelve, in observability/runbooks/:

Runbook	For
api-down	The API isn’t serving
mongo-unavailable	Database unreachable
r2-degraded	Object storage degraded
queue-backlog	Events piling up
upload-failures	Uploads failing at rate
checksum-failure	Integrity verification failing
ledger-discrepancy	Metering diverging from reality
billing-webhook-failure	Provider events not landing
backup-failure	Backups failing
disk-full	Out of disk
high-cpu / high-memory	Resource saturation

Tip

Runbooks are only useful if they’re current. When you resolve an incident the runbook didn’t cover, add what you learned before you close the ticket.

Local observability stack

cd observability
docker compose -f docker-compose.observability.yml up

Brings up Prometheus, Grafana with the dashboards provisioned, Loki, Tempo, Alloy and Alertmanager against your local CAVS stack. The fastest way to see whether an instrumentation change works.

Retention

Guidance in observability/RETENTION.md. Broadly: high-resolution metrics for weeks, downsampled for months, traces sampled and short-lived, logs sized to your compliance requirement.

Caution

The audit log is a separate concern from your observability retention. The API returns only the last 100 events and is not an archive — export it if you have a retention obligation. See Audit log.

Wiring a new deployment

1. Set METRICS_ENABLED=true, METRICS_ADDR, CAVS_REGION; keep the port private.
2. Point Prometheus (or Alloy) at METRICS_ADDR on every instance.
3. Load the recording and alert rules.
4. Import the dashboards.
5. Run verify-metrics.sh and confirm cavs_* series arrive.
6. Wire Alertmanager to your pager.
7. Enable tracing at 10% and confirm spans land.
8. Deploy the R2 exporter with R2_RECONCILE=true.
9. Set CHECKLIST.md as your go-live gate.

Next

• Configuration reference — the variables.
• Scaling — what the metrics will tell you to do.
• Deploying — the deployment path.