Troubleshooting

Find your symptom. Each entry gives the cause and the fix.

Start here

Three commands answer most questions:

cav doctor                                    # environment + connectivity + repo wiring
curl -sS https://cavsnode.com/readyz          # is the platform up?
curl -sS https://cavsnode.com/api/v1/users/me -H "Authorization: Bearer $CAVS_TOKEN"

The last one tells you exactly which organizations and roles your credential sees — which is the answer to most permission puzzles.

Authentication

401 unauthorized — “missing bearer token”

No Authorization header. Check the variable is actually set and exported:

echo "${CAVS_TOKEN:0:12}…"     # should print a cavs_ prefix

401 unauthorized — “invalid or expired token”

The token isn’t in the organization’s token list, has expired, or was revoked. Check the list, and recreate if unsure. A truncated or whitespace-padded copy-paste gives the same error.

401 from the dashboard, but the API works with a token

Your Firebase session expired, or you’re on a non-canonical origin. Firebase persists auth per origin — a link on a different host reads as logged out. Sign in at https://cavsnode.com.

403 forbidden — “account is suspended”

The user status isn’t ACTIVE. Contact an organization owner.

Permissions

403 forbidden on something that used to work

Almost always one of:

1. Your role changed. GET /users/me.
2. The token scope is narrower than the operation. Scopes are a hard cap — a repo:read token can never push.
3. A repository override was set to NONE (or a narrower level). An override wins over the role.
4. You’re calling something no token scope can reach — billing, audit, token management, storage connections, organization update or delete. Those need a session-authenticated role.

Checklist: Roles & permissions.

400 bad_request — “invalid repository id”

You passed a slug where a UUID is required. Repository paths take UUIDs only:

curl -sS "$API/organizations/acme-ai/repositories" -H "$H" \
  | jq -r '.repositories[] | "\(.slug)\t\(.id)"'

400 bad_request on a body that looks correct

The API rejects unknown JSON fields. Check every field name against the reference — one typo, or one field your client added before the server knew about it, gives a 400.

Quotas

402 quota_exceeded

Read details.reason:

reason	Fix
max_repositories	Free allows one. Upgrade or delete one.
max_members	Upgrade, or use repository invitations instead — they don’t consume member slots.
max_tokens	Revoke unused tokens. Check the last-used column.
max_object_size	The file exceeds your plan’s per-object limit.
storage_quota	Free up space or upgrade. See reducing your bill.
max_storage_connections	Upgrade, or reuse an existing connection.

Storage didn’t drop after deleting a branch or repository

Expected. Garbage collection doesn’t reclaim bytes yet — the stage coordinates but doesn’t delete. Plan capacity on what you’ve written. See Lifecycle & maintenance.

Transfers

git push succeeded but the dashboard shows nothing

Danger

This is the failure mode most worth knowing. The transfer agent reports objects complete during ingest, then mirrors and finalizes at terminate. If finalize fails, git push still succeeds — Git got its commits, your Git host is happy, but CAVS never registered the objects.

Never trust the push exit code. Verify:

curl -sS "$API/repositories/$REPO" -H "$H" | jq .repository.generation
curl -sS "$API/repositories/$REPO/objects?limit=5" -H "$H"

If the generation didn’t bump, the push didn’t land. The most common cause is the next entry.

Mirror fails with 429 on a repository with many packs

The agent HEAD-guards every tree file before uploading, and does not back off on 429. Once a repository has enough packs, that burst exceeds the API’s 30 req/s limit and the mirror fails — silently, per the entry above.

• Self-hosted: raise RATE_LIMIT_RPS and RATE_LIMIT_BURST.
• Managed: retry the push; a second attempt often lands because much of the tree is already present.
• Either way, verify the generation bumped.

The agent’s output is invisible

git-lfs suppresses the standalone agent’s stderr during push. Wrap it:

cat > /tmp/cavs-agent-wrapper.sh <<'EOF'
#!/usr/bin/env bash
exec cavs-lfs-agent "$@" 2>>/tmp/agent.err
EOF
chmod +x /tmp/cavs-agent-wrapper.sh
git config lfs.customtransfer.cavs.path /tmp/cavs-agent-wrapper.sh
tail -f /tmp/agent.err

git lfs push prints “0 B, done” and doesn’t run the agent

git-lfs caches push state. Push a brand-new object to exercise the agent, or drive its NDJSON protocol directly. git lfs push --object-id gives spurious “Unable to stat local media path” errors — don’t rely on it.

smudge filter lfs failed on clone

LFS tried to fetch before CAVS was configured. Re-clone with the smudge filter off:

GIT_LFS_SKIP_SMUDGE=1 git clone <url>
cd <repo>
cav repo connect <org>/<repo>
git lfs pull

Uploads fail with 403 from object storage

The presigned URL expired — they last 15 minutes. Re-request from the authorize step; don’t retry the expired URL.

An upload session is stuck

Sessions expire after 6 hours and their unfinalized objects are swept. Either finalize it or start a new one with the same Idempotency-Key.

The agent refuses to upload over http://

By design — your token would travel in cleartext. Use https://, or for local development only:

export CAVS_LFS_ALLOW_INSECURE_HTTP=1

git-lfs upload reports a verify failure

The verify action the batch response advertises isn’t routed. The bytes probably landed — confirm with cav verify <file> or GET …/objects. Prefer the CAVS agent, which doesn’t use that path.

Deduplication

Physical size ≈ logical size

In order of likelihood:

1. The agent isn’t wired. cav doctor, then git config --get lfs.standalonetransferagent — it must print cavs. Standard git-lfs does no chunking at all.
2. Your format resists chunking. Solid archives (.tar.gz, .tar.zst, solid .zip) re-encode the whole stream on any change. Ship a plain .tar and let CAVS compress it — usually the single biggest win.
3. The content genuinely changed. A re-encoded video or a retrained-from-scratch model shares nothing with its predecessor.

Diagnose properly: Deduplication explained.

The Benchmarks tab is empty

Benchmark records only exist for transfers the CAVS agent handled. Standard git-lfs and direct API uploads produce none.

Dedup got worse after a config change

Did you change --profile? Chunk boundaries are part of a profile’s identity, so new pushes won’t dedup against objects stored under the old one. Pick one profile per repository and leave it alone.

Dashboard

Files, Commits and Branches are empty

The repository hasn’t been indexed. Responses carry "indexed": false.

cav repo index --full

Files look stale after a history rewrite

Re-index from scratch: cav repo index --full.

Usage looks stale right after a push

Reconciliation is asynchronous — seconds, not minutes. Object count and generation update immediately; byte totals follow.

An invitation link bounces to /login

The link’s origin doesn’t match where you sign in. Firebase persists auth per origin. Self-hosted: set APP_BASE_URL to the canonical origin and confirm your deploy tooling doesn’t overwrite it. Managed: have an admin regenerate the link.

An invitation email never arrived

Check spam. Then have an admin Copy link from Settings → Invitations — that rotates the token, so the new link works and the old one stops — and send it over a channel you trust.

SDKs

NotFoundError on client.artifacts.download(reference)

Logical-reference download needs GET /resolve, which isn’t deployed. Use a bare oid with project=, or an immutable @sha256: reference. The TypeScript SDK’s download() always calls /resolve, so it doesn’t work at all yet — see TypeScript SDK.

NotFoundError on client.runs.* or client.service_accounts.*

Not deployed. Use /graph/edges for lineage and cavs_ci_ tokens instead of service accounts.

deduplication_ratio is None / undefined

The finalize response doesn’t include dedup statistics yet. Read GET /repositories/{id}/usage instead.

404 on upload from the TypeScript SDK

project must be the repository UUID, not a slug — the TS SDK doesn’t resolve slugs. The Python SDK does.

Webhooks

Deliveries stopped

The circuit breaker fired after 10 consecutive failures.

curl -sS "$API/organizations/$ORG/webhooks" -H "$H" \
  | jq '.webhooks[] | {url, active, consecutive_failures, last_error}'

Fix the endpoint, then re-enable with PATCH {"active": true} — that also clears the breaker.

Signature verification always fails

Three usual causes:

1. You’re hashing a re-serialized body. Hash the raw bytes.
2. Wrong secret. It’s generated by CAVS (whsec_…) and shown only at creation.
3. You’re comparing against the header including the sha256= prefix inconsistently.

Working examples: Webhooks.

Duplicate deliveries

Expected — delivery is at-least-once. Deduplicate on X-CAVS-Delivery.

Timeouts

You have 10 seconds per attempt. Acknowledge with 204 and process asynchronously.

CLI

cav: command not found

The installer used ~/.local/bin. Add it to PATH:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.profile

cav talks to localhost:8080

That’s the shipped default. cav config set api https://cavsnode.com, or export CAVS_API.

Exit code 13 — ORG_AMBIGUOUS

You belong to several organizations. Pass --org <slug>.

Exit code 12 — REPO_NOT_CONNECTED

cav upload, download and verify read cavs.repo-id from the Git config. Run cav repo connect <org>/<repo>, or use the HTTP API in a bare CI checkout.

cavs-lfs-agent not found on PATH

Install it from the engine releases, or pass --agent-path.

Self-hosted

The API won’t boot

Message	Cause
S3-related	The API refuses to start in production without S3_ACCOUNT_ID and keys
“invalid issuer” / Firebase	FIREBASE_PROJECT_ID empty or mismatched
MongoDB	Check MONGODB_URI and network reachability; /readyz pings it

Everything 500s with S3 “connection refused”

MinIO died. Ctrl-C on ./run.sh all sends SIGINT to the whole process group and takes MinIO with it. Check :9000 before debugging anything else.

The dashboard says “Firebase not configured”

Vite started without the root .env:

set -a && . ./.env && set +a && cd web && npm run dev

A new deploy didn’t take effect

The worker revision may not be Ready — Cloud Run keeps serving the previous one. Check all three services report Ready = True. And confirm your working tree was the ref you intended: deploy.sh builds the local tree, not a git ref. See Deploying.

External /healthz returns 404

Google’s front end reserves that path. Use /readyz for external checks; container probes hit /healthz directly and still work.

Getting help

Include:

• The request_id from the error body — this is the single most useful thing.
• The exact code and status.
• Your client and version (cav --version, SDK version).
• cav doctor output, if relevant.
• What you expected versus what happened.

Danger

Never include tokens or presigned URLs in a support request, an issue or a paste. If you already have, revoke the token first, then continue.

Next

• FAQ — questions rather than symptoms.
• Errors — the code reference.
• Glossary — terminology.