CAVS LFS agent

cavs-lfs-agent is a standalone custom transfer agent for Git LFS. It replaces LFS’s whole-file transfer and storage with CAVS chunk-level dedup, which is where the real savings come from.

	Vanilla Git LFS	CAVS agent
Push a 22 MiB file with ~3 MiB changed	22 MiB stored + sent	~3.3 MiB
Pull that version where the previous one is cached	22 MiB	~0.5 MiB

How it works

git push                                  git pull / clone
   │                                         │
   ▼ NDJSON over stdin/stdout                ▼
upload:                                   download:
  pack the blob into one raw track          fetch the asset manifest
  ingest into the shared store  (dedup!)    diff against the local chunk cache
  export the content-addressed tree         concurrent range GETs for the gaps
  mirror it to the repository's /lfs        BLAKE3 + sha256(oid) verified
   │                                         │
   ▼                                         ▼
<repo>/cas/assets/<oid>/… + chunks/…      git-lfs moves the file into .git/lfs

Details that matter:

• The asset name and track name are the LFS oid (SHA-256), and the packed container carries sha256:<oid> as metadata — so the engine’s end-to-end verification checks the LFS oid for free on every download.
• Uploads land in one shared store per remote, so chunks dedup across all objects, all versions and all pushers.
• Publication is session-batched. A push’s objects become fetchable when the agent finalizes at the end of the transfer, not per object. A push killed before finalize publishes nothing; retrying re-ingests and repairs. There is no half-published state.
• Downloads reuse a local chunk cache keyed by BLAKE3. Only missing chunks travel, fetched as parallel HTTP ranges.

Installation

The CLI installer brings both binaries:

curl -fsSL https://raw.githubusercontent.com/orelvis15/cavs-hub-cli/main/install.sh | sh
which cavs-lfs-agent

Or download it from the engine releases and put it on your PATH.

Wiring it up

Easiest:

cav repo connect <org>/<repo>     # connect + wire the agent + install the hook
# or, in an already-connected repository:
cav install-lfs

By hand:

git lfs install --local
git config lfs.url                             "https://cavsnode.com/api/v1/repositories/$REPO/lfs"
git config lfs.standalonetransferagent         cavs
git config lfs.customtransfer.cavs.path        cavs-lfs-agent
git config lfs.customtransfer.cavs.concurrent  false

Caution

Keep concurrent set to false. Transfers within one agent invocation are sequential by design — downloads still parallelize internally per object, and a lock serializes concurrent pushes to a shared remote. Setting it to true invites corruption.

Authentication

The agent finds a token in $CAVS_TOKEN, then the token field of ~/.config/cav/config.toml (i.e. whatever cav login stored). It sends it as Authorization: Bearer … on the …/lfs/* requests.

export CAVS_TOKEN=cavs_ci_...    # CI
cav login                        # interactive

If no token is found, the agent fails before any transfer starts, with a message telling you which to set.

Plaintext HTTP is refused

Over https:// the token is always sent. Over plain http:// the agent refuses to upload unless CAVS_LFS_ALLOW_INSECURE_HTTP is set to a truthy value (1, true, yes). That escape hatch is for local development against a dev stack — never set it against a real endpoint.

Options

Pass them through lfs.customtransfer.cavs.args:

git config lfs.customtransfer.cavs.args "--profile fastcdc-32k --compression zstd-6 --connections 16"

Flag	Default	What it does
--remote <path|url>	the remote git-lfs announces	Where objects live. Also $CAVS_LFS_REMOTE.
--cache-dir <dir>	<git-dir>/lfs/cavs/cache	Chunk cache. Also $CAVS_LFS_CACHE; falls back to ~/.cache/cavs-lfs-agent.
--profile <p>	auto	Chunking: fastcdc-16k/32k/64k/128k/256k[-n3], fixed-256k/512k/1m. auto picks by file size: < 128 MiB → 16k, < 512 MiB → 64k, else 128k.
--compression <c>	zstd-3	none, or zstd-<1..22>.
--no-bg4	off	Disable the per-chunk BG4 byte-grouping pretransform. It’s on by default and helps numeric payloads — model weights, vertex buffers, audio samples.
--connections <n>	8	Parallel download connections.
--pubkey <hex>	—	Require Ed25519-signed manifests on download.
--sign-key <file>	—	Sign uploads (64-hex secret key).

Environment:

Variable	Default	Effect
CAVS_TOKEN	—	Access token
CAVS_LFS_REMOTE	—	Remote override
CAVS_LFS_CACHE	—	Cache directory override
CAVS_LFS_ALLOW_INSECURE_HTTP	unset	Permit plaintext-HTTP pushes
CAVS_FETCH_MAX_INFLIGHT_BYTES	128 MiB	Process-wide cap on in-flight range bytes, so many parallel downloads can’t stack unbounded buffers

Tuning

Change one thing, push twice, and read the benchmark records. Guessing is worse than measuring.

Symptom	Try
Small edits transfer more than they should	a smaller profile: --profile fastcdc-16k
Push is CPU-bound (ingest_ms dominates duration_ms)	a larger profile, or --compression zstd-1
Storage is higher than you’d like and pushes are rare	--compression zstd-9 or higher
Payload is float/int-heavy (weights, meshes, audio)	keep BG4 on — it’s the default, don’t pass --no-bg4
Payload is already compressed (mp4, jpg)	--compression none saves CPU for no size cost
Downloads are slow on a fat pipe	--connections 16
Downloads use too much memory	lower CAVS_FETCH_MAX_INFLIGHT_BYTES
Many small files	bundle into an uncompressed TAR first — per-object overhead dominates

Changing the profile doesn't rewrite history

Chunk boundaries are part of a profile’s identity. Changing --profile means new pushes chunk differently and won’t dedup against objects stored under the old profile. Pick one per repository and leave it alone; if you must change, expect one expensive push while the new chunking establishes itself.

Verifying it’s active

cav doctor
git config --get lfs.standalonetransferagent      # → cavs
git config --get lfs.customtransfer.cavs.path     # → cavs-lfs-agent

Then push and look at the evidence:

# A benchmark record only exists if the agent handled the transfer
curl -sS "https://cavsnode.com/api/v1/repositories/$REPO/transfers?kind=push&limit=1" \
  -H "Authorization: Bearer $CAVS_TOKEN" | jq

Empty benchmarks plus a dedup ratio near zero means the agent isn’t in the path.

Signing

For content you distribute, the agent can sign manifests and clients can require a signature:

# Push, signed
git config lfs.customtransfer.cavs.args "--sign-key /secure/cavs-signing.key"

# Pull, enforcing the signature
git config lfs.customtransfer.cavs.args "--pubkey 3f9a…"

A download whose manifest isn’t signed by that key fails. Keep the secret key out of the repository and out of CI logs.

Limitations

• Sequential within one invocation. Keep concurrent false. Downloads parallelize internally per object.
• Session-batched publication. Objects appear at finalize, not per object. A push killed early publishes nothing — retry.
• Storage duplication on directory remotes. A filesystem remote keeps the packed data twice (store + static export) in exchange for a CDN-syncable tree. Against a CAVS Node HTTP remote — the normal case — only the export is mirrored.
• Advisory locks on NFS. A file lock serializes concurrent pushes to a shared filesystem remote; advisory locks can be unreliable on NFS. Not a concern for the HTTP remote path.
• Profile changes break dedup continuity. See the caution above.

Static export and CDN

The assets/ + chunks/ tree is a static CAVS export: content-addressed immutable files served by plain GET plus Range. That means you can rsync or aws s3 sync it to a CDN and point read-only clients at the URL — clones then pull over HTTP with no server at all, while pushes keep going to CAVS.

Useful for public datasets and game clients. See Engine SDKs.

Next

• Deduplication explained — what the profiles do.
• Storage insights — reading the benchmark records.
• Engine documentation — the agent’s own reference.