Deduplication explained

Deduplication is the whole point, so it’s worth understanding well enough to predict. This page explains the mechanism and, more usefully, when it fails.

The mechanism

1. Chunk. The object is split with content-defined chunking (FastCDC). Boundaries are chosen by a rolling hash over the content, not at fixed offsets. This is the key property: insert a byte near the front of a file and only the chunk containing it changes — every following boundary lands in the same place. Fixed-size blocks would shift by one byte and every block downstream would look new.

2. Address. Each chunk is named by its BLAKE3-256 hash. Identical chunks have identical names, so identity is decided by content, never by position or filename.

3. Pretransform and compress. By default a per-chunk BG4 byte-grouping pretransform runs before compression — it interleaves bytes so numeric payloads (model weights, vertex buffers, audio samples) compress much better — then zstd (level 3 by default).

4. Store once. Chunks are written to a shared store, packed into immutable content-addressed packfiles. A chunk already present costs nothing. This is why ten similar checkpoints don’t cost ten times one checkpoint.

5. Verify. Per-chunk BLAKE3, a global Merkle root per object, and the LFS oid (SHA-256) checked on every download. Reconstruction is byte-identical or it fails — never halfway.

Chunk profiles

The transfer agent picks a profile per file by size (--profile auto, the default):

File size	Profile	Average chunk
< 128 MiB	fastcdc-16k	~16 KiB
< 512 MiB	fastcdc-64k	~64 KiB
≥ 512 MiB	fastcdc-128k	~128 KiB

Smaller chunks find finer-grained similarity but produce more metadata; larger chunks are cheaper to track but miss small edits. These thresholds come from the engine’s benchmark suite. Fixed-size profiles (fixed-256k, fixed-512k, fixed-1m) exist for payloads where content-defined boundaries buy nothing.

Override per repository through the agent’s arguments:

git config lfs.customtransfer.cavs.args "--profile fastcdc-32k --compression zstd-6"

See CAVS LFS agent for the full option table.

Where dedup happens

Scope	Deduplicated
Within one object	yes — repeated content inside a single file
Across versions of a file	yes — this is the big win
Across different files in a repository	yes — shared content is stored once
Across repositories	no — namespaces are isolated
Across organizations	no

Cross-tenant dedup would be a side channel on content, so it isn’t done. See Security model.

Standard git-lfs dedups differently

Without the CAVS transfer agent there is no chunking in the path. Identical whole files are still stored once (they share an oid), but a one-byte change produces a completely new object. If your dedup ratio is near zero, check whether the agent is actually wired: cav doctor.

What to expect

Measured on real payloads:

Change	Logical	Stored / transferred
First push of a 22 MiB file	22 MiB	22 MiB, compressed
Edit ~3 MiB inside it, push again	22 MiB	~3.3 MiB
Pull that version where the previous one is cached	22 MiB	~0.5 MiB
32 MiB compressible object, first push	32 MiB	~2.1 MiB (~93% saved)
Pull that object	32 MiB logical	~2.1 MiB on the wire

Your own figures land in the repository’s Benchmarks tab, per transfer. See Storage insights.

Formats that dedup well

Format	Why
safetensors, raw tensor dumps	contiguous layout; changing some weights leaves the rest byte-identical
Parquet, ORC	column chunks are independent; appending rewrites little
Uncompressed TAR	member bytes stay put; only the changed member’s chunks change
Sparse disk images, VM images	most blocks never change between snapshots
WAV, uncompressed audio, PNG frames	edits are local
Append-only logs and datasets	by construction
Checkpoint sequences from the same run	successive epochs differ only partially

Formats that dedup badly

Format	Why	What to do
Solid archives (.zip with a single stream, .tar.gz, .tar.zst)	one changed byte re-encodes the whole stream	store uncompressed TAR and let CAVS compress, or archive per-file
Encrypted blobs	ciphertext is designed to look random	encrypt after CAVS, or use per-chunk-friendly modes
Already-compressed media (.mp4, .jpg, .webp)	high entropy; a re-encode changes everything	dedup across versions still works if you don’t re-encode; don’t expect within-file gains
Formats with a distributed table of contents (some game package formats)	any insert cascades every offset	check cavs analyze in the engine; repack the payload before publishing
Databases rewritten wholesale on save	no content stability	export logically (dumps, Parquet) instead of shipping the file
Many small files	per-object overhead dominates	bundle into an uncompressed TAR first

The one change that usually pays for itself

If you ship .tar.gz or .tar.zst, switch to a plain .tar and let CAVS chunk and compress it. You keep the same compression ratio on the first push and gain near-total reuse on every push after. Gzip’s whole-stream coupling is the single most common reason a dedup ratio looks disappointing.

Diagnosing a bad ratio

1. Check the ratio. GET /repositories/{id}/usage → is physical close to logical?
2. Check the agent is in the path. cav doctor, then git config --get lfs.standalonetransferagent (should be cavs). No agent means no chunking.
3. Check a transfer record. GET /repositories/{id}/transfers?kind=push&limit=1 → what is reused_chunks / chunk_count? Near zero on a small edit means the format isn’t chunk-stable.
4. Identify the culprit. GET /repositories/{id}/objects?sort=size&limit=20, then look at the extensions. One solid archive can dominate a repository’s footprint.
5. Test the hypothesis. Push version A, then version B with a small change. Compare stored_bytes on the second push against the file size. If they’re similar, the format is the problem, not CAVS.
6. Change the packaging, or accept it. Some payloads genuinely resist dedup. CAVS still gives you versioning, access control, verification and a browsable index — it just won’t be cheap.

Reading the storage numbers

• Logical — the sum of your objects’ nominal sizes.
• Physical — the unique compressed bytes actually stored. This is what you pay for.
• Dedup ratio — 1 − physical / logical.

Caution

Per-object physical_size is that object’s own footprint. Don’t sum it across a repository — a chunk shared by two objects would be counted twice. Repository physical size is the unique packed bytes under the namespace, which is the figure the usage worker reports.

Going deeper

The chunking, packing and delivery engine is open source, with a benchmark suite, format specifications and local analysis tools (cavs sweep, cavs analyze, cavs bench) that let you measure a payload’s update behaviour before you publish it: github.com/orelvis15/cavs.

Next

• CAVS LFS agent — tune the profile and compression.
• Storage insights — measure what you actually get.
• Lifecycle & maintenance — reclaiming chunks nothing references.