jzip-compressor

KO | EN

gitlite — search

#react #go #android #rust #deep-learning #docker #nodejs #macos #kubernetes #awesome-list #security #pytorch #mcp #open-source

jzip-compressor

(★ 82)

Compression for unit-norm embedding vectors using spherical coordinates

File Explorer

Download Latest Version (.zip)

.gitignore
jzip.c
Makefile
pipeline.png
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# jzip

Near-lossless compression for unit-norm embedding vectors using spherical coordinates, 1.5x compression ratio.

> [!NOTE]
> Near-lossless means the reconstruction error is below 1e-7 (the float32 machine epsilon). So indistinguishable at float32 precision, but we still can't call it bit-exact. [Read our paper for more details](https://jina.ai/embedding-compression.pdf).

![jzip pipeline](https://raw.githubusercontent.com/jina-ai/jzip-compressor/main/pipeline.png)

## Build

Requires zstd library.

```bash
# macOS
brew install zstd

# Ubuntu/Debian
apt install libzstd-dev

# Build
make
```

## Usage

```bash
# Compress: input.bin (N vectors of D dimensions) -> output.jz
jzip -c input.bin output.jz N D [LEVEL]

# Decompress: output.jz -> output.bin
jzip -d output.jz output.bin
```

LEVEL is the zstd compression level (1-22, default: 1). Higher levels are slower with negligible compression gain because the spherical transform already minimizes entropy.

## Example

```bash
# Compress 1000 vectors of 384 dimensions
jzip -c embeddings.bin compressed.jz 1000 384

# Compress with higher zstd level (slower, same ratio)
jzip -c embeddings.bin compressed.jz 1000 384 19

# Decompress
jzip -d compressed.jz restored.bin
```

## Python/NumPy Integration

Export embeddings to binary format:

```python
import numpy as np

# Ensure float32 and unit-normalized
embeddings = embeddings.astype(np.float32)
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)

# Save to binary
embeddings.tofile('embeddings.bin')
n, d = embeddings.shape  # use these for jzip -c
```

With sentence-transformers:

```python
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(texts).astype(np.float32)
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
embeddings.tofile('embeddings.bin')
# jzip -c embeddings.bin out.jz {len(texts)} {model.get_sentence_embedding_dimension()}
```

Load decompressed embeddings back:

```python
restored = np.fromfile('restored.bin', dtype=np.float32).reshape(n, d)
```

## File Format

Input: raw float32 binary (N x D floats, row-major)

Output: 16-byte header + zstd-compressed spherical angles

The input embeddings must be unit-normalized (L2 norm = 1).

## Algorithm

1. Convert Cartesian coordinates to spherical angles (N x D -> N x D-1)
2. Transpose angle matrix to group same-position angles
3. Byte-shuffle to group IEEE 754 exponent bytes
4. Compress with zstd

Decompression reverses these steps. Reconstruction error is ~7e-8, below float32 machine epsilon.

jzip-compressor

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

jzip-compressor

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README