shrynk - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #go #android #rust #linux #java #windows #awesome #cli #awesome-list #ios #mcp #openai

shrynk

(★ 114)

Using Machine Learning to learn how to Compress :zap:

File Explorer

Download Latest Version (.zip)

- .gcloudignore
- app.yaml
- footer.html
- main.py
- material_datatable.css
- material_datatable.js
- requirements.txt
- table.html
- - __init__.py
  - bytes_default.jsonl.gzip
  - json_default.jsonl.gzip
  - pandas_default.jsonl.gzip
- __init__.py
- Dockerfile
- analysis.py
- benchmark.py
- experiment.py
- first.py
- memtest.py
- rcsvs.py
- - __init__.py
  - bytes.py
  - json_.py
  - pandas_.py
- __init__.py
- __main__.py
- compressor.py
- predictor.py
- utils.py
- all_test.py
.gitignore
.travis.yml
conftest.py
deploy.py
README.md
setup.cfg
setup.py
Shrynk.png
shrynk_all.py
tox.ini

# Installation Guide

1. Get the code

git clone https://github.com/kootenpv/shrynk

Downloads the entire project code from GitHub to your computer.

cd shrynk

Moves into the project folder you just downloaded.

2. Official Install Script

Easy Recommended

Prerequisites

Python 3 Python is required to use pip.

pip install shrynk

Installs the package published on PyPI directly — no need to clone the source.

✅ After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.

Pulled directly from this repo's README.

3. Docker

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Docker Desktop Needed to build and run containers. Install it and keep it running in the background.

docker run --rm -v ~/.shrynk:/root/.shrynk -v ~/Downloads:/data kootenpv/shrynk shrynk --help

Runs the built image as an actual container.

docker run --rm -v ~/.shrynk:/root/.shrynk -v ~/Downloads:/data kootenpv/shrynk \

Runs the built image as an actual container.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

Pulled directly from this repo's README.

4. Python

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install shrynk

Installs the package published on PyPI directly — no need to clone the source.

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<p align="center">
  <img src="./Shrynk.png" width="400rem"/>
</p>

[![Build Status](https://travis-ci.org/kootenpv/shrynk.svg?branch=master)](https://travis-ci.org/kootenpv/shrynk)
[![PyPI](https://img.shields.io/pypi/pyversions/shrynk.svg?style=flat-square&logo=python)](https://pypi.python.org/pypi/shrynk/)
[![PyPI](https://img.shields.io/pypi/v/shrynk.svg?style=flat-square&logo=pypi)](https://pypi.python.org/pypi/shrynk/)
[![HitCount](http://hits.dwyl.io/kootenpv/shrynk.svg)](http://hits.dwyl.io/kootenpv/shrynk)

You can read the [introductory blog post](https://vks.ai/2019-12-05-shrynk-using-machine-learning-to-learn-how-to-compress) or try it live at https://shrynk.ai

### Features

- ✓ Compress your data smartly based on **Machine Learning**
- ✓ Takes **User Requirements** in the form of weights for `size`, `write_time` and `read_time`
- ✓ Trains & caches a model based on **compression methods available** in the system, using packaged data
- ✓ **CLI** for compressing and decompressing
- ✓ Works with `CSV`, `JSON` and `Bytes` in general

### CLI

shrynk compress myfile.json       # will yield e.g. myfile.json.gz or myfile.json.bz2
    shrynk decompress myfile.json.gz  # will yield myfile.json

shrynk compress myfile.csv --size 0 --write 1 --read 0

shrynk benchmark myfile.csv                  # shows benchmark results
    shrynk benchmark --predict myfile.csv        # will also show the current prediction
    shrynk benchmark --save --predict myfile.csv # will add the result to the training data too

### Usage in Docker

To test shrynk out quickly yourself, you can use the official docker image from DockerHub. It is great not to interfere with an existing python installation.

You can also build the image from scratch by going to [the docker folder here](./docker/) and doing `docker build -t shrynk .` and use `shrynk` instead of `kootenpv/shrynk` above.

In the following commands, replace `~/Downloads` with the folder you want to share with the container (where the file you want to compress is).

```bash
# To see help
docker run --rm -v ~/.shrynk:/root/.shrynk -v ~/Downloads:/data kootenpv/shrynk shrynk --help

# To compress a file called train.csv in your ~/Downloads folder
docker run --rm -v ~/.shrynk:/root/.shrynk -v ~/Downloads:/data kootenpv/shrynk \
   shrynk compress /data/train.csv

# To benchmark and predict the train.csv file in your ~/Downloads folder
docker run --rm -v ~/.shrynk:/root/.shrynk -v ~/Downloads:/data kootenpv/shrynk \
   shrynk benchmark --predict /data/train.csv
```

### Usage in Python

Installation:

pip install shrynk

Then in Python:

```python
import pandas as pd
from shrynk import save, load

# save dataframe compressed
my_df = pd.DataFrame({"a": [1]})
file_path = save(my_df, "mypath.csv")
# e.g. mypath.csv.bz2

# load compressed file
loaded_df = load(file_path)
```

If you just want the prediction, you can also:

```python
import pandas as pd
from shrynk import infer

infer(pd.DataFrame({"a": [1]}))
# {"engine": "csv", "compression": "bz2"}
```

### Add your own data

If you want more control you can do the following:

```python
import pandas as pd
from shrynk import PandasCompressor

df = pd.DataFrame({"a": [1, 2, 3]})

pdc = PandasCompressor("default")
pdc.run_benchmarks(df) # adds data to the default

pdc.train_model(size=3, write=1, read=1)

pdc.predict(df)
```