LongMamba
Some preliminary explorations of Mamba's context scaling.
File Explorer
Download Latest Version (.zip)- heatmap_16384.png
- heatmap_32768.png
- longlora_passkey.png
- LongMamba_16384_bs128_step400.csv
- LongMamba_16384_bs128_step400.csv.pdf
- LongMamba_16384_bs128_step400.csv.png
- LongMamba_16384_bs128_step400_large.csv.png
- mamba-length-extrapolation.png
- mamba_byte.png
- mamba_half_delta_training.csv
- mamba_half_delta_training.csv.pdf
- mamba_half_delta_training.csv.png
- Mamba_issue.png
- original_mamba.csv
- original_mamba.csv.pdf
- original_mamba.csv.png
- original_mamba_delta_ratio_0.5.csv
- original_mamba_delta_ratio_0.5.csv.pdf
- original_mamba_delta_ratio_0.5.csv.png
- Transformer-XL.png
- mamba_lm.py
- mamba_module.py
- mamba_lm.py
- mamba_module.py
- mamba_torch.py
- .gitignore
- eval.py
- pass_key.py
- plot.py
- README.md
- requirements.txt
- script.sh
- train-infinite.py
- train.py
# Installation Guide
git clone https://github.com/jzhang38/LongMamba
Downloads the entire project code from GitHub to your computer.
cd LongMamba
Moves into the project folder you just downloaded.
2. Official Install Script
Easy Recommended- Python 3 Python is required to use pip.
pip3 install torch --index-url https://download.pytorch.org/whl/cu118
Installs the package published on PyPI directly β no need to clone the source.
pip install causal-conv1d>=1.1.0
Installs the package published on PyPI directly β no need to clone the source.
pip install mamba-ssm
Installs the package published on PyPI directly β no need to clone the source.
Pulled directly from this repo's README.
3. Python
Easypip3 install torch --index-url https://download.pytorch.org/whl/cu118
Installs the package published on PyPI directly β no need to clone the source.
pip install causal-conv1d>=1.1.0
Installs the package published on PyPI directly β no need to clone the source.
pip install mamba-ssm
Installs the package published on PyPI directly β no need to clone the source.
pip install -r requirements.txt
Installs the Python libraries listed in requirements.txt (or similar).
Pulled directly from this repo's README.