lit-llama

(★ 6,084)

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

Owner Repos

pytorch-lightning ★ 31,297

Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.

Python
litgpt ★ 13,619

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Python
LitServe ★ 3,929

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

Python
torchmetrics ★ 2,460

Machine learning metrics for distributed, scalable PyTorch applications.

Python
lightning-thunder ★ 1,469

PyTorch compiler that accelerates training and inference. Get built-in optimizations for performance, memory, parallelism, and easily write your own.

Python
deep-learning-project-template ★ 1,288

Pytorch Lightning code guideline for conferences

Python

File Explorer

ZIP Download
  • .gitignore
  • generate.py
  • LICENSE
  • pyproject.toml
  • README.md
  • setup.py

# Installation Guide

python
Prerequisites

Setup Steps

git clone https://github.com/Lightning-AI/lit-llama && cd lit-llama

Clone the repository and navigate into the project folder.

pip install -e ".[all]"

Install the package with all optional dependencies.

python scripts/download.py

Download the pretrained model weights.

python generate.py --prompt "Hello, my name is"

Run the model to test text generation.

Key Commands

pip install -e ".[all]"

Install the package and all dependencies in editable mode.

python generate.py --prompt "Hello, my name is"

Perform text generation using downloaded model weights.

python quantize/gptq.py --output_path checkpoints/lit-llama/7B/llama-gptq.4bit.pth --dtype bfloat16 --quantize gptq.int4

Convert model weights using GPTQ-style int4 quantization.

python scripts/prepare_alpaca.py

Download and prepare the Alpaca dataset for finetuning.

python finetune/lora.py

Finetune the model using the LoRA technique.

Running large language models requires at least 14GB~26GB of GPU memory (A100 or RTX 3090+ recommended); use quantization options if memory is limited.

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Link
Example

Command Glossary

Commands referenced in this DOCs, explained below.

🔍

git clone

View Details ▼

Clone an existing repository.

git clone {{remote_repository_location}} {{path/to/directory}}

Clone an existing repository into a new directory (the default directory is the repository name):

git clone --recursive {{remote_repository_location}}

Clone an existing repository and its submodules:

git clone {{[-n|--no-checkout]}} {{remote_repository_location}}

Clone only the `.git` directory of an existing repository:

🔍

pip install

View Details ▼

Install Python packages.

pip install {{package1 package2 ...}}

Install one or more packages:

pip install {{package1 package2 ...}} {{[-U|--upgrade]}}

Upgrade all specified packages to the latest version, installing any that are not already present:

pip install {{package}}=={{version}}

Install a specific version of a package:

🔍

python

View Details ▼

Python language interpreter.

python

Start a REPL (interactive shell):

python {{path/to/file.py}}

Execute a specific Python file:

python -i {{path/to/file.py}}

Execute a specific Python file and start a REPL:

# Project Badges

// repository documentation