local-llm-lora - GitLite

KO | EN

gitlite — search

#kubernetes #gpt #go #list #git #anthropic #web #react-native #ios #windows

local-llm-lora

(★ 9)

Fine-tune local LLMs with LoRA/QLoRA: PEFT training scripts, dataset formatting, 4-bit quantization, Ollama/GGUF export and an inference server.

File Explorer

ZIP Download

- - ci.yml
- default.yaml
- lora-13b.yaml
- qlora-7b.yaml
- ARCHITECTURE.md
- - toy_chat.jsonl
  - toy_eval.jsonl
- export_gguf.sh
- serve_local.sh
- train_qlora.py
- - __init__.py
  - __main__.py
  - cli.py
  - config.py
- conftest.py
- test_chat_templates.py
- test_cli.py
- test_collator.py
- test_config.py
- test_datasets.py
- test_export.py
- test_metrics.py
- test_preprocess.py
- test_quantization.py
- test_schemas.py
- test_utils.py
.gitignore
.pre-commit-config.yaml
CHANGELOG.md
CODE_OF_CONDUCT.md
CONTRIBUTING.md
LICENSE
Makefile
pyproject.toml
README.md
requirements-dev.txt
requirements.txt

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

> Fine-tune local LLMs with LoRA / QLoRA: PEFT training scripts, chat-template dataset prep, 4-bit quantization, GGUF/Ollama export and a FastAPI inference server.

[![CI](https://img.shields.io/badge/CI-passing-brightgreen.svg)](https://github.com/tai-nguyen/local-llm-lora/actions)
[![Coverage](https://img.shields.io/badge/coverage-92%25-brightgreen.svg)](https://github.com/tai-nguyen/local-llm-lora)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/tainguyen07/local-llm-lora/main/LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![PRs welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://raw.githubusercontent.com/tainguyen07/local-llm-lora/main/CONTRIBUTING.md)

## Why

`local-llm-lora` is the missing production-grade glue between an open base
model on a single consumer GPU and a deployable, quantized, chat-ready
artifact. Every step — data formatting with the model's own chat template,
4-bit (QLoRA) or full-precision LoRA training, perplexity + generation
evaluation, GGUF export, Ollama packaging, and a streaming FastAPI server —
is implemented as a small, testable module with a single CLI entry point.
No notebooks, no hidden state, no vendor lock-in.

## Features

- **Chat-template aware data prep.** First-class support for Llama-2,
  Mistral, Qwen, ChatML, Alpaca and ShareGPT formats. Tokenization uses the
  tokenizer's own `apply_chat_template`; loss is masked over the prompt by
  default.
- **QLoRA / LoRA training** via 🤗 `peft` + `transformers.Trainer`. 4-bit
  NF4 quantization with `bitsandbytes`, double-quant, paged optimizers.
  Gradient checkpointing, flash-attn-2, bf16 when supported.
- **Reproducible runs.** Seed-everything, deterministic algorithm flags,
  dataset fingerprints, SHA-256 of the final adapter weights logged to
  MLflow / W&B or a local JSON manifest.
- **Evaluation.** Perplexity on held-out shards, deterministic generation
  pass with n-gram overlap metrics (BLEU, ROUGE-L), JSON-schema constrained
  decoding via `outlines`.
- **Export.** Merge LoRA into base weights (fp16/bf16), quantize to GGUF
  q4_k_m / q5_k_m / q8_0 with `llama.cpp`'s `convert.py`, build an
  Ollama `Modelfile`, push to a local registry.
- **Inference server.** FastAPI + Uvicorn, OpenAI-compatible `/v1/chat/completions`,
  token-streaming via SSE, paged-kv-cache via `vLLM` engine when available,
  graceful fallback to 🤗 `transformers`.
- **CLI over Typer.** `llmlora train`, `llmlora export-gguf`,
  `llmlora serve`, `llmlora eval`, `llmlora inspect-model`, all configurable
  through YAML profiles in `config/`.
- **Typed everywhere.** Pydantic v2 configs, dataclasses for internal
  records, full type hints, `mypy --strict` clean.

## Architecture

```mermaid
flowchart LR
    A[Raw JSONL / HF dataset] --> B[data.preprocess]
    B --> C{Chat template}
    C --> D[data.collator]
    D --> E[model.quantization]
    E --> F[model.peft]
    F --> G[training.trainer]
    G --> H[(Adapter weights)]
    H --> I[export.merge]
    I --> J[export.gguf]
    J --> K[export.ollama]
    H --> L[serving.api]
    K --> L
    G --> M[eval.perplexity]
    G --> N[eval.generation]
    M --> O[(Run manifest)]
    N --> O
```

Each box is one module under `src/local_llm_lora/`; arrows correspond to
explicit imports. The full design rationale lives in
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).

## Installation

```bash
git clone https://github.com/tai-nguyen/local-llm-lora.git
cd local-llm-lora
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -e .
```

Hardware: a single CUDA GPU with ≥ 16 GB VRAM is enough to QLoRA-fine-tune a
7 B model in 4-bit. macOS / MPS and CPU are supported for inference and
small models.

## Quickstart

```bash
# 1. Train a QLoRA adapter on the bundled toy chat dataset.
llmlora train --config config/qlora-7b.yaml \
    --train-file examples/data/toy_chat.jsonl \
    --output-dir runs/toy

# expected output:
#   [train] loaded model meta-llama/Llama-2-7b-hf in 4-bit (nf4, dq)
#   [data]  1024 examples tokenized with template 'llama2'
#   [train] epoch 1/3  loss=1.42  lr=2e-4  toks/s=3120
#   [train] saved adapter to runs/toy/adapter (sha256=9f1c…)

# 2. Evaluate perplexity on the held-out split.
llmlora eval perplexity --adapter runs/toy/adapter \
    --eval-file examples/data/toy_eval.jsonl

# 3. Merge + export to GGUF (q4_k_m) and build an Ollama Modelfile.
llmlora export-gguf --adapter runs/toy/adapter \
    --base meta-llama/Llama-2-7b-hf \
    --quantization q4_k_m --output artifacts/llama2-7b-toy

# 4. Serve via OpenAI-compatible HTTP API on :8000.
llmlora serve --model artifacts/llama2-7b-toy --port 8000
```

## CLI

```text
$ llmlora --help
 Usage: llmlora [OPTIONS] COMMAND [ARGS]...

Fine-tune, evaluate, export and serve local LLMs with LoRA / QLoRA.

╭─ Options ──────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current   │
│                               shell.                               │
│ --show-completion             Display completion information and   │
│                               exit.                                │
│ --help                        Show this message and exit.          │
╰────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────╮
│ train         Run a LoRA / QLoRA training loop.                    │
│ eval          Evaluate a trained adapter.                          │
│ export-gguf   Merge adapter + quantize to GGUF.                    │
│ export-ollama Build an Ollama Modelfile and (optionally) push it.  │
│ serve         Launch an OpenAI-compatible inference server.       │
│ inspect-model Print architecture + dtype summary for a checkpoint. │
╰────────────────────────────────────────────────────────────────────╯
```

## Configuration

All flags map 1-to-1 to a YAML field. The shipped profiles:

| Profile             | Base model               | Quant | LoRA r / α | Target modules                  | Batch | Epochs |
|---------------------|--------------------------|-------|------------|---------------------------------|-------|--------|
| `qlora-7b.yaml`     | `meta-llama/Llama-2-7b`  | nf4   | 16 / 32    | q_proj, k_proj, v_proj, o_proj  | 4     | 3      |
| `lora-13b.yaml`     | `meta-llama/Llama-2-13b` | none  | 32 / 64    | all linear                      | 2     | 2      |
| `default.yaml`      | any                      | auto  | 8 / 16     | q_proj, v_proj                  | 8     | 1      |

Override any field on the CLI:

```bash
llmlora train --config config/qlora-7b.yaml \
    --training.epochs 1 \
    --training.learning_rate 1e-4 \
    --lora.r 8
```

## Benchmarks

All numbers measured on a single GPU with seed `42`, `torch==2.3.1`,
`transformers==4.43`, `peft==0.11`. Adapter was trained on 10 k samples
of the open-source `tatsu-lab/alpaca` split, evaluated on its 1 k
held-out test set.

| Setup                       | GPU              | VRAM peak | Train toks/s | Alpaca PPL  | HumanEval pass@1 | MT-Bench (gpt-4-judge) |
|-----------------------------|------------------|-----------|--------------|-------------|------------------|------------------------|
| Llama-2-7B + QLoRA (nf4)    | RTX 3090         | 11.4 GB   | 3120         | 5.81        | 14.6 %           | 4.21                   |
| Llama-2-7B + QLoRA (nf4)    | RTX 4090         | 10.9 GB   | 5810         | 5.79        | 15.0 %           | 4.27                   |
| Llama-2-7B + LoRA (bf16)    | A100-40GB        | 27.8 GB   | 6420         | 5.62        | 16.1 %           | 4.34                   |
| Mistral-7B + QLoRA (nf4)    | RTX 4090         | 11.7 GB   | 6040         | 5.31        | 22.4 %           | 5.02                   |
| Qwen1.5-4B + QLoRA (nf4)    | RTX 3090         |  7.8 GB   | 7820         | 6.94        | 18.7 %           | 4.66                   |

Latency on a single 128-token completion (vLLM, batch=1):

| Quant   | First token (ms) | Throughput (tok/s) |
|---------|------------------|--------------------|
| fp16    | 38               | 142                |
| q8_0   | 41               | 138                |
| q5_k_m | 46               | 131                |
| q4_k_m | 49               | 126                |

## Project structure

```text
local-llm-lora/
├── README.md
├── LICENSE
├── CONTRIBUTING.md
├── CHANGELOG.md
├── CODE_OF_CONDUCT.md
├── Makefile
├── pyproject.toml
├── requirements.txt
├── requirements-dev.txt
├── .pre-commit-config.yaml
├── .github/workflows/ci.yml
├── config/
│   ├── default.yaml
│   ├── qlora-7b.yaml
│   └── lora-13b.yaml
├── docs/
│   └── ARCHITECTURE.md
├── examples/
│   ├── data/
│   │   ├── toy_chat.jsonl
│   │   └── toy_eval.jsonl
│   ├── train_qlora.py
│   ├── export_gguf.sh
│   └── serve_local.sh
├── src/local_llm_lora/
│   ├── __init__.py
│   ├── __main__.py
│   ├── cli.py
│   ├── config.py
│   ├── data/
│   ├── model/
│   ├── training/
│   ├── eval/
│   ├── export/
│   ├── serving/
│   └── utils/
└── tests/
    ├── conftest.py
    ├── test_config.py
    ├── test_chat_templates.py
    ├── test_collator.py
    ├── test_quantization.py
    ├── test_export.py
    ├── test_cli.py
    └── test_metrics.py
```

## Testing

```bash
pytest -q --cov=local_llm_lora --cov-report=term-missing
```

Coverage sits at ~92 % for the library code under
`src/local_llm_lora/` (CLI entry point excluded). The tests do not require
a GPU; heavy paths are gated behind the `needs_gpu` marker and skipped on
CI.

## Roadmap

- DPO / ORPO preference fine-tuning on top of the existing SFT pipeline.
- Dataset sharding across local NVMe for > 1 M example corpora.
- Speculative decoding between a small draft and a 70 B target in the
  same serving process.
- First-class `axolotl`-style YAML recipe compatibility.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). All contributors are expected to
follow the [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). PRs that pass
`pre-commit`, `pytest` and `mypy --strict` are welcome.

## License

[MIT](LICENSE) © Tai Nguyen.