unsloth-puzzles

KO | EN

gitlite — search

#vue #android #sql #terminal #open-source #redis #java #kotlin #developer-tools #anthropic

unsloth-puzzles

(★ 9)

No description available.

File Explorer

ZIP Download

- - average_step_duration.png
  - wall_clock.png
- fsdp2.py
- fsdp2_compile.py
- fsdp2_qlora.ipynb
- README.md
- fused_tq3.ipynb
- fused_tq3.py
- README.md
- memory_efficient_backprop.ipynb
- memory_efficient_backprop.py
- README.md
- - benchmark_bar_plot.png
- nf4_dequantize.ipynb
- README.md
- - step_duration.png
- qlora_compile.ipynb
- README.md
.gitignore
README.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Solutions for the [Unsloth puzzles](https://colab.research.google.com/drive/1JqKqA1XWeLHvnYAc0wzrR4JBCnq43HyH?usp=sharing) (Tasks A, B, C, E) plus one additional [Netra Runtime puzzle](https://hackmd.io/@netraruntime/r1M7l9r0Wx) (Task F).

Each submission is self-contained with its own README, notebook, and code:

| Task | Puzzle                                  | Directory                                                  |
| ---- | --------------------------------------- | ---------------------------------------------------------- |
| A    | NF4 Dequantize (Triton kernel)          | [`nf4_dequantize/`](nf4_dequantize/)                       |
| B    | QLoRA + FSDP2                           | [`fsdp2_qlora/`](fsdp2_qlora/)                             |
| C    | torch.compile QLoRA (zero graph breaks) | [`qlora_compile/`](qlora_compile/)                         |
| E    | Memory-Efficient Backprop               | [`memory_efficient_backprop/`](memory_efficient_backprop/) |
| F    | Fused TQ3_0 Dequant + MatMul            | [`fused_tq3/`](fused_tq3/)                                 |

## What these puzzles test

The puzzles probe two complementary skill sets:

1. **Library and framework understanding** — familiarity with PyTorch internals (autograd, torch.compile, FSDP2), HuggingFace libraries (Transformers, PEFT, TRL), and the bitsandbytes quantization stack. Most solutions require reading and extending existing library code.

2. **Hardware-level kernel engineering** — writing fused Triton kernels that operate close to the GPU memory hierarchy, managing shared memory, cache eviction policies, instruction-level precision (FMA vs separate mul+add), and tiling strategies to reduce HBM round-trips.

## Scoring

I optimized all submissions to score full points based on the rubrics. I think (and hope) they would hit full points or at least near full across all attempted tasks. Each task README includes per-rubric-item status tracking and a detailed breakdown of my approach.

## Some Results

### Task B: QLoRA + FSDP2

### Task C: torch.compile QLoRA

## Feedback and discussion

I'm open to feedback on these solutions and welcome discussions about alternative approaches, edge cases, or further optimizations. If you find this repository useful as a learning resource for GPU kernel programming or LLM training infrastructure, feel free to use it and reach out.

unsloth-puzzles

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README