llm-from-scratch
Build LLM from scratch
File Explorer
- deploy-site.yml
- FUNDING.yml
- alpaca_sft.prompt
- question_only.prompt
- r1_zero.prompt
- zero_shot_system_prompt.prompt
- __init__.py
- args.py
- dataset.py
- drgrpo_grader.py
- evaluate.py
- grpo.py
- r1_prompt.py
- sft.py
- sft_tokenizer_saver.py
- train_rl.py
- util.py
- __init__.py
- bench_mark_atten.py
- bench_mark_atten_jit.py
- bench_mark_ddp.py
- bench_mark_flash_attention.py
- bench_mark_model.py
- __init__.py
- deduplicate.py
- harmful_detect.py
- html_process.py
- language_identification.py
- mask_pii.py
- quality_classfier.py
- quality_filter.py
- data-pipeline.md
- distributed-training.md
- flash-attention.md
- index.md
- project-overview.md
- rlft-gsm8k.md
- sft-gsm8k.md
- tokenizer.md
- training-loop.md
- transformer-core.md
- 2026-06-17-astro-github-pages-project-site.md
- 2026-06-18-astro-docs-chapter-redesign.md
- 2026-06-17-astro-github-pages-project-site-design.md
- 2026-06-18-astro-docs-chapter-redesign.md
- data-pipeline.md
- distributed-training.md
- flash-attention.md
- index.md
- project-overview.md
- rlft-gsm8k.md
- sft-gsm8k.md
- tokenizer.md
- training-loop.md
- transformer-core.md
- site.md
- loss.png
- lr.png
- __init__.py
- flash_attention_mock.py
- flash_attention_triton.py
- __init__.py
- args.py
- bpe_tokenizer.py
- checkpoint.py
- generating.py
- training.py
- transformer.py
- __init__.py
- ddp.py
- sharded_optimizer.py
- loss.png
- lr.png
- favicon.svg
- ArchitectureFlow.astro
- DocsChapterNav.astro
- FeaturePortalGrid.astro
- HeroBackground.astro
- HeroSection.astro
- MermaidInit.astro
- MetricCards.astro
- ProofBand.astro
- SectionIntro.astro
- SiteFooter.astro
- SiteHeader.astro
- site.ts
- BaseLayout.astro
- DocsLayout.astro
- docs.ts
- locale.ts
- [slug].astro
- index.astro
- [slug].astro
- index.astro
- architecture.astro
- benchmarks.astro
- index.astro
- sft-rlft.astro
- architecture.astro
- benchmarks.astro
- index.astro
- sft-rlft.astro
- global.css
- site.spec.ts
- content.spec.ts
- docs.spec.ts
- package-lock.spec.ts
- routes.spec.ts
- .npmrc
- astro.config.mjs
- package-lock.json
- package.json
- playwright.config.ts
- tsconfig.json
- test_4d_scaled_dot_product_attention.npz
- test_adamw.npz
- test_compute_entropy.npz
- test_compute_group_normalized_rewards_no_normalize_by_std.npz
- test_compute_group_normalized_rewards_normalize_by_std.npz
- test_compute_grpo_clip_loss_large_cliprange.npz
- test_compute_grpo_clip_loss_small_cliprange.npz
- test_compute_naive_policy_gradient_loss.npz
- test_compute_policy_gradient_loss_grpo_clip.npz
- test_compute_policy_gradient_loss_no_baseline.npz
- test_compute_policy_gradient_loss_reinforce_with_baseline.npz
- test_embedding.npz
- test_get_response_log_probs.npz
- test_grpo_microbatch_train_step_grpo_clip.npz
- test_grpo_microbatch_train_step_grpo_clip_10_steps.npz
- test_linear.npz
- test_masked_mean_dim0.npz
- test_masked_mean_dim1.npz
- test_masked_mean_dimlast.npz
- test_masked_mean_dimNone.npz
- test_masked_normalize_dim0.npz
- test_masked_normalize_dim1.npz
- test_masked_normalize_dimlast.npz
- test_masked_normalize_dimNone.npz
- test_multihead_self_attention.npz
- test_multihead_self_attention_with_rope.npz
- test_positionwise_feedforward.npz
- test_rmsnorm.npz
- test_rope.npz
- test_scaled_dot_product_attention.npz
- test_sft_microbatch_train_step.npz
- test_sft_microbatch_train_step_10_steps.npz
- test_sft_microbatch_train_step_normalize.npz
- test_swiglu.npz
- test_tokenize_prompt_and_output.npz
- test_train_bpe_special_tokens.pkl
- test_transformer_block.npz
- test_transformer_lm.npz
- test_transformer_lm_truncated_input.npz
- doc1.txt
- doc2.txt
- doc3.txt
- doc4.txt
- doc5.txt
- pytorch_license.txt
- rails_mit_license.txt
- react_mit_license.txt
- doc1.txt
- doc2.txt
- doc3.txt
- doc4.txt
- doc5.txt
- special_tokens_map.json
- tokenizer.json
- tokenizer_config.json
- config.json
- generation_config.json
- model.safetensors
- config.json
- generation_config.json
- model.safetensors
- model.pt
- model_config.json
- address.txt
- corpus.en
- ddp_test_data.pt
- ddp_test_labels.pt
- german.txt
- gpt2_merges.txt
- gpt2_vocab.json
- high_quality_wiki_reference.txt
- low_quality_cc.txt
- moby.html
- moby_extracted.txt
- sft_sample.jsonl
- special_token_double_newlines_non_whitespace.txt
- special_token_trailing_newlines.txt
- tinystories_sample.txt
- tinystories_sample_5M.txt
- tokenized_sft_sample.json
- train-bpe-reference-merges.txt
- train-bpe-reference-vocab.json
- __init__.py
- adapters.py
- common.py
- conftest.py
- test_attention.py
- test_data.py
- test_ddp.py
- test_ddp_individual_parameters.py
- test_deduplication.py
- test_dpo.py
- test_extract.py
- test_generating.py
- test_grpo.py
- test_langid.py
- test_metrics.py
- test_model.py
- test_nn_utils.py
- test_optimizer.py
- test_pii.py
- test_quality.py
- test_serialization.py
- test_sft.py
- test_sharded_optimizer.py
- test_tokenizer.py
- test_toxicity.py
- test_train_bpe.py
- .gitignore
- BENCHMARK.md
- LICENSE
- pyproject.toml
- README.md
- README_cn.md
- uv.lock
# Use via CDN
jsDelivrjsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.
Command Glossary
Commands referenced in this DOCs, explained below.
gunzip
View Details ▼
gunzip
This command is an alias of `gzip --decompress`.
tldr gzip
View documentation for the original command:
npm
View Details ▼
npm
JavaScript and Node.js package manager.
Manage Node.js projects and their module dependencies.
Some subcommands such as `install`, `run`, etc. have their own usage documentation.
npm init {{[-y|--yes]}}
Create a `package.json` file with default values (omit `--yes` to do it interactively):
npm {{[i|install]}}
Download all the packages listed as dependencies in `package.json`:
npm {{[i|install]}} {{package_name}}@{{version}}
Download a specific version of a package and add it to the list of dependencies in `package.json`:
parallel
View Details ▼
parallel
Run commands on multiple CPU cores.
See also: `xargs`.
parallel gzip ::: {{path/to/file1 path/to/file2 ...}}
Gzip several files at once, using all cores:
ls *.txt | parallel {{[-j|--jobs]}} 4 gzip
Read arguments from `stdin`, run 4 jobs at once:
parallel convert {} {.}.png ::: *.jpg
Convert JPEG images to PNG using replacement strings:
pytest
View Details ▼
pytest
Run Python tests.
pytest {{path/to/test_file1.py path/to/test_file2.py ...}}
Run tests from specific files:
pytest -k {{expression}}
Run tests with names matching a specific [k]eyword expression:
pytest {{[-x|--exitfirst]}}
Exit as soon as a test fails or encounters an error:
python
View Details ▼
python
Python language interpreter.
python
Start a REPL (interactive shell):
python {{path/to/file.py}}
Execute a specific Python file:
python -i {{path/to/file.py}}
Execute a specific Python file and start a REPL:
uv run
View Details ▼
uv run
Run a command or script in the project environment.
uv run {{path/to/script.py}}
Run a Python script:
uv run {{[-m|--module]}} {{module_name}}
Run a Python module:
uv run {{[-w|--with]}} {{package}} {{command}}
Run a command with additional packages installed temporarily:
