BoLT - GitLite

KO | EN

gitlite — search

#python #hacktoberfest #react #go #golang #machine-learning #nodejs #macos #windows #awesome-list #security #pytorch #mcp #open-source

BoLT

(★ 134)

Code for "Reasoning to Learn from Latent Thoughts"

파일 탐색기

최종 버전 다운로드 (.zip)

- - __init__.py
  - eval.py
  - finetune.py
  - gen_latents.py
  - generate.py
  - latent_bootstrap.py
  - prepare_data.py
  - train.py
  - transformer.py
- __init__.py
- paradigm_illustration.jpg
- - bootstrap_fixed_data.sh
  - slurm.sh
  - synth_data_method_compare.sh
- install_requirements.sh
- prepare_data.sh
- prepare_hf_ckpts.sh
- prepare_hf_datasets.sh
- run_bootstrap_latents.sh
- run_finetune_eval.sh
- run_gen_latents.sh
- run_train_bootstrap.sh
- run_train_warmstart.sh
- - finemath_4plus_final_2b_separated_bootstrap_split_cached.json
  - finemath_4plus_final_500m_warmstart_split_cached.json
  - finemath_4plus_final_fixed_val_split_cached.json
  - finemath_4plus_raw.json
- - cpt_eval.yaml
- - gsm8k_finetune_vllm.yaml
  - math_finetune_vllm.yaml
- - bootstrap_latents_cpt.yaml
  - gpt_latent_thoughts.yaml
  - wrap_baseline.yaml
  - wrap_cot.yaml
- - tinyllama_1B_cpt.yaml
- __init__.py
- args.py
- checkpoint.py
- data.py
- distributed.py
- float8.py
- logger.py
- metrics.py
- optim.py
- probe.py
- profiling.py
- stool.py
- tokenizer.py
- transformer.py
- plot_bootstrap_fix_data.ipynb
- plot_finetune_eval_results.ipynb
- plot_synth_data_method_compare.ipynb
- plot_utils.py
- convert_consolidated_lingua_ckpt_to_hf.py
- download_cached_datasets.py
- download_hf_ckpt.py
- download_hf_data.py
.gitignore
environment.yaml
LICENSE
README.md
requirements.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/ryoungj/BoLT

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd BoLT

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

jupyter notebook

브라우저에서 노트북(.ipynb) 파일들을 열람하고 실행할 수 있는 Jupyter 화면을 켭니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Reasoning to Learn by Bootstrapping Latent Thoughts

<div align="center" style="font-size: 28px;">
    <a href="https://arxiv.org/abs/2503.18866">📃 Paper</a> &nbsp;&nbsp;|&nbsp;&nbsp;
    <a href="https://huggingface.co/datasets/ryoungj/bootstrap-latent-thought-data">🤗 Data</a> &nbsp;&nbsp;|&nbsp;&nbsp;
    <a href="https://x.com/YangjunR/status/1904943713677414836">🐦 Twitter</a>
</div>

<br>

This repo contains the code for our paper [Reasoning to Learn from Latent Thoughts](https://arxiv.org/abs/2503.18866).

<div align="center">
  <img src="assets/paradigm_illustration.jpg" width="65%">
</div>

We introduce a "reasoning to learn" paradigm for data-efficient pretraining of language models (LMs) and model self-improvement on task-agnostic data.
Motivated by how humans apply deliberate thinking to learn from limited data, we train an LM to infer (or “decompress”) latent thoughts underlying the highly compressed observed data. These synthesized latent thoughts augment the raw observed data during pretraining, improving the LM’s data efficiency. This procedure can be iteratively applied through an Expectation-Maximization algorithm and form a model
self-improvement loop where increasingly capable LMs synthesize more effective latent thoughts, which in turn train more capable models -- we call this apporach **Bo**otstrapping **L**atent **T**houghts (**BoLT**).

Our implementation is built upon the [Meta Lingua](https://github.com/facebookresearch/lingua) codebase for LM pretraining.

## Preparation

### Setup Environment

1. Create a conda environment:
```
conda env create -f environment.yaml
conda activate bootstrap-latents
```

2. Install requirements: `bash bin/install_requirements.sh`

3. Create soft links to store large datasets and experiment checkpoints:
```
ln -s [YOUR_SCRATCH_DIR]/data data
ln -s [YOUR_SCRATCH_DIR]/exp_logs exp_logs
```

4. Create a `.env` file with your API keys for synthetic data generation and HuggingFace access:
```
OPENAI_API_KEY=[YOUR_OPENAI_API_KEY]
HF_TOKEN=[YOUR_HF_TOKEN]
```

5. If you are using sphinx, specify the slurm config in `bin/slurm.sh`.

### Prepare Data and Pretrained Checkpoints

1. Download pretrained TinyLlama checkpoints and convert to Lingua-compatible format: `bash bin/prepare_hf_ckpts.sh`

2. Download and preprocess FineMath-4+ dataset: `bash bin/prepare_hf_datasets.sh [DOWNLOAD_PREPROCESSED_DATASETS]`. Setting `DOWNLOAD_PREPROCESSED_DATASETS=true` will download our processed/generated datasets from HuggingFace used in our experiments, otherwise the data processing/generation will be done from scratch. We recommend setting it to either `true` or `false` for the whole experiment setup to ensure consistency (such as the train/val split).

Note that for the specified number of tokens in the dataset name, we have rounded them to the nearest multiple of 500M, and the actual number should be multiplied by 0.96 - so for example, `2b` corresponds to ~1.92B tokens.

## Synthetic Latent Generation Experiments

Here we include the code for synthetically generating latent thoughts with GPT-4o-mini and comparing it with other alternatives in the data-constrained setup such as training repeatedly on the raw data or synthetically generating paraphrases.

1. Prepare the raw data corpus: `bash bin/prepare_data.sh warmstart [DOWNLOAD_PREPROCESSED_DATASETS]`

2. Generate synthetic data using GPT-4o-mini with batch API: `bash bin/run_gen_latents.sh [METHOD] [DOWNLOAD_PREPROCESSED_DATASETS]` with the `[METHOD]` argument specifying the synthetic data generation method selected from {`latent_thoughts`, `wrap_baseline`, `wrap_cot`}.

3. Run the training script with different synthesized data: `bash ./bin/run_train_warmstart.sh [MODE] synth_data_method_compare.sh [DATA_TYPE]`, where:
- `[MODE]` is the mode for executing the script, selected from `cat` (for printing the commands only), `run` (for running the commands locally), `launch` (for submitting the jobs to slurm).
- `[DATA_TYPE]` is the data used for training, selected from {`latent_thoughts`, `raw_repeat`, `raw_fresh`, `wrap_baseline`, `wrap_cot`}.
- The default training setups work for 4xH200 GPUs, adjust `PER_GPU_BATCH_SIZE` in the script to match your GPU setup.

4. Plot the results with [`plot_synth_data_method_compare.ipynb`](plots/plot_synth_data_method_compare.ipynb).

## Bootstrapping Latent Experiments

Here we include the code for bootstrapping latent thought models with self-generated latents on fixed data.

1. Prepare the raw data corpus (~1.92B tokens): `bash bin/prepare_data.sh bootstrap_fixed_data [DOWNLOAD_PREPROCESSED_DATASETS]`.

2. Train warmstart model on a small corpus (~240M raw tokens) with GPT-4o-mini synthetic latents: `bash bin/run_train_warmstart.sh [cat|run|launch] bootstrap_fixed_data.sh`.

3. Start the bootstrapping loop by training with self-generated latents on the fixed raw data corpus. For each iteration `BOOTSTRAP_ITERATION`:
    - Generate the latents from the model trained at the previous iteration. You can choose to:
        - Generate the latents locally by first running `bash bin/run_bootstrap_latents.sh launch bootstrap_fixed_data.sh [BOOTSTRAP_ITERATION]` to launch distributed synthetic data generation jobs (by default, over 200 async jobs) and then `bash bin/run_bootstrap_latents.sh merge bootstrap_fixed_data.sh [BOOTSTRAP_ITERATION]` to merge the generated data.
        - Download our generated datasets: `bash bin/run_bootstrap_latents.sh download bootstrap_fixed_data.sh [BOOTSTRAP_ITERATION]`.
    - Train the model on the generated data: `bash bin/run_train_bootstrap.sh [cat|run|launch] bootstrap_fixed_data.sh [DATA_SETUP] [BOOTSTRAP_ITERATION]`. Setting `DATA_SETUP` to `bootstrap_latent` to train on data augmented with bootstrap latents, and `raw_token_match_baseline` or `raw_flops_match_baseline` to train on raw data. Due to the high run-to-run variance on downstream evaluations, we recommend running at least 3 trials as being set in the script. The default training setup works for 8xH200 GPUs, adjust `PER_GPU_BATCH_SIZE` in the script to match your GPU setup.

4. Run raw-token baseline training: `bash bin/run_train_bootstrap.sh [cat|run|launch] bootstrap_fixed_data.sh [raw_token_match_baseline|raw_flops_match_baseline]`.

5. Plot the results with [`plot_bootstrap_fix_data.ipynb`](plots/plot_bootstrap_fix_data.ipynb).

6. Due to the high variance of few-shot prompting evaluation, we recommend running finetuning-based evaluation that demonstrates smoother performance metrics. 
    - Evaluate the model on the finetune datasets: `bash bin/run_finetune_eval.sh [cat|run|launch] math bootstrap_fixed_data.sh [DATA_SETUP] [BOOTSTRAP_ITERATION]`. To further reduce variance, increase the number of seeds (5 by default) and the number of samples per prompt (`num_samples_per_prompt=16` by default).
    - Plot the results with [`plot_finetune_eval_results.ipynb`](plots/plot_finetune_eval_results.ipynb).

## Citation

```
@article{ruan2025reasoning,
  title={Reasoning to Learn from Latent Thoughts},
  author={Ruan, Yangjun and Band, Neil and Maddison, Chris J and Hashimoto, Tatsunori},
  journal={arXiv preprint arXiv:2503.18866},
  year={2025}
}
```