stoa

KO | EN

gitlite — search

#kubernetes #artificial-intelligence #command-line #terminal #ai-agents #shell #github #kotlin #bash #markdown

(★ 9)

RL post-training with your Apple-silicon Macs as the rollout fleet: Mac workers generate experience, GPUs learn, synced over Cloudflare R2

File Explorer

ZIP Download

- - ci.yml
- how-it-works.png
- how-it-works.svg
- how_it_works_fig.py
- stoa-banner.svg
- ab_kvopt.py
- analyze_ab.py
- README.md
- run_ab.sh
- async_uploader.py
- logging_setup.py
- pulse_chain.py
- pulse_delta.py
- pulse_quant.py
- pulse_sync.py
- r2_commit.py
- r2_io.py
- r2_rollout_log.py
- r2_stats_log.py
- filtered_training_set.jsonl
- README.md
- architecture.md
- async-learner-design.md
- r2-setup.md
- testing.md
- - README.md
  - run_qwen25_gsm8k_decoupled.sh
- - eval_serve_sglang.sh
  - generate_with_search_lfm2.py
  - launch_fleet.env.example
  - preflight_check.py
  - prep_dataset.py
  - prep_eval.py
  - prep_worker_dataset.py
  - psqa_env.py
  - qa_em.py
  - README.md
  - resume_from_full.sh
  - run_lfm25_psqa_fleet.sh
- - patch_mlx_lfm2_moe_padslab.py
  - patch_mlx_lfm2_moe_router.py
- - retrieval_server.py
- clean_node.sh
- detect_mac_tier.sh
- install_supervisor.sh
- install_worker_key.sh
- join_and_run.sh
- mac_fleet.hosts.example
- provision_mac_fleet.sh
- run.env.example
- run.env.fleet-16gb.example
- run.env.fleet-24gb.example
- run.env.fleet.example
- setup_mac_worker.sh
- setup_retriever.sh
- setup_retriever_linux.sh
- apply_slime_patches.py
- - heldout_full5000_n8.csv
  - REPRODUCE.md
- build_pubmed_bm25.py
- fetch_data.sh
- hf_pull.py
- README.md
- - _fake_s3.py
  - _pulse_test_ids.py
  - test_producer_consumer.py
  - test_pulse_chain.py
  - test_pulse_edge.py
  - test_pulse_failclosed.py
  - test_pulse_gaps.py
  - test_pulse_plane.py
  - test_pulse_quant.py
  - test_pulse_realr2.py
  - test_pulse_realr2_correctpath.py
  - test_pulse_roundtrip.py
  - test_pulse_scenarios.py
  - test_pulse_sparsity.py
  - test_pulse_taxonomy.py
  - test_pulse_v2_protocol.py
  - test_pulse_wedge.py
  - test_r2_commit.py
  - test_r2_io.py
  - test_r2_io_hardening.py
  - test_r2_rollout_log.py
- - test_apply_slime_patches.py
- - test_adaptive_fetch.py
  - test_ckpt_export.py
  - test_cursor_handoff.py
  - test_drgrpo_reducer.py
  - test_full_ckpt.py
  - test_math_rm_grading.py
  - test_math_rm_threading.py
  - test_publish_async.py
  - test_pulse_stats.py
  - test_replay_buffer.py
  - test_rollout_metrics.py
- - test_cb_native.py
  - test_cb_scheduler.py
  - test_class_ds.py
  - test_dynamic_sampling.py
  - test_multigroup.py
  - test_native_kernels.py
  - test_paged_cache.py
  - test_paged_fullscale.py
  - test_paged_ledger_safety.py
  - test_paged_race.py
  - test_paged_refcount_lock.py
  - test_paged_table_guards.py
  - test_plane_native_load.py
  - test_prefix_store.py
  - test_psqa_streaming_phase2.py
  - test_quant_gap.py
  - test_search_floor.py
  - test_stop_sequences.py
  - test_stop_signal.py
  - test_streaming_e2e_mlx.py
  - test_streaming_phase2.py
  - test_streaming_upload.py
  - test_weight_prefetch.py
  - test_worker_config.py
  - test_worker_curriculum.py
  - test_worker_identity.py
  - test_worker_stats.py
- conftest.py
- mlx-lm-LICENSE
- PaperSearchQA-LICENSE
- Search-R1-LICENSE
- slime-LICENSE
- verl-LICENSE
- ckpt_export.py
- dppo_gate.py
- drgrpo_reducer.py
- eval_curve.py
- eval_gsm8k.py
- eval_pass_k.py
- full_ckpt.py
- gsm8k_rm.py
- inspect_rollouts.py
- learning_curve.py
- live_anchor_eval.py
- math_rm.py
- prep_gsm8k.py
- prep_math.py
- psqa_anchor_watcher.py
- psqa_eval_shakedown.py
- r2_rollout.py
- replay_buffer.py
- wordle_advantage.py
- canary_oracle.py
- cb_scheduler.py
- dRL_producer.py
- ds_gate.py
- native_kernels.py
- paged_cache.py
- paged_pool.py
- prefix_store.py
- producer_decode.py
- producer_weights.py
- psqa_mlx_smoke.py
- psqa_mlx_smoke_streaming.py
- psqa_retriever.py
- psqa_rollout.py
- requirements.txt
- run_b0_battery.sh
- run_dRL_worker_gsm8k.sh
- run_dRL_worker_psqa.sh
- run_local_e2e.py
- stop_sequences.py
- sweep_telemetry.py
- turn_mask.py
- worker_config.py
- worker_curriculum.py
- worker_stats.py
.env.example
.gitignore
.pre-commit-config.yaml
.python-version
.r2env.example
CITATION.cff
CLAUDE.md
CONTRIBUTING.md
LICENSE
NOTICE
pyproject.toml
README.md
SECURITY.md
THIRD_PARTY_NOTICES.md
uv.lock

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

*RL post-training with your Apple-silicon Macs as the MLX rollout fleet.*

[![status: proof of concept](https://img.shields.io/badge/status-proof_of_concept-96700F?style=flat-square)](https://raw.githubusercontent.com/PluralisResearch/stoa/main/runs/psqa-decoupled/REPRODUCE.md)
[![License: MIT](https://img.shields.io/badge/License-MIT-022548?style=flat-square)](https://raw.githubusercontent.com/PluralisResearch/stoa/main/LICENSE)
[![Python 3.12](https://img.shields.io/badge/Python-3.12-022548?style=flat-square)](https://raw.githubusercontent.com/PluralisResearch/stoa/main/.python-version)
[![rollouts: MLX](https://img.shields.io/badge/rollouts-MLX-022548?style=flat-square)](https://github.com/ml-explore/mlx)
[![trainer: slime](https://img.shields.io/badge/trainer-slime-022548?style=flat-square)](https://github.com/THUDM/slime)

</div>

stoa disaggregates RL post-training into two halves. Disposable Apple-silicon
Macs, anywhere from one to thousands, generate rollouts in MLX; one or more CUDA GPUs learn through Megatron /
[slime](https://github.com/THUDM/slime). The two sides meet only at a Cloudflare R2 bucket, where workers
stream rollouts up and the trainer streams weights back. A Mac can join, leave, or crash mid-run and the
trainer keeps learning from whatever landed.

> **Status: proof of concept.** This project is just a proof of concept at this stage, and it is not production-hardened.
> A production-grade version is in development and will be released.

## How it works

`rollouts/` holds immutable per-worker records. The weight plane is a `current.json` version pointer plus
`anchors/` checkpoints and per-version `versions/` objects, with PULSE sparse-lossless deltas between
anchors; a worker polls the pointer and pulls only when the version advances. The trainer filters rollouts
by staleness and reuse, then applies dual-clip PPO plus the DPPO gate on each worker's recorded logprobs
before one Megatron step. GRPO stays the RL algorithm. The full design is in
[`docs/architecture.md`](./docs/architecture.md).

## Before you start

You need:

- **[`uv`](https://astral.sh/uv)** on each Mac. It installs Python 3.12 and the worker dependencies.
- **A Cloudflare R2 bucket and S3 token.** See [`docs/r2-setup.md`](./docs/r2-setup.md).
- **A CUDA GPU with NVIDIA drivers, model and checkpoint storage, and SSH.** The 1.5B quickstart trains on an A100 or H100; the 8B reference run used one B200.
- **Apple-silicon Macs.** Run `ops/detect_mac_tier.sh` for each Mac's memory tier and decode batch `G`.

Optional: a Hugging Face token for gated models, Weights & Biases for training and eval curves, and a separate GPU for held-out eval.

## Quickstart

**1. Test rollouts on your Mac.** This runs Qwen2.5-1.5B in MLX, scores GSM8K rollouts, and reads the
records back through a temporary R2 prefix. No trainer yet.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra worker --extra data
cp .r2env.example ~/.r2env && chmod 600 ~/.r2env && $EDITOR ~/.r2env
uv run python worker/run_local_e2e.py 2 4 128
```

The arguments are 2 groups, 4 samples, 128 tokens. Omit them for the default 3 groups, 4 samples, 256
tokens. Set `MODEL` to use another model: `MODEL=Qwen/Qwen2.5-3B-Instruct uv run python worker/run_local_e2e.py`.

**2. Run the trainer and worker.** Bring up a slime/Megatron container and convert the base model as in
[`envs/gsm8k/README.md`](./envs/gsm8k/README.md). The trainer repo must live at `/root/dRL` (the scripts
hardcode `.r2env`, `patches/`, and `PYTHONPATH` there); the detailed step 2 sets that up. Then:

```bash
DRL_RUN=my-gsm8k DRL_PULSE_PLANE=bf16 bash envs/gsm8k/run_qwen25_gsm8k_decoupled.sh

# on your Mac
uv run python trainer/prep_gsm8k.py ~/dRL_data
DRL_RUN=my-gsm8k DRL_PULSE_PLANE=bf16 bash worker/run_dRL_worker_gsm8k.sh
```

Use the same `DRL_RUN`, `DRL_PULSE_PLANE`, and R2 bucket on both. GSM8K reward climbs over a few tens of
steps. [`envs/gsm8k/README.md`](./envs/gsm8k/README.md) covers GPU setup, SSH, model conversion, the curve,
and teardown; [`runs/psqa-decoupled/REPRODUCE.md`](./runs/psqa-decoupled/REPRODUCE.md) is the same shape at
8B.

## Run a fleet

This is the PaperSearchQA reference fleet: `ops/join_and_run.sh` brings up the PSQA retriever and worker. GSM8K has no retriever, so for a GSM8K fleet run `worker/run_dRL_worker_gsm8k.sh` on each Mac (per the quickstart) instead of `join_and_run.sh`.

1. **Start the trainer** in the slime/Megatron container with `--rollout-num-gpus 0 --rollout-function-path r2_rollout.generate_rollout`. It publishes the initial anchor, then consumes rollouts and publishes PULSE weights.
2. **Join each Mac** with `bash ops/join_and_run.sh --name "<label>"`. It starts the task retriever if needed, syncs weights from R2, and streams rollouts. No registration, and you can add or drop boxes any time.
3. **Watch** the trainer's W&B streams under `train/*`, `rollout/*`, and `worker/*`, plus a held-out eval on a separate GPU.
4. **Stop** the trainer with `ray stop --force`. Stop a Mac with `pkill -f dRL_producer.py` or uninstall its supervisor. The trainer keeps whatever rollouts already landed.

> **Renting Macs is optional.** Use Macs you own, or rent by the hour from [Scaleway](https://www.scaleway.com/), [Flow Swiss](https://flow.swiss/), or [AWS EC2 Mac](https://aws.amazon.com/ec2/instance-types/mac/) (24-hour minimum per dedicated host).

Our reference recipe (the fleet launcher and preflight, per-tier `run.env`, and the trainer setup we used) is in [`runs/psqa-decoupled/REPRODUCE.md`](./runs/psqa-decoupled/REPRODUCE.md).

## Reference run

Our reference run trains **LFM2.5-8B-A1B** with GRPO on **PaperSearchQA** across a mixed Apple-silicon
fleet plus one B200. On the full 5,000-question held-out test set, cover-EM pass@1 climbs from about 0.29 to
0.63 over tens of versions (pass@8 to about 0.83), and search rate rises from about 0.22 to 0.85.
[`runs/psqa-decoupled/REPRODUCE.md`](./runs/psqa-decoupled/REPRODUCE.md) is the full runnable recipe: the
LFM2 slime plugin ships in-repo and the trainer runs on the public slime base, so you can rerun it end to
end on your own trainer GPU plus Macs. You retrain from the public base model, so you reproduce the curve,
not our exact checkpoint (that one stays private, and off-policy RL over a live fleet is stochastic). The
GSM8K quickstart above is the faster smoke test.

## License

stoa is [MIT licensed](./LICENSE). A few vendored or adapted files keep their upstream Apache-2.0 terms,
recorded in [`NOTICE`](./NOTICE) and [`THIRD_PARTY_NOTICES.md`](./THIRD_PARTY_NOTICES.md), with full texts
in [`third_party_licenses/`](./third_party_licenses/). The reference run's model and datasets carry their
own licenses, noted in [`runs/psqa-decoupled/REPRODUCE.md`](./runs/psqa-decoupled/REPRODUCE.md).

## Citation

If you use stoa in your research, please cite it:

```bibtex
@software{stoa2026,
  author  = {Miahi, Erfan},
  title   = {{stoa: RL post-training with Apple-silicon Macs as the rollout fleet}},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/PluralisResearch/stoa}
}
```

Companion write-up: Erfan Miahi, "RL Post-Training on Macs", Pluralis Research Blog, July 2026.

## References

- **GRPO**, Shao et al., DeepSeekMath: <https://arxiv.org/abs/2402.03300>
- **DPPO** (the off-policy gate, `trainer/dppo_gate.py`), Qi et al.: <https://arxiv.org/abs/2602.04879>
- **slime**: <https://github.com/THUDM/slime> · **MLX**: <https://github.com/ml-explore/mlx>

stoa

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

stoa

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README