MARS_preview

KO | EN

gitlite — search

#database #open-source #cloud-native #claude-code #mcp #reactjs #claude #computer-vision #react #golang

(★ 9)

Preview Code for MARS Paper

File Explorer

ZIP Download

- - test_workload.jsonl
- - command_guard.py
  - common.py
  - llm_trace_bridge.py
  - openhands_runner.py
  - request_bootstrap.py
  - request_worker.py
  - run_task_workload.py
  - sandbox_shell.sh
  - test_command_guard.py
  - test_run_task_workload.py
  - test_sandbox_shell.py
  - test_tool_trace_bridge.py
  - tool_trace_bridge.py
  - trace_writer.py
  - vllm_api_server_mars.py
- - common_runtime.sh
  - run_baseline_continuum.sh
  - run_openhands_qwen3_local.sh
  - run_policy_smoke_gpu1.sh
- - cpu_extension.cmake
  - hipify.py
  - utils.cmake
- - continuum.base.yml
- - activation_kernels.cu
  - cache.h
  - cache_kernels.cu
  - cuda_compat.h
  - cuda_utils.h
  - cuda_utils_kernels.cu
  - cuda_view.cu
  - cumem_allocator.cpp
  - custom_all_reduce.cu
  - custom_all_reduce.cuh
  - custom_all_reduce_test.cu
  - custom_quickreduce.cu
  - dispatch_utils.h
  - layernorm_kernels.cu
  - layernorm_quant_kernels.cu
  - ops.h
  - permute_cols.cu
  - pos_encoding_kernels.cu
  - sampler.cu
  - torch_bindings.cpp
  - type_convert.cuh
- - Dockerfile
  - Dockerfile.cpu
  - Dockerfile.nightly_torch
  - Dockerfile.ppc64le
  - Dockerfile.rocm
  - Dockerfile.rocm_base
  - Dockerfile.s390x
  - Dockerfile.tpu
  - Dockerfile.xpu
- - build.txt
  - common.txt
  - cpu-build.txt
  - cpu.txt
  - cuda.txt
  - dev.txt
  - docs.txt
  - kv_connectors.txt
  - lint.txt
  - nightly_torch_test.txt
  - rocm-build.txt
  - rocm-test.txt
  - rocm.txt
  - test.in
  - test.txt
  - tpu.txt
  - xpu.txt
- - create_continuum_vllm_conda_env.sh
  - start_vllm_service.sh
- - check_init_lazy_imports.py
  - check_pickle_imports.py
  - check_repo.sh
  - check_spdx_header.py
  - check_triton_import.py
  - enforce_regex_import.py
  - generate_cmake_presets.py
  - generate_nightly_torch_test.py
  - install_deepgemm.sh
  - install_nixl.sh
  - mypy.sh
  - png-lint.sh
  - report_build_time_ninja.py
  - shellcheck.sh
  - update-dockerfile-graph.sh
  - validate_config.py
- - __init__.py
  - _bc_linter.py
  - _custom_ops.py
  - _ipex_ops.py
  - _version.py
  - beam_search.py
  - collect_env.py
  - connections.py
  - env_override.py
  - envs.py
  - forward_context.py
  - logger.py
  - logits_process.py
  - logprobs.py
  - outputs.py
  - pooling_params.py
  - py.typed
  - sampling_params.py
  - scalar_type.py
  - scripts.py
  - sequence.py
  - tasks.py
  - test_utils.py
  - tracing.py
  - version.py
- .clang-format
- .dockerignore
- .gitignore
- .markdownlint.yaml
- .pre-commit-config.yaml
- .readthedocs.yaml
- .shellcheckrc
- .yapfignore
- CMakeLists.txt
- CODE_OF_CONDUCT.md
- CONDA_SETUP.md
- CONTRIBUTING.md
- DCO
- find_cuda_init.py
- format.sh
- LICENSE
- MANIFEST.in
- mkdocs.yaml
- pyproject.toml
- README.md
- RELEASE.md
- SECURITY.md
- setup.py
- use_existing_torch.py
.gitattributes
.gitignore
LICENSE.txt
README.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**MARS** is a serving-system prototype for heterogeneous agentic workloads. It
extends vLLM with scheduling policies for workloads that interleave GPU
inference with CPU-side tool execution, and provides an OpenHands-based
integration path for end-to-end agent evaluation.

This repository is a preview release for the
[MARS paper](https://arxiv.org/abs/2604.26963). It includes the code required
to run the experiments, the MARS-modified vLLM source code, the
OpenHands workload runner, and a compact hybrid workload.

Supported scheduling policies:

- `mars`: adaptive co-scheduling for heterogeneous agentic workloads.
- `fcfs`: first-come, first-served request scheduling.
- `continuum` and `continuum_dy`: baselines built from the official
  [vLLM-Continuum](https://github.com/Hanchenli/vllm-continuum) implementation;
  `continuum_dy` extends it with dynamically computed TTL values.
- `autellix`: an Autellix baseline implemented with the PLAS scheduling policy
  proposed in the [Autellix paper](https://arxiv.org/abs/2502.13965).
- `infercept`: an InferCept baseline implemented following the
  [InferCept paper](https://arxiv.org/abs/2402.01869)'s min-waste principle
  across tool-call interceptions.

---

## How It Works

### MARS-Modified vLLM

`vllm-continuum/` contains the local vLLM source code used by the experiments.
It adds the scheduler implementations and runtime hooks needed to compare
MARS with the bundled baselines.

### Agentic Workloads

This preview release includes a compact test workload at
`benchmark/hybrid_workloads/test_workload.jsonl`. This file is the default
workload consumed by the vLLM replay and OpenHands launch scripts.

### Evaluation Paths

The repository provides two execution paths:

- **vLLM replay**: runs the workload through an in-process vLLM engine with
  simulated tool-side execution.
- **OpenHands integration**: starts a local OpenAI-compatible vLLM server and
  runs OpenHands workers against it for end-to-end agent execution.

---

## Project Structure

```text
MARS/
|-- vllm-continuum/                  # MARS-modified vLLM source tree
|-- experiments/
|   |-- replay/vllm_tool_baseline/   # vLLM replay runtime
|   |-- openhands/                   # OpenHands integration and tests
|   `-- scripts/                     # Maintained launch scripts
|-- benchmark/
|   |-- hybrid_workloads/test_workload.jsonl # Test workload
|   |-- GitTaskBench/                # Referenced task assets
|   `-- terminal-bench/              # Referenced task assets
`-- LICENSE.txt
```

---

## Installation

**Requirements:** `conda`, `git`, `cmake`, `ninja`, `python>=3.12`,
`torch==2.8.0+cu128`; Datacenter-level NVIDIA GPU. Tested on H100 and H200.

```bash
# Clone the repository and enter the repo root.
git clone <repo-url> MARS
cd MARS

# Build the MARS-modified vLLM environment.
cd vllm-continuum
MODE=source bash scripts/create_continuum_vllm_conda_env.sh
conda activate mars
cd ..

# If CUDA auto-detection fails, rebuild with explicit CUDA settings:
# cd vllm-continuum
# MODE=source CUDA_VERSION_OVERRIDE=12.8 TORCH_CUDA_TAG=cu128 \
#   bash scripts/create_continuum_vllm_conda_env.sh
# conda activate mars
# cd ..

# Verify that Python imports vLLM from this repository and sees the local
# FlashAttention extension modules.
PYTHONDONTWRITEBYTECODE=1 \
PYTHONPATH="$PWD:$PWD/vllm-continuum" \
python - <<'PY'
import importlib
import vllm

print("vllm:", vllm.__file__)
for name in ("vllm.vllm_flash_attn._vllm_fa2_C", "vllm.vllm_flash_attn._vllm_fa3_C"):
    mod = importlib.import_module(name)
    print(name, mod.__file__)
PY

# Create the OpenHands worker environment.
# The vLLM service runs in conda env `mars`; OpenHands workers use this venv.
# OpenHands 1.14 requires Python >=3.12. These commands use python3.12 because
# python3.13 is not installed by default on many servers; override as needed.
OPENHANDS_VENV="${OPENHANDS_VENV:-$HOME/.venvs/openhands312}"
OPENHANDS_VENV_PYTHON="${OPENHANDS_VENV_PYTHON:-python3.12}"
"$OPENHANDS_VENV_PYTHON" -m venv "$OPENHANDS_VENV"
"$OPENHANDS_VENV/bin/python" -m pip install --upgrade pip
"$OPENHANDS_VENV/bin/python" -m pip install \
  openhands-sdk==1.14.0 \
  openhands-tools==1.14.0 \
  litellm==1.82.3
export OPENHANDS_PYTHON="$OPENHANDS_VENV/bin/python"

# Verify the OpenHands runtime.
OPENHANDS_SUPPRESS_BANNER=1 "$OPENHANDS_PYTHON" - <<'PY'
from openhands.sdk import Agent, Conversation, LLM, Tool
from openhands.tools.terminal import TerminalTool
from openhands.tools.file_editor import FileEditorTool
print("OpenHands runtime OK")
PY
```

---

## Usage

Set `MODEL_PATH` to a local model directory before running GPU experiments. The
scripts default to `$HOME/models/Qwen3-Coder-30B-A3B-Instruct`.

### 1. vLLM Replay Experiment

Runs the test workload through the vllm replay baseline with the selected
scheduling policy:

```bash
conda activate mars
cd /path/to/MARS
CUDA_VISIBLE_DEVICES=0 \
MODEL_PATH=$HOME/models/Qwen3-Coder-30B-A3B-Instruct \
TASKS_PATH=$PWD/benchmark/hybrid_workloads/test_workload.jsonl \
NUM_REQUESTS=100 RPS=0.2 SCHEDULING_POLICY=mars \
bash experiments/scripts/run_baseline_continuum.sh
```

### 2. OpenHands Integration Experiment

Starts a local vLLM OpenAI-compatible server and runs the OpenHands workload
runner:

```bash
conda activate mars
cd /path/to/MARS
CUDA_VISIBLE_DEVICES=0 \
MODEL_PATH=$HOME/models/Qwen3-Coder-30B-A3B-Instruct \
TASKS_PATH=$PWD/benchmark/hybrid_workloads/test_workload.jsonl \
MAX_REQUESTS=20 RPS=0.2 \
bash experiments/scripts/run_openhands_qwen3_local.sh
```

### 3. Scheduling Policy Smoke Test

Runs all the scheduling policies on one GPU. By default, the smoke test uses the
first 200 requests for the vLLM replay target and the first 50 requests for the
OpenHands target.

```bash
conda activate mars
cd /path/to/MARS
bash experiments/scripts/run_policy_smoke_gpu1.sh
```

---

## End-to-End Walkthrough

The following commands run a compact verification pipeline on the bundled
workload:

```bash
conda activate mars
cd /path/to/MARS
export MODEL_PATH=$HOME/models/Qwen3-Coder-30B-A3B-Instruct

# Step 1: run a small vLLM replay job
CUDA_VISIBLE_DEVICES=0 NUM_REQUESTS=100 RPS=0.2 SCHEDULING_POLICY=mars \
  bash experiments/scripts/run_baseline_continuum.sh

# Step 2: run a small OpenHands job
CUDA_VISIBLE_DEVICES=0 MAX_REQUESTS=20 RPS=0.2 SCHEDULING_POLICY=mars \
  bash experiments/scripts/run_openhands_qwen3_local.sh

# Step 3: compare selected policies on GPU 0
GPU=0 POLICIES="fcfs mars" TARGETS="vllm openhands" \
  VLLM_REQUESTS=200 OPENHANDS_REQUESTS=50 \
  bash experiments/scripts/run_policy_smoke_gpu1.sh
```

---

## Reproducibility Notes

- All launch scripts are path-relative to the repository root.
- Override `MODEL_PATH`, `TASKS_PATH`, `PYTHON_BIN`, `MARS_PYTHON`,
  `OPENHANDS_PYTHON`, and output variables as needed for your machine.
- CPU binding defaults are `CPU_GPU=0-1`, `CPU_TOOL=16-23`, and
  `CPU_CORES=0-1,16-23`; override them if your node has a different CPU layout.

---

## Troubleshooting

`ValueError: Unsupported FA version: None`

: The local `vllm-continuum` tree was not built correctly, or Python is
  importing a source tree without `_vllm_fa2_C.abi3.so` and
  `_vllm_fa3_C.abi3.so`. Rebuild `vllm-continuum` in the active `mars`
  environment and rerun the installation verification.

`OPENHANDS_PYTHON not found`

: Create the OpenHands environment above or pass
  `OPENHANDS_PYTHON=/path/to/openhands/python`.

`AttributeError: Qwen2Tokenizer has no attribute all_special_tokens_extended`

: The `mars` environment resolved an incompatible tokenizer stack. Reinstall
  the tested versions with
  `python -m pip install transformers==4.55.2 tokenizers==0.21.1 huggingface-hub==0.36.2 lmcache==0.4.4`.

`Model path not found`

: Download or mount the model locally and set `MODEL_PATH=/path/to/model`.

---

## Coming Soon

- **Full experiment reproduction** — end-to-end scripts to reproduce the
  complete experiment results reported in the MARS paper.
- **Analysis and visualization utilities** — scripts for result analysis,
  plotting, figure generation, and workload construction.

---

## Acknowledgements

This repository builds on the following open-source projects:

- [vLLM](https://github.com/vllm-project/vllm)
- [vLLM-continuum](https://github.com/Hanchenli/vllm-continuum)
- [OpenHands](https://github.com/All-Hands-AI/OpenHands)
- [GitTaskBench](https://github.com/git-task-bench/GitTaskBench)
- [Terminal-Bench](https://github.com/laude-institute/terminal-bench)

We thank the authors and contributors of these projects for their valuable work.

MARS_preview

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README