VISTA_Bench

KO | EN

gitlite — search

#bash #golang #claude-code #nlp #command-line #privacy #shell #react #tensorflow #pdf

(★ 9)

No description available.

파일 탐색기

ZIP 다운로드

파일 수가 많아 일부만 표시됩니다. 전체 파일은 위 다운로드 버튼으로 확인해 주세요.

- leaderboard.png
- - .DS_Store
  - 10_streaming_music-streaming_anchors.cleaned.json
  - 10_streaming_music-streaming_anchors.json
  - 10_streaming_music-streaming_anchors.raw.json
  - description.md
  - manifest.json
  - sitemap.md
- - .DS_Store
  - 1_newsletter_anchors.cleaned.json
  - 1_newsletter_anchors.json
  - 1_newsletter_anchors.raw.json
  - description.md
  - manifest.json
  - sitemap.md
- - .DS_Store
  - 2_real-estate_anchors.cleaned.json
  - 2_real-estate_anchors.json
  - 2_real-estate_anchors.raw.json
  - description.md
  - interaction_index.json
  - interaction_index.verified.json
  - manifest.json
  - sitemap.md
- - .DS_Store
  - 3_job-board_anchors.cleaned.json
  - 3_job-board_anchors.json
  - 3_job-board_anchors.raw.json
  - description.md
  - manifest.json
  - sitemap.md
- - .DS_Store
  - 4_forum_anchors.cleaned.json
  - 4_forum_anchors.json
  - 4_forum_anchors.raw.json
  - description.md
  - interaction_index.json
  - interaction_index.verified.json
  - manifest.json
  - sitemap.md
- - .DS_Store
  - 5_travel-booking_anchors.cleaned.json
  - 5_travel-booking_anchors.json
  - 5_travel-booking_anchors.raw.json
  - description.md
  - interaction_index.json
  - interaction_index.verified.json
  - manifest.json
  - sitemap.md
- - .DS_Store
  - 6_chat_anchors.json
  - 6_chat_anchors.raw.json
  - description.md
  - manifest.json
  - sitemap.md
- - .DS_Store
  - 7_cloud-storage_anchors.json
  - 7_cloud-storage_anchors.raw.json
  - description.md
  - interaction_index.json
  - interaction_index.verified.json
  - manifest.json
  - sitemap.md
- - .DS_Store
  - 8_ecommerce_anchors.json
  - 8_ecommerce_anchors.raw.json
  - description.md
  - interaction_index.json
  - interaction_index.verified.json
  - interaction_summary.md
  - manifest.json
  - sitemap.md
- - .DS_Store
  - 9_project-management_anchors.json
  - 9_project-management_anchors.raw.json
  - description.md
  - interaction_index.json
  - interaction_index.verified.json
  - manifest.json
  - sitemap.md
- - .DS_Store
- - .DS_Store
- - .DS_Store
- - analyze_codex_run.py
  - analyze_edits.py
  - analyze_eval_results.py
  - analyze_run.py
  - analyze_task_workflows.py
  - analyze_workflow_taxonomy_v2.py
  - anchor_marker.html
  - annotate_pages.py
  - build_c4_from_c3.py
  - build_interaction_index.py
  - build_interaction_summary.py
  - build_scaffold_cache.py
  - camel_runner.py
  - camel_runner_single.py
  - camel_trace_viewer.py
  - camel_usage_report.py
  - clean_anchor_json.py
  - clip_eval.py
  - cursor_3x_report.py
  - cursor_finalize.py
  - discover_routes.py
  - eval_run.py
  - eval_run_vlm.py
  - insert_anchor_tags.py
  - match_pages.py
  - page_type_marker.html
  - page_types_preannotated.json
  - plot_action_sequences.py
  - plot_new_results.py
  - plot_workflow_taxonomy_v2.py
  - preload_scaffolds.py
  - repair_manifest.py
  - repurpose_c2_from_scratch.py
  - sync_descriptions.py
  - tee_jsonl.py
  - vision_verify_index.py
- _run_deepseek_v4pro.sh
- agent_system_promt_c0.md
- agent_system_promt_c1.md
- agent_system_promt_c1_preload.md
- agent_system_promt_c2.md
- agent_system_promt_c3.md
- agent_system_promt_c3_preload.md
- agent_system_promt_c4.md
- collect_trajectories.sh
- eval_all_r1.sh
- eval_all_r2.sh
- eval_all_runs.sh
- run_all.sh
- run_eval.sh
- run_eval_antigravity.sh
- run_eval_claude.sh
- run_eval_codex.sh
- run_eval_cursor.sh
- run_eval_gemini.sh
.gitignore
README.md

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

파일 목록이 많아 일부만 표시됩니다.

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**VISTA** ranks LLM coding agents on **end-to-end web-app generation from visual specs**: each task gives an agent a product's design and asks it to build a *runnable* full-stack app. We score how faithfully the result matches the spec — not just visually, but behaviorally — by matching every human-annotated UI anchor to a live DOM element and checking that it's both **placed correctly** and **actually works**.

---

## 🏆 Leaderboard — Combined Score (condition C4)

<a href="https://vista-benchmark.org/">Explore the latest rankings, costs, speed, trajectories, and replays →</a>

---

## How it's scored

```
visual spec ─▶ agent (model × harness) ─▶ runnable app ─▶ per-anchor DOM match ─▶ S
```

For each app the spec carries **critical UI anchors** annotated by humans. The evaluator brings the agent's app up under Docker, then for every anchor:

- **Localization (L)** — is the right element present, in the right place? (DOM match + bounding-box IoU/distance.)
- **Behavior (B)** — does it *do* the right thing? (Click navigates, input accepts, toggle flips, dialog opens…)

The per-anchor score is **L × B**, and the app's score is the mean over its critical anchors. **Combined score S** is the mean across all 10 apps (a missing or broken element scores 0). Because `S = L × B`, an app that looks right but doesn't work scores near zero — behavior is usually the bottleneck.

### Condition C4

The agent gets the **richest spec** and the **freest hand**: the page's **rendered Figma image** (a screenshot mockup) **and** its **pruned Figma structure** (the layout tree as JSON) — but **no target framework**. It picks its own stack.

---

## Running the benchmark

All runners live in [`tasks/`](tasks/). **[`run_eval.sh`](tasks/run_eval.sh)** is the base runner — it launches **one agent against one task/variant**, materializes a run dir (`inputs/`, `workspace/`, system prompt, `meta.json`, event logs), and runs post-hoc analysis. The four `run_eval_<cli>.sh` scripts are thin wrappers that just fix `--cli`.

### Prerequisites

- **Docker** running (each app is built + brought up on fixed host ports `38000`/`38001`).
- The chosen agent CLI **installed and authenticated** (see per-CLI notes below).

### One run — base script

```bash
cd tasks

# ./run_eval.sh --task <task> --variant <c0|c1|c2|c3|c4> [--cli <cli>] [--model <m>]
./run_eval.sh --task 1_newsletter --variant c4 # default --cli claude
./run_eval.sh --task 4_forum --variant c4 --cli codex
```

| Flag | Meaning |
|------|---------|
| `--task` | task folder name, e.g. `1_newsletter`, `4_forum`, `7_cloud-storage` |
| `--variant` | spec condition: `c0`–`c4` (the leaderboard uses `c4`) |
| `--cli` | `claude` (default) · `codex` · `gemini` · `antigravity` · `cursor` |
| `--model` | model slug (CLI-specific default if omitted) |
| `--preload` | pre-seed the workspace with a scaffold (`c1`/`c3` only) |
| `--skill` | tell the agent it may freely use installed skills |

### One run — per-CLI wrappers

Each wrapper is `run_eval.sh` with `--cli` pinned, so it takes the same flags:

```bash
cd tasks

./run_eval_claude.sh --task 1_newsletter --variant c4                        # → Claude Code   (default model: sonnet)
./run_eval_codex.sh  --task 1_newsletter --variant c4 --model gpt-5.5        # → OpenAI Codex  (default: gpt-5-codex)
./run_eval_gemini.sh --task 1_newsletter --variant c4 --model gemini-2.5-pro # → Gemini CLI    (default: gemini-2.5-pro)
./run_eval_cursor.sh --task 1_newsletter --variant c4                        # → cursor-agent  (default model: composer-2.5)
```

| Wrapper | Agent CLI | Install / auth |
|---------|-----------|----------------|
| [`run_eval_claude.sh`](tasks/run_eval_claude.sh) | Claude Code | `claude /login` (Max-plan token) |
| [`run_eval_codex.sh`](tasks/run_eval_codex.sh) | OpenAI Codex | `codex /login` (OAuth) **or** `OPENAI_API_KEY` |
| [`run_eval_gemini.sh`](tasks/run_eval_gemini.sh) | Gemini CLI (`@google/gemini-cli ≥ 0.11`) | run `gemini` once (Google OAuth) **or** `GEMINI_API_KEY` |
| [`run_eval_cursor.sh`](tasks/run_eval_cursor.sh) | Cursor (`cursor-agent`) | `curl https://cursor.com/install -fsS \| bash`, then `cursor-agent login` |

### Pinning the harness version (reproducibility)

The harness matters as much as the model, so point each runner at an **exact** pinned binary via its `*_BIN` env var (recorded into `meta.json`):

```bash
CLAUDE_BIN=~/.claude-pinned/2.1.152/node_modules/.bin/claude ./run_eval_claude.sh --task 4_forum --variant c4
CODEX_BIN=~/.codex-pinned/0.134/node_modules/.bin/codex ./run_eval_codex.sh --task 4_forum --variant c4
CURSOR_BIN=~/.local/share/cursor-agent/versions/<ver>/cursor-agent ./run_eval_cursor.sh --task 4_forum --variant c4
```

> `cursor-agent` has no built-in run timeout and can hang after finishing; `run_eval.sh` wraps it with an inactivity watchdog (`CURSOR_IDLE_KILL`, default 180s) and a hard cap (`CURSOR_TIMEOUT`, default 45m).

### Batch + scoring

```bash
./run_all.sh --variant c4 --cli cursor --skip-existing # run every task for one agent
FILTER='*c4*' ./eval_all_runs.sh "$PWD/_runs_<config>" # build + bring up + score each run dir → eval_summary.csv
```

## Trajectories

The `trajectories/` directory contains agent trajectories recorded during benchmark runs. These files preserve the model-generated outputs produced throughout execution, including intermediate reasoning or status messages exposed by the agent CLI, tool calls, code-generation steps, command outputs, and final responses.

You can use `tasks/collect_trajectories.sh` to collect trajectories from your benchmark runs. Contributions are welcome—feel free to submit a merge request (MR) to upload and share your collected trajectories.

## Citation

```bibtex
@misc{guo2026vistaendtoendbenchmarkvisual,
      title={VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents},
      author={JunJia Guo and Yuhang Yao and Jiawei and Zhou and Jingdi Chen},
      year={2026},
      eprint={2605.26144},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2605.26144},
}
```

VISTA_Bench

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Download README