scbench - GitLite

KO | EN

gitlite — search

#api #computer-vision #tensorflow #rag #security #windows #claude #claude-code #automation #list

scbench

(★ 29)

Benchmark for agentic single cell data analysis

파일 탐색기

ZIP 다운로드

- adding_evals.md
- CLI_USAGE.md
- specification.md
- - T04a_endothelin_niche_sources.json
- - tapestri_ccus_clustering_12_largest_mutant_clone.json
- - DE01_pseudobulk_de.json
- - dr_05_pca_preprocessing_sentinels.json
- - NRM01_sparse_normalization.json
- - bd_rhapsody_tnbc_panel_aware_qc.json
- manifest.json
- category_results.json
- model_results.csv
- model_steps.csv
- platform_results.json
- __init__.py
- cli.py
- types.py
.gitignore
CONTRIBUTING.md
LICENSE
pyproject.toml
README.md
uv.lock

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

명령어 용어집

이 문서에서 사용된 명령어를 모아봤습니다. 낯선 명령어가 있다면 펼쳐서 확인해보세요.

🔍

`pip install`

설명 보기 ▼

Install Python packages.

pip install {{package1 package2 ...}}

Install one or more packages:

pip install {{package1 package2 ...}} {{[-U|--upgrade]}}

Upgrade all specified packages to the latest version, installing any that are not already present:

pip install {{package}}=={{version}}

Install a specific version of a package:

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**Evaluating AI Agents on Single-Cell RNA-seq Analysis**

scBench is a benchmark of 195 verifiable problems derived from practical single-cell RNA-seq workflows. Each problem pairs a data snapshot (AnnData `.h5ad`) with a natural-language task prompt and a deterministic grader that maps the agent's structured output to pass/fail.

## Key Findings

| model_name | harness | Accuracy (%) | Cost ($) |
|---|---|---:|---:|
| claude-opus-4-8 | claude-code | 57.95 | 1.4792 |
| gpt-5.5 | mini-swe-agent | 57.95 | 1.1136 |
| gpt-5.5 | openai-codex | 57.78 | 2.4685 |
| claude-opus-4-8 | mini-swe-agent | 57.44 | 1.7000 |
| gpt-5.4 | mini-swe-agent | 57.44 | 0.8240 |
| gemini-3.5-flash | pi | 56.92 | 1.7708 |
| claude-opus-4-7 | mini-swe-agent | 55.21 | 1.5378 |
| claude-opus-4-7 | claude-code | 54.02 | 1.1465 |
| gemini-3.1-pro-preview | mini-swe-agent | 53.85 | 0.8948 |
| claude-opus-4-6 | mini-swe-agent | 52.65 | 1.1917 |
| gpt-5.2 | mini-swe-agent | 52.31 | 0.8874 |
| claude-sonnet-4-6 | mini-swe-agent | 50.26 | 0.9872 |
| claude-opus-4-5 | mini-swe-agent | 47.18 | 0.6553 |
| grok-4.20-beta-0309-reasoning | mini-swe-agent | 44.44 | 0.2957 |
| grok-4.3 | mini-swe-agent | 44.27 | 0.2147 |
| gpt-5.1 | mini-swe-agent | 38.80 | 0.2177 |
| claude-sonnet-4-5 | mini-swe-agent | 33.16 | 0.2682 |
| grok-4-1-fast-reasoning | mini-swe-agent | 30.26 | 0.0282 |
| gemini-2.5-pro | mini-swe-agent | 23.59 | 0.1368 |

Full results with per-task and per-platform breakdowns are in [`results/`](results/).

## Benchmark Structure

195 evaluations across:
- **6 platforms**: BD Rhapsody, Chromium, CSGenetics, Illumina, MissionBio, ParseBio
- **6 task categories**: QC, Normalization, Dimensionality Reduction, Clustering, Cell Typing, Differential Expression

Tasks require empirical interaction with the agents that rely on prior knowledge without performing the requisite analysis fail.

## Canonical Examples

Six canonical examples are in [`evals/`](evals/). The sample covers all current platforms and task categories. The full 195-evaluation benchmark is withheld to prevent training contamination.

| Task | Platform | Eval |
|---|---|---|
| QC | BD Rhapsody | `bd_rhapsody_tnbc_panel_aware_qc` |
| Dimensionality Reduction | Chromium | `dr_05_pca_preprocessing_sentinels` |
| Normalization | CS Genetics | `NRM01_sparse_normalization` |
| Cell Typing | Illumina snRNA | `T04a_endothelin_niche_sources` |
| Clustering | Mission Bio Tapestri | `tapestri_ccus_clustering_12_largest_mutant_clone` |
| Differential Expression | Parse Bio | `DE01_pseudobulk_de` |

## Quick Start

```bash
pip install -e .

# Validate an evaluation
scbench validate evals/qc/bd_rhapsody_tnbc_panel_aware_qc.json

# Run with mini-swe-agent
export ANTHROPIC_API_KEY=your_key
scbench run evals/qc/bd_rhapsody_tnbc_panel_aware_qc.json --agent minisweagent --model anthropic/claude-opus-4-5
```

### Custom Agent

```python
from scbench import EvalRunner

def my_agent(task_prompt, work_dir):
    import json
    answer = {
        "n_pbmcs_retained": 14346,
        "median_genes_per_pbmc": 68,
        "n_monocytes_pbmc": 2592,
    }
    (work_dir / "eval_answer.json").write_text(json.dumps(answer))
    return answer

runner = EvalRunner("evals/qc/bd_rhapsody_tnbc_panel_aware_qc.json")
result = runner.run(agent_function=my_agent)
print(f"Passed: {result['passed']}")
```

## Graders

Five grader families handle different answer types:

| Grader | Use Case |
|--------|----------|
| NumericTolerance | QC metrics, counts, expression values |
| MultipleChoice | Discrete interpretation questions |
| MarkerGenePrecisionRecall | Gene lists (P@K, R@K) |
| LabelSetJaccard | Cell type sets |
| DistributionComparison | Cell type proportions |

See [latch-eval-tools](https://github.com/latchbio/latch-eval-tools) for implementations.

## Citation

```bibtex
@article{scbench2026,
  title={scBench: Evaluating AI Agents on Single-Cell RNA-seq Analysis},
  author={Workman, Kenny and Yang, Zhen and Muralidharan, Harihara and Abdulali, Aidan and Le, Hannah},
  year={2026},
  note={LatchBio}
}
```

## License

Apache 2.0