paper-lantern-challenges

KO | EN

gitlite — search

#javascript #python #typescript #llm #rust #deep-learning #linux #awesome #vue #awesome-list #security #pytorch #mcp #open-source

paper-lantern-challenges

(★ 9)

No description available.

paper-lantern-challenges Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# <img src="paper_lantern_logo_no_name.png" alt="" width="32"> Paper Lantern Challenges

Your coding agent makes technical decisions. **Paper Lantern reads hundreds of research papers per decision and turns them into implementation guidance.**

We ran 9 everyday tasks - same agent, same model, same data. The only difference: one agent could call Paper Lantern before building its solution.

> **[Test generation](experiments/test_generation/) (+39% mutation score)**  
> Writing tests that actually catch bugs. Paper Lantern found a technique that targets specific mutation points instead of guessing at coverage ([MuTAP](https://arxiv.org/abs/2308.16557))

> **[Code review](experiments/code_review/) (+13% F1)**  
> Finding real issues in PRs, not hallucinated ones. Paper Lantern surfaced a multi-pass strategy - real bugs show up every time, false positives don't ([Multi-Review Aggregation](https://arxiv.org/abs/2509.01494))

> **[Text-to-SQL](experiments/text_to_sql/) (+6% accuracy)**  
> Generating SQL that returns the right answer. Paper Lantern surfaced 5 candidate approaches; the winner was self-consistency voting ([SQL-PaLM](https://arxiv.org/abs/2306.00739))

> **[Document extraction](experiments/document_extraction/) (+72% F1)**  
> Pulling structured data from long documents. Paper Lantern found a retrieve-then-validate pipeline from papers published weeks before the experiment ([BEAVER](https://arxiv.org/abs/2603.19635), [PAVE](https://arxiv.org/abs/2603.20673))

## Try it

```
npx paperlantern@latest
```

Setup guide and docs at [paperlantern.ai/code](https://paperlantern.ai/code)

## All results

| Task | Baseline | With PL | Delta | What PL surfaced |
|---|---|---|---|---|
| [Document extraction](experiments/document_extraction/) | 0.444 | **0.764** | +72% | [BEAVER](https://arxiv.org/abs/2603.19635) section selection + [PAVE](https://arxiv.org/abs/2603.20673) validation |
| [PDF extraction](experiments/pdf_extraction/) | 0.318 | **0.572** | +80% | Section-level decomposition + self-verification ([PARSE](https://arxiv.org/abs/2510.08623), [Deep Reflective Reasoning](https://arxiv.org/abs/2603.20435)) |
| [Test generation](experiments/test_generation/) | 0.625 | **0.870** | +39% | Mutation-aware prompting via AST analysis ([MuTAP](https://arxiv.org/abs/2308.16557), [MUTGEN](https://arxiv.org/abs/2506.02954)) |
| [Text classification](experiments/text_classification/) | 0.505 | **0.666** | +32% | Retrieval-first classification ([Retrieval-ICL](https://arxiv.org/abs/2406.17534), [LLM-Select-P](https://arxiv.org/abs/2412.05137)) |
| [Prompt examples](experiments/prompt_examples/) | 0.193 | **0.324** | +68% | MMR diversity + hierarchical parent-then-child prompting |
| [Text-to-SQL](experiments/text_to_sql/) | 0.650 | **0.690** | +6% | Self-consistency voting ([SQL-PaLM](https://arxiv.org/abs/2306.00739), [MCS-SQL](https://arxiv.org/abs/2405.07467)) |
| [Code review](experiments/code_review/) | 0.351 | **0.395** | +13% | Consensus aggregation - 3 passes, majority vote ([Multi-Review Aggregation](https://arxiv.org/abs/2509.01494)) |
| [LLM routing](experiments/llm_routing/) | 0.744 | **0.761** | +2% | Cross-validated model selection ([CARGO](https://arxiv.org/abs/2509.14899)) |
| [LLM-as-judge](experiments/llm_as_judge/) | 0.623 | **0.633** | +2% | Dimension-specific multi-pass evaluation ([HypoEval](https://arxiv.org/abs/2504.07174), [PAIRS](https://arxiv.org/abs/2403.16950)) |

The biggest gains came when Paper Lantern surfaced an architecturally different approach - not a better prompt, but a different way to structure the solution. The smaller gains came on tasks where the baseline was already structurally sound. We're showing all 9 because that's the honest picture.

## How the experiments work

The baseline and `with_pl` prompts are identical except for one section - `baseline` says "use your own knowledge," `with_pl` says "use Paper Lantern MCP tools to research first." Same agent, same model ([Gemini Flash 3](https://ai.google.dev/)), same data, same evaluation script. Everything else downstream was the agent's doing.

Every prompt, every line of agent code, every prediction is in this repo. Diff them yourself:

```bash
diff experiments/test_generation/prompts/baseline.md experiments/test_generation/prompts/with_pl.md
```

## Run it yourself

Results are already committed - no API keys needed to browse. But if you want to verify:

```bash
git clone https://github.com/paperlantern-ai/paper-lantern-challenges.git
cd paper-lantern-challenges
uv sync

uv run python run.py setup test_generation
uv run python run.py run test_generation baseline
uv run python run.py run test_generation with_pl
uv run python run.py compare test_generation
```

<details>
<summary>Prerequisites</summary>

- **[uv](https://docs.astral.sh/uv/)** - `curl -LsSf https://astral.sh/uv/install.sh | sh`
- **[Claude Code](https://docs.anthropic.com/en/docs/claude-code)** - or any coding agent
- **Gemini API key** - `export GEMINI_API_KEY=your_key`
- **Paper Lantern** (for `with_pl` runs) - [setup instructions](https://paperlantern.ai/code)

</details>

<details>
<summary>CLI reference</summary>

```bash
uv run python run.py list                                          # Show available challenges
uv run python run.py setup <challenge>                             # Download data
uv run python run.py run <challenge> <submission>                  # Run + evaluate
uv run python run.py run <challenge> <submission> --dataset cfpb   # Run one dataset
uv run python run.py eval <challenge> <submission>                 # Evaluate only
uv run python run.py compare <challenge>                           # Side-by-side comparison
```

</details>

<details>
<summary>Repo structure</summary>

```
experiments/<challenge>/
├── CHALLENGE.md          # Problem description
├── evaluate.py           # Scoring script
├── setup.py              # Data download
├── starter/              # Starter code
├── prompts/
│   ├── baseline.md       # Exact prompt we gave the baseline agent
│   └── with_pl.md        # Exact prompt we gave the PL agent
├── data/                 # Challenge data (after setup)
└── submissions/
    ├── baseline/
    │   ├── *.py              # Code the agent wrote
    │   ├── predictions.jsonl
    │   └── approach.md       # Strategy, decisions, reasoning
    └── with_pl/
        ├── *.py
        ├── predictions.jsonl
        └── approach.md       # Strategy, decisions, papers cited
```

</details>

## License

MIT

paper-lantern-challenges

paper-lantern-challenges Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

paper-lantern-challenges

paper-lantern-challenges Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README