swebench-harness

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #llm #android #machine-learning #deep-learning #awesome-list #pytorch #security #ios #openai #open-source

swebench-harness

(★ 15)

Coding agent harness, achieved SWE Bench Lite resolve rate of 52.4%

swebench-harness 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**Run #1: Achieved a 52.4% resolve rate on SWE-bench Lite, using Claude Haiku 4.5 for context gathering and Sonnet 4.6 for implementation.**

<h3>For-Loop Subsessions &amp; Tool Calls</h3>

<p>Agent subsessions are for-loops, with each iteration calling an LLM API to generate a <code>json</code> file that specifies the tool call the agent wants to execute and the parameters for it</p>

<ol>
  <li>An LLM can only produce text, so for each tool call, we have it generate a string-format <code>json</code> file, then use <code>json.loads()</code> to convert it from string to dict
    <ul class="circle" type="circle">
      <li>In the LLM prompt, we inject prior conversation and tool call history</li>
      <li>We also provide descriptions of each available tool in <code>tools=json_schema</code>
        <ul class="square" type="square">
          <li><code>json_schema</code> contains function name, expected params, and summary</li>
          <li>All available tools are stored in <code>registry.tools</code> map, loaded at start up keyed by tool name to <code>ToolSpec</code> dataclass</li>
        </ul>
      </li>
      <li>Make sure <code>temperature=0</code> for benchmark determinism</li>
    </ul>
  </li>
  <li>From the response object, we check if the <code>finish_reason</code> property equals “tool_calls”, if so, our single-sample completion request will produce <code>response.choices[0].message.tool_calls</code> — a list of tool call objects to loop over
    <ul class="disc" type="disc">
      <li>For each <code>tc</code> in <code>tool_calls</code>, we can find the right function with <code>tc.function.name</code> and the right inputs with <code>tc.function.arguments</code>, a json-encoded string</li>
    </ul>
  </li>
  <li>Use <code>tc.function.name</code> as a key for <code>registry_tools</code> map, fetching <code>ToolSpec</code> object for that tool call, which has the python function as a property
    <ul class="disc" type="disc">
      <li>The code of the tool call is run by <code>spec.fn(args)</code></li>
      <li>The actual function itself is either <code>subprocess.run(cmd, shell=True)</code> for bash, or <code>open(path, “w\”).write()</code> for write, and similar process for read</li>
    </ul>
  </li>
  <li>Collect the results of the tool call as a string, then append to the message history
    <ul class="disc" type="disc">
      <li>We organize each tool call and corresponding result string by an ID number</li>
      <li>Continue repeating loop until model responds with plain text instead of tool call</li>
      <li>At loop turn=N, we can create logic to add/modify the system prompt to encourage specific behavior with the tool calls</li>
    </ul>
  </li>
</ol>

<h3>The Node Graph Structure</h3>

<p>Each node on the graph is a Node class that, after executing, returns another Node class that represents the next node in the sequence to traverse to. The node traversal is purely deterministic based on if-statements and implementations/test-results produced by LLM.</p>

<ol>
  <li>Build <code>AgentState</code> dataclass with ~40 fields of metadata (task info, retry counters, current node name, etc) – the information it stores triggers which node to be ran and with what data, it also acts as a place for nodes to hand off information to one another
    <ul class="circle" type="circle">
      <li>Examples include tracking to-do lists of tests to be ran, gathered context bundles after reading relevant files, failure-type flags</li>
      <li><code>AgentState</code> is accompanied with serialization functions <code>to_dict/from_dict</code> so it can be persistently stored via SQLite</li>
    </ul>
  </li>
  <li>We use <code>state.current_node</code> as a string key into a node registry, a dict mapping name to node class, then <code>node.run()</code> triggers a subsession or deterministic result
    <ul class="disc" type="disc">
      <li><code>node.run(state)</code> outputs <code>NodeResult(next_node, state_update)</code> where <code>state_update</code> is a dict of <code>{field_name: new_value}</code></li>
      <li>We then iterate over each entry to update <code>AgentState</code> via <code>state.apply_update()</code></li>
    </ul>
  </li>
  <li>In <code>BlueprintEngine</code>, we loop through running each node, gathering results, and switching to the next node until <code>state.current_node</code> equals <code>“END”</code>
    <ul class="disc" type="disc">
      <li><code>next_node</code> is outputted deterministically inside the node's own code, often based on the outcome of running tests or code the LLM wrote, but NOT an LLM's decision of whether it's done.</li>
      <li>Example: Verify_Node routes to Implement_Node if F2P tests are still failing, but routes to Gather_Context_Node if P2P tests regressed</li>
    </ul>
  </li>
</ol>

<h3>Harness Design</h3>

<ol>
  <li><strong><code>LOAD_TASK</code></strong>: read issue text and pass_to_pass tests, store in <code>state.task_text</code> and <code>state.pass_to_pass</code>, automatically set <code>next_node</code> to reproduce issue, we store but never use <code>state.fail_to_pass</code> until the very last official swe-bench eval</li>
  <li><strong><code>REPRODUCE_ISSUE</code></strong>:
    <ul class="disc" type="disc">
      <li>One LLM call takes issue text and creates list of discrepancies between observed and expected behavior, creates <code>{observed, expected}</code> pairs</li>
      <li>70-turn subsession is launched to write one test case to objectively express each discrepancy, forming <code>_repro_tests.py</code> — tool set: bash, read, write
        <ul class="circle" type="circle">
          <li>At turn=0, prompt encourages tool calls to explore issue relevant files</li>
          <li>At turn=8, add a nudging statement to the prompt to encourage writing failing tests for each discrepancy</li>
          <li>At turn=25, make grep call for each written test, searching for any sibling functions or call-sites that also need fixing</li>
          <li>At turn=50, explicitly run pytest and confirm every written test actually fails</li>
          <li>At turn=60, nudge to consider any edge cases unaccounted for</li>
        </ul>
      </li>
    </ul>
  </li>
  <li><strong><code>GATHER_CONTEXT</code></strong>:
    <ul class="disc" type="disc">
      <li>Parse traceback text from failing repro tests for relevant functions and file-paths using regexes for python traceback frames, pytest format, and assert lines</li>
      <li>These file-paths are seeded into a read-queue, <code>GatherQueueState</code>: queue (list), seen (set), notes (list), current (str|None), and line-numbers from the call-trace</li>
      <li>22-turn subsession is launched to dequeue these files and take notes on each one with an LLM call, if an adjacent relevant file is spotted, it is enqueued
        <ul class="circle" type="circle">
          <li>Uses a new tool set: enqueue, dequeue, note, bash</li>
        </ul>
      </li>
      <li>on_finish_check blocks make sure that queue is empty before stopping, and that notes are not concentrated solely on the _repro_tests file</li>
      <li>After the subsession, notes are grouped by file-path, "not relevant" ones dropped, the rest are organized into <code>ContextBundle.task_adjacent_files</code></li>
      <li><code>ContextBundle</code> is a dataclass and <code>task_adjacent_files</code> is <code>list[dict]</code>, a dict for each file the agent took noted – {“filepath”: … “why”: … “notes_content”: …}</li>
      <li>The dicts are grouped by file-paths and sorted by line number for easy readability</li>
    </ul>
  </li>
  <li><strong><code>IMPLEMENT</code></strong>:
    <ul class="disc" type="disc">
      <li>30-turn subsession is launched to resolve each failing test case given the context we’ve gathered in the previous step – tools: read, write, bash, replace</li>
      <li>Once turns exceed 20, we call an LLM to compress all conversation history into a summary in the <code>messages</code> list except for the latest 6 turns, which stay verbatim</li>
      <li>If Verify step fails, the Implement may get triggered up to 5 times total
        <ul class="circle" type="circle">
          <li>1st attempt: system prompt includes all failing tests and tracebacks</li>
          <li>later attempts: current failing tests, regressed p2p tests, current git diff</li>
        </ul>
      </li>
      <li>Agent can also write its own scripts to reduce repetitive work or test suspicions
        <ul class="circle" type="circle">
          <li>we also provide premade helper script <code>mro_check.py</code> which dumps a python class’s full method-resolution-order chain</li>
          <li><code>import_graph.py</code> is another premade helper that shows all dependencies a given file imports, as well as what other files depend on it (converse)</li>
        </ul>
      </li>
      <li>Once subsession finishes (iteration ends without a tool call), we go to Verify</li>
    </ul>
  </li>
  <li><strong><code>VERIFY</code></strong>:
    <ul class="disc" type="disc">
      <li>We run every failing test that we wrote in Reproduce Issue, as well as all the baseline pass_to_pass tests individually – for each test, we store the result and traceback to <code>state.todo_list</code>, which tracks our entire list of tests to pass</li>
      <li>If we encounter a p2p test regression, we traverse back to Gather Context step to recollect information about relevant files on this new issue</li>
      <li>If not all failing tests we wrote are passing, we traverse back to Implement step for the subsession to try again, with an extra nudge on currently failing cases</li>
      <li>If all tests pass or max verify attempts exceeded, we traverse to the very last step, which is starting a separate container to run the official swebench eval</li>
      <li>If tests are still failing after max attempts, we track which retry attempt yielded the most number of tests passing and submit that one for official evaluation</li>
      <li>Verify step is purely deterministic code</li>
    </ul>
  </li>
</ol>

<h3>Running SWE-Bench Evals</h3>

<ul class="disc" type="disc">
  <li>SWE-Bench Lite spans 12 unique repos (astropy, django, flask, matplotlib…) across 300 instances, but each instance also pins a specific repo version, so the real unit of environment is (repo, version), not just repo, giving 64 distinct environments</li>
  <li>We pre-build all 64 envs once via <code>setup_scripts/setup_env_cache.sh</code>, store each as its own snapshot in Modal Volume swebench-envs-v2
    <ul class="circle" type="circle">
      <li>This way, every instance run just mounts the matching (repo, version) snapshot instead of resolving deps fresh</li>
      <li>This guarantees the exact same interpreter/compiler + exact same dependency versions every run, for every instance sharing that (repo, version) pair</li>
    </ul>
  </li>
  <li>Retrieval is keyed off each instance's declared (repo, version) at <code>LOAD_TASK</code> time, used to pull the right cached snapshot out of the volume for that instance's container</li>
  <li>Each instance run is a Modal container that mounts the volume, does its work, then spins up a nested container for official eval scoring
    <ul class="circle" type="circle">
      <li>Single instance: <code>modal run modal_run.py::run_one --instance-id &lt;id&gt;</code><br><code>[--provider anthropic]</code></li>
      <li>Subset by repo: <code>modal run modal_run.py::run_repo --repo sympy/sympy</code><br><code>--limit 50 --concurrency 10</code></li>
      <li>Full 300-instance suite: <code>modal run modal_run.py::run_batch --ids-file</code><br><code>ids.txt --results-file test_traces/results_full300.jsonl</code></li>
    </ul>
  </li>
  <li><code>--concurrency</code> controls how many instances run in parallel (each holding ~2 containers, per the nested-eval-container point above), default 10 assumes an OpenAI Tier 2 key; lower-tier keys should drop to 4 or less to avoid rate-limit 429s</li>
</ul>

<h3>Citations</h3>

<p>This project builds on SWE-bench and SWE-bench Lite. SWE-bench Lite is a 300-instance curated subset of the original benchmark introduced by the same authors (no separate paper), so both are cited via the original SWE-bench paper:</p>

<pre>@inproceedings{
    jimenez2024swebench,
    title={{SWE}-bench: Can Language Models Resolve Real-world Github
Issues?},
    author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu
Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
    booktitle={The Twelfth International Conference on Learning
Representations},
    year={2024},
    url={https://openreview.net/forum?id=VTF8yNQM66}
}</pre>

<hr>

<h3>Additional References Consulted</h3>

<ul class="disc" type="disc">
  <li>opencode</li>
  <li>openharness <a href="https://github.com/HKUDS/OpenHarness">https://github.com/HKUDS/OpenHarness</a></li>
  <li>openautocoder/live-swe-agent</li>
  <li>openautocoder/agentless</li>
  <li>walkie-talkie (<a href="https://github.com/xyuzh/walkie-talkie">https://github.com/xyuzh/walkie-talkie</a>)</li>
  <li>claude code behavior: read/bash/edit (we could stick with this but direct it to Reproduce)</li>
  <li>disecting swe bench leaderboard: <a href="https://arxiv.org/pdf/2506.17208v1">https://arxiv.org/pdf/2506.17208v1</a></li>
  <li>for formatting repo: <a href="https://github.com/SWE-agent/mini-swe-agent">https://github.com/SWE-agent/mini-swe-agent</a></li>
</ul>

<hr>

<h3>Setup Instructions</h3>

<pre>git clone &lt;your-repo-url&gt; multitask-oss &amp;&amp; cd multitask-oss
pip install -e ".[dev,swebench]"
modal setup                                              # auth (browser login)
modal secret create openai-keys OPENAI_API_KEY=sk-...
modal secret create modal-token MODAL_TOKEN_ID=&lt;id&gt; MODAL_TOKEN_SECRET=&lt;secret&gt;   # from ~/.modal.toml
bash setup_scripts/setup_env_cache.sh                     # builds/caches all 63 envs into Modal Volume "swebench-envs-v2"</pre>

<p>Wait for that script to finish (~5-10 min) before running anything else — it's what makes every repo's VERIFY signal correct.</p>

<p><strong>Generate the instance list</strong></p>

<pre>python3 -c "
from datasets import load_dataset
ds = load_dataset('princeton-nlp/SWE-bench_Lite', split='test')
open('ids.txt', 'w').write('\n'.join(r['instance_id'] f
"</pre>

<pre>modal run modal_run.py::run_batch --ids-file ids.txt --concurrency 4 --results-file test_traces/results_full300.jsonl</pre>

<p><strong>Why <code>--concurrency 4</code>, not the default 10</strong></p>

<p>Each in-flight instance actually holds two concurrent Modal containers at once — <code>run_instance</code> itself, plus the nested container the officieval spins up (<code>evaluate_patch(..., --modal true)</code>) to grans ~8 containers in flight, <code>--concurrency 10</code> (thedefault) means ~20 — comfortably under Modal's Starter-plan container cap, but the real constraint on a free plan is usually OpenAI's own rate limit for a lower-tier API key, which is what --coing (<code>modal_run.py</code>'s own comment: default 10 is "matchedto OpenAI Tier 2 headroom" — a free/low-tier key should go lower). If he still hits 429s at 4, drop to 2; if it's smooth, he can nudge up.</p>

<p>Progress streams live and writes to <code>test_traces/results_full300.jsonl</code> incrementally, so it's safe to Ctrl-C and resume-by-diffing <code>ids.txt</code> against what's already in the results file if needed.</p>

</div>

swebench-harness

swebench-harness 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

swebench-harness

swebench-harness 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README