Search-SWE

Run Search-SWE

Full guide

Run Search-SWE from your terminal or through your coding agent. Both use the same evaluation pipeline.

CLI

Configure and launch an evaluation from your terminal.

CLI reference

  1. Install the CLI

    Install the CLI in your Python environment:

    pip install search-swe

  2. Choose your setting and tasks

    search-swe run

    The wizard finds your checkout or helps you clone one to ~/Search-SWE or a custom path, then remembers its location.

    Choose Codex, Claude Code or Pi with OpenRouter or a supported direct provider. Select the model and reasoning effort, enter missing keys, then select tasks and configure their services.

    Default setting: Codex + OpenRouter + GPT-6 Luna (max).

    See supported agents and providers. For Pi, search-swe pi-models lists providers and their model counts; add --provider NAME for exact model IDs.

  3. Review resources and start

    Review the task and cache locations, disk-space estimate and resource requirements. Choose sequential or parallel execution, then confirm to download and verify assets, allocate resources and launch the run.

    The Hugging Face cache takes space in addition to task files. Set HF_HUB_CACHE in your shell or selected .env to move it to another disk; task copies stay in the checkout. See the storage guide.

    Use search-swe info to inspect your checkout. Between evaluation batches, preview updates with search-swe update --check, then apply them with search-swe update. See the update guide.

Coding agent

Let your agent configure the run, monitor progress and report results.

Agent guide

  1. Prepare the evaluation host

    Select or clone a checkout and save your settings without starting an evaluation:

    pip install search-swe
search-swe configure

  2. Add the evaluation skill

    Run this in your agent’s workspace, then reload its skills or start a new session:

    npx skills add VectorSpaceLab/Search-SWE --skill run-searchswe-evaluation -a codex

    Requires Node.js 22.20+. The example uses Codex; replace codex with your supported agent, such as claude-code.

    For other installers and setup options, see the installation guide.

  3. Describe your experiment

    Give your agent the checkout location, Python environment, model settings and tasks. For example:

    Use run-searchswe-evaluation with the checkout at /path/to/Search-SWE. Run agentic-search and agentic-search-vectorless with Codex + OpenRouter + openai/gpt-6-luna, max reasoning and concurrency 2. Reuse my .env, monitor progress and report scores, verifier health and result paths.

    The agent coordinating the run can differ from the agent and model being evaluated.

    If keys are missing, the agent guides you through local prompts or your private .env. Keep key values out of chat.

Read the results

Both workflows save results in the selected checkout:

jobs/evaluations/<run>/

Open the printed path and follow run.json to each task’s results and logs. Check Harbor scores and verifier health; process completion alone does not mean a passing score.

Harbor isolates coding and verification. Scores reflect held-out search quality, correctness, resource limits and evaluation integrity.

Explore published scores and trajectories