Search-SWE
GPU TASK Valid result

Instruction-following Embedding

DeepSeek V4.1 Flash 1017 · Codex (v0.164.0) · reasoning effort max

Task description ↗Task at evaluation revision ↗← Previous taskNext task →
Task score (%)89.3
Input Tokens20,907,536including 20,721,280 cached
Output Tokens156,626
Agent time1h 31min
Cost$0.18
Recorded steps170

Cost is estimated from recorded agent tokens at official API rates in USD (2026-11-01; DeepSeek off-peak pricing).

Loading trajectory…

Select this tab to browse the saved files.

Coding time & token usage
Original agent completion
normal
Coding started
2026-10-24T03:12:05.272158Z
Coding finished
2026-10-24T04:43:45.597514Z
Input (includes cache)
20,907,536
Cached input (subset)
20,721,280
Cache writes (subset)
0
Output
156,626
Reasoning (reported separately; not added)
77,754
Recorded responses
167

Input includes cached input; output is counted once. Auxiliary reasoning counts are shown as recorded. Regrading does not repeat coding usage.

Select this tab to browse the saved files.

Configuration

Evaluation settings summary
{
  "agent": "Codex",
  "agent_timeout_sec": 7200.0,
  "coding_commit": null,
  "effort": "max",
  "model": "deepseek/deepseek-v4.1-flash-1017",
  "provider": "OpenRouter",
  "resources": null,
  "run_mode": "full_agent_and_verifier",
  "task_commit": null,
  "verifier_commit": "bb34e915be5c2b2948146c2d7862cb58652d6300",
  "version": "0.164.0"
}

Select this tab to browse the saved files.

Resolved configuration

Select this tab to browse the saved files.

Trial log

Select this tab to browse the saved files.

Exceptions

Select this tab to browse the saved files.

Select this tab to browse the saved files.

Trajectory downloads & provenance
Exported ATIF trajectory (.json.gz) ↓Final evaluation (.json) ↓

Public text redacts private values and normalizes host paths. Source hashes identify the original records.

Trial ID
instruction-following-embedding-deepseek-v4.1-flash-1017-max-ab0b0e0df236
Source trajectory SHA-256
fe10f6264e2c56670ae0073d85ccdd8d9a957644581c46a10acafbe616b65c5e
Final reward SHA-256
114c9900467129988d8924388ff9d3c39cb1e7cc9e1416f7ee915b63773862fe