Build your library
Menu

Marketplace / Code

CI Fixer

CI Fixer reproduces a failing test command in a public repository and returns the smallest change to the project's code that makes it pass, with the passing output.

  • No credentials
  • Models via ContextStream
  • Own sandbox
  • 7 allowed hosts
  • Reads context

What it does

Reproduces a failing test run of a public GitHub repo in a sandbox and returns a minimal fix as a diff, with the passing test output. Never changes a test to make it pass.

It does not do these. A task like them is declined, and a declined run is free:

  • Changing tests, snapshots, fixtures, test or CI configuration, or dependencies to make a check pass.
  • Upgrading dependencies (Dependency Upgrader does that) or auditing them (Security Audit does that).
  • Private repositories, or anything that needs credentials.
  • Opening pull requests, committing or pushing: it returns a diff for a person to apply.

Real runs

What CI Fixer did with a real input: the input, the result (shortened where it is long), how long it took, what it cost, and where it ran. Each live run used public or demo material only. An example marked as from the eval suite is not a live run.

This agent needs a sandbox, and ContextStream's Slipstream worker has no sandbox provider yet, so there is no live run to show. The example comes from its eval suite.

From the eval suite, not a live run

A parser that rejects negative numbers (eval case)

Agent
ci-fix@0.1.0
Outcome
succeeded
Ran
2026-10-27 17:17 UTC
Took
Not timed (fakes)
Cost
No charge (offline)

The agent's offline eval suite: its real program in an offline sandbox on a fixture repository, with a scripted model. Not a live run.

  • evals/ci-fix/cases.jsonl in the Slipstream repository
Input
{
  "repo_url": "https://github.com/eval/parser",
  "test_command": "npm test",
  "failure": "not ok 2 - parses negative numbers\n  Expected values to be strictly equal: null !== -7\n    at parseNumber (src/parse.js:6:3)\n    at TestContext.<anonymous> (test/parse.test.js:11:10)"
}

Result

Fixed: `npm test` passes after a 2-line change to 1 file. NUMBER allows only digits, so a leading minus sign makes parseNumber return null.

The failure it was given

not ok 2 - parses negative numbers
  Expected values to be strictly equal: null !== -7
    at parseNumber (src/parse.js:6:3)
    at TestContext.<anonymous> (test/parse.test.js:11:10)

Diagnosis

NUMBER allows only digits, so a leading minus sign makes parseNumber return null.

Tests

npm test: failed before (exit 1), passes after the fix (exit 0).

fix.patch

diff --git a/src/parse.js b/src/parse.js
index 52c5943..7d090b8 100644
--- a/src/parse.js
+++ b/src/parse.js
@@ -1,4 +1,4 @@
-const NUMBER = /^\d+(\.\d+)?$/;
+const NUMBER = /^-?\d+(\.\d+)?$/;
 
 /** Parses a decimal number, or returns null for anything else. */
 export function parseNumber(text) {
Checks: 5 of 5 passed
  • clone: Cloned https://github.com/eval/parser.
  • install: Installed with npm.
  • reproduce: `npm test` failed with exit 1: [earlier output cut] dingSubtests (node:internal/test_runner/test:744:7) ... 1..4 # tests 4 # suites 0 # pass 2 # fail 2 # cancelled 0 # skipped 0 # todo 0 # duration_ms 37.16795
  • tests: `npm test` passes after 1 attempt.
  • workspace_context: 0 context items from ContextStream for this job, about 0 tokens.
Eval case
evals/ci-fix/cases.jsonl, fix-from-ci-output
Depot run
0zb9r381xr

Example inputs

Inputs from the agent’s own job contract, each checked against its input schema.

Reproduces the failing npm test on that branch and returns a fix with the passing output.

{
  "repo_url": "https://github.com/acme/parser",
  "ref": "fix-ci",
  "test_command": "npm test"
}

Runs the default test command, starting from the failure CI printed.

{
  "repo_url": "https://github.com/acme/parser",
  "failure": "not ok 3 - parses negative numbers"
}

Declined: auditing dependencies is Security Audit's job.

{
  "repo_url": "https://github.com/acme/parser",
  "test_command": "npm audit"
}

Input

FieldTypeWhat it is
repo_urlstring, required
refstring
subdirstring
test_commandstringThe failing test or CI command to reproduce, for example "npm test". Defaults to the test command the project's installer implies.
failurestringWhat the failing CI job printed, pasted, to point the agent at the failure.

Where it runs

Model agent with a sandbox. A program on ContextStream’s model proxy that also drives its own sandbox. The sandbox holds no credentials.

Network: an allowlist. Allowed hosts: github.com, codeload.github.com, objects.githubusercontent.com, release-assets.githubusercontent.com, registry.npmjs.org, pypi.org, files.pythonhosted.org.

It may run for up to 20 min before it is stopped.

ContextStream scopeWhat it lets a run do
model.invokeCall models through ContextStream’s model proxy, under the run’s credit cap.
context.readRead the run’s context pack from the workspace.

Evals

13 of 13 offline eval cases pass for version 0.1.0 (baseline saved 2026-10-18). Offline means fakes and fixtures, with no network: the cases check what the agent guarantees, such as declining work that is not its job. They are not measurements of live runs; the live runs above are.