Run your agent

This guide connects an agent command you already have to a Versalist challenge. You record a baseline run, check its output with a verifier, change one thing, and let versalist compare decide whether the candidate improved. run, evaluate, and compare work locally and write to .versalist/ in your repository. Nothing is uploaded until you submit, and submit sends only the project URL and metadata.

The example uses illustrative support requests, where each case has an expected category and priority. Supply your own agent command and adapt the verifier to your challenge. For a complete local example without an agent, use the quickstart. The command reference lists every command, what it writes, and its exit codes.

Before you start

The terminal path needs:

• Node.js 18.17 or later, for the CLI.
• An agent command you can run from a shell. For the verifier below, it must write its answers to output.json in the working directory.
• Python 3, for the example verifier only.
• A Versalist API key for start and submit. An admin reviews every new account before it can create keys, so request access early. Browsing with list needs no key or account.

1. Pick a challenge

You do not need an account to browse. A challenge is a fixed task with inputs, expected results, and a scoring rubric. For a useful comparison, keep the test cases, scoring rules, model settings, and runtime settings the same.

npx -y @versalist/cli list
npx -y @versalist/cli list --search "support"

Note the slug of the challenge you want. Add --json for machine-readable output. The same catalog is on the challenges page.

2. Create an API key

1. Sign in with work email, Google, or GitHub. An admin reviews every new account before it can be used. You get an email when yours is ready.
2. Open Developer API and create a key with read:challenges and submit:solutions. That is all this page needs.
3. Copy the value once. It starts with vk_live_.

export VERSALIST_API_KEY=vk_live_...

Other scopes are explained on the API keys page. You do not need them yet.

3. Load the challenge into your repository

Install the CLI once so the remaining steps can call versalist directly. start writes files. It does not run anything.

npm install -g @versalist/cli
versalist start <challenge-slug>

• CHALLENGE.md: the brief your agent should read.
• .versalist.json: the challenge id and local metadata.
• eval/examples.json: the public test cases with expected results, when the challenge publishes them. Private cases are never downloaded.

start also adds .versalist/ to .gitignore. If the challenge publishes no public cases, start deletes an existing eval/examples.json, so keep your own cases in a different file.

Point your agent at the working directory. Agents that read the repository pick up CHALLENGE.md without an extra call.

4. Run your agent and record it

Wrap the command you already use. Versalist records the command, exit status, duration, Git revision, and a hash of the challenge files, plus stdout.log and stderr.log.

versalist run --command "python agent.py" --label baseline

Replace python agent.py with your agent command. Versalist does not create this script. For the verifier below, your command must write output.json, an object keyed by case ID. Each value must contain the fields in that case’s expected_output object.

The command times out after 900 seconds unless you pass --timeout <seconds>. When it fails or times out, versalist run still records the run and exits non-zero.

5. Evaluate the run

A local evaluation attaches a verifier result to a recorded run. The command reads files in your current working directory. The verifier exits 0 when every check passes and non-zero otherwise. Write it to print what it compared, because that output is the evidence you will read later.

Save the following script as check_output.py. It requires nonempty test cases with object-shaped expected results. It rejects missing results. For another output format, use a verifier that checks that format.

import json
import sys
from pathlib import Path

cases = json.loads(Path("eval/examples.json").read_text())
actual = json.loads(Path("output.json").read_text())
if not isinstance(cases, list) or not cases:
    sys.exit("FAIL: no test cases")
if not isinstance(actual, dict):
    sys.exit("FAIL: output must be an object keyed by case ID")

failures = 0
seen = set()
for case in cases:
    item_id = case.get("item_id") if isinstance(case, dict) else None
    want = case.get("expected_output") if isinstance(case, dict) else None
    if not isinstance(item_id, str) or not item_id or item_id in seen:
        sys.exit("FAIL: case IDs must be nonempty, unique strings")
    seen.add(item_id)
    if not isinstance(want, dict) or not want:
        sys.exit(f"FAIL: {item_id} has no expected fields")
    got = actual.get(item_id)
    if not isinstance(got, dict):
        sys.exit(f"FAIL: {item_id} has no result object")
    for field, value in want.items():
        ok = field in got and got[field] == value
        failures += int(not ok)
        print(f"{'ok  ' if ok else 'FAIL'} {item_id} {field}: expected {value!r}, actual {got.get(field)!r}")
print(f"{len(cases)} cases checked, {failures} failed field(s)")
sys.exit(1 if failures else 0)

versalist evaluate --run latest --command "python check_output.py"

Illustrative verifier output for three support cases:

ok   case-1 category: expected 'billing', actual 'billing'
FAIL case-1 priority: expected 'high', actual 'medium'
ok   case-2 category: expected 'access', actual 'access'
ok   case-2 priority: expected 'high', actual 'high'
ok   case-3 category: expected 'billing', actual 'billing'
ok   case-3 priority: expected 'low', actual 'low'
3 cases checked, 1 failed field(s)

The full verifier output is in .versalist/runs/<run-id>/evaluations/<evaluation-id>/stdout.log. This is the local counterpart of the failed check you would see on a hosted run. Understand your results covers both.

6. Fix one thing and compare

Change one thing, for example the instruction that decides priority. Then run and evaluate again with the same verifier. The first run is your baseline, the current version. The second is the candidate, the proposed version.

versalist run --command "python agent.py" --label candidate
versalist evaluate --run latest --command "python check_output.py"

versalist compare --baseline <baseline-run-id> --candidate <candidate-run-id>

Use the run IDs from your command output. If a baseline fails and a candidate passes, their default scores are 0 and 100. The comparison reports a 100-point improvement. This example describes the scoring rule. It is not evidence of a measured agent improvement.

decision is one of improved, unchanged, below_threshold, or regressed. The command exits 1 when the candidate fails its verifier or scores lower, so you can use it as a gate in CI. Add --min-delta 5 to require a minimum improvement.

Evaluate both runs with the same verifier command. If the commands differ, compare stops with Both evaluations must use the same verifier command.A usage error, such as a missing run ID, also exits 1. In CI, add --json and read gate_passed and decision rather than relying on the exit code alone.

Evidence stays in .versalist/. Run logs stay in your repository.

7. Optional: record a finer score

When your verifier computes a score, pass that value with --score. Create results.json before you attach it with --metrics. The values below are illustrative:

versalist evaluate --run latest --command "python check_output.py" \
  --score 83 --metrics results.json

8. Submit

versalist submit \
  --url https://github.com/you/solution \
  --title "Candidate run"

Submit sends the project URL and metadata, not the contents of .versalist/. One submission per challenge per account. A second submit returns 409.

Command reference

Every command below accepts --json except start and submit. Every command also exits non-zero on a usage error, such as a missing required flag. The CLI reference lists every flag.

Troubleshooting

Use the web app

A hosted run gives you more than a local evaluation: a score for each rubric dimension, results for public test cases, and aggregate progress that includes private cases. Traces require capture and access to be enabled.

1. Sign in. Company workspaces that require approval use enterprise sign-in.
2. Open a challenge on Challenges. Read the task, constraints, and rubric.
3. Select Run, then choose the skill and model. The card shows a Runtime block when the challenge needs a sandbox. If that runtime is unavailable on Versalist, the Run button is disabled and the card says why.
4. Open the result. Score by rubric dimension shows where points were lost, Task-by-task shows public test cases, and Execution trace shows recorded calls when available.
5. Change one thing, run again, and compare the two results.

Connect a coding agent

If you want the challenge tools inside Cursor, Claude Code, or another host, start the Model Context Protocol (MCP) server instead of, or in addition to, the command-line interface (CLI):

npx -y @versalist/cli mcp

Host configs are on coding agents. The tools are listed on MCP tools.

Next steps

• Understand your results: scores, failed checks, execution errors, timeouts, and traces.
• Test a change before you release it: save a failure as a test case, compare baseline and candidate, record the decision.
• Where your agent runs: hosted sandbox, your own machine, and local records.
• CLI reference: every flag, including VERSALIST_BASE_URL for staging.
• API reference: HTTP if you are not using the CLI.