Complete Technical Reference for Polyman CLI Tool
A TypeScript-based CLI tool for Codeforces problem setters that automates problem preparation workflows including test generation, validation, solution verification, and checker integration.
Polyman follows a layered architecture:
┌─────────────────────────────────────┐
│ CLI Layer (cli.ts) │
│ Commander.js command parsing │
└─────────────────────────────────────┘
↓
┌─────────────────────────────────────┐
│ Action Layer (actions.ts) │
│ High-level workflow orchestration│
└─────────────────────────────────────┘
↓
┌─────────────────────────────────────┐
│ Helper Layer (helpers/) │
│ Domain-specific logic modules │
└─────────────────────────────────────┘
↓
┌─────────────────────────────────────┐
│ Execution Engine (executor.ts) │
│ Process management & I/O │
└─────────────────────────────────────┘
↓
┌─────────────────────────────────────┐
│ Formatter (formatter.ts) │
│ Terminal output styling │
└─────────────────────────────────────┘
The CLI entry point is src/cli.ts. It uses Commander.js to parse argv and delegate to action functions in src/actions.ts. See GUIDE.md - CLI Commands Reference for user-facing command descriptions, or read src/cli.ts for the wiring.
All TypeScript types live in src/types.d.ts. The key shapes are ConfigFile (the on-disk Config.json schema), LocalSolution, LocalGenerator, LocalChecker, LocalValidator, LocalTestset, plus SolutionTag and the source-type unions for C++/Java/Python. See src/types.d.ts for the full definitions and JSDoc, or GUIDE.md - Configuration Reference for usage.
Action functions in src/actions.ts orchestrate workflows by composing helpers. Each action is structured as a sequence of independent step… calls (see src/steps.ts) which compile, run, and verify components. See GUIDE.md - CLI Commands Reference for what each action does from the user perspective.
Each helper module owns a single domain. Function-level documentation is on TypeDoc; this section lists the responsibility of each module so contributors know where to look.
src/helpers/utils.ts — Cross-cutting utilities: config file reading, directory management, C++ and Java compilation, error logging, and the shared exit/throw helpers used by every action.src/helpers/generator.ts — Generator compilation and test file production. Drives Polygon-format scripts: parses, resolves to concrete test indices, then runs each generator (or copies each manual test) into the testset's output directory.src/helpers/validator.ts — Validator compilation, validation of generated test inputs, and validator self-testing against validator_tests.json.src/helpers/checker.ts — Checker compilation (custom or standard from assets/checkers/), checker invocation, and checker self-testing. Owns the verdict mapping from checker stdout to CheckerVerdict.src/helpers/solution.ts — Solution compilation, execution against a testset, and verdict comparison. Responsible for enforcing the solution-tag contract (e.g. a WA solution must produce WA or PE on at least one test, as on Polygon).src/helpers/create-template.ts — Materialises the bundled template/ tree into a new problem directory when polyman new runs.src/helpers/testlib-download.ts — Fetches testlib.h from the upstream GitHub repo for the polyman download-testlib command.src/helpers/testset.ts — Testset and group resolution: given filters from CLI options, returns the matching testsets, groups, and indices.src/helpers/script-parser.ts — Parses Polygon-format generation scripts (gen [args] > N|$|{indices}, FreeMarker comments and <#list> loops, <#-- @group X --> headers) and resolves them, together with manualTests[], into a flat list of test-production instructions with concrete indices.src/helpers/remote/ — Polygon integration. pulling.ts and pushing.ts implement the per-component download/upload steps; utils.ts handles credentials, line-ending normalization, and problem-id extraction; viewer.ts formats polyman remote view output.For function-level docs see TypeDoc at https://hamzahassanain.github.io/polyman/.
src/executor.tsLow-level process execution with timeout and memory limit support.
executor.executeWithTimeout(command: string, args: string[], timeoutMs: number, memoryLimitMB: number): Promise<ExecutionResult>Executes command with resource limits.
Parameters:
command: Executable path or commandargs: Command-line argumentstimeoutMs: Maximum execution timememoryLimitMB: Maximum memory usageReturns:
interface ExecutionResult {
stdout: string;
stderr: string;
exitCode: number;
timedOut: boolean;
memoryExceeded: boolean;
}
Platform Support:
ulimit for memory limitingImplementation:
// Memory limiting (Java gets -Xmx instead)
if (memoryLimitMB > 0 && process.platform === 'darwin') {
// macOS: ulimit -v fails there, so the limit is skipped and a warning is shown once
} else if (memoryLimitMB > 0 && process.platform !== 'win32') {
const memoryLimitKB = memoryLimitMB * 1024;
command = `(ulimit -v ${memoryLimitKB}; ${command})`;
}
// Spawn process with timeout
const child = spawn(command, args, { shell: true });
const timeout = setTimeout(() => {
child.kill();
timedOut = true;
}, timeoutMs);
src/formatter.tsTerminal output styling with Codeforces theme.
The actual colors come from src/theme.ts: a dark theme and a light theme. A non-empty NO_COLOR turns all colors off. POLYMAN_THEME=dark or POLYMAN_THEME=light forces a theme. When POLYMAN_THEME is unset or auto, resolveTheme() detects the terminal background with src/helpers/terminal-background.ts: first COLORFGBG, then an OSC 11 query followed by a DA1 query (the DA1 answer tells a terminal without OSC 11 support apart from a slow one), with a 100 ms timeout, and dark as the fallback. Detection is skipped when stdin or stdout is not a TTY, when chalk has no colors, when TERM=dumb, for background jobs, and on Windows.
Detection is async, but fmt is created when the module loads, so src/cli.ts awaits resolveTheme() once before program.parse() and applies the result with fmt.setTheme(theme) and printComprehensiveHelp(program, theme). The sync getTheme(env) does no detection (unset means dark); it is the default for new Formatter() and for tests.
Colors:
#1E88E5 (blue)#FF6B6B (red)#4CAF50 (green)#FFC107 (yellow)section(title: string)
Prints section header with box.
╭─────────────────────────────────╮
│ 📁 SECTION TITLE │
╰─────────────────────────────────╯
step(stepNumber: number, title: string)
Prints numbered step header.
╭─ STEP 1: Creating Directory
stepComplete(message: string)
Prints step completion (╰─ ✓ Done).
Other methods: success, error, warning, info, log, successBox, errorBox.
primary(text: string): Blue colorhighlight(text: string): Cyan colordim(text: string): Dimmed textsuccessIcon(): ✓ iconerrorIcon(): ✗ iconwarningIcon(): ⚠ iconinfoIcon(): ℹ iconThe on-disk Config.json schema is described in GUIDE.md - Configuration File Reference for users, and defined as TypeScript interfaces in src/types.d.ts (ConfigFile).
C++:
g++ -O2 -std=<cppStandard> <compiler.flags...> -iquote <problem-root> -o output source.cpp
The executable comes from POLYMAN_CXX, then compiler.cpp in Config.json, then g++ (resolveCppCompiler in src/helpers/utils.ts). The flags come from resolveCppFlags: a -std= item in compiler.flags replaces -std=<cppStandard> and an -O item replaces -O2. For a solution, compileSolution in src/helpers/solution.ts passes the standard named in its sourceType (cpp.g++17 gives c++17) as the fallback when Config.json has no cppStandard. Solutions are compiled with { onlineJudge: true }, which adds -DONLINE_JUDGE after the standard (like Codeforces), unless compiler.flags already defines it. Checkers, validators and generators do not get it. The resolved executable and flags are part of the compile-cache key.
The problem root (the current working directory) is passed via -iquote so sources in subdirectories such as generators/ can #include "testlib.h" from the root-level copy without a system-wide install. Every path and argument that reaches the shell is quoted with quoteShellArgument (src/helpers/shell.ts), so problem directories containing spaces or shell metacharacters work.
Java:
javac source.java
java ClassName < input.txt > output.txt
javac comes from POLYMAN_JAVAC, then compiler.javac, then javac (resolveJavaCompiler). java is always taken from PATH.
Python:
python3 source.py < input.txt > output.txt
.cpp → compileCPP.java → compileJava.py → Return interpreter commandThe validator is a C++ program built against testlib that reads a test input on stdin and exits 0 for VALID or 3 for INVALID (any other code is an error). The validation workflow compiles the validator once and runs it over each test file in the requested testset, logging per-test results. Validator self-tests live in validator/validator_tests.json as {stdin, expectedVerdict} pairs and are exercised by polyman test validator to verify the validator agrees with its own contract before being used on generated tests.
Each solution is compiled, then run against every test in the active testset under the configured time and memory limits. The runner inspects the first line of the captured output for the sentinel verdicts Time Limit Exceeded, Memory Limit Exceeded, and Runtime Error; otherwise it invokes the checker on the (input, output, answer) triple to obtain OK/WA/PE. Aggregated VerdictTracker flags (didWA, didPE, didTLE, didMLE, didRTE) are then matched against the solution's declared SolutionTag so that, e.g., a WA solution must get WA or PE on at least one test (as on Polygon) and a MA solution must pass them all.
Standard checkers ship in assets/checkers/*.cpp (vendored testlib checkers — do not modify). Run polyman list checkers for the runtime list, or see GUIDE.md - Standard Checkers.
Location: checker/chk.cpp
Interface:
int main(int argc, char* argv[]) {
registerTestlibCmd(argc, argv);
// Read input
// Read output (ouf)
// Read answer (ans)
// Compare and return verdict:
// quitf(_ok, "Correct");
// quitf(_wa, "Wrong Answer");
// quitf(_pe, "Presentation Error");
}
Command:
./checker <input_file> <output_file> <answer_file>
Output Parsing:
ok/OK → OKwrong/WA → WApresentation/PE → PEGenerators are testlib C++ programs that take parameters as positional argv[] and write the test content to stdout. The pipeline is:
script-parser.ts reads the testset's generatorScript.script (or scriptFile) as Codeforces-Polygon syntax: each non-comment line is gen-name [args...] > target where target is N, $ (smallest unused), or {1-3,7} (multi-output). FreeMarker <#-- comments -->, <#list a..b as i> ... ${i} ... </#list>, and <#list ["a", "b"] as s> blocks are supported, and <#-- @group <name> --> headers tag every following line.script-parser.ts:resolveTests merges parsed lines with the testset's manualTests[], assigning concrete Polygon indices to every $. Manual indices and explicit indices reserve slots; $ walks the smallest free integer. Duplicate indices are an error.generator.ts:executeResolvedTests compiles each referenced generator once and, per resolved test, either copies the manual .in file into testsets/<name>/test<index>.txt, runs the generator with stdout redirected to that file, or (for > {indices} lines) runs the generator with cwd set to the testset directory and verifies it wrote the promised files itself.Generator names with extensions (gen.exe, gen.cpp) are rejected — Polygon strips them on the server side, so polyman keeps the same constraint.
Polyman provides comprehensive error handling with formatted output and proper exit codes.
PolymanError(message, exitCode, hint?) from src/errors.ts (codes: 1 problem failed, 2 usage error, 3 environment error, 4 remote error).--json report with the code, and throws the error again..catch on program.parseAsync in cli.ts prints the message (and the hint) and exits with exitCodeFor(error). Any error that is not a PolymanError exits with 1.Only two places call process.exit on their own: the SIGINT/SIGTERM handler in executor.ts (exit 130/143, a signal does not go through an action) and node-version-check.ts (exit 3, it runs before anything else loads). See GUIDE.md for the full table.
Configuration Errors:
Compilation Errors:
Execution Errors:
Validation Errors:
Checker Errors:
Polyman provides comprehensive integration with the Codeforces Polygon system through a type-safe TypeScript SDK and remote operations.
Location: src/polygon.ts
The Polygon SDK is a complete TypeScript implementation of the Polygon API v1, providing type-safe methods for all Polygon operations.
class PolygonSDK {
private apiKey: string;
private apiSecret: string;
private baseUrl: string;
constructor(config: PolygonConfig) {
this.apiKey = config.apiKey;
this.apiSecret = config.apiSecret;
this.baseUrl = config.baseUrl || 'https://polygon.codeforces.com/api';
}
// 54+ API methods for complete Polygon integration
}
The SDK exposes 50+ methods grouped under Problem Management, Statements, Solutions, Checker & Validator, Generators, Tests, Metadata, Packages, and Contests. See src/polygon.ts for the full method signatures and JSDoc, or TypeDoc for the rendered API.
Polygon API uses SHA-512 signature-based authentication.
API Credentials:
{
apiKey: 'your-api-key',
apiSecret: 'your-api-secret'
}
Request Signing:
// Generate 6-character random prefix
const rand = generateRandomString(6);
// Sort parameters alphabetically
const sortedParams = Object.keys(params).sort();
// Build signature string
const sigString = `${rand}/${methodName}?${paramString}#${apiSecret}`;
// Compute SHA-512 hash
const hash = crypto.createHash('sha512').update(sigString).digest('hex');
// Final signature
const apiSig = rand + hash;
Request Parameters:
apiKey - Your API keytime - Current Unix timestampapiSig - Computed signatureAPI Call:
POST https://polygon.codeforces.com/api/{method}
Content-Type: application/x-www-form-urlencoded
apiKey=xxx&time=xxx&apiSig=xxx&problemId=xxx&...
~/.polyman/credentials.jsonUser Command
↓
polyman remote pull 123456 ./my-problem
↓
remotePullProblemAction
↓
┌────────────────────────────────────┐
│ Step 1: Read Credentials │
│ - Load from ~/.polyman/ │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Step 2: Initialize SDK │
│ - Create PolygonSDK instance │
│ - Configure authentication │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Step 3: Fetch Problem Info │
│ - Get limits, I/O files │
│ - Get problem metadata │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Step 4: Create Directory │
│ - Create problem directory │
│ - Create subdirectories │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Step 5: Download Components │
│ ┌──────────────────────────────┐ │
│ │ Solutions (parallel) │ │
│ │ Checker + tests │ │
│ │ Validator + tests │ │
│ │ Generators │ │
│ │ Statements (all languages) │ │
│ │ Tests (parallel fetch) │ │
│ │ Metadata │ │
│ └──────────────────────────────┘ │
│ - Normalize line endings (→ Unix) │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Step 6: Generate Config.json │
│ - Build complete configuration │
│ - Include all metadata │
└────────────────────────────────────┘
↓
Success! Problem ready for local work
User Command
↓
polyman remote push ./my-problem
↓
remotePushProblemAction
↓
┌────────────────────────────────────┐
│ Step 1: Read Credentials │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Step 2: Initialize SDK │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Step 3: Read Config.json │
│ - Get problem ID │
│ - Get all configurations │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Step 4: Update Problem Info │
│ - Upload time/memory limits │
│ - Upload I/O file settings │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Step 5-11: Upload Components │
│ ┌──────────────────────────────┐ │
│ │ Solutions with tags │ │
│ │ Checker + set active │ │
│ │ Validator + tests + set │ │
│ │ Generators │ │
│ │ Statements (all languages) │ │
│ │ Metadata │ │
│ │ Testsets: │ │
│ │ - Clear existing │ │
│ │ - Enable groups │ │
│ │ - Upload tests (parallel) │ │
│ │ - Upload script │ │
│ └──────────────────────────────┘ │
│ - Normalize line endings (→ Win) │
└────────────────────────────────────┘
↓
Success! Changes uploaded to Polygon
(Don't forget to commit!)
Parallel Test Operations:
Both pull and push operations fetch/upload manual tests in parallel for significant performance improvements:
Pull (Parallel Fetch):
// Fetch test metadata without inputs (fast)
const tests = await sdk.getTests(problemId, testsetName, true);
const manualTests = tests.filter(t => t.manual);
// Fetch all inputs in parallel
const promises = manualTests.map(test =>
sdk.getTestInput(problemId, testsetName, test.index)
);
const results = await Promise.all(promises);
Push (Parallel Upload):
// Create upload promises for all manual tests
const promises = manualTests.map(test =>
sdk.saveTest(problemId, testsetName, index, input, options)
);
// Upload all in parallel
await Promise.all(promises);
Performance Impact:
polyman chat and the polyman ai commands live in src/agent/. The user side is in GUIDE.md - Chat (AI agent). The design docs are in design/agent/; this section is a map of the code, not a copy of them.
The main rule (design decision D1): the agent never imports actions.ts or steps.ts. It runs the polyman CLI as a child process with --json, like a user would. A crash or a process.exit in the CLI cannot kill the chat, and the CLI stays the one source of truth.
cli.ts `chat` and `ai` load the agent with import(), only when they run
↓
agent/index.ts createChatRuntime(): bus, gate, runner, tools, sandbox, prompt, session
↓
agent/core/ AgentSession (the loop), event bus, context trimming, system prompt
↓
agent/tools/ registry, permission gate, sandbox, the tools themselves
↓
agent/runner/ NodeCliRunner: runs dist/cli.js as a child process
↓
agent/providers/ the model APIs (streaming, retries, keys)
↓
agent/tui/ or plain.ts Ink terminal UI, or the line mode; both only read events
cli.ts loads agent/index.js and agent/ai-commands.js with a dynamic import(). Other commands never load the agent code, so they do not start slower.agent/index.ts: resolveChatStart() decides the problem folder (a folder with Config.json, or "new problem" mode). createChatRuntime() wires everything and needs no terminal and no key, so tests call it with a mock provider. startChat() adds the real provider and the UI.agent/core/:
session.ts: AgentSession, the agent loop. One turn asks the model, runs the tool calls it asks for, and asks again, until the model stops, the step limit, a cancel or an error. It has a repeat guard (the same call with the same arguments at most twice per turn, reset after a successful write) and it redacts secrets from tool output and errors. The chat and every workflow worker use this same class.event-bus.ts, events.ts: the event protocol, see below.context.ts: trims a copy of the history to the context budget before each model call. The history itself is never changed.prompt.ts: ChatPromptBuilder builds the system prompt before each call: the rules (workflow/prompts.ts), the skills index, the environment (OS, g++, polyman version) and the problem state (notes, a Config.json summary, file names).text-tool-call.ts: finds a tool call the model wrote as text, so the session can ask it to use the real format.pasted-key.ts: blocks a message that looks like a Polygon key before it is sent.agent/tools/:
registry.ts: every call goes through the same checks: the tool exists, the arguments are valid JSON and match the schema (schema.ts), there is a problem folder when the tool needs one, the permission gate allows it, it runs with a time limit, and its output is cut to a size the model can read. A failure is an error output the model reads, never a throw (only a cancel throws).permissions.ts: DefaultPermissionGate. Each tool has a risk (read, run, write, remote) and the mode (ask, auto-edit, plan-approved) decides what asks the user. remote always asks. Asking emits approval_needed and waits for resolve().sandbox.ts: FolderSandbox checks every path by its real path (symlinks followed). It refuses paths outside the problem folder and writes to .polyman/cache/, .polyman/agent/sessions/ and testlib.h. A workflow worker gets a sandbox with allowWrite set to the files its task owns.fs-tools.ts, cli-tools.ts, remote-tools.ts, notes-tools.ts, ideas-tool.ts, workflow-tool.ts, and skills.ts one level up: the tools. skills.ts reads template/instructions/*.md from the installed package.agent/runner/: NodeCliRunner spawns process.execPath dist/cli.js ..., never a shell. All runs go through one FIFO lock (lock.ts), so parallel workers never compile or time solutions at the same time. The child gets POLYMAN_AGENT=1, NO_COLOR=1 and the environment without API keys. On POSIX the child leads its own process group and kill-tree.ts kills the whole group on a timeout or a cancel; on Windows it uses taskkill /T /F. The full output goes to <problem>/.polyman/agent/logs/, the tool result keeps only the start and the end.agent/providers/: see Providers, keys and models. mock.ts (MockProvider) plays a fixed script and is what the tests use.agent/tui/: the Ink UI. It is the only ESM part of polyman (src/agent/tui/package.json has "type": "module", and npm run build copies it to dist/), because Ink is ESM only. agent/index.ts loads it with a variable path. store.ts is a pure reducer from events to screen state, so most UI logic is tested without a terminal. keybindings.ts has the keys and /help. dev/run.tsx starts the UI with a fake session: npm run build && node dist/agent/tui/dev/run.js.agent/plain.ts: the line mode (--plain) with readline. It is also the fallback when the TUI fails to load.agent/providers/ turns a provider and a model into a Provider and the budgets of the chat.
presets.ts: the known providers (base URL, env var, quirks, known models). Two adapters: openai-compatible.ts for every OpenAI-compatible server and anthropic.ts. Both stream through sse-parser.ts, and retry.ts retries rate limits and server errors before the first byte.key-store.ts: ~/.polyman/ai.json (keys, the default, overrides). keyForRequest only gives a key for its own base URL.model-catalog.ts: the live model list. ModelCatalog calls listModels() of the OpenAI-compatible adapter without a key, caches the result for 24 hours in ~/.polyman/agent/models/<provider>.json and falls back to the old cache when offline. Only OpenRouter uses it today.capabilities.ts: what a model can do. Lookup order, last one wins: DEFAULT_CAPABILITIES, the preset model, the live list, the ai.json override. The Capabilities table of a session also turns tools off for a model that failed with tools_not_supported.chat-provider.ts: createChatProvider picks the provider (--provider, the ai.json default, openrouter) and the model (--model, the default, DEFAULT_CHAT_MODEL), checks the key and returns the provider with its capabilities. budgetsFor sizes the chat from the capabilities: maxOutputTokens = min(8000, maxOutput, contextWindow / 4) and contextBudgetTokens = min(32000, (contextWindow - maxOutputTokens) * 0.75), then the ai.json override (chatMaxSteps, contextBudgetTokens, maxOutputTokens) wins.The workflow uses the same provider and model. ModelManager and ModelWorker get the model's max output, and a worker keeps min(16000, chat context budget) of context.
A session with a chat-only model sends no tools and emits one notice event with the reason. When the provider answers tools_not_supported in the middle of a turn, the session turns tools off and runs the step again without them.
Everything the core reports is an event on the bus (core/events.ts). The TUI, the plain mode, the evals and the tests read the same events. SimpleEventBus gives each event a seq number.
| Event | When |
|---|---|
turn_start, turn_end |
A user message starts a turn. turn_end has the reason: done, cancelled, max_steps or error. |
text_delta, reasoning_delta |
A chunk of the answer or of the model's reasoning. |
tool_call, tool_progress, tool_result |
A tool call, a progress line (stderr of the CLI), and its result. Events from a workflow worker carry a taskId. |
approval_needed, approval_resolved |
A call waits for the user, and the answer (allow_once, allow_session, or deny with a reason). |
usage |
Token counts of one model call, and its cost when the provider reports it. |
notes_updated, problem_dir_changed, compacted |
The spec changed, create_problem made a new folder, old messages were trimmed. |
plan_created, plan_approval_needed |
The manager made a plan, and the plan waits for the user. |
task_started, task_checkpoint, task_done |
A workflow task started an attempt, ran its check, and ended. |
workflow_done |
The workflow ended, with ok, a summary and the cost. |
notice |
A line from the session for the user, never sent to the model, like "this model can only chat". |
The UI answers approvals with session.resolveApproval(), which calls the gate's resolve().
The idea is in 10-workflow-manager.md: do not make the model smarter, make each of its tasks smaller and check every task with real tools. The contract is workflow/types.ts.
tools/workflow-tool.ts: the start_workflow tool. It reads the spec from the notes, checks that the spec has the sections the recipe needs, builds a WorkflowScheduler and runs it inside the chat turn, so esc stops it. Its risk is run, because the user approves the plan instead. It puts the gate's mode back when the workflow ends.workflow/manager.ts: the manager. It fills a recipe into a plan with the submit_plan tool, and with the decide tool it chooses what to do with a task that failed 3 times (retry_with_hint, redo_task or ask_user). It does not use a session, only a few plain model calls. It never checks the plan.workflow/scheduler.ts: WorkflowScheduler, our code, no model. It gets the plan (a second try with the errors, then a default plan from recipes.ts), checks it with checkPlan(), asks the user once (gate.askPlan(), which puts the gate in plan-approved), and runs the tasks in dependency order with at most WORKFLOW_LIMITS.parallel workers at a time. The retry ladder: attempt 2 continues the worker session with the error digest, attempt 3 is a fresh worker with a short note, then the manager decides. A verify task runs up to WORKFLOW_LIMITS.fixRounds fix rounds. A failed task skips the tasks that depend on it.workflow/worker.ts: one task as a small AgentSession with its own prompt, tools, sandbox (allowWrite = the task's files), gate and a private bus. A worker gets only a few tools: read_file, list_files, load_skill, the write tools when the task writes, verify for a fix task, review for the statement critic, and finish_task. finish_task runs the task's real check first, so the worker can fix a failure in the same session.workflow/checkpoints.ts: CliCheckpoints runs the real check of each task kind with the compiler and the CLI (test validator --json, generate --all --json, validate --all --json, verify --json), and runs the tasks that need no model (scaffold, checker, generate_validate, verify and the prove_idea checks). MemoCheckpoints skips a check that already passed when the task's files and Config.json did not change.workflow/config-patch.ts: the single writer of Config.json during a workflow. Workers return a ConfigPatch, the scheduler applies the patches one at a time, and a patch that makes the config invalid is refused with nothing written. dropTemplateExamples() removes the template's examples in the scaffold task.workflow/recipes.ts: the recipes (build_problem, fix_verify, strengthen_tests, prove_idea, find_ideas): which task kinds a plan may have, which it must have, and the order rules. Also checkPlan(), addRecipeDependencies() and the fixed or default plans (buildProblemSkeleton(), fixVerifyPlan(), strengthenTestsSkeleton(), proveIdeaPlan()).workflow/prove.ts: the prove_idea checks (stress, samples, max test, kill wrong) on the quick programs in .polyman/agent/prove/. compile.ts compiles and runs them without the CLI.workflow/ideas.ts: IdeaWorkflow for find_ideas. It emits the same plan and task events, so the UI shows the same tree.workflow/spec.ts: parses the spec. The format is in design/agent/briefs/SPEC-FORMAT.md.workflow/prompts.ts: the prompts of the chat agent, the manager and each worker kind.Tool (tools/types.ts) in the file of its group, or a new file in src/agent/tools/: a snake_case name, a plain description, a JSON Schema in parameters, a risk, needsProblem, and describe() for the approval title and command.run(), pass every path through ctx.sandbox.resolve() and every polyman command through ctx.runner with --json. Return a short content for the model and the full data in data for the UI. Return isError: true for a failure, do not throw.createChatRuntime() in src/agent/index.ts. If workers need it, add it to the tools of modelSchedulerFactory() in tools/workflow-tool.ts and to workerToolNames() in workflow/worker.ts.remote.chatSystemPrompt() in workflow/prompts.ts.tests/agent/tools/, and a safety test in tests/agent/safety/ if it touches files or secrets.Plan['recipe'] in workflow/types.ts, and any new task kind to TaskKind.RECIPES in workflow/recipes.ts: allowed kinds, required kinds and order rules. If the plan has a fixed shape, write a plan function (like fixVerifyPlan()) and use it in makePlan() in scheduler.ts, so no model call is needed. Add a default plan if the manager can fail.CliCheckpoints.check() (worker kinds) or runNoModelTask() (no model kinds) in checkpoints.ts, its worker prompt and info in workflow/prompts.ts, its tools in workerToolNames(), and the kind lists in recipes.ts (NO_MODEL_KINDS, MUST_OWN_FILES).recipeDescription() in workflow/prompts.ts.recipe enum of start_workflow and to workflowSpecProblems() in tools/workflow-tool.ts, and tell the chat agent when to use it in chatSystemPrompt().tests/agent/workflow/ (recipes.test.ts, scheduler.test.ts, checkpoints.test.ts).npm test runs the agent tests with the rest. They need no key and no network.
MockProvider plays a script of model turns (text, tool calls, usage, errors). Most core, tool and workflow tests use it with createChatRuntime().tests/fixtures/agent/openai/ and tests/fixtures/agent/anthropic/ have HTTP responses in the providers' wire formats, close to what curl -i prints. The provider tests parse them.tests/agent/fixtures/fake-cli.mjs is a small fake polyman for the runner tests: it prints lines, sleeps, starts children and exits with a code.tests/agent/tui/ renders the components and the whole App with ink-testing-library and sends keys to it. store.test.ts tests the reducer alone.tests/agent/safety/ has path escapes, protected files, permissions, secrets, limits and bad tool calls.tests/agent/e2e/build-problem.test.ts runs a scripted model through the real chat runtime, tools, scheduler, checkpoints and dist/cli.js to build a problem in a temp folder, and the real polyman verify must pass. It needs npm run build and g++, and is skipped without them.The evals run the agent with a real model. They are not part of npm test. They need npm run build and g++. The probe, real, baseline and wf runs also need an OpenRouter key in OPENROUTER_API_KEY (only read from the environment, never written to a file). They copy corpus/vendor/testlib.h instead of downloading it.
They cost real money. A probe run is a few cents, but a real or baseline run can cost up to about $0.60 per case. Use the --max-cost option, and run one case before --all.
| Folder | What it does | Run |
|---|---|---|
evals/cases/ |
The test problems (pairs-sum, max-subarray, bracket-depth): a spec.md and a hidden reference/ (main, brute, wrong solutions, tests, answers) the agent never sees. check.mjs checks the cases themselves, with no model. |
node evals/cases/check.mjs [--case <id>] |
evals/probe/ |
One model call per case, no agent loop: does the model call the right tool with valid arguments? | node evals/probe/probe.mjs --runs 3 |
evals/real/ |
The whole build_problem workflow on a case, then verify and hidden checks against the reference. |
node evals/real/run.mjs --case pairs-sum |
evals/baseline/ |
The same build with one long chat session and no workflow, to compare with evals/real/. compare.mjs puts them side by side. |
node evals/baseline/run.mjs --case pairs-sum |
evals/broken/ |
Three copies of pairs-sum, each with one planted bug that verify catches. Used to test fix_verify and strengthen_tests, no model needed to run verify on them. |
see evals/broken/README.md |
evals/wf/ |
drive.mjs, a small script that runs the real build_problem workflow on one case. |
node evals/wf/drive.mjs --case pairs-sum |
Each folder's README.md has the options, what one run does and the last results. The results go to evals/*/results/, which is git-ignored.
See GUIDE.md - Directory Structure for the canonical layout of a Polyman problem directory and the bundled template tree.
git clone https://github.com/HamzaHassanain/polyman.git
cd polyman
npm install
npm run build
npm link
Entry Points:
src/cli.ts - CLI commandssrc/actions.ts - Action functionsCore Logic:
src/helpers/ - Domain modulessrc/executor.ts - Process executionsrc/formatter.ts - Output formattingAgent (polyman chat, polyman ai), see Agent Architecture:
src/agent/index.ts - startChat() and createChatRuntime()src/agent/core/ - Agent loop, event bus, context, system promptsrc/agent/tools/ - Tools, registry, permission gate, sandboxsrc/agent/runner/ - Runs the CLI as a child processsrc/agent/providers/ - Model providers and keyssrc/agent/workflow/ - Workflow manager, scheduler, workers, checkpointssrc/agent/tui/ - Ink terminal UI (ESM)evals/ - Evals with a real model, not part of npm testdesign/agent/ - Design docs of the agentTypes:
src/types.d.ts - TypeScript type definitionsAssets:
assets/checkers/ - Standard checkerstemplate/ - Problem templateNew Command:
src/actions.tssrc/cli.tsNew Helper Module:
src/helpers/src/actions.tsNew Solution Type:
src/types.d.tsgetExpectedCheckerVerdictsrc/helpers/solution.ts# Lint code
npm run lint
# Build
npm run build
# Unit tests (the agent tests need no key and no network)
npm test
# Manual testing
polyman new test-problem
cd test-problem
# Test commands...
See eslint.config.js for linting rules.
Key Rules:
_)any (warning)TypeDoc generates the full API reference: https://hamzahassanain.github.io/polyman/. Key entry points: src/cli.ts (commands), src/actions.ts (orchestration), src/polygon.ts (Polygon SDK), src/types.d.ts (types).
Important considerations for development, deployment, and platform-specific behavior.
ulimitLinux:
macOS:
Windows:
spawn with shell: true; sanitize user input if accepting external configsREADME.mdGUIDE.mddesign/agent/MIT License - See LICENCE file for details.
This technical documentation is auto-generated alongside TypeDoc API documentation. For the latest version, visit the online documentation.