Model service and judge

The model service Oko asks#

Every model call Oko makes as itself — the independent judge, the reading of operator messages, brama complete and oko gateway send --prompt — asks Brama as Oko by default. A machine without Brama points them at any OpenAI-compatible chat-completions service instead — OpenAI, OpenRouter, a local server — so tasks are judged where Brama is not installed:

printf '%s' "$KEY" | oko model-service set --origin https://api.openai.com --model gpt-5
oko model-service show
oko model-service clear

set reads the service's API key from standard input only and keeps it owner-only in ~/Library/Application Support/Oko/judge-provider.json (OKO_JUDGE_PROVIDER_PATH names another file); the key is never printed. Every verb prints the choice that stands afterwards as JSON, or with --plain as key: value lines: provider (brama or openai-compatible), model, and for a declared service origin and declaredIn. Oko then asks that service for that model alone; oko tasks judge --probe and every verify pass use it. Refusals: an --origin that is not HTTPS, or HTTP on loopback, with no user, query or fragment; an empty --model; an empty standard input; a declaration missing its origin, model or key, which names the file and set/clear; and clear when nothing is declared (no model service is declared in <file>; Oko already asks Brama). The group was oko tasks judge-provider with unset; it chooses for every model call, not only the judge's, and clears the way oko contact clear does.

Desktop's Model access → Judge provider shows the same choice and supports Save provider and Use Brama, each with confirmation of its scope. The API key is entered in a secure field and is never read back. The form clears it after submission or when leaving the screen. A failure retains the last read choice with the actual refusal; it does not present that choice as a fresh read. The native API uses tasks.judge_provider with a typed operation object: {"action":"show"}, {"action":"set","origin":"https://api.openai.com","model":"gpt-5","key":"<API key>"}, or {"action":"unset"}. These operations share the CLI implementation and need neither Brama nor a database connection to configure the choice. Saving does not call a model or invalidate the fact that an older observation was recorded. Use Probe judge to obtain an observation of the new choice.

The same declaration serves every other model call Oko makes as itself: oko gateway send --prompt TEXT (unless --router URL names a gateway, and with the declared model unless --model names another), and the completions behind assignment estimates, context advice, goal evals and suggestions. Tama's model-judged guards ask through oko gateway send --prompt, so on a machine without Brama they are answered by the declared service too. A refused call names the service's origin: <origin> answered no route asked: <route>: HTTP <status>: <answer>.

An explicitly selected OpenAI-compatible service always receives POST /v1/chat/completions, including when its model is named decision-model. Only Brama's own decision alias uses /v1/decisions. A successful HTTP status alone does not make the judge healthy: a chat probe must return nonblank message text, and a typed probe must return the requested ready.noul probability between zero and one. An empty message or empty answers object is reported as returned no usable probe answer; no successful health observation is recorded for it. A successful probe proves model access, not completion of any task. CLI verification and Desktop's Verify current result use the same judge.

oko gateway check does not ask a model. It reports credential reads, bearer acceptance and the gateway's readiness observation; --json prints the same report, and the command exits 0 only when the gateway accepts Oko's bearer. An attempted readiness failure retains its operation, HTTP status and response in gatewayFailure; it is not reported as a request never made. Agent routing and automatic sign-in failures remain visible. Desktop's Model access screen shows the same diagnostics separately from Task judge. Opening or refreshing the screen reads saved judge health without calling a model. Probe judge explicitly asks the selected service through Oko's resident API and shows the recorded observation or the actual failure.

A probe returns its own observation. It does not read an older healthy row and call that a successful request. Both the shared observation and local gate copy are attempted. A failed write identifies its destination, cause and whether the other write succeeded; the command or API operation fails even when the model answered. A probe is not a task-completion verdict.

Task decisions must satisfy the complete reply contract: one JSON object, an exact declared verdict label, a nonblank reason, and a string list of nonblank missing items. done has no missing items; not_done names at least one. Oko does not recover a verdict from broken JSON, prose, a code fence or an isolated label. A typed decision must supply confidence between zero and one. Unreadable decisions save no verdict and make CLI and native verification fail, while retaining any valid decisions already saved in the pass. Earlier saved verdict bindings are reassessed under this complete-response contract. A later operator statement still takes precedence.

Model failures name the selected service address. Only Brama's own refusals include its grant or subscription guidance. A successful probe also records the service address with the model name, so its history identifies what was actually asked.

Real judge verification

tests/judge/journey.rs in the Oko repository exercises a released CLI against the real model and shared database. It requires a separate home signed into a dedicated test account, that account's organization, and fresh real task sessions: at least one completed request and one incomplete request. It refuses the operator's account and previously judged results. No provider, transcript result or database response is simulated.

Run the journey with OKO_JUDGE_TEST_CONFIGURATION naming a JSON file, then:

cargo test --manifest-path rust/Cargo.toml --test judge-journey -- --nocapture

The configuration fields are release (the directory containing oko, SOURCE_REVISION and SHA256SUMS), test_home, organization, optional provider (a real judge-provider.json), cases, and one_off_cases. Each task case contains session, task and expected (done or not done). Prepare fresh results through the real harness before each run; do not write verdicts to prepare the cases. To exercise an explicit service whose model is named decision-model, supply its real provider declaration.

Each one_off_cases entry contains a real session, tool-use key, and expected: either null for quoted or help-only text, or an object containing category, outcome and followed_by_commit. Include repair attempts with and without a completed later commit, and a quoted non-execution case. Each session needs a positive repair control so an unindexed session cannot pass as a negative result. These must be actual isolated harness records, not fabricated transcripts.

The journey probes the model, reads the persisted health from another CLI process, checks the local gate copy, and refuses an unsafe HTTP credential destination without changing the selected provider. The repair journey under tests/oneoff/ checks the case identities against real Transcript Lake events, runs oko index sync, reads persisted classifications from separate processes, and checks meaningful report refusals. When the real Lake holds a replayed operator event, it also compares the saved catalogue's operator-message count with the distinct events rather than the inflated raw row count. No synthetic duplicate is inserted into operator data. The judge journey then verifies both task outcomes against separately read saved verdicts. It retains source revision, binary checksum, commands, exit statuses and observations under .build/judge/. tests/judge/persistence.rs also makes a real gate-copy write fail against an isolated directory, verifies that the original copy is unchanged, and checks the next valid probe's recovery. Observation times are compared as instants, not by the spelling of their UTC offset. tests/judge/native.rs starts the same released binary as an isolated resident service using the dedicated test account. It probes through the native API, reads the saved result through the CLI and local gate file, and checks that an invalid probe request does not change history. Its linked report under .build/api-*/ retains the service identity, exchanges and actual exit state. With a real external provider, the native journey also sets that service in an isolated declaration, checks owner-only permissions and CLI readback, refuses an unsafe origin and empty key without changing the file, removes it, and restores it through the CLI before the resident service probes the model. It removes the isolated declaration afterward; the supplied credential file is never changed. Native request reports redact the key, and an answer exposing the submitted key fails before that answer is retained. A Brama-only run records that the external-provider lifecycle was not exercised; both configurations are needed to qualify both provider modes. When Brama is selected, tests/judge/readiness.rs also reads the real gateway, requests an unsupported path on it, exercises a refused TCP connection, and checks an invalid origin. It distinguishes an HTTP refusal, a transport failure and a request not attempted, without a fake HTTP service. Each command's origin override is retained in the report. These Brama checks do not apply when the configuration explicitly selects an external judge provider. A missing account, release or live case fails the run; it is not a skipped or passing test.