Recoverable local worker jobs For an existing reusable InferOptimal endpoint and the built-in worker. Needs Node 22+. First use the endpoint's worker setup command to save its exact URL, model ID and key. Review the endpoint’s spending/runtime limits before running work. For default attach mode, Start it first. For owned mode below, leave it idle or stopped. Install the optional job client: mkdir -p ~/.inferoptimal curl -fsSL https://infraoptimal.com/codex/install-worker-jobs.mjs -o ~/.inferoptimal/install-worker-jobs.mjs node ~/.inferoptimal/install-worker-jobs.mjs This downloads both modules into an isolated versioned folder. It does not replace your main agent MCP helper or change the saved key in agents.json. Create tasks.json containing independent tasks, for example: [ {"task":"Read the auth code and report reproducible bugs; do not edit files.","cwd":"/absolute/path/to/repo"}, {"task":"Read billing code and report reproducible bugs; do not edit files.","cwd":"/absolute/path/to/repo"} ] Run (default is one worker; choose --in-flight explicitly): node ~/.inferoptimal/worker-jobs.mjs run --id security-review-01 --tasks tasks.json --time-limit-hours 1 --in-flight 2 Inspect or resume after the process exits: node ~/.inferoptimal/worker-jobs.mjs status --id security-review-01 node ~/.inferoptimal/worker-jobs.mjs resume --id security-review-01 Keep the process and computer running while it works. The output is JSON. Completed task results survive a restart. Interrupted tasks become unknown and are never silently replayed: inspect their files/tool effects before creating new work. Resume runs only tasks that never started and never extends the original deadline. The same ID with different tasks, settings, duration or concurrency is refused. When readiness is still pending, run resume again after the private GPUs are ready. Scope and spending By default, this client attaches to your existing endpoint. It does not call Start or Stop, increase the budget, create a GPU rental, schedule recurring work or publish results. The wall deadline limits worker execution; it is not a separate dollar spending cap. Stopping this local job does not stop GPU billing. Use the endpoint's Stop control when you are finished and no other application needs it. The endpoint's configured idle/wake and fallback policy still applies to inference. The client checks readiness and generation before each queued task. Those reads are not an exclusive compute lease or atomic fencing of every model request in a task. If someone stops/restarts the endpoint, queued work pauses; create a new reviewed job rather than silently moving the old job to a different generation. Task briefs and results are saved locally under ~/.inferoptimal/worker-jobs with private file permissions. The endpoint key is read from the existing local setup and is not written into the job journal. Treat briefs, files and tool results as sensitive; the journal is local recovery, not an exactly-once tool execution system. Optional private traces for your own workload Add --trace full when creating a new run to save the built-in worker's logical model requests/replies (including returned reasoning), tool calls/results and timings locally. Capture is off by default and cannot be toggled on an existing job. It adds no model output limit and never uploads a trace. Use it only for data you are authorized to capture; credential redaction is best effort. Each task attempt writes a separate private 0600 JSONL file and receipt under the job's traces folder (0700). Status shows trace completeness and capture errors. Resume preserves the original policy, never appends to an interrupted attempt, and never replays started work. A crash can leave an incomplete file without a final receipt. Capture failure is reported separately from task success. These are logical worker events, not raw streamed bytes or hidden provider internals. Aborted model calls cannot reconstruct a full answer. Tool results are exactly what the worker observed, including any normal tool clipping; source files outside those results are not automatically copied. Traces remain on your machine until you delete them, so review storage and retention for long jobs. Optional exclusive job lifecycle Only for an idle/stopped reusable endpoint configured with one private-only replica and serverless overflow off. This mode reserves the endpoint for this job; other applications cannot send inference until the job ends. Never use a shared customer endpoint for it. Review the current quoted rate and limits before authorizing it. node ~/.inferoptimal/worker-jobs.mjs run --id owned-review-01 --tasks tasks.json --time-limit-hours 1 --in-flight 2 --lifecycle owned --max-spend-usd 5 This is explicit authorization to allocate up to $5, bounded by the endpoint's existing accepted limits. The deadline is fixed when the journal is created and includes startup. Use at least 15 minutes. This example is not a quote: a rate or provider minimum that exceeds the allowance can prevent startup. The server persists ownership before requesting hardware. The client waits for private readiness and sends a separate secret job token with every model request. On completion/cancellation it ends only its own allocation. An owner can always Stop the service; an old job cannot stop a later generation. Lost Start replies reuse the same intent. Resume never increases the deadline or dollar allowance. The server drains/releases the allocation at the fixed deadline even if the local runner crashes. Provider cleanup may take time: cleanup=pending is not proof that billing ended. cleanup=confirmed requires a provider stop and settled charge. If the End request fails, the report says cleanup is unconfirmed; resume retries it. The job does not restart itself if its allocation fails. Create a new reviewed job. Owned mode retains the machine through tool waits until completion or deadline; the ordinary idle-stop setting does not interrupt an exclusive job. Saved URL/key survive cleanup. A private job-auth.json stores the job admission token; it is not included in reports. No recurring/hosted task scheduler is provided. Replenishing a read-only worker queue Use this when a coordinator has more work to submit during the same work session. It reuses the worker, journal, concurrency queue and endpoint ownership above. It is not a second Fleet optimization controller or a hosted recurring scheduler. Fleet still proposes/evaluates optimization candidates and controls promotion. Prepare an isolated source snapshot without credentials, outside the saved worker profile and journal. The queue refuses a root containing its own profile. Every task directory must resolve inside it. The queue forces builtin read-only tools: no shell, file writes, web add-on, forks or deployment actions. Task IDs are stable 8–80 character letters/numbers/hyphens/underscores starting with a letter or number. For example: [{"id":"auth-read-01","task":"Read auth flows; report evidence and uncertainties."}] node ~/.inferoptimal/worker-jobs.mjs run --id ongoing-review-01 --tasks tasks.json --queue-root /absolute/path/to/snapshot --queue-idle-seconds 60 --queue-max-tasks 100 --time-limit-hours 1 --in-flight 8 Use --lifecycle owned --max-spend-usd for exclusive ownership and compute cleanup. In attach mode, queue idle exit does NOT stop or cap billing. The original endpoint, root, concurrency, task allowance and deadline are fixed. The ordinary queue requires a reusable service URL. Raw deployment URLs need the explicit externally supervised attachment described below; they cannot claim a reusable service generation or ownership lease. While the runner is alive, another local process can submit more work: node ~/.inferoptimal/worker-jobs.mjs append --id ongoing-review-01 --submission-id billing-batch-01 --tasks more-tasks.json [{"id":"billing-read-01","task":"Inspect billing cancellation accounting."}, {"id":"auth-summary-01","task":"Summarize accepted auth findings.","depends_on":["auth-read-01"]}] Submission returns submitted, not executed. Inspect status.queue.receipts for accepted or rejected. Retrying the same submission/task ID with identical content is idempotent; changed content is refused. Each queue accepts at most 128 immutable commands and 1,000 total tasks (or the smaller original maxTasks). Each command is at most 1 MiB. Dependencies must refer to earlier tasks; forward edges/cycles are refused. Running tasks keep their slots while the queue admits more independent work up to the original in-flight limit. Completed results appear in review_queue. A coordinator must inspect them before accepting a dependency; a completed model answer is not proof that it is correct: node ~/.inferoptimal/worker-jobs.mjs review --id ongoing-review-01 --submission-id auth-review-01 --task-id auth-read-01 --decision accepted Use rejected for unsupported results. Review decisions cannot be reversed in the same job. Dependents run only after all parents completed and were accepted. They receive the retained reviewed results as source data. If their combined brief exceeds 100,000 characters, the task fails visibly; explicitly prepare a reviewed summary instead of silently dropping evidence. Failed/unknown/rejected parents never authorize dependent work. This local review is not permission to merge or publish: production changes remain with the operator and normal release checks. Idle time is measured after useful dispatched work finishes. If there is no runnable work for the configured 1–300 seconds, the queue closes, retaining blocked work/results. The fixed wall deadline also closes it. Explicit cancellation or an endpoint fence error closes admission. Owned mode ends its allocation; cleanup errors remain visible. No new generation is rented by resuming a closed queue. Review submissions remain possible after closure; resume applies these reviews locally and retries existing owned cleanup, without executing more tasks. After a process crash, resume recovers unfinished commands. Tasks that had been claimed are unknown and never replayed; untouched independent tasks may run within the original deadline and endpoint generation. A single runner lock and atomic journal/inbox protect local ownership. The computer must remain awake. This is bounded at-most-once task dispatch, not exactly-once model/tool execution or an automatically supervised daemon. Monitor cleanup and service status independently. Operator-only attachment to an existing native allocation For an already allocated exact native endpoint, add --native-allocation-dir to the same queue command. This path is externally supervised: it never provisions, renews, replaces or Stops compute. The existing allocation owner and watchdog do that. It does not accept platform/cloud credentials. Use an isolated INFEROPTIMAL_HOME with the exact native URL/model/key and endpointDeadlineAt saved in its agents.json; the source snapshot must exclude that profile and allocation receipt directory. The directory must contain state.json, provider-cleanup.json, protocol-result.json and readiness-execution.json for the exact approved request, provider app, URL, model and source. State must retain noReplacement, supplier cap, watchdog ID and budget authority. Profile deadline must leave at least 300 seconds before stopDeadline for cleanup. Protocol success must explicitly bind deploymentId, appId, baseUrl and model and pass both streaming and a tool round trip. The external owner also maintains native-observation.json from fresh provider and native-health observations, marking failure/stopping explicitly. Required fields: {"schema":"inferoptimal.native-worker-observation.v1", "deploymentId":"req_<24 hex>","appId":"ap-", "baseUrl":"https://ep-<10 hex>.inferoptimal.com/v1","model":"", "source":"<40 hex source revision>","observedAt":, "expiresAt":,"ready":true,"state":"running"} Before each task, the runner checks that fresh observation and unchanged binding, reads the live lease using the existing endpoint-key /console/agent-context API, requires /health ready and the exact /v1/models entry, then rechecks freshness. Redirects are refused. Missing/expired/failed evidence admits no task. A shorter live lease shortens execution; it never extends the original queue deadline. These are allocation/readiness checks, not proof of a particular process incarnation: the native runtime does not expose a process-generation identifier. Unlike an owned reusable queue, native idle grace switches to waiting_review when a result awaits acceptance, or waiting_work when none does, and keeps the process alive for new submissions within its original deadline. It does not close an eight-hour allocation just because the coordinator is reviewing. Process crash recovery still never replays claimed/unknown work. At the deadline, this runner closes its queue; the external watchdog must confirm provider cleanup. The report names external_watchdog ownership and never invents a service generation. Literature review preset (local text corpus) Use your main agent to collect papers and convert PDFs to text first. This preset does not search the web, fetch PDFs or claim exhaustive literature coverage. Put papers.json beside the source text files: {"question":"When does prefix caching improve agent throughput?","papers":[ {"id":"paper-a","title":"Paper title","path":"paper-a.txt","url":"https://example.org/paper"}, {"id":"paper-b","title":"Another paper","path":"paper-b.txt"} ]} Start your existing endpoint, then run these phases in order. Each invocation requires its own explicit time allowance. These commands do not rent or stop GPUs. The endpoint's spending limit still applies; elapsed phase limits are not a dollar budget and do not stop GPU billing. Stop the endpoint when no other work needs it. node ~/.inferoptimal/worker-jobs.mjs literature --phase extract --manifest papers.json --id papers-extract-01 --time-limit-hours 0.25 --in-flight 4 node ~/.inferoptimal/worker-jobs.mjs literature --phase verify --manifest papers.json --id papers-verify-01 --extract-id papers-extract-01 --time-limit-hours 0.25 --in-flight 4 node ~/.inferoptimal/worker-jobs.mjs literature --phase synthesize --manifest papers.json --id papers-synthesis-01 --extract-id papers-extract-01 --verify-id papers-verify-01 --time-limit-hours 0.25 --in-flight 4 For large evidence sets, synthesize produces bounded partial reviews in parallel. Inspect status: if there is more than one synthesis task, run the explicit combine phase (with its own allowance) to produce the cross-corpus review: node ~/.inferoptimal/worker-jobs.mjs literature --phase combine --manifest papers.json --id papers-combined-01 --extract-id papers-extract-01 --verify-id papers-verify-01 --synthesis-id papers-synthesis-01 --time-limit-hours 0.25 --in-flight 1 Extractors read one paper each and retain claims with exact passages and line numbers. The next phase first rejects quotations that do not match the saved text, then uses a fresh worker conversation per paper to judge whether each retained passage supports its claim. Synthesis compares supported findings, contradictions, conditions and gaps. A quote match proves that text exists, not that a conclusion is correct; independent checks are still model judgments. Review the final draft. New synthesis and combine jobs validate the returned JSON before marking a task completed: bounded review/questions, every expected evidence ID or numeric part exactly once, and citations referring to existing source IDs and line ranges. Reviews with supported claims need a citation; empty evidence can report no findings without inventing one. These checks establish structure and valid ranges, not that every sentence follows from its citations. Malformed output is retained with literature_output_invalid; resume does not silently rerun it. Inspect it and use a new job ID for an intentional retry. Older journals without this validation contract retain their old status; completed does not retroactively certify them. All phases force the built-in worker into read-only local tools: no shell, file writes or web tools. Paper text is treated as evidence, never as instructions. The model can read only the snapshotted corpus folder. Source snapshots, briefs and results are private local files under ~/.inferoptimal. No settings/key are changed. Source IDs, titles and URLs establish provenance; final synthesis cites source IDs and line numbers. Keep the manifest with the result so readers can find originals. Use status/resume with each phase's ID exactly as above. Completed tasks are reused; unknown interrupted tasks require inspection. Changed sources create a new corpus and cannot consume an older extraction. Invalid JSON, missing verdicts or oversized combined summaries fail visibly rather than silently skipping papers. Evidence is split into bounded synthesis shards without dropping claims. Combining checks that every shard accounts for each evidence ID, including rejected claims; that coverage check does not prove the prose is complete or scientifically correct. Limits are 100 papers/review, 256 KB text/paper and 12 claims/paper; split larger questions. With Claude or Codex coordinating, ask it to run these phases and inspect the findings between them. Without a main agent, run the same phases yourself (combine when needed): the endpoint model extracts, audits and synthesizes; the fixed workflow coordinates it. This is not autonomous scientific judgment or a new self-directing swarm. If a model exhausts its output-token budget, the task fails with output_budget_exhausted, finish_reason length, partial visible output and reported usage. Reasoning tokens consume that budget too. It is not an empty successful answer; truncated tool calls are not executed and the request is not replayed. Inspect the result and explicitly retry with an appropriate budget when justified.