## Run a command

`POST /v1/infra/servers/:id/exec`

Runs a shell command on a BLACKHOLE server through the tunnel agent. The full standard output (`stdout`), error stream (`stderr`), and return code are passed to the client. By default the response is JSON, returned after the command finishes (recommended for AI agents and scripts). On `STANDALONE`, if you need line-by-line progress, pass `?stream=true` and read the `stdout`/`stderr`/`exit` SSE events. A failure during command execution arrives in streaming mode as an `error` event.

**For AI agents and MCP clients, always use JSON mode (without `?stream=true`)** — they cannot parse SSE. The official Vibecode MCP client (the `manage_server_deploy` tool) drops this parameter itself and always works in JSON mode.

Streaming gives you live progress, but it does **not** raise the execution time limit: the synchronous execution limit is the same in JSON and streaming modes and equals `timeout + 30` seconds, while `timeout` itself is capped at 600 seconds. On `STANDALONE`, for longer work with a retained result, use `async=true` — see "Durable exec" below. A client-managed background job is another option.

## Which servers it runs on

The server kind arrives in the `kind` field of the [`GET /v1/infra/servers`](/docs/infra/servers/list) and [`GET /v1/infra/servers/:id`](/docs/infra/servers/get) responses.

| `kind` | Command runs | What to keep in mind |
|--------|--------------|----------------------|
| `STANDALONE` — a standalone virtual machine | yes | The contract of this page applies in full, including `workdir` and `env` |
| `GALAXY_APP` — an application in a galaxy | yes, inside the application container | The `workdir` and `env` fields are not supported — `400 GALAXY_EXEC_NO_WORKDIR_ENV`. Use `cd /opt/app; FOO=bar node script.js` only for non-secret values. Load secrets inside the container from an existing protected file or the application’s configured secret store; keep their values out of command text. Failures during execution arrive with status `502`, not `200`. There are two exceptions: `EXEC_BUSY` arrives as `409`, and `EXEC_NO_EXIT` arrives as `200` with `success: false`, because it is not a channel failure but a completed call with no verdict (see below). Only synchronous exec is supported, with timeout up to 600 seconds. `async=true` receives `400 DURABLE_EXEC_UNSUPPORTED_SERVER_KIND` without execution or fallback. |
| `GALAXY` — the machine carrying application containers | no — `403 GALAXY_HOST_EXEC_FORBIDDEN` | The machine carries containers belonging to different keys of a single Bitrix24 account, so it is not a command target. Run the command on the application, by its own ID (`kind` equals `GALAXY_APP`). For the machine's free disk space, look at the galaxy card in your Vibecode account — see [Galaxy app](/docs/infra/galaxy). Durable requests receive `400 DURABLE_EXEC_UNSUPPORTED_SERVER_KIND` before dispatch. |

## Parameters

| Parameter | In | Type | Required | Default | Description |
|----------|---|-----|:-----:|-----------|----------|
| `id` | path | string (UUID) | yes | — | BLACKHOLE server ID, `status: running`, `blackholeStatus: CONNECTED` |
| `stream` | query | string | no | — | `true` — synchronous SSE mode on `STANDALONE`. Omit for JSON. Galaxy exec returns blocking JSON without a live SSE stream |
| `async` | query | string | no | `false` | `true` — standalone durable execution with a retained result and a `202` response. Other kinds: `400 DURABLE_EXEC_UNSUPPORTED_SERVER_KIND`, without synchronous fallback. Cannot be combined with `stream=true` |

## Request fields (body)

| Field | Type | Required | Description |
|------|-----|:-----:|----------|
| `command` | string | **yes** | Shell command. 1–10,000 characters |
| `timeout` | number | no | Execution timeout in seconds: 1–600 synchronously; 1–86400 with `async=true` on `STANDALONE` only. Default 300 |
| `workdir` | string | no | Standalone only. Up to 500 characters. Must be an existing directory whose cleaned absolute path is `/opt`, `/srv`, `/tmp`, `/root`, `/var/log`, `/var/lib`, `/etc/systemd`, `/etc/nginx`, or below one of them. Relative paths, remaining `..` in the cleaned path, and sensitive prefixes listed below are denied. Omit to use the execution process’s default directory |
| `env` | object | no | Environment variables: `{ "KEY": "value" }`. Only string values |

**OpenAPI validation.** The standard `requestBody` permits `timeout` up to 86400 seconds so generic validators and generated clients accept supported `async=true` requests on `STANDALONE`. The body schema does not account for the `async` query parameter: the synchronous API still rejects `timeout` above 600 seconds even when the standard schema accepts the body. The mode-specific contracts are `#/components/schemas/InfraSyncExecRequest` (up to 600 seconds) and `#/components/schemas/InfraDurableExecRequest` (up to 86400 seconds). The `x-vibe-request-schema` extension validates assembled `{query,body}` by mode; server kind is checked at runtime.

The `x-vibe-request-schema` extension validates the assembled `{query, body}` object: mode-dependent timeout bounds and the allowed `async`/`stream` combination. The API checks server kind at request time; choosing the durable body schema does not remove the `STANDALONE` prerequisite.

For `workdir`, path cleaning precedes validation. The following prefixes are denied even inside an allowed root: `/etc/shadow`, `/etc/passwd`, `/etc/sudoers`, `/etc/cron`, `/etc/ssh/`, `/root/.ssh/`, `/root/.gnupg/`, `/root/.bashrc`, `/root/.profile`, `/root/.bash_history`. A denied path causes `EXEC_WORKDIR_DENIED`; an unavailable directory causes `EXEC_WORKDIR_NOT_FOUND`, and a non-directory causes `EXEC_WORKDIR_NOT_DIR`. Durable admission can refuse a denied path with HTTP `502` before launch; an accepted job reports later directory checks through its operation. See [durable outcome refusals](./operation-status.md#durable-exec-refusal-codes).

## Examples

### curl — personal key

```bash
curl -X POST "https://vibecode.bitrix24.com/v1/infra/servers/SERVER_ID/exec" \
  -H "X-Api-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"command": "ls -la /opt/app", "timeout": 30}'
```

### curl — OAuth application

```bash
curl -X POST "https://vibecode.bitrix24.com/v1/infra/servers/SERVER_ID/exec" \
  -H "X-Api-Key: YOUR_APP_KEY" \
  -H "Authorization: Bearer USER_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"command": "npm ci --production", "workdir": "/opt/app", "timeout": 180}'
```

### JavaScript — personal key

```javascript
const res = await fetch(
  `https://vibecode.bitrix24.com/v1/infra/servers/${serverId}/exec`,
  {
    method: 'POST',
    headers: {
      'X-Api-Key': 'YOUR_API_KEY',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      command: 'node -v',
      timeout: 10,
    }),
  }
)
const { data } = await res.json()
console.log(`exit ${data.exitCode} in ${data.duration}ms:\n${data.stdout}`)
```

### JavaScript — OAuth application

```javascript
// A command with environment variables and a working directory
const res = await fetch(
  `https://vibecode.bitrix24.com/v1/infra/servers/${serverId}/exec`,
  {
    method: 'POST',
    headers: {
      'X-Api-Key': 'YOUR_APP_KEY',
      'Authorization': 'Bearer USER_SESSION_TOKEN',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      command: 'npx tsx scripts/seed.ts',
      workdir: '/opt/app',
      env: { DATABASE_URL: 'postgresql://localhost/mydb' },
      timeout: 60,
    }),
  }
)
```

## Response fields

| Field | Type | Mode | Description |
|-------|------|------|-------------|
| `success` | boolean | Both | Synchronous `200`: `true` if the command ran, including a nonzero exit code. Durable `202`: accepted for reconciliation; this does not prove launch |
| `data.exitCode` | number | Synchronous `200` | Process exit code |
| `data.stdout` | string | Synchronous `200` | Standard output |
| `data.stderr` | string | Synchronous `200` | Error stream |
| `data.duration` | number | Synchronous `200` | Execution time in milliseconds |
| `data.truncated` | boolean | Synchronous `200` | Whether output reached the shared 1 MiB (1,048,576 bytes) stdout+stderr limit |
| `data.operationId` | string | Both | Run identifier; on refusal use `error.operationId`. Required in durable `202`: no dispatch without a stored record. Synchronous execution can continue without it if recording fails |
| `data.status` | string | Durable `202` | `running`: launch acknowledgement may be unconfirmed; poll the same operation |
| `data.statusUrl` | string | Durable `202` | URL for reading this operation and its retained result |

The durable `202` response contains no command output or exit code. Read retained fields through the authorized [operation outcome](./operation-status.md) endpoint.

## Response example

```json
{
  "success": true,
  "data": {
    "exitCode": 0,
    "stdout": "Linux epd65hdv07p8g89c0c06 6.8.0-107-generic #107-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 13 19:51:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux\n",
    "stderr": "",
    "duration": 5,
    "truncated": false,
    "operationId": "clz9k2m4x0001qw8h3f7d2n5q"
  }
}
```

Standalone durable command accepted (`202`):

```json
{
  "success": true,
  "data": {
    "operationId": "clz9k2m4x0001qw8h3f7d2n5q",
    "status": "running",
    "statusUrl": "/v1/infra/operations/clz9k2m4x0001qw8h3f7d2n5q"
  }
}
```

## Error response example

409 `EXEC_BUSY` — exec admission is occupied. A platform-lock refusal returns the retry signals shown in this example:

```json
{
  "success": false,
  "error": {
    "code": "EXEC_BUSY",
    "message": "Another operation is running on this server",
    "retryable": true,
    "retryAfter": 10
  }
}
```

When the response includes `retryable: true` and `retryAfter`, it also carries the `Retry-After` HTTP header with the same interval in seconds. Follow that interval; `error.hint.autoExpiresInSeconds`, when present, gives the lock's full upper bound. A durable agent refusal instead returns `error.operationId` without `retryable`, `retryAfter`, or `Retry-After`. Read that operation before another exec request; an unknown outcome requires reconciliation and independent confirmation that the command has stopped.

## Errors

The endpoint's errors fall into two groups, and they are delivered differently. A failure before the command starts arrives with a real HTTP status. A failure during execution arrives in the response body on a standalone virtual machine (`kind: "STANDALONE"`). On a Galaxy application (`kind: "GALAXY_APP"`) it arrives with the status `502`, and there are two exceptions there: a busy exec channel (`EXEC_BUSY`) arrives as a real `409`, because it is a "busy" refusal rather than a failure, and `EXEC_NO_EXIT` arrives with `200` and `success: false`.

### Errors before the command starts

| HTTP | Code | Description |
|------|-----|----------|
| 400 | `VALIDATION_ERROR` | The request schema is violated (empty command, invalid timeout, `workdir` longer than 500 characters, invalid `async` value or `async=true` combined with `stream=true`). |
| 400 | `DURABLE_EXEC_UNSUPPORTED_SERVER_KIND` | `async=true` targets a server whose kind is not `STANDALONE`, including `GALAXY_APP` and `GALAXY`. Refused before locking, operation creation or command dispatch; no operation ID or synchronous fallback. Galaxy remains available through a separate synchronous request with timeout up to 600 seconds |
| 400 | `NOT_BLACKHOLE` | Server in OPEN mode — the Deploy API is unavailable |
| 400 | `COMMAND_TOO_LONG` | The command is longer than 10,000 characters. On a standalone server, send large payloads and scripts via [`/upload`](./upload.md), then run the script: `bash /path/script.sh`. For a Galaxy host, `/upload` returns `GALAXY_HOST_NOT_A_DEPLOY_TARGET`. For a Galaxy app, it returns `GALAXY_APP_USE_GALAXY_ROUTE`. Include app files in the deploy archive through the `source` field of `POST /v1/infra/servers/:id/deploy` |
| 400 | `GALAXY_EXEC_NO_WORKDIR_ENV` | A command for a Galaxy application (`kind: "GALAXY_APP"`) carried `workdir` or `env`. Container execution does not accept them — use `cd` and non-secret assignments such as `FOO=bar`; load secrets from an existing protected file or the application’s configured secret store inside the container |
| 401 | `MISSING_API_KEY` | The `X-Api-Key` header was not provided |
| 401 | `INVALID_API_KEY` | The key is not recognized — no such key exists on the platform |
| 402 | `ACCOUNT_FROZEN` | The Vibecode balance is frozen. The request is rejected before the operation and a sleeping server is not woken — top up the balance and repeat |
| 403 | `SERVER_WAKE_BLOCKED` | The server is asleep and waking is blocked — for example, platform access has ended or an administrative block is in place. A command does not bring such a server up, and a retry will not help until the block is lifted. For details on the block, see [Wake a server](/docs/infra/lifecycle/wake) |
| 403 | `WRONG_KEY` | The server exists but is not yours. Operations on the application's content — deploy, exec, upload and log reads — are open on any one of three grounds: the server's managing key, a key whose application is bound to that server (`Application.serverId`), or membership in that server's development team (both roles, Developer and Administrator). Access tokens and the icon upload require the managing key. Managing the machine itself is also open to the Administrator role on the team. The response carries a `hint` with a two-step recovery. See [Server access recovery](/docs/infra/server-access-recovery) |
| 403 | `GALAXY_HOST_EXEC_FORBIDDEN` | The command was sent to the ID of a galaxy host machine (`kind: "GALAXY"`). Run it on the application, by its own ID — see "Which servers it runs on" above |
| 403 | `INFRA_FORBIDDEN_FOR_COWORK_KEY` | The call was made with a Cowork key — such a key works with data only and cannot perform write operations. To issue a key that can, see [Project key for deploy](/docs/cowork/deploy-key) |
| 403 | `EXTERNAL_COLLABORATOR_KEY_SERVER_MISMATCH` | The call was made with an external-collaborator key, and the address names a server other than the one the key was issued for. Such a key is valid for exactly one server, and the response does not reveal whether the requested server exists. Working with another server requires the key issued for that external membership |
| 403 | `EXTERNAL_COLLABORATOR_NOT_A_MEMBER` | An external-collaborator key addresses its own server, but access is not granted right now: the membership was removed, it expired, or external development collaborators are switched off for the Bitrix24 account that owns the server. The code does not tell these reasons apart and reveals nothing about the server. Membership is re-read on every request, so revoked access stops working immediately |
| 404 | `NOT_FOUND` | There is no server with this `id`, or it was deleted. A server that exists but is not yours answers `403 WRONG_KEY` — the code tells "no such server" apart from "no rights to it" |
| 409 | `EXEC_BUSY` | A sleeping server with a live plan-change operation (`resize`) returns `409 EXEC_BUSY` before waking; wait for that operation to finish. Another command or deploy holds admission. A platform lock refusal carries `Retry-After` and `retryable: true` / `retryAfter`; follow that interval. A standalone durable start refused by the agent also returns `409` with `error.operationId`, but without these retry fields. Read the operation before retrying. A running standalone durable job normally keeps the channel busy after `202`, without `autoExpiresInSeconds`; poll its result instead of calling `/unstick` or deleting the lock. On Galaxy the channel is shared by every application and deploy on the host. For an unknown outcome, reconcile state and logs and contact support; an absent `error.hint.holder` does not prove the host is idle. Lock recovery is covered by [`/unstick`](/docs/infra/servers/unstick) and [`DELETE /lock`](./lock.md) and requires independently confirmed idle state. Galaxy application owners cannot force recovery |
| 409 | `SERVER_NOT_READY` | The server is not ready for the operation: it is not running, the tunnel is not connected, or a successful non-empty Gateway snapshot confirms that a tunnel stored as `CONNECTED` is absent. This response is returned only when repair is not confirmed successful; an unavailable or empty snapshot does not confirm absence and the API uses the stored status. The platform attempts to restore a confirmed-missing tunnel before starting the command; if it cannot, the response carries a `hint`. This code does not describe a sleeping server — the platform wakes that one itself. Restore the tunnel with [`/repair`](/docs/infra/lifecycle/repair), then repeat the request. A sleeping server with a live plan-change operation (`resize`) returns `409 EXEC_BUSY` before waking; wait for that operation to finish. |
| 409 | `WAKE_IN_PROGRESS` | Another request is already waking this sleeping server. Wait for it to finish and repeat |
| 422 | `VM_MISSING` | The server record has no virtual machine on the provider side, so there is nothing to wake. Replace the server: delete it and create a new one. `error.hint` names the endpoint of each step only if that step is available to this request, and for an unavailable step it gives the reason for the refusal and what to change. The `error.userMessage` field arrives only when both steps are available |
| 422 | `VM_POINTER_LOST` | The record has no `externalId`, but a cloud scan does find a machine under this server's cloud name: the link is lost while the machine itself is most likely running and billed. Do NOT delete the server and create a new one — that would leave the machine running with no record at all. The server is not moved to `error`; contact support to have the link restored |
| 502 | `WAKE_FAILED` | Server readiness could not be confirmed. This response does not mean that the server was stopped. Check the [current server state](/docs/infra/servers/get) before taking another action |
| 502 | `PROVIDER_ERROR` | The cloud provider returned an error while starting the sleeping server's virtual machine. The tunnel is not involved here, so [`/repair`](/docs/infra/lifecycle/repair) will not help |
| 409 | `GALAXY_APP_NOT_READY` | Galaxy applications only (`kind: "GALAXY_APP"`): the platform did not resolve the name of the application's container on the host and did not run the command. Two conditions. The container name failed validation — `message` then reads `invalid on-host name`. Or the application has no subdomain or no link to its galaxy host — `message` then reads `Galaxy app is missing its subdomain or host link`. Read the application's current state with [`GET /v1/infra/servers/:id`](/docs/infra/servers/get) |
| 429 | `RATE_LIMITED` | The limit of 10 operations per minute per server was exceeded. The current limit is returned in the `x-ratelimit-limit` header |
| 503 | `EXEC_JOB_STORAGE_UNAVAILABLE` | A durable operation record or its target could not be stored; no command was dispatched |
| 502 | `EXEC_JOB_CAPACITY` | Durable admission reached 64 retained job directories on the host. The command did not start. Read `error.operationId`; wait for eligible seven-day retention cleanup or contact support. No automatic resend |
| 502 | `EXEC_JOB_UNAVAILABLE` | Durable storage, admission or systemd launch is unavailable. Read `error.operationId` and contact support; do not substitute a synchronous execution |
| 502 | `EXEC_JOB_INVALID` | The agent rejected invalid durable launch data. The command did not start. Read `error.operationId`; check the recorded refusal and contact support if a valid request keeps failing |
| 502 | `EXEC_JOB_START_WINDOW` | The server clock disagrees with the platform clock, so the launch window fell outside what the agent accepts. The command did not start. Read `error.operationId`, check the time on the server and retry; contact support if a valid request keeps failing |
| 502 | `EXEC_WORKDIR_DENIED` | The standalone working directory violates the path policy above. No command ran. Read `error.operationId`, then choose an allowed existing directory |
| 502 | `EXEC_JOB_REJECTED` | The agent refused before execution without naming a code. Read the failed operation and contact support; do not substitute a synchronous launch |
| 502 | `UNKNOWN_TYPE` | The agent does not support durable exec; command refused before launch, without synchronous fallback |
| 503 | `WAKE_NO_CAPACITY` | Startup has already been refused for lack of capacity. Read the [current server state](/docs/infra/servers/get) before trying again |
| 503 | `WAKE_TIMEOUT` | The sleeping server did not come up within the allotted 6.5 minutes — either the machine never started or the tunnel never connected. The server returns to `sleeping`, so a repeat request is safe. See "Known specifics" |

### Refusals after durable acceptance

HTTP `202` can precede a refusal detected before the user command starts. Poll the returned operation; `status: "failed"` and `data.error.code` identify a proven refusal, while `unknown` does not authorize another exec. An expired launch window or a refused working directory is described in the [durable refusal table](./operation-status.md#durable-exec-refusal-codes). An unknown outcome has `data.error: null`; do not infer success from the absence of an error code.

### Errors during synchronous execution

> **The HTTP status does not reflect the command's outcome.** On a standalone virtual machine (`kind: "STANDALONE"`) the connection is held open for the whole run, and the `200` status is sent before execution begins — so both a successful and a failed command arrive with the code `200`. The failure marker is the `success: false` field and the `error` object in the response body. **Check `success` in the body, not the HTTP status:** a client that branches on the status will treat a failed command as successful. For a Galaxy application (`kind: "GALAXY_APP"`) the same error arrives with the status `502`. There are two exceptions, the same ones as in the server-kind table above: `EXEC_BUSY` arrives as `409`, and `EXEC_NO_EXIT` arrives as `200` with `success: false`. So `success` in the body must be checked on both server kinds.

| Code | Description |
|-----|----------|
| `EXEC_TIMEOUT` | The `timeout` was exceeded (or the default 300 seconds). The agent terminates the whole process group forcibly (SIGKILL, no grace period). The response carries a `hint` object — its fields are described below |
| `EXEC_WORKDIR_DENIED` | The standalone working directory violates the path policy above. The agent refused before starting the command; reconcile the recorded outcome before using an allowed directory |
| `EXEC_WORKDIR_NOT_FOUND` | The standalone working directory is absent or cannot be inspected. The agent refused before starting the command; reconcile the recorded outcome before using an existing accessible directory |
| `EXEC_WORKDIR_NOT_DIR` | The standalone working path is not a directory. The agent refused before starting the command; reconcile the recorded outcome before using a directory |
| `EXEC_FAILED` | An execution error on the agent |
| `EXEC_NO_EXIT` | The stream ended without sending an exit status. What the command managed to do on the server is unknown, so this is not a success: the agent may have stopped responding, the tunnel may have dropped mid-command, or very large output may have overrun the channel after the command had already finished. `data` carries the output collected up to the break (`stdout`, `stderr`); the `exitCode`, `duration` and `truncated` fields are absent — the platform does not know their values. Returned on both server kinds. On a standalone virtual machine (`kind: "STANDALONE"`) the response additionally carries a `hint` with the recovery path; a Galaxy application gets none: the hint names virtual-machine recovery, and an application inside a galaxy has no machine of its own |
| `EXEC_BUSY` | The agent execution channel is occupied by a synchronous command, or on a standalone server by a durable job. The synchronous refusal is `success: false` with HTTP `200` on `STANDALONE`, or `409` with `Retry-After` and `retryable: true` / `retryAfter` on `GALAXY_APP`. Its `hint.recoveryAction` points first to `GET /v1/infra/servers/:id/operations?kind=exec`. Read durable outcomes and poll running jobs; an unknown outcome needs support and full-stop confirmation. An absent `autoExpiresInSeconds` does not prove a leaked mutex. Use [`/unstick`](/docs/infra/servers/unstick) only for a confirmed leaked synchronous mutex on an independently confirmed idle standalone server. A Galaxy owner cannot recover the shared host channel; another application’s job may be invisible in their list, so contact support if contention persists |
| `CONTAINER_EXEC_OPTIONS_DENIED` | Synchronous Galaxy exec refused host `workdir` or `env` options before execution. The public API normally catches these options earlier with `400 GALAXY_EXEC_NO_WORKDIR_ENV`. Reconcile the recorded outcome and correct the synchronous request |
| `CONTAINER_NOT_READY` | The galaxy host answered, but the application container the command was meant to run in is not there yet: the slot was created and never deployed, a deploy is in flight, or the container is restarting. Arrives with the status `502`. The response carries a `hint` that leads to deploying the application rather than to the host — the host is healthy, so waking or repairing it changes nothing. Applies to a Galaxy application (`kind: "GALAXY_APP"`) only |

#### The `hint` fields on `EXEC_TIMEOUT`

| Field | Type | Description |
|-------|------|-------------|
| `hint.reason` | string | What happened: the command did not finish within the allotted time, and the whole process group was terminated immediately, with no grace period |
| `hint.recovery` | string | What to do: in synchronous mode the upper bound for `timeout` is 600 seconds, so run longer operations as a background job (`systemd-run --unit=<name>`) and follow it via [`GET /v1/infra/servers/:id/logs`](./logs.md) with the `?service=<name>` parameter; for progress within the timeout pass `?stream=true` |
| `hint.recoveryAction` | string | A pointer to the documentation: `docs: /docs/infra/deploy/exec` |

The hint arrives in the JSON response and the SSE `error` event. Agent-origin `EXEC_BUSY` uses `GET /v1/infra/servers/:id/operations?kind=exec` as its `recoveryAction` on both supported server kinds. Read the operation outcome first. Only a confirmed leaked synchronous mutex on an independently confirmed idle standalone server calls for [`/unstick`](/docs/infra/servers/unstick). On `STANDALONE`, a running durable job keeps the execution channel busy after `202` releases the platform lock. Poll its operation; an unknown outcome needs support and independently confirmed full termination. Missing `autoExpiresInSeconds` does not prove a leaked mutex. Tunnel recovery and lock deletion do not clear durable admission. Galaxy commands run synchronously with a 600-second limit and share the host channel across all applications and deploys; your own operation list may not show another application’s job, so an empty list does not prove the host is idle.

A tunnel communication failure arrives in the same shape — with its own code in the `error.code` field (for example `TUNNEL_NOT_FOUND` or `GATEWAY_TIMEOUT: …`). Some gateway codes carry details after a colon (`GATEWAY_UNREACHABLE: …`, `GATEWAY_TIMEOUT: …`), so match `error.code` by prefix, not by an exact comparison. In streaming mode (`?stream=true`) the same errors arrive as an SSE `error` event. The event data carries `code` and `message`, and for `EXEC_TIMEOUT`, `GATEWAY_TIMEOUT` and an agent-side `EXEC_BUSY` it also carries a `hint`. The `GATEWAY_TIMEOUT` hint states the key point: no exit status came back, so the outcome of the command is unknown and it may still be running on the server — do not re-run it blindly, check the state first. These events carry no `success` field — the `error` event itself is the failure marker.

For the full list of common API errors, see [Errors](/docs/errors).

## Known specifics

- **A sleeping standalone virtual machine is woken by the command itself.** If the server (`kind: "STANDALONE"`) is in `sleeping` status, the platform starts the wake and waits for it within the same request — up to 6.5 minutes — and only then runs the command. A separate [`/wake`](/docs/infra/lifecycle/wake) call is not needed, but the client's request timeout must be longer than that window. If the server does not come up, `503 WAKE_TIMEOUT` arrives and the server returns to `sleeping` — the command never ran, so a repeat request is safe. A sleeping server with a live plan-change operation (`resize`) returns `409 EXEC_BUSY` before waking; wait for that operation to finish.
- **The command runs through `/bin/sh`.** The `command` field is executed by the `/bin/sh` interpreter — a minimal POSIX shell (`dash`), not `bash`. Constructs specific to `bash` (`set -o pipefail`, `[[ … ]]`, arrays) are not available there and fail with an error. Run them explicitly — via `command: "bash -c 'set -o pipefail; …'"` or as a script file `command: "/bin/bash /opt/app/script.sh"`.
- **Server-level locking.** Synchronous `/exec` and `/deploy` acquire a platform lock; synchronous exec has a `timeout + 90` second lock window and deploy has a 15-minute window. Standalone durable exec releases that platform lock after `202`, but keeps its execution channel busy throughout the job, with requested timeout up to 86400 seconds. Completion checks can extend admission beyond the command timeout. An unknown outcome needs reconciliation and confirmed termination; `/unstick` and [`DELETE /lock`](./lock.md) do not clear the durable gate. Galaxy exec is synchronous and capped at 600 seconds, and its host channel is shared by all applications and deploys. Recover a leaked synchronous lock only after independently confirming idle server state and logs.
- **The total output limit is 1 MiB (1,048,576 bytes), shared across `stdout` and `stderr`.** It is one budget for both streams, not 1 MiB each: if the command writes to both `stdout` and `stderr`, truncation is measured against their combined size. When the output is larger than the limit, `truncated: true` arrives and the tail is cut off. The command still ran to completion, and `exitCode` is correct. For large output, redirect to a file: `command: "my-cmd > /opt/app/output.log 2>&1"` and then read it via `/exec cat /opt/app/output.log`.
- **The `.env` from `/deploy` is not picked up automatically.** Systemd loads `.env` when the service starts, but not for one-off commands via `/exec`. If you need `DATABASE_URL` / API keys, pass them in the `env` field: `{ env: { DATABASE_URL: "..." } }`.
- **The two timeouts together give `timeout + 30` seconds.** The agent kills the process exactly at `timeout`; the Gateway waits another 30 seconds for the final events and only then returns `EXEC_TIMEOUT`.
- **Keeping the connection alive in JSON mode.** If the command runs longer than 15 seconds, the server periodically sends spaces in the response body — this prevents timeouts in nginx and intermediate proxies. The spaces go **inside the already-opened JSON object**, so the body looks like JSON (`{`) from its very first byte instead of starting with spaces: strict clients and ordinary parsers read it the same way, and you do not need to trim anything by hand.
- **Client timeout: make sure it is inter-byte, not total.** The keepalive spaces reset "silence" timeouts, so the classic Python socket timeouts (`urlopen(..., timeout=300)`, the `requests` read timeout) will not by themselves abort a long command in JSON mode. Two other things do abort it. First — clients and wrappers with a true wall-clock deadline for the whole request (for example, `aiohttp` with `ClientTimeout(total=…)` or an agent framework's own deadline): keepalive cannot help against those. Second — the server-side `timeout` of the command itself (300 seconds by default): once it expires, the agent kills the process and you get `EXEC_TIMEOUT` regardless of the client settings — do not mistake it for a client-side disconnect. In practice: in Python set split timeouts — `requests.post(url, json=body, headers=headers, timeout=(10, 60))` — 10 seconds for connect and 60 for the inter-byte pause. In synchronous mode, if the client drops the connection, the server-side lock persists for up to `timeout + 90` seconds (`EXEC_BUSY` for repeated calls). Commands longer than a few minutes are better run as a background job — see "Background jobs" below. **On a sleeping server this rule does not apply right away:** the spaces start only after the wake, so up to 6.5 minutes of complete silence precede the first byte of the response, and the `timeout=(10, 60)` recipe aborts the request before the command even starts. Either raise the inter-byte timeout above that window, or wake the machine in advance with [`/wake`](/docs/infra/lifecycle/wake) and `wait=true`, then call `/exec` on a machine that is already up. A sleeping server with a live plan-change operation (`resize`) returns `409 EXEC_BUSY` before waking; wait for that operation to finish. Do not use manual wake to bypass resize.

## Unsticking a stuck channel

On `STANDALONE`, an `EXEC_BUSY` without `autoExpiresInSeconds` can mean a durable job is still running; it does not prove a leaked agent mutex. First read the [operation outcome](./operation-status.md), or discover the run in the [server operation list](./operations.md) with `kind=exec`. Wait while it is running. An `unknown` outcome does not confirm termination: reconcile state and logs and contact support. Do not bounce the tunnel or delete the lock for a running or unconfirmed durable job; `/unstick` does not clear its admission gate. Use [`POST /v1/infra/servers/:id/unstick`](/docs/infra/servers/unstick) only for lock recovery after independently confirming idle state, subject to that endpoint’s guards.

## Choosing the mode

Four call modes and when each one fits:

| When | How | Monitoring |
|------|-----|------------|
| The command fits into 600 seconds, no progress needed | `{ "command": "...", "timeout": 600 }` without `?stream=true` — a blocking JSON response | `exitCode`, `stdout`, `stderr` in the response body |
| A standalone command fits into `timeout`, live progress needed | the same request with `?stream=true` — SSE events `stdout` / `stderr` / `exit` | read the events as they arrive |
| Retained standalone result; its channel stays busy throughout the job | `?async=true`, timeout up to 86400 on `STANDALONE` only | Poll `statusUrl` for the outcome and retained stdout/stderr |
| A standalone command whose channel must be released before the job finishes | a client-managed background job `systemd-run --unit=<name>` — `/exec` returns immediately | [`GET /v1/infra/servers/:id/logs?service=<name>`](./logs.md) |

For synchronous exec, do not set the client-side overall request deadline lower than the command's `timeout` — the client would cut the connection before the server can answer. A sleeping server adds up to 6.5 minutes of wake time to that deadline — the example below assumes a machine that is already up. An example of split timeouts in Python:

```python
import requests
r = requests.post(
    f"{VIBE_URL}/v1/infra/servers/{SERVER_ID}/exec",
    headers={"X-Api-Key": VIBE_API_KEY},
    json={"command": "npm ci --production", "workdir": "/opt/app", "timeout": 600},
    timeout=(10, 60),   # 10 seconds to connect, 60 for a pause between bytes
)
```

Raising the `timeout` of `/exec` does not help the `install` step — deploy steps have their own 300-second limit, separate from the synchronous 600-second `/exec` limit.

**Important:** the `timeout` default is **300 seconds**, not 600. The synchronous 600-second ceiling exists, but to use it the field must be passed explicitly. A command planned for nine minutes and started without `timeout` is killed at the five-minute mark.

## If the wait was interrupted

A transport timeout, a dropped connection, or a client restart end only the wait for the response. The command on the server is not stopped and can run to completion after the client has already received an error. That is why re-running a state-changing command may apply the change a second time.

To avoid that, an accepted run usually gets an identifier — but in synchronous mode that is not a guarantee: when the platform could not open the record there is none, and the command still runs. Standalone durable execution requires a stored record and returns the identifier with `202`. When it exists, it arrives through three channels:

- the `X-Vibe-Operation-Id` response header — on a standalone virtual machine it is sent before the command starts executing, so it survives to the client even when the response body is lost. For synchronous Galaxy exec it arrives only with the terminal response: that branch opens no stream, so an interrupted wait never receives the id and the run is found in the [server operation list](./operations.md) with `?kind=exec`,
- the first `event: operation` frame with `?stream=true` — synchronous exec on standalone virtual machines only. A browser `EventSource` does not expose response headers, so this is its only early channel,
- the `operationId` field in the body — `data.operationId` on success, `error.operationId` on failure.

The recovery order:

1. Do not re-run a state-changing command blindly.
2. Call [`GET /v1/infra/operations/:operationId`](./operation-status.md). If the identifier is lost, find the run in the [server operation list](./operations.md) with `?kind=exec`.
3. Status `succeeded` means the command ran to completion, and its own verdict is in `exitCode`. There is nothing to repeat.
4. Status `failed` is set by the platform only when it can prove the command never ran — for example on `EXEC_BUSY`. That proves only this attempt never ran. For `EXEC_BUSY`, first reconcile the occupying operation, including any earlier attempt whose acknowledgement was lost; an unknown occupying outcome does not authorize another exec.
5. Status `unknown` means the outcome is not proven and the command may be running right now. Reconcile the server state and the [logs](./logs.md) before deciding to repeat it.

For synchronous exec, the record stores the outcome of the attempt — status, exit code and the reason for a refusal — but not command output. Use `async=true` to retain stdout/stderr with the operation; a client-managed background job with journal monitoring is another option.

## Background jobs

For synchronous JSON and streaming exec, `timeout` is capped at 600 seconds, and when it expires the agent terminates the whole process group forcibly (SIGKILL, no grace period). On a standalone machine, for longer work — package installation, a database dump restore, a heavy build — prefer durable `async=true` when you need retained output. Alternatively, manage a background job yourself: `/exec` returns instantly, the channel is freed (a second call will not hit `EXEC_BUSY`), and the process keeps running independently of the HTTP connection.

### Recommended — a transient systemd unit

```json
{ "command": "systemctl reset-failed restore-db 2>/dev/null; systemd-run --unit=restore-db /bin/bash /opt/data/restore.sh", "timeout": 30 }
```

The job gets its own cgroup and lives independently of the exec session. Manage it with standard calls:

- status: `{ "command": "systemctl is-active restore-db" }` — `active` (running), `inactive` (finished successfully), `failed` (crashed).
- logs: journald picks up the unit's output automatically — read it via [`GET /v1/infra/servers/:id/logs?service=restore-db`](./logs.md).
- `systemctl reset-failed <name>` before a rerun — otherwise systemd refuses to create a unit with the name of the failed job.

### Lightweight — background with output redirect

```json
{ "command": "(cd /opt/app && npm run build > /tmp/build.log 2>&1 &)", "timeout": 10 }
```

Redirecting both streams to a file is mandatory — it releases the exec session's pipes. Progress: `{ "command": "tail -20 /tmp/build.log" }`.

### Anti-pattern — bare `nohup`

`nohup cmd &` **without a redirect** does not work: in a pipe (rather than a terminal) `nohup` does not redirect the output, the background process inherits the exec session's pipe, and the agent waits for it to close right up to the `timeout` — then forcibly terminates the whole process group, including your "background" process. Always add `> file 2>&1`.

Keep secret values out of `command`. On a standalone server, pass them via `env`; on a Galaxy application, load them inside the container from an existing protected file or the application’s configured secret store. Synchronous exec logs the first 200 characters of command text on the agent. The durable worker disables that command log, but commands can still expose secrets through their own output or child processes. Also keep in mind that the `install`/`preStart` steps of [`/deploy`](./deploy.md) have their own 300-second timeout — move heavy installations into a one-off `/exec` or a background job, not into the install script.

## Durable exec

Use `POST /v1/infra/servers/:id/exec?async=true` for a long command on a `STANDALONE` server only. It returns HTTP `202`, `X-Vibe-Operation-Id` and `data: {operationId, status: "running", statusUrl}`. Poll `GET /v1/infra/operations/:operationId`; do not resend `/exec` after a lost acknowledgement. If the id was lost, discover the run through the server operation list with `kind=exec`.

The body accepts `timeout` from 1 to 86400 seconds (default 300). `async=true` cannot be combined with `stream=true`. Every other server kind receives `400 DURABLE_EXEC_UNSUPPORTED_SERVER_KIND` before locking, operation creation or dispatch, without a synchronous fallback. Galaxy exec remains synchronous with a 600-second limit. An execution service that does not support durable jobs refuses the request before execution; no synchronous fallback runs.

The job survives a lost HTTP response and reconnection. Its execution channel remains busy throughout the job even after the platform releases its lock on returning `202`; other exec calls and deploy commands receive `EXEC_BUSY`. Poll the operation instead of resending exec or calling `/unstick`. A completed operation returns `stdout`, `stderr`, `duration` in milliseconds, `truncated` and its proven `exitCode`, with a shared 1 MiB output cap. Output is encrypted at rest and retained within the seven-day outcome window. Private output is returned only to a scoped key owned by the operation creator; a same-owner rotation preserves access, a transfer to another owner does not. `succeeded` includes nonzero exit codes; `unknown` does not authorize a repeat. A host reboot or lost process can leave an unknown outcome requiring reconciliation and confirmed termination. Status reads reconcile the result without starting a command. Cancellation and automatic retries are unavailable.

The execution service retains at most **64 job records per standalone host**, including completed jobs during their seven-day retention window. A new request at capacity is refused before execution with HTTP `502 EXEC_JOB_CAPACITY` and `error.operationId`. Read that operation first, then wait for eligible retention cleanup or contact support; do not automatically resend. Cleanup runs on admission and removes only expired results with confirmed termination. If a job is interrupted before its result is saved, confirmed termination leaves its outcome `unknown`, but makes its record eligible for cleanup after seven days. Active jobs and unconfirmed termination are not cleared just because seven days elapsed. Neither `/unstick` nor lock deletion clears the durable admission gate.

## See also

- [Full deploy](./deploy.md)
- [Operation outcome](./operation-status.md)
- [Recent server operations](./operations.md)
- [Release a stuck lock](./lock.md)
- [Service logs](./logs.md)
- [Upload a file](./upload.md)
- [Repair the tunnel](/docs/infra/lifecycle/repair)
- [A fast release cycle](./fast-cycle.md)
- [Galaxy app](/docs/infra/galaxy)