Skip to content

One command across many hosts

SARA can run the same command on several SSH hosts at once, with one approval for all of them. Use it to compare driver versions, check disk space on a fleet, or apply the same fix to a set of training nodes.

Choose the hosts

  1. Click the Scope pill above the message box. It reads "Scope: none selected" when no host is in scope.
  2. The Chat scope panel lists your hosts grouped by environment. Tick hosts one by one, or tick an environment to take all of its hosts.
  3. Optionally click ★ Make active default on the host SARA should use for single-host commands.
  4. Click Done. Clear scope empties the selection.

Naming a host in your message also adds it to the scope. SARA can only run commands on hosts in scope.

From the CLI, choose several hosts with /server gpu-01 gpu-02 in the interactive terminal, or repeat --server with skyportalai chat send.

Approve once for all hosts

SkyPortal checks each host on its own: your team role, Read-Only Mode, sensitive files and the host's permit list. If every host allows the command, it runs without asking. If any host needs an approval, you get one card for all of them.

  • The card lists each host with a checkbox, all ticked. Untick hosts to run the command on fewer of them.
  • Leave at least one host ticked, or click Reject.
  • ✓ Approve runs the command once on the ticked hosts.

Some commands are stopped before the card:

  • Read-only hosts in scope. A destructive command is blocked for the whole run while a host with a read-only policy (Production, by default) is in scope. The message names those hosts. Narrow the scope to run it on the others. Reads still run on every host.
  • Sensitive files. Reading or changing a sensitive file always needs an approval on a multi-host run, even when a permit list matches.

See Approvals and policies for the rules behind each check.

Read the results

  • The hosts run at the same time. A failure on one host doesn't stop the others.
  • The result shows "N/M succeeded" and, when something failed, an "X failed" chip.
  • Failed hosts are listed first and expanded, with their exit codes.
  • Secrets in the output are redacted, as everywhere in SkyPortal.

Batches of read-only commands

When SARA needs several read-only commands across your hosts, it can run them as one batch without approval cards. Anything that needs an approval is asked one command at a time.

Limits

  • SSH hosts only. A Kubernetes cluster can't be part of a multi-host command.
  • Each host has up to 30 minutes to finish.
  • A command can be up to 8,192 characters long.