Skip to content

In-cluster agent

The SkyPortal agent runs inside your Kubernetes cluster and reports to SkyPortal over outbound HTTPS. Use it when the cluster's API server isn't reachable from the internet, or when you'd rather not give SkyPortal a kubeconfig. Through the agent, SARA can only run read-only kubectl.

What gets installed

The Helm chart installs the agent with kubernetes.enabled=true:

Workload Pods What it does Kubernetes access
<release>-cluster Deployment 1 Runs the read-only kubectl commands SkyPortal asks for: nodes, namespaces, pods, warning events, deployments and statefulsets, kubectl top, and recent logs of failing pods A read-only ClusterRole
<release>-node DaemonSet 1 per node Reads the node's CPU, memory, disk, load and network from the host's /proc, disk space, and GPU data from nvidia-smi None
<release> Deployment 1 Reports ML experiment runs from Weights & Biases and MLflow. You can turn its scanners off None

SkyPortal decides what to collect and asks every 30 seconds. If SkyPortal can't be reached, uploads wait on disk (up to 512 MiB) and are sent in order later.

The cluster role's ClusterRole grants:

  • get, list and watch on nodes, namespaces, pods and events;
  • get on pod logs;
  • get and list on deployments and statefulsets, and on metrics.k8s.io nodes and pods.

It grants nothing on Secrets, Services or proxies, and no write verbs. The agent also refuses any command that isn't read-only kubectl, whatever it is asked to run.

Pod specs are sent as they are

The agent sends full pod specs (kubectl get pods -A -o json), including environment variable values written directly in the spec. Keep secrets in Kubernetes Secrets, which the agent can't read.

Before you start

  • helm 3 and kubectl with access to the cluster.
  • Outbound TCP 443 from the cluster to app.skyportal.ai. Nodes also pull the image from ghcr.io, with its files served from pkg-containers.githubusercontent.com. The chart and image are public, so no login or pull secret is needed.
  • The agent's namespace must allow privileged pods, because the node DaemonSet mounts host paths. The steps below add the label.
  • metrics-server, for CPU and memory usage. Without it, usage shows as unknown.
  • On GPU nodes, the runtime class that exposes the NVIDIA driver, usually nvidia (kubectl get runtimeclass).
  • The API server's port: run kubectl get endpoints kubernetes -n default. If it isn't 443 or 6443, add it to kubernetes.apiServerPorts, because the chart's NetworkPolicy only allows those ports.

Create an agent token

The agent signs in with an agent token. Agent tokens start with agt_ and belong to one cluster.

On the Agent Tokens page (/agents/, Agent Tokens in the account sidebar):

  1. Fill Agent name and Cluster name, and optionally Expires. The cluster name becomes the cluster's name in SkyPortal, so it can't match another of your hosts.
  2. Click Create token.
  3. Copy the token. It is shown only once.

From a chat: ask SARA to connect a cluster that isn't reachable from the internet. SARA asks for approval with an Agent token card, which only offers approve and reject. Auto-approve never answers it. After you approve, the token is handed over once:

  • in the web app, a card shows the token with Copy and the commands to create the Secret;
  • in the interactive terminal, the token can go straight into your cluster without being shown. See Install from a chat.

The handover is valid for 10 minutes. After that, rotate the token on the Agent Tokens page. A session signed in with a short-lived access token (skt_) can't create agent tokens: run skyportalai login with an API key first.

Manage tokens

The Agent Tokens page lists each token with its cluster, status, agent version, creation date and expiry.

Status Meaning
never The agent hasn't reported yet
active The agent reported in the last 10 minutes
stale The agent stopped reporting
expired The token passed its expiry date
  • Rotate issues a new token and turns off the old one. The expiry date carries over if it is still in the future.
  • Revoke turns the token off. The agent using it stops reporting right away.
  • A token stops working when its owner is deactivated, or when the owner's organization requires two-factor authentication and the owner hasn't set it up.

Install with Helm

Replace 0.3.3 with the latest chart version if a newer one is out. Chart 0.3.3 installs agent image 0.3.0.

kubectl create namespace skyportal
kubectl label namespace skyportal pod-security.kubernetes.io/enforce=privileged

read -rs AGT   # paste the agt_ token; it is not echoed or kept in history
printf '%s' "$AGT" | kubectl -n skyportal create secret generic skyportalai-agent-token \
  --from-file=SKYPORTALAI_AGENT_TOKEN=/dev/stdin
unset AGT

helm upgrade --install skyportalai-agent oci://ghcr.io/skyportalai/charts/skyportalai-agent \
  --version 0.3.3 -n skyportal \
  --set token.existingSecret=skyportalai-agent-token \
  --set kubernetes.enabled=true \
  --set kubernetes.node.gpu.runtimeClassName=nvidia
  • Leave out kubernetes.node.gpu.runtimeClassName on a cluster without GPUs.
  • No Weights & Biases or MLflow in the cluster? Add --set config.enableWandb=false --set config.enableMlflow=false.
  • The chart refuses kubernetes.enabled with an agent image older than 0.3.0.

Install from a chat

With skyportalai 0.3.1 or later, the interactive terminal can install the token for you when SARA creates one. It keeps the token in memory only, shows your current kubectl context and asks:

Create Secret skyportalai-agent-token in namespace skyportal and set that namespace's Pod Security enforcement to privileged, using kubectl context '<context>'? [y/N]

If you answer yes, it creates the namespace, sets the label and creates the Secret, passing the token on standard input. The token is never printed or written to disk. The terminal then offers the helm upgrade --install command. If you answer no, or kubectl is missing or fails, it prints the token once with the commands to run yourself.

Check the connection

kubectl -n skyportal get pods -o wide

You should see one -cluster- pod and one -node- pod per node.

In SkyPortal, the cluster appears in your host list with a ⎈ Kubernetes badge and Waiting for agent until the first upload arrives. You can also ask SARA whether the agent is connected: it reports waiting, connected (with node and GPU counts) or stale. A cluster counts as stale after about 90 seconds without an upload.

What SARA can do through the agent

  • Run read-only kubectl: get, logs, top, describe, api-resources, api-versions, version, cluster-info and explain. Anything else is refused with "This cluster is connected through the in-cluster SkyPortal agent, which is read-only." Helm isn't available.
  • Commands go through the same namespace scope and approval rules as a kubeconfig cluster.
  • The agent picks up a command within a few seconds. SkyPortal gives up on it after 60 seconds.
  • If a cluster has both a kubeconfig and an agent, SkyPortal uses the kubeconfig.

Current limits

  • Ask in chat. Clusters connected only through the agent don't appear in the Monitor Dashboard or on the Kubernetes clusters page yet; the dashboard shows "Cluster not reachable" for them. Cluster health, pods, logs, events and pod right-sizing work through SARA in a chat.
  • The 30-second collection interval is set by SkyPortal and can't be changed.
  • vLLM serving metrics and DCGM exporter metrics are not collected through the agent.
  • Kubernetes monitoring needs the Helm chart. The plain manifests in the repository only cover experiment tracking.
  • A token created on the Agent Tokens page always registers a Kubernetes cluster in SkyPortal, even if you only use it for experiment tracking. That entry stays at Waiting for agent.

Helm values

Value Default Purpose
token.existingSecret "" Name of a Secret that holds the token. Takes precedence over token.value
token.value "" The token itself; the chart stores it in a Secret
token.secretKey SKYPORTALAI_AGENT_TOKEN Key of the token inside the Secret
config.baseUrl "" (uses https://app.skyportal.ai) SkyPortal address
kubernetes.enabled false Install the cluster Deployment and the node DaemonSet
kubernetes.apiServerPorts [443, 6443] API server ports the NetworkPolicy allows
kubernetes.node.gpu.runtimeClassName "" Runtime class for GPU access, usually nvidia
kubernetes.node.tolerations tolerate every taint Limit which nodes run the DaemonSet
kubernetes.node.nodeSelector {} Limit which nodes run the DaemonSet
kubernetes.node.spoolHostPath /var/lib/skyportalai-agent Host directory for uploads waiting to be sent
networkPolicy.enabled true Deny inbound traffic; allow DNS and outbound 443
config.enableWandb, config.enableMlflow true Experiment scanners
image.tag the chart's app version Agent image tag

The full reference, including mirroring the image to your own registry, is in the agent README.