Skip to content

Monitoring

SkyPortal collects health and usage data from your SSH hosts and Kubernetes clusters. You can see it in the Monitor Dashboard, and SARA uses the same data when you ask about a host or a cluster in a chat.

Open the Monitor Dashboard

On the agent page, click Monitor Dashboard and pick a host or a cluster. SSH hosts open the host monitor, and Kubernetes clusters open the cluster monitor. Only a host's owner can open its dashboard.

The Observability button next to it is a different view, for ML experiments. See ML experiments.

SSH hosts

SkyPortal checks every SSH host with saved credentials every 30 seconds. Each check runs about 40 read-only commands, each limited to 8 seconds.

What it collects:

  • Usage: CPU, memory, disk /, load, processes, uptime, and average GPU utilization and memory.
  • Hardware: CPU model and cores, GPU count and model, swap, mounts, virtualization.
  • Software: OS, kernel, Python, Docker, CUDA version, and running and failed services.
  • Health: journal errors and warnings from the last 24 hours, OOM kills, failed logins, zombie processes and open connections.

The host monitor shows:

  • tiles for GPU, GPU util, GPU mem, CPU, Memory (RAM), Disk /, Net latency, Load (1m) and Uptime;
  • a per-GPU table from nvidia-smi: model, memory, utilization, power, temperature and the processes using each GPU;
  • charts for per-GPU memory and utilization, and for host CPU and memory;
  • host details for CPU, memory, every mounted disk and the network;
  • Host events: the last 50 warnings and errors from journalctl.

The tiles and the GPU table are read live over SSH when you open the dashboard. The charts show the stored history. Net latency is a ping from the host to 8.8.8.8.

GPU data on SSH hosts comes from nvidia-smi, so it covers NVIDIA GPUs only. A host without one shows "No GPUs detected on this host."

Kubernetes clusters

Clusters connected with a kubeconfig are checked every 20 seconds with read-only kubectl. Clusters connected only through the in-cluster agent don't appear in the Monitor Dashboard yet; ask SARA about them in a chat.

The cluster monitor has two views, Infrastructure and Workloads, and a Namespace filter. System namespaces are hidden.

  • Tiles: Nodes ready, CPU utilization, Memory used, Peak GPU mem, GPU utilization, GPUs idle, GPU headroom and Allocation, each compared with its 24-hour median.
  • Charts: cluster GPU memory and utilization, and node CPU and memory.
  • Sections: per-GPU details, nodes, and warning events from kubectl get events.
  • Workloads: each workload's namespace, pods, CPU, GPUs, peak GPU memory, GPU utilization, power, restarts and status.

What the cluster needs:

  • metrics-server for node and pod CPU and memory. Without it, the monitor shows "metrics-server not installed" and those values show as n/a.
  • A GPU exporter for GPU data. SkyPortal looks for the NVIDIA DCGM exporter (app=nvidia-dcgm-exporter in the gpu-operator or kube-system namespace), nvidia_gpu_exporter (app.kubernetes.io/name=nvidia-gpu-exporter) or the AMD GPU exporter (app=amd-gpu-exporter) on port 9400, and reads it through the API server's service proxy. Without one, the monitor shows "No GPU exporter (DCGM/etc.) is reachable on this cluster."

The cluster charts cover the recent period while the dashboard is in use. They aren't a long-term history.

Ask SARA

SARA reads the same data, so you can ask questions like "How is gpu-01 doing?" or "Which pods keep restarting?". Every 15 minutes SkyPortal also compares each metric with the previous 15 minutes, and SARA uses that summary when you ask what changed.

SkyPortal doesn't send alerts or notifications about metrics.

Pod right-sizing

Ask SARA whether pods are sized well, for example "Are any pods in ml-serving over-provisioned?". It works for Kubernetes clusters connected either way.

For each pod, SARA compares the latest CPU and memory usage it saw in the last 10 minutes with the pod's requests and limits, summed over its containers. It flags:

Finding Rule
OOM-kill risk Memory at 90% or more of the limit
Throttled CPU at 90% or more of the limit
Eviction risk Usage above the request, depending on the node's free capacity
No memory limit A container without a memory limit
No requests A container without requests
Over-provisioned Usage under 20% of the request

The answer starts with a summary such as "Pod right-sizing — 42 pods: 30 ok, 9 flagged, 3 usage unknown", followed by a table with the flagged pods first and a short conclusion.

  • It reports findings. It doesn't suggest new request or limit values.
  • It uses the latest sample, not an average over time, and it doesn't size GPUs.
  • Platform namespaces such as kube-*, *-system, gpu-operator, monitoring, cert-manager and ingress-nginx are left out.
  • Without metrics-server, pods show as "usage unknown — sizing not assessed", never as ok.

ML experiments

The Observability page shows ML training runs reported by the observability agent from Weights & Biases and MLflow. See the SDK & CLI overview to install the agent.