Kubernetes Support¶
To use LPUs in an existing Kubernetes cluster, prepare the driver and toolkit on the nodes,
add ha-lpu-operator to the cluster, and request the number of LPUs in your workloads.
| Location | What to add or change |
|---|---|
| LPU nodes | Install hyperaccel-lpu and configure CDI in the runtime |
| Cluster | The ha-lpu-operator Helm chart |
| Existing Pods and Deployments | Add hyperaccel.ai/lpu to the container's resources.limits |
Preparing the nodes and cluster¶
In a cluster running Kubernetes 1.31 or later, check the following on each node with an LPU:
- Is
hyperaccel-lpuinstalled so the driver and toolkit are available? - Is the existing containerd or CRI-O runtime configured to read CDI specifications? See Container support.
If cert-manager manages the webhook certificates, it must already be available in the cluster. If you provide your own certificate, use the settings below.
Adding the operator to the cluster¶
Install ha-lpu-operator if it is not already present in the cluster.
The chart and the component images are pulled with the account HyperAccel issued. Log Helm in first,
and create the Secret the component images are pulled with. The chart looks it up by the name harbor-secret.
1 2 3 4 | |
1 2 | |
The chart creates the operator and a default BerthaCluster. The operator manages the
LPU components, including device-plugin. The chart includes NFD (node-feature-discovery).
If it is already installed, check its status rather than installing it again.
1 2 3 | |
Once BerthaCluster is Ready and the node advertises the hyperaccel.ai/lpu resource, Pods can request LPUs.
The validator checks the node's driver and device files before allowing device-plugin and feature-discovery to run.
Using existing certificate settings¶
To change the defaults, update berthaCluster.spec in the Helm values.
Use the webhook certificate path that fits your environment:
| Setting | Certificate |
|---|---|
webhook.tlsSecretName |
A Secret you provide |
webhook.certManager.issuerRef |
The specified cert-manager Issuer |
| Neither (default) | The operator creates a self-signed ClusterIssuer |
Using cards in a Pod¶
In spec.containers[] of an existing Pod or spec.template.spec.containers[] of a Deployment,
add hyperaccel.ai/lpu to resources.limits. Keep the existing CPU and memory settings.
1 2 3 | |
To check access to a single card using the SDK image, use this Pod. The namespace you create it in needs harbor-secret too:
| lpu-test.yaml | |
|---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
1 2 3 | |
The connection check passes when the Pod completes and prints a card count of 1.
Serving with vLLM¶
In your existing Deployment, change the vLLM container's image, execution options, and LPU request.
The example below shows only the fields to change in spec.template.spec.containers[].
To pull the image, add harbor-secret to spec.template.spec.imagePullSecrets and create the same Secret as above in that namespace.
1 2 3 4 5 | |
Set the model and context length to suit your existing workload. To split the model across two LPUs,
set both hyperaccel.ai/lpu: 2 and --pipeline-parallel-size 2.
For serving options, see the vLLM configuration guide.
Check these points in your existing operational configuration:
| Existing setting | What to check when using LPUs |
|---|---|
| Model cache | Use the existing volume. If you reference a PVC, it must be available in the same namespace. Check storage constraints to determine whether it can also be used on another node |
| Startup and readiness checks | Allow for model downloading and kernel preparation in startupProbe. Startup check failures can trigger a restart; readinessProbe failures remove the Pod from traffic but do not themselves restart it |
| Update strategy | If the existing Pod holds all LPUs, a new Pod may wait for resources. Updates without downtime require spare LPU and storage capacity for the new Pod |
To replace the existing Pod by stopping it first when no spare LPU is available, add the following under the Deployment's spec.
The service will be unavailable until the new model is ready.
1 2 | |
Node labels¶
feature-discovery adds these labels. Use them directly in nodeSelector or affinity rules.
| Label | Example | Meaning |
|---|---|---|
hyperaccel.ai/lpu.present |
true |
The node has cards |
hyperaccel.ai/lpu.count |
8 |
Card count |
hyperaccel.ai/lpu-max-contiguous |
2 |
Maximum number of cards under a single PCIe switch |
For Pods requesting two or more cards, the webhook adds a preference for nodes whose lpu-max-contiguous is at least the requested card count.
When cards share a switch, communication between them stays within that switch. This is a preference, so the Pod can still run if no node matches.
Metrics¶
Add ha-lpu-exporter as a scrape target in your existing Prometheus Operator setup.
The exporter serves /metrics from Pods in bertha-system on the port named metrics (9401).
Set metadata.namespace and metadata.labels in the PodMonitor below to match your existing Prometheus
podMonitorNamespaceSelector and podMonitorSelector.
The example values monitoring and release: kube-prometheus-stack are for environments using those selection criteria.
spec.namespaceSelector points to bertha-system, where the exporter runs.
If you changed the BerthaCluster name, update app.kubernetes.io/instance to the actual value as well.
| ha-lpu-exporter-podmonitor.yaml | |
|---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | |
1 | |
Include this PodMonitor without overwriting the selection criteria for existing scrape targets.
In Prometheus Targets, check that the exporter is UP, then query lpu_device_up.
UP indicates a successful scrape; lpu_device_up indicates whether a card responds.
| Metric | Meaning |
|---|---|
lpu_device_up |
Whether the card responds |
lpu_device_wedged |
Whether the card is stuck |
lpu_fw_errors_total |
Firmware error count |
lpu_temperature_celsius |
Temperature |
lpu_power_watts |
Power |
lpu_dram_used_bytes |
Card memory usage |
lpu_pcie_link_degraded |
Whether the PCIe link is running below its maximum speed |
lpu_pod_lpu_count |
Number of cards allocated per Pod |
Troubleshooting¶
| Symptom | What to check |
|---|---|
Pod is Pending |
Events in kubectl describe pod. If the node has no hyperaccel.ai/lpu, check kubectl get berthacluster -n bertha-system and the validator logs |
BerthaCluster is not Ready |
kubectl get pods -n bertha-system. If the validator fails, start with the driver on that node |
No /dev/ha* inside the container |
Run ha-ctk precheck on the node and check the runtime's CDI configuration |
| Allocatable count is lower than the card count | Check whether device-plugin has excluded faulty cards from new allocations. Check the lpu_device_up and lpu_device_wedged metrics |
| Exporter is missing from Prometheus | PodMonitor namespace and label selectors, and permission to list Pods. If the target is DOWN, check the metrics port, path, and network |