Skip to content

Multiple devices

This page covers what operators need to know when a host has several LPUs. It explains the rules the driver uses to number cards and how to identify a card without relying on its number, what operates independently on each card, how to check the data paths between cards through sysfs, how the IOMMU mode and the PCIe topology affect those paths, what other cards show when one card is reset or removed, and the trust boundary when several cards are divided up for use. The output format of each node is in sysfs nodes.


Card numbers

For each card that it probes successfully, the driver creates the device node /dev/haN and the sysfs directory /sys/class/ha/haN/. On this page, a "card registered with the driver" means a card whose nodes were created this way.

N is 0-7, and the driver handles at most 8 cards per host. No nodes are created for a card that gets no number. Such a card appears in lspci but is not bound to the driver, and the kernel log shows only Probing device for that card, not Probe successful.

Numbers are assigned by the following rules.

  • When the module is loaded, the driver numbers the LPUs present at that time from 0 in PCI address order. It counts them whether or not they are bound to the driver. So as long as the same cards are installed, each card keeps its number across module reloads and host reboots.
  • The number of a card whose probe fails, or of a card held by another driver (such as vfio-pci), stays empty, and the numbers of the cards after it do not move down. When that card is later bound to ha, it receives the number reserved for it.
  • While the module stays loaded, if a card is removed and then reappears at the same PCI address (rebind, rescan, ha-hotreset), it receives its previous number again, provided no other card has taken that number in the meantime.
  • A card that appears after the module is loaded first receives a number not reserved for any card. Only when there is no such number does it take the number of a card that is currently absent, and that card may not get its previous number back when it returns.
  • A card whose PCI address changed (because it was moved to another slot, or because re-enumeration changed its bus number) is treated as a new card.
  • If you add or remove a card and then reload the module (including a reboot), the cards with a higher PCI address than that card get different numbers.
  • If there are more than 8 LPUs, the kernel logs ha: more than 8 cards; the extra cards take any free number, so card numbers can change across reloads when the module is loaded, and after a reload the numbers and the cards that get nodes can change. Keep 8 or fewer LPUs on one host.
  • While a card is removed or has failed its probe, a gap appears in the numbers. When counting cards, check every existing number with ls /sys/class/ha/.

All /dev/haN nodes share one major device number, and the minor is the card number. The kernel picks the major when the module is loaded, so it can change when the module is reloaded or the host reboots. A container keeps the device numbers it was created with, so recreate the container if the major changes. Read the current device number from /sys/class/ha/haN/dev.

A number is not a card's identity

Numbers follow the PCI address order of the installed cards, so they change when you add, remove, or move a card. Where you must point to a specific card, such as a monitoring label, a container assignment, or a firmware update target, use the PCI address or serial. The CDI device names given to containers (Container support) and hyperaccel-smi's -i and HYPERACCEL_VISIBLE_DEVICES (hyperaccel-smi) also select cards by number.

Identifying a card by PCI address and serial

/sys/class/ha/haN/device is a link to the PCI device of that card. The following command prints the number, PCI address, and serial of each card, one line per card.

1
2
3
for d in /sys/class/ha/ha*; do
    echo "$(basename "$d") $(basename "$(readlink -f "$d/device")") $(cat "$d/serial")"
done
1
2
ha0 0000:1a:00.0 HA-5f3a0c21-0012-0007-0001-00a400000000
ha1 0000:1b:00.0 unknown
  • serial is a unique number recorded on the chip, so it stays the same even if you move the card to another slot. A card whose firmware does not report this value, or a card that is not yet ready, shows unknown; in that case, use the PCI address.
  • Write the PCI address in the format that includes the domain (0000:1a:00.0). On a host with several PCI domains, two or more cards can have the same bus, device, and function numbers.
  • The Bus-Id column of hyperaccel-smi, the -d option of ha-fw-update, and the --bdf option of ha-hotreset use the same PCI address.

To also see cards that are not bound to the driver, check the PCI bus directly.

1
2
3
4
# All installed HyperAccel cards
lspci -D -d 20fb:
# Cards bound to this driver (links named by PCI address)
ls /sys/bus/pci/drivers/ha/

What operates independently on each card

Item Scope
Device node and telemetry Per card. /dev/haN, /sys/class/ha/haN/
FATAL state and health Per card. The other cards keep working even if one card becomes FATAL
Reset Per card. Only the work on the card that was reset is cleared
Firmware update Per card. ha-fw-update writes to the one card you specify

There are two exceptions even among the items that operate independently on each card.

  • The data path between cards spans two cards. When you reset or remove one card, work on other cards that was exchanging data with that card is also affected (When one card is reset or removed).
  • PCIe error recovery targets everything below the port where the kernel detected the error. If an error is detected at a PCIe switch or at a port above it, the cards below it may be reset together (FATAL and recovery).

Data paths between cards

Cards can exchange data with each other over PCIe without going through host memory. There are two paths. One is device-to-device copy, in which the DMA engine of one card moves data to and from the memory of another card. The other is access in which compute code running on a card (called a device kernel on this page; it is different from the Linux kernel) directly reads and writes the core SRAM of another card. Core SRAM is the SRAM that each compute core has, and it differs from the host-shared SRAM that sram_size and sram_used in telemetry refer to.

Device-to-device copy SRAM access by a device kernel
What moves the data The DMA engine of one of the two cards A device kernel running on one card
What it reaches on the peer card The memory of the peer card, through BAR2 The core SRAM of the peer card, through BAR0
What starts it The SDK's device-to-device copy A device kernel that reads and writes the core SRAM of another card
Node to check peer_map peer_sram
  • The two paths are independent of each other. peer_map reports only on device-to-device copy, and peer_sram reports only on access by a device kernel.
  • The verdict on the PCIe route between two cards (peer_route) is common to both paths.
  • The three telemetry nodes (peer_map, peer_sram, peer_route) read only host-side state, so you can read them even while a card has stopped, and you can read them without root privileges.

Device-to-device copy

When a card is registered with the driver, the driver prepares, in both directions between that card and the cards already registered, the DMA addresses through which a DMA engine reaches the BAR of the peer card. peer_map shows the result, with one card as the reference, for the direction toward the other cards.

1
cat /sys/class/ha/ha0/peer_map
1
2
peer1 mapped
peer2 mapped
  • There is one line for each other registered card, and the N in peerN is the number of that card. If there are no other cards, the output is none.
  • It is directional. peer1 mapped on ha0 means that the DMA engine of ha0 has an address that reaches ha1. Which card's DMA engine performs a copy between two cards depends on the application's operating configuration, so check that the peer card shows mapped in the peer_map of both cards.
  • A device-to-device copy in an unmapped direction ends with an error. The driver does not retry a failed copy over another path or send it by another route.
  • If the address could not be prepared in some direction, dma_map_resource failed for peer[N] is logged in the kernel log when the later-registered of the two cards is registered. The PCI address in front of the message is that of the card that shows unmapped, and N is the number of the peer card.
  • When the peer card is removed, its line disappears, and when it is registered again, the line reappears.

When copies concentrate on one card. Device-to-device copy pins a window of the peer card's memory and uses it for the duration of the copy. A card can pin 12 windows at the same time, and a single copy can use several of them, so when several cards copy to one card at the same time, this limit is reached first. A copy that reaches the limit waits until a window is free, and when ha_tw_pin_wait_ms (default 12000 ms) elapses, it fails without moving anything. In that case, a warning such as TW pin: 12/12 slots pinned for 12000ms, giving up (da=0x…) is logged in the kernel log of the card that provides the windows. If this warning repeats, reduce the number of simultaneous copies concentrated on one card. The range of the parameter is in the Parameter reference.

Translation window boundaries. A copy that straddles a boundary of the translation window size of the peer card's memory (tw_window_size) is cut at the boundary by the driver, which pins one window, transfers, and releases it, repeatedly. The data is moved intact, but the copy can take longer than a copy that finishes within one window.

SRAM access by a device kernel

peer_sram shows the address at which a device kernel running on this card reaches the core SRAM of each LPU installed in the host. There is one line per installed LPU, in PCI address order, and it includes cards that are not bound to the driver and this card itself.

1
cat /sys/class/ha/ha0/peer_sram
1
2
3
4
ha0 0000:1a:00.0 base=- (bar0=0x2000000000 route=self)
ha1 0000:1b:00.0 base=0xe010000000 (bar0=0x2010000000 route=bus)
ha2 0000:5e:00.0 base=0xf800000000 (bar0=0x1f8040000000 route=remap:14)
- 0000:af:00.0 base=- (bar0=0x1f8080000000 route=none:not-bound)
  • The first field is the number of that LPU (haN), or - for an LPU that is not registered with the driver.
  • The only value that a device kernel uses is the one written after base=. The part in parentheses is for reference.
  • Read route as follows.
route Meaning
bus, iova, remap:K A device kernel on this card has an address that reaches the core SRAM of that LPU. base is its start address. The differences among the three are in IOMMU mode and the device-to-device path
self This card itself. There is no base
none:<reason> The driver does not give the device kernel on this card an address for that LPU. There is no base
unknown The verdict cannot be trusted right now. This card or that LPU is being reset, is not yet ready, is in the FATAL state, or is not responding, so the telemetry poller has stopped. It is recalculated when the state recovers

The none reasons that an operator can act on right away are as follows. The remaining reasons are described in sysfs nodes.

Reason Meaning Action
not-bound That card is not bound to any driver. A card beyond the 8 cards that the driver handles also falls under this Bind that card to ha. A card beyond 8 is not bound
other-driver That card is bound to another driver (for example, vfio-pci) To use it for this path, rebind that card to ha

The address windows and IOMMU mappings that this path needs are created by the driver when it probes the device, and they are readjusted after a reset. The operator has nothing to configure separately.

The per-core address that a device kernel uses is obtained from base as follows.

1
base + (c << 23) + w * 128
  • c is the core number: 4 × core group number + core number within the group (0-31).
  • w is the number of the 128-byte word in that core SRAM (0-65535).

PCIe route and ACS

peer_route shows whether the traffic between this card and another card stays inside a PCIe switch or passes through the root complex.

1
cat /sys/class/ha/ha0/peer_route
1
2
peer1 direct
peer2 host-mem
Value Meaning
direct The two cards have a common upstream bridge (a PCIe switch), and no bridge on the route has an ACS setting that sends device-to-device requests upstream or blocks them. Traffic stays inside the switch
host-mem Traffic passes through the root complex. This applies when the two cards are under different root ports (including different CPU sockets), when they are in different PCI domains, or when the ACS of a bridge on the route is set to send requests upstream or block them
unknown The ACS setting of a bridge on the route could not be read. ACS state unreadable; P2P route stays unknown is logged in the kernel log together with the PCI address of that bridge. It can also appear briefly while a card is being registered
  • If there are no other cards, the output is none.
  • The verdict is made when the card is registered with the driver. If you change the ACS setting afterwards, the change takes effect only after you rebind that card or reload the module. The verdict at registration is logged in the kernel log, as in P2P peer[1]: direct.
  • It is for observation only. No value blocks copying or access. Even with host-mem, both paths are still used, and because traffic passes through the root complex, it can be slower than with direct.
  • On a host with the IOMMU enabled, the Linux kernel normally turns on ACS redirect on PCIe bridges even when the IOMMU is left in passthrough, so two cards under the same switch can also show host-mem.

You can check the basis of the verdict yourself. The resolved path of the /sys/class/ha/haN/device link lists, in order, the bridges from the root port down to that card.

1
2
readlink -f /sys/class/ha/ha0/device
readlink -f /sys/class/ha/ha1/device
1
2
/sys/devices/pci0000:17/0000:17:01.0/0000:18:00.0/0000:19:00.0/0000:1a:00.0
/sys/devices/pci0000:17/0000:17:01.0/0000:18:00.0/0000:19:01.0/0000:1b:00.0

In this example, the common upstream bridge of the two cards is 0000:18:00.0, and the bridges on the route between the two cards are 0000:19:00.0, 0000:19:01.0, and 0000:18:00.0. View the ACS setting of each bridge as follows.

1
sudo lspci -vvv -s 0000:19:00.0 | grep ACSCtl

If a bridge on the route has at least one of ReqRedir+, CmpltRedir+, and EgressCtrl+ on its ACSCtl line, the verdict is host-mem. A bridge without the ACS capability has no such line.

A prepared address does not mean the path works

mapped in peer_map and base in peer_sram mean that the driver has prepared an address that reaches the peer card, and peer_route is a verdict derived from the topology and the ACS setting. None of them guarantees that the PCIe fabric actually delivers traffic between the two cards. In particular, whether a host-mem route works depends on the platform's root complex. Before deployment, transfer data of the real size between each pair of cards you plan to use, and compare the data.


IOMMU mode and the device-to-device path

The IOMMU mode determines how the addresses that a card sends out over PCIe are interpreted. Check it for each card as follows.

1
2
readlink /sys/class/ha/ha0/device/iommu_group
cat /sys/class/ha/ha0/device/iommu_group/type
Result Mode
type is DMA or DMA-FQ translation. The IOMMU translates the addresses that the card sends out
type is identity passthrough (for example, the kernel command line iommu=pt)
readlink prints nothing (there is no iommu_group link) The IOMMU is off

The type file exists from kernel 4.19 onward, and also in the 4.18.0-553 kernel of the RHEL 8 family. On a kernel without this file, the presence of an iommu_group link tells you only that the IOMMU is on; to tell whether it is translation or passthrough, check the IOMMU setting on the kernel command line (for example, iommu=pt).

The HyperAccel driver supports all three IOMMU configurations (Supported environments).

Depending on the mode, the way the two paths obtain the peer card's address and the range that a card can reach differ.

translation passthrough or IOMMU off
Device-to-device copy The peer card's BAR2 is mapped into this card's IOMMU address space and used The peer card's physical address is used as is
route in peer_sram iova. The driver maps the peer's BAR0 into this card's IOMMU address space and keeps it for as long as this card stays bound bus if the peer's BAR0 ends within 240 GiB of host address space; if it lies above that, remap:K (K is the window number), where the driver has reconfigured one of this card's address windows to match that BAR0
Range the card can reach Limited to the addresses that the IOMMU has mapped for that card. Because the driver maps the BARs of other LPUs, the card can reach other cards Not restricted by the IOMMU. A device kernel can also reach addresses other than those of the other LPUs
  • The 240 GiB boundary exists because the address window that the card sends out over PCIe points directly at the first 240 GiB of PCIe addresses. In passthrough, this is the host physical address, and in translation, it is the IOMMU address. If the address that the IOMMU assigned exceeds this range in translation, the result is none:iova-above-240g.
  • In passthrough, the window used for remap:K is chosen only from windows whose address range contains neither system RAM nor the memory of another PCI device. On a host with a large amount of system RAM, there may be no window to choose, and the result can be none:no-free-region.
  • The IOMMU setting also affects peer_route through the ACS setting (PCIe route and ACS).

When one card is reset or removed

The other cards keep working. However, the work that was exchanging data with that card, and the peer nodes of the other cards, show the following changes.

What happened to that card What the other cards show
Being reset In peer_sram, that card's line is unknown. It is recalculated when the reset finishes. peer_map and peer_route are unchanged. A device-to-device copy targeting that card is not started and ends with a retryable error (EAGAIN). If the card is reset during a transfer, the copy ends with an error instead of success
FATAL In peer_sram, that card's line stays unknown until a reset clears the FATAL state. A device-to-device copy targeting a card that is FATAL, or whose telemetry polling has stopped, is not started and ends with a no-such-device error (ENODEV). This prevents touching a card that may not respond over PCIe and thereby resetting the host
Removal (unbind, hot-unplug, the removal step of ha-hotreset) In peer_map and peer_route, that card's peerN line disappears, and the output becomes none if no other card remains. In peer_sram, the first field of that card's line becomes - and base becomes unusable. A device-to-device copy targeting that card fails
Registered again (rebind, rescan, the re-probe step of ha-hotreset) The lines reappear. With the same PCI address, the card usually returns to its previous number (Card numbers)
  • A reset reclaims all of the memory of that card. Memory allocated on that card before the reset cannot be used for copies with other cards either, so the application must allocate it again through the SDK.
  • If a device-to-device copy from another card targeting that card is in progress, the removal waits for that copy to finish before proceeding.
  • If the BAR0 address of the card that was registered again has changed, that card's line appears as none:unsupported in the peer_sram of other cards whose IOMMU is in translation mode. Rebinding the card whose peer_sram shows that line, or reloading the module, recalculates it with the new address. Check the BAR0 address in the bar0 field of peer_sram.
  • ha-hotreset removes only the card you specify, issues an SBR on the parent bridge, and then has the driver probe the card again. It does not touch the other cards or the module, and it refuses to reset if there are other devices below the parent bridge (ha-hotreset).

Stop the work between cards before a reset or removal

A reset or removal affects not only the processes that have that card open but also the work on other cards that exchanges data with it. First stop the device-to-device copies that target that card and the device kernels that read and write that card's core SRAM. The results of a device kernel that accessed that card during a reset or removal cannot be trusted.


Trust boundary

Handing out cards does not by itself isolate workloads

  • The default udev rules open the device nodes to all local users (0666).
  • Processes that open the same device node are not isolated from each other's device memory.
  • If you can run a device kernel on a card, you can read and write the core SRAM of the other LPUs that card can reach. Every LPU that has a base in that card's peer_sram falls under this, even if another user or another container is using that LPU.
  • With device-to-device copy, opening the node of just one card is enough to use memory allocated on another card as the source or destination.
  • If the IOMMU is in passthrough or off, the range that a device kernel can reach is not limited to the other LPUs.

When you place workloads that must be isolated from each other on several cards of one host, design on the assumption that these paths exist. How to restrict the users who can open the nodes is described in Restricting access.


See also