Multiple devices¶
This page covers what operators need to know when a host has several LPUs. It explains the rules the driver uses to number cards and how to identify a card without relying on its number, what operates independently on each card, how to check the data paths between cards through sysfs, how the IOMMU mode and the PCIe topology affect those paths, what other cards show when one card is reset or removed, and the trust boundary when several cards are divided up for use. The output format of each node is in sysfs nodes.
Card numbers¶
For each card that it probes successfully, the driver creates the device node /dev/haN and the sysfs directory /sys/class/ha/haN/. On this page, a "card registered with the driver" means a card whose nodes were created this way.
N is 0-7, and the driver handles at most 8 cards per host. No nodes are created for a card that gets no number. Such a card appears in lspci but is not bound to the driver, and the kernel log shows only Probing device for that card, not Probe successful.
Numbers are assigned by the following rules.
- When the module is loaded, the driver numbers the LPUs present at that time from
0in PCI address order. It counts them whether or not they are bound to the driver. So as long as the same cards are installed, each card keeps its number across module reloads and host reboots. - The number of a card whose probe fails, or of a card held by another driver (such as
vfio-pci), stays empty, and the numbers of the cards after it do not move down. When that card is later bound toha, it receives the number reserved for it. - While the module stays loaded, if a card is removed and then reappears at the same PCI address (rebind, rescan,
ha-hotreset), it receives its previous number again, provided no other card has taken that number in the meantime. - A card that appears after the module is loaded first receives a number not reserved for any card. Only when there is no such number does it take the number of a card that is currently absent, and that card may not get its previous number back when it returns.
- A card whose PCI address changed (because it was moved to another slot, or because re-enumeration changed its bus number) is treated as a new card.
- If you add or remove a card and then reload the module (including a reboot), the cards with a higher PCI address than that card get different numbers.
- If there are more than 8 LPUs, the kernel logs
ha: more than 8 cards; the extra cards take any free number, so card numbers can change across reloadswhen the module is loaded, and after a reload the numbers and the cards that get nodes can change. Keep 8 or fewer LPUs on one host. - While a card is removed or has failed its probe, a gap appears in the numbers. When counting cards, check every existing number with
ls /sys/class/ha/.
All /dev/haN nodes share one major device number, and the minor is the card number. The kernel picks the major when the module is loaded, so it can change when the module is reloaded or the host reboots. A container keeps the device numbers it was created with, so recreate the container if the major changes. Read the current device number from /sys/class/ha/haN/dev.
A number is not a card's identity
Numbers follow the PCI address order of the installed cards, so they change when you add, remove, or move a card. Where you must point to a specific card, such as a monitoring label, a container assignment, or a firmware update target, use the PCI address or serial. The CDI device names given to containers (Container support) and hyperaccel-smi's -i and HYPERACCEL_VISIBLE_DEVICES (hyperaccel-smi) also select cards by number.
Identifying a card by PCI address and serial¶
/sys/class/ha/haN/device is a link to the PCI device of that card. The following command prints the number, PCI address, and serial of each card, one line per card.
1 2 3 | |
1 2 | |
serialis a unique number recorded on the chip, so it stays the same even if you move the card to another slot. A card whose firmware does not report this value, or a card that is not yet ready, showsunknown; in that case, use the PCI address.- Write the PCI address in the format that includes the domain (
0000:1a:00.0). On a host with several PCI domains, two or more cards can have the same bus, device, and function numbers. - The
Bus-Idcolumn ofhyperaccel-smi, the-doption ofha-fw-update, and the--bdfoption ofha-hotresetuse the same PCI address.
To also see cards that are not bound to the driver, check the PCI bus directly.
1 2 3 4 | |
What operates independently on each card¶
| Item | Scope |
|---|---|
| Device node and telemetry | Per card. /dev/haN, /sys/class/ha/haN/ |
FATAL state and health |
Per card. The other cards keep working even if one card becomes FATAL |
| Reset | Per card. Only the work on the card that was reset is cleared |
| Firmware update | Per card. ha-fw-update writes to the one card you specify |
There are two exceptions even among the items that operate independently on each card.
- The data path between cards spans two cards. When you reset or remove one card, work on other cards that was exchanging data with that card is also affected (When one card is reset or removed).
- PCIe error recovery targets everything below the port where the kernel detected the error. If an error is detected at a PCIe switch or at a port above it, the cards below it may be reset together (FATAL and recovery).
Data paths between cards¶
Cards can exchange data with each other over PCIe without going through host memory. There are two paths. One is device-to-device copy, in which the DMA engine of one card moves data to and from the memory of another card. The other is access in which compute code running on a card (called a device kernel on this page; it is different from the Linux kernel) directly reads and writes the core SRAM of another card. Core SRAM is the SRAM that each compute core has, and it differs from the host-shared SRAM that sram_size and sram_used in telemetry refer to.
| Device-to-device copy | SRAM access by a device kernel | |
|---|---|---|
| What moves the data | The DMA engine of one of the two cards | A device kernel running on one card |
| What it reaches on the peer card | The memory of the peer card, through BAR2 | The core SRAM of the peer card, through BAR0 |
| What starts it | The SDK's device-to-device copy | A device kernel that reads and writes the core SRAM of another card |
| Node to check | peer_map |
peer_sram |
- The two paths are independent of each other.
peer_mapreports only on device-to-device copy, andpeer_sramreports only on access by a device kernel. - The verdict on the PCIe route between two cards (
peer_route) is common to both paths. - The three telemetry nodes (
peer_map,peer_sram,peer_route) read only host-side state, so you can read them even while a card has stopped, and you can read them without root privileges.
Device-to-device copy¶
When a card is registered with the driver, the driver prepares, in both directions between that card and the cards already registered, the DMA addresses through which a DMA engine reaches the BAR of the peer card. peer_map shows the result, with one card as the reference, for the direction toward the other cards.
1 | |
1 2 | |
- There is one line for each other registered card, and the
NinpeerNis the number of that card. If there are no other cards, the output isnone. - It is directional.
peer1 mappedon ha0 means that the DMA engine of ha0 has an address that reaches ha1. Which card's DMA engine performs a copy between two cards depends on the application's operating configuration, so check that the peer card showsmappedin thepeer_mapof both cards. - A device-to-device copy in an
unmappeddirection ends with an error. The driver does not retry a failed copy over another path or send it by another route. - If the address could not be prepared in some direction,
dma_map_resource failed for peer[N]is logged in the kernel log when the later-registered of the two cards is registered. The PCI address in front of the message is that of the card that showsunmapped, andNis the number of the peer card. - When the peer card is removed, its line disappears, and when it is registered again, the line reappears.
When copies concentrate on one card. Device-to-device copy pins a window of the peer card's memory and uses it for the duration of the copy. A card can pin 12 windows at the same time, and a single copy can use several of them, so when several cards copy to one card at the same time, this limit is reached first. A copy that reaches the limit waits until a window is free, and when ha_tw_pin_wait_ms (default 12000 ms) elapses, it fails without moving anything. In that case, a warning such as TW pin: 12/12 slots pinned for 12000ms, giving up (da=0x…) is logged in the kernel log of the card that provides the windows. If this warning repeats, reduce the number of simultaneous copies concentrated on one card. The range of the parameter is in the Parameter reference.
Translation window boundaries. A copy that straddles a boundary of the translation window size of the peer card's memory (tw_window_size) is cut at the boundary by the driver, which pins one window, transfers, and releases it, repeatedly. The data is moved intact, but the copy can take longer than a copy that finishes within one window.
SRAM access by a device kernel¶
peer_sram shows the address at which a device kernel running on this card reaches the core SRAM of each LPU installed in the host. There is one line per installed LPU, in PCI address order, and it includes cards that are not bound to the driver and this card itself.
1 | |
1 2 3 4 | |
- The first field is the number of that LPU (
haN), or-for an LPU that is not registered with the driver. - The only value that a device kernel uses is the one written after
base=. The part in parentheses is for reference. - Read
routeas follows.
route |
Meaning |
|---|---|
bus, iova, remap:K |
A device kernel on this card has an address that reaches the core SRAM of that LPU. base is its start address. The differences among the three are in IOMMU mode and the device-to-device path |
self |
This card itself. There is no base |
none:<reason> |
The driver does not give the device kernel on this card an address for that LPU. There is no base |
unknown |
The verdict cannot be trusted right now. This card or that LPU is being reset, is not yet ready, is in the FATAL state, or is not responding, so the telemetry poller has stopped. It is recalculated when the state recovers |
The none reasons that an operator can act on right away are as follows. The remaining reasons are described in sysfs nodes.
| Reason | Meaning | Action |
|---|---|---|
not-bound |
That card is not bound to any driver. A card beyond the 8 cards that the driver handles also falls under this | Bind that card to ha. A card beyond 8 is not bound |
other-driver |
That card is bound to another driver (for example, vfio-pci) |
To use it for this path, rebind that card to ha |
The address windows and IOMMU mappings that this path needs are created by the driver when it probes the device, and they are readjusted after a reset. The operator has nothing to configure separately.
The per-core address that a device kernel uses is obtained from base as follows.
1 | |
cis the core number:4 × core group number + core number within the group(0-31).wis the number of the 128-byte word in that core SRAM (0-65535).
PCIe route and ACS¶
peer_route shows whether the traffic between this card and another card stays inside a PCIe switch or passes through the root complex.
1 | |
1 2 | |
| Value | Meaning |
|---|---|
direct |
The two cards have a common upstream bridge (a PCIe switch), and no bridge on the route has an ACS setting that sends device-to-device requests upstream or blocks them. Traffic stays inside the switch |
host-mem |
Traffic passes through the root complex. This applies when the two cards are under different root ports (including different CPU sockets), when they are in different PCI domains, or when the ACS of a bridge on the route is set to send requests upstream or block them |
unknown |
The ACS setting of a bridge on the route could not be read. ACS state unreadable; P2P route stays unknown is logged in the kernel log together with the PCI address of that bridge. It can also appear briefly while a card is being registered |
- If there are no other cards, the output is
none. - The verdict is made when the card is registered with the driver. If you change the ACS setting afterwards, the change takes effect only after you rebind that card or reload the module. The verdict at registration is logged in the kernel log, as in
P2P peer[1]: direct. - It is for observation only. No value blocks copying or access. Even with
host-mem, both paths are still used, and because traffic passes through the root complex, it can be slower than withdirect. - On a host with the IOMMU enabled, the Linux kernel normally turns on ACS redirect on PCIe bridges even when the IOMMU is left in passthrough, so two cards under the same switch can also show
host-mem.
You can check the basis of the verdict yourself. The resolved path of the /sys/class/ha/haN/device link lists, in order, the bridges from the root port down to that card.
1 2 | |
1 2 | |
In this example, the common upstream bridge of the two cards is 0000:18:00.0, and the bridges on the route between the two cards are 0000:19:00.0, 0000:19:01.0, and 0000:18:00.0. View the ACS setting of each bridge as follows.
1 | |
If a bridge on the route has at least one of ReqRedir+, CmpltRedir+, and EgressCtrl+ on its ACSCtl line, the verdict is host-mem. A bridge without the ACS capability has no such line.
A prepared address does not mean the path works
mapped in peer_map and base in peer_sram mean that the driver has prepared an address that reaches the peer card, and peer_route is a verdict derived from the topology and the ACS setting. None of them guarantees that the PCIe fabric actually delivers traffic between the two cards. In particular, whether a host-mem route works depends on the platform's root complex. Before deployment, transfer data of the real size between each pair of cards you plan to use, and compare the data.
IOMMU mode and the device-to-device path¶
The IOMMU mode determines how the addresses that a card sends out over PCIe are interpreted. Check it for each card as follows.
1 2 | |
| Result | Mode |
|---|---|
type is DMA or DMA-FQ |
translation. The IOMMU translates the addresses that the card sends out |
type is identity |
passthrough (for example, the kernel command line iommu=pt) |
readlink prints nothing (there is no iommu_group link) |
The IOMMU is off |
The type file exists from kernel 4.19 onward, and also in the 4.18.0-553 kernel of the RHEL 8 family. On a kernel without this file, the presence of an iommu_group link tells you only that the IOMMU is on; to tell whether it is translation or passthrough, check the IOMMU setting on the kernel command line (for example, iommu=pt).
The HyperAccel driver supports all three IOMMU configurations (Supported environments).
Depending on the mode, the way the two paths obtain the peer card's address and the range that a card can reach differ.
| translation | passthrough or IOMMU off | |
|---|---|---|
| Device-to-device copy | The peer card's BAR2 is mapped into this card's IOMMU address space and used | The peer card's physical address is used as is |
route in peer_sram |
iova. The driver maps the peer's BAR0 into this card's IOMMU address space and keeps it for as long as this card stays bound |
bus if the peer's BAR0 ends within 240 GiB of host address space; if it lies above that, remap:K (K is the window number), where the driver has reconfigured one of this card's address windows to match that BAR0 |
| Range the card can reach | Limited to the addresses that the IOMMU has mapped for that card. Because the driver maps the BARs of other LPUs, the card can reach other cards | Not restricted by the IOMMU. A device kernel can also reach addresses other than those of the other LPUs |
- The 240 GiB boundary exists because the address window that the card sends out over PCIe points directly at the first 240 GiB of PCIe addresses. In passthrough, this is the host physical address, and in translation, it is the IOMMU address. If the address that the IOMMU assigned exceeds this range in translation, the result is
none:iova-above-240g. - In passthrough, the window used for
remap:Kis chosen only from windows whose address range contains neither system RAM nor the memory of another PCI device. On a host with a large amount of system RAM, there may be no window to choose, and the result can benone:no-free-region. - The IOMMU setting also affects
peer_routethrough the ACS setting (PCIe route and ACS).
When one card is reset or removed¶
The other cards keep working. However, the work that was exchanging data with that card, and the peer nodes of the other cards, show the following changes.
| What happened to that card | What the other cards show |
|---|---|
| Being reset | In peer_sram, that card's line is unknown. It is recalculated when the reset finishes. peer_map and peer_route are unchanged. A device-to-device copy targeting that card is not started and ends with a retryable error (EAGAIN). If the card is reset during a transfer, the copy ends with an error instead of success |
| FATAL | In peer_sram, that card's line stays unknown until a reset clears the FATAL state. A device-to-device copy targeting a card that is FATAL, or whose telemetry polling has stopped, is not started and ends with a no-such-device error (ENODEV). This prevents touching a card that may not respond over PCIe and thereby resetting the host |
Removal (unbind, hot-unplug, the removal step of ha-hotreset) |
In peer_map and peer_route, that card's peerN line disappears, and the output becomes none if no other card remains. In peer_sram, the first field of that card's line becomes - and base becomes unusable. A device-to-device copy targeting that card fails |
Registered again (rebind, rescan, the re-probe step of ha-hotreset) |
The lines reappear. With the same PCI address, the card usually returns to its previous number (Card numbers) |
- A reset reclaims all of the memory of that card. Memory allocated on that card before the reset cannot be used for copies with other cards either, so the application must allocate it again through the SDK.
- If a device-to-device copy from another card targeting that card is in progress, the removal waits for that copy to finish before proceeding.
- If the BAR0 address of the card that was registered again has changed, that card's line appears as
none:unsupportedin thepeer_sramof other cards whose IOMMU is in translation mode. Rebinding the card whosepeer_sramshows that line, or reloading the module, recalculates it with the new address. Check the BAR0 address in thebar0field ofpeer_sram. ha-hotresetremoves only the card you specify, issues an SBR on the parent bridge, and then has the driver probe the card again. It does not touch the other cards or the module, and it refuses to reset if there are other devices below the parent bridge (ha-hotreset).
Stop the work between cards before a reset or removal
A reset or removal affects not only the processes that have that card open but also the work on other cards that exchanges data with it. First stop the device-to-device copies that target that card and the device kernels that read and write that card's core SRAM. The results of a device kernel that accessed that card during a reset or removal cannot be trusted.
Trust boundary¶
Handing out cards does not by itself isolate workloads
- The default udev rules open the device nodes to all local users (
0666). - Processes that open the same device node are not isolated from each other's device memory.
- If you can run a device kernel on a card, you can read and write the core SRAM of the other LPUs that card can reach. Every LPU that has a
basein that card'speer_sramfalls under this, even if another user or another container is using that LPU. - With device-to-device copy, opening the node of just one card is enough to use memory allocated on another card as the source or destination.
- If the IOMMU is in passthrough or off, the range that a device kernel can reach is not limited to the other LPUs.
When you place workloads that must be isolated from each other on several cards of one host, design on the assumption that these paths exist. How to restrict the users who can open the nodes is described in Restricting access.
See also¶
- sysfs nodes — definitions of
peer_map,peer_route, andpeer_sram, and allnonereasons - Parameter reference —
ha_tw_pin_wait_ms - Telemetry — how nodes are updated and what
unknownmeans - FATAL and recovery — reset, unbind, PCIe error recovery
- ha-hotreset — hot reset of a single card
- hyperaccel-smi —
Bus-Idandserialof each card - Container support — CDI device names
- Kernel log messages
- Supported environments — supported IOMMU configurations and number of cards
- Troubleshooting