Skip to content

Support Matrix

Items verified with hapti 0.0.6. Anything not listed here is not verified.

Legend — ✅ supported · 🟡 limited · ❌ not supported

Full stack combinations, including OS, kernel, driver, and firmware, are in the SDK Installation Guide. This page covers what the profiler itself supports.


Platform

Item Status Notes
Linux, x86_64 ✅ Works through LD_PRELOAD
Python 3.10 – 3.12 ✅
Runtime (hart) 0.0.6 ✅
Device tracing ✅
Windows, macOS ❌
Statically linked targets ❌ Cannot be intercepted with LD_PRELOAD

Collection

Feature Status Notes
Runtime API activity ✅
Memory allocation activity ✅ Included in runtime activity with a byte count. The allocating device is not recorded
Lower-layer call activity ✅
Device kernel timeline ✅ --runtime-trace or --device-trace
Device memory transfer timeline ✅
Memory transfer bandwidth track ✅ H2D and D2H, 4 KiB or more
Per-kernel hardware counters 🟡 --device-trace. Default is 20 of core 0, up to 32
Signal logger ✅ --device-trace
Host-device correlation ✅
User annotations ✅ haprof path
Profiler overhead ❌ The record type exists but is not recorded
Per-domain activity selection ❌ Flags are parsed but ignored
Profiling an already running process ❌ Configuration is fixed at process start
Start and stop during execution ❌ haprof works per process. Use the Kineto plugin for region-level control
More than one device per trace ❌ Collects from one device only. Select it with HAPTI_TOOL_DEVICE_ID

CLI Options

The status and effect of each option are in CLI: Option Status.

Option Status Notes
--runtime-trace, --device-trace ✅
--profile-children, --preload, --no-execv ✅
-o, -d, -f, --log-level ✅
Per-domain --*-trace, --external-correlation ❌ Parsed but silently ignored
haprof merge ✅

Output

Format Status Notes
Perfetto .pftrace ✅
Chrome JSON .json ✅
Selecting a single format ✅ -f
Custom path or name ✅ -d, -o

Counters

Counter blocks 6 — AXIBRG, CDMA, DMC, UTIL, CGDMA, MPU
Events 41
Registered per execution Up to 32, 20 by default
Sampling Per kernel execution
Representation Applies to
Per-execution delta AXIBRG, CDMA, CGDMA, MPU
Raw value DMC
Percentage of sample_total UTIL
Derived bandwidth and stall breakdown CGDMA, AXIBRG
Bank skew DMC

The catalog and formulas are in the Counter Reference.

Multi-Process

Feature Status Notes
fork and exec children ✅ --profile-children. One trace per process
Merging traces from the same machine ✅ haprof merge. Aligns exactly because the clock is shared
Straggler report ✅ haprof merge. Requires per-rank .json
Merging traces across machines 🟡 Concatenation works but clocks are not corrected. The report prints a cross-node: yes warning and per-process clock offsets

Integrations

Integration Status Notes
Perfetto UI ✅
Chrome tracing ✅
PyTorch profiler through Kineto ✅ hapti.kineto.register()
Device timeline on the Kineto path ✅
Hardware counters on the Kineto path ❌ Use haprof --device-trace
hapti annotations on the Kineto path ❌ Use torch.profiler.record_function()
Using haprof and the Kineto plugin together ❌ Only one device record ring can be open
Writing a custom collection tool (C++ API) ❌ No public API is provided. Use haprof or the Kineto plugin
Trace loading or query API ❌ Use Perfetto

Known Limitations

  • Per-domain trace flags are parsed but not passed through. They are silently ignored, and host-side activity kinds are always collected in full. Choose whether to include device data with --runtime-trace or --device-trace.
  • Configuration is fixed at process start. You cannot profile an already running process, start or stop during execution, or change settings. If you need region-level collection, use the profile() region of the Kineto plugin, or mark the analysis range with annotations.
  • One device per trace. Select another device with HAPTI_TOOL_DEVICE_ID, and for multi-rank runs with one device per rank, produce per-rank traces with --profile-children.
  • The trace is written at process exit. A forced kill leaves no usable result. Send an interrupt signal instead of SIGKILL so the process exits normally.
  • Profiler overhead is not recorded. Compare against a run without the profiler to estimate the overhead.
  • Memory allocation records do not include the allocating device. In a multi-device process, the allocation location cannot be distinguished from the trace alone, and there is currently no workaround.
  • CGDMA bandwidth and UTIL utilization are estimates, not measurements. Use them only to compare executions, and cross-check UTIL utilization against the stall breakdown metrics.
  • The first execution is not a measurement target. It includes kernel loading, and counter deltas have no valid baseline. Profile steady-state executions after warm-up.
  • In a Kineto session that uses multiple devices, only the lowest-numbered device gets named rows. Many combinations are unverified, so haprof --profile-children is recommended for multiple devices.

The impact and workaround for each limitation are summarized in a table in the Known Issues section of the release notes.


See Also