7. Troubleshooting¶
Before guessing at a cause, search this page for the error message printed to the console or the symptom you see in the trace viewer.
Each entry in this chapter has three parts: Symptom, Cause, and Remedy. The cause is stated so that you have a technical basis for judging similar problems that are not documented here. Most problems in this chapter follow directly from the behavior described in Chapter 2 Capturing a Trace through Chapter 4 Trace Data Analysis Guide. If the same symptom recurs, refer back to those chapters.
Contents¶
| Section | Objective |
|---|---|
| Before You Start | Basic diagnostic information to collect before troubleshooting or requesting support |
| No Trace File | Files not created or missing, and multi-process environments |
| Empty or Partial Trace | Missing device tracks, counters, or annotations, and clock synchronization errors |
| Options and Commands | CLI flags not applied, file path errors, and merge errors |
| Numbers That Look Wrong | Checking counter, bandwidth, utilization, and overhead values |
| Trace Does Not Open | Perfetto rejecting a file and oversized JSON |
| Execution Interference | LD_PRELOAD conflicts and exit code control |
| Requesting Support | What to send to the HyperAccel support team if the problem is not resolved |
Before You Start¶
When requesting support, collect the following diagnostic information. These are the first items checked in technical support.
1 2 3 4 | |
No Trace File¶
Nothing Is Created¶
Symptom — The program finished, but no .pftrace or .json output file was created.
Cause — Most likely the process was forcibly terminated, the output directory was not where you expected, or a required tool library is missing.
Remedy
- Check whether the process was forcibly terminated — Trace data is flushed to the file all at once when the
process exits normally. Nothing is left after
SIGKILL, a segmentation fault, a forced container stop, and similar situations. Do not force-kill the process; send an interrupt signal so that it exits normally. - Check the output path — The default location is the working directory where
haprofwas run, or the path given with-d. - Check that the tool library exists — If a required library is missing,
haprofprints an error about failing to findlibhapti_tool.soand exits before running the program. In that case, reinstall the HyperAccel SDK.
Only One Trace in a Multi-Process Run¶
Symptom — In a multi-rank distributed run, only one pair of trace files is created.
Cause — Child processes are excluded from profiling by default. The tool clears its sentinel variable at startup to prevent child processes from creating duplicate traces.
Remedy — Run with the --profile-children option. A separate <hostname>_<pid>_<starttime> file pair is created
for each rank.
Empty or Partial Trace¶
The Device Track Is Empty¶
Symptom — Host API calls are shown normally, but no kernel or memory transfer slices appear on the LPU Device
track.
Cause — Device tracing was not enabled, or the device profiling configuration was rejected.
Remedy
- Check that you passed the
--runtime-traceor--device-traceflag. Without it, only the host side is recorded by design. -
Look for the following strings in the standard error (
stderr) output.Diagnostic message Root cause Action S6 DEVICE RECORDS DISABLED (configure rc=...)The driver and firmware do not support profiling Upgrade to a driver and firmware that support profiling no LPU device to profile/HaSetDevice(N) failedThe target device for profiling was not found Check that the device number set in HAPTI_TOOL_DEVICE_IDis validdevice timeline: device reset or teardown before setupA device reset or runtime shutdown happened before profiling was set up To profile the range after a reset, start a new run after the reset -
If the program exits before the device finishes its work, records do not exist for batches that have not completed. Synchronize the stream explicitly right before the program exits (see Device Tracing: When Device Records Are Collected).
Kernel Positions Make No Sense¶
Symptom — Kernel slices appear at times that do not match the host timeline at all (far in the past or future).
Cause — The clock synchronization that converts device cycles to nanoseconds failed, so kernel timestamps were
left as raw cycle values. stderr records kernel clock_sync=FAILED(kernel ts left raw).
Remedy — Check in the Support Matrix that your driver and firmware versions support clock synchronization. Kernel times from a failed clock synchronization cannot be compared with host records.
Gaps in the Device Timeline¶
Symptom — Timeline ranges where kernels should be are empty, and a [hapti-tool] warning is printed at process
exit.
Cause — Records were lost, for example through an overrun, before the reader thread collected them.
Remedy — See the warning message table in Device Tracing: Cost and Loss. If short kernels run at very high frequency, narrow the range being profiled.
Only One Device's Activity Is Visible¶
Symptom — The process runs in a multi-device environment, but only one device track appears on the timeline.
Cause — Device records are collected from only one device per session, index 0 by default.
Remedy — Specify another device index with the HAPTI_TOOL_DEVICE_ID environment variable, or in multi-rank runs
use the --profile-children option so that each rank profiles its own device.
User Python Code Is Not Visible¶
Symptom — Python functions or internal model steps do not appear on the trace timeline.
Cause — The profiler intercepts the runtime API and its lower-layer calls, not the inside of the Python interpreter.
Remedy — Explicitly add annotations to the key code ranges you want to analyze (see Chapter 5 Annotations).
--device-trace Is Set but the Desired Hardware Counters Are Missing¶
Symptom — Kernel slices are collected normally, but CGDMA or MPU counters, or values for core 1 and above,
are not shown.
Cause — The default configuration registers only 20 key events: AXIBRG, DMC, and UTIL of core 0. If there
are no kernel slices at all, the cause is the same as "The Device Track Is Empty" above.
Remedy — Specify the events you need with the
HAPTI_COUNTERS environment variable.
Invalid tokens are printed to stderr as [ha_profiling] HAPTI_COUNTERS: unknown ... — skipped and safely
ignored.
No Runtime API Calls in the torch.profiler Trace¶
Symptom — The trace collected with the Kineto plugin shows no Ha* runtime API calls or device activity.
Cause — The hart runtime was loaded more than once in one process (for example, when hart installed in
different environments is loaded together), and the plugin attached to an instance other than the one the
application actually uses.
Remedy — Clean up the environment so that the process loads only the hart module installed in a single
environment. Also check that the ProfilerActivity.PrivateUse1 activity is in the collection list
(see Running the Profiler).
hapti Annotations Do Not Appear in the torch.profiler Trace¶
Symptom — The trace collected with the Kineto plugin does not contain hapti.annotate() ranges.
Cause — The Kineto collection path does not collect hapti annotation activity by design.
Remedy — Use the standard PyTorch API torch.profiler.record_function() instead
(see PyTorch Profiler Integration).
Options and Commands¶
Per-Domain Flags Such as --kernel-trace Do Not Work¶
Symptom — Options such as --kernel-trace and --driver-trace do not change the trace contents.
Cause — Per-domain flags are parsed but not yet passed to the tool. Host-side activity is always collected in full regardless of the flags.
Remedy — Control whether device data is collected with the --runtime-trace or --device-trace option. For the
detailed support status of each option, see the CLI Reference.
must be a bare file name, not a path¶
Symptom — A command of the form haprof -o traces/run1 -- ... is rejected before it runs.
Cause — The -o option accepts only a bare file name without a directory path. Specifying the directory is the
job of the -d option.
Remedy — Specify the directory and file name separately, as in haprof -d traces -o run1 -- ....
--profile-children without the json format¶
Symptom — Running with the -f pftrace --profile-children combination prints a warning.
Cause — The straggler report of haprof merge reads and analyzes the .json file of each rank's trace.
Remedy — If you need the straggler report, include json in the -f option. If you need only the merged
timeline, you can ignore the warning.
[haprof merge] error: no traces found¶
Symptom — haprof merge exits without finding any traces.
Cause — There are no per-rank trace files in the -d path (default: current directory).
Remedy — Specify the directory where the trace files are actually stored with -d, or pass the files directly as
arguments.
Numbers That Look Wrong¶
Counter Values of the First Execution Are Empty or Abnormally Large¶
Symptom — The hardware counter values of the first kernel execution are much higher than later values or distorted.
Cause — Free-running hardware counters are computed as the difference (delta) from the previous execution. The first execution has no valid previous baseline, so values accumulated by work done before profiling started are mixed in.
Remedy — Analyze steady-state executions after a sufficient warm-up phase, and discard the first execution's results.
CGDMA Bandwidth Differs from a Value You Computed¶
Symptom — A kernel's CGDMA.c<n>.read BW (GB/s) or CGDMA.c<n>.write BW (GB/s) metric does not match the bandwidth computed from transferred bytes and
duration.
Cause — This counter value is not a measured byte count but a utilization metric scaled by the
peak defined by the firmware. The memory bandwidth (GB/s) on transfer slices, on the other hand, is a measured
value.
Remedy — Use the kernel bandwidth metric only for relative comparison between executions, not as an absolute transfer rate (see Counter Reference: Display Rules).
Utilization Is Lower Than Expected¶
Symptom — The UTIL.*% metric of a unit you expected to be heavily loaded is very low.
Cause — UTIL counters are aggregated by sampling, not by cycle-exact measurement. Units that run and finish in
short bursts between samples are under-measured.
Remedy — Treat utilization metrics as duty cycle estimates, and verify them by cross-checking against the stall breakdown metrics.
The Profiled Run Is Much Slower¶
Symptom — Total run time increases significantly when the application runs under haprof.
Cause — This is expected profiler overhead. In particular, with --device-trace enabled on workloads that run
small kernels at high frequency, the fixed per-call overhead accumulates and total time increases significantly.
Remedy — Use profiled runs to diagnose where time is spent, and always compare wall-clock results from runs without the profiler to measure how much faster things actually got.
Trace Does Not Open¶
Perfetto Rejects the File¶
Symptom — The ui.perfetto.dev web viewer cannot open the trace file or shows only part of it.
Cause — Usually the process terminated abnormally, skipping the final file write, so the file is truncated.
Remedy — Check the size of the generated file, run the program again so that it exits normally, and open the new trace file.
The JSON Trace File Is Too Large¶
Symptom — The generated .json file is so large that chrome://tracing freezes or becomes very slow.
Cause — The Chrome JSON format is structurally very verbose, and long runs with device tracing enabled produce a large number of records.
Remedy — Prefer the .pftrace (Perfetto Protobuf) binary format. If needed, skip JSON generation with the
-f pftrace option, or narrow the profiled range with annotations.
Execution Interference¶
The Application Behaves Differently Under haprof¶
Symptom — Library loading or initialization that works without the profiler fails under haprof.
Cause — The profiler uses the LD_PRELOAD mechanism and prepends the package directory to the library search
path so that only one copy of the runtime library is loaded. Programs that use their own LD_PRELOAD or have an
unusual library layout can conflict with this.
Remedy — Do not set the LD_PRELOAD environment variable yourself; pass additional libraries through the
--preload option of haprof. This keeps the library loading order correct.
The Process Exit Code Is Not What You Expect¶
Symptom — A parent shell script that wraps haprof does not receive the target program's exit code as expected.
Cause — By default, haprof replaces its own process image with the target command through the execvpe system
call, so the final exit code is the target program's. In wrapper scripts that need a separate subprocess boundary,
this may not be what you intend.
Remedy — Use the --no-execv option to run the target as a separate child process and propagate its exit code
explicitly.
Requesting Support¶
If the issue is not resolved by the troubleshooting entries above, contact the HyperAccel support team with all of the following diagnostic information attached.
- The output of the basic diagnostic information collected in Before You Start
- The full
haprofcommand you ran and the complete standard error (stderr) output of a rerun with the--log-level infooption added - Whether a trace file was created, and the size of the generated files
- The HyperAccel SDK version you are using (
pip show hapti) - Whether the program exits normally without the profiler