Skip to content

5. Error Handling and Reset

Every driver API function returns a standard HaError status code. An execution failure in an asynchronous operation is not reported at the call site. It remains on the stream as an error state and is detected at a later wait. A device reset invalidates all child handles of the target device at once. For the full list of error codes, see the Error Code Reference.


Contents

Section Objective
5.1 Return Contract State of output arguments and the sequence number (seq) on failure
5.2 Stream Errors (Sticky Error) Sticky error behavior, where an asynchronous execution failure remains on the stream
5.3 Device Reset Scope of child object invalidation on reinitialization, and waking of blocked calls

5.1 Return Contract

  • Return type: Most driver API functions return a HaError that indicates success or failure. Some query functions (ha_drv_scan_devices, ha_drv_get_device_id, ha_drv_is_ready) return the result value directly.
  • Output arguments on failure: A failed call may leave its output arguments untouched or may write some values. Use output arguments only when the return value is HaSuccess. The exception is out_synced_seq of ha_drv_wait, which is written even when the synchronization fails (Synchronization in Streams and Events).
  • Failed enqueue: An operation that fails at the enqueue stage is not added to the stream and does not consume a sequence number (seq).
  • Returning is not completion: HaSuccess from an enqueue function only means that the operation was added to the queue. Check the actual execution result at a later synchronization point.

5.2 Stream Errors (Sticky Error)

If an asynchronous operation running on a stream fails, the stream records only the first error and does not record later errors. After that, ha_drv_wait calls and enqueues on that stream keep returning the same error code. No function clears the error. To recover, destroy the explicit stream and create a new one. An error on a default stream is cleared only by a device reset.

An asynchronous failure is reported at the next synchronization wait, not by the function that enqueued the operation, so always check the result of synchronization.

1
2
3
4
5
6
7
8
9
ha_drv_kernel_launch(dev, stream, k, args);   // The enqueue itself succeeds

HaError err = ha_drv_wait(dev, stream, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);
if (err != HaError::HaSuccess) {
  // Failure during device execution.
  // Later enqueues and waits on this stream return the same error. The destroy call may also return it.
  ha_drv_obj_destroy(dev, HA_OBJ_STREAM, stream);
  ha_drv_obj_create(dev, HA_OBJ_STREAM, reinterpret_cast<void **>(&stream));
}

5.3 Device Reset

ha_drv_reset(dev) stops the target device's work, drains the commands left in its queues, releases the memory pool, and forcibly reinitializes the device. Finally, it releases every stream (default streams included), event, and kernel handle of that device.

Child handles are invalidated by reset

Streams, events, memory allocations, and kernel handles obtained before the reset are all invalid after the reset. Only the top-level device handle (HaDrvDev) remains valid. To use the device again after a reset, recreate the child objects you need and reload the kernels. Kernel library handles (HaLibrary_t) and library kernel handles (HaKernel_t), which do not belong to a device, remain valid after a reset, so library kernels only need to be reloaded with ha_drv_kernel_load_from_library().

1
2
3
4
5
if (ha_drv_reset(dev) == HaError::HaSuccess) {
  // dev is still valid. Recreate only the invalidated child objects.
  ha_drv_obj_create(dev, HA_OBJ_STREAM, reinterpret_cast<void **>(&stream));
  ha_drv_kernel_load_from_file(dev, "kernels/add.elf", &k);
}

Handle validity before and after reset

Handles that remain valid after a reset and child objects that must be recreated

  • Calls during reset: While a reset is in progress, functions that operate on that device (allocation, copy, enqueue, wait, object creation, and so on) return HaDeviceNotReady.
  • Blocked calls: When the reset stops the device, calls blocked on that device (synchronization waits, kernel upload waits, and so on) wake up and return HaDeviceNotReady. The reset waits a fixed time for these calls to return; for a call that has not returned by then, it keeps the objects that call uses instead of releasing them. Those objects are released at the next reset or at shutdown.
  • Reset failure: If the reset itself fails, it returns HaDeviceResetFailed, but it still releases the resources described above.
API Sync / Async Behavior
ha_drv_reset() Sync Forcibly reinitializes the device and invalidates all child handles
ha_drv_is_ready() Sync Checks that the device has not been released and the handle is valid. Does not reflect whether a reset is in progress

Next Steps