4. Streams and Events¶
When the host calls an asynchronous API, the API enqueues the work on a stream and returns without waiting for it to complete. A stream is a queue that holds the work to run on the device in order; work enqueued on the same stream runs in the order it was enqueued. The work may not be finished on the device when the call returns, so confirm that it has finished before using its results.
There are two ways to confirm completion. Synchronizing a stream waits until all work enqueued so far has finished. An event marks a point in a stream; use it to check completion only up to that point or to make another stream wait for that point.
Consequences of Missing Synchronization¶
Streams and events are not only a performance tool; they are core infrastructure for correctness.
- Data consistency: Without an explicit execution order, operations can run in an unintended order and corrupt results. A missing synchronization boundary can also let the host access a buffer that the device is still using, corrupting data.
- Debugging difficulty: Bugs from missing synchronization often look correct under light load and show up only intermittently under heavy load, such as in production. They usually appear as silently wrong results without a crash, which makes the cause especially hard to find, so design synchronization carefully from the start when you write asynchronous code.
Contents¶
| Section | Objective |
|---|---|
| 4.1 Stream | Ordering scope, default stream, count limit, asynchronous allocation |
| 4.2 Event | Marking a point in a stream and waiting for it |
| 4.3 Inter-Stream Dependencies and Choosing Synchronization Points | Building dependencies between streams; what to wait for and what not to |
| 4.4 Lifetime & Device Reset | When handles become invalid |
4.1 Stream¶
A stream is a strictly ordered queue whose work is processed serially on the target device.
- Intra-stream: Commands issued to the same stream run strictly in program order.
- Inter-stream: For operations on different streams, there is no architectural guarantee about their relative execution order unless you build an explicit dependency with an event object.
Order is guaranteed within a stream but not between streams
1 2 3 4 5 6 7 8 | |
1 2 3 4 5 6 7 8 9 | |
| API (Python / C++) | Sync / Async | Behavior |
|---|---|---|
stream_create() / HaStreamCreate() |
— | Creates a stream on the current device |
stream_destroy() / HaStreamDestroy() |
Sync | Waits for the enqueued work, then destroys the stream. The stream is destroyed even if that work failed, and that error is returned |
stream_synchronize() / HaStreamSynchronize() |
Sync | Returns when all enqueued work has completed |
stream_query() / HaStreamQuery() |
Sync | HaSuccess if idle, HaStreamNotReady if work is in progress (Python: returns True / False) |
stream_wait_event() / HaStreamWaitEvent() |
Async | Makes the stream wait for an event. Only device execution is serialized |
stream_get_device() / HaStreamGetDevice() |
Sync | Returns the device the stream belongs to |
Default Stream¶
If you pass the reserved constant HA_STREAM_DEFAULT (C++:
nullptr, Python: 0) instead of creating an explicit stream handle, the default
stream instance managed internally by the runtime is bound implicitly.
The default stream is per thread on each device, not shared by all threads that use the device, and has the following properties.
- Per-thread independence: Each host thread has its own default stream on each
device. When different host threads each use
HA_STREAM_DEFAULTon the same device, the work of each thread is processed independently and is not ordered against the others. - Comparison with CUDA: This matches CUDA's per-thread default stream behavior and differs from the legacy default stream, which synchronized all host threads.
- Guidance: To order work across threads, share an explicitly created stream or control the order directly with events.
Per-thread default stream (PTDS) compared with the CUDA legacy default
CUDA lets you select PTDS with a compile option, but hart always uses PTDS.
Stream Count Limit¶
A device can have at most 10 streams created with stream_create() (C++: HaStreamCreate()).
The limit applies only to explicit streams; the per-thread default stream
does not count toward it.
- Recommended pattern (Pre-allocation): Create the set of explicit streams the pipeline needs up front during initialization and reuse them for the lifetime of the process.
- Anti-pattern: Do not create streams dynamically per request; this fails once the limit is exceeded.
Asynchronous Allocation¶
Streams can bring memory management, not just compute commands, into the asynchronous pipeline. Like kernel launches, memory allocation and deallocation are enqueued directly on the stream's ordered execution queue and processed asynchronously.
1 2 3 | |
1 2 3 4 | |
Asynchronous memory operations follow two key rules.
- Using the allocated pointer (
mem_alloc_async): The pointer returned bymem_alloc_async()(C++:HaMemAllocAsync()) can be used right away to enqueue later operations on the same stream. However, the allocation becomes usable only when the stream reaches that point, so synchronize the stream before touching the memory from the host or passing the pointer to a synchronous API or an operation on another stream. - Ownership after free (
mem_free_async): After you callmem_free_async()(C++:HaMemFreeAsync()), the application's ownership of the pointer ends immediately at the call, even though the hardware reclaims the memory later in stream order, so do not reuse it.
Memory allocated asynchronously can be freed with the synchronous interface
mem_free() (C++: HaMemFree()) after stream synchronization has fully completed, or freed asynchronously by calling
mem_free_async() (C++: HaMemFreeAsync()) on the same stream.
4.2 Event¶
An event is a synchronization object that marks a point on a stream. Once recorded on a stream, it lets the host thread wait until that point has completed, or makes another stream run only after that point.
The basic form of a point-to-point dependency, in which one stream records an event and another stream waits for it, is shown below.
Stream-event point-to-point dependency (C++ API. Python: event_record(), stream_wait_event())
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
| API (Python / C++) | Sync / Async | Description |
|---|---|---|
event_create() / HaEventCreate() |
— | Creates an event object |
event_destroy() / HaEventDestroy() |
Sync | Destroys an event object. A recorded event is destroyed after its record has been processed on the stream |
event_record() / HaEventRecord() |
Async | Enqueues an event marker on the stream |
event_synchronize() / HaEventSynchronize() |
Sync | Waits until the stream has executed up to the point where the event was recorded (HaEventNotRecorded if it was never recorded) |
event_query() / HaEventQuery() |
Sync | Queries the event state immediately (HaSuccess / HaEventNotReady / HaEventNotRecorded) |
Destroying events
Do not destroy an event object early while a stream is still waiting on it. Make sure the stream has finished processing so that the event stays valid.
Behavior and Error Handling of Event Record (event_record)¶
- Asynchronous record (Async):
event_record()(C++:HaEventRecord()) leaves a completion marker on the stream and returns immediately without waiting for work to complete. The event moves to the completed state only after all prior work on the stream has actually finished. - Recorded once: An event can be recorded only once. Calling
event_record()again on an event that was already recorded returnsHaEventInvalidArgument. To mark another point, create a new event. - State after a failure: If recording fails, the event stays unrecorded. You can record the same event again without destroying or recreating it.
Waiting on and Querying Events (event_synchronize / event_query)¶
- Synchronous wait (
event_synchronize):event_synchronize()(C++:HaEventSynchronize()) blocks the calling thread until the event completes. If the event has never been recorded, it does not wait indefinitely but immediately returnsHaEventNotRecorded. Unlikestream_synchronize(), which waits for all work on the stream, it is useful for waiting on one specific point. - Immediate query (
event_query):event_query()(C++:HaEventQuery()) queries the event state immediately and returns two distinct not-complete states.HaEventNotReadymeans the event was recorded but the work has not completed yet, andHaEventNotRecordedmeansevent_record()was never called. Handle the two states separately in polling loops to avoid incorrect wait logic.
4.3 Inter-Stream Dependencies and Choosing Synchronization Points¶
Building Inter-Stream Dependencies with stream_wait_event()¶
As noted in Stream, execution order across different streams
is not guaranteed without an explicit dependency built through an event.
stream_wait_event() (C++: HaStreamWaitEvent()) is an asynchronous (Async) call that does not block the host
thread; it inserts a device-level inter-stream dependency command into the queue of the
target stream and returns immediately.
This asynchronous dependency control is the key tool for fully overlapping background data transfers and kernel compute on the timeline, removing pipeline bottlenecks and maximizing throughput.
Example of an event-based dependency between a copy stream and a compute stream
Choosing a Synchronization Point¶
Choose the synchronization API based on the completion scope you need to check and the execution dependency. Synchronizing only the minimum scope avoids unnecessary blocking of the host thread and maximizes overall throughput.
| Criterion | Recommended API (Python / C++) |
|---|---|
| Has all work enqueued on this stream finished? | stream_synchronize(stream) / HaStreamSynchronize(stream) |
| Has work up to a specific point finished? | event_record()/HaEventRecord(), then event_synchronize()/HaEventSynchronize() |
| Check whether work has finished without waiting | stream_query()/HaStreamQuery() · event_query()/HaEventQuery() |
| Stream B must start after a specific point in stream A | event_record()/HaEventRecord(), then stream_wait_event()/HaStreamWaitEvent() |
4.4 Lifetime & Device Reset¶
The lifetime of stream and event objects depends on the device that created them. Resetting a
device releases all of its resources, destroying every stream and event object that belongs to it.
To use the reset device again, each thread must call set_device() (C++: HaSetDevice())
again.
While a reset is in progress, every new operation issued
to the device (memory allocation, free, and transfer; kernel launches;
stream and event operations; and so on) is rejected immediately with the HaDeviceNotReady error
code.
State changes of the device, streams, and events during a device reset transition (C++ API. Python: launch_kernel_async_with_binary_path(), reset_device())
Streams and events are invalidated by a reset
After reset_device() (C++: HaResetDevice()) or reset_all_devices() (C++:
HaResetAllDevices()) is called, every stream and
event handle that belonged to the device immediately becomes stale. Calls with an invalidated handle
are rejected with HaStreamInvalidHandle or HaEventInvalidHandle, so explicitly destroy the related
handles before the reset and create new handles as needed after the reset completes.
Responsibility for stale handles
New operations issued while a reset is in progress are blocked with HaDeviceNotReady, and stale stream and event handles
that remain after the reset completes are rejected with the error codes above. Device pointers are
rejected with HaMemoryInvalidAddress by a transfer and HaMemoryFreeFailed by a free, but if a new allocation after the reset receives the same address, the old pointer
refers to the new allocation and is not rejected. You must therefore manage lifetimes strictly
yourself.