Skip to content

4. Streams and Events

Every asynchronous operation in the driver API is enqueued in order on a stream. To wait for completion or query status from the host, use ha_drv_wait for streams, events, and devices alike. Streams and events are created and destroyed with the same functions (ha_drv_obj_create / ha_drv_obj_destroy), and stream operations that have no dedicated function are enqueued with ha_drv_stream_post.


Contents

Section Objective
4.1 Stream Stream creation, the default stream (HA_STREAM_DEFAULT), count limit, and stream operations
4.2 Event Event recording and the rules for cross-stream dependency waits
4.3 Synchronization The five synchronization modes of ha_drv_wait and completion checks based on sequence numbers (seq)

4.1 Stream

A stream is an ordered queue in which work is processed serially on the device. Kernel launches and memory copies enqueued on the same stream run in the order they were enqueued. Across different streams, execution order is not guaranteed unless the streams are linked with events.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
HaStream_t stream = HA_STREAM_DEFAULT;
ha_drv_obj_create(dev, HA_OBJ_STREAM, reinterpret_cast<void **>(&stream));

ha_drv_memcpy(dev, dev_in, reinterpret_cast<HaDevicePtr>(host_in), n, HaMemCpyDirection::HostToDevice, stream);
ha_drv_kernel_launch(dev, stream, k, args);
ha_drv_memcpy(dev, reinterpret_cast<HaDevicePtr>(host_out), dev_out, n, HaMemCpyDirection::DeviceToHost, stream);

// All three enqueued operations have completed at this synchronization point.
ha_drv_wait(dev, stream, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);
ha_drv_obj_destroy(dev, HA_OBJ_STREAM, stream);
API Sync / Async Behavior
ha_drv_obj_create() + HA_OBJ_STREAM Sync Creates an explicit stream on the device
ha_drv_obj_destroy() + HA_OBJ_STREAM Sync Waits until the work left on the stream finishes, then destroys it
ha_drv_stream_post() Async Adds one stream operation (HaDrvStreamOp) to the queue
ha_drv_wait() + HA_WAIT_STREAM_SYNC Sync Waits on the host until all enqueued work completes
ha_drv_wait() + HA_WAIT_STREAM_QUERY Sync Returns the stream error if there is one; otherwise HaSuccess if idle, HaStreamNotReady if in progress

Default Stream

Passing HA_STREAM_DEFAULT as the stream argument refers to the default stream that the calling thread uses on that device (equivalent to CUDA's CU_STREAM_PER_THREAD). The default stream is created automatically on first use. When the thread exits, the stream is reclaimed after its remaining work has finished; until then, HA_WAIT_DEVICE_SYNC still synchronizes it.

  • Per-thread independence: If two different threads each use HA_STREAM_DEFAULT, they are different streams and execution order is not guaranteed. If you need ordering across threads, share an explicit stream or synchronize with events.
  • Relationship to the runtime API: It is the same stream as the default stream the runtime API uses for the same thread and device in the same process. However, do not mix the two APIs on one device (see Section 1.2).

Stream Count Limit

A device can hold up to 10 explicit streams at a time. Default streams do not count toward this limit; streams created with the runtime API on the same device do. Once the limit is reached, ha_drv_obj_create returns HaStreamCreationFailed.

Stream Operations

Stream operations other than kernel launches, including memory copies, are enqueued with ha_drv_stream_post by putting the operation type and payload in a HaDrvStreamOp structure.

Operation type Behavior
HA_STREAM_OP_MEMCPY Copies data between the host and the device, or within the device
HA_STREAM_OP_FREE Returns memory asynchronously after the preceding work completes (see Section 3.1)
HA_STREAM_OP_LOAD_FILE Loads directly by DMA from a file descriptor into device memory (see Section 3.3)
HA_STREAM_OP_MARKER Consumes a sequence number (seq) and does nothing else. Used as a synchronization reference point
HA_STREAM_OP_SET_SAMPLING Sets token sampling parameters in stream order, before subsequent kernel launches
HA_STREAM_OP_RECORD_EVENT Records an event at the current point of the stream (see Section 4.2)
HA_STREAM_OP_WAIT_EVENT Holds subsequent work on this stream until the event completes (see Section 4.2)
1
2
3
4
5
HaDrvStreamOp op{HA_STREAM_OP_MEMCPY, {}};
op.memcpy = {dst, src, n, HaMemCpyDirection::DeviceToDevice};

uint64_t seq = 0;
ha_drv_stream_post(dev, stream, &op, &seq);   // seq: sequence number assigned to this operation within the stream

4.2 Event

An event is an object that marks a specific execution point in a stream. When one stream records an event, another stream or the host can wait until the work up to that point completes. Driver events (HaDrvEvent) are separate objects from runtime events (HaEvent_t).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
HaDrvEvent done = nullptr;
ha_drv_obj_create(dev, HA_OBJ_EVENT, reinterpret_cast<void **>(&done));

// 1. Launch a kernel on the producer stream and record an event.
ha_drv_kernel_launch(dev, producer, k1, args1);
HaDrvStreamOp record{HA_STREAM_OP_RECORD_EVENT, {}};
record.event.event = done;
ha_drv_stream_post(dev, producer, &record, nullptr);

// 2. Make the consumer stream wait for that point on the producer to complete.
HaDrvStreamOp wait{HA_STREAM_OP_WAIT_EVENT, {}};
wait.event.event = done;
ha_drv_stream_post(dev, consumer, &wait, nullptr);
ha_drv_kernel_launch(dev, consumer, k2, args2);

// 3. Or wait on the host.
ha_drv_wait(dev, HA_STREAM_DEFAULT, done, HA_WAIT_EVENT_SYNC, -1, nullptr);
ha_drv_obj_destroy(dev, HA_OBJ_EVENT, done);
API Sync / Async Behavior
ha_drv_obj_create() + HA_OBJ_EVENT Sync Creates an event object
ha_drv_stream_post() + HA_STREAM_OP_RECORD_EVENT Async Adds an event record operation to the stream
ha_drv_stream_post() + HA_STREAM_OP_WAIT_EVENT Async Makes the stream's later work wait for this event. The host does not wait
ha_drv_wait() + HA_WAIT_EVENT_SYNC Sync Waits on the host until the work up to the recorded point completes
ha_drv_wait() + HA_WAIT_EVENT_QUERY Sync HaSuccess if complete, HaEventNotReady if in progress
ha_drv_obj_destroy() + HA_OBJ_EVENT Sync Destroys the event. If it was recorded, waits until the record operation reaches the device, then releases it

Event Record and Wait Rules

  • Single record: An event can be recorded only once in its lifetime. Recording the same event again is rejected with HaEventInvalidArgument.
  • No self-wait: Enqueuing a wait on a stream for its own recorded event would never finish, so it is rejected with HaStreamInvalidArgument.
  • No wait before record: Waiting on an event that has never been recorded, with a stream operation or with ha_drv_wait, returns HaEventNotRecorded.

4.3 Synchronization

Host-side completion waits and status queries go through ha_drv_wait. The kind argument determines the wait target and whether the call blocks. Stream modes take only stream, and event modes take only event; leave the other argument as HA_STREAM_DEFAULT or nullptr. The device mode leaves both empty. Any other combination returns HaInvalidArgument. timeout_ms applies only to the SYNC modes; a negative value waits without a host-side time limit. With HA_WAIT_STREAM_SYNC, the last argument of ha_drv_wait(), out_synced_seq, optionally returns the sequence number this synchronization took.

Synchronization mode (kind) Target Behavior
HA_WAIT_STREAM_SYNC Stream Blocks until all work enqueued before the call completes
HA_WAIT_STREAM_QUERY Stream Queries status without blocking. Returns the stream error if there is one; otherwise HaSuccess if complete, HaStreamNotReady if in progress
HA_WAIT_EVENT_SYNC Event Blocks until the work at the recorded point completes
HA_WAIT_EVENT_QUERY Event Queries status without blocking. HaSuccess if complete, HaEventNotReady if pending
HA_WAIT_DEVICE_SYNC Device Synchronizes every stream on the device at the time of the call: explicit streams, the default streams of all threads, and the default streams of exited threads. Returns the first error

Note: Work enqueued after the call

In every synchronization mode, work newly enqueued after the ha_drv_wait call is not included in the wait.

Completion Checks with Sequence Numbers (seq)

Compare the sequence number returned by ha_drv_stream_post in out_seq with the sequence number returned by ha_drv_wait(HA_WAIT_STREAM_SYNC) in out_synced_seq to determine whether a given stream operation has completed. If the synchronization returned HaSuccess, every operation with a seq smaller than out_synced_seq has completed. out_synced_seq is written before the wait, so it is filled in even when the synchronization fails. An operation that fails to enqueue does not consume a sequence number.

Stream sequence numbers and the range ha_drv_wait waits for

Sequence numbers of stream operations and the out_synced_seq that ha_drv_wait returns

1
2
3
4
5
6
7
HaDrvStreamOp marker{HA_STREAM_OP_MARKER, {}};
uint64_t marker_seq = 0;
ha_drv_stream_post(dev, stream, &marker, &marker_seq);

uint64_t synced_seq = 0;
ha_drv_wait(dev, stream, nullptr, HA_WAIT_STREAM_SYNC, -1, &synced_seq);  // last argument = out_synced_seq
// marker_seq < synced_seq: all work enqueued before the marker has completed

Next Steps