Skip to content

Sampling API

Token sampling parameters programmed onto the device.


HaSetSamplingParams

Sets sampling parameters (top-p/top-k/temperature, etc.).

Parameter Direction Default Description
core_id in — Target core ID.
entry_index in — Target entry index.
entry_num in — Number of consecutive entries to configure.
top_p in — Nucleus sampling (p).
top_k in — Top-k sampling (k).
temperature in — Temperature.
repetition_penalty in — Repetition penalty.
pseudo_random in — Pseudo-random seed forwarded to the SMP module's PRNG.

Returns — Success or an error code.

Signature · Example
1
HaError HaSetSamplingParams(uint32_t core_id, int entry_index, int entry_num, float top_p, int top_k, float temperature, float repetition_penalty, float pseudo_random);

Example

1
2
3
HaSetSamplingParams(core_id, entry_index, entry_num,
                    /*top_p=*/0.9F, /*top_k=*/50, /*temperature=*/0.8F,
                    /*repetition_penalty=*/1.1F, /*pseudo_random=*/0.0F);

Note

Used for configuring runtime sampling strategy for generation. The meaning of core_id and entry_index follows the runtime request-table contract.

Note

The runtime caches these on the device, so re-setting unchanged parameters every step is wasted work.


HaSetSamplingParamsAsync

Asynchronously sets sampling parameters (top-p/top-k/temperature, etc.) on a stream.

Parameter Direction Default Description
core_id in — Target core ID.
entry_index in — Target entry index.
entry_num in — Number of consecutive entries to configure.
top_p in — Nucleus sampling (p).
top_k in — Top-k sampling (k).
temperature in — Temperature.
repetition_penalty in — Repetition penalty.
pseudo_random in — Pseudo-random seed forwarded to the SMP module's PRNG.
stream in HA_STREAM_DEFAULT Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device.

Returns — HaSuccess if enqueued successfully; the write may not be applied on return.

Signature · Example
1
HaError HaSetSamplingParamsAsync(uint32_t core_id, int entry_index, int entry_num, float top_p, int top_k, float temperature, float repetition_penalty, float pseudo_random, HaStream_t stream = HA_STREAM_DEFAULT);

Example

1
2
3
HaSetSamplingParamsAsync(core_id, entry_index, entry_num,
                         /*top_p=*/0.9F, /*top_k=*/50, /*temperature=*/0.8F,
                         /*repetition_penalty=*/1.1F, /*pseudo_random=*/0.0F, stream);

Note

Posted onto the stream's op queue in the same order as launches, so it is applied before any launch enqueued after it on that stream and after any launch enqueued before it -- unlike HaSetSamplingParams, which only orders correctly against the default stream.

Note

Use HaStreamSynchronize() or HaEventSynchronize() to confirm completion and observe a device-side failure (e.g. an invalid parameter) -- returning does not mean it was applied.