Sampling API¶
Token sampling parameters programmed onto the device.
HaSetSamplingParams¶
Sets sampling parameters (top-p/top-k/temperature, etc.).
| Parameter | Direction | Default | Description |
|---|---|---|---|
core_id |
in | — | Target core ID. |
entry_index |
in | — | Target entry index. |
entry_num |
in | — | Number of consecutive entries to configure. |
top_p |
in | — | Nucleus sampling (p). |
top_k |
in | — | Top-k sampling (k). |
temperature |
in | — | Temperature. |
repetition_penalty |
in | — | Repetition penalty. |
pseudo_random |
in | — | Pseudo-random seed forwarded to the SMP module's PRNG. |
Returns — Success or an error code.
Signature · Example
1 | |
Example
1 2 3 | |
Note
Used for configuring runtime sampling strategy for generation. The meaning of core_id and entry_index follows the runtime request-table contract.
Note
The runtime caches these on the device, so re-setting unchanged parameters every step is wasted work.
HaSetSamplingParamsAsync¶
Asynchronously sets sampling parameters (top-p/top-k/temperature, etc.) on a stream.
| Parameter | Direction | Default | Description |
|---|---|---|---|
core_id |
in | — | Target core ID. |
entry_index |
in | — | Target entry index. |
entry_num |
in | — | Number of consecutive entries to configure. |
top_p |
in | — | Nucleus sampling (p). |
top_k |
in | — | Top-k sampling (k). |
temperature |
in | — | Temperature. |
repetition_penalty |
in | — | Repetition penalty. |
pseudo_random |
in | — | Pseudo-random seed forwarded to the SMP module's PRNG. |
stream |
in | HA_STREAM_DEFAULT |
Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device. |
Returns — HaSuccess if enqueued successfully; the write may not be applied on return.
Signature · Example
1 | |
Example
1 2 3 | |
Note
Posted onto the stream's op queue in the same order as launches, so it is applied before any launch enqueued after it on that stream and after any launch enqueued before it -- unlike HaSetSamplingParams, which only orders correctly against the default stream.
Note
Use HaStreamSynchronize() or HaEventSynchronize() to confirm completion and observe a device-side failure (e.g. an invalid parameter) -- returning does not mean it was applied.