Skip to content

Python Sampling API

Function Signature
set_sampling_params (core_id: int, entry_index: int, entry_num: int, top_p: float, top_k: int, temperature: float, repetition_penalty: float, pseudo_random: float) -> None
set_sampling_params_async (core_id: int, entry_index: int, entry_num: int, top_p: float, top_k: int, temperature: float, repetition_penalty: float, pseudo_random: float, stream: int = HA_STREAM_DEFAULT) -> None

Example

set_sampling_params

The parameters are cached on the device, so writing unchanged values again every step is wasted work.

Example
1
2
3
4
5
6
import hart

hart.set_sampling_params(
    core_id, entry_index, entry_num,
    top_p=0.9, top_k=50, temperature=0.8, repetition_penalty=1.1, pseudo_random=0.0,
)

set_sampling_params_async

Enqueues the setting on the stream and returns immediately. It is applied before any launch enqueued after it on the same stream. Check completion with stream_synchronize().

Example
1
2
3
4
5
6
hart.set_sampling_params_async(
    core_id, entry_index, entry_num,
    top_p=0.9, top_k=50, temperature=0.8, repetition_penalty=1.1, pseudo_random=0.0,
    stream=stream,
)
hart.launch_kernel_async_with_binary_path(kernel_path, args, stream)

See also