Salman Mohammed 75454b96da feat: use tiktoken-rs instead of tokenizers, single global tokenizer (#3115) ...

Co-authored-by: jack <>

2025-07-01 14:42:51 -04:00

..

benches

feat: use tiktoken-rs instead of tokenizers, single global tokenizer (#3115)

2025-07-01 14:42:51 -04:00

examples

feat: use tiktoken-rs instead of tokenizers, single global tokenizer (#3115)

2025-07-01 14:42:51 -04:00

src

feat: use tiktoken-rs instead of tokenizers, single global tokenizer (#3115)

2025-07-01 14:42:51 -04:00

tests

pipe the argument to storage (#3184)

2025-06-30 16:55:31 -07:00

.gitignore

feat: V1.0 (#734)

2025-01-24 13:04:43 -08:00

Cargo.toml

feat: use tiktoken-rs instead of tokenizers, single global tokenizer (#3115)

2025-07-01 14:42:51 -04:00