2.4 KiB

Raw Blame History

tokenizer

Fast text tokenization with BPE support. Python counterpart: HuggingFace Tokenizers.

Header: tokenizer/tokenizer.h Source: tokenizer/tokenizer.c Status: API Complete - Pending Verification

Overview

The tokenizer module provides:

BPE (Byte-Pair Encoding) tokenization
HuggingFace JSON format loading
Special token handling (BOS, EOS, PAD, UNK)
Batch encoding/decoding

Quick Start

#include "tokenizer/tokenizer.h"

// Load tokenizer from HuggingFace format
ft_tokenizer* tok = ft_load("tokenizer.json");

// Encode text
int len;
int* ids = ft_encode(tok, "Hello, world!", &len);

// Decode back
char* text = ft_decode(tok, ids, len);
printf("%s\n", text);  // "Hello, world!"

// Cleanup
free(text);
free(ids);
ft_free(tok);

API Reference

Loading

ft_tokenizer* ft_load(const char* path);                              // HuggingFace JSON
ft_tokenizer* ft_load_bpe(const char* vocab_path, const char* merges_path);  // BPE files
ft_tokenizer* ft_new(void);                                            // Empty tokenizer
void ft_free(ft_tokenizer* tok);

Vocabulary

int ft_add_token(ft_tokenizer* tok, const char* token);   // Returns token ID
void ft_add_merge(ft_tokenizer* tok, const char* a, const char* b);
int ft_vocab_size(ft_tokenizer* tok);
const char* ft_id_to_token(ft_tokenizer* tok, int id);
int ft_token_to_id(ft_tokenizer* tok, const char* token);

Special Tokens

void ft_set_unk_token(ft_tokenizer* tok, const char* token);
void ft_set_bos_token(ft_tokenizer* tok, const char* token);
void ft_set_eos_token(ft_tokenizer* tok, const char* token);
void ft_set_pad_token(ft_tokenizer* tok, const char* token);

int ft_unk_id(ft_tokenizer* tok);  // -1 if not set
int ft_bos_id(ft_tokenizer* tok);
int ft_eos_id(ft_tokenizer* tok);
int ft_pad_id(ft_tokenizer* tok);

Encoding

int* ft_encode(ft_tokenizer* tok, const char* text, int* length);
int* ft_encode_with_special(ft_tokenizer* tok, const char* text, int* length,
                            bool add_bos, bool add_eos);
int** ft_encode_batch(ft_tokenizer* tok, const char** texts, int n, int** lengths);

Decoding

char* ft_decode(ft_tokenizer* tok, int* ids, int length);
char* ft_decode_skip_special(ft_tokenizer* tok, int* ids, int length);

2.4 KiB Raw Blame History