161 lines
5.1 KiB
Markdown
161 lines
5.1 KiB
Markdown
# Quickstart
|
|
|
|
Build Pantheon and generate text in under a minute.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
git clone --recursive https://git.teamide.dev/David/Pantheon.git
|
|
cd Pantheon
|
|
make USE_BLAS=1 pantheon
|
|
```
|
|
|
|
Foundry comes in as a git submodule at `foundry/`; in a clone made without
|
|
`--recursive`, run `git submodule update --init` first.
|
|
|
|
Build flags:
|
|
|
|
| Flag | What | When |
|
|
|------|------|------|
|
|
| `USE_BLAS=1` | Accelerate (macOS) / OpenBLAS (Linux) | Always |
|
|
| `USE_THREADS=1` | Thread pool for parallel attention | 1B+ models |
|
|
| `DEBUG=1` | `-O0 -g` debug build | Debugging |
|
|
|
|
## Download a Model
|
|
|
|
Any GGUF model from HuggingFace works. RWKV-X safetensors models are also supported. Start small:
|
|
|
|
```bash
|
|
# SmolLM2 135M (fast, good for testing)
|
|
huggingface-cli download bartowski/SmolLM2-135M-Instruct-GGUF \
|
|
SmolLM2-135M-Instruct-Q8_0.gguf --local-dir ../models/base/
|
|
|
|
# RWKV-X 0.2B (hybrid recurrent model, safetensors format)
|
|
huggingface-cli download howard-hou/RWKV-X \
|
|
RWKV-X-0.2B-64k-Base.safetensors --local-dir ../models/base/
|
|
|
|
# Phi-3 mini (3.8B, good quality for size)
|
|
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-gguf \
|
|
Phi-3-mini-4k-instruct-Q4_K_M.gguf --local-dir ../models/base/
|
|
|
|
# Mistral 7B v0.3 (larger, needs USE_THREADS=1)
|
|
huggingface-cli download bartowski/Mistral-7B-Instruct-v0.3-GGUF \
|
|
Mistral-7B-Instruct-v0.3-Q4_K_M.gguf --local-dir ../models/base/
|
|
```
|
|
|
|
See [reference/supported-models.md](../reference/supported-models.md) for the full list of verified models.
|
|
|
|
## Generate Text
|
|
|
|
```bash
|
|
# GGUF model
|
|
./pantheon --model ../models/base/SmolLM2-135M-Instruct-Q8_0.gguf \
|
|
--prompt "The meaning of life is" --max-tokens 64
|
|
|
|
# RWKV-X safetensors model (requires --vocab)
|
|
./pantheon --model ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
|
|
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
|
|
--prompt "Hello world" --max-tokens 32
|
|
|
|
# RWKV-X with INT8 quantization (~3x faster on ARM)
|
|
./pantheon --model ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
|
|
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
|
|
--prompt "Hello world" --max-tokens 32 --quantize
|
|
```
|
|
|
|
## CLI Flags
|
|
|
|
| Flag | Default | Description |
|
|
|------|---------|-------------|
|
|
| `--model <path>` | (required) | Model file (GGUF or safetensors) |
|
|
| `--vocab <path>` | none | RWKV World tokenizer vocab file (required for safetensors) |
|
|
| `--prompt <text>` | `"Hello"` | Input prompt |
|
|
| `--max-tokens <n>` | 128 | Maximum tokens to generate |
|
|
| `--temperature <f>` | 0.8 | Sampling temperature (0 = greedy) |
|
|
| `--top-p <f>` | 0.9 | Nucleus sampling threshold |
|
|
| `--top-k <n>` | 40 | Top-k sampling |
|
|
| `--quantize` | off | INT8 quantize weights at load time (RWKV-X) |
|
|
| `--chat` | off | Interactive multi-turn chat mode |
|
|
| `--system <text>` | none | System prompt (chat mode only) |
|
|
| `--bench` | off | Report tok/s, load time, memory |
|
|
| `--help` | | Show usage |
|
|
| `--version` | | Print version |
|
|
|
|
## Benchmark
|
|
|
|
```bash
|
|
./pantheon --model ../models/base/SmolLM2-135M-Instruct-Q8_0.gguf \
|
|
--prompt "The meaning of life is" --max-tokens 64 --bench --temperature 0
|
|
```
|
|
|
|
```
|
|
=== Benchmark ===
|
|
load: 41.6 ms
|
|
decode: 64 tokens in 787.3 ms (81.3 tok/s)
|
|
memory: 552.7 MB RSS
|
|
=================
|
|
```
|
|
|
|
## C API
|
|
|
|
Pantheon exposes a minimal C API for embedding into applications:
|
|
|
|
```c
|
|
#include "pantheon.h"
|
|
|
|
int main(void) {
|
|
/* GGUF model */
|
|
pn_model *model = pn_load("model.gguf");
|
|
|
|
/* Or: RWKV-X safetensors model */
|
|
pn_load_opts opts = { .vocab_path = "rwkv_vocab.txt", .quantize = true };
|
|
pn_model *rwkvx = pn_load_ex("model.safetensors", &opts);
|
|
|
|
pn_state *state = pn_state_new(model);
|
|
pn_sampling_params params = pn_sampling_defaults();
|
|
params.temperature = 0.0f; /* greedy */
|
|
|
|
char *output = pn_generate(model, state, "Hello world", 64, ¶ms);
|
|
printf("%s\n", output);
|
|
|
|
free(output);
|
|
pn_state_free(state);
|
|
pn_model_free(model);
|
|
return 0;
|
|
}
|
|
```
|
|
|
|
### Streaming
|
|
|
|
```c
|
|
void on_token(const char *token, void *user_data) {
|
|
printf("%s", token);
|
|
fflush(stdout);
|
|
}
|
|
|
|
pn_stream(model, state, "Hello", 128, ¶ms, on_token, NULL);
|
|
```
|
|
|
|
### API Functions
|
|
|
|
| Function | Description |
|
|
|----------|-------------|
|
|
| `pn_load(path)` | Load GGUF model, returns `pn_model*` |
|
|
| `pn_load_ex(path, opts)` | Load any model (GGUF or safetensors) with options |
|
|
| `pn_model_free(model)` | Free model and weights |
|
|
| `pn_state_new(model)` | Create inference state (KV cache + arena, or RWKV-X state) |
|
|
| `pn_state_free(state)` | Free state |
|
|
| `pn_state_reset(state)` | Clear state and position (reuse state) |
|
|
| `pn_generate(model, state, prompt, max_tokens, params)` | Generate text, returns `char*` (caller frees) |
|
|
| `pn_stream(model, state, prompt, max_tokens, params, cb, ctx)` | Streaming generation via callback |
|
|
| `pn_stream_tokens(model, state, tokens, n, max, params, cb, ctx)` | Streaming from pre-tokenized input (for chat) |
|
|
| `pn_sampling_defaults()` | Default sampling params (temp=0.8, top_p=0.9, top_k=40) |
|
|
|
|
## Run Tests
|
|
|
|
```bash
|
|
make USE_BLAS=1 test
|
|
```
|
|
|
|
62 tests across 7 suites: build sanity (5), GGUF dequant (11), arena (4), quantized matvec (4), SIMD kernels (6), chat templates (20), RWKV-X (12).
|