Pantheon/docs/guides/pth-convert.md

PyTorch .pth Converter

Convert PyTorch checkpoint files (.pth) to safetensors format in pure C. No Python, no pip, no dependencies.

Why?

PyTorch's .pth format uses Python's pickle module, which is fundamentally insecure — pickle.load() can execute arbitrary code. Every existing conversion tool requires Python + PyTorch + transformers (2-10 GB of dependencies).

Pantheon reads .pth files via a sandboxed C pickle interpreter that extracts tensor data without executing any embedded Python. It then writes safetensors, a secure mmap-friendly format.

Build

cd Pantheon
make USE_BLAS=1 pantheon-convert

Or from Foundry (standalone, no Pantheon needed):

cd foundry
make pth-convert

Usage

Convert

# Convert .pth to safetensors
./pantheon-convert model.pth -o model.safetensors

Inspect

# Print tensor table (names, shapes, dtypes, sizes)
./pantheon-convert model.pth --inspect

Output:

=== PyTorch Tensor Inspection ===
File: RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth
Tensors: 399

  #  Name                              DType   Shape                      Size
---  --------------------------------  ------  -------------------------  --------
  0  emb.weight                        BF16    [65536, 768]               96.0 MB
  1  blocks.0.att.x_r                  BF16    [768]                       1.5 KB
  2  blocks.0.att.x_w                  BF16    [768]                       1.5 KB
  ...

Example: RWKV-7 0.1B

# 1. Download the .pth model from HuggingFace
huggingface-cli download BlinkDL/rwkv-7-world \
  RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
  --local-dir models/base/

# 2. Convert to safetensors
./pantheon-convert \
  models/base/RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
  -o models/base/RWKV-7-0.1B.safetensors

# 3. Run inference
./pantheon --model models/base/RWKV-7-0.1B.safetensors \
  --vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
  --prompt "Hello" --max-tokens 32 --temperature 0

How It Works

A .pth file is a ZIP archive containing:

archive/
  data.pkl          # Pickle: OrderedDict mapping tensor names -> storage refs
  data/0            # Raw bytes for storage #0
  data/1            # Raw bytes for storage #1
  ...

The converter:

1. ZIP reader — mmaps the file, finds End-of-Central-Directory, parses entries
2. Pickle interpreter — stack-based opcode VM that extracts tensor metadata (names, shapes, dtypes, storage references). Security-whitelisted: only allows torch._utils._rebuild_tensor_v2, collections.OrderedDict, and torch storage classes. Unknown GLOBAL/REDUCE targets abort immediately.
3. Storage resolver — maps storage IDs to ZIP data entries, computes data pointers (zero-copy into mmap)
4. Safetensors writer — writes header JSON + aligned tensor data via Foundry's fs_writer_* API

Supported Dtypes

PyTorch Storage Class	Safetensors Dtype	Bits
BFloat16Storage	BF16	16
HalfStorage	F16	16
FloatStorage	F32	32

Security

The pickle interpreter is deliberately minimal:

• No code execution — only extracts tensor metadata and raw data
• Whitelisted classes — GLOBAL only allows known torch/collections classes
• Whitelisted callables — REDUCE only allows _rebuild_tensor_v2 and OrderedDict
• Immediate abort — any unrecognized opcode or unauthorized class triggers abort with warning

This is not a general-purpose pickle reader. It handles the specific pickle structure that torch.save() produces for state dicts. Adversarial pickles with embedded code will be rejected.

C API

The pth reader is a reusable Foundry module (pt_ prefix):

#include "pth/pth.h"

pt_file *f = pt_open("model.pth");
int n = pt_tensor_count(f);

for (int i = 0; i < n; i++) {
    const pt_tensor *t = pt_tensor_at(f, i);
    printf("%s: %s [", t->name, fs_dtype_name(t->dtype));
    for (int d = 0; d < t->ndim; d++)
        printf("%s%lld", d ? ", " : "", (long long)t->shape[d]);
    printf("] %zu bytes\n", t->data_size);
    // t->data points directly into mmap'd storage (zero-copy)
}

pt_close(f);