Foundry/eval/README.md

eval/

C implementation of LLM evaluation and benchmarking.

Python counterpart

lm-eval-harness - https://github.com/EleutherAI/lm-evaluation-harness

Why lm-eval-harness?

lm-eval-harness is the standard for evaluating language models:

• Standardized benchmarks - MMLU, HellaSwag, ARC, TruthfulQA, etc.
• Reproducible results - Consistent evaluation methodology
• Multiple metrics - Accuracy, perplexity, F1, exact match
• Few-shot evaluation - Test with N examples in context

Alternatives (HELM, OpenAI evals) cover different benchmark suites.

Scope

This module implements:

1. Core metrics - Accuracy, perplexity, exact match
2. Multiple choice - For benchmarks like MMLU, HellaSwag
3. Generation eval - For open-ended tasks
4. Few-shot prompting - Construct prompts with examples
5. Common benchmarks - MMLU, HellaSwag, ARC, Winogrande

Getting Started

Dependencies

• tensor/ module (for model inference)
• tokenizer/ module (for text processing)
• data/ module (for loading benchmark data)

Key Files to Study

From lm-eval-harness:

• lm_eval/tasks/ - Task definitions
• lm_eval/api/model.py - Model interface
• lm_eval/evaluator.py - Evaluation loop

Core API

// Initialize evaluator with model
fe_evaluator* fe_new(tm_model* model, ft_tokenizer* tokenizer);

// Run specific benchmark
fe_result* fe_run_task(fe_evaluator* e, const char* task, int num_fewshot);

// Available tasks
const char** fe_list_tasks(void);

// Results
typedef struct {
    const char* task;
    float accuracy;
    float stderr;
    int num_samples;
} fe_result;

void fe_print_result(fe_result* r);
void fe_save_results(fe_result** results, int n, const char* path);

// Metrics
float fe_accuracy(int* predictions, int* labels, int n);
float fe_perplexity(tm_model* model, const char** texts, int n);
float fe_exact_match(const char** predictions, const char** references, int n);

// Multiple choice evaluation
int fe_loglikelihood_choice(tm_model* model, const char* context,
                             const char** choices, int num_choices);

Benchmark Format

Tasks are defined as JSON:

{
  "task": "hellaswag",
  "type": "multiple_choice",
  "dataset": "hellaswag",
  "prompt_template": "{context}",
  "choices_key": "endings",
  "label_key": "label",
  "num_fewshot": 10
}

Implementation Order

1. Core metrics (accuracy, perplexity)
2. Multiple choice evaluation
3. Few-shot prompt construction
4. HellaSwag benchmark
5. MMLU benchmark
6. ARC benchmark
7. Results reporting

References

• lm-eval-harness
• HELM
• Open LLM Leaderboard