108 lines
2.7 KiB
Markdown
108 lines
2.7 KiB
Markdown
# eval/
|
|
|
|
C implementation of LLM evaluation and benchmarking.
|
|
|
|
## Python counterpart
|
|
|
|
**lm-eval-harness** - https://github.com/EleutherAI/lm-evaluation-harness
|
|
|
|
## Why lm-eval-harness?
|
|
|
|
lm-eval-harness is the standard for evaluating language models:
|
|
|
|
- **Standardized benchmarks** - MMLU, HellaSwag, ARC, TruthfulQA, etc.
|
|
- **Reproducible results** - Consistent evaluation methodology
|
|
- **Multiple metrics** - Accuracy, perplexity, F1, exact match
|
|
- **Few-shot evaluation** - Test with N examples in context
|
|
|
|
Alternatives (HELM, OpenAI evals) cover different benchmark suites.
|
|
|
|
## Scope
|
|
|
|
This module implements:
|
|
|
|
1. **Core metrics** - Accuracy, perplexity, exact match
|
|
2. **Multiple choice** - For benchmarks like MMLU, HellaSwag
|
|
3. **Generation eval** - For open-ended tasks
|
|
4. **Few-shot prompting** - Construct prompts with examples
|
|
5. **Common benchmarks** - MMLU, HellaSwag, ARC, Winogrande
|
|
|
|
## Getting Started
|
|
|
|
### Dependencies
|
|
|
|
- tensor/ module (for model inference)
|
|
- tokenizer/ module (for text processing)
|
|
- data/ module (for loading benchmark data)
|
|
|
|
### Key Files to Study
|
|
|
|
From lm-eval-harness:
|
|
- `lm_eval/tasks/` - Task definitions
|
|
- `lm_eval/api/model.py` - Model interface
|
|
- `lm_eval/evaluator.py` - Evaluation loop
|
|
|
|
### Core API
|
|
|
|
```c
|
|
// Initialize evaluator with model
|
|
fe_evaluator* fe_new(tm_model* model, ft_tokenizer* tokenizer);
|
|
|
|
// Run specific benchmark
|
|
fe_result* fe_run_task(fe_evaluator* e, const char* task, int num_fewshot);
|
|
|
|
// Available tasks
|
|
const char** fe_list_tasks(void);
|
|
|
|
// Results
|
|
typedef struct {
|
|
const char* task;
|
|
float accuracy;
|
|
float stderr;
|
|
int num_samples;
|
|
} fe_result;
|
|
|
|
void fe_print_result(fe_result* r);
|
|
void fe_save_results(fe_result** results, int n, const char* path);
|
|
|
|
// Metrics
|
|
float fe_accuracy(int* predictions, int* labels, int n);
|
|
float fe_perplexity(tm_model* model, const char** texts, int n);
|
|
float fe_exact_match(const char** predictions, const char** references, int n);
|
|
|
|
// Multiple choice evaluation
|
|
int fe_loglikelihood_choice(tm_model* model, const char* context,
|
|
const char** choices, int num_choices);
|
|
```
|
|
|
|
## Benchmark Format
|
|
|
|
Tasks are defined as JSON:
|
|
```json
|
|
{
|
|
"task": "hellaswag",
|
|
"type": "multiple_choice",
|
|
"dataset": "hellaswag",
|
|
"prompt_template": "{context}",
|
|
"choices_key": "endings",
|
|
"label_key": "label",
|
|
"num_fewshot": 10
|
|
}
|
|
```
|
|
|
|
## Implementation Order
|
|
|
|
1. Core metrics (accuracy, perplexity)
|
|
2. Multiple choice evaluation
|
|
3. Few-shot prompt construction
|
|
4. HellaSwag benchmark
|
|
5. MMLU benchmark
|
|
6. ARC benchmark
|
|
7. Results reporting
|
|
|
|
## References
|
|
|
|
- [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness)
|
|
- [HELM](https://crfm.stanford.edu/helm/)
|
|
- [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|