Frontier AI
Turing
Jul 2024 — Aug 2025
CUDA-EVAL — GPU-programming evaluations for frontier models
A year on CUDA-EVAL, designing and reviewing GPU-programming evaluation tasks used to train and benchmark advanced language models. The hard part is not writing CUDA — it is writing a problem that has exactly one defensible answer, and a rubric that can tell a correct solution from a merely convincing one.
- Authored evaluation tasks with explicit inputs, assumptions, constraints and objective acceptance criteria, so grading is reproducible rather than a matter of opinion.
- Produced reference solutions and validation checks, confirming correctness analytically and by execution rather than by inspection.
- Peer-reviewed other engineers' submissions for technical correctness and completeness, feeding consistent quality standards back into the task set.
- Applied the same discipline used in safety-relevant engineering review, where a plausible-looking wrong answer is the expensive failure mode.
Directly relevant to simulation-led AI training work: problem definition, reference modelling, analytical validation and peer quality assessment — the same loop, whether the domain is CUDA kernels or electrical machines.
CUDAEvaluation designReference solutionsPeer reviewTechnical writing