We Got Claude to Build CUDA Kernels and teach open models!

Summary

Domain-specific coding tasks like writing optimized CUDA kernels often require capable, expensive frontier models, leading to high operational costs. The upskill tool addresses this challenge by capturing specialized workflows from teacher models such as Claude Opus 4.5 into portable skill files that can be consumed by smaller or locally hosted models. The framework automatically generates input-output test cases from execution traces to benchmark student model accuracy and token efficiency with and without the skill. In an evaluation targeting HuggingFace kernel-builder development, a local GLM-4.7 Flash model improved its pass rate from 40% to 85% when supplied with the generated skill. Measuring token usage across different models remains essential, as skills can lower token consumption for some architectures while introducing unnecessary overhead for others.

Takeaways

1. 1

   The upskill tool evaluates skill effectiveness using a teacher-student framework where teacher models generate assertion test cases to measure task lift on student models.

2. 2

   Adding an agent skill does not uniformly improve efficiency; some models reduce token consumption while others experience increased token usage without accuracy gains.

3. 3

   Skills follow a standard agent directory structure ({agent}/skills/{skill_name}/SKILL.md) and package specialized domain requirements into roughly 500 tokens that load on demand.