Loading…
GitHub Does My Operations Homework: A Ruby Speed Story
2023-10-18
- Source
- Shopify
- Published
- Added to Yomu
Summary
YJIT’s benchmarking site combines open-source benchmark runners with Jekyll, GitHub Pages, and GitHub Actions to publish frequently refreshed results. The author wanted to compare Ruby configurations, expand reporting beyond console output, and avoid recurring operational work. Benchmarks run on an AWS instance rather than in GitHub Actions, while Actions schedules jobs, uses repository secrets for SSH, and commits generated data back to the repository; Jekyll builds the Pages site from that data. Reports include D3.js and Ruby-generated SVG graphs, tables, textual summaries, and JSON tripwire data that can trigger an issue when performance drops too far. The reported results show YJIT 20.6% faster than interpreted CRuby and 17.4% faster than MJIT, with Railsbench gains of 18.5% and 21.0% respectively.
Context
The existing benchmark repository had a runner that printed results to the console, but the author wanted comparisons between Ruby configurations, more reporting, and a way to avoid constant operational work for a frequently updated website. Benchmarking also requires a machine whose performance can be controlled rather than a GitHub Actions runner.
Approach / What changed
The setup uses GitHub Pages and a customized Jekyll configuration for the site, GitHub Actions as a scheduled automation layer, and an AWS instance to run benchmarks. Generated data, graphs, reports, and JSON tripwire files are committed to the repository, after which GitHub builds the site. The workflow uses repository secrets and SSH to reach AWS, and can file an issue when benchmark performance drops beyond a threshold.
Takeaways
- GitHub Pages can host a Jekyll site for free, while GitHub Actions schedules benchmark runs, rebuilds the site, and sends email only when something fails.
- Benchmark execution is delegated to an AWS instance because the author wanted more control over machine performance than GitHub Actions provides; SSH from Actions requires handling repository secrets, host keys, and environment variables.
- YJIT is reported as 20.6% faster than interpreted CRuby and 17.4% faster than MJIT overall; on Railsbench, the reported differences are 18.5% and 21.0% respectively.