Updating Illustrations at Scale

Summary

Shopify’s Polaris team needed to update an illustration style across a product whose assets had become fragmented across more than 6,000 repositories. Illustrators and developers could not reliably identify an image’s repository, code references, reuse, or file type, while GitHub API limits and search behavior prevented dependable large-scale discovery. An initial Node.js monorepo cloned repositories, used globby to find PNG, SVG, and WebP files, searched source lines, and served JSON results through a Preact interface, but rebuilding 24 repositories took more than 35 minutes and produced inaccurate data. The team replaced it with @shopify/get-repo-images, a Go executable that parallelizes file operations, bundles crawling and site generation, and exposes a searchable Next.js interface through NPM. Search and build time fell to eight minutes, with repository cloning as the remaining bottleneck, and the tool was open-sourced.

Context

The Polaris team was updating Shopify’s illustration style, but illustration usage had become fragmented across more than 6,000 repositories. Images appeared in multiple repositories and code paths, used inconsistent file types, and could also come from API requests. Existing GitHub API searches encountered limits, false positives, wildcard handling issues, and incomplete results, making it difficult to identify references and prioritize updates.

Approach / What changed

The team first cloned repositories with Node.js, located matching image extensions with globby, searched source files for references, and presented JSON results in a Preact application. After that approach became too slow and inaccurate, they consolidated repository cloning, searching, data generation, and application startup into a Go binary. The tool performs file operations in parallel, uses a Next.js interface with a simple API, runs through NPM, and supports browsing, filtering, and sorting image results.

Takeaways

• The initial Node.js implementation needed daily rebuilds and took more than 35 minutes to search 24 repositories, while also producing inaccurate data and complicating repository additions.
• The Go rewrite reduced search and website build time from more than 35 minutes to 8 minutes by running operations in parallel across multiple cores; cloning 24 monolithic repositories remained the bottleneck.
• @shopify/get-repo-images finds images and their code references across multiple GitHub repositories, then lets users browse, filter, and sort the results through a generated website.

Related reading