Building hermetic notebook images for Open Data Hub and Red Hat OpenShift AI

Summary

Open Data Hub and Red Hat OpenShift AI adopted hermetic container builds to eliminate network failures, satisfy Conforma compliance, and ensure reproducible notebook environments. Instead of pulling packages during image creation, dependencies are pinned in lockfiles, prefetched into a local cache using tools like Cachi2, and installed with the network disabled. The workflow relies on a unified Containerfile capable of running locally, within GitHub Actions, and on Konflux via Tekton tasks. To pass audit checks and avoid generic tarball downloads, the team packaged awkward binaries like ripgrep and Pandoc into multi-arch Python wheels while patching Node postinstall scripts that fetch files at install time. Upstream CentOS and downstream Red Hat Enterprise Linux dependencies are maintained in distinct lock trees to prevent subtle packaging mismatches.

Takeaways

1. 1

   Hermetic builds decouple caching from image compilation by prefetching all dependencies into a local cache using tools like Cachi2 or Hermeto before running builds without network access.

2. 2

   Typed package ecosystems such as rpm, pip, npm, and gomod are preferred over generic tarball downloads to produce verifiable SBOMs and meet Conforma compliance requirements.

3. 3

   Upstream and downstream lock trees should remain separate to prevent incompatibilities between CentOS packages and subscribed Red Hat Enterprise Linux or AIPCC base images.