Caching Without Marshal Part 2: The Path to MessagePack

Summary

Rails caching relied heavily on Marshal, but cached Ruby-specific objects could break after code changes when old payloads were loaded by code without the former classes. MessagePack provided a generic binary format whose core types exclude Ruby-specific Object and instance-variable encodings, while extension types allowed production-specific serializers such as the compact Date representation. During a roughly six-month migration, the team ran both formats, prefixed MessagePack payloads with a version byte, and used encoding failures to identify unsupported cache values. To scale beyond 128 extension types, they added an Object catchall based on as_pack and from_pack, plus Struct and T::Struct modules whose attribute digests detect refactors and turn stale data into cache misses. After logs showed that Marshal was no longer used, the core monolith moved exclusively to MessagePack, with the Paquito gem extracting much of the migration work.

Context

Rails used Marshal extensively for cache serialization, but cached payloads containing Ruby classes could break after deployments when code changes removed or renamed those classes. The migration also had to account for cache values outside MessagePack’s built-in types and scale across Shopify’s large developer population without exhausting the format’s 128 extension-type limit.

Approach / What changed

The team introduced a MessagePack factory with custom extension types, ran Marshal and MessagePack side by side for about six months, and added a version-byte prefix to payloads. Unsupported objects caused encoding failures that could be logged and investigated. To scale the design, an Object catchall used as_pack and from_pack, while reusable Struct and T::Struct modules encoded attribute data and digests so changed structures became cache misses. Marshal was removed after logs showed it was no longer used.

Takeaways

• MessagePack omits Ruby-specific Object and instance-variable encodings by default, preventing unsupported Ruby objects from being silently serialized.
• MessagePack extension types can serialize application classes such as Date, but the format allows at most 128 explicitly registered extension types.
• Struct attribute digests let deserialization detect refactors and treat changed cached data as stale, while class-name changes are handled as cache misses.

Related reading