Code Ranges: A Deeper Look at Ruby Strings

Summary

Ruby's string model supports mutable and immutable operations, per-object or per-file freezing, and more than 100 encodings, while permitting byte sequences invalid for their attached encoding. This flexibility helps legacy and unusual platforms but makes encoding checks, validity scans, and character-boundary handling part of many operations. To reduce repeated O(n) validation scans, Ruby implementations cache each string's state in a four-valued code range: unknown, 7bit, valid, or broken. The post explains how these values map to public methods, how MRI stores them in two header bits, and how JRuby and TruffleRuby represent them differently. It also describes code ranges as inputs to fast and slow paths and as useful but fallible cache information, especially for native extensions and string-heavy metaprogramming.

Takeaways

1. 1

   Ruby code ranges have four mutually exclusive values: ENC_CODERANGE_UNKNOWN, ENC_CODERANGE_7BIT, ENC_CODERANGE_VALID, and ENC_CODERANGE_BROKEN.

2. 2

   MRI stores a string's code range in two object-header bits, so the constants must be compared as enum-like values rather than combined or tested directly as bitmasks.

3. 3

   Code ranges can avoid repeated string scans, but native extensions that fail to update them properly can leave cached information wrong and cause correctness or performance issues.