Character Prefix Conditioning

Summary

The post poses a problem in code completion: generating a language-model completion that begins with the characters already typed by the user. Because modern models process token sequences rather than characters, naively tokenizing the input can produce the wrong result when the cursor falls inside a token. It introduces character prefix conditioning, defined as sampling tokens from an autoregressive distribution while constraining their decoded character string to start with the given prefix. The challenge is to construct an efficient autoregressive algorithm for this conditional distribution while minimizing calls to the original language model; the post requests either an algorithm description or an implementation.

Takeaways

1. 1

   Character prefix conditioning samples a token sequence conditional on a prefix of characters rather than a prefix of tokens.

2. 2

   The sampled token sequence must decode to a character string beginning with the user's supplied prefix.

3. 3

   The requested algorithm should sample autoregressively while minimizing calls to the original language model.