Training mRNA Language Models Across 25 Species for $165

Summary

OpenMed describes an end-to-end protein engineering pipeline that connects ESMFold structure prediction, ProteinMPNN sequence design, and custom mRNA codon optimization. The codon-language-model study compares CodonBERT, ModernBERT, and several CodonRoBERTa variants using 250,000 E. coli coding sequences, a 69-token codon vocabulary, masked-language modeling, and training on four 80GB A100 GPUs. CodonRoBERTa-large-v2 achieved perplexity 4.10, CAI Spearman correlation 0.404, and 7.7% synonymous recovery, while ModernBERT-base reached perplexity 26.24 and correlation 0.070. The work then trained a multi-species suite on 381,283 sequences spanning 25 species, producing four production models in 55 GPU-hours and documenting planned extensions such as larger training data and additional prediction heads.

Takeaways

1. 1

   The tokenizer represents each codon as one token, using 64 codons plus PAD, UNK, CLS, SEP, and MASK rather than learned BPE subwords.

2. 2

   All architecture comparisons used the same data and evaluation protocol, with 15% masking and 15,000–25,000 training steps on four A100 80GB GPUs with FSDP sharding.

3. 3

   Planned work includes training on mRNABERT’s public 36-million-sequence dataset, adding ProtT5-XL cross-modality alignment, and extending species-specific fine-tuning to YEAST, PICHIA, and MOUSE.