# SOTA OCR with Core ML and dots.ocr

[Hugging Face](https://yomu.fyi/company/hugging-face) · Christopher Fleetwood, Pedro Cuenca · Oct 2, 2025

**Type:** Tutorial

## Summary

Running competitive models like RedNote's 3B parameter dots.ocr on-device provides zero-cost, network-free optical character recognition while leveraging power-efficient hardware like Apple's Neural Engine. Because the Neural Engine requires Core ML, converting the 1.2B parameter NaViT vision encoder from PyTorch requires resolving multiple graph tracing incompatibilities. The initial conversion pipeline targets FLOAT32 execution on GPU by capturing the execution graph with torch.jit.trace and compiling it via coremltools. Simplifying the model for single-image inference eliminates dynamic sequence masking, dynamic tensor iteration, and multi-attention complexity. Although the resulting Core ML model accurately matches PyTorch baseline numerical precision, the unoptimized artifact spans over 5GB and takes over one second per forward pass.

## Context

Executing OCR models on-device avoids API costs and network requirements while taking advantage of power-efficient hardware like Apple's Neural Engine. However, the Neural Engine is only accessible via Core ML, which introduces graph conversion hurdles when translating PyTorch implementations with unsupported operators, dynamic shapes, and strict data type constraints.

## Approach / What changed

The authors trace the 1.2B NaViT vision encoder using torch.jit.trace and coremltools, targeting FLOAT32 on GPU with static shapes. They simplify the graph for single-image inference by standardizing on scaled\_dot\_product\_attention, casting torch.arange outputs to match target dtypes, replacing dynamic sequence masking with constant float masks, and eliminating dynamic loop iteration over tensor shapes.

## Takeaways

- coremltools ignores the dtype argument on torch.arange and outputs int32 by default, requiring an explicit type cast to prevent matmul dtype mismatch errors.
- Apple's Neural Engine does not support boolean tensors, necessitating the replacement of boolean attention masks with float masks containing zero values.
- An initial unoptimized Core ML FLOAT32 conversion of the 1.2B parameter vision encoder produced accurate outputs but exceeded 5GB in size and required over one second per forward pass.

**Tags:** [iOS](https://yomu.fyi/topic/ios), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/dots-ocr-ne)