Introducing the Palmyra-mini family: Powerful, lightweight, and ready to reason!

Summary

WRITER released three open models in the Palmyra-mini family ranging between 1.5B and 1.7B parameters, designed for efficient inference and specialized reasoning tasks. Built on the Qwen architecture, the release includes the standard palmyra-mini base model alongside two Chain of Thought variants, palmyra-mini-thinking-a and palmyra-mini-thinking-b. In benchmark evaluations, palmyra-mini achieved 52.6% on Big Bench Hard, while palmyra-mini-thinking-a reached 82.87% on GSM8K and palmyra-mini-thinking-b reached 92.5% on AMC23. For palmyra-mini-thinking-b, applying reinforcement learning fine-tuning to an OpenReasoning-Nemotron-1.5B base improved single-shot pass@1 accuracy while reducing sampling diversity and majority@64 performance. The models are available in GGUF and MLX-BF16 quantizations and support inference engines including vLLM, SGLang, TRTLLM, and TGI.

Context

WRITER aims to deliver powerful, lightweight, and performant models in the 1.5B to 1.7B parameter range for efficient inference across general generative tasks, complex logic, and mathematical reasoning.

Approach / What changed

The team developed three Qwen-architecture models: the base palmyra-mini, palmyra-mini-thinking-a trained with Chain of Thought, and palmyra-mini-thinking-b built by applying reinforcement learning fine-tuning on OpenReasoning-Nemotron-1.5B. Quantizations are provided in GGUF and MLX-BF16 formats, supporting inference on engines like vLLM, SGLang, TRTLLM, and TGI.

Takeaways

• Palmyra-mini-thinking-a achieved 82.87% on GSM8K (strict match) and held the highest overall average benchmark score among the released models.
• Palmyra-mini-thinking-b scored 92.5% on AMC23 and led average benchmark scores across AIME24, AIME25, GPQA, HMMT25, HLE, MMLU_PRO, MATH500, and LCB.
• Reinforcement learning fine-tuning improved pass@1 single-shot accuracy on palmyra-mini-thinking-b but reduced sampling diversity, resulting in lower majority@64 performance compared to the SFT base model.

Related reading

• Hugging Face · Sep 22, 2025

  Gaia2 and ARE: Empowering the community to study agents

  Existing AI agent evaluation environments are often tightly coupled to specific tasks and fail to model real-world challenges such as API failures, spontaneous events, and asynchronous conditions. To address this limitation, the Gaia2 benchmark and the Meta Agents Research Environments (ARE) framework introduce interactive read-and-write evaluation. Gaia2 incorporates 1,000 human-created scenarios spanning multi-step execution, cross-source search, ambiguity handling, adaptability, temporal reasoning, agent collaboration, and noise tolerance. Using a simulated smartphone interface equipped with 101 tools, evaluations showed that GPT-5 with high reasoning scored highest overall, while Kimi K2 was the leading open-source model. The results demonstrated that instruction following and search do not reliably predict performance on closer-to-real-world tasks.

  Clémentine Fourrier, Grégoire Mialon, Maxime Lecanu, Pierre Andrews, Adrien Carreira, frere thibaud, Avijit Ghosh, Romain Froger, Dheeraj Mekala, Caroline Pascal, Ulyana Piterbarg

• Hugging Face