ML based ranking using Nrtsearch

Summary

Nrtsearch’s Inference Plugin embeds machine-learning ranking in the Lucene-based search engine, addressing scalability problems in a two-stage design where a separate inference service scored candidates. Previously, large candidate sets and feature payloads created network-transfer and serialization overhead; the new flow extracts features from index documents or request parameters and scores documents within replica nodes. An ML Scorer selects models, transforms features, runs MLeap-based inference, and aggregates scores, with support for XGBoost, neural networks, multi-model execution, batch inference, and custom Java scorers. Models are loaded from MLflow through cluster configuration, while teams can use Lucene expressions or implement modular interfaces; testing, canary deployment, rolling updates, and Prometheus metrics support operations, and future work includes GPU-based ranking and additional inference engines.

Takeaways

1. 1

   Co-locating feature storage and inference in Nrtsearch reduces network transfer, serialization overhead, and latency associated with sending large candidate feature payloads to a standalone inference service.

2. 2

   The ML Scorer orchestrates document-level ranking by selecting models, transforming configured features, running MLeap inference, and aggregating scores for multi-model requests.

3. 3

   Teams can choose a generic scorer using Lucene expressions or implement custom Java modules; model updates can roll through nodes without restarting the cluster, while scorer changes use canary deployment and rolling restarts.

FAQ