---
title: "Catalog as a Pipeline: Five Checkpoints to Trustworthy Entries"
description: "A catalog is a pipeline, not a list. This five-step guide shows how to fix entry identity, score candidates in tiers, bind policy per entry, and refresh without rescoring everything."
---

# Catalog as a Pipeline: Five Checkpoints to Trustworthy Entries

**Type:** [How to](https://yomu.fyi/visual-explainers/category/how-to.md)

![Five numbered panels left to right: fix the identity key, generate candidate pairs, score in tiers, bind policy per entry, and refresh with cached scores. Thin arrows connect the panels, a dashed return arrow loops back, and the bottom-right corner is left empty.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/67df5e3d-3b27-4d86-88f2-2e45d610e092/image-114cc577-b367-4d65-9592-cfca28224cb0.webp)

> A catalog is a pipeline, not a list. This five-step guide shows how to fix entry identity, score candidates in tiers, bind policy per entry, and refresh without rescoring everything.

## Explanation

A catalog is only as useful as the entries it can keep distinct and current, so treat it as a pipeline rather than a static list. Start by fixing the identity key that decides when two records describe the same thing, then generate candidate pairs cheaply instead of comparing every pair. Score those candidates in escalating tiers so expensive models only touch ambiguous cases, bind authorization to each entry, and refresh continuously while cached scores absorb the churn. The payoff is a catalog that stays trustworthy as it grows, whether it indexes products, tools, integrations, or skills.

## FAQ

### Why not just compare every pair of records when building a catalog?

Full pairwise comparison grows quadratically, so Shopify generates candidate pairs with approximate nearest-neighbor search over billions of embeddings and scores only those. That reserves expensive inference for plausible matches instead of the whole corpus.

### When should catalog scoring escalate to a heavier model?

Use a tiered ladder: a fast classifier handles most pairs, a heavier cross-encoder reads ambiguous records together, and image models run last. Each tier only sees survivors from the one before, and scores are cached so daily updates do not re-run everything.

### Should authorization live at the gateway or on each catalog entry?

Bind policy to entries. An MCP gateway validates OIDC or JWT credentials, then authorizes per-tool access from identity provider claims, with network policy and rate limits around it. That lets you revoke or narrow one capability without taking the whole catalog offline.

**Tags:** [Catalog](https://yomu.fyi/topic/catalog.md), [Data Governance](https://yomu.fyi/topic/data-governance.md), [Entity Resolution](https://yomu.fyi/topic/entity-resolution.md), [MCP](https://yomu.fyi/topic/mcp.md), [Self-Service](https://yomu.fyi/topic/self-service.md), [Vector Search](https://yomu.fyi/topic/vector-search.md)

## Related reading

### [Real-time energy trading and analytics on Databricks](https://yomu.fyi/post/real-time-energy-trading-and-analytics-on-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Victoria Sicking, Jack Yallop
- Published: Oct 23, 2026
- Type: Problem & solution

Energy trading desks face increasing price volatility and the EU’s move from hourly to 15-minute market intervals, multiplying market periods by four. Separating operational and analytical systems creates two versions of truth, stale risk controls, duplicated governance, and extra synchronization. Databricks addresses this with a unified platform: Spark Structured Streaming Real-Time Mode streams market data with millisecond latency into Lakebase, a serverless Postgres serving as the operational database. The same governed data supports real-time trading apps in Databricks Apps, machine learning models for price forecasts, and historical analysis via Genie One, all under Unity Catalog governance. This eliminates separate data copies and synchronization pipelines, enabling risk monitoring and analytics to work from a single consistent data foundation.


### [Locking down MCP at the gateway](https://yomu.fyi/post/locking-down-mcp-at-the-gateway.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: David Martin, Jason Madigan
- Published: Oct 23, 2026
- Type: Tutorial

Model Context Protocol (MCP) allows an agent to use tools across enterprise systems, but a shared MCP gateway requires a policy boundary beyond authentication. The article describes using Red Hat Connectivity Link (RHCL) AuthPolicy to validate OIDC or JWT credentials and then authorize per-tool access based on roles in the identity provider claims. NetworkPolicy controls which workloads can communicate with the gateway and MCP servers, while RateLimitPolicy limits request volume. Gateway isolation can be achieved with separate gateway instances or a shared gateway with isolated listeners, each with its own MCPGatewayExtension. These controls map to OWASP risks for LLM and agentic applications, including excessive agency, unbounded consumption, tool misuse, and identity abuse.


### [Auth0-Powered SaaS Providers Unlock Growth with the Okta Integration Network](https://yomu.fyi/post/auth0-powered-saas-providers-unlock-growth-with-the-okta-integration-n.md)
- Company: [Auth0](https://yomu.fyi/company/auth0.md)
- Published: Oct 23, 2026
- Type: Announcement

Enterprise SaaS providers must differentiate core offerings while meeting enterprise identity security and interoperability requirements for winning RFPs. The Okta Integration Network, a catalog of over 8,000 integrations used by 20,000 Okta customers, allows Auth0-powered SaaS providers to signal enterprise-readiness. By publishing Express Configuration and Universal Logout integrations, companies like Jamf and Augment Code automate onboarding and enhance security without pausing their product roadmap. Express Configuration reduces SSO setup from days of manual OIDC application creation to a few clicks, as Jamf demonstrated for its 78,000 customers. Universal Logout enables instant token revocation across the platform, allowing providers to respond to session hijack immediately, eliminating support ticket workflows.


### [Donating Agent Skills to the Agentic AI Foundation](https://yomu.fyi/post/donating-agent-skills-to-the-agentic-ai-foundation.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Author: Anthropic
- Published: Oct 22, 2026
- Type: Announcement

Anthropic is donating Agent Skills, an open standard for AI agent specialized knowledge and workflows, to the Agentic AI Foundation under the Linux Foundation. A skill consists of a folder containing a SKILL.md file that teaches an agent a specific task or domain, along with any scripts or reference material it needs. Over 70 agent platforms, including ChatGPT, Cursor, GitHub Copilot, and VS Code, have integrated Agent Skills since the standard was published in December 2025, leading to one million skills published and nearly 280 million installations. The donation transfers ownership of the Agent Skills name, logo, GitHub organization, and agentskills.io domain to the AAIF, ensuring vendor-neutral governance. Engineers from Anthropic, Google, OpenAI, and other companies will continue maintaining the standard independently, with David Soria Parra and Mahesh Murag serving as Lead Maintainers.


### [Agentic partners integration: AI support agents for Microsoft Azure Red Hat OpenShift](https://yomu.fyi/post/agentic-partners-integration-ai-support-agents-for-microsoft-azure-red.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Sharon Dashet, Carlos Camacho Gonzalez
- Published: Oct 22, 2026
- Type: Explainer

The operations team troubleshooting production issues on Azure Red Hat OpenShift must already know which tools to reach for, switching between portals, CLI, and dashboards, with tickets arriving incomplete and triage taking hours. The agentic-partners-integration quickstart solves this by connecting a central orchestration layer with domain-specific agents, including a Microsoft Azure MCP Server that exposes Azure resource management operations as MCP tools and an agent that discovers them at runtime. In an OOMKilled investigation, the agent correlates resource health, CPU metrics, and recommendations using a discover-then-execute pattern to produce a data-backed diagnosis. Security is layered: the service principal defaults to Azure Reader role, Open Policy Agent policies gate every request, and each tool call is logged for audit. This pattern extends beyond Azure; any infrastructure backend exposing an MCP server can become an agent capability without framework changes.


### [Cross-store product clustering across millions of stores (2026) - Shopify](https://yomu.fyi/post/cross-store-product-clustering-across-millions-of-stores-2026-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Ilia Kuchumov
- Published: Oct 20, 2026
- Type: Problem & solution

Shopify’s cross-store product clustering system groups listings of the same product across different merchants into a single global Universal Product Identifier. The approach avoids comparing every pair by generating candidate pairs using approximate nearest-neighbor search across billions of vector embeddings. Scoring uses three models of increasing cost: L1 is a fast GBDT classifier, L2 is a fine-tuned multilingual cross-encoder that reads both listings together, and L3 adds images. The UPI builder evaluates proposed merges using a trained model that weighs evidence, handling non-transitive pairwise judgments. The system caches scores and uses priority queues for expensive inference, enabling daily cluster updates while background GPU inference continues refining matches.
