Traditional customer data ingestion via SFTP, APIs, or managed connectors carries hidden costs like pipeline drift, backfill complexity, and duplicate storage. OpenSharing is an open protocol built on table formats such as Delta Lake and Iceberg that lets consumers read a customer’s table in place instead of copying it. By exposing a change data feed, OpenSharing enables incremental pulls of only new, updated, or deleted records without manual timestamp tracking or per-customer connector babysitting. Backfills become simple queries against the shared historical table, eliminating the need to ask customers to re-export data or hit API rate limits. Customers benefit by sharing tables they already maintain, gaining an audit trail of access via Databricks system tables, and avoiding export babysitting, while consumers use their own compute and avoid duplicate storage costs.
Context
The post discusses the hidden costs of traditional customer data ingestion patterns (SFTP, APIs, managed connectors) and introduces OpenSharing, an open protocol for reading tables in place, as a solution. It is Part 1 of a series, focusing on the ‘why’ before the ‘how’ in Part 2.
Takeaways
1
OpenSharing uses open table formats like Delta Lake and Iceberg to let consumers read a customer’s table in place, eliminating the need for file exports or API pulls.
2
By exposing a change data feed, OpenSharing enables incremental data pulls that track only new, updated, or deleted records, avoiding manual timestamp bookkeeping.
3
Customers gain an audit trail of data access through Databricks system tables, replacing blind trust with verifiable logs.
FAQ
How does OpenSharing enable incremental data pulls?
OpenSharing exposes a change data feed on shared tables. When your pipeline runs, it reads the feed to know which records were created, deleted, or updated, so you only pull changes instead of reprocessing entire files.
How does OpenSharing simplify backfills?
If the customer shares the entire historical table, you can query the data you need to backfill directly. You do not need to ask the customer to re-export data or hit API rate limits, because you have direct access to the table.
What audit capabilities does OpenSharing provide to customers?
With Databricks system tables, customers can see when a recipient queries a shared table. This replaces blind trust with a verifiable audit trail of every access.
Clang's constant interpreter evaluates constant expressions at compile time, so the bill lands on your build, not your binary. This five-checkpoint guide shows where that memory goes and how to keep it small.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.