Data Catalog Platforms Built on Open-Source Metadata Engines

Summary

We build data catalog platforms on open-source metadata engines. A catalog streamlines enterprise-wide data discovery by collecting metadata about every data asset an organization holds and making it searchable in one place. Rather than storing data itself, it stores descriptions of it: datasets, tables, columns, dashboards, reports, documents, pipelines and machine learning models, along with ownership, business glossary terms, classification tags and lineage. Work of this kind spans engine selection, deployment, interface extension and access model design.

The Challenge

Building a catalog from scratch is rarely the right call, because the hard and unglamorous work is connector coverage, and mature open-source platforms already provide it. The real difficulty is choosing an engine and then customizing it in a way that survives. Customization expressed as configuration and plug-in extension carries forward through upgrades; customization expressed as a source fork does not, and a catalog that falls several versions behind loses exactly the connector coverage it was chosen for.

Because a catalog concentrates knowledge about sensitive data, it also becomes an access-control problem in its own right. Policies have to distinguish who may see that an asset exists, who may see its schema and profiling statistics, and who may request or grant access to the underlying data. Where the catalog is hosted for multiple organizations, tenant isolation has to be enforced at the metadata store and search layers, not only in the interface. Two further traps are common: full-crawl ingestion that eventually outgrows its schedule window, and an uncurated glossary that leaves the catalog as a search index over noise. In research-heavy environments such as medical centres, the platform must also cover research papers and business intelligence reports alongside conventional database assets.

The Solution

Choosing and customizing an engine

We evaluate candidates on scalability under complex data estates, breadth of customization, native integration with the tools already in use such as object storage, cloud data warehouses, relational databases and streaming platforms, ease of extension toward AI-assisted search, and cost-to-performance. DataHub is a strong default because its metadata model is entity-and-aspect based, which means new asset types and custom attributes can be added without forking the core schema.

Ingestion, storage and search

• Ingestion connectors crawl source systems on a schedule, emit metadata change events, and push them through a stream into the metadata service.
• The metadata store persists entities and their aspects with full version history, so the state of any asset description at any past point remains reconstructable.
• A search index, typically Elasticsearch or equivalent, serves discovery queries with faceting across tags, ownership, platform and domain.
• Lineage comes from the connector where the source exposes it, or from parsing query logs and transformation definitions, which is the harder and more valuable path.
• Ingestion is kept incremental rather than full-crawl.

Access control, identity and multi-tenancy

Single sign-on against the organization's identity provider over OIDC is the baseline, with group membership mapped to catalog roles and provisioning automated so leavers lose access without manual intervention. Where the deployment is a hosted product, each tenant's identity provider configuration is kept independent and isolation is enforced in the storage and search layers.

Deployment and interface work

Deployment runs on Kubernetes with a private container registry, which keeps environments reproducible, and interface work is layered as custom components over the platform's extension points rather than as edits to core views.

How the build runs

We elicit the target capability set through structured interviews, then deploy a working instance early against real source systems, because connector behaviour against actual estates is where the surprises live. Whether the deployment is a product or an internal platform is settled up front, since that shapes tenancy and upgrade strategy. Documentation is part of the work, since a user guide covering sign-in, ingestion, search and access requests determines whether a catalog is adopted.

What This Delivers

People find the right asset, understand where it came from and who is accountable for it, and request access through one surface instead of asking around. Lineage makes the effect of a change traceable before it is made. Access policies keep a concentrated view of sensitive data governed, and upgrade-safe customization means the platform keeps gaining connector coverage rather than falling behind it.

Technologies and Tools

• DataHub or an equivalent open-source metadata engine with an entity-and-aspect model
• Elasticsearch or equivalent for discovery and faceted search
• Kubernetes with a private container registry; Azure and other cloud hosting
• OIDC single sign-on with automated group-to-role provisioning