Project case study
Governed self-service data platform for discovery, query access, transformation, and BI.
Created a governed self-service data path where teams could discover data, transform it, query it, and consume BI outputs without bypassing access control or central metadata.
Context
Teams needed self-service access to shared data assets, but the platform still had to preserve authorization, secrets, metadata, transformation ownership, and a usable BI layer.
The platform used Apache Kyuubi as the governed access layer with custom changes for engine selection, RBAC, and secrets management; Trino as a query engine; dbt for transformation; DataHub as the central metadata base; and Metabase as the BI layer.
System trace
A high-level operating path: where the request starts, how the system shapes it, and how other teams consume the result.
Kyuubi served as the governed access gateway, with custom platform changes for engine selection, RBAC, and secrets management.
Trino handled interactive query execution, while Spark supported heavier processing paths.
dbt provided transformation structure, DataHub served as the central metadata base, and notebooks supported exploration.
Platform model
Governed self-serve
The platform connected access control, query engines, metadata, transformations, notebooks, and BI instead of treating them as disconnected tools.
Access gateway
Kyuubi customizations
Custom changes covered engine behavior, RBAC, and secrets management around shared query access.
Architecture
Ownership
Lessons
Engineering decisions
Kyuubi was the right place to centralize access behavior because it sits close to users, engines, identity, and workload routing.
DataHub became the shared metadata base so discovery and governance did not depend on scattered project knowledge.
dbt and Metabase served different parts of the workflow: transformation ownership and BI consumption.
What can be shown
The internal systems stay private. This section keeps the public parts: my role, system boundaries, technology context, scale, decisions, constraints, and what I learned.
Platform shape
Governed self-serve
The shareable story is the operating model: query access, authorization, secrets, metadata, transformation, and BI connected into one platform path.
Access layer
Kyuubi + Trino
Kyuubi handled governed entry points and engine behavior while Trino provided interactive query execution.
Supporting context
High-level architecture
Can be shown as Kyuubi access gateway, custom engine routing, RBAC, secrets management, Trino query execution, dbt transformations, DataHub metadata, Metabase BI, notebooks, and scheduled platform workflows.
Open-source reference
The case study can cite the named open-source projects as technology context without implying ownership of enterprise deployment details.
Related case studies
Continue through related work or return to the full project index.
Related projects
Spark + Airflow + Platform engineering
Built browsing-log ingestion and analytics pipelines for safe-browsing classification, audience management, cohort creation, and pattern-based downstream data products.
Trino + DataHub + Platform engineering
I architected and built a governed conversational data and visualization agent: it retrieves business knowledge, answers business questions, runs governed queries from that context, reasons over results, and builds charts without making the LLM the data boundary.
Apache Ranger + Trino + Platform engineering
Extended enterprise data access governance around Apache Ranger-based RBAC, an external attribute store, DataHub tag-driven policies, row-level security, masking, Trino integration, audit clarity, and local/containerized development paths.