Project case study
A migration path for running enterprise Spark workloads on Kubernetes with Spark Operator and shared CI/CD.
Reduced infrastructure cost by about 30% while improving workload isolation, resource management, deployment repeatability, and scalability for data workloads.
Context
Data workloads needed a more repeatable runtime with better resource management, cleaner deployment mechanics, and lower infrastructure cost without forcing every team to invent its own Kubernetes pattern.
The work moved shared data-processing workloads from older execution patterns into Kubernetes-managed Spark execution with Spark Operator, CI/CD templates, Helm/Docker packaging, and operational controls.
System trace
A high-level operating path: where the request starts, how the system shapes it, and how other teams consume the result.
Existing Spark and data workloads were moved toward containerized execution patterns.
Spark Operator handled Kubernetes-native Spark application submission and lifecycle management.
CI/CD templates, Docker packaging, and Helm deployment conventions made the migration repeatable.
Migration scope
50+ workloads
Moved more than 50 Spark and data workloads toward Kubernetes-managed execution.
Cost outcome
~30% reduction
Reduced infrastructure cost by about 30% through better workload placement and resource management.
Architecture
Ownership
Lessons
Engineering decisions
Spark Operator gave the platform a clearer control point for submission, lifecycle, and resource behavior than one-off execution scripts.
Shared packaging and CI/CD conventions mattered because migrating one workload well is different from migrating dozens consistently.
The cost benefit came from better workload placement, scaling, and resource management rather than a cosmetic platform move.
What can be shown
The internal systems stay private. This section keeps the public parts: my role, system boundaries, technology context, scale, decisions, constraints, and what I learned.
Migration scope
50+ workloads
The migration covered more than 50 Spark and data workloads across enterprise data-processing use cases.
Cost impact
~30% reduction
The move to Kubernetes-managed Spark execution reduced infrastructure cost by about 30%.
Supporting context
High-level architecture
Can show CI/CD, container packaging, Helm deployment, Spark Operator submission, Kubernetes resource management, and workload monitoring without exposing internal clusters.
Scale signal
50+ migrated workloads and roughly 30% infrastructure cost reduction are shareable rounded signals.
Related case studies
Continue through related work or return to the full project index.
Related projects
In-house CI/CD YAML + Jenkins + Backend engineering
Created an in-house YAML-driven CI/CD framework that let teams onboard projects with very little friction while keeping validation, security scans, deployment behavior, and Jira status updates standardized.
Spark + Kubernetes (OCP) + Backend engineering
Designed and built services that keep Hive metadata consistent across independent environments using real-time listener sync, daily reconciliation, expiry cleanup, one-time interval jobs, observability, and deployment hardening.
Kubernetes (OCP) + Helm + Backend engineering
Extended enterprise data access governance around Apache Ranger-based RBAC, an external attribute store, DataHub tag-driven policies, row-level security, masking, Trino integration, audit clarity, and local/containerized development paths.