Data Engineer | Data Analyst | Platform Engineer | DevOps.
About me
Hello! I am a Data Analytics evangelist and dedicated data engineer with a passion for transforming raw data into valuable insights. With expertise in data modeling, ETL processes, and data visualization, I thrive on streamlining complex workflows and uncovering hidden patterns.
As a constant learner, I stay updated with the latest trends and technologies in the data analytics field. I believe collaboration is key to solving challenges, and I'm excited to connect with like-minded professionals and projects.
Let's harness the power of data together and shape a smarter, data-driven future! If you're seeking a data engineer with a keen eye for detail and a commitment to excellence, I'm here to help you unlock the true potential of your data. Let's make a difference, one dataset at a time!
Top projects
Observability.
- Elastic Stack
- Node.js
- MS Teams
- Kafka
Designed and implemented inhouse alerting and monitoring framework using elastic stack and custom applications.
Setup of Datamesh.
- Apache Kyuubi
- Trino
- Alluxio
- Datahub
- Jupyter
Enabled DataMesh architecture using kyuubi, ranger, trino, s3, airflow, datahub, dbt & spark.
Automatic engine selection for Kyuubi.
- Apache Kyuubi
- java
- Apache Spark
- Trino
Patched codebase for dynamically allocating interactive or batch engine based on user AD groups.
Portfolio website
- React
- Next.js
- MongoDB
- Tailwind
- Prisma
Created my website using nextJS. This is a portfolio website having a blog integration with WIX.
Context Aware Rule Engine
- Flink
- Kafka
- Java
- Elastic Stack
- Influx
- Airflow
Developed a context aware platforn, which takes deisions based on user past activity in real time and to trigger business rules.
Telecom - Network Datalake
- Flink
- Kafka
- Java
- Elastic Stack
- Influx
- Airflow
- NiFi
Designed and implemented a datalake, warehouse for network data comin from mobile towers. Managed data at petabyte scale with >2Trillion events coming every day.
Retail - Cross Shopping Behaviour
- Spark
- Python
- Power-BI
- Airflow
Developed solution for biggest retailers worldwide to analyse on how customers cross shop b/w departments, categories or products. This was enhanced with multiple customer segmentations
Retail - Association Rule Engine
- Spark
- Python
- Power-BI
- Airflow
- Statistics-Apriori
Developed solution for biggest retailers worldwide to analyse on how products are related, and which products, categories, or even departments are shopped together, Used Apriori algorithm at big data scale
Retail - Category Uplift & Cannabalization
- Spark
- Python
- Power-BI
- Airflow
- Statistics-Apriori
Developed solution for biggest retailers worldwide to analyse on what is the impact of new product launch, how it isi adding to the sales and how much it cannibalises the other products in the same category.
My skills
- Spark
- Flink
- Kubernetes (OCP)
- Python
- Java
- MongoDB
- Cloud (GCP, AWS)
- Kafka
- MongoDB
- Elastic Stack
- Nifi
- Datahub
- DBT
- Airflow
- Influx
- Grafana
My experience
Data Engineer
Airtel DigitalI work as a full-stack Data Engineer designing PetaByte scale data pipelines. My expertise is in Distributed Systems, and designing efficient data pipelines and data product APIs.
2021 - present
Data Science Engineer
dunnhumbyI worked as bridge b/w data scientist and big data platforms. I worked on multitude of ML and Statistic Analysis problems. I created products and solutions to be reused by diverse client base across world, including some of the biggest retailers in the world. I created Customer Segmentation, Setup data marts, reporting platform, on demand analytics product
2018 - 2021
Software Engineer
MphasisI worked in Mainframe and Big data ecosystem as a developer for Insurance and Telecom client. My role was instrumental in migraiton from Mainfram to Spark, along with some key automations which saved more than 1000 man hours per year.
2014 - 2018Contact me
Please contact me directly at contact@vijayjangir.com or through this form.