Led and contributed to a large-scale data platform migration, designing and developing ETL pipelines using Apache Spark, PySpark, Spark SQL, and Python, orchestrated with Apache Airflow on AWS EMR, processing up to 3 billion records and 1 TB of data per execution. Optimized distributed Spark workloads, by tuning Spark jobs, SQL transformations, partitioning strategies, and execution parameters, reducing pipeline runtimes from 2 hours to 1 hour (50% improvement) and enabling faster data availability. Implemented Infrastructure as Code (IaC) using Terraform to define and manage AWS service configurations and resource capacities, ensuring scalability, consistency, and reproducibility across environments. Monitored Spark job execution on AWS EMR using Amazon CloudWatch, and used Jenkins pipelines to deploy Infrastructure as Code (Terraform/CloudFormation), provisioning and configuring EMR clusters and AWS resources consistently across environments. Performed data validation and reconciliation using Snowflake, Amazon Athena, and Great Expectations, enforcing data quality, accuracy, and consistency throughout the migration process.

Designed and implemented a cloud-based data lakehouse architecture on AWS, following an initial evaluation of Azure and GCP and ultimately selecting AWS to accelerate delivery timelines and enable self-service analytics for business users. Built end-to-end data pipelines ingesting data from on-premise databases, storing Parquet datasets on Amazon S3, and transforming data into curated layers using AWS Glue, Apache Spark, Python, and Pandas. Implemented data governance and data quality processes, embedding validation rules and controls in Python to ensure accuracy, consistency, and reliability across lakehouse layers. Leveraged Amazon S3 with AWS Glue Data Catalog to manage table metadata, and Amazon Athena to validate, reconcile, and query curated datasets prior to consumption. Optimized SQL queries and stored procedures used in existing pipelines, reducing execution times from 10 minutes to 3 minutes (70% improvement) and improving both ETL runtimes and dashboard refresh performance, while delivering tactical Power BI dashboards to support immediate business needs.

Designed and implemented data pipelines using Python, SQL, Pentaho, Apache Spark, and AWS, selecting technologies based on project requirements to process and deliver analytical datasets at scale. Developed automated monitoring pipelines using Python, Linux Bash scripting, and SQL to populate a centralized database tracking server and platform status, improving operational visibility and proactive incident detection. Automated data ingestion and process orchestration using Python and AWS services, reducing manual execution and improving service and pipeline startup times by 10%. Optimized pipeline performance by tuning SQL queries, transformation logic, and Pentaho jobs, achieving 15% faster execution times across data processing workflows. Implemented data quality checks and validations using Python and SQL, and built enterprise BI dashboards with MicroStrategy on top of curated and reliable analytical datasets.

Designed and implemented ETL pipelines orchestrated with Python, leveraging SQL and PL/SQL packages on Oracle for large-scale data extraction and transformation, processing approximately 1 million records per table. Developed and optimized complex PL/SQL logic on Oracle, improving data transformation performance and reducing dashboard data load times by 20%. Built and managed analytical datasets stored on AWS, enabling scalable and reliable access to curated data for enterprise reporting. Developed and maintained enterprise BI dashboards using MicroStrategy, powered by optimized Oracle-based queries and dimensional data models to support institutional decision-making.

Designed and developed end-to-end systems using C#, .NET, JavaScript, and SQL, optimizing ticketing and operational workflows and improving service response times by approximately 30%. Designed and implemented relational and dimensional data models (fact and dimension tables), enabling structured analytical reporting and KPI tracking. Built Power BI dashboards and optimized SQL queries to transform operational data into actionable insights, laying the foundation for data analytics and engineering practices.