Data Consultant

Design, develop, and optimize batch and streaming data pipelines using Spark, PySpark, Databricks, and AWS Glue. Architect and maintain data lakes and warehouses (Snowflake, Redshift) and implement dbt-based data modeling for analytics-ready datasets. Build and manage orchestration workflows using Apache Airflow, and automate deployments with CI/CD pipelines (GitHub Actions, Terraform, Jenkins). Implement data governance, security, and compliance controls, including PII masking, RBAC, and encryption. Collaborate with cross-functional teams (product managers, analysts, data scientists) to deliver datasets powering BI dashboards and ML models. Optimize Spark jobs and SQL queries for performance and cost efficiency on large-scale datasets (billions of rows). Develop containerized solutions using Docker and Kubernetes, ensuring high availability and scalability. Monitor and troubleshoot pipelines using CloudWatch, Splunk, and Azure Monitor to maintain SLA compliance.

Created: 2026-03-17