Lead Data Engineer

Job Description The Lead Data Engineer will provide both handsu2011on development and technical leadership while guiding a team of data engineers while building, reviewing, and optimizing endu2011tou2011end data pipelines on Databricks (PySpark) within the AWS Enterprise Data Platform (EDP). This person will build and optimize data pipelines in Databricks, and ingest and transform data from ERP sources such as Salesforce, SAP, Oracle, and SQL Server. They will handle production deployments, ensuring data quality across AWS DWH environments like Redshift and Athena. They will collaborate within a multiu2011vendor Agile environment and participat3 in sprint planning through Jira. They will help to enforce DevOps/Data Ops standards using GitHub, ensuring that deliverables meet timelines and enterprise architecture frameworks. The Lead will be responsible for ensuring junior and midu2011level engineers deliver effectively within each sprint cycle. We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: Skills and Requirements 6u20138+ years of experience in Data Engineering, including 2+ years in a Lead or Technical Leadership role, owning delivery while mentoring engineers and setting engineering best practices. Deep, handsu2011on expertise with Databricks on AWS, with demonstrated experience designing and operating productionu2011grade data pipelines on the Databricks Lakehouse Platform, while owning Databricks job orchestration, cluster configuration, and performance optimization. Advanced proficiency with PySpark and Apache Spark to build scalable, distributed data transformations for largeu2011volume enterprise datasets Strong experience integrating Databricks with AWS services such as S3, IAM, Redshift, Athena, and Glue Proven ability to own endu2011tou2011end data pipelines, which includes ingesting and transforming data from complex ERP and operational sources (Salesforce, SAP, Oracle, SQL Server). Strong programming and data modeling skills, using SQL and Python for data transformation and analytics use cases. DevOps / DataOps mindset, with experience using GitHub for version control, pull requests, and code reviews. Experience leading delivery in Agile environments, including active participation in sprint planning, estimation, and backlog refinement using Jira.

Created: 2026-02-18