Sr. Data Engineer – Lakehouse Architecture

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We’re looking for people who are determined to make life better for people around the world.Tech@Lilly is seeking a highly skilled Senior Data Engineer who can implement and optimize large-scale Lakehouse solutions and drive the evolution of our modern data platform while providing technical leadership to a growing team. The ideal candidate will have hands-on experience with modern data engineering technology stack and a proven track record of managing engineering talent in fast-paced environments.What you will be doingDesign and implement comprehensive Lakehouse architecture solutions using technologies like Databricks, Snowflake, or equivalent platformsBuild and maintain real-time and batch data processing systems using Apache Spark, Kafka, and similar technologiesArchitect scalable data pipelines that handle structured, semi-structured, and unstructured data to deliver AI ready data.Develop data transformation workflows using tools like DBT, Airflow, or DatabricksLead the technical strategy for data lake and data warehouse integration, ensuring optimal performance and cost efficiency.Implement data governance frameworks, including data quality monitoring, lineage tracking, data time travel and security protocols.Implement centralized data catalog system and enhance data discovery using technologies like Elastic Search / Open Search.Establish monitoring and alerting systems for data pipeline health using technologies like Apache SupersetDrive adoption of modern data engineering best practices including Infrastructure as Code, CI/CD, and automated testingCollaborate with data scientists, analysts, and business stakeholders to translate requirements into robust technical solutionsMentor a team of 3-5 data engineersFoster a collaborative team culture focused on continuous learning and innovationHow you will succeedProven ability to mentor junior engineers and facilitate knowledge sharingStrong project management skills with experience leading cross-functional initiativesCoordinate cross-functional projects and ensure effective communication between technical and business teamsDemonstrated ability to make architectural decisions and drive technical consensusEmbrace a growth mindset and actively seek opportunities to expand your leadership capabilities and technical masteryWhat you should bringExperience with streaming data technologies (Kafka)Familiarity with data cataloging tools (Apache Atlas or DataHub)Familiarity with high performance data service framework (Arrow Flight)Industry certifications in cloud platforms or big data technologiesExpert-level proficiency in Python and SQL for data transformation and pipeline developmentStrong experience with Apache Spark for big data processing and analyticsHands-on experience with cloud platforms (AWS or Azure) and their data servicesProficiency with Infrastructure as Code tools (Terraform , CloudFormation)Experience with containerization (Docker , Kubernetes) and orchestration platformsKnowledge of data modeling techniques for both analytical and operational workloadsUnderstanding of data governance, security, and compliance requirementsKnowledge in the pharmaceutical or life sciences domainBasic QualificationMaster’s degree in computer science, Engineering, or related technical field3+ years of hands-on experience with Lakehouse architectures (Databricks, Snowflake, or similar)7+ years of overall data engineering experience with large-scale distributed systemsOrganization OverviewLilly IT builds and maintains capabilities using cutting edge technologies like most prominent tech companies. What differentiates Lilly IT is that we redefine what’s possible through tech to advance our purpose – creating medicines that make life better for people around the world, like data driven drug discovery and connected clinical trials. We hire the best technology professionals from a variety of backgrounds, so they can bring an assortment of knowledge, skills, and diverse thinking to deliver innovative solutions in every area of our business.Research Tech unites science with technology to accelerate the Research and Development of medicines and to deliver therapeutic innovations. The team leverages technology and platforms to streamline scientific experimentation to help Researchers follow the science, to understand the disease and identify potential therapies. They are at the forefront of advanced analytics to enable data driven drug discovery, to innovate so Scientists can rapidly analyze and accelerate the discovery of new medicines.Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form ( for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status.Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include: Africa, Middle East, Central Asia Network, Black Employees at Lilly, Chinese Culture Network, Japanese International Leadership Network (JILN), Lilly India Network, Organization of Latinx at Lilly (OLA), PRIDE (LGBTQ+ Allies), Veterans Leadership Network (VLN), Women’s Initiative for Leading at Lilly (WILL), enAble (for people with disabilities). Learn more about all of our groups.Actual compensation will depend on a candidate’s education, experience, skills, and geographic location. The anticipated wage for this position is$70,500 – $200,200Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly’s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.WeAreLilly #J-18808-Ljbffr

Created: 2025-09-21