Databricks Lead

Databricks Lead Index Analytics, LLC, is a rapidly growing, Baltimore-based small business providing health-related consulting services to the federal government. At the center of our company culture is a commitment to instilling a dynamic and employee-friendly place to work. We place a priority on promoting a supportive and collegial team environment and enhancing staff experience through career development and educational opportunities. Position Overview The incumbent will lead the development and implementation of Databricks pipelines across multiple environments with access from the Client's AWS Government Cloud. As part of a cross-functional project team that includes data engineers, database architects, analysts and analytic SMEs, the Databricks Lead will implement a scalable Databricks environment capable of ingesting diverse data sources to support advanced data analytics. Responsibilities Architect, develop and implement data models and data engineering pipelines to support ingestion of diverse CMS data sources, applying complex business rules derived from Python/R code reviews and enable analytics through optimized Apache Spark datasets while maintaining strong security and governance practices. Engineer and manage end-to-end data processing workflows including receiving data files in multiple formats, applying validated business logic, loading transformed data into production pipelines, delivering recurring and ad-hoc data extracts to meet stakeholder and program needs. Support and administer the Databricks environment including user and access control management for clusters, jobs, notebooks, DDL deployments, optimizing performance, monitoring usage and delivery of cost and utilization reporting. Ensure data quality and maintain documentation conducting comprehensive QA validation on all ingested and delivered datasets, managing codebooks and user training guides as well as maintaining complete metadata, source-to-target mappings and technical artifacts according to established schedules. Qualifications U.S. citizen or otherwise authorized to work in the United States and able to demonstrate physical residency in the U.S. for at least three (3) of the past five (5) years. Bachelor's degree or higher in computer science or relevant discipline with eight (8) or more years of professional work experience, including six (6) or more years of data engineering experience designing and building pipelines across a variety of database technologies. Databricks certified Data Engineer certifications are a plus. Three (3) or more years of hands-on experience with the Databricks Platform and developer tools including performance tuning, use of Databricks CLI and REST APIs, and cost optimization for Databricks compute and storage. Two (2) or more years of hands-on programming experience with SAS, Python, JavaScript or similar tools. Working experience with Databricks integrations, including Delta Lake, LakeFlow and Lakehouse Federation. Proven experience with Databricks Unity Catalog and related governance/security frameworks as well as working knowledge of database security, including Access Control Lists (ACLs), Unity Catalog permissions, and RBAC controls. Proficiency with cloud platforms such as AWS or Azure, including database deployment automation and CI/CD pipelines in an AWS environment. Experience working with unstructured and semi-structured data (XML, JSON, Parquet, video, audio, images, PDF) is a plus. Strong communication skills and commitment to continuous learning. We're dedicated to ensuring a safe and transparent recruitment process for all candidates and have implemented robust measures to protect your personal information. If you are selected for an interview, please be advised that Index Analytics LLC reserves the right to prohibit the use of artificial intelligence (AI) tools, including but not limited to AI-generated responses, real-time transcription, or automated assistance during the interview process. We value authentic interactions and the opportunity to engage directly with candidates. Any unauthorized use of AI may result in disqualification from consideration. The salary range provided represents the estimated compensation for new hires in this position, applicable across all locations. Actual offers may vary based on factors such as the candidate's skills, qualifications, experience, and market conditions. Index complements its base salary offering with a competitive package that includes health and retirement benefits, discretionary bonuses, and reimbursement for professional development opportunities. Index Analytics provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

Created: 2026-03-04