Sr. Spark / Scala Developer

***THIS POSITION WILL START OFF REMOTELY BUT WILL EVENTUALLY SIT ONSITE***Job Details: We are looking for a Senior Spark/Scala Develoepr that has hands on experience working on the Hadoop Ecosystem. Roles and Responsibilities: Develop end to end data pipeline using Spark, Hive and ImpalaWrite SPARK jobs to fetch large data volumes from sourceUnderstand business needs, analyze functional specifications and map those to development and designing Apache Spark programs and algorithms.Optimizing Spark code, Impala queries and Hive partitioning strategy for better scalability, reliability and performance.Work on leading BI technologies like MSTR, Tableau over Hadoop Ecosystem through ODBC/JDBC connectionWork on hive performance optimizations like using distributed cache for small datasets, Partition, Bucketing in Hive and Map Side joins.Build Machine Learning Algorithms using Spark.Design and deploy enterprise-wide scalable operationsData wrangling and creating workable datasets and work on different file formats like Parquet, ORC, Sequence files and different serialization formats like AvroBuild applications using Maven, SBT and integrated with continuous integration servers like Jenkins to build jobs.Execution of Hadoop ecosystem and Applications through Apache HUEFeasibility Analysis (For the deliverables) - Evaluating the feasibility of the requirements against complexity and time lines.Performance tuning of Impala queriesDesign and documented operational problems by following standards and procedures using software reporting tool JIRAInstalling, configuring, and using Hadoop components like Spark, Spark Job server, Spark Thrift server, Phoenix on HBase, Flume, SqoopExpertise in Shell-Scripts, Cron Automation and Regular ExpressionsCoordinating for the Development, Integration and Production deployments.Use Rest services to access HBASE data and used the data for further processing in the downstream systemsGood experience in debugging issues using the Hadoop, Spark Log filesResponsible for preparing technical specifications, analyzing functional specs, development and maintenance of codeCreate mapping documents to outline data flow from source to target.Perform migration from Legacy Databases RDBMS to Hadoop EcosystemUse Cloudera Manager, an end-to-end tool to manage Hadoop operations in Cloudera ClusterCreate various database objects like tables, views, functions, and triggers using SQLExperience with Spark and Spark SQLMust have hands on experience in Java, Spark, Scala, AKKA,Hive, Maven/SBT, Amazon S3Experience in Kafka, ReST services is a plus.Experience in Hadoop, HBase, MongoDB, or other NoSQL platformsGood knowledge of Big Data querying tools, such as Pig, Hive, and ImpalaKnowledge in Sqoop, Flume preferredExcellent communication skills with both Technical and Business audienceExperience in Apache Phoenix, Text Search (Solr, ElasticSearch, CloudSearch)Education: Bachelors in Computer Science Degree plus 4years experience or MS in Computer science with 2years experienceRequired Technical Skills3+ years strong native SQL skills3+ years strong experience in database and data warehousing concepts and techniques. Must understand: relational and dimensional modeling, star/snowflake schema design, BI, Data Warehouse operating environments and related technologies, ETL, MDM, and data governance practices.3+ years experience working in Linux3+ years experience with Spark3+ years experience with Scala1+ years experience with Hadoop, Hive, Impala, HBase, and related technologies1+ years strong experience with low latency (near real time) systems and working with Tb data sets, loading and processing billions of records per day1+ years experience with MapReduce/YARN1+ years experience with Lambda architectures1+ years experience with MPP, shared nothing database systems, and NoSQL systemsQualifications: Ability to work in a fast-paced, team-oriented environmentAbility to complete the full lifecycle of software development and deliver on timeAbility to work with end-users to gather requirements and convert them to working documentsStrong interpersonal skills, including a positive, solution-oriented attitudeMust be passionate, flexible and innovative in utilizing the tools, their experience, and any other resources, to effectively deliver to very challenging and always changing business requirements with continuous successMust be able to interface with various solution/business areas to understand the requirements and prepare documentation to support developmentHealthcare and/or reference data experience is a plusA willingness and ability to travelRight to work in the recruiting country. - provided by Dice

Created: 2025-11-01