Insurance Company in the Midwest has a need for a Data Engineer. within the Data and Analytics Team. This position is responsible for data engineering processes supporting analytics and reporting solutions for Agency, Sales and Operations. This position will be part of a team to design and build data pipelines into data platforms that meets business requirements. This position will also help execute a strategic roadmap to a cloud data platform.
● Apply all Phases of Software Development Life Cycle (Analysis, Design, Development, Testing and Maintenance) using Waterfall and Agile methodologies.
● Utilize AWS platform tools for creating tables, data distribution by implementing Partitioning and Bucketing.
● Develop DataFrame and RDD (Resilient Distributed Datasets) to achieve unified transformations on the data load.
● Develop scheduling and monitoring Rundeck or Oozie workflows
● Transfer data from different platforms into and from AWS platform.
● Manage the source code in Phabricator or Github.
● Track and deliver requirements in Jira.
● Build distributed, scalable, and reliable data pipelines that ingest and process data at scale and in real-time.
● Create metrics and apply business logic using Spark, Scala, R, Python, and/or Java.
● Support Data Engineering team in designing and building effective and efficient data pipelines for ingestion for analytics and reporting.
● Drive the collection of new data and the refinement of existing data sources.
● Define, improve, and maintain data infrastructure and any related architecture.
● Locate, profile, cleanse, map, import and merge data from various sources
● Model, design, develop, code, test, debut, document and deploy application to production through standard processes.
● Harmonize, transform, and move data from a raw format to consumable and curated views.
● Apply strong Data Governance principles, standards, and frameworks
● Bachelor’s degree in Computer Science, Computer Engineering, Programming, Management Information Systems, or related field. Insurance industry experience is a plus.
● Minimum of five years of prior data engineer experience. (Client will consider a strong P&C Actuary who wants to transition into Data Engineering)
● Proficient in working on Apache Hadoop ecosystem components with AWS EC2 cloud computing.
● Expert in SQL and DataFrames using R and Python for Distributed Data Processing.
● Expertise in various scripting languages like Linux/Unix shell scripts, R,and Python.
● Experience in working with cloud environment AWS EMR, EC2, S3 and Athena and GCP BigQuery.