Data Engineer

Job Description

We are looking to hire an experienced Data Engineer who we want to come with creativity, expertise, flexibility and drive, but above all a desire to learn and keep learning We want you to want to understand the big picture and how your work makes a difference.

Responsibilities

• Helping design, build, maintain and operate the data pipeline.
• Defining and building modular data pipeline components.
• Ensuring that solid development practices, such as proper use of
• source control, full testing processes and automated deployment
• mechanisms, are followed
• Collaborate with our data scientists, product and business team
• to discover where business value can be found within the data
• we have available.
• Maintaining existing systems and supporting migration to our
• new data pipeline architecture.
• Acting as a subject matter expert on all aspects of the data
• pipeline. Identifying potential performance issues, bottlenecks
• and pain points and recommend new and creative ways of
• resolving them

Requirements

• 3+ years of proven experience using Python to build data
• pipelines, including familiarity with python's core big data/data
• science libraries: e.g. pandas, pyspark, scikit-learn etc
• Solid understanding of database design and SQL
• Experience working in cross-functional agile teams, particularly
• teams including Data Scientists, Software Engineers and Business
• Analysts
• The ability to communicate complicated technical solutions to
• non-technical users
• Take ownership of feature development and ongoing
• maintenance
• Technical understanding of infrastructure components, their
• dependencies, and interactions between servers, virtual systems,
• networks, databases, web applications, etc
• Distributed data processing, for example, Spark
• NoSQL Databases, such as MongoDB or Couchbase
• Cloud computing platforms, such as Google Cloud Platform or
• AWS
• Pipeline orchestration, for example, Airflow
• Technical understanding of infrastructure components, their
• dependencies, and interactions between servers, virtual systems,
• networks, databases, web applications.