We are looking for an experienced Senior Data Engineer for a rapidly expanding Data Engineering team in Sydney. They are Innovative and growing within a disruptive technology sector and are looking for their next hire within their Data Engineering team.
We are looking for an experienced Senior Data Engineer for a rapidly expanding Data Engineering team in Sydney. They are a reputable and growing within a disruptive technology sector and are looking for their next hire within their Data Engineering team.
This job will have the following responsibilities:
Responsible for design and development of high performing micro-service style server applications, REST APIs and distributed processing systems using Spark/Scala.
Analyzes business/ functional requirements and prepares development project schedule, tasks, and estimates
Identify and build automated solutions for the acquisition, processing and management of data that would otherwise require manual effort
Build distributed, scalable, and reliable data pipelines that ingest and process data at scale
Build the API layer to access Machine Learning Models
Ensures application design, development schedule, and implementation meet or exceed documented application scope/timeline while adhering to established standards
Qualifications & Requirements:
BS or higher degree in Computer Science/Engineering or related field
Must have experience with Big Data / Analytics technologies like Hadoop, Spark, Python, Scala, R, Machine Learning. Machine Learning Models using REST APIs. Designing and building highly concurrent and high volume REST APIs using Microsoft.Net Core using C#. Addressing concerns with big data management (Governance, Role Based Access Control using LDAP etc.)
A solid foundation in data structures, algorithms, design patterns
Highly desired knowledge of Azure components like Azure Data Lake Store, SQL Server, HDInsight, Web Jobs, Functions, Key vault, Data Factory is preferred. NoSQL databases like MongoDB is a big plus.Caching technologies like Redis is a plus. Knowledge of React, ExpressJS, Node is a plus.
Strong aptitude for problem-solving, particularly to modify and enhance processes and workflows. Outstanding communicator with both business and technology audiences
Experience with statistical methodologies and tools (R, SAS, SPSS, etc) and predictive modeling
Important Functional Abilities
Identifies unique opportunities to collect new data and mine existing data.
Experience in disparate data sources and bridging various sources to create a market view that can be used as a proprietary source of market data.
Experienced at leveraging both structured and unstructured data sources.
Responsible for validation, quality control and integration of 3rd party data into systems ensuring cohesive, accurate reporting
Designs new processes and is able to build large, complex data sets.
Develop strategies new uses for data and its interaction with data design using a variety of techniques and applications across various data sources.