For our client we are looking for a Streaming Data Specialist, at the department of Autonomous System, we are developing transport solutions for the future. Autonomous transport environments such as autonomous mining sites or autonomous transport hubs are examples of cyber-physical systems that amongst other things, produce large amounts of data. We are looking for you, a data engineer who is enthusiastic to make this big data useful via appropriate data streaming, modeling, visualization and analysis. You will work closely together with a team of inspiring and engaged colleagues. Our main development task is to support operations in autonomous transport system via effective utilization of data.
Responsibilities
- Helping the team to design and develop appropriate data pipelines, from different sources of data to the final users.
- Creating and consuming Kafka Topics on the cloud as well on-premise.
- Defining scenarios within autonomous transport systems to apply data-driven methods.
- Developing software and services.
- Designing algorithms for data analysis within autonomous transport system domain.
Requirements
- Solid experience in Apache Kafka (Stream Processing Platform) and Event-Driven Systems.
- Design, develop, and maintain scalable data pipelines and solutions using Databricks, integrating with AWS, to process and analyze large datasets for business insights.
- Demonstrated experience in Alarming, Monitoring and Logging for Event-driven Systems.
- Solid experience in AWS (Amazon Web Services) Cloud Computing Platform. Experience in Other cloud computing platforms (e.g. Microsoft Azure) is a plus.
- Knowledge and experience in Terraform (Infrastructure as Code tool for cloud).
- Knowledge and experience in CI/CD and GitLab.
- Solid experience in Java Programming Language.
- Solid experience in Python (with focus on Machine Learning and Data Analytics).
- Knowledge and experience in Machine Learning and Data Analytics tools such as Databricks, Apache Spark, TensorFlow etc.
- Good theoretical background in Machine Learning, Big Data Analytics and Statistics.
