We are looking for a motivated Big Data Engineer that will take ownership of our Big Data Infrastructure. Ideally someone that has experience in dealing with data pumping out from high throughput, low latency mission critical systems. With your strong technical capabilities, you will participate in the architecturing of the infrastructure which is capable of handling the ingress of such data flow in realtime and hence trigger necessary business action/decision; but also persistence of such data for analytics and business intelligence.
This is a full-time position that can be remote from any location.
Responsibilities
- Capture expectation from stakeholders and translate it into technical vision and goal
- Design, deploy and operate the data platform and infrastructure
- Continuous monitoring and improvement of the data service’s performance, capability, availability & scalability
- Response to incidents promptly and identify potential issues
- Improve oneself and service to the latest technology
Qualifications
- 5-6+ years of Big Data DevOps experience
- Familiar with data processing including ETL, ELT, realtime streaming & transformation
- Familiar with various data structure / messaging format specification
- Demonstration knowledge of various big data facilities, features and suitability of different use cases
- Experience on at least 1 relational database storage programming level skills - MySQL, Bootstrap, etc.
- Knowledge on common big data components such as Hadoop, Hive, Spark, Spark Streaming, Presto, HBase, ElasticSearch, Kafka, ZooKeeper, Redis, Airflow will be a big plus
- Solid experience on big data processing, covering data capturing, transformation and data provisioning
- Good experience in monitoring, optimizing and troubleshooting big data infrastructure
- Proficient in shell scripting, Python and data query languages
- Experiences in AWS data services is a plus
- Strong teamwork, analytical mind, and keep striving for a better solution
- Dedicated to commitment and quality of delivery