Senior, Data Engineer (Crawler Service)

Binance logo

Binance

View Salaries, Reviews, and more  

Job Summary


Job Type
Permanent

Seniority
Senior

Years of Experience
At least 4 years

Tech Stacks
HTTP Data Extraction Strategy Scrapy Linux Python Java

Job Description

Are you looking to be a part of the most influential company in the blockchain industry and contribute to the crypto-currency revolution that is changing the world?

Responsibility

  • Architecture design and research and development of distributed crawler and data acquisition system
  • Design and development of distributed crawler module service architecture and data storage architecture
  • Realization of daily network data capture requirements and quality monitoring of collected data
  • Reptile data extraction, cleaning, weight elimination, statistics, etc
  • Optimize the crawling strategy, make full use of bandwidth resources, avoid various restrictions, and improve the crawling effect
Requirements

  • More than three years of Internet or enterprise-level web crawler development experience
  • Work conscientiously, meticulously and practically, have strong learning ability, take solving technical problems as fun, have ideas and dare to challenge
  • Familiar with Linux platform, solid basic skills in Java or Python, able to design and write crawler system independently is preferred
  • Familiar with the principles and techniques of web crawling, regular expressions, multithreading, HTTP protocol, and be able to obtain information from structured and unstructured data
  • Familiar with the concepts and processes of crawling, seed, parsing, downloading, deduplication, extraction, filtering, scheduling, asynchronous processing, etc
  • Familiar with one or more open source technologies in WebMagic/Scrapy/Heritrix/HtmlParser/Jsoup/HttpClient
  • Experience in verification code cracking, anti-crawling, distributed crawler architecture, data mining, and building data warehouses is preferred
  • A background in data mining, natural language processing, information retrieval, and machine learning is preferred
  • Have a good team spirit and cooperative spirit, full of enthusiasm for work and a sense of responsibility
Conditions
  • Do something meaningful; Be a part of the future of finance technology and the no.1 company in the industry
  • Fast moving, challenging and unique business problems
  • International work environment and flat organisation
  • Great career development opportunities in a growing company
  • Possibility for relocation and international transfers mid-career
  • Competitive salary
  • Flexible working hours, Casual work attire

banner icon
Interested in common interview questions?
Let's prepare for potential interview questions tailored to your job description and work experience.
Get Started!

Achieve your dream job with our top-notch tools!

Resume Checker Illustration

Resume Checker

Our free resume checker analyzes the job description and identifies important keywords and skills missing from your resume in just a minute!

Check Now
Resume Checker Illustration

Interview Preparation

Utilizing advanced AI, our tool generates tailored interview questions based on your industry, role, and experience. Practice and receive feedback on your answers in real time!

Let's Prepare
Resume Checker Illustration

Resume Builder

Let us show you the differences between a bad, good, and great resume, and guide you in building a resume that helps you stand out to employers, ensuring you land your next position faster!

Build Resume