Hadoop Administrator
Summary
| Title: | Hadoop Administrator |
|---|---|
| ID: | 1115 |
| Department: | Engineering |
Description
Hadoop Administrator
Job Summary
The Hadoop Administrator responsible for several Hadoop clusters which ingest, analyze, and serve data on hundreds of terabytes and billions of lines of open source code we continuously spider. The ideal candidate has several years’ experience maintaining a stream and batch processing infrastructure, using current Hadoop (YARN, Storm, Kafka, Hive, HBase, Phoenix, Solr) technologies, and highly available containerized REST services. As DevOps Engineer you will streamline deployments, work with developers and engineers to identify and assess new technologies, monitor and tune performance to handle increasing workloads, and contribute to the evolution of the platform architecture. This position reports to the VP Engineering - KnowledgeBase.
Responsibilities include, but are not limited to:
· Understand, anticipate, and plan for Black Duck’s data processing / analytics requirements
· Rollout current Hadoop components on our clusters (from Hortonworks distribution and upstream)
· Spec, requisition, and provision servers (real or virtual, on premise and cloud) for dev/test/prod roles
· With Engineering and IT, maintain and evolve redundant infrastructure for serving data globally behind REST APIs
· Monitor performance of stream and batch processes and tune clusters
· Automate processes including log analysis, integration with build, test, deployment systems
· Evaluate tools and technologies to improve the overall data ingestion, transformation, serving process
Requirements:
· 5+ years’ experience in IT working in Linux environments
· 2+ years’ experience as a Hadoop administrator
· Expert Linux command line and shell programming, scripting skills (Bash, Python)
· Experience configuring and tuning YARN, MapReduce, Hive, HBase, Oozie
· Experience with traditional RDBMS (e.g. Postgres), and moving data between systems
· Understanding of agile software development life cycle
· Excellent communication skills - able to interface with multiple groups across the company
About Black Duck
Black Duck provides the world’s only end-to-end platform for OSS Logistics, enabling enterprises of every size to optimize the opportunities and solve the logistical challenges that come with open source adoption, governance, and management. As part of the greater open source community, Black Duck connects developers to comprehensive OSS resources through the Black Duck Open Hub (formerly Ohloh.net), and to the latest commentary from industry experts through the Open Source Delivers blog. Black Duck is headquartered near Boston and has offices in San Mateo, London, Paris, Frankfurt, Hong Kong, Tokyo, Seoul, and Beijing.
Interested applicants may apply via: Black Duck Careers
Job Summary
The Hadoop Administrator responsible for several Hadoop clusters which ingest, analyze, and serve data on hundreds of terabytes and billions of lines of open source code we continuously spider. The ideal candidate has several years’ experience maintaining a stream and batch processing infrastructure, using current Hadoop (YARN, Storm, Kafka, Hive, HBase, Phoenix, Solr) technologies, and highly available containerized REST services. As DevOps Engineer you will streamline deployments, work with developers and engineers to identify and assess new technologies, monitor and tune performance to handle increasing workloads, and contribute to the evolution of the platform architecture. This position reports to the VP Engineering - KnowledgeBase.
Responsibilities include, but are not limited to:
· Understand, anticipate, and plan for Black Duck’s data processing / analytics requirements
· Rollout current Hadoop components on our clusters (from Hortonworks distribution and upstream)
· Spec, requisition, and provision servers (real or virtual, on premise and cloud) for dev/test/prod roles
· With Engineering and IT, maintain and evolve redundant infrastructure for serving data globally behind REST APIs
· Monitor performance of stream and batch processes and tune clusters
· Automate processes including log analysis, integration with build, test, deployment systems
· Evaluate tools and technologies to improve the overall data ingestion, transformation, serving process
Requirements:
· 5+ years’ experience in IT working in Linux environments
· 2+ years’ experience as a Hadoop administrator
· Expert Linux command line and shell programming, scripting skills (Bash, Python)
· Experience configuring and tuning YARN, MapReduce, Hive, HBase, Oozie
· Experience with traditional RDBMS (e.g. Postgres), and moving data between systems
· Understanding of agile software development life cycle
· Excellent communication skills - able to interface with multiple groups across the company
About Black Duck
Black Duck provides the world’s only end-to-end platform for OSS Logistics, enabling enterprises of every size to optimize the opportunities and solve the logistical challenges that come with open source adoption, governance, and management. As part of the greater open source community, Black Duck connects developers to comprehensive OSS resources through the Black Duck Open Hub (formerly Ohloh.net), and to the latest commentary from industry experts through the Open Source Delivers blog. Black Duck is headquartered near Boston and has offices in San Mateo, London, Paris, Frankfurt, Hong Kong, Tokyo, Seoul, and Beijing.
Interested applicants may apply via: Black Duck Careers
This opening is closed and is no longer accepting applications
