Senior Java Big Data Engineer
About the project
- We do streaming data ingestion and online analytics using our custom made database (written in Java). We design tests that ensure the integrity of all the processed data. We run automatic builds, deployments and performance tests with feedback from profilers. Deployments happen every day. We strive to pay technical debt on time. We automate all possible QA testing steps and update them when a new feature is developed. Developers write all unit and integration tests.
- We run on Java 14, use EC2 and S3 services. We DO NOT run on Kubernetes. Our DB manages more than 300 terabytes of data. We use libraries like RxJava, Disruptor, Agrona, Aeron, Netty, Kotlin, Spock.
About the role
- You are looking to join a small, highly intelligent team where your voice is heard. Where each of your teammates is an expert you can learn from but is also eager to learn from you. Where you can redefine processes and tools without the need to argue with architects or bureaucrats. You show mechanical sympathy to all hardware, large and small. You want to apply this knowledge at scale while processing gigabytes of data in milliseconds. You want to build something that can make an impact on the company and waiting to deploy it to production for one hour seems unacceptably long. Building software that handles hundreds of thousands of requests per second is in your comfort zone. Sounds like you? Keep reading.
- Take active part in designing architecture of the system
- Analyse requirements from (internal) client perspective while taking into consideration performance, time and cost
- Implement new services and APIs used by all products in the company
- Design and conduct experiments to improve our algorithms and overall architecture
- Ensure applications have proper end to end tests, monitoring and documentation
- Create production clusters (Terraform), deploy to production (EC2), verify deployments
- Provide support for production services (PagerDuty)
- Investigate and implement new technologies (e.g. JVM features) related to data processing, application performance, performance monitoring, infrastructure
- Ability to look at the system from the perspective of clients (other dev teams)
- Capacity for learning and investigating how things work
- Advanced programming skills in Java
- Knowledge about garbage collection
- General knowledge of the issues of cloud computing and data processing
- Good command of written and spoken English
Nice to have
- Experience with Amazon Web Services
- Understanding of modern web app development
- Knowledge of Linux OS architecture (basic command line tools, networking, filesystems)