We are looking for an experienced Senior Data Engineer to join our 'Data & AI' group. As a key member of our team, you will play a crucial role in designing and building data models, developing infrastructure, and optimizing data processes to support our data analysis, data science and engineering teams. You will collaborate with various stakeholders to ensure efficient data ingestion, processing, and analysis, contributing to the overall success of our data-driven initiatives.
Key Responsibilities:
Design and build robust data models and infrastructure to support data science and engineering teams.
Develop and optimize data processes, including ETL/ELT pipelines, for efficient data ingestion and processing.
Identify and understand business processes to derive actionable insights from the data.
Collaborate with BI analysts to create comprehensive reports and dashboards.
Implement and maintain scalable data solutions in a high-volume production environment.
Utilize modern technologies such as M3DB, Open-Search, Grafana, etc., for data processing and analysis.
Lead design initiatives for new products and contribute to best practices.
Work closely with cloud providers like AWS for data storage, computation, and orchestration.
Stay updated with advancements in data engineering, big data, and cloud technologies.
Key Responsibilities:
Design and build robust data models and infrastructure to support data science and engineering teams.
Develop and optimize data processes, including ETL/ELT pipelines, for efficient data ingestion and processing.
Identify and understand business processes to derive actionable insights from the data.
Collaborate with BI analysts to create comprehensive reports and dashboards.
Implement and maintain scalable data solutions in a high-volume production environment.
Utilize modern technologies such as M3DB, Open-Search, Grafana, etc., for data processing and analysis.
Lead design initiatives for new products and contribute to best practices.
Work closely with cloud providers like AWS for data storage, computation, and orchestration.
Stay updated with advancements in data engineering, big data, and cloud technologies.
Requirements:
Bachelor's degree in computer science, engineering, or a related field.
5+ years of hands-on experience in data engineering and BI development.
Proficiency in SQL and Python for data manipulation and pipeline development.
Experience with complex ETL/ELT processes and big data technologies
Experience with Airflow.
Strong analytical skills and attention to detail
Excellent communication and collaboration abilities.
Preferred Qualifications:
Experience with machine learning algorithms and AI implementation.
Previous work with distributed systems, real-time analytics, and data visualization tools.
Familiarity with the AWS cloud platform and big data technologies.
Experience with Spark, Kafka, Elasticsearch, DynamoDB, and K8s.
Bachelor's degree in computer science, engineering, or a related field.
5+ years of hands-on experience in data engineering and BI development.
Proficiency in SQL and Python for data manipulation and pipeline development.
Experience with complex ETL/ELT processes and big data technologies
Experience with Airflow.
Strong analytical skills and attention to detail
Excellent communication and collaboration abilities.
Preferred Qualifications:
Experience with machine learning algorithms and AI implementation.
Previous work with distributed systems, real-time analytics, and data visualization tools.
Familiarity with the AWS cloud platform and big data technologies.
Experience with Spark, Kafka, Elasticsearch, DynamoDB, and K8s.
This position is open to all candidates.
