The position
We are seeking a skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have extensive experience in building and optimizing data pipelines, handling large datasets, and working within cloud-based and on-premise environments. You will be responsible for designing and implementing efficient, scalable solutions to manage, process, and analyze high volumes of data, ensuring the reliability and performance of our data infrastructure.
What will your job look like:
Design, build, and maintain scalable data pipelines to process large datasets efficiently.
Develop and implement data models and architectures that support both real-time and batch data processing.
Ensure data integrity, security, and accuracy across all systems.
Collaborate with data scientists, analysts, and other engineers to ensure data availability and quality.
Optimize data retrieval and storage processes to handle large volumes of data seamlessly.
Work with structured, semi-structured, and unstructured data, integrating various data sources.
Troubleshoot and resolve data issues, ensuring continuous operation of the data infrastructure.
Maintain and enhance ETL processes, ensuring scalability and performance in handling large datasets.
Stay up-to-date with industry best practices and emerging technologies related to big data engineering.
All you need is:
Bachelor’s degree in Computer Science, Engineering, or a related field.
At least 4 years experience
in data engineering, with a focus on large-scale data processing and big data technologies.
Strong proficiency in programming languages such as Python.
Experience with data pipeline and workflow management tools
Hands-on experience with large-scale data processing frameworks like Apache Spark, Hadoop, or similar.
Familiarity with data modeling, ETL processes, and data warehousing concepts.
good knowledge of relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., Cassandra, MongoDB).
Significant leadership ability
Experience in building teams and processes from scratch
Nice to have:
Experience working with real-time data streaming technologies (Kafka, Flink).
Knowledge of machine learning frameworks and integrating data pipelines for model training and deployment.
Experience with version control systems (e.g., Git), CI/CD pipelines, and automation tools.
Experience with containerization (Docker, Kubernetes) is a plus.
