As a Senior Data Engineer at CodeValue, you'll play a crucial role in our data team, contributing to the development and maintenance of scalable data pipelines and systems. You'll be at the forefront of designing, implementing, and optimizing data solutions that drive our business objectives and empower data-driven decision-making processes. Collaboration with data analysts, data scientists, and other stakeholders will be essential to ensure data integrity, reliability, and accessibility. Responsibilities:
* Design, develop, and optimize data pipelines and architectures using a variety of technologies and tools including Python, SQL, Spark, Kafka, and Airflow.
* Develop efficient data models to facilitate storage, retrieval, and processing, with a focus on schema design for NoSQL databases, data lakes, and data warehouses.
* Ensure scalability of big data architecture to handle large data volumes and high traffic loads.
* Implement data quality checks, validation processes, and data governance measures to ensure accuracy and consistency of data.
* Monitor, troubleshoot, and debug data issues and performance bottlenecks.
* Implement security measures such as encryption, access controls, and data masking to protect sensitive data.
* Research and evaluate new data technologies and best practices to enhance data engineering processes and solutions.
* Design, develop, and optimize data pipelines and architectures using a variety of technologies and tools including Python, SQL, Spark, Kafka, and Airflow.
* Develop efficient data models to facilitate storage, retrieval, and processing, with a focus on schema design for NoSQL databases, data lakes, and data warehouses.
* Ensure scalability of big data architecture to handle large data volumes and high traffic loads.
* Implement data quality checks, validation processes, and data governance measures to ensure accuracy and consistency of data.
* Monitor, troubleshoot, and debug data issues and performance bottlenecks.
* Implement security measures such as encryption, access controls, and data masking to protect sensitive data.
* Research and evaluate new data technologies and best practices to enhance data engineering processes and solutions.
Top Profiles:
Top Profiles
Requirements:
* 4+ years of experience in data engineering or related roles.
* Proficiency in Python and SQL, with familiarity in other programming languages such as Java, Scala, etc.
* Experience with big data technologies and frameworks like Spark, Hadoop, Hive, etc.
* Familiarity with data streaming and messaging platforms such as Kafka, RabbitMQ, etc.
* Expertise in high-performance, near real-time ETL/ELT processes incorporating current and emerging data stack tools like Airflow, AWS, Kubernetes, Databricks, dbt, Spark, and Kafka.
* Experience with cloud platforms and services like AWS, GCP, Azure, etc.
* Ability to work independently and as part of a team in a fast-paced and dynamic environment.
* Bachelor's degree in Computer Science, Engineering, or related field, or equivalent work experience.
* 4+ years of experience in data engineering or related roles.
* Proficiency in Python and SQL, with familiarity in other programming languages such as Java, Scala, etc.
* Experience with big data technologies and frameworks like Spark, Hadoop, Hive, etc.
* Familiarity with data streaming and messaging platforms such as Kafka, RabbitMQ, etc.
* Expertise in high-performance, near real-time ETL/ELT processes incorporating current and emerging data stack tools like Airflow, AWS, Kubernetes, Databricks, dbt, Spark, and Kafka.
* Experience with cloud platforms and services like AWS, GCP, Azure, etc.
* Ability to work independently and as part of a team in a fast-paced and dynamic environment.
* Bachelor's degree in Computer Science, Engineering, or related field, or equivalent work experience.
This position is open to all candidates.
