You'll own the platform end to end: the infrastructure it runs on (Spark on Kubernetes, Apache Iceberg, AWS Glue, Airflow), the frameworks and tooling that let dozens of other engineers build on it without reinventing the wheel, and the design of the data pipelines themselves. Everything you build becomes leverage for the teams around you – your abstractions, base images, CI/CD flows, and operational patterns are what make the platform usable at scale.
You'll also own one of the hardest ongoing trade-offs in a high-scale data platform: balancing cost and performance. Compute sizing, storage layout, partitioning and compaction strategy, job scheduling – every decision has a price tag and a latency profile, and you'll be the one making those calls with data.
This role is ideal for an engineer who is equally comfortable debugging a Spark executor OOM on Kubernetes at 10am, designing a clean Python framework API at noon, and modeling the cost impact of a table layout change in the afternoon.
What You'll Do
Platform & Infrastructure
– Design, deploy, and operate our Spark-on-Kubernetes compute platform, including autoscaling, resource tuning, and multi-tenancy considerations.
– Own the lakehouse storage layer built on Apache Iceberg and AWS Glue catalog – table design, partitioning, compaction, schema evolution, and retention.
– Build and operate orchestration on Airflow: DAG standards, deployment flows, environment promotion, and reliability.
– Own production operations of the platform: monitoring, alerting, incident response, and continuous hardening.
Frameworks & Developer Enablement
– Build the code frameworks, libraries, and templates that other engineers use to write pipelines – so that spinning up a new production-grade Spark job is measured in hours, not weeks.
– Define and enforce standards for pipeline structure, testing, observability, and deployment across teams.
– Own CI/CD for data workloads: image builds, artifact promotion, and GitOps-based delivery.
– Act as a technical partner to product and research teams building on the platform – your customers are other engineers.
Data Pipelines & Architecture
– Design and build scalable batch and streaming pipelines processing complex, high-volume datasets from diverse sources.
– Lead large-scale backfills and migration initiatives, ensuring data consistency and integrity across evolving storage and compute platforms.
– Design event-driven data flows over large-scale queue systems (Kafka) for reliable, efficient data movement.
Cost & Performance
- Continuously balance cost against performance: right-size compute, tune queries and jobs, optimize storage layout and file sizes, and choose the correct engine for each workload.
– Build cost visibility and attribution into the platform so trade-offs are made with data, not guesswork.
– 5+ years of experience in software engineering, with meaningful time spent building and operating large-scale data platforms.
– Strong hands-on experience with distributed processing engines (Spark strongly preferred), including performance tuning and debugging in production.
– Practical experience deploying and operating workloads in Kubernetes-based environments – you're not afraid of infra work; you enjoy it.
– Experience building shared frameworks, libraries, or internal tooling used by other engineers, with the product mindset that comes with it (clean APIs, docs, versioning, backward compatibility).
– Strong proficiency in SQL and data modeling: complex analytical queries, query tuning, partitioning strategies.
– Solid software engineerin המשרה מיועדת לנשים ולגברים כאחד.
