ViewJobs
Listed 11 days ago

Senior Data Engineer (Databricks, Data Modeling)

EPAM Systems

Database Development & Administration
Location
Sydney, NSW
Type
Full-time

We are seeking a Senior Data Engineer (Databricks, Data Modeling). You will model large-scale batch and streaming data into governed Bronze, Silver and Gold layers to support regulatory reporting data products. You wi…

About the role

We are seeking a Senior Data Engineer (Databricks, Data Modeling). You will model large-scale batch and streaming data into governed Bronze, Silver and Gold layers to support regulatory reporting data products. You will build on Databricks using Spark, Python and SQL, partner with stakeholders on definitions and use AI-assisted tools to speed up delivery, testing and documentation.

Responsibilities

  • Design trusted Bronze, Silver and Gold layers using medallion architecture principles
  • Model large-scale batch and streaming datasets into reusable, governed data models for reporting use cases
  • Build Databricks pipelines using Delta Lake, Delta Live Tables or Lakeflow aligned to quality and lineage needs
  • Develop Spark workloads using PySpark and Spark SQL for performance, reliability and maintainability
  • Implement streaming ingestion and processing using Spark Structured Streaming and Kafka or similar
  • Partner with business and regulatory stakeholders to translate reporting needs into data products and model definitions
  • Strengthen governance with cataloging, lineage and data quality checks using Unity Catalog or equivalent patterns
  • Contribute to CI/CD for data pipelines, Git-based workflows, testing and release practices

Requirements

  • Data modeling background across conceptual, logical and physical models using dimensional, Data Vault or similar approaches
  • Databricks experience, including Delta Lake plus governance concepts such as Unity Catalog
  • Apache Spark skill sets using PySpark and Spark SQL in production environments
  • Streaming data experience using Spark Structured Streaming, Kafka or similar technologies
  • Python capability for pipeline development and orchestration patterns
  • Advanced SQL skill sets for transformation logic, validation and performance tuning
  • Data governance experience covering quality, lineage, controls and well-defined data contracts
  • AI-assisted development tooling exposure, with Claude preferred, to accelerate build, testing and documentation

Nice to have

  • Regulatory reporting data exposure such as EMIR, MiFIR or ASIC derivative reporting
  • Data mesh experience delivering domain-owned data products
  • Data Vault 2.0 experience in large-scale enterprise models
  • Apache Kafka experience and strong understanding of streaming platform operations
Apply

via careers.epam.com

Similar jobs

  • Data Engineer

    XPT Software Australia PtySydney, NSW

    Full-time2 weeks ago

    Job Details: Role Overview We are looking for a Senior Data Engineer with strong hands-on experience in Azure Databricks and the Microsoft Azure data ecosystem. The role will design, build, optimise, and support scala…

  • Senior Databricks Engineer

    SynogizeMelbourne, VIC

    Full-time4 weeks ago

    Are you a Databricks expert who thrives on building high-performing teams and delivering cutting-edge data and AI solutions? Join us as a Senior Databricks Consultant to lead projects and mentor consultants as a Datab…

  • Senior Data Engineer

    DX1Melbourne, VIC

    $120k – $180kFull-time6 weeks ago

    Role Purpose The Senior Data Engineer is responsible for the design, development, testing, maintenance, and documentation of data pipelines and data platforms in accordance with client requirements, system specificati…

  • Senior Consultant - Data Engineer

    SynogizeMelbourne, VIC

    Full-time6 weeks ago

    Are you a Databricks expert who thrives on building high-performing teams and delivering cutting-edge data and AI solutions? Join us as a Senior Databricks Consultant to lead projects and mentor consultants as a Datab…

  • Senior Data Modeller

    Intelligen GroupSydney, NSW

    Full-time10 weeks ago

    Description ABOUT US At Intelligen, we’re building something different. Over the past three years, we’ve evolved from a bold startup to a trusted partner for some of Australia’s most complex data transformations. We’v…

  • Senior Data Engineer

    XPT Software Australia PtySydney, NSW

    Full-time9 weeks ago

    Job Description: Must Have Skills:  Strong hands‑on experience with Azure Data Lake Gen2, Synapse, Databricks  Solid experience in PySpark / Spark SQL, Databricks performance tuning & job optimization  Proficient i…