Data Test Engineer, Paya Lebar

Experience Requirements: Must have QA/Testing Experience. Data Testing Experience specifically in Big Data, Hadoop, or Cloud Data Warehouse environments. Databricks Experience of testing pipelines within a Databricks environment. Automation Focus: Proven track record of moving from manual SQL checks to automated Python-based testing frameworks. Mandatory: Databricks Certified Data Engineer Associate (at minimum). Preferred: ISTQB Foundation or Advanced Level (Test Automation Engineer). Core Technical Skills: Data Validation & Frameworks Great Expectations / Pandera: Proficiency in using Python-based libraries to define data 'contracts' and automated validation suites. DLT Expectations: Deep understanding of Delta Live Tables (DLT) expectations (Fail, Drop, Quarantining bad records). Advanced SQL: Expert-level SQL for complex data reconciliation, identifying duplicates, and null-value analysis across billions of records. Python for QA (PySpark): Pytest-Spark: Experience using pytest to write unit tests for PySpark transformations and logic. Notebook Testing: Ability to write automated test notebooks that validate Medallion Architecture transitions (Bronze to Silver, Silver to Gold). Data Reconciliation: Building Python scripts to perform 'source-to-target' counts and checksums across distributed file systems. Performance & Integration Testing: Scalability Testing: Ability to validate that data pipelines meet performance SLAs when data volume spikes. End-to-End Orchestration Testing: Testing the reliability of Databricks Workflows and handling of job failures/retries. Schema Evolution: Testing how pipelines handle upstream schema changes without breaking downstream Gold tables. Governance & Security Testing: Unity Catalog Validation: Testing Row-Level Security (RLS) and Column-Level Masking to ensure unauthorized users cannot see sensitive data. Data Lineage: Validating that data lineage in Unity Catalog correctly reflects the movement of data across the Lakehouse.