Responsibilities:

• Date Architecture & Engineering: Design and implement enterprise-scale date architectures, including data lakes, warehouses, and real-time streaming platforms. Develop and maintain ETL/ELT pipelines that efficiently process large volumes of structured and unstructured date from diverse sources. Ensure date quality, governance, and security standards are embedded throughout all the data engineering processes.
• Technical Implementation: Hands-on development of Databricks notebooks using PySpark, Python, and SQL for ETL automation. Create and optimize PySpark scripts for efficient data extraction, transformation, and loading from large datasets. Implement custom data manipulation, validation, and error handling solutions to enhance ETL robustness.
• Technical Guidance: Provide technical mentorship to junior date engineers and analysts. Lead code review, establish best practices, and drive adoption of modern data engineering tools and methodologies. Collaborate with cross-function teams including data scientists, analysts, and software engineers to deliver integrated solutions.
• Performance & Optimisation: Monitor and optimise data pipeline and performance, implementing solutions for scalability and cost-effectiveness. Conduct testing, debugging, and troubleshooting of data transformation processes. Verify data integrity throughout pipeline stages and resolve complex technical issues.

Requirements:

• Bachelor's Degree in Computer Science, Engineering, Data Science, or related field.
• Master's Degree in Data Engineering, Computer Science, or MBA (preferred)
• Relevant diploma or certificate in AWS cloud technologies or data engineering.
• 5-7 years of data engineering experience with AWS-native implementations.
• Proven experience working as part of a date engineering team on ETL migration projects.
• Hand-on experience implementing Databricks notebooks using PySpark, Python, and SQL for ETL automation.
• Demonstrated expertise developing PySpark scripts for efficient data extraction, transformation, and loading from large datasets.
• Experience utilising Python for custom data manipulation, validation, and error handling ETL processes.
• Proficiency employing SQL for complex joins, aggregations, and database operations within Databricks environments.
• Track record of testing, debugging, and optimising data transformation processes for accuracy and performance.
• Experience verifying data integrity throughout pipeline stages and resolving troubleshooting issues.
• Proven ability collaborating with cross-functional teams to align ETL migrations tasks with project goals and deadlines.
• Experience in government sector data projects and compliance requirements (preferred).
• Experience in HR domain including workforce analytics, payroll systems, and employee data management (preferred)
• Experience providing technical mentorship to junior team members (preferred)
• Technical Skills:
• AWS cloud platform expertise (RedShift, S3, Glue, EMR, Kinesis, Lambda)
• Databricks platform proficiency with PySpark, Python, and SQL implementation.
• Data warehouse design and implementation (Amazon RedShift, Snowflake on AWS)
• ETL/ELT migration project experience and pipeline development.
• Real-time streaming technologies (Kinesis Data Streams, Kafka)
• Data exchange platforms and API integration (AWS Data Exchange, Partner APIs)