Lead Data Engineer

Lead Data Engineer

• Contract duration: 12 + 12 + 12 + 12 months 

• Location of work: Anywhere in Australia / Remote 

• Application closing date: Wednesday, 26 March 2025

• Estimated start date: Tuesday, 01 July 2025

• Security Clearance: Able to obtain baseline clearance

Our federal government client, IP Australia (IPA), is seeking an ETL Developer/Data Engineer to provide technical leadership to carry out development of and provide production support for a series of data load or data replication workflows on an AWS-based Data Lake platform.

Key Skills:

• Data pipelines/ETL/ELT in PySpark (demonstrated in AWS Glue)

• Management of data in a variety of table formats including Parquet and Iceberg

• Management of data ingestion/egress using REST APIs

• Management of data ingestion using AWS native technologies such as DMS and OpenSearch

• Data manipulation in SQL and Python

• Management and deployment of AWS infrastructure using CloudFormation

• Management of lifecycle policies and backups in AWS S3

• Management of data workflows using a combination of AWS Lambda and Sagemaker

Selection CriteriaEssential 

1. Demonstrate experience working in tight-knit Agile teams, providing data capabilities and services to technical and business stakeholders. Communicate well (both verbally and in writing), share knowledge and experiences and document requirements.

2. Have a strong knowledge or and demonstrate high-level experience in data analysis, problem solving and ETL development, including designing, building, testing and deployment of data workflows.

3. Demonstrate experience with the following technologies: o AWS Glue o AWS DMS o Apache Parquet o AWS Lambda o AWS Sagemaker o Opensearch/ElasticSearch o PySpark o Logstash o Iceberg

Desirable 

1. Experience in the following areas: o Database design o Oracle DBMSs, Elasticsearch DBMSs and Azure services o Data load/migration projects including data profiling and data quality analysis.

2. Use of Informatica IICS, PowerCenter, Developer, Analyst or MDM tools