Data Engineer

We are seeking an experienced Data Engineer to join our team and lead the design, development, and implementation of our Databricks data Lakehouse solution. This role involves setting up and managing data ingestion pipelines from various source systems, structuring data to meet the specifications required by our data scientists, and ensuring the scalability, performance, and security of our data platform. You will be responsible for employing the medallion architecture pattern and leveraging Azure technologies to support our data infrastructure.

You will have at least 5 years of experience in data engineering, proven experience with Databricks, and proficiency in Python, SQL, and Apache Spark. Excellent communication and problem-solving skills to collaborate effectively with the team are essential. You should have the willingness to learn new technologies to continuously improve data engineering practices.

This is a fully remote or hybrid role embedded with our client, a Swedish supply chain company.

Responsibilities

1. Design and Develop Data Solutions: Lead the design and implementation of scalable data Lakehouse solutions on Databricks, ensuring data is efficiently ingested, processed, and structured according to the medallion pattern (Bronze, Silver, Gold layers).
2. Data Pipeline Management: Build, manage, and optimise data ingestion pipelines using Azure Data Factory (ADF) and other relevant tools, ensuring the seamless integration of data from various source systems into Azure Data Lake Storage Gen2 (ADLS G2).
3. Data Structuring and Optimisation: Collaborate with data scientists and analysts to understand data requirements and ensure the data is structured in a way that meets their needs, providing clean and well-organised datasets.
4. Azure Ecosystem Expertise: Utilise your in-depth knowledge of Azure technologies such as Azure Databricks, ADLS G2, ADF, and Identity and Access Management (IAM) to design and maintain a robust data infrastructure.
5. Performance Tuning and Optimisation: Continuously monitor and optimise the performance of data pipelines and storage solutions, ensuring data processing is efficient and scalable.
6. Advanced Data Engineering Tools: Leverage advanced tools and technologies such as Python, Apache Spark, Unity Catalog, Delta Tables, and SQL to build and manage robust data solutions.
7. Security and Compliance: Implement best practices for data security and compliance, managing access controls, and ensuring that data is protected in accordance with organisational and regulatory requirements.
8. Documentation and Best Practices: Document technical solutions, data workflows, and architecture designs clearly and concisely, ensuring that the data engineering team adheres to best practices.
9. Troubleshooting and Support: Provide advanced troubleshooting for complex data issues, collaborating with stakeholders across the organisation to resolve challenges in data ingestion, processing, and storage.
10. Stay Current with Industry Trends: Keep up to date with the latest developments in data engineering, Azure technologies, and big data solutions to continually enhance the capabilities of our data platform.

Requirements

1. Education and Experience: At least 5 years of experience in data engineering, with a strong focus on building and managing data Lakehouse architectures on Microsoft Azure.
2. Technical Expertise: Proven experience with Databricks, Azure Data Factory, Azure Data Lake Storage Gen2, and other Azure services. Strong knowledge of IAM (Identity and Access Management) in Azure environments.
3. Data Architecture Knowledge: Expertise in data architecture patterns, especially the medallion architecture, and experience with designing data solutions that are both scalable and performant.
4. Programming and Scripting: Proficiency in Python, SQL, and Apache Spark, with a deep understanding of Unity Catalog and Delta Tables.
5. Communication Skills: Excellent communication skills, with the ability to articulate complex technical concepts to both technical and non-technical stakeholders.
6. Problem-Solving Skills: Strong analytical and problem-solving skills, with the ability to troubleshoot complex data-related issues and implement effective solutions.
7. Team Collaboration: Ability to work effectively as part of a cross-functional team, providing technical guidance and mentorship to junior engineers.
8. Adaptability and Learning: Willingness to learn new technologies and stay abreast of industry trends to continuously improve data engineering practices.

Sounds interesting? We are excited to get to know you!

If you have any questions you would like to ask or if there is any additional information you would like to receive, please feel free to get in touch via [email protected].