As a Lead Data Engineer, you will play a critical role in building and optimizing data infrastructure to support analytics and machine learning initiatives. You will design scalable data pipelines, develop ETL workflows, and ensure the efficient integration of data across various platforms using industry-standard technologies and tools. Your work will enable cross-functional teams to derive insights and make data-driven decisions.Beyond technical expertise, you will also be expected to take on a leadership role, mentoring team members, shaping technical direction, and fostering collaboration across the organization.
Responsibilities:
- Design, build, and maintain scalable data pipelines and workflows using multiple Cloud services and tools
- Collaborate with data scientists, machine learning engineers, and business stakeholders to understand data requirements and deliver appropriate solutions.
- Optimize data storage solutions and implement best practices for data governance, security, and performance.
- Implement Python-based solutions for data processing and analysis.
- Build and refine CI/CD processes to improve data workflows and ensure seamless deployments.
- Monitor and troubleshoot data pipelines to ensure reliability and minimize downtime.
- Provide technical leadership for the data engineering team, guiding architectural decisions and best practices.
- Mentor and coach other data engineers, fostering a culture of continuous learning and improvement.
- Drive cross-functional collaboration, ensuring alignment between engineering, product, and business goals.
- Influence strategic decisions related to data platforms and analytics capabilities.
Required Skills and Expertise:
- Bachelor's or Master's degree in Data Engineering, Computer Science or a related field (or equivalent practical experience).
- 5+ years of experience in designing and implementing data pipelines and ETL workflows.
- Proven track record of delivering production-grade software.
- Strong problem-solving skills and attention to detail.
- Excellent English communication.
- Strong problem-solving and analytical skills, with the ability to work collaboratively in a team environment.
- Proven ability to lead projects and teams, including mentoring other engineers and driving alignment across functions.
- Strong experience with Python programming language.
- Exceptional SQL proficiency for querying and managing complex data structures.
- Deep experience with Databricks for managing and optimizing large-scale data systems.
- Strong experience with at least one cloud service (AWS, Azure, or GCP)
- Deep understanding of database systems, data warehousing, and data modeling techniques.
- Familiarity with distributed computing and Big Data frameworks like Apache Spark or Hadoop.
- Demonstrated leadership experience in guiding teams and projects while working with software engineer teams and business teams.
- Excellent communication and collaboration skills, with the ability to influence and inspire both technical and non-technical stakeholders.
Preferred Experience:
- Experience in the transportation and logistics industry.
- Knowledge of streaming data pipelines and real-time data processing.
- Familiarity with MLOps workflows and supporting machine learning teams.
- Experience with data visualization tools like Power BI or Tableau.