Senior Specialist - Data Engineering (Contract- For Immediate Hire)

JOB DESCRIPTION SUMMARY

This role will be responsible for designing, developing, and maintaining scalable data pipelines, ensuring data flow across systems, and optimizing data infrastructure. You will collaborate with cross-functional teams, assist with production support, and drive process optimizations to ensure a seamless data ecosystem for business operations. Strong Databricks experience is a must.

KEY DUTIES & RESPONSIBILITIES

1.Data Pipeline Development: Design, build, and maintain robust data pipelines using Azure Data Factory (ADF), Azure Databricks, ADLS Gen 2 and PySpark, ensuring efficient data collection, processing, and storage from various sources.

2.DataBricks Expertise: Have extensive experience working with DataBricks; including resource provisioning, understanding tools and modules available, and being able to share knowledge and expertise in DataBricks to contribute to optimization of this platform.

3.Data Integration: Integrate data from multiple structured and unstructured sources, including APIs, third-party systems, and databases (SQL, Oracle), ensuring smooth data flow across the ecosystem.

4.Data Warehousing: Design, develop, and optimize data lakes and warehouses on platforms like Databricks, Azure Synapse and ADLS, supporting analytics and reporting needs.

5.ETL/ELT Development: Develop and optimize ETL processes for efficient data extraction, transformation, and loading, using tools like ADF and custom pipelines.

6.Process Optimization: Automate and streamline data workflows (incremental loads, email notifications, pipeline runs) to enhance system performance and reduce costs.

7.Production Support & Resolution: Troubleshoot and resolve production failures within SLAs, ensuring minimal downtime and high data availability. Provide support during business hours and extended hours, (and if critical P1 failures on weekends in rotation with others), as required.

8.Go-Live & Project Support: Support the successful migration and implementation of new code and solutions. Utilizing Azure DevOps CI/CD.

9.Team Management: Lead and manage vendor staff, ensuring smooth execution of projects or operations support as assigned. Ensuring SLAs or project milestones are met on time, and providing mentorship to junior team members.

10.Innovation & Optimization: Design scalable, high-performance data architectures and promote innovation, cost-efficiency, and operational excellence.

11. Documentation & Knowledge Transfer: Create and maintain clear documentation for processes, operations, and technical solutions, including knowledge transfer documents and data catalogue updates.

12.Monitoring & System Oversight: Monitor system performance, identify opportunities for optimization, and ensure platform scalability to meet future needs.

JOB REQUIREMENTS

Education & Skills (Required, Preferred)

1.Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field Required

2. Proven experience working with cloud-based infrastructure, particularly Azure. Required

3.Proven experience working with DataBricks. Required

4.Experience with containerization tools (Docker, Kubernetes) and CI/CD pipelines. Required

5.Knowledge of machine learning workflows and model deployment. Preferred

6.Proficient in English Required

7.Proficient in Arabic Preferred

Experience & Knowledge (Required, Preferred)

1.6+ years of experience in data engineering, production support, and pipeline management. Required

2.Cloud Platforms: ADLS Gen 2, Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and DevOps, Azure Key Vault. Required

3.Programming/Querying: Python, PySpark, SQL. Required

4.Automation & Monitoring: Custom automation tools, failure email triggers, performance monitoring solutions. Required

5.Experience in project management. Preferred

6.Excellent leadership, communication, and problem-solving skills. Preferred

Abilities & Specific Requirements (Required, Preferred)

1.Able to deal with varied tasks and to work under pressure to meet deadlines and targets Required

2.Able to work effectively and responsibly without close supervision Required

3.Able to negotiate with all types of vendors Required

4.Capable of agile thinking to provide fast, accurate answers to questions Preferred

5.Applies a structured and systematic approach to problems Preferred

6.Demonstrates critical thinking and problem-solving capabilities Preferred

7.Able to foster a culture of collaboration, innovation, transparency, and accountability Preferred

8.Able to execute clear strategies and corporate requirements Required

9.Able to articulate ideas to both technical and non-technical audiences Required

10.Professional demeanor, leads by example Required

11.Demonstrates ability to manage change and foster teamwork Required