Tech Lead

3 weeks ago

Dublin, Dublin City, Ireland The Citco Group Limited Full time

Job Description

Company Overview

Citco is a global leader in financial services, delivering innovative solutions to some of the world's largest institutional clients. We harness the power of data to drive operational efficiency and informed decision-making. We are looking for a Tech Lead - Data Engineering with extensive Databricks expertise and AWS experience to lead mission-critical data initiatives.

Role Summary

As the Tech Lead - Data Engineering, you will be responsible for architecting, implementing, and optimizing end-to-end data solutions on Databricks (Spark, Delta Lake, MLflow, etc.) while integrating with core AWS services (S3, Glue, Lambda, etc.). You will lead a technical team of data engineers, ensuring best practices in performance, security, and scalability. This role requires a deep, hands-on understanding of Databricks internals and a track record of delivering large-scale data platforms in a cloud environment.

Responsibilities

Key Responsibilities

  1. Databricks Platform & Architecture
    • Architect and maintain Databricks Lakehouse solutions using Delta Lake for ACID transactions and efficient data versioning.
    • Leverage Databricks SQL Analytics for interactive querying and report generation.
    • Manage cluster lifecycle (provisioning, sizing, scaling) and optimize Spark jobs for cost and performance.
    • Implement structured streaming pipelines for near real-time data ingestion and processing.
    • Configure and administer Databricks Repos, notebooks, and job scheduling/orchestration to streamline development workflows.
  2. AWS Cloud Integration
    • Integrate Databricks with AWS S3 as the primary data lake storage layer.
    • Design and implement ETL/ELT pipelines using AWS Glue catalog, AWS Lambda, and AWS Step Functions where needed.
    • Ensure proper networking configuration (VPC, security groups, private links) for secure and compliant data access.
    • Automate infrastructure deployment and scaling using AWS CloudFormation or Terraform.
  3. Data Pipeline & Workflow Management
    • Develop and maintain scalable, reusable ETL frameworks using Spark (Python/Scala).
    • Orchestrate complex workflows, applying CI/CD principles (Git-based version control, automated testing).
    • Implement Delta Live Tables or similar frameworks to handle real-time data ingestion and transformations.
    • Integrate with MLflow (if applicable) for experiment tracking and model versioning, ensuring data lineage and reproducibility.
  4. Performance Tuning & Optimization
    • Conduct advanced Spark job tuning (caching strategies, shuffle partitions, broadcast joins, memory optimization).
    • Fine-tune Databricks clusters (autoscaling policies, instance types) to manage cost without compromising performance.
    • Optimize I/O performance and concurrency for large-scale data sets.
  5. Security & Governance
    • Implement Unity Catalog or equivalent Databricks features for centralized governance, access control, and data lineage.
    • Ensure compliance with industry standards (e.g., GDPR, SOC, ISO) and internal security policies.
    • Apply IAM best practices across Databricks and AWS to enforce least-privilege access.
  6. Technical Leadership & Mentorship
    • Lead and mentor a team of data engineers, conducting code reviews, design reviews, and knowledge-sharing sessions.
    • Champion Agile or Scrum development practices, coordinating sprints and deliverables.
    • Serve as a primary technical liaison, working closely with product managers, data scientists, DevOps, and external stakeholders.
  7. Monitoring & Reliability
    • Configure observability solutions (e.g., Datadog, CloudWatch, Prometheus) to proactively identify performance bottlenecks.
    • Set up alerting mechanisms for latency, cost overruns, and cluster health.
    • Maintain SLAs and KPIs for data pipelines, ensuring robust data quality and reliability.
  8. Innovation & Continuous Improvement
    • Stay updated on Databricks roadmap and emerging data engineering trends (e.g., Photon, Lakehouse features).
    • Evaluate new tools and technologies, driving POCs to improve data platform capabilities.
    • Collaborate with business units to identify data-driven opportunities and craft solutions that align with strategic goals.
Qualifications

Qualifications
  1. Educational Background
    • Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or equivalent experience.
  2. Technical Experience
    • Databricks Expertise: 5+ years of hands-on Databricks (Spark) experience, with a focus on building and maintaining production-grade pipelines.
    • AWS Services: Proven track record with AWS S3, EC2, Glue, EMR, Lambda, Step Functions, and security best practices (IAM, VPC).
    • Programming Languages: Strong proficiency in Python (PySpark) or Scala; SQL for analytics and data modeling.
    • Data Warehousing & Modeling: Familiarity with RDBMS (e.g., Postgres, Redshift) and dimensional modeling techniques.
    • Infrastructure as Code: Hands-on experience using Terraform or AWS CloudFormation to manage cloud infrastructure.
    • Version Control & CI/CD: Git-based workflows (GitHub/GitLab), Jenkins or similar CI/CD tools for automated builds and deployments.
  3. Leadership & Soft Skills
    • Demonstrated experience leading a team of data engineers in a complex, high-traffic data environment.
    • Outstanding communication and stakeholder management skills, with the ability to translate technical jargon into business insights.
    • Adept at problem-solving, with a track record of quickly diagnosing and resolving data performance issues.
  4. Certifications (Preferred)
    • Databricks Certified Associate/Professional (e.g., Databricks Certified Professional Data Engineer).
    • AWS Solutions Architect (Associate or Professional).
#J-18808-Ljbffr

  • Tech Lead

    2 weeks ago

    Dublin, Dublin City, Ireland realTime Recruitment Full time

    Job Opening Tech Lead Permanent Dublin 14-06-2023 TECH LEAD ROLEOne of my clients is a company that helps large organizations sustain or extend their advantage through new innovations. They are looking for a Tech Lead with a view to progressing to a manager role in the future.Start-up environment working on a Fintech Application and the ideal...

  • Tech Lead

    4 weeks ago

    Dublin, Dublin City, Ireland TN Ireland Full time

    Social Network: Please specify the social network you want to login/join with.Position: Tech LeadOne of my clients is a company that helps large organizations sustain or extend their advantage through new innovations. They are looking for a Tech Lead with a view to progressing to a manager role in the future.Environment: Start-up environment working on a...

  • Tech Lead

    4 days ago

    Dublin, Dublin City, Ireland TN Ireland Full time

    One of my clients is a company that helps large organizations sustain or extend their advantage through new innovations. They are looking for a Tech Lead with a view to progressing to a manager role in the future.Start-up environment working on a Fintech Application and the ideal candidate will be 100% responsible for the tech.You need to love the start-up's...

  • Food Tech Sales Specialist

    4 days ago

    Dublin, Dublin City, Ireland Stelfox Tech Recruitment Full time

    Are you a highly motivated sales professional looking for a new challenge? Do you have a passion for building relationships and driving business growth?This is an excellent opportunity to join a leading company in the restaurant tech and food delivery space. As a sales development representative, you will play a key role in expanding our client's business...

  • Global Tech Product Lead

    7 days ago

    Dublin, Dublin City, Ireland Engineeringuk Full time

    We're seeking an experienced Global Tech Product Lead to join our Loss Prevention Tech team. As a key member of our team, you will be responsible for designing, developing, and maintaining software applications that support our field LP, security, and operations teams.About the Role:Lead cross-functional teams to develop and implement scalable global tech...

  • Pharmaceutical Validation and Tech Transfer Lead

    2 weeks ago

    Dublin, Dublin City, Ireland LSC, Life Science Consultants Full time

    Pharmaceutical Validation and Tech Transfer LeadAbout the Role:We are seeking a highly experienced Pharmaceutical Validation and Tech Transfer Lead to join our team at LSC, Life Science Consultants. As a key member of our Technical Services department, you will be responsible for leading the development and implementation of validation and tech transfer...

  • Tech Lead in DevOps Engineering

    7 days ago

    Dublin, Dublin City, Ireland Playrix Full time

    Playrix is a leading mobile game development company, with one of the most successful games in the world.Our teams work tirelessly to ensure that our games are available to millions of players every day, which requires a robust and efficient CI/CD process.We're looking for a Tech Lead to join our CI/CD team and help us take our process to the next...

  • Azure Tech Lead

    1 week ago

    Dublin, Dublin City, Ireland Tn Ireland Full time

    About the Role:This is an exciting opportunity to join our team in Tn Ireland as a Cloud Engineer – Tech Lead. You will be responsible for implementing and managing technical execution on the Azure cloud for our products.As a key member of our team, you will work closely with our Product, UI/UX, and Architecture teams to design and develop new...

  • Tech Transfer Lead

    1 day ago

    Dublin, Dublin City, Ireland PE Global Full time

    : PE Global is currently recruiting for a Tech Transfer Lead on behalf of a leading biotech company based in Dublin South. This is an initial 12-month contract role. This is a shift position. The shift patterns are as follows: 07:00 to 19:00, and 19:00 to 07:00. (4 on, 4 off). Overview: This role will require the candidate to provide process development...

  • Staff Software Engineer

    5 days ago

    Dublin, Dublin City, Ireland TN Ireland Full time

    Reliable and Scalable SolutionsStaff Software Engineer - Tech Lead for AnalyticsWe're seeking a staff software engineer to take on a tech lead role and drive the development and maintenance of our analytics platform. As a tech lead, you'll be responsible for defining the technical direction of the platform, ensuring seamless integration across various...

  • Tech Lead for Future Projects

    1 week ago

    Dublin, Dublin City, Ireland Contentful Full time

    Required Skills and QualificationsExtensive experience as a Senior Full Stack Software Engineer with substantial experience using TypeScript, React, and containerization technologies like Docker and Kubernetes.Demonstrable leadership skills, including experience in a senior role, such as a tech lead, performing responsibilities similar to this role.Solid...

  • Lawyer for Emerging Tech

    1 week ago

    Dublin, Dublin City, Ireland Hunter Savage Full time

    We are seeking a Lawyer for Emerging Tech to join our team at Hunter Savage. This role offers the opportunity to work with top-tier clients in fast-evolving industries and to engage in thought leadership on AI, blockchain, and emerging tech trends.ResponsibilitiesAdvising clients on technology contracts, regulatory compliance, intellectual property, and data...

  • Senior Tech Lead

    1 week ago

    Dublin, Dublin City, Ireland Wayflyer Limited Full time

    Job DescriptionAs a Senior Tech Lead - E-commerce Apps, you will be responsible for leading the development and maintenance of mobile features for commerce merchants. You will work with cross-functional teams to translate UI/UX designs into high-quality code, collaborate with backend developers to ensure a high-performance product, and act as a leading voice...

  • Tech Lead

    3 weeks ago

    Dublin, Dublin City, Ireland Apple Inc. Full time

    Do you love crafting elegant solutions to highly complex challenges? Can you intrinsically see the importance of every detail? At Apple, our compute team is responsible for designing and building the foundational pieces of our data center software. In this role, you will collaborate with engineers across Apple to build and deploy forward-looking...

  • Tech Law Specialist

    1 week ago

    Dublin, Dublin City, Ireland Hunter Savage Full time

    We are seeking a Tech Law Specialist to join our team at Hunter Savage. This is an exciting opportunity for a solicitor with a passion for technology law, data protection, and emerging digital trends to work with top-tier clients in fast-evolving industries.Key ResponsibilitiesAdvising clients on technology contracts, regulatory compliance, intellectual...

  • Senior Azure Engineer

    2 weeks ago

    Dublin, Dublin City, Ireland UnitedHealth Group Full time

    We're looking for a highly skilled Cloud Engineer – Tech Lead to join our team at UnitedHealth Group. As a Cloud Engineer – Tech Lead, you'll be responsible for implementing and managing technical execution on the Azure cloud for our small set of products.About the PositionThis position will focus on the development, implementation, and maintenance of...

  • Backend Tech Lead

    7 days ago

    Dublin, Dublin City, Ireland Playrix Full time

    Backend Tech Lead ) Playrix is in the global top 3 in terms of profit among mobile game developers.We have created the hit games Gardenscapes, Fishdom, Manor Matters, Homescapes, Wildscapes, and Township.They have been downloaded by more than two billion players, and have a monthly audience of 130 million people.We are looking for a Backend Tech Lead for our...

  • Devops Tech Lead

    7 days ago

    Dublin, Dublin City, Ireland Playrix Full time

    DevOps Tech Lead Playrix  is ??one of the most successful mobile game development companies in the world.In terms of mobile application revenue, we rank first in Europe and are in the top 3 worldwide.Our games have been installed 2.3 billion times (that's a third of the world's population)and more than  30 million people  around the world play them every...

  • Data Scientist Lead

    6 days ago

    Dublin, Dublin City, Ireland Stelfox Tech Recruitment Full time

    Role OverviewWe are seeking a highly skilled Data Scientist Lead to join our team at Stelfox Tech Recruitment. As a key member of our organization, you will be responsible for developing and overseeing the delivery of scalable machine learning pipelines and robust production models.

  • Public Cloud Delivery Lead

    1 week ago

    Dublin, Dublin City, Ireland European Tech Recruit Full time

    Job ProfileWe are looking for a Public Cloud Delivery Lead to join our team at Euro Tech Recruit. As a senior-level professional, you will be responsible for leading the delivery of infrastructure and new services for our global public cloud projects.Key Responsibilities: