Highly Skilled Big Data Specialist

Senior Data Engineer Role

As a Senior Data Engineer, you will play a pivotal role in managing and optimizing the Databricks Platform on AWS. Your responsibilities will include setting up, configuring, administering, and ensuring the scalability and performance of secure Databricks environments.

This role requires close collaboration with engineering, infrastructure, and compliance teams to guarantee that the Databricks platform meets enterprise data and regulatory requirements.

Key Responsibilities:

• Provision, configure, and manage Databricks Workspaces across multiple environments.
• Define and enforce cluster policies, compute tagging, and usage controls to optimize cost and resource utilization.
• Administer Unity Catalog, including creation and management of catalogs, schemas, access control, and data lineage.
• Manage SCIM integrations for user/group provisioning with IdPs like Okta or Azure/AWS SSO.
• Configure and maintain CI/CD pipelines for notebook deployment and workflow promotion using GitHub, GitLab, or similar tools.

Security, Governance & Compliance:

• Implement and enforce role-based access controls (RBAC) and fine-grained data permissions using Unity Catalog and Lake Formation.
• Ensure auditability and lineage tracking for compliance with Open Banking, GDPR, and PSD2 regulations.
• Configure and manage token scopes, secrets management, and credential passthrough for secure access to underlying data.
• Work with InfoSec and compliance teams to ensure platform security aligns with regulatory frameworks.

Monitoring, Support & Troubleshooting:

• Monitor workspace performance, cluster health, job execution, and user activity using Databricks native tools, CloudWatch, and third-party integrations.
• Provide L2/L3 support for Databricks usage issues, including notebook errors, job failures, and workspace anomalies.
• Maintain operational runbooks, automation scripts, and alerting mechanisms for platform health and governance events.

Automation & Best Practices:

• Build and manage Terraform/IaC scripts for environment provisioning and infrastructure consistency.
• Define naming conventions, resource tagging standards, and workspace governance guidelines.
• Promote standardization of reusable notebook templates, shared cluster configurations, and workflow orchestrations.
• Drive internal knowledge sharing, onboarding, and platform enablement sessions.

Must-have Skills:

• 6+ years of experience in Databricks administration on AWS or multi-cloud environments.
• Deep understanding of Databricks workspace architecture, Unity Catalog, and cluster configuration best practices.
• Strong experience in managing IAM policies, SCIM integration, and access provisioning workflows.
• Hands-on experience with monitoring, cost optimization, and governance of large-scale Databricks deployments.
• Hands-on experience with infrastructure-as-code (Terraform) and CI/CD pipelines.
• Experience with ETL orchestration and collaboration with engineering teams (Databricks Jobs, Workflows, Airflow).

Good-to-Have Skills:

• Understanding of Delta Lake, Lakehouse architecture, and data governance patterns.
• Experience in AWS Glue and database technologies SQL (Amazon Aurora) and NoSQL (MongoDB/DynamoDB).
• Exposure to regulated industries such as banking, fintech, or healthcare.
• Experience with incident response, audit logging, and security remediation.

Preferred Certifications:

• Databricks Certified Data Engineer Associate/Professional
• Databricks Certified Lakehouse Platform Administrator
• AWS Certified Solutions Architect – Associate

Corporate Security Responsibility:

All activities involving access to Mastercard assets, information, and networks come with an inherent risk to the organization. It is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must abide by Mastercard's security policies and practices; ensure the confidentiality and integrity of the information being accessed; report any suspected information security violation or breach; and complete all periodic mandatory security trainings in accordance with Mastercard's guidelines.

Seniority level:

• Mid-Senior level

Employment type:

• Full-time

Job function:

• Information Technology

Industries:

• Financial Services, IT Services and IT Consulting, and Technology, Information and Internet

---