Principal Engineer – Distributed Systems Architecture

Job Description:

We are looking for a highly experienced Technical Architect to lead the design, strategy, and implementation of observability and SRE frameworks for enterprise-scale, microservices-based applications. The ideal candidate will bring deep technical knowledge of both Splunk Observability Stack and Open Source tools (like OpenTelemetry, Prometheus, Grafana, Jaeger), and be capable of defining and executing architecture strategies for complex distributed systems.

This role requires hands-on ability to create architecture blueprints, lead technical teams, and work directly with stakeholders and platform owners to embed observability and reliability practices across the SDLC.

Key Responsibilities:

• Design and deliver end-to-end observability architecture (Metrics, Logs, Traces, Events) for cloud-native and hybrid environments.
• Create technical architecture diagrams, data flow maps, and integration blueprints using tools like Lucidchart or Visio.
• Lead the definition of SLIs, SLOs, and Error Budgets aligned with business KPIs and DORA metrics.
• Architect telemetry pipelines using OpenTelemetry Collector and Splunk Observability Cloud (SignalFx, APM, RUM, Log Observer).
• Define tool adoption strategy and integration roadmap for OSS tools (Prometheus, Loki, Grafana, Jaeger) and Splunk-based stacks.
• Reliability Engineering Enablement
• Lead adoption of SRE principles including incident management frameworks, resiliency testing, and runbook automation.
• Collaborate with DevOps to integrate observability into CI/CD pipelines (e.g., Jenkins, ArgoCD, GitHub Actions).
• Define health checks, golden signals, and SPoG dashboards.
• Exposure to AIOps, ML-based anomaly detection, or business observability.
• Stakeholder Management & Governance
• Serve as a technical liaison between client leadership, SREs, developers, and infrastructure teams.
• Run workshops, assessments, and evangelize observability-first culture across teams.
• Provide guidance on data retention, access control, cost optimization, and compliance (especially with Splunk ingestion policies).
• Performance & Optimization
• Continuously monitor and fine-tune observability data flows to prevent alert fatigue and ensure actionability.
• Implement root cause analysis practices using telemetry correlation across metrics, logs, and traces.
• Lead efforts to build self-healing systems using automated playbooks and AIOps integrations (where applicable).

• 15+ years in IT, with 5 years in Observability/SRE architecture roles
• Proven experience designing architecture for microservices, containers (Docker, Kubernetes), and distributed systems
• Strong hands-on expertise with:
• Splunk Observability Cloud (SignalFx, Log Observer, APM)
• OpenTelemetry (SDKs + Collector)
• Jaeger / Zipkin for distributed tracing
• CI/CD tools: Jenkins, GitHub Actions, ArgoCD
• Ability to build and present clear architecture diagrams and solution roadmaps
• Working knowledge of cloud environments (AWS, Azure, GCP) and container orchestration (K8s/OpenShift)
• Familiarity with SRE and DevOps best practices (error budgets, release engineering, chaos testing)

The client is on a journey to mature its Observability and SRE ecosystem, and this role will be critical in unifying legacy and modern telemetry stacks, driving reliability-first mindset and tooling, and establishing a scalable blueprint for production excellence.

As a trusted global innovator of business and technology services, we serve top clients worldwide and commit to helping them innovate, optimize and transform for long term success. Our diverse experts operate in more than 50 countries and partner with established and start-up companies to drive digital innovation. We are one of the leading providers of digital and AI infrastructure in the world. Our commitment to R&D enables us to help organizations move confidently and sustainably into the digital future.