Site Reliability ArchitectCompany: IncedoHybridAustin, TX, USAThis job is currently
not accepting new applications
not accepting new applications
Site Reliability Engineer (SRE Architect) – Unified Observability & AIOps
Location- Austin, TX
Role Summary
We are seeking a Senior SRE with strong expertise in Unified Observability, proactive detection, AIOps, and GenAI-driven operations to support complex, distributed financial services platforms. The role requires hands-on experience designing SLI/SLO-driven monitoring, dynamic thresholds, intelligent alerting, and AI/ML-based anomaly detection across multi-stream architectures.
Key Responsibilities
Observability & Reliability Engineering
- Design and implement unified observability dashboards across metrics, logs, traces, events, and topology
- Define and manage SLIs, SLOs, and error budgets aligned to business outcomes
- Build actionable dashboards for operations, engineering, and leadership
- Implement alerting strategies using static and dynamic thresholds
Proactive Detection & AIOps
- Leverage AI/ML/AIOps to detect anomalies, predict incidents, and reduce MTTR
- Transition monitoring from reactive alerts to proactive insights
- Implement noise reduction, alert correlation, and root cause analysis
- Apply baseline modeling, seasonality detection, and anomaly scoring
Distributed Systems & Dependency Analysis
- Monitor and troubleshoot multi-service architectures involving:
- Microservices
- Downstream APIs
- Kafka / streaming platforms
- Cloud infrastructure (Terraform, IaC)
- Identify whether issues originate from:
- Upstream/downstream dependencies
- Streaming platform
- Infrastructure
- Application code
Tooling & Platforms
- Deep hands-on experience with Dynatrace (mandatory)
- Experience with:
- OpenTelemetry
- Prometheus / Grafana
- ELK / EFK
- Cloud-native monitoring (AWS/Azure/GCP)
- Strong JSON-based telemetry manipulation and enrichment
GenAI & LLM Enablement
- Apply GenAI / LLMs for:
- Incident summarization
- Root cause explanation
- Runbook recommendations
- Auto-remediation suggestions
- Collaborate with platform teams to operationalize GenAI safely
Required Skills & Experience
✅ 15+ years in SRE / Production Engineering
✅ Strong Unified Observability background (not infra-only)
✅ Hands-on Dynatrace experience (metrics, traces, logs, Davis AI)
✅ SLI/SLO engineering experience in production systems
✅ Experience implementing dynamic thresholds and anomaly detection
✅ Knowledge of AI/ML concepts applied to Ops (AIOps)
✅ Distributed systems troubleshooting expertise
✅ Experience with Kafka or streaming data platforms
Differentiators (Highly Valued)
- Experience in financial services or regulated environments
- Proven reduction of alert noise and MTTR using AIOps
- GenAI / LLM integration into operations workflows
Interview Question Bank (Mapped to LPL Expectations)
1. Dashboards, SLAs, and Reliability Targets
Purpose: Identify true SREs vs dashboard builders
- How do you design dashboards differently for engineers vs leadership?
- Explain how SLIs and SLOs differ from SLAs. Which do you operationalize?
- How do you map SLOs to alerting without creating noise?
- What KPIs would you track for a critical trading or advisor-facing platform?
Red Flag: Talks only about CPU, memory, uptime
2. Alerting Strategy & Threshold Design
Purpose: Assess signal-to-noise maturity
- How do you decide when to use static vs dynamic thresholds?
- Explain how you prevent alert storms during high traffic or seasonal spikes.
- What makes an alert actionable?
- How do you design alerts for early symptom detection?
Follow-up
- What happens after an alert fires? Walk me through the lifecycle.
3. Dynamic Thresholds & Anomaly Detection
Purpose: Validate AIOps fundamentals
- How do dynamic thresholds work under the hood?
- How do you account for baseline drift and seasonality?
- What risks do dynamic thresholds introduce?
- How would you tune sensitivity to avoid false positives?
Expected Concepts ✅ Baselines
✅ ML models
✅ Adaptive learning
✅ Time-series analysis
4. Multiplexing (Metrics, Signals, Streams)
Purpose: Test system observability depth
- What is multiplexing in observability?
- How do multiple telemetry signals strengthen diagnosis?
- Provide an example where one signal was misleading.
- How do you correlate metrics, traces, logs, and events?
5. JSON Tooling & Proactive Detection
Purpose: Ensure hands-on operational telemetry skills
- How have you used JSON-based event payloads to enrich observability?
- How do you normalize data across heterogeneous sources?
- How do structured logs improve proactive detection?
- How do you extract signals from high-volume telemetry?
6. Proactive vs Reactive Detection
Purpose: Directly aligned to LPL concern
- Give an example where you predicted an incident before customer impact.
- What indicators help you identify impending failures?
- How do you measure the success of proactive detection?
7. Multi-Service Failure Diagnosis (Critical Question)
Purpose: Core differentiator at LPL
Scenario Question
A user-facing issue is reported. The architecture includes:
- Frontend
- Backend microservices
- Downstream APIs
- Kafka streams
- Terraform-managed infrastructure
Ask:
- How do you determine if the issue is:
- Application-related?
- Kafka or streaming lag?
- Downstream API latency?
- Infrastructure drift via Terraform?
Expected Approach ✅ Dependency mapping
✅ Golden signals
✅ Trace correlation
✅ Change analysis
8. Dynatrace (Mandatory)
Purpose: Address explicit gap in feedback
- What Dynatrace features have you used most?
- How does Davis AI determine root cause?
- How do you implement service-level baselining in Dynatrace?
- How do you reduce alert noise using Dynatrace?
Red Flag: “I’ve mostly used dashboards”
9. AI/ML & AIOps Fundamentals
Purpose: Ensure non-theoretical knowledge
- What ML techniques are commonly used in AIOps?
- How do supervised vs unsupervised models differ in Ops?
- Where does AI fail in observability?
- How do you validate AI-based decisions?
10. GenAI & LLM Use Cases for SRE
Purpose: Explicit LPL requirement
- Where do you see GenAI adding value in SRE?
- Have you used LLMs for incident response?
- How would you integrate GenAI without introducing risk?
- What data would you restrict from LLM exposure?
Expected Use Cases ✅ Incident summarization
✅ RCA explanation
✅ Runbook suggestions
✅ MTTR reduction