Senior Principal Engineer - Applications Development

Company:

Marsh Risk

Description:

 

POSITION OVERVIEW

We are seeking a Dev Ops to design and implementation of a comprehensive, verifier-gated agentic software delivery platform. This platform will orchestrate the complete enterprise software lifecycle—from requirements management through production verification—with AI agents accelerating work while independent verifiers and deterministic controls ensure compliance, security, and governance. This is a hands-on technical leadership role requiring expertise in CI/CD architecture, security integration, policy-as-code, release management, and enterprise orchestration.

 

KEY RESPONSIBILITIES

1.       Design and implement the end-to-end verifier-gated software delivery architecture, spanning requirements intake, test engineering, development, verification, release approval, deployment, and runtime evidence collection.

2.       Lead Excel-to-Azure DevOps (ADO) ingestion pipeline design: define canonical schema, implement idempotent synchronization, handle hierarchy mapping, and ensure source-of-truth integrity.

3.       Architect requirements verification and enrichment system: design quality gates for completeness, testability, security requirements, data classification, and compliance.

4.       Build test engineering pipeline: design test generation from acceptance criteria, implement test-case verification gates, ensure coverage metrics, and establish traceability from story → AC → test → code.

5.       Implement independent verifier gates for software quality, security (Snyk, SAST/SCA, secrets, IaC), architecture (policy-as-code, standards compliance), and enterprise policy (logging, AI governance, data controls).

6.       Design and implement the release evidence package assembly: define required evidence domains, implement evidence hashing, build release-readiness assessment, and integrate risk/confidence scoring engine.

7.       Architect the approval workflow orchestration: design approval matrix (change type/risk tier/environment), implement routing logic, build decision-brief generation, and manage approval token lifecycle.

8.       Lead secure deployment authorization: design OIDC/workload identity integration, implement artifact signing/verification, manage token validation, and enforce environment promotion controls.

9.       Architect runtime verification and feedback system: design production-signal correlation to release ID, implement health-check automation, integrate SIEM/telemetry, and automate rollback/incident/backlog actions.

10.   Define the Enterprise Skills Registry: structure security, identity, architecture, observability, AI governance, and release-management skills for local and CI/CD verification.

11.   Lead agent orchestration framework design: define agent contracts, least-privilege permissions, separation of duties (creator vs verifier), and bounded remediation scopes.

12.   Establish observability and audit infrastructure: design correlation IDs/workflow tracking, implement metrics for gates/approvals/deployments, and ensure full auditability.

13.   Lead architecture decisions on tooling: ADO process templates, GitHub Actions vs Azure Pipelines orchestration, evidence-store implementation, policy-as-code technology, and artifact signing standards.

 

REQUIRED TECHNICAL SKILLS & EXPERTISE

Skill Domain

Specific Requirements

Enterprise CI/CD Architecture

7+ years with Jenkins, Azure Pipelines, GitHub Actions, or Spinnaker. Deep experience with multi-stage pipelines, artifact management, SBOM generation, and immutable release candidates.

Azure DevOps Mastery

Expert-level ADO: work-item hierarchy, custom fields, REST API, Test Plans, release management, webhook orchestration, idempotent sync patterns. 5+ years hands-on.

Security & Policy-as-Code

Hands-on experience with Snyk, SAST/SCA tools, secrets management (HashiCorp Vault, Azure Key Vault), OPA/Gatekeeper, policy-as-code frameworks. Real production security gate design.

Workflow Orchestration & State Machines

Designing complex, stateful workflows with deterministic controls. Experience with approval routing, conditional branching, retry logic, escalation. Not just sequential pipelines.

Evidence & Audit Architecture

Design immutable evidence stores, traceability chains, hashing/signing, compliance audit trails. Familiar with evidence correlation patterns.

AI/Agent Governance

Understanding of agentic systems, least-privilege agent permissions, separation of duties (creator ≠ verifier), bounded remediation scopes. AI security and governance requirements.

Cloud & Identity (AWS/Azure/GCP)

OIDC/workload identity, IAM policy-as-code, secrets rotation, vault integration. Multi-cloud architecture experience.

Observability & Correlation

Designing release-correlated telemetry, SIEM integration, distributed tracing, incident correlation. Experience with observability platforms (DataDog, Splunk, Azure Monitor).

 

EXPERIENCE LEVEL & SENIORITY

·       8+ years total in software delivery, CI/CD, or DevOps engineering

·       8+ years designing enterprise-scale CI/CD/release pipelines

·       5+ years with policy-as-code, security verification gates, or compliance automation

·       3+ years with Azure DevOps or GitHub Enterprise (not just Git, but orchestration/work-item management)

·       Demonstrable experience leading architectural decisions on complex, multi-team platforms

·       Track record of building verifier-gated or policy-driven delivery systems

·       Experience with agentic/AI-accelerated workflows (helpful but can be learned on job)

 

KEY DELIVERABLES (9 months)

14.   Excel intake schema, validator, and idempotent ADO sync service (Phase 1: Month 1)

15.   Requirements verification and enrichment agent architecture (Phase 2: Month 2-3)

16.   Test generation and verification gate implementation (Phase 2: Month 2-3)

17.   Enterprise Skills Registry structure and VS Code integration framework (Phase 2: Month 2-3)

18.   Independent verifier gates: quality, security, architecture, policy (Phase 4: Month 4-5)

19.   Release evidence package and risk/confidence scoring engine (Phase 4: Month 4-5)

20.   Approval orchestration workflow and token lifecycle (Phase 5: Month 6-7)

21.   Secure deployment authorization with OIDC and token validation (Phase 5: Month 6-7)

22.   Runtime verification, telemetry correlation, and incident automation (Phase 6: Month 8-9)

23.   Full platform integration and calibration on pilot team (Phase 6: Month 8-9)

 

SUCCESS CRITERIA

24.   Excel intake seamlessly creates/updates ADO stories without duplicates; idempotency proven over 1000+ test items

25.   All user stories have versioned acceptance criteria and requirement-verifier outcomes before development

26.   Generated test cases trace to acceptance criteria; coverage reaches >90% of mandatory criteria

27.   Mandatory verifier failures block deployments; confidence scores cannot override policy blockers

28.   Remediation is independently re-verified by different system/verifier

29.   Release evidence package is complete, hashable, and immutable for every release

30.   Approval routing follows configured matrix; <24hr approval SLA for low-risk changes

31.   Approval tokens bind commit, artifact, evidence, and policy versions; stale-approval rules enforced

32.   Deployment rejects missing/invalid tokens; zero unauthorized deployments to protected environments

33.   Production telemetry correlates to release ID; rollback/incident automation triggered for critical signals

34.   Platform supports 5+ teams, 10+ applications, 100+ releases/week at launch

35.   Pilot team shows measurable improvement: change-failure rate reduction, security incident reduction, mean-time-to-deployment improvement

 

PREFERRED QUALIFICATIONS

·       Published work on CI/CD architecture, release engineering, or supply-chain security

·       Open-source contributions to CI/CD/DevOps tooling (Jenkins plugins, Terraform providers, etc.)

·       Experience with CNCF projects (Tekton, Flux, ArgoCD) or GitOps platforms

·       Background in regulated industries (financial services, healthcare, government) with compliance/audit requirements

·       Familiarity with software composition analysis (SBOM, provenance, attestation standards)

·       Public speaking at conferences on DevOps, security, or release engineering

·       Mentorship of junior engineers or architectural guidance on team transformations

·       Experience with LLM-based AI agents or agentic workflows

·       Certification in enterprise architecture, security, or cloud (AWS SA-Pro, AZ-305, CISSP, etc.)

 

SOFT SKILLS & WORKING STYLE

·       Strong technical communication: can explain complex architecture to executives, architects, developers, and operators

·       Collaborative but opinionated: drives architectural decisions while incorporating feedback from security, compliance, and operations teams

·       Evidence-driven thinking: designs for measurable outcomes; comfortable with instrumentation and metrics

·       Systems thinking: understands end-to-end workflow; sees tradeoffs between automation, governance, and developer velocity

·       Ownership mindset: takes accountability for platform reliability and user adoption

·       Adaptability: comfortable with emergent requirements and continuous calibration of controls

·       Documentation discipline: writes clear design documents, architectural decision records (ADRs), and runbooks

 

 

Marsh (NYSE: MRSH) is a global leader in risk, reinsurance and capital, people and investments, and management consulting, advising clients in 130 countries. With annual revenue of over $27 billion and more than 95,000 colleagues, Marsh helps build the confidence to thrive through the power of perspective. For more information, visit marsh.com, or follow us on LinkedIn and X.

Marsh is committed to embracing a diverse, inclusive and flexible work environment. We aim to attract and retain the best people and embrace diversity of age, background, caste, disability, ethnic origin, family duties, gender orientation or expression, gender reassignment, marital status, nationality, parental status, personal or social status, political affiliation, race, religion and beliefs, sex/gender, sexual orientation or expression, skin color, or any other characteristic protected by applicable law.

Marsh is committed to hybrid work, which includes the flexibility of working remotely and the collaboration, connections and professional development benefits of working together in the office. All Marsh colleagues are expected to be in their local office or working onsite with clients at least three days per week. Office-based teams will identify at least one “anchor day” per week on which their full team will be together in person.