Director, Engineering AI Evaluation & Quality

Your Team's Impact 

AI has evolved from an isolated feature into a capability that spans our entire platform. The engineering infrastructure required to measure, monitor, and validate it must now evolve as well.

Today, teams instrument AI systems independently, with no shared visibility when something behaves unexpectedly in production and no common framework for validating changes before they ship. You will build that foundation.

You will deploy and operate LangSmith in our own infrastructure, develop the evaluation tooling on top of it, and own the observability layer for AI systems running across FactSet. When an AI workflow fails in a client-facing context, your platform is how the organization detects it — and proves it has been resolved.

This is a founding engineering role. You will inherit a small team and be responsible for consolidating organically developed evaluation solutions into a single, authoritative capability. Adoption will be earned, not mandated — your platform's influence across the organization will grow through the value it demonstrably delivers.

What You Will Own

LangSmith (Self-Hosted):  Own the full lifecycle — deployment, operations, upgrades, and security. This includes Kubernetes infrastructure, datastores, SSO, access control, data isolation, and capacity management. Lead the vendor relationship and partner with Cloud and Central Technology to ensure compliant, secure hosting of client-adjacent evaluation data.

Evaluation Tooling:  Build the tooling that makes the platform practical — eval harnesses for offline and production traffic, LLM-as-judge scoring pipelines, versioned golden datasets, and regression suites that integrate into existing CI pipelines. Your product counterpart defines what gets measured; you build what does the measuring.

AI Production Observability:  Own tracing, metrics, and alerting across AI systems in production covering latency, cost, model and prompt versioning, retrieval behavior, and drift detection. This layer transforms a client escalation into a diagnosable, reproducible event.

Production Reliability:  The team owns the pager for what it builds. You set the operational standard with a first-year objective of building a sustainable on-call rotation as adoption scales.

Technical Leadership:  You are stepping into a hands-on technical leadership role, not a pure management position. You will inherit a team of 4 engineers today, with the expectation of growing to 8–10 as the platform scales and proves its value. Beyond people leadership, you will drive architectural decisions, review critical code and design changes, and hire deliberately -bringing on the right people at the right time rather than growing for the sake of it.

What We're Looking For :

Minimum Requirements:

• 10+ years of production software experience, with significant time on platform or infrastructure teams that other engineers depended on
• Hands-on technical leadership: you have led small teams while staying in the code, and you prefer it that way
• Strong Python proficiency is required. The LLM evaluation and observability ecosystem is Python-native. 

Critical Skills:

• Platform track record: you have built internal platforms that teams chose to adopt, and you understand why the ones that failed were slower than what they replaced
• LLM production experience: you can reason about tracing, retrieval behavior, prompt and model versioning, and regression analysis; familiarity with tooling such as LangSmith, LangFuse, Braintrust, or Weights & Biases is a plus
• Kubernetes and cloud infrastructure depth, including operating relational and analytical datastores under real load
• Self-hosted platform operations: you have run stateful third-party platforms in production, including upgrades, data migration, capacity planning, and security and compliance requirements
• Operational judgment: you have established on-call practices on small teams and know how to keep them sustainable
• Cross-functional influence: you can operate as a peer to product leadership and drive alignment across distributed engineering teams without direct authority

Nice to Have:

• Experience in financial services or capital markets
• Familiarity with regulated or audited production environments
• Exposure to data residency and tenant isolation requirements

What's In It For You

At FactSet, our people are our greatest asset, and our culture is our biggest competitive advantage. Being a FactSetter means:

• The opportunity to join an S&P 500 company with over 45 years of sustainable growth, powered by the entrepreneurial spirit of a start-up.
• Support for your total well-being. This includes health, life, and disability insurance, as well as retirement savings plans and a discounted employee stock purchase program, plus paid time off for holidays, family leave, and company-wide wellness days.
• Flexible work accommodations. We value work/life harmony and offer our employees a range of accommodations to help them achieve success both at work and in their personal lives.
• A global community dedicated to volunteerism and sustainability, where collaboration is always encouraged, and individuality drives solutions.
• Career progression planning with dedicated time each month for learning and development.
• Business Resource Groups open to all employees that serve as a catalyst for connection, growth, and belonging.

Learn more about our benefits here.

Salary is just one component of our compensation package and is based on several factors, including but not limited to education, work experience, and certifications.