Principal Technical Program Manager - Applied Science

AI evaluations areat a critical inflection point. Static benchmarks are saturated;benchmarks likeMMLU,HumanEval, and SWE-Benchhavereached their limitsas models become increasingly familiar with public test data, and capable ofautonomously finding answer keys online.The gap between benchmark scores and real developerexperienceis growing, making it hard tounderstand which problems are truly 'solved',and which are worthdeeper investment.

GitHub is uniquely positioned to lead the industry through this transition. We have direct feedback and deep insight into real production workflows from millions of developers, and the scale to build evaluation systems that truly reflect developer success. We're looking for a Principal Technical Program Manager to help us build the future of AI evaluation.

The Applied Science teamforGitHub Copilot sits at the intersection of frontier AI research and the world's largest developer platform. We ship AI-powered experiences(ex:code completion, code review,coding agents)used by millions of professional developers every day.As a member of the team, you willhelp leadGitHub Copilot's AI evaluation strategy end-to-end-from benchmark design and lifecycle governance, through evaluation infrastructure and internal adoption, to community engagement and public transparency. You are the person who ensures that everymodelswap,product harness,and feature launch is measured against whatactually mattersto developers - and that the world can see the results.

In this role you'll:

• Partner with Applied Science researchers to translatecutting-edgeevaluation research into production systems: adaptive testing (IRT), agent-centric co-evolution, adversarial benchmarking, and telemetry-driven benchmark generation.

• Lead the deprecation of saturated benchmarks and design their next-generation replacements - including procedurally-generated code evaluations thatcan'tbe memorized and adaptive testing systems that skip trivial questions for frontier models.

• Build GitHub's community benchmark submission program - enabling external researchers, enterprises, and open-source developers to contribute domain-specific evaluations - and publish GitHub's first external benchmark transparency reports showing how models perform on real developer workflows.

• Design and operationalize multi-tier evaluation frameworks - from fast automated regression suites and LLM-as-judge systems, through expert human evaluation, to production A/B testing - so teams can iterate in hours, not weeks.

• Design feedback-to-benchmark pipelines that convert thumbs-down signals, user frustrations, and support tickets into candidate regression tests - systematizing informal practices into scalable, automated systems.

• Establish evaluation as a first-class discipline across GitHub Copilot - creating the rituals, dashboards, and communication cadences that make evaluation results accessible and actionable for every team.

Required Qualifications

• Bachelor's Degree AND 6+ years experience in engineering, product/technical program management, data analysis, or product development OR equivalent experience.
• 3+ years of experience managing cross-functional and/or cross-team projects.

• Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years

Preferred Qualifications