Engineering Manager - Machine Learning, Model Evaluations and Data Curation (AI Foundations)
Netflix is one of the world's leading entertainment services, with over 300 million paid memberships in over 190 countries enjoying TV series, films and games across a wide variety of genres and languages. Members can play, pause and resume watching as much as they want, anytime, anywhere, and can change their plans at any time.
The Opportunity
Fast-paced innovation in large language models (LLMs) and generative AI is reshaping personalization and discovery at Netflix. Our current bet is that an in-house LLM, customized on Netflix data, will be the cornerstone of this transformation. Model Evaluations and Data Curation (“Evals & Data”) are central to the development of LLMs and other high leverage foundation models at Netflix. Our Evals & Data team builds the benchmarks, evaluators, and baselines that guide LLM progress, and the data infrastructure that delivers high-quality, reproducible training and evaluation datasets to our AI/ML researchers. Together, these capabilities create the flywheel of continuous improvement between data, evals and modeling that drives innovation in LLMs that will power personalization and discovery in the future. We are incubating a centralized, first-class evaluation discipline, creating shared language, tools, and standards that enable application teams to measure progress consistently and with confidence, with an emphasis on Netflix’s LLM customized for personalization.
We are looking for a seasoned engineering leader to build and scale this team. You will lead a world-class group of AI/ML researchers and engineers working at the intersection of evaluation science, data systems, and foundation models. Your team’s work will shape the trajectory of Netflix’s foundation model development by driving the continuous improvement loop between data, evaluation, and modeling.
Questions Your Team Will Help Answer
The Model Evaluations and Data Curation team exists to give Netflix clarity and confidence in the foundation models that power personalization and discovery. Some of the strategic questions your team will help answer include:
Are our LLMs and other foundation models improving in ways that meaningfully benefit members?
How do we rigorously measure LLM capabilities across personalization, content understanding, and text generation?
What are the strongest baselines we can build with off-the-shelf SOTA models?
How do data composition, diversity, and quality influence model performance?
Which evaluation signals most reliably predict real-world product outcomes?
How can we iterate quickly on data and evaluations while ensuring reproducibility and trust?
How can we evaluate agentic systems for recommendations and content understanding?
In this role, you will:
Partner with downstream AI application teams to define shared evaluations that codify application expectations of LLMs and other foundation models, ensuring progress can be transparently tracked against real-world needs.
Design rigorous benchmarks and evaluation methodologies across ranking & recommendations, content understanding, and language/text generation — grounded in a deep technical understanding of LLMs, their strengths, limitations, and failure modes.
Lead the development of evaluators and strong baselines to ensure in-house LLMs and other foundation models demonstrate clear advantages over off-the-shelf alternatives.
Build scalable, reproducible data and evaluation systems that make dataset creation and evaluation design as nimble and experiment-friendly as model development itself.
Hire, grow, and nurture a world-class team, fostering an inclusive, high-performing culture that balances research innovation with engineering excellence.
Work closely with the teams developing Netflix’s foundation models (including our core LLM) to ensure evaluation and data insights are folded back into the cadence of model development. Proactively influence the ML Platform and Data Engineering teams at key interfaces.
Minimum Qualifications
Experience building and leading high-performing teams of ML researchers and engineers.
Proven track record of leading machine learning initiatives from research to production, ideally involving evaluation frameworks, ML infrastructure, or data-intensive systems.
Strong technical expertise in LLMs, their evaluation, and practical methods for ensuring robustness, reproducibility, and quality.
Broad knowledge of machine learning fundamentals and evaluation methodologies, including benchmark design, model-based evaluators, and offline/online metrics.
Experience driving cross-functional projects, including close collaboration with AI application teams to translate product needs into evaluation frameworks.
Excellent written and verbal communication skills, able to bridge technical and non-technical audiences.
Advanced degree in Computer Science, Statistics, or a related quantitative field.
Preferred Qualifications
8+ years of overall experience, including 3+ years in engineering management.
Experience with large-scale ML systems and foundation models, especially LLMs.
Background in building evaluation frameworks, model benchmarking, or data infrastructure for LLM training.
Familiarity with multi-modal data and evaluation.
Culture & Benefits
Netflix's culture is an integral part of our success, and we approach diversity and inclusion seriously and thoughtfully. We are an equal opportunity employer and celebrate diversity, recognizing that bringing together different perspectives and backgrounds helps build stronger teams. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.
Our compensation structure consists solely of an annual salary; we do not have bonuses. You choose each year how much of your compensation you want in salary versus stock options. To determine your personal top-of-market compensation, we rely on market indicators and consider your specific job family, background, skills, and experience to determine your compensation in the market range. The range for this role is $190,000 - $920,000.
Netflix provides comprehensive benefits including Health Plans, Mental Health support, a 401(k) Retirement Plan with employer match, Stock Option Program, Disability Programs, Health Savings and Flexible Spending Accounts, Family-forming benefits, and Life and Serious Injury Benefits. We also offer paid leave of absence programs. Full-time hourly employees accrue 35 days annually for paid time off to be used for vacation, holidays, and sick paid time off. Full-time salaried employees are immediately entitled to flexible time off. See more details about our Benefits here.
Netflix has a unique culture and environment. Learn more here.
Inclusion is a Netflix value and we strive to host a meaningful interview experience for all candidates. If you want an accommodation/adjustment for a disability or any other reason during the hiring process, please send a request to your recruiting partner.
We are an equal-opportunity employer and celebrate diversity, recognizing that diversitybuilds stronger teams. We approach diversity and inclusion seriously and thoughtfully. We do not discriminate on the basis of race, religion, color, ancestry, national origin, caste, sex, sexual orientation, gender, gender identity or expression, age, disability, medical condition, pregnancy, genetic makeup, marital status, or military service.
Job is open for no less than 7 days and will be removed when the position is filled.
About the job
Apply for this position
Engineering Manager - Machine Learning, Model Evaluations and Data Curation (AI Foundations)
Netflix is one of the world's leading entertainment services, with over 300 million paid memberships in over 190 countries enjoying TV series, films and games across a wide variety of genres and languages. Members can play, pause and resume watching as much as they want, anytime, anywhere, and can change their plans at any time.
The Opportunity
Fast-paced innovation in large language models (LLMs) and generative AI is reshaping personalization and discovery at Netflix. Our current bet is that an in-house LLM, customized on Netflix data, will be the cornerstone of this transformation. Model Evaluations and Data Curation (“Evals & Data”) are central to the development of LLMs and other high leverage foundation models at Netflix. Our Evals & Data team builds the benchmarks, evaluators, and baselines that guide LLM progress, and the data infrastructure that delivers high-quality, reproducible training and evaluation datasets to our AI/ML researchers. Together, these capabilities create the flywheel of continuous improvement between data, evals and modeling that drives innovation in LLMs that will power personalization and discovery in the future. We are incubating a centralized, first-class evaluation discipline, creating shared language, tools, and standards that enable application teams to measure progress consistently and with confidence, with an emphasis on Netflix’s LLM customized for personalization.
We are looking for a seasoned engineering leader to build and scale this team. You will lead a world-class group of AI/ML researchers and engineers working at the intersection of evaluation science, data systems, and foundation models. Your team’s work will shape the trajectory of Netflix’s foundation model development by driving the continuous improvement loop between data, evaluation, and modeling.
Questions Your Team Will Help Answer
The Model Evaluations and Data Curation team exists to give Netflix clarity and confidence in the foundation models that power personalization and discovery. Some of the strategic questions your team will help answer include:
Are our LLMs and other foundation models improving in ways that meaningfully benefit members?
How do we rigorously measure LLM capabilities across personalization, content understanding, and text generation?
What are the strongest baselines we can build with off-the-shelf SOTA models?
How do data composition, diversity, and quality influence model performance?
Which evaluation signals most reliably predict real-world product outcomes?
How can we iterate quickly on data and evaluations while ensuring reproducibility and trust?
How can we evaluate agentic systems for recommendations and content understanding?
In this role, you will:
Partner with downstream AI application teams to define shared evaluations that codify application expectations of LLMs and other foundation models, ensuring progress can be transparently tracked against real-world needs.
Design rigorous benchmarks and evaluation methodologies across ranking & recommendations, content understanding, and language/text generation — grounded in a deep technical understanding of LLMs, their strengths, limitations, and failure modes.
Lead the development of evaluators and strong baselines to ensure in-house LLMs and other foundation models demonstrate clear advantages over off-the-shelf alternatives.
Build scalable, reproducible data and evaluation systems that make dataset creation and evaluation design as nimble and experiment-friendly as model development itself.
Hire, grow, and nurture a world-class team, fostering an inclusive, high-performing culture that balances research innovation with engineering excellence.
Work closely with the teams developing Netflix’s foundation models (including our core LLM) to ensure evaluation and data insights are folded back into the cadence of model development. Proactively influence the ML Platform and Data Engineering teams at key interfaces.
Minimum Qualifications
Experience building and leading high-performing teams of ML researchers and engineers.
Proven track record of leading machine learning initiatives from research to production, ideally involving evaluation frameworks, ML infrastructure, or data-intensive systems.
Strong technical expertise in LLMs, their evaluation, and practical methods for ensuring robustness, reproducibility, and quality.
Broad knowledge of machine learning fundamentals and evaluation methodologies, including benchmark design, model-based evaluators, and offline/online metrics.
Experience driving cross-functional projects, including close collaboration with AI application teams to translate product needs into evaluation frameworks.
Excellent written and verbal communication skills, able to bridge technical and non-technical audiences.
Advanced degree in Computer Science, Statistics, or a related quantitative field.
Preferred Qualifications
8+ years of overall experience, including 3+ years in engineering management.
Experience with large-scale ML systems and foundation models, especially LLMs.
Background in building evaluation frameworks, model benchmarking, or data infrastructure for LLM training.
Familiarity with multi-modal data and evaluation.
Culture & Benefits
Netflix's culture is an integral part of our success, and we approach diversity and inclusion seriously and thoughtfully. We are an equal opportunity employer and celebrate diversity, recognizing that bringing together different perspectives and backgrounds helps build stronger teams. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.
Our compensation structure consists solely of an annual salary; we do not have bonuses. You choose each year how much of your compensation you want in salary versus stock options. To determine your personal top-of-market compensation, we rely on market indicators and consider your specific job family, background, skills, and experience to determine your compensation in the market range. The range for this role is $190,000 - $920,000.
Netflix provides comprehensive benefits including Health Plans, Mental Health support, a 401(k) Retirement Plan with employer match, Stock Option Program, Disability Programs, Health Savings and Flexible Spending Accounts, Family-forming benefits, and Life and Serious Injury Benefits. We also offer paid leave of absence programs. Full-time hourly employees accrue 35 days annually for paid time off to be used for vacation, holidays, and sick paid time off. Full-time salaried employees are immediately entitled to flexible time off. See more details about our Benefits here.
Netflix has a unique culture and environment. Learn more here.
Inclusion is a Netflix value and we strive to host a meaningful interview experience for all candidates. If you want an accommodation/adjustment for a disability or any other reason during the hiring process, please send a request to your recruiting partner.
We are an equal-opportunity employer and celebrate diversity, recognizing that diversitybuilds stronger teams. We approach diversity and inclusion seriously and thoughtfully. We do not discriminate on the basis of race, religion, color, ancestry, national origin, caste, sex, sexual orientation, gender, gender identity or expression, age, disability, medical condition, pregnancy, genetic makeup, marital status, or military service.
Job is open for no less than 7 days and will be removed when the position is filled.