Intermediate Machine Learning Engineer, AI Powered: Custom Models
An overview of this role
The Custom Models team is responsible for Duo Self-Hosted, a key component in GitLab AI that allows customers to run GitLab Duo features on completely private environments, connecting GitLab to their own AI models. They work collaboratively with numerous teams to ensure a complete lifecycle of assessing models, evaluating features, implementing controls for customization and guaranteeing a smooth experience for our largest customers.
Why us? This isn't just a job; it's your chance to shape the future of AI at GitLab. Your expertise in backend development will be critical to your success. Ready to dive into the future of AI at GitLab? Apply now! We're excited to meet potential candidates like you and welcome a new star to our team. Let's shape the future together!
What you’ll do
Develop evaluation techniques to assist feature teams on guaranteeing the quality of their features on new models.
Evolve the Evaluation Runner, our internal tool for scaling AI Feature evaluation
Keep up with the industry to explore brand new models.
Deploy and manage LLMs internally for evaluation and development.
Collaborate with product managers, engineers, and other stakeholders as a machine learning specialist, guiding them on effective implementation of AI features.
Advocate for improvements to product quality, security, and performance.
Solve technical problems of moderate scope and complexity.
Craft code that meets our internal standards for style, maintainability, and best practices for a high-scale machine-learning environment. Maintain and advocate for these standards through code review.
Confidently ship small features and improvements with minimal guidance and support from other team members. Collaborate with the team on larger projects.
Participate as a reviewer or project maintainer in one or more engineering projects.
What you’ll bring
A relevant Master’s degree and 2 or more years of experience in ML or PhD degree with a focus on Machine Learning or Data Science.
Professional experience with Python.
Experience with performance and optimization problems and a demonstrated ability to both diagnose and prevent these problems
Comfort working in a highly agile, intensely iterative software development process
Demonstrated ability to onboard and integrate with an organization long-term
Positive and solution-oriented mindset
Effective communication skills: Regularly achieve consensus with peers, and clear status updates
An inclination towards communication, inclusion, and visibility
Experience owning a project from concept to production, including proposal, discussion, and execution.
Self-motivated and self-managing, with strong organizational skills.
Demonstrated ability to work closely with other parts of the organization
Share our Values, and work in accordance with those values
Ability to thrive in a fully remote organization
Two of more of
Professional experience with prompt engineering and Retrieval Augmented Generation (RAG)
Experience with evaluation of Machine Learning models or AI features.
Experience with model deployments on cloud platforms like AWS and GCP.
Proven experience designing and implementing LLM evaluation systems.
Strong understanding of ML model architectures.
Expertise in ML evaluation metrics and dataset management.
Demonstrated ability to build production-grade ML infrastructure.
Practical experience with Python-based ML frameworks and evaluation tools (e.g., Langsmith, DSpy).
Bonus qualifications
Have contributed a Merge Request to GitLab
Have contributed to ML open source projects
About the team
The Custom Models team is a team formed of GitLab team members from around the globe. Engineers are primarily located across various European countries, with some distributions in Australia, New Zealand, America, and Canada.
The team works closely with these other teams within the organization:
AI Framework
MLOps
Model Validation
Duo Chat
How GitLab will support you
All remote, asynchronous work environment
Home office support
Please note that we welcome interest from candidates with varying levels of experience; many successful candidates do not meet every single requirement. Additionally, studies have shown that people from underrepresented groups are less likely to apply to a job unless they meet every single qualification. If you're excited about this role, please apply and allow our recruiters to assess your application.
About the job
Apply for this position
Intermediate Machine Learning Engineer, AI Powered: Custom Models
An overview of this role
The Custom Models team is responsible for Duo Self-Hosted, a key component in GitLab AI that allows customers to run GitLab Duo features on completely private environments, connecting GitLab to their own AI models. They work collaboratively with numerous teams to ensure a complete lifecycle of assessing models, evaluating features, implementing controls for customization and guaranteeing a smooth experience for our largest customers.
Why us? This isn't just a job; it's your chance to shape the future of AI at GitLab. Your expertise in backend development will be critical to your success. Ready to dive into the future of AI at GitLab? Apply now! We're excited to meet potential candidates like you and welcome a new star to our team. Let's shape the future together!
What you’ll do
Develop evaluation techniques to assist feature teams on guaranteeing the quality of their features on new models.
Evolve the Evaluation Runner, our internal tool for scaling AI Feature evaluation
Keep up with the industry to explore brand new models.
Deploy and manage LLMs internally for evaluation and development.
Collaborate with product managers, engineers, and other stakeholders as a machine learning specialist, guiding them on effective implementation of AI features.
Advocate for improvements to product quality, security, and performance.
Solve technical problems of moderate scope and complexity.
Craft code that meets our internal standards for style, maintainability, and best practices for a high-scale machine-learning environment. Maintain and advocate for these standards through code review.
Confidently ship small features and improvements with minimal guidance and support from other team members. Collaborate with the team on larger projects.
Participate as a reviewer or project maintainer in one or more engineering projects.
What you’ll bring
A relevant Master’s degree and 2 or more years of experience in ML or PhD degree with a focus on Machine Learning or Data Science.
Professional experience with Python.
Experience with performance and optimization problems and a demonstrated ability to both diagnose and prevent these problems
Comfort working in a highly agile, intensely iterative software development process
Demonstrated ability to onboard and integrate with an organization long-term
Positive and solution-oriented mindset
Effective communication skills: Regularly achieve consensus with peers, and clear status updates
An inclination towards communication, inclusion, and visibility
Experience owning a project from concept to production, including proposal, discussion, and execution.
Self-motivated and self-managing, with strong organizational skills.
Demonstrated ability to work closely with other parts of the organization
Share our Values, and work in accordance with those values
Ability to thrive in a fully remote organization
Two of more of
Professional experience with prompt engineering and Retrieval Augmented Generation (RAG)
Experience with evaluation of Machine Learning models or AI features.
Experience with model deployments on cloud platforms like AWS and GCP.
Proven experience designing and implementing LLM evaluation systems.
Strong understanding of ML model architectures.
Expertise in ML evaluation metrics and dataset management.
Demonstrated ability to build production-grade ML infrastructure.
Practical experience with Python-based ML frameworks and evaluation tools (e.g., Langsmith, DSpy).
Bonus qualifications
Have contributed a Merge Request to GitLab
Have contributed to ML open source projects
About the team
The Custom Models team is a team formed of GitLab team members from around the globe. Engineers are primarily located across various European countries, with some distributions in Australia, New Zealand, America, and Canada.
The team works closely with these other teams within the organization:
AI Framework
MLOps
Model Validation
Duo Chat
How GitLab will support you
All remote, asynchronous work environment
Home office support
Please note that we welcome interest from candidates with varying levels of experience; many successful candidates do not meet every single requirement. Additionally, studies have shown that people from underrepresented groups are less likely to apply to a job unless they meet every single qualification. If you're excited about this role, please apply and allow our recruiters to assess your application.