Data Engineer - Personalization
Personalization’s Hulk squad produces Human Understandable Language Knowledge to enrich content understanding. We utilize Large Language Models to understand podcasts and audiobooks, building reliable, scalable systems to distribute that knowledge to Spotify internal teams, users, and creators. We are looking for a Data Engineer with interest in ML techniques to join our team and help build the future of podcast and audiobook listening experiences for millions of listeners at Spotify. This is a unique opportunity to help develop and shape Spotify recommendations. You’ll grow your skills in engineering at scale, work with a cross-functional team of Machine Learning Engineers, Backend Engineers, Data Engineers, and researchers, and join a super motivated and supportive team.
What You'll Do
Work with large-scale data pipelines using data processing frameworks like Scio (built on Apache Beam), Hendrix ML, Ray, PyTorch, and Dataflow and other Google Cloud Platform offerings
Support streaming and batch inference of core LLM models
Develop new data pipelines to meet emerging needs and maintain our existing data systems as necessary to ensure they are reliable, scalable, and well tested
Work with ML engineers to develop training and evaluation datasets for Generative LLM models
Own the development of data solutions from architecture to delivery
Collaborate with other backend and data engineers, ML engineers, data scientists, researchers, and other stakeholders
Develop monitoring tools to automatically track AI product quality. This likely involves the use of LLMs to evaluate results as well as closer monitoring of performance indicators, and creation and maintenance of metrics datasets and dashboards
Work in an agile team to continuously experiment, iterate and deliver on new product objectives
Share knowledge, promote standard methodologies, making your team the best version of itself through mentorship and constructive accountability
Who You Are
You are passionate about data and proficient in data modeling, data access, and data storage optimization
You have experience building high-scale production data pipelines with one or more data processing frameworks such as Scio (built on Apache Beam), and major props if you know Ray
You might have worked with orchestration frameworks such as Flyte
You care about agile software processes, data-driven development, reliability, and responsible experimentation
You value collaboration and partnership within teams
Experience, exposure, or curiosity about ML Data processing and inference pipelines, LLM fine-tuning, and/or using LLMs for data analysis
Bonus if you have experience with NRT (near real time) systems
Bonus if you experience with prompt engineering for LLMs and prompt optimization tools like DSPy
Where You'll Be
We offer you the flexibility to work where you work best! For this role, you can be within the North America region as long as we have a work location.
This team operates within the Eastern Standard time zone for collaboration.
The United States base range for this position is $160,091- $228,702 plus equity. The benefits available for this position include health insurance, six month paid parental leave, 401(k) retirement plan, monthly meal allowance, 23 paid days off, 13 paid flexible holidays, paid sick leave. These ranges may be modified in the future.
Spotify is an equal opportunity employer. You are welcome at Spotify for who you are, no matter where you come from, what you look like, or what’s playing in your headphones. Our platform is for everyone, and so is our workplace. The more voices we have represented and amplified in our business, the more we will all thrive, contribute, and be forward-thinking! So bring us your personal experience, your perspectives, and your background. It’s in our differences that we will find the power to keep revolutionizing the way the world listens.
At Spotify, we are passionate about inclusivity and making sure our entire recruitment process is accessible to everyone. We have ways to request reasonable accommodations during the interview process and help assist in what you need. If you need accommodations at any stage of the application or interview process, please let us know - we’re here to support you in any way we can.
Spotify transformed music listening forever when we launched in 2008. Our mission is to unlock the potential of human creativity by giving a million creative artists the opportunity to live off their art and billions of fans the chance to enjoy and be passionate about these creators. Everything we do is driven by our love for music and podcasting. Today, we are the world’s most popular audio streaming subscription service.
About the job
Apply for this position
Data Engineer - Personalization
Personalization’s Hulk squad produces Human Understandable Language Knowledge to enrich content understanding. We utilize Large Language Models to understand podcasts and audiobooks, building reliable, scalable systems to distribute that knowledge to Spotify internal teams, users, and creators. We are looking for a Data Engineer with interest in ML techniques to join our team and help build the future of podcast and audiobook listening experiences for millions of listeners at Spotify. This is a unique opportunity to help develop and shape Spotify recommendations. You’ll grow your skills in engineering at scale, work with a cross-functional team of Machine Learning Engineers, Backend Engineers, Data Engineers, and researchers, and join a super motivated and supportive team.
What You'll Do
Work with large-scale data pipelines using data processing frameworks like Scio (built on Apache Beam), Hendrix ML, Ray, PyTorch, and Dataflow and other Google Cloud Platform offerings
Support streaming and batch inference of core LLM models
Develop new data pipelines to meet emerging needs and maintain our existing data systems as necessary to ensure they are reliable, scalable, and well tested
Work with ML engineers to develop training and evaluation datasets for Generative LLM models
Own the development of data solutions from architecture to delivery
Collaborate with other backend and data engineers, ML engineers, data scientists, researchers, and other stakeholders
Develop monitoring tools to automatically track AI product quality. This likely involves the use of LLMs to evaluate results as well as closer monitoring of performance indicators, and creation and maintenance of metrics datasets and dashboards
Work in an agile team to continuously experiment, iterate and deliver on new product objectives
Share knowledge, promote standard methodologies, making your team the best version of itself through mentorship and constructive accountability
Who You Are
You are passionate about data and proficient in data modeling, data access, and data storage optimization
You have experience building high-scale production data pipelines with one or more data processing frameworks such as Scio (built on Apache Beam), and major props if you know Ray
You might have worked with orchestration frameworks such as Flyte
You care about agile software processes, data-driven development, reliability, and responsible experimentation
You value collaboration and partnership within teams
Experience, exposure, or curiosity about ML Data processing and inference pipelines, LLM fine-tuning, and/or using LLMs for data analysis
Bonus if you have experience with NRT (near real time) systems
Bonus if you experience with prompt engineering for LLMs and prompt optimization tools like DSPy
Where You'll Be
We offer you the flexibility to work where you work best! For this role, you can be within the North America region as long as we have a work location.
This team operates within the Eastern Standard time zone for collaboration.
The United States base range for this position is $160,091- $228,702 plus equity. The benefits available for this position include health insurance, six month paid parental leave, 401(k) retirement plan, monthly meal allowance, 23 paid days off, 13 paid flexible holidays, paid sick leave. These ranges may be modified in the future.
Spotify is an equal opportunity employer. You are welcome at Spotify for who you are, no matter where you come from, what you look like, or what’s playing in your headphones. Our platform is for everyone, and so is our workplace. The more voices we have represented and amplified in our business, the more we will all thrive, contribute, and be forward-thinking! So bring us your personal experience, your perspectives, and your background. It’s in our differences that we will find the power to keep revolutionizing the way the world listens.
At Spotify, we are passionate about inclusivity and making sure our entire recruitment process is accessible to everyone. We have ways to request reasonable accommodations during the interview process and help assist in what you need. If you need accommodations at any stage of the application or interview process, please let us know - we’re here to support you in any way we can.
Spotify transformed music listening forever when we launched in 2008. Our mission is to unlock the potential of human creativity by giving a million creative artists the opportunity to live off their art and billions of fans the chance to enjoy and be passionate about these creators. Everything we do is driven by our love for music and podcasting. Today, we are the world’s most popular audio streaming subscription service.