Sr AI Language Engineer, Amazon Rufus

Language Engineer Opportunity

The Conversational Shopping team is looking for a Language Engineer to drive efficiencies and innovation in its efforts to deliver a seamless, fluent, and engaging experience for AI-assisted shopping. This role is inherently high-visibility and highly cross-functional, requiring collaboration and influence across global product, design, science, and engineering teams. We are looking for candidates who are passionate about the intersection of language and technology and who are keen to use their technical abilities to develop automated, scalable solutions to questions in the Large Language Model (LLM) space.

In this role, they will act as one of the driving forces behind our evaluation-driven product development strategy. They will design processes to facilitate the production of high quality editorial data which will allow us to evaluate and improve the Shopping AI experience in different languages. To do so, they will be tasked with the creation and development of LLM-assisted editorial tools, automated verification scripts and automated annotations (e.g. LLM-as-a-judge) to support the humans-in-the-loop (HITL) work of the broader Editorial team.

They will lead and drive the requirements behind data annotation tasks and tooling, writing intuitive annotation guidelines and guiding the creation of the tools adapted to these workflows. They will employ their data processing and analysis skills to track team productivity and measure output quality.

This role requires strong analytical and technical skills as well as experience in language technology to help us measure, analyze and solve complex problems. They should have experience in creating technical solutions for automating and processing data workflows at scale and have the ability to do so while upholding the highest linguistic quality standards.

They should also have exceptional writing and communication skills with the ability to interface between both technical and non-technical teams.

Key Job Responsibilities

• Produce, process and manipulate different types of language data, analyze, and provide efficient solutions
• Automate operations and perform data analysis using coding/scripting language
• Develop LLM-assisted workflows and annotations solutions to support Human-in-the-loop evaluations
• Design and lead editorial data production/collection by defining scope with internal customer teams
• Define clear editorial workflows to meet or exceed the quality bar
• Adopt and design control mechanisms, metrics and methodologies for editorial and annotation quality
• Maximize productivity, process efficiency and quality through streamlined workflows, process standardization, documentation, audits and investigations
• Collaborate with editors, applied scientists, engineers, and product managers to deliver the optimal customer experience
• Establish processes and mechanisms to onboard and train editors on an ongoing basis
• Handle work prioritization and deliver based on business priorities
• Be flexible in changes to conventions deployed in response to customers' requests and change workflows accordingly

Basic Qualifications

• Knowledge of and proficiency in the use of Python scripting language
• Experience communicating technical concepts and processes using clear, simple language and visuals
• Experience working in a fast-paced environment similar to a high-tech start-up
• Experience prioritizing and handling multiple assignments at any given time while maintaining commitment to deadlines
• Master's degree, or a PhD and experience in building speech recognition, machine translation and natural language processing systems
• Knowledge of command line tools to troubleshoot protocols, analyze log outputs, or automate basic tasks
• Strong experience in Natural Language Processing, Machine Learning, AI or Large Language Models
• Knowledge of Regex, SQL, MS Excel, Git
• Familiarity with annotation tools and workflows

Preferred Qualifications

• PhD in a quantitative field, or PhD and 4+ years of data scientist experience
• Experience with language annotation and other forms of data markup
• Experience in several of the following areas: machine learning, statistics, deep learning, natural language processing, or information retrieval
• Knowledge of user experience principles and techniques
• Experience in online retail operations or similar field
• Experience with AWS services (S3, Sagemaker, ML language services, etc.)

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.