LLM Ops Engineer

LLM Ops Engineer

We are seeking a skilled and proactive LLM Ops Engineer with expertise in Large Language Model (LLM) Operations to join our growing AI and software development team. This role is critical in ensuring the reliability, performance, and evolution of LLMs and data pipelines in production environments.

You'll understand the roles of Platform, AI Engineering, Governance and Product. We have built a technical stack that can be expressed as three main categories; tracing, Annotations and experiments with plans to build a Data Science Workspace. We are looking for an engineer who can help us maintain the pipelines, experimentation platform and data science workspace. Ideally you'll have experience with Langfuse (or similar tools) and how data science workspaces.

The LLM Ops team is a new team and we are building out the use case, so you'll be part of a team that is pushing on new frontiers in the area of LLM Ops. You'll be working with other LLM Ops engineers, AI engineers, data scientists, operations personnel and our product manager.

Responsibilities

• Data & System Integration

  • You'll be responsible for configuring the platform, integration. You'll understand the roles of Platform, AI Engineering, Governance and Product. You'll be responsible for completion of the LLM Ops stack, including Platform (DSQ, Langfuse), Langfuse integrations for AI components, LLM Ops workflows, help with data annotations, basic monitoring and analytics.

  • Work with data scientists and engineers to ensure high-quality, real-time data feeds into LLM systems.

  • Build and maintain observability tools for data quality, model behaviour, and system health.

• Model Lifecycle Management

  • Manage the deployment, versioning, and rollback of LLMs in production.

  • Help to coordinate and execute model upgrades, ensuring minimal disruption to services. Supported by data scientists.

  • Implement CI/CD pipelines for model deployment and testing.

• LLM Monitoring & Maintenance

  • Help to monitor LLM performance in production, including latency, accuracy, and user feedback.

  • Help to detect and address LLM drift and data drift using automated tools and custom metrics.

  • Collaborate with data scientists to define and refine model evaluation criteria.

Required Skills & Experience

• Proven experience in LLM Ops, MLOps, or AI infrastructure roles.

• Strong understanding of LLM behaviour, fine-tuning, and prompt engineering.

• Experience with monitoring tools (e.g., Prometheus, Grafana, MLflow, Evidently AI).

• Experience in AWS Bedrock LLM models

• Proficiency in Python and familiarity with ML frameworks (e.g., Hugging Face Transformers, LangChain).

• Experience with cloud platforms (AWS, Azure, or GCP) and containerization (Docker, Kubernetes).

• Experience in Model Optimisation and familiarity with data drift detection, model drift analysis, and automated retraining pipelines.

• Excellent communication and collaboration skills.

Desired Skills & Experience

• Experience working in agile software development environments.
• Knowledge of data privacy, model interpretability, and responsible AI practices.
• Contributions to open-source LLM or MLOps projects.
• Experience in Finance sector and/or risk management

Career Stage: Senior Associate

London Stock Exchange Group (LSEG) Information: Join us and be part of a team that values innovation, quality, and continuous improvement. If you're ready to take your career to the next level and make a significant impact, we'd love to hear from you.

LSEG is a leading global financial markets infrastructure and data provider. Our purpose is driving financial stability, empowering economies and enabling customers to create sustainable growth.

Our purpose is the foundation on which our culture is built. Our values of Integrity, Partnership, Excellence and Change underpin our purpose and set the standard for everything we do, every day. They go to the heart of who we are and guide our decision making and everyday actions.

Working with us means that you will be part of a dynamic organisation of 25,000 people across 65 countries. However, we will value your individuality and enable you to bring your true self to work so you can help enrich our diverse workforce.

We are proud to be an equal opportunities employer. This means that we do not discriminate on the basis of anyone's race, religion, colour, national origin, gender, sexual orientation, gender identity, gender expression, age, marital status, veteran status, pregnancy or disability, or any other basis protected under applicable law. Conforming with applicable law, we can reasonably accommodate applicants' and employees' religious practices and beliefs, as well as mental health or physical disability needs.

You will be part of a collaborative and creative culture where we encourage new ideas. We are committed to sustainability across our global business and we are proud to partner with our customers to help them meet their sustainability objectives. Our charity, the LSEG Foundation provides charitable grants to community groups that help people access economic opportunities and build a secure future with financial independence. Colleagues can get involved through fundraising and volunteering.

LSEG offers a range of tailored benefits and support, including healthcare, retirement planning, paid volunteering days and wellbeing initiatives.