View All Jobs 133765

AI Automation Engineer

Develop automated pipelines to evaluate and improve conversational AI response quality at scale
Delhi
Mid-Level
yesterday
Netomi

Netomi

An AI platform specializing in customer service automation through intelligent chatbots and virtual assistants for businesses.

AI Automation Engineer

Netomi is the leading agentic AI platform for enterprise customer experience. We work with the largest global brands like Delta Airlines, MetLife, MGM, United, and others to enable agentic automation at scale across the entire customer journey. Our no-code platform delivers the fastest time to market, lowest total cost of ownership, and simple, scalable management of AI agents for any CX use case. Backed by WndrCo, Y Combinator, and Index Ventures, we help enterprises drive efficiency, lower costs, and deliver higher quality customer experiences.

An AI Automation Engineer focuses on building automated systems to evaluate, test, and enhance the performance of conversational AI models (LLMs, chatbots, voice assistants). They sit at the intersection of machine learning, software engineering, and QA automation, ensuring that AI systems produce high-quality, safe, and reliable responses at scale.

Responsibilities

  • Build Python-based pipelines for automated quality testing of AI responses.
  • Integrate LLMs into automated evaluation frameworks.
  • Automate regression and stress testing for conversational AI flows.
  • Define evaluation metrics (relevance, factuality, coherence, safety, empathy).
  • Implement both rule-based and AI-driven quality checks.
  • Monitor model drift, bias, and hallucinations using automated workflows.
  • Work with APIs, SDKs, and CI/CD pipelines to embed automated AI evaluation in production.
  • Develop monitoring dashboards to visualize conversation quality.
  • Collaborate with ML engineers, product managers, and QA teams to close the feedback loop.
  • Experiment with prompt engineering and automated prompt-testing frameworks.
  • Explore reinforcement learning, self-critique models, or human-in-the-loop automation for continuous improvement.
  • Automate compliance and policy adherence checks for enterprise AI systems.

Requirements

  • Strong 2-5 years of experience in Python development (automation, scripting, data handling).
  • Experience with LLMs/NLP frameworks.
  • Understanding of MLOps / AI deployment pipelines.
+ Show Original Job Post
























AI Automation Engineer
Delhi
Engineering
About Netomi
An AI platform specializing in customer service automation through intelligent chatbots and virtual assistants for businesses.