AI Engineer, Product, (le Chat)

AI Engineer, Product, (Le Chat)

Embedded directly in Le Chat product team, you will build the evaluation and A/B testing framework, add end-to-end observability, and run a reliable model release process. Work with Science to ship measurable improvements to quality, latency, safety, and reliability.

• Build and maintain an LLM evaluation framework (reference tests, heuristics, model-graded checks).

• Define and track metrics: task success, helpfulness, hallucination proxies, safety flags, latency/cost.

• Run A/B tests for prompts, models, and system prompts, analyze results, recommend rollout or rollback.

• Set up observability for LLM calls: structured logging, tracing, dashboards, alerts.

• Operate the model release: canary and shadow traffic, sign-offs, SLO-based rollback criteria, regression detection.

• Improve core behaviors: memory write/retrieve policies and evals, intent classification, follow-ups, routing, tool-call reliability.

• Create templates and docs so other teams can author evals and ship safely.

• Partner with Science, diagnose regressions and lead post-mortems.

About you:

• Strong TypeScript or Python skills

• Production LLM experience: prompts, tool/function calling, and system prompts.

• Hands-on with evals and A/B testing, you can design metrics and make rollout decisions from data.

• Observability: logging, tracing, dashboards, alerting

• Product mindset: form hypotheses, run experiments, interpret results, iterate.

• Clear written and spoken communication, autonomous; and product-oriented.

Now it would be ideal if you have experience with:

• Safety systems: moderation, PII handling/redaction, guardrails.

• Release operations: canary/shadowing, automated rollbacks, experiment platforms.

Hiring Process:

• Introduction call - 45 min

• Hiring Manager interview - 30 min

• Technical Rounds

- Live-coding Interview (Machine Learning) - 45 min

- Live-coding Interview (Algorithm) - 45 min

• Culture-fit discussion - 30 min

• References

Location & Remote:

This role is primarily based at our HQ in Paris, France. We will prioritize candidates who either reside in Paris or are open to relocating. We strongly believe in the value of in-person collaboration to foster strong relationships and seamless communication within our team. Our remote work policy is designed to offer flexibility, enhance work-life balance, and boost productivity. The number of remote workdays is determined by each manager, taking into account individual autonomy and specific circumstances—such as increased flexibility during the summer months. Regardless of the arrangement, we expect all employees to maintain open lines of communication with their teams and be available during core working hours.

In certain specific situations, we will also consider remote candidates based in one of the countries listed in this job posting (currently France, UK, Germany, Netherlands, Spain and Italy). In that case, we ask all new hires to visit our Paris office:

• for the first month of their onboarding (accommodation and travelling covered)

• then at least 3 days per month

What we offer:

Competitive salary and equity

Health insurance

Transportation allowance

Sport allowance

Meal vouchers

Private pension plan

Generous parental leave policy

Visa sponsorship