We’re all building AI features now – or will be soon. But leading teams who are building with LLMs brings its own challenges – namely: How can we bring in the latest research, consider AI ethics, and consider the cost of different models without blowing past delivery dates. Not to mention making sure that the features we build will be stable, reliable and maintainable in the future. We recently built our first LLM feature – showing the quality of feedback given in code reviews. In just 1 month with a lean team of 2 full-time team members, we built the feature – including a literature review, consultation with academic experts, data labelling, model experimentation, a cost assessment, and finally, all the ML engineering to launch them into production. The outcome: < 1% extreme misclassification and zero hallucinations, plus – the most important part – a feature loved by customers.

In this talk, we’ll share our approach to building LLM features – how we partnered with academia (without being delayed by their timelines), what tooling we used, and how we made the cost and money tradeoffs to keep business stakeholders happy. As one example, we’ll share how important evaluation data was for building our features, because it helped us improve our definitions and revealed gender differences on how people perceive feedback. We’ll share perspectives from both data and product & engineering leadership, and how we thought about balancing academic rigor and robust practices with cost and timeline considerations. We’ll share which frameworks actually helped us make the right calls, avoid expensive do-overs, and figure out which AI ethics issues needed fixing right away versus later.

You'll walk away with practical strategies for leading your own teams through AI implementations, identifying ethical issues early, addressing them efficiently, and still meeting your business objectives on time and on budget.