AI Evals and Analytics

Cohort 3

Cohort 1

Cohort 2

Cohort 2 - skipped

Cohort 4

Cohort 5

LLM-as-a-judge is widely used as a low-cost proxy for human or business ground truth, but uncalibrated judge scores can be statistically misleading, even reversing model rankings. This creates real production risk. Eddie Landesberg, an AI Evals researcher, introduces a calibration method to better align LLM-as-a-judge with human judgment and real-world decisions. This deck is from a guest Lightning Lesson by <a href="https://www.linkedin.com/in/eddie-landesberg/" rel="noopener noreferrer" target="_blank">Eddie Landesberg</a>. Check out Lightning Lesson the recording <a href="https://maven.com/p/ed75d8/calibrate-llm-as-a-judge-for-real-world-impact?utm_medium=ll_share_link&amp;utm_source=instructor" rel="noopener noreferrer" target="_blank">here</a>.Also check out Eddie's post "<a href="https://www.cimolabs.com/blog/metrics-lying" rel="noopener noreferrer" target="_blank">Your AI Metrics Are Lying to You</a>".

Calibrate LLM-as-a-judge for Real-world Impact

AI product teams often struggle to decide which metrics actually matter. We introduce AI evaluations and AI product analytics as two distinct but complementary approaches to measuring AI product success. We explain what AI evals measure before and during deployment, how product analytics reflects real-world behavior after launch, and how combining both across the product lifecycle helps teams build AI systems that are reliable, safe, and impactful in production.

AI Evals vs. Product Metrics

The first step in building a solid AI Evals practice is team alignment. Before you run tests or build dashboards, you need to convince leadership and stakeholders that AI Evals matter.Not everyone’s on the same page (yet). Let’s talk about how to have that conversation, and how to influence effectively.

Bring AI Evals to Product Leadership

Product-focused. Immediately executable AI Evals playbook for your product.

A Preview of Our AI Evals Playbook and Framework

Product managers who drive strategy, development, and launch of products.

Engineers and engineering leaders who build, deploy, and scale software systems.

Data scientists who apply statistical and ML techniques to extract insights from data.

Measuring and evaluating AI model performance, quality, and reliability for production use cases.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Using data and metrics to measure performance, run experiments, and inform decisions.

Learn how to evaluate, ship, and monitor GenAI products that perform reliably in production.

Head of AI Applied Science

Stella Liu

Cofounder, AI Evals & Analytics

Amy Chen

Cofounder at AI Evals & Analytics

AI Evals and Analytics Playbook

Accessibility Technical Manager

Product Manager

Technology Advisor

Data Scientist

New Grad

Founder

AI Engineer/ AI Scientist

Staff Machine Learning Engineer

AI product manager

Senior Data Scientist

Shopify

Carvana

Harvard University

System1

Uptake

Seasalt.ai

UCLA

University of Washington

<blockquote>Building AI is easier than ever. Building AI that is reliable, safe, and trusted is not.AI evaluations help teams measure quality, detect failures, and build guardrails around your GenAI systems that will survive in the real world.</blockquote><blockquote>🔹</blockquote><h3>Who this course is for</h3>Data scientists, PMs and QA/QE professionals who:<ul><li>have no prior background in AI evals, or</li><li>have tried LLM-as-a-judge but want a clearer framework.</li></ul><blockquote>🔹How the course works</blockquote><ul><li>Self-paced lectures you can watch anytime</li><li>Hands-on exercises using a pre-built GenAI product</li><li>Dedicated office hours for questions and guidance</li></ul><blockquote>🔹</blockquote><h3>What makes this course different</h3><ul><li>Playbook-driven approach</li><li>Evaluation-driven development</li><li>Product-focused</li><li>Hands-on experience</li><li>Flexible learning + expert support</li></ul>🔹Curious about the ideas behind this course?You can explore related topics in the <a target="_blank" rel="noopener noreferrer nofollow" href="https://datasciencexai.substack.com/">Data Science × AI Substack</a>, where we share perspectives on evaluating real GenAI products.🔹💰 Purchasing Power Parity pricing adjustments upon request. ✉️ Email us: StellaxAmy@gmail.com

Learn How to Evaluate GenAI Products Confidently in 3 Weeks

Execution-ready Testing & Monitoring Playbook

GenAI Risk Management Strategy

Hands-On AI Evals Exercise

Product LeadersBuilding GenAI products and need a clear playbook for evaluation frameworks, success metrics, and shipping with confidence.

Data ScientistsLeveling up with AI product skills: learn AI evals, LLM-as-a-judge, and design metrics for GenAI systems.

QA and Test EngineerAdapting traditional testing practices to GenAI systems.

Learn directly from AI Evals and Analytics

AI Evals and Analytics

Courses

AI Evals and Analytics Playbook

Lightning Lessons

The Analytics Mindset Behind Effective AI Evals

Collaborative AI Evals with Human Feedback

Evals for Voice AI: Learnings from Google Evals Team

Calibrate LLM-as-a-judge for Real-world Impact

AI Evals vs. Product Metrics: A 30min Crash Course

Pragmatic AI Evals: Beyond Vendor Promises

Resources

A Preview of Our AI Evals Playbook and Framework

AI Evals vs. Product Metrics

Bring AI Evals to Product Leadership

Calibrate LLM-as-a-judge for Real-world Impact