Skip to main content Open menu Open navigation Go to Reddit Home r/ollama Get App Get the Reddit app Log In Log in to Reddit Expand user menu Open settings menu Go to ollama r/ollama • Ok_Constant_9886 हिन्दीEspañol (Latinoamérica) What do you guys test LLMs in CI/CD? r/AIEval • What do you guys test LLMs in CI/CD? Something that our team is thinking about is how to test LLMs (apps, not just foundational models) in CI envs like GitHub Actions. In one standpoint, there's the concept of testing for functionality, while on the other hand there is the concept of testing for responsible AI such as bias and toxicity. Unlike traditional unit testing, tests for LLMs seem to be more scattered, with criteria that are not clearly defined (due to non-deterministically). The approaches we're doing right now includes separating tests by: Functionality: Test files and directories on whether APIs to our LLM app returns correctly Responsibility: Test files that tackle responsible AI - our app is user facing so it must comply with local regulations in our region Performance: Test latency, tokens per second, cost, deterministic metrics Specific business criteria: Custom LLM-as-a-judge criteria that is more subjective but gives us a good idea of how things are performing. We also found open-source tools like deepeval useful since it integrates with Pytest for CI, and offers the breadth of LLM-as-a-judge metrics we need. Curious how other people are doing it? Read more upvotes Share New to Reddit? Create your account and connect with a world of communities. Continue with Email By continuing, you agree to our User Agreement and acknowledge that you understand the Privacy Policy. Public Anyone can view, post, and comment to this community 0 0 Reddit Rules Privacy Policy User Agreement Accessibility Reddit, Inc. © 2026. All rights reserved. Expand Navigation Collapse Navigation
==============
Обсуждается вопрос тестирования LLM в CI/CD, включая функциональное, ответственное (избежание предвзятости и токсичности) и производительное тестирование, а также специфические бизнес-критерии. Предлагаются инструменты, такие как deepeval, для интеграции с Pytest, и подчеркивается важность определения четких критериев тестирования из-за непредсказуемости LLM.
