RAG-LLM Evaluation & Test Automation for Beginners
- Descrição
- Currículo
- FAQ
- Revisões
LLMs are everywhere! Every business is building its own custom AI-based RAG-LLMs to improve customer service. But how are engineers testing them? Unlike traditional software testing, AI-based systems need a special methodology for evaluation.
This course starts from the ground up, explaining the architecture of how AI systems (LLMs) work behind the scenes. Then, it dives deep into LLM evaluation metrics.
This course shows you how to effectively use the RAGAS framework library to evaluate LLM metrics through scripted examples. This allows you to use Pytest assertions to check metric benchmark scores and design a robust LLM Test/evaluation automation framework.
What will you learn from the course?
-
High level overview on Large Language Models (LLM)
-
Understand how Custom LLM’s are built using Retrieval Augmented Generation (RAG) Architecture
-
Common Benchmarks/Metrics used in Evaluating RAG based LLM’s
-
Introduction to RAGAS Evaluation framework for evaluating/test LLM’s
-
Practical Scripts generation to automate and assert the Metrics Score of LLM’s.
-
Automate Scenarios such as Single turn interactions and Multi turn interactions with LLM’s using RAGAS Framework
-
Generate Test Data for evaluating the Metrics of LLM using RAGAS Framework.
By end of the course, you will be able to create RAGAS Pytest Evaluation Framework to assert the Metrics of RAG- (Custom) LLM’s
Important Note:
This course covers Top 7 Metrics which are commonly used to Evaluate and test the LLM’s. Same logic can be applied to rest of any other metric evaluations.
Handson Experience:
Course provides Practice RAG -LLM for you for Handson, but at scripting phase, you need a basic subscription of Open AI to access their API’s (Minimal 10$ credit will suffice)
Course Prerequisites:
-
Python, PyTest basics are required to understand the Framework.
We have 2 dedicated sections at the end of this course which gives you necessary knowledge on Python & Pytest required to follow the course.
-
Basic knowledge on API Testing.
-
1
What this course offers? FAQ"s -Must Watch -
2
Course outcome - Setting the stage of expectation -
3
Introduction to Artificial Intelligence and LLM's - How they work
-
4
Overview of popular LLM"s and Challenges with these general LLM's
-
5
What is Retrieval Augmented Generation (RAG)? Understand its Architecture
-
6
End to end flow in RAG Architecture and its key advantages
-
10
Course resources download
-
11
Demo of Practice RAG LLM's to evaluate and write test automation scripts
-
12
Understanding implementation part of practice RAG LLM's to understand context
-
13
Understand conversational LLM scenarios and how they are applied to RAG Arch
-
14
Understand the Metric benchmarks for Document Retrieval system in LLM
-
19
Making connection with OpenAI using Langchain Framework for RAGAS
-
20
End to end -Evaluate LLM for ContextPrecision metric with SingleTurn Test data
-
21
Metrics document download
-
22
Communicate with LLM's using API Post call to dynamically get responses
-
23
Evaluate LLM for Context Recall Metric with RAGAS Pytest Test example
-
27
Understand LLM's Faithfulness and Response relevance metrics conceptually
-
28
Build LLM Evaluation script to test Faithfulness benchmarks using RAGAS
-
29
Reading Test data from external json file to LLM evaluation scripts
-
30
Understand how Metrics are used at different places of RAG LLM Architecture
-
31
Factual Correctness - Build a single Test to evaluate multiple LLM metrics
-
32
Understand EvaluationDataSet and how it help in evaluating Multiple metrics
-
33
Upload the LLM Metrics evaluation results into RAGAS dashboard portal visually
-
34
How to evaluate RAG LLM with multi conversational history chat
-
35
Build LLM Evaluation Test which can evaluate multi conversation - example
-
36
How to Create Test Data using RAGAS Framework to evaluate LLM
-
37
Load the external docs into Langchain utils to analyze and extract test data
-
38
Install and configure NLTK package to scan the LLM documents & generating tests
-
39
Generate Rubrics based Criteria Scoring to evaluate the quality of LLM responses
-
42
Python hello world Program with Basics
-
43
Datatypes in python and how to get the Type at run time
-
44
List Datatype and its operations to manipulate
-
45
Tuple and Dictionary Data types in Python with examples
-
46
If else condition in python with working examples
-
47
How to Create Dictionaries at run time and add data into it
-
48
How loops work in Python and importance of code idendation
-
49
Programming examples using for loop - 1
-
50
Programming examples using While loop - 2
-
51
What are functions? How to use them in Python
-
52
OOPS Principles : Classes and objects in Python
-
53
What is Constructor and its role in Object oriented programming
-
54
Inheritance concepts with examples in Python
-
55
Strings and its functions in python
