Ragas Elasticsearch Demo
Evaluating your Elasticsearch LLM applications with Ragas
Installing the required packages
You will first want to install the packages required for this application. These include the following:
-
The Elasticsearch Python client will be helpful for authenticating and connecting to Elasticsearch and for vector search.
-
Ragas is used for evaluating the quality of the LLM applications using standard metrics
-
The Hugging Face library datasets is used to create a robust evaluation dataset
-
Langchain-OpenAI, which is used both for generating answers to user questions and for evaluation
[notice] A new release of pip is available: 25.1.1 -> 25.2 [notice] To update, run: pip install --upgrade pip
Import modules
After installing the required packages, import modules for handling environment variables, parsing JSON data, securely entering sensitive values, connecting to Elasticsearch, evaluating RAG applications and their metrics, creating evaluation datasets, and enabling chat model capabilities.
Host: ········ API Key: ········
Your model will need to be deployed first before the following code could run. To learn more about this be sure to check out our documention on the subject.
Deleted existing index 'ragas-books' Created index 'ragas-books'
If you encounter any problems running this in Colab or locally, it might be due to the dataset requiring a separate download. It can be found in the same folder as this example.
Indexed 1: Lucky 7 Indexed 2: Salvation Lost Indexed 3: Alien Warrior's Mate Indexed 4: On the Steel Breeze Indexed 5: Salvage Marines Indexed 6: Trade Secret Indexed 7: There Will Be Time Indexed 8: Only in Death Indexed 9: His To Claim Indexed 10: Savage Drift Indexed 11: Light of the Jedi Indexed 12: Mega Robo Bros Indexed 13: Transmetropolitan, Vol. 1: Back on the Street Indexed 14: The Queen of Traitors Indexed 15: The Island of Doctor Moreau Indexed 16: Human Nature Indexed 17: Legion Indexed 18: Wolfsbane Indexed 19: Lamb: The Gospel According to Biff, Christ's Childhood Pal Indexed 20: Our Pet Indexed 21: The Aylesford Skull Indexed 22: Ghosts of War Indexed 23: The Book of Time Indexed 24: Because It Is My Blood Indexed 25: The Annals of the Heechee
Implementing the RAG generation component
This section sets up the RAG generation pipeline by configuring the OpenAI API key, creating a chat model, defining functions to generate answers from retrieved context, analyze question intent, score books against user preferences, and produce ground truth responses for evaluation.
Running the demo
This section runs the RAG demo by asking sample questions, retrieving context, generating answers, creating ground truths, building an evaluation dataset, running Ragas metrics, and printing and saving the evaluation results.
🚀 Demo: 📚 Question 1: What's a good science fiction book with high ratings? Answer: "Light of the Jedi" by Charles Soule... Ground Truth: For a highly-rated science fiction book, I recommend 'Light of the Jedi' by Charles Soule (rating: 4.2). You might also consider 'Legion' by Dan Abnett. 📚 Question 2: Can you suggest a fantasy book by a popular author? Answer: Based on the context provided, I recommend "The Book of Time" by Guillaume Prévost.... Ground Truth: A good fantasy book from the available options is 'Our Pet' by S.M. Matthews. You might also consider 'The Book of Time' by Guillaume Prévost. 📚 Question 3: What's a highly rated mystery novel? Answer: Human Nature by Jonathan Green... Ground Truth: For a highly-rated mystery book, I recommend 'Human Nature' by Jonathan Green (rating: 3.6). You might also consider 'Lamb: The Gospel According to Biff, Christ's Childhood Pal' by Christopher Moore. 📚 Question 4: Recommend a book with good reviews Answer: The Island of Doctor Moreau by H.G. Wells... Ground Truth: Based on the available books, 'The Island of Doctor Moreau' by H.G. Wells would be a good choice. You might also consider 'Savage Drift' by Emmy Laybourne. ✨ Running Ragas evaluation...
Evaluating: 0%| | 0/12 [00:00<?, ?it/s]
✨ Ragas Evaluation Results:
user_input \
0 What's a good science fiction book with high r...
1 Can you suggest a fantasy book by a popular au...
2 What's a highly rated mystery novel?
3 Recommend a book with good reviews
retrieved_contexts \
0 [Light of the Jedi by Charles Soule: Two hundr...
1 [Alien Warrior's Mate by Vi Voxley: He's damn ...
2 [The Island of Doctor Moreau by H.G. Wells: Ra...
3 [The Island of Doctor Moreau by H.G. Wells: Ra...
response \
0 "Light of the Jedi" by Charles Soule
1 Based on the context provided, I recommend "Th...
2 Human Nature by Jonathan Green
3 The Island of Doctor Moreau by H.G. Wells
reference context_precision \
0 For a highly-rated science fiction book, I rec... 0.0
1 A good fantasy book from the available options... 0.5
2 For a highly-rated mystery book, I recommend '... 0.0
3 Based on the available books, 'The Island of D... 1.0
faithfulness context_recall
0 1.000000 0.0
1 1.000000 1.0
2 0.666667 0.0
3 0.666667 1.0
\✨ Averages:
context_precision: 0.375
faithfulness: 0.833
context_recall: 0.500
Results saved to 'ragas_evaluation.csv'
🎉 Demo completed successfully!