Openai Embeddings Eval
agentsartificial-intelligencellmsmongodb-genai-showcaseevalsnotebooksgenerative-airag
Export
How to choose the right embedding model for your RAG application
This notebook evaluates the text-embedding-3-large model.
Step 1: Install required libraries
- datasets: Python library to get access to datasets available on Hugging Face Hub
- openai: Python library to interact with OpenAI APIs
- numpy: Python library that provides tools to perform mathematical operations on arrays
- pandas: Python library for data analysis, exploration and manipulation
- tdqm: Python module to show a progress meter for loops
[49]
Step 2: Setup pre-requisites
Set OpenAI API key as environment variable, and initialize the OpenAI client.
Steps to obtain an OpenAI API Key can be found here
[50]
[51]
OpenAI API Key: ········
Step 3: Download the evaluation dataset
We will use MongoDB's cosmopedia-wikihow-chunked dataset, which has chunked versions of WikiHow articles from the Cosmopedia dataset released by Hugging Face. The dataset is pretty large, so we will only grab the first 2k records for testing.
[52]
Step 4: Data analysis
Make sure the length of the dataset is what we expect (2k), preview the data, drop Nones etc.
[53]
2000
[54]
[55]
[56]
352
Step 5: Creating embeddings
Define the embedding function, and run a quick test.
[57]
[58]
[59]
[60]
3072
Step 6: Evaluation
Measuring embedding latency
Create a local vector store (list) of embeddings for the entire dataset.
[61]
[62]
[63]
0%| | 0/2000 [00:00<?, ?it/s]
Measuring retrieval quality
- Create embedding for the user query
- Get the top 5 most similar documents from the local vector store using cosine similarity as the similarity metric
[ ]
[43]
[44]
[45]
Query: Give me some tips to improve my mental health. Score: 0.5319 Step 2: Reach Out to Someone Trustworthy Connect with someone who cares about you—a friend, family member, mental health professional, or support group. Talking openly about your struggles can lighten your emotional burden, making it easier to manage. Sharing your feelings may also lead to helpful suggestions and advice from others. Guideline: Make sure to choose someone trustworthy who has shown empathy towards you before. If possible, reach out to more than one person to build a strong support network around you. Step 3: Engage in Physical Activity Exercise releases endorphins, which improve mood and reduce stress levels. Go for a walk, jog, bike ride, swim, or engage in any physical activity that suits your abilities. Exercising regularly can significantly impact overall wellbeing by reducing symptoms of depression and anxiety. Key Tip: Start small - aim for just five minutes of exercise if needed, then gradually increase duration over time. The most important thing is to get started! -------- Score: 0.5309 Step 2: Improve Your Physical Health Explanation: Taking care of your physical health can significantly contribute to your overall mood and energy levels. Regular exercise, proper nutrition, and adequate sleep play crucial roles in maintaining your well-being. Key Tips & Guidelines: * Exercise regularly (e.g., walking, jogging, yoga, dancing). * Eat balanced meals rich in fruits, vegetables, lean proteins, and whole grains. * Limit processed foods, sugars, and unhealthy fats. * Stay hydrated throughout the day. * Get at least seven hours of sleep per night. Step 3: Enhance Your Personal Style Explanation: Dressing in a way that makes you feel confident and comfortable can boost your self-esteem and reflect your vibrant personality. Key Tips & Guidelines: -------- Score: 0.4906 Key Tips & Guidelines: * Monitor your inner dialogue and identify any recurring negative thoughts. * Flip the script by reframing those thoughts into more balanced and realistic alternatives. For example, if you think, "I always mess up," try telling yourself, "I made a mistake, but I can learn from it." * Practice affirmations – short, empowering statements that help build confidence and resilience. Repeat them regularly throughout the day. Why it works: Negative self-talk reinforces low self-esteem and pessimism, while positive thinking promotes mental wellbeing and contentment. Step 3: Foster Social Connections Explanation: Humans are social creatures who thrive on connection and support. Building strong relationships with friends, family, and community contributes to feelings of belonging and happiness. Key Tips & Guidelines: --------
[46]
Query: Give me some tips for writing good code. Score: 0.5893 Step 6: Improve Code Quality Strive for clean, readable, maintainable code. Adopt consistent naming conventions, indentation styles, and formatting rules. Utilize version control systems like Git to track changes and collaborate effectively. Leverage linters and static analyzers to enforce style guides automatically. Document your work using comments and dedicated documentation tools. High-quality code facilitates collaboration, promotes longevity, and simplifies troubleshooting. Step 7: Embrace Best Practices Follow established best practices relevant to your chosen language and domain. Examples include Object-Oriented Design Principles, SOLID principles, Test-Driven Development (TDD), Dependency Injection, Asynchronous Programming, etc. While seemingly overwhelming initially, integrating them gradually enhances design patterns, scalability, and extensibility. Consult authoritative blogs, books, and articles to stay updated on current trends and recommendations. -------- Score: 0.4586 Conclusion: Becoming a good programmer requires dedication, persistence, and patience. By methodically progressing through these steps, mastering core concepts, practicing diligently, engaging with peers, and committing to continuous improvement, you'll be well on your way to achieving your goal. Remember, every expert was once a beginner - keep pushing forward! -------- Score: 0.4490 Step 4: Work On Real Projects Theory alone won't make you a proficient programmer; hands-on experience does. Apply what you've learned by working on real projects. Begin with small tasks like creating calculators, text-based games, or automating repetitive tasks. Gradually move towards larger projects related to your field of interest. Participate in open-source initiatives, contribute to existing repositories, or develop your application ideas. These experiences enhance problem-solving skills and prepare you for professional challenges. Step 5: Master Debugging Techniques Debugging is inevitable when writing code. Identifying and resolving errors early improves productivity and reduces frustration. Use debuggers provided within IDEs, print statements, or log messages to trace execution flow and pinpoint issues. Familiarize yourself with common error types and causes, especially those specific to your chosen language. Over time, proactive debugging becomes second nature. --------
[47]
Query: How do I create a basic webpage? Score: 0.4838 Step 6: Choose a Template After logging in, select a template that suits your preferences by browsing through various categories such as business, personal, blog, etc., located on the left sidebar under "Template Categories". Once you find a suitable design, hover over it and click on the green "Use this template" button below the preview image. Step 7: Customize Your Site You can customize different aspects of your site like its layout, color scheme, background image, font styles, and more via the editor dashboard on the left panel. Remember to save changes made before navigating away from any editing screen. Step 8: Add Pages To add pages to your website, go to the "Pages" tab on the editor dashboard. Here, choose between predefined page types (e.g., Home, About Us, Services) or create custom ones according to your requirements. Don't forget to assign appropriate titles and URL slugs to these pages. -------- Score: 0.4658 Step 9: Edit Page Content For each added page, utilize the content editor to input text, images, videos, links, tables, dividers, or other elements needed. Format texts using headings, bullet points, numbered lists, indentation, alignment, and colors as necessary. Step 10: Configure Settings Navigate to the "Settings" tab on the editor dashboard. Fill in essential details about your site including name, description, keywords, contact info, social media profiles, analytics tracking codes, and SEO settings. Save changes once completed. Step 11: Preview Your Site Before publishing your site, preview it first by clicking on the eye icon near the top right corner of the editor interface. Review your entire site carefully, checking for errors or inconsistencies. If satisfied, proceed to the next step; otherwise, make adjustments accordingly. -------- Score: 0.4604 Title: How to Get a Free Website with Way2Webscape.com Introduction: Way2Webscape.com offers free websites for users who want to create their own online presence without any cost. This tutorial will guide you through the process of getting a free website using this service. By following these steps, you'll be able to create an attractive and functional website that meets your needs. Step 1: Visit Way2Webscape.com Open your web browser and navigate to <https://www.way2webscape.com/>. This will take you to the homepage of Way2Webscape.com. Key tip: Make sure you have a stable internet connection while completing the following steps. Step 2: Click on "Create Your Free Website" Button On the top right corner of the homepage, click on the green button labeled "Create Your Free Website." This action will redirect you to the registration page. --------
[48]
Query: What are some environment-friendly practices I can incorporate in everyday life? Score: 0.6116 a) Use public transportation, carpool, bike, walk, or telecommute whenever possible to minimize fuel consumption. b) If purchasing a vehicle, consider electric or hybrid options that emit fewer greenhouse gases than traditional gasoline-powered cars. c) Improve home energy efficiency through insulation, LED lighting, Energy Star appliances, and renewable energy sources like solar panels. d) Limit air travel and opt for video conferencing when feasible. e) Be mindful of your dietary choices – consume less red meat, choose locally sourced foods, and reduce food waste. f) Plant trees and support reforestation projects as they absorb CO2 from the atmosphere. g) Advocate for policies that promote clean energy and reduced emissions at local, national, and international levels. Key Tip: Calculate your carbon footprint using online tools (such as the EPA's Household Carbon Footprint Calculator) to identify areas where you can make improvements. -------- Score: 0.5926 a) Bring reusable bags, containers, utensils, and water bottles when shopping or dining out. b) Purchase items with minimal packaging, prioritize products made from recycled materials, and avoid single-use plastics. c) Participate in composting programs or start composting at home to turn organic waste into nutrient-rich soil amendments. d) Donate unwanted items rather than discarding them, and repurpose household goods when possible. e) Learn about local recycling rules and sort waste accordingly. Key Tip: Familiarize yourself with your city's waste management system to optimize recycling efforts and divert waste from landfills. Step 4: Protect Biodiversity Biodiversity underpins the health and resilience of ecosystems, providing vital services such as pollination, decomposition, pest control, and nutrient cycling. Safeguard biodiversity through the following actions: -------- Score: 0.5780 Step 2: Conserve Water Freshwater scarcity is becoming increasingly prevalent due to population growth, urbanization, agricultural practices, and climate change. Follow these strategies to conserve water: a) Fix leaks promptly and install low-flow faucets, showerheads, and dual-flush toilets. b) Only run full loads in washing machines and dishwashers. c) Collect rainwater for irrigation purposes and use drought-resistant plants in landscaping. d) Avoid hosing down driveways and sidewalks; instead, sweep debris away. e) Support policies promoting efficient water use and protecting watersheds. Key Guideline: The average American uses about 80-100 gallons of water per day. Strive to reduce this amount through conscious conservation efforts. Step 3: Minimize Waste and Recycle Reducing waste production conserves natural resources, prevents pollution, and reduces strain on landfill capacity. Implement these measures: --------