Notebooks
M
MongoDB
Openai Embeddings Eval

Openai Embeddings Eval

agentsartificial-intelligencellmsmongodb-genai-showcaseevalsnotebooksgenerative-airag

Open In Colab

View Article

How to choose the right embedding model for your RAG application

This notebook evaluates the text-embedding-3-large model.

Step 1: Install required libraries

  • datasets: Python library to get access to datasets available on Hugging Face Hub
  • openai: Python library to interact with OpenAI APIs
  • numpy: Python library that provides tools to perform mathematical operations on arrays
  • pandas: Python library for data analysis, exploration and manipulation
  • tdqm: Python module to show a progress meter for loops
[49]

Step 2: Setup pre-requisites

Set OpenAI API key as environment variable, and initialize the OpenAI client.

Steps to obtain an OpenAI API Key can be found here

[50]
[51]
OpenAI API Key: ········

Step 3: Download the evaluation dataset

We will use MongoDB's cosmopedia-wikihow-chunked dataset, which has chunked versions of WikiHow articles from the Cosmopedia dataset released by Hugging Face. The dataset is pretty large, so we will only grab the first 2k records for testing.

[52]

Step 4: Data analysis

Make sure the length of the dataset is what we expect (2k), preview the data, drop Nones etc.

[53]
2000
[54]
[55]
[56]
352

Step 5: Creating embeddings

Define the embedding function, and run a quick test.

[57]
[58]
[59]
[60]
3072

Step 6: Evaluation

Measuring embedding latency

Create a local vector store (list) of embeddings for the entire dataset.

[61]
[62]
[63]
  0%|          | 0/2000 [00:00<?, ?it/s]

Measuring retrieval quality

  • Create embedding for the user query
  • Get the top 5 most similar documents from the local vector store using cosine similarity as the similarity metric
[ ]
[43]
[44]
[45]
Query: Give me some tips to improve my mental health.
Score: 0.5319
Step 2: Reach Out to Someone Trustworthy
Connect with someone who cares about you—a friend, family member, mental health professional, or support group. Talking openly about your struggles can lighten your emotional burden, making it easier to manage. Sharing your feelings may also lead to helpful suggestions and advice from others.

Guideline: Make sure to choose someone trustworthy who has shown empathy towards you before. If possible, reach out to more than one person to build a strong support network around you.

Step 3: Engage in Physical Activity
Exercise releases endorphins, which improve mood and reduce stress levels. Go for a walk, jog, bike ride, swim, or engage in any physical activity that suits your abilities. Exercising regularly can significantly impact overall wellbeing by reducing symptoms of depression and anxiety.

Key Tip: Start small - aim for just five minutes of exercise if needed, then gradually increase duration over time. The most important thing is to get started!
--------
Score: 0.5309
Step 2: Improve Your Physical Health
Explanation: Taking care of your physical health can significantly contribute to your overall mood and energy levels. Regular exercise, proper nutrition, and adequate sleep play crucial roles in maintaining your well-being.
Key Tips & Guidelines:

* Exercise regularly (e.g., walking, jogging, yoga, dancing).
* Eat balanced meals rich in fruits, vegetables, lean proteins, and whole grains.
* Limit processed foods, sugars, and unhealthy fats.
* Stay hydrated throughout the day.
* Get at least seven hours of sleep per night.

Step 3: Enhance Your Personal Style
Explanation: Dressing in a way that makes you feel confident and comfortable can boost your self-esteem and reflect your vibrant personality.
Key Tips & Guidelines:
--------
Score: 0.4906
Key Tips & Guidelines:

* Monitor your inner dialogue and identify any recurring negative thoughts.
* Flip the script by reframing those thoughts into more balanced and realistic alternatives. For example, if you think, "I always mess up," try telling yourself, "I made a mistake, but I can learn from it."
* Practice affirmations – short, empowering statements that help build confidence and resilience. Repeat them regularly throughout the day.

Why it works: Negative self-talk reinforces low self-esteem and pessimism, while positive thinking promotes mental wellbeing and contentment.

Step 3: Foster Social Connections
Explanation: Humans are social creatures who thrive on connection and support. Building strong relationships with friends, family, and community contributes to feelings of belonging and happiness.

Key Tips & Guidelines:
--------
[46]
Query: Give me some tips for writing good code.
Score: 0.5893
Step 6: Improve Code Quality
Strive for clean, readable, maintainable code. Adopt consistent naming conventions, indentation styles, and formatting rules. Utilize version control systems like Git to track changes and collaborate effectively. Leverage linters and static analyzers to enforce style guides automatically. Document your work using comments and dedicated documentation tools. High-quality code facilitates collaboration, promotes longevity, and simplifies troubleshooting.

Step 7: Embrace Best Practices
Follow established best practices relevant to your chosen language and domain. Examples include Object-Oriented Design Principles, SOLID principles, Test-Driven Development (TDD), Dependency Injection, Asynchronous Programming, etc. While seemingly overwhelming initially, integrating them gradually enhances design patterns, scalability, and extensibility. Consult authoritative blogs, books, and articles to stay updated on current trends and recommendations.
--------
Score: 0.4586
Conclusion:
Becoming a good programmer requires dedication, persistence, and patience. By methodically progressing through these steps, mastering core concepts, practicing diligently, engaging with peers, and committing to continuous improvement, you'll be well on your way to achieving your goal. Remember, every expert was once a beginner - keep pushing forward!
--------
Score: 0.4490
Step 4: Work On Real Projects
Theory alone won't make you a proficient programmer; hands-on experience does. Apply what you've learned by working on real projects. Begin with small tasks like creating calculators, text-based games, or automating repetitive tasks. Gradually move towards larger projects related to your field of interest. Participate in open-source initiatives, contribute to existing repositories, or develop your application ideas. These experiences enhance problem-solving skills and prepare you for professional challenges.

Step 5: Master Debugging Techniques
Debugging is inevitable when writing code. Identifying and resolving errors early improves productivity and reduces frustration. Use debuggers provided within IDEs, print statements, or log messages to trace execution flow and pinpoint issues. Familiarize yourself with common error types and causes, especially those specific to your chosen language. Over time, proactive debugging becomes second nature.
--------
[47]
Query: How do I create a basic webpage?
Score: 0.4838
Step 6: Choose a Template
After logging in, select a template that suits your preferences by browsing through various categories such as business, personal, blog, etc., located on the left sidebar under "Template Categories". Once you find a suitable design, hover over it and click on the green "Use this template" button below the preview image.

Step 7: Customize Your Site
You can customize different aspects of your site like its layout, color scheme, background image, font styles, and more via the editor dashboard on the left panel. Remember to save changes made before navigating away from any editing screen.

Step 8: Add Pages
To add pages to your website, go to the "Pages" tab on the editor dashboard. Here, choose between predefined page types (e.g., Home, About Us, Services) or create custom ones according to your requirements. Don't forget to assign appropriate titles and URL slugs to these pages.
--------
Score: 0.4658
Step 9: Edit Page Content
For each added page, utilize the content editor to input text, images, videos, links, tables, dividers, or other elements needed. Format texts using headings, bullet points, numbered lists, indentation, alignment, and colors as necessary.

Step 10: Configure Settings
Navigate to the "Settings" tab on the editor dashboard. Fill in essential details about your site including name, description, keywords, contact info, social media profiles, analytics tracking codes, and SEO settings. Save changes once completed.

Step 11: Preview Your Site
Before publishing your site, preview it first by clicking on the eye icon near the top right corner of the editor interface. Review your entire site carefully, checking for errors or inconsistencies. If satisfied, proceed to the next step; otherwise, make adjustments accordingly.
--------
Score: 0.4604
Title: How to Get a Free Website with Way2Webscape.com

Introduction:
Way2Webscape.com offers free websites for users who want to create their own online presence without any cost. This tutorial will guide you through the process of getting a free website using this service. By following these steps, you'll be able to create an attractive and functional website that meets your needs.

Step 1: Visit Way2Webscape.com
Open your web browser and navigate to <https://www.way2webscape.com/>. This will take you to the homepage of Way2Webscape.com.

Key tip: Make sure you have a stable internet connection while completing the following steps.

Step 2: Click on "Create Your Free Website" Button
On the top right corner of the homepage, click on the green button labeled "Create Your Free Website." This action will redirect you to the registration page.
--------
[48]
Query: What are some environment-friendly practices I can incorporate in everyday life?
Score: 0.6116
a) Use public transportation, carpool, bike, walk, or telecommute whenever possible to minimize fuel consumption.
b) If purchasing a vehicle, consider electric or hybrid options that emit fewer greenhouse gases than traditional gasoline-powered cars.
c) Improve home energy efficiency through insulation, LED lighting, Energy Star appliances, and renewable energy sources like solar panels.
d) Limit air travel and opt for video conferencing when feasible.
e) Be mindful of your dietary choices – consume less red meat, choose locally sourced foods, and reduce food waste.
f) Plant trees and support reforestation projects as they absorb CO2 from the atmosphere.
g) Advocate for policies that promote clean energy and reduced emissions at local, national, and international levels.

Key Tip: Calculate your carbon footprint using online tools (such as the EPA's Household Carbon Footprint Calculator) to identify areas where you can make improvements.
--------
Score: 0.5926
a) Bring reusable bags, containers, utensils, and water bottles when shopping or dining out.
b) Purchase items with minimal packaging, prioritize products made from recycled materials, and avoid single-use plastics.
c) Participate in composting programs or start composting at home to turn organic waste into nutrient-rich soil amendments.
d) Donate unwanted items rather than discarding them, and repurpose household goods when possible.
e) Learn about local recycling rules and sort waste accordingly.

Key Tip: Familiarize yourself with your city's waste management system to optimize recycling efforts and divert waste from landfills.

Step 4: Protect Biodiversity
Biodiversity underpins the health and resilience of ecosystems, providing vital services such as pollination, decomposition, pest control, and nutrient cycling. Safeguard biodiversity through the following actions:
--------
Score: 0.5780
Step 2: Conserve Water
Freshwater scarcity is becoming increasingly prevalent due to population growth, urbanization, agricultural practices, and climate change. Follow these strategies to conserve water:

a) Fix leaks promptly and install low-flow faucets, showerheads, and dual-flush toilets.
b) Only run full loads in washing machines and dishwashers.
c) Collect rainwater for irrigation purposes and use drought-resistant plants in landscaping.
d) Avoid hosing down driveways and sidewalks; instead, sweep debris away.
e) Support policies promoting efficient water use and protecting watersheds.

Key Guideline: The average American uses about 80-100 gallons of water per day. Strive to reduce this amount through conscious conservation efforts.

Step 3: Minimize Waste and Recycle
Reducing waste production conserves natural resources, prevents pollution, and reduces strain on landfill capacity. Implement these measures:
--------