Notebooks
M
MongoDB
Gemini Embeddings Eval

Gemini Embeddings Eval

agentsartificial-intelligencellmsmongodb-genai-showcaseevalsnotebooksgenerative-airag

Open In Colab

View Article

How to choose the right embedding model for your RAG application

This notebook evaluates the gemini-embedding-001 model.

Step 1: Install required libraries

  • datasets: Python library to get access to datasets available on Hugging Face Hub
  • google-genai: Google’s GenAI Python SDK
  • numpy: Python library that provides tools to perform mathematical operations on arrays
  • pandas: Python library for data analysis, exploration and manipulation
  • tdqm: Python module to show a progress meter for loops
[25]

Step 2: Setup pre-requisites

Set the Gemini API key as an environment variable, and initialize the Gemini client.

Steps to obtain a Gemini API Key can be found here

[26]
[27]
Gemini API Key: ········

Step 3: Download the evaluation dataset

We will use MongoDB's cosmopedia-wikihow-chunked dataset, which has chunked versions of WikiHow articles from the Cosmopedia dataset released by Hugging Face. The dataset is pretty large, so we will only grab the first 2k records for testing.

[28]

Step 4: Data analysis

Make sure the length of the dataset is what we expect (2k), preview the data, drop Nones etc.

[29]
2000
[30]
[31]
[32]
352

Step 5: Creating embeddings

Define the embedding function, and run a quick test.

[33]
[34]
[35]
[36]
3072

Step 6: Evaluation

Measuring embedding latency

Create a local vector store (list) of embeddings for the entire dataset.

[37]
[38]
[39]
  0%|          | 0/2000 [00:00<?, ?it/s]

Measuring retrieval quality

  • Create embedding for the user query
  • Get the top 5 most similar documents from the local vector store using cosine similarity as the similarity metric
[18]
[19]
[20]
[21]
Query: Give me some tips to improve my mental health.
Score: 0.6919
Key Tips & Guidelines:

* Monitor your inner dialogue and identify any recurring negative thoughts.
* Flip the script by reframing those thoughts into more balanced and realistic alternatives. For example, if you think, "I always mess up," try telling yourself, "I made a mistake, but I can learn from it."
* Practice affirmations – short, empowering statements that help build confidence and resilience. Repeat them regularly throughout the day.

Why it works: Negative self-talk reinforces low self-esteem and pessimism, while positive thinking promotes mental wellbeing and contentment.

Step 3: Foster Social Connections
Explanation: Humans are social creatures who thrive on connection and support. Building strong relationships with friends, family, and community contributes to feelings of belonging and happiness.

Key Tips & Guidelines:
--------
Score: 0.6789
Step 2: Reach Out to Someone Trustworthy
Connect with someone who cares about you—a friend, family member, mental health professional, or support group. Talking openly about your struggles can lighten your emotional burden, making it easier to manage. Sharing your feelings may also lead to helpful suggestions and advice from others.

Guideline: Make sure to choose someone trustworthy who has shown empathy towards you before. If possible, reach out to more than one person to build a strong support network around you.

Step 3: Engage in Physical Activity
Exercise releases endorphins, which improve mood and reduce stress levels. Go for a walk, jog, bike ride, swim, or engage in any physical activity that suits your abilities. Exercising regularly can significantly impact overall wellbeing by reducing symptoms of depression and anxiety.

Key Tip: Start small - aim for just five minutes of exercise if needed, then gradually increase duration over time. The most important thing is to get started!
--------
Score: 0.6748
### Step 5: Stay Active

**Explanation:** Regular exercise is essential for maintaining both physical and mental health. Exercise releases endorphins that boost energy levels, reduce anxiety, and promote better sleep.

* **Key Tips:**
	+ Choose exercises that suit your fitness level and preferences, such as yoga, Pilates, dance classes, or bodyweight training routines.
	+ Allocate time for regular walks outside if possible, taking advantage of parks or scenic routes nearby.
	+ Utilize free resources available online, such as workout videos or apps, to guide your routine.

### Step 6: Maintain Social Connections

**Explanation:** Strong social relationships contribute significantly to emotional well-being. Nurturing connections with friends and family members prevents feelings of isolation and loneliness.
--------
[22]
Query: Give me some tips for writing good code.
Score: 0.7023
Title: How to Become a Good Programmer

Introduction:
Programming is an essential skill in today's digital world. It opens up various opportunities in different industries such as software development, web development, data science, artificial intelligence, and many more. This comprehensive guide will provide you with actionable steps and valuable insights to become a good programmer. By following this tutorial, you'll learn programming fundamentals, best practices, and strategies for continuous improvement.

Step 1: Choose Your Programming Language
Choosing the right programming language is crucial when starting your coding journey. Key factors include your interests, career goals, and available resources. Some popular choices are Python (general-purpose), JavaScript (web development), Java (enterprise applications), or C# (game development). Research and select one that aligns with your objectives. For beginners, Python and JavaScript are great options due to their simplicity and versatility.
--------
Score: 0.6997
Step 6: Improve Code Quality
Strive for clean, readable, maintainable code. Adopt consistent naming conventions, indentation styles, and formatting rules. Utilize version control systems like Git to track changes and collaborate effectively. Leverage linters and static analyzers to enforce style guides automatically. Document your work using comments and dedicated documentation tools. High-quality code facilitates collaboration, promotes longevity, and simplifies troubleshooting.

Step 7: Embrace Best Practices
Follow established best practices relevant to your chosen language and domain. Examples include Object-Oriented Design Principles, SOLID principles, Test-Driven Development (TDD), Dependency Injection, Asynchronous Programming, etc. While seemingly overwhelming initially, integrating them gradually enhances design patterns, scalability, and extensibility. Consult authoritative blogs, books, and articles to stay updated on current trends and recommendations.
--------
Score: 0.6727
Conclusion:
Becoming a good programmer requires dedication, persistence, and patience. By methodically progressing through these steps, mastering core concepts, practicing diligently, engaging with peers, and committing to continuous improvement, you'll be well on your way to achieving your goal. Remember, every expert was once a beginner - keep pushing forward!
--------
[23]
Query: How do I create a basic webpage?
Score: 0.6636
Step 6: Choose a Template
After logging in, select a template that suits your preferences by browsing through various categories such as business, personal, blog, etc., located on the left sidebar under "Template Categories". Once you find a suitable design, hover over it and click on the green "Use this template" button below the preview image.

Step 7: Customize Your Site
You can customize different aspects of your site like its layout, color scheme, background image, font styles, and more via the editor dashboard on the left panel. Remember to save changes made before navigating away from any editing screen.

Step 8: Add Pages
To add pages to your website, go to the "Pages" tab on the editor dashboard. Here, choose between predefined page types (e.g., Home, About Us, Services) or create custom ones according to your requirements. Don't forget to assign appropriate titles and URL slugs to these pages.
--------
Score: 0.6482
Step 9: Edit Page Content
For each added page, utilize the content editor to input text, images, videos, links, tables, dividers, or other elements needed. Format texts using headings, bullet points, numbered lists, indentation, alignment, and colors as necessary.

Step 10: Configure Settings
Navigate to the "Settings" tab on the editor dashboard. Fill in essential details about your site including name, description, keywords, contact info, social media profiles, analytics tracking codes, and SEO settings. Save changes once completed.

Step 11: Preview Your Site
Before publishing your site, preview it first by clicking on the eye icon near the top right corner of the editor interface. Review your entire site carefully, checking for errors or inconsistencies. If satisfied, proceed to the next step; otherwise, make adjustments accordingly.
--------
Score: 0.6396
Step 12: Publish Your Site
Once everything is set up correctly, hit the orange "Publish" button located at the upper right corner of the editor interface. Confirm publication when prompted. Congratulations! Your free website is now live on the internet.

Key Tips & Guidelines:

* Always remember to save changes after making edits.
* Use descriptive titles and URL slugs for better search engine optimization (SEO).
* Utilize header tags (H1, H2, etc.) appropriately for improved readability and ranking.
* Optimize visual content by compressing large files and adding alt attributes.
* Regularly update your site with fresh content to maintain user engagement and improve rankings.
--------
[24]
Query: What are some environment-friendly practices I can incorporate in everyday life?
Score: 0.7323
Title: How to Save Our Environment: A Comprehensive Guide

Introduction:
The environment is the foundation of all life on Earth. It provides us with air to breathe, water to drink, food to eat, and countless other resources that are essential for our survival. However, human activities have led to severe environmental degradation, including climate change, deforestation, pollution, and loss of biodiversity. To ensure a sustainable future for ourselves and generations to come, we must take action now to save our environment. This comprehensive guide offers practical steps you can take to reduce your impact on the planet and contribute to global efforts towards sustainability.

Step 1: Reduce Your Carbon Footprint
A carbon footprint refers to the total greenhouse gas emissions produced directly or indirectly by an individual, organization, event, or product. By reducing your carbon footprint, you help mitigate climate change and its devastating impacts on ecosystems and communities worldwide. Here's how:
--------
Score: 0.7112
a) Use public transportation, carpool, bike, walk, or telecommute whenever possible to minimize fuel consumption.
b) If purchasing a vehicle, consider electric or hybrid options that emit fewer greenhouse gases than traditional gasoline-powered cars.
c) Improve home energy efficiency through insulation, LED lighting, Energy Star appliances, and renewable energy sources like solar panels.
d) Limit air travel and opt for video conferencing when feasible.
e) Be mindful of your dietary choices – consume less red meat, choose locally sourced foods, and reduce food waste.
f) Plant trees and support reforestation projects as they absorb CO2 from the atmosphere.
g) Advocate for policies that promote clean energy and reduced emissions at local, national, and international levels.

Key Tip: Calculate your carbon footprint using online tools (such as the EPA's Household Carbon Footprint Calculator) to identify areas where you can make improvements.
--------
Score: 0.7055
Step 2: Conserve Water
Freshwater scarcity is becoming increasingly prevalent due to population growth, urbanization, agricultural practices, and climate change. Follow these strategies to conserve water:

a) Fix leaks promptly and install low-flow faucets, showerheads, and dual-flush toilets.
b) Only run full loads in washing machines and dishwashers.
c) Collect rainwater for irrigation purposes and use drought-resistant plants in landscaping.
d) Avoid hosing down driveways and sidewalks; instead, sweep debris away.
e) Support policies promoting efficient water use and protecting watersheds.

Key Guideline: The average American uses about 80-100 gallons of water per day. Strive to reduce this amount through conscious conservation efforts.

Step 3: Minimize Waste and Recycle
Reducing waste production conserves natural resources, prevents pollution, and reduces strain on landfill capacity. Implement these measures:
--------