Notebooks
W
Weaviate Similarity Search Byov Modernbert

Similarity Search Byov Modernbert

vector-searchvector-databaseretrieval-augmented-generationllm-frameworksweaviate-featuresnomic-aifunction-callingweaviate-recipesmodel-providersPythongenerative-ai
alph-notebooks/weaviate-recipes / similarity_search_byov_modernbert.ipynb

Open in Colab

Cover image

Generating Embeddings for Vector Search with ModernBERT in Weaviate

A 100% open source recipe 🧑‍🍳 💚

By Mary Newhauser, MLE @ Weaviate

This is a code recipe that uses Nomic AI's modernbert-embed-base model to generate text embeddings for machine learning papers, inserts them into Weaviate and performs similarity search over the documents.

In this notebook, we accomplish the following:

  • Load and transform the ML-ArXiv-Papers dataset
  • Generate text embeddings for a random sample of 100 articles using sentence-transformers and modernbert-embed-base
  • Perform a basic similarity search over the dataset

About ModernBERT

ModernBERT is the biggest improvement in years to the BERT model. ModernBERT features:

  • 16x longer sequence length
  • Faster inference
  • SOTA performance across tasks like classification and retrieval

For more information, check out Hugging Face's ModernBERT blog post.

Requirements

To run this notebook, we used Python version 3.9.6 and Transformers 4.48.0.

[1]
[2]

Load and transform dataset

[ ]
[4]

The original dataset contains over ~100k titles and abstracts for ML papers from arXiv. For this demo, we'll just take a random sample of 100 papers.

[ ]

Generate embeddings with modernbert-embed-base

We'll use the sentence-transformers library to load and embed the concatenated titles and abstracts with the modernbert-embed-base embedding model, adding them to their own column in the original dataset.

[ ]

Next, we'll convert the dataset to a pandas DataFrame for insertion into Weaviate.

[ ]

Insert the embeddings into Weaviate

Create and configure an embedded Weaviate collection

Embedded Weaviate allows you to spin up a Weaviate instance directly from your application code, without having to use a Docker container.

If you're interested in other deployment methods, like using Docker-Compose or Kubernetes, check out this page in the Weaviate docs.

[ ]

Next, we define the collection and its properties.

[ ]

Finally, we insert the embeddings and metadata into our Weaviate collection.

[11]

Query the data using similarity search

Here, we perform a simple similarity search to return the most similar embedded chunks to our search query.

[12]
[ ]

☁️ Want to scale this notebook?

😍 Get 14 days of free access to Weaviate Cloud's Sandbox by creating an account here.

No name, no credit card required.