Scalable late interaction vectors in Elasticsearch: Average Vectors

In this notebook, we will be looking at how scale search with late interaction models. We will be taking the average vector over our late interaction multi-vectors and use Elasticsearchs vector search capabilities to achieve scalable search over billions of vectors.

This notebook builds on part 1 where we downloaded the images, created ColPali vectors and saved them to disk. Please execute this notebook before trying the techniques in this notebook.

Also check out our accompanying blog post on Scaling Late Interaction Models for more context on this notebook.

This is the key part of this notebook. We use to_avg_vector(vectors) to convert our 2d array into a single vector that holds the "average meaning" of the document.

Here we are defining our mapping for our Elasticsearch index. Note how we are using the dense_vector field type for our average vector.
This allows us to leverage the highly optimized HNSW indexing structures in Elasticsearch with which we can scale our search to billions of documents.

Here we are looping over all our vectors and convert them into our average vectors.
We still save our full fidelity ColPali vectors as bit vectors as we can use them for reranking. More on that later.

Again we create our query vector by using the ColPali model and calculating the average vector. We can now use Elasticsearches knn query to compare our single vector to the average vectors within our index. Notice that the document that we have found in our previous examples with the title Social Media for Recruitment is now in 5th position. See the next cell on how we can handle this.

In the cell above we have seen that the document Social Media for Recruitment is not considered the most relevant anymore.

What we want to do to handle this is to run our KNN vector stage as a first stage retrieval algorithm. This is the knn retriever in the example below. We then run a first stage retrieval rescoring algorithm on a smaller set of these documents. For this we can use higher fidelity ColPali vectors. This is the rescore section in the example below.