Scalable late interaction vectors in Elasticsearch: Token Pooling

In this notebook, we will be looking at how scale search with late interaction models. We will be looking a token pooling - a technique to reduce the dimensionality of the late interaction multi-vectors by clustering similar information. This technique can of course be combined with the other techniques we have discussed in the previous notebooks.

This notebook builds on part 1 where we downloaded the images, created ColPali vectors and saved them to disk. Please execute this notebook before trying the techniques in this notebook.

Also check out our accompanying blog post on Scaling Late Interaction Models for more context on this notebook.

We will be using the HierarchicalTokenPooler from the colpali-engine to reduce the dimensions of our vector.
The authors recommend a pool_factor=3 for most cases, but you should always tests how it impact the relevancy of your dataset.