INFO:weaviate-client:Started /Users/dudanogueira/.cache/weaviate-embedded: process ID 83300
Started /Users/dudanogueira/.cache/weaviate-embedded: process ID 83300

{"action":"startup","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-11-13T15:16:51-03:00"}
{"action":"startup","auto_schema_enabled":true,"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"module offload-s3 is enabled","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"warning","msg":"Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"open cluster service","servers":{"Embedded_at_8079":51826},"time":"2024-11-13T15:16:51-03:00"}
{"address":"192.168.28.127:51827","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"starting cloud rpc server ...","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"starting raft sub-system ...","time":"2024-11-13T15:16:51-03:00"}
{"address":"192.168.28.127:51826","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"loading local db","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"local DB successfully loaded","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"schema manager loaded","n":0,"time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"Embedded_at_8079","time":"2024-11-13T15:16:51-03:00"}
{"action":"raft","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","index":154,"level":"info","msg":"raft initial configuration","servers":"[[{Suffrage:Voter ID:Embedded_at_8079 Address:192.168.28.127:51713}]]","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","last_snapshot_index":0,"last_store_applied_index_on_start":157,"level":"info","msg":"raft node constructed","raft_applied_index":0,"raft_last_index":157,"time":"2024-11-13T15:16:51-03:00"}
{"action":"raft","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"raft entering follower state","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","hasState":true,"level":"info","msg":"raft init","time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"attempting to join","remoteNodes":["192.168.28.127:51826"],"time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"attempted to join and failed","remoteNode":"192.168.28.127:51826","status":8,"time":"2024-11-13T15:16:51-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"attempting to join","remoteNodes":["192.168.28.127:51826"],"time":"2024-11-13T15:16:52-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"attempted to join and failed","remoteNode":"192.168.28.127:51826","status":8,"time":"2024-11-13T15:16:52-03:00"}
{"action":"raft","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","last-leader-addr":"","last-leader-id":"","level":"warning","msg":"raft heartbeat timeout reached, starting election","time":"2024-11-13T15:16:52-03:00"}
{"action":"raft","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"raft entering candidate state","node":{},"term":46,"time":"2024-11-13T15:16:52-03:00"}
{"action":"raft","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"raft pre-vote successful, starting election","refused":0,"tally":1,"term":46,"time":"2024-11-13T15:16:52-03:00","votesNeeded":1}
{"action":"raft","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"raft election won","tally":1,"term":46,"time":"2024-11-13T15:16:52-03:00"}
{"action":"raft","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","leader":{},"level":"info","msg":"raft entering leader state","time":"2024-11-13T15:16:52-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"reload local db: update schema ...","time":"2024-11-13T15:16:52-03:00"}

INFO:httpx:HTTP Request: GET http://localhost:8079/v1/.well-known/openid-configuration "HTTP/1.1 404 Not Found"
HTTP Request: GET http://localhost:8079/v1/.well-known/openid-configuration "HTTP/1.1 404 Not Found"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/meta "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/meta "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/.well-known/ready "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/.well-known/ready "HTTP/1.1 200 OK"

{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","docker_image_tag":"localhost","level":"info","msg":"configured versions","server_version":"1.26.6","time":"2024-11-13T15:16:53-03:00"}
{"action":"grpc_startup","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"grpc server listening at [::]:50050","time":"2024-11-13T15:16:53-03:00"}
{"address":"192.168.28.127:51826","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"current Leader","time":"2024-11-13T15:16:53-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"starting migration from old schema","time":"2024-11-13T15:16:53-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"legacy schema is empty, nothing to migrate","time":"2024-11-13T15:16:53-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"migration from the old schema has been successfully completed","time":"2024-11-13T15:16:53-03:00"}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"attempting to join","remoteNodes":["192.168.28.127:51826"],"time":"2024-11-13T15:16:53-03:00"}
{"action":"raft","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","command":0,"level":"info","msg":"raft updating configuration","server-addr":"192.168.28.127:51826","server-id":"Embedded_at_8079","servers":"[[{Suffrage:Voter ID:Embedded_at_8079 Address:192.168.28.127:51826}]]","time":"2024-11-13T15:16:53-03:00"}
{"action":"restapi_management","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","docker_image_tag":"localhost","level":"info","msg":"Serving weaviate at http://127.0.0.1:8079","time":"2024-11-13T15:16:53-03:00"}

INFO:httpx:HTTP Request: GET https://pypi.org/pypi/weaviate-client/json "HTTP/1.1 200 OK"
HTTP Request: GET https://pypi.org/pypi/weaviate-client/json "HTTP/1.1 200 OK"

{"action":"telemetry_push","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:aaa598be-f7d1-48c4-9981-8eabbeee03a0 Type:INIT Version:1.26.6 NumObjects:0 OS:darwin Arch:arm64 UsedModules:[]}","time":"2024-11-13T15:16:53-03:00"}

INFO:httpx:HTTP Request: GET http://localhost:8079/v1/meta "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/meta "HTTP/1.1 200 OK"
Client: 4.9.3, Server: 1.26.6

INFO:httpx:HTTP Request: DELETE http://localhost:8079/v1/schema/BlogPost "HTTP/1.1 200 OK"
HTTP Request: DELETE http://localhost:8079/v1/schema/BlogPost "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:8079/v1/schema "HTTP/1.1 200 OK"
HTTP Request: POST http://localhost:8079/v1/schema "HTTP/1.1 200 OK"
Collection was created.

{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"warning","msg":"prop len tracker file /Users/dudanogueira/.local/share/weaviate/blogpost/Yenb2lylXSIW/proplengths does not exist, creating new tracker","time":"2024-11-13T15:17:01-03:00"}
{"action":"hnsw_prefill_cache_async","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"not waiting for vector cache prefill, running in background","time":"2024-11-13T15:17:01-03:00","wait_for_cache_prefill":false}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"Created shard blogpost_Yenb2lylXSIW in 3.126083ms","time":"2024-11-13T15:17:01-03:00"}
{"action":"hnsw_vector_cache_prefill","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","count":1000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-11-13T15:17:01-03:00","took":95583}

INFO:httpx:HTTP Request: DELETE http://localhost:8079/v1/schema/PodCast "HTTP/1.1 200 OK"
HTTP Request: DELETE http://localhost:8079/v1/schema/PodCast "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:8079/v1/schema "HTTP/1.1 200 OK"
HTTP Request: POST http://localhost:8079/v1/schema "HTTP/1.1 200 OK"
Collection was created.

{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"warning","msg":"prop len tracker file /Users/dudanogueira/.local/share/weaviate/podcast/PgqgLoHorvKd/proplengths does not exist, creating new tracker","time":"2024-11-13T15:17:03-03:00"}
{"action":"hnsw_prefill_cache_async","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"not waiting for vector cache prefill, running in background","time":"2024-11-13T15:17:03-03:00","wait_for_cache_prefill":false}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"Created shard podcast_PgqgLoHorvKd in 3.057791ms","time":"2024-11-13T15:17:03-03:00"}
{"action":"hnsw_vector_cache_prefill","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","count":1000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-11-13T15:17:03-03:00","took":71375}

INFO:httpx:HTTP Request: DELETE http://localhost:8079/v1/schema/MeetingNotes "HTTP/1.1 200 OK"
HTTP Request: DELETE http://localhost:8079/v1/schema/MeetingNotes "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:8079/v1/schema "HTTP/1.1 200 OK"
HTTP Request: POST http://localhost:8079/v1/schema "HTTP/1.1 200 OK"
Collection was created.

{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"warning","msg":"prop len tracker file /Users/dudanogueira/.local/share/weaviate/meetingnotes/AvU2hkhsYsdf/proplengths does not exist, creating new tracker","time":"2024-11-13T15:17:08-03:00"}
{"action":"hnsw_prefill_cache_async","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"not waiting for vector cache prefill, running in background","time":"2024-11-13T15:17:08-03:00","wait_for_cache_prefill":false}
{"build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","level":"info","msg":"Created shard meetingnotes_AvU2hkhsYsdf in 3.211ms","time":"2024-11-13T15:17:08-03:00"}
{"action":"hnsw_vector_cache_prefill","build_git_commit":"ab0312d5d","build_go_version":"go1.23.1","build_image_tag":"localhost","build_wv_version":"1.26.6","count":1000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-11-13T15:17:08-03:00","took":77000}

title: Multi-Tenancy Vector Search with millions of tenants


Large-scale setups were always a great reason to choose Weaviate. Last year we wrote about the first time [a Weaviate setup ran with a billion objects & vectors](/blog/sphere-dataset-in-weaviate). What was a mere experiment back then is a regular production case today. But earlier this year, we saw a shift in usage patterns: As we onboarded more and more large-scale and enterprise users, the definition of scale shifted from the number of vectors to the number of individual tenants that can run on a single setup.

Previously, Weaviate offered multiple ways to tackle multi-tenancy, but none were intended for a massive scale. Weaviate `v1.20` - coming in July 2023 - changes this once and for all: Native multi-tenancy support that scales to millions of tenants with 10s of thousands of active tenants per node. Yet scale is not the only point that makes the new multi-tenancy feature great; we put a lot of emphasis on compliance and a smooth UX. GDPR-compliant deletes with one command are just one of the many features. Let me walk you through what’s coming in the next Weaviate release and show you why I’m incredibly excited about this one.

## The need for multi-tenancy
We define multi-tenancy as the need to serve multiple distinct users or user groups from a single application. Imagine a fictional company called ACME Accounting Group that offers online accounting services that use AI to make accounting easy and fun. The company has over one million customers. Each customer is a company that can have many users and even more documents. Alice, who works for AliceCorp, should never be able to see the accounting information of Bob, who works for BobInc. Therefore, AliceCorp and BobInc are tenants from the perspective of ACME Accounting. Besides access isolation, ACME Accounting has other requirements for a multi-tenancy setup:

* **Speed**: With millions of tenants, narrowing a request down to a single tenant should not take much work.

* **Easy on and offboarding**: Just because ACME Accounting already has a million customers doesn’t mean that adding your business to ACME should take a considerable computational load. If AliceCorp cancels its contract, this should not affect other tenants.

* **Resource boundaries**: If members of BobInc all get together to produce their annual report, this can put a lot of load onto ACME’s system. This should not interfere with AliceCorp, which might also have essential accounting deadlines.

* **Cost-efficiency**: With usage peaking around tax season but most tenants inactive for many days of the month, ACME shouldn’t have to pay for a large setup that is essentially idle most of the time.

* **Diversity of tenants**: ACME has both large and small customers. Their setup needs to be able to handle tenants of vastly different sizes. While most tenants are small, a few tenants can make up a large bulk of the data.

## A time before multi-tenancy support
Before Weaviate `v1.20`, you had two options to model a multi-tenancy landscape. Both had considerable drawbacks which made us completely rethink multi-tenancy:

### Using Classes
Classes were already isolation units within Weaviate, so you were already halfway there: Good isolation, easy deletes, and no filter required to scope a request to a tenant. However, there was one major problem: With classes, it was incredibly difficult to run with more than 5 to 10,000 tenants. Workarounds were needed to make it scale further, and there was a lot of duplication in the schema.

### Using filters
Another approach was to use a single class and use Weaviate’s built-in filtering feature to scope a request to a tenant. However, this came with multiple problems. From a performance perspective, you would build a giant monolithic vector index with potentially billions of vectors, yet you would only ever query a tiny fraction of it. With a median tenant storing between 1,000 and 100,000 objects, you would typically query less than 0.01% of the index. What a waste of resources. Additionally, dropping many tenants simultaneously would lead to expensive repair operations on the shared index. Resource isolation for tenants was also not possible.

## No more workarounds: Native multi-tenancy support for the largest of cases

![multi-tenancy](./img/multi-tenancy-dark.png#gh-dark-mode-only)
![multi-tenancy](./img/multi-tenancy-light.png#gh-light-mode-only)

Let me walk you through the best parts of the native multi-tenancy feature in Weaviate `v1.20` and even some of the plans that we have for upcoming releases. As you will see, Multi-Tenancy is not just an additional feature. It is rethinking from the ground up how your business, with its many customers, scales with Weaviate.

### Ease of use of your favorite Weaviate features
Weaviate’s new multi-tenancy mode is opt-in. If you don’t require it, there are no changes for you. But if you have a multi-tenancy case, all you need to do is select multi-tenancy for your class (or classes). You can keep using all your favorite features, including super-fast queries, cross-references, and replication. The only change? For every operation, specify the tenant key – a user-defined property in your schema. That’s all that changes. Weaviate will handle the rest for you.

### Built-in isolation through lightweight shards
To make sure we have isolation for both technical and compliance reasons, Weaviate must also store the data per tenant separately. This allows quick deletes, reduces resource sharing, restricts access, and more. Weaviate uses partition shards for this. With `v1.20`, shards have become a lot more lightweight because we expect you to have a lot of them. You can easily run with 50,000+ active shards per node. With just 20 nodes, you can support 1M concurrently active tenants. And with support for inactive tenants (coming soon - read below), there is no limit per node at all.

### Fast and efficient querying
You don’t need to set a filter to restrict a query to a tenant. The simple addition of a tenant key is enough, and Weaviate will find the right tenant’s shard for you. More importantly, every tenant has a dedicated high-performance vector index providing query speeds as if the tenant was the only user on your cluster. With more features in the pipeline, such as tenant-specific rate limiting or tenant-specific replication factors, you can customize performance per tenant even further.

### GDPR-compliant and efficient deletes
When discussing solutions made for many users, our first intuition is to worry about how we onboard and serve them. But deleting them is equally important – both for technical and legal reasons. Take GDPR as an example. Can you guarantee that all data related to a specific user is gone? With Weaviate’s multi-tenancy compliant deletes, you can! All data specific to a tenant is isolated in a dedicated shard. Deleting the entire tenant is as easy as deleting a file from your system.

But deleting tenants is also interesting from a technical perspective; certain events can lead to mass onboarding followed by mass offboarding. For example, when your app goes viral on social media, you might get a lot of sign-ups for a free trial. While we wish you a high conversion rate, some of your trial users must be removed after 2 weeks. This should be quick and painless. It definitely shouldn’t affect your users who stay onboard. With a shared single index, it would. With Weaviate’s multi-tenancy isolation, it won’t.

### Massive Scale
Weaviate’s multi-tenancy support was built to scale with your business from the ground up. Supporting over 50,000+ active tenants per node, you need just a 20-node cluster for 1 million active tenants with billions of vectors in total. Run out of space, but want to onboard more tenants? Simply add a new node to your existing cluster. Weaviate will automatically schedule new tenants on the node with the least resource usage. With Weaviate’s rebalancing features (coming soon), you can ensure tenants are distributed across nodes exactly how you want them to – or leave it to Weaviate to do it for you.

### Active and inactive tenants
With traditional search and other vector search solutions, infrastructure is typically sized for the number of objects or vectors that could be served. But why should you pay for expensive compute and memory resources for users who aren’t currently active? Enabled by Weaviate’s strict isolation between tenants, Weaviate allows you to distinguish between active and inactive tenants. A user hasn’t been logged in for an hour? Their tenant shouldn’t take up valuable memory or file descriptors. A user hasn’t been logged in for days? Let’s offload their data to cloud storage and reload it on demand. Whether you want to control which of your users should be active or let Weaviate decide to activate and deactivate tenants automatically, you’ll save on infrastructure costs for sure!

*Support for inactive tenants is coming later in 2023.*

## We couldn’t have done it without you
We know we built exactly what you need the most because you helped us design it. We’re incredibly grateful for all the community support and feedback, the out-in-the-open design discussions, and the valuable feedback from our existing and future customers. I can’t wait to demo the best parts of the Weaviate multi-tenancy future to all of you. Whether you’ve been part of the design discussions or are hearing about this for the first time, we appreciate all your feedback, so drop us a note with your thoughts and questions.  I’m incredibly proud of what our team has built and can’t wait for July 2023 when we go live with Weaviate `v1.20`!


import WhatNext from '/_includes/what-next.mdx'

<WhatNext />

"thank you so much for watching the wevia\npodcast I'm super excited to welcome we\nV8 CTO and co-founder Eddie and dilocker\nfor the weeviate 1.20 release podcast\nthis is another packed release Abate\nwith all sorts of cool things\nmulti-tenancy PQ rescoring Auto cut\nre-rankers a new hybrid uh rank Fusion\nalgorithm and some Cloud monitoring\nmetrics every time we do these release\npodcasts it's always so much fun I\nalways learn so much so Eddie and\nfirstly thank you so much for joining\nthe podcast\nthanks so much for having me same for me\nI love talking about these things I love\nthe the great questions that you always\nprepare and I'm very very excited as\nwell to talk about this one\nyeah amazing yeah I love just like the\nbreadth of it going through all the\ndifferent topics in weaviate and so\nstarting off with I think just a super\nexciting topic diving into the database\nthing multi-tenancy can you tell us uh\nmaybe just to begin in the highest level\nabstraction like the overview of what\nmulti-tenancy is yeah yeah what it is\nand sort of why why you need it even in\nthe first place so multi-tenancy uh for\nus and I always feel like I need to sort\nof because when people hear\nmulti-tenancy they think of cloud\noperations sort of share resources and\nCloud operations and of course we have a\ncloud service so multi-tenancy for us is\nnot about how we run the VB account\nservice yes you can use multi-tenancy on\nthe cloud service but it's actually\nabout multi-tenancy for you the user so\nlet's say you have an application your\napplication has separate users and they\nhave somehow data that needs to be\nseparated from one way or another so for\nexample let's say you build an app and\nyour app allows you to index documents\nthat you have on your hard drive so\nmaybe just sort of install something on\nyour let's say something like Dropbox or\nso you want to search to all your your\ndocuments you only want to be able to\nsearch through them yourself you\ndefinitely don't want other users who\nhappen to be using Dropbox to be able to\nsearch through your documents so that's\nkind of the the idea of scoping that to\nto individual tenants so you as a\nDropbox user in this case would be a\ntenant or it could be that multiple\nusers maybe instead of Dropbox it would\nbe a workspace on a notion or an\natlassian Confluence or something so in\nthis in this setup basically a group of\nusers could be attended so it doesn't\nnecessarily have to be an individual\nuser but it needs to be some kind of\nisolation unit basically and this is\nthis is so far this is not even like a\ntechnical requirement that's essentially\njust an application Level how do you\ncurate what do you have access to\nrequirement uh but then where this gets\nsuper interesting for Vector search is\nthat it kind of mixes with the technical\nrequirements and it's almost like like\nit's sort of it perfectly aligns because\nin in Vector search we have the problem\nthat we need to somehow figure out how\nto sort of limit the vector space and\nwe've talked about H and SW the sort of\nthe indexing graph in the past and now\nthink of this whole graph that contains\nmaybe a billion vectors but these\nbillion vectors are spread out over a\nmillion tenants now you would have to\nsort of assuming there was no\nmulti-tenancy you would have to\nessentially cut that graph into a very\nsmall chunk that only contains about a\nthousand objects each though for a\nmillion tenants have a total of billion\nobjects that's only a thousand each uh\nchances are this graph becomes either it\nbecomes disconnected or you have to\ntravel a lot through that graph without\nsort of hitting notes basically that\nyou're not allowed to hit or you're not\nsupposed to hit so this single graph and\nfilter kind of approach is at best the\nworkaround like you can we have this in\nmediate we have this flat surge cutoff\nbasically where if the the filter\nbecomes too specific you actually do a\nflat search so this would be one way to\nwork around it but then um you sort of\nlose the whole benefits of of uh the\nfast paid or the high throughput and low\nlatency kind of search that you expect\nfrom hnsw so there is a need to to\nbasically also do this kind of\nseparation from a technical level and\nthis is where we said okay enough\nworkarounds another work around that\nthat users have semi-successfully used\nin the past was to separate this on a\nclass level because in deviate a class\nis already a an isolation unit so you\ncould sort of uh\nsay per tenant you create one class and\nthey would all have all these classes\nwould have an identical schema because\nthey're it's all the same use case\nbasically you just copy it for\nindividual users so your schema keeps on\ngrowing and growing and growing and like\neach schema update made the whole thing\nslower and slower and this worked\nokayish for maybe two to three maybe\nfive thousand tenants and one workaround\nand then really we're talking about work\nround after work round uh another one\nwas then you could turn off graphql\nbecause part of the the part of\nrebuilding the schema part of what took\nso much time was rebuilding graphql so\nyou could turn off graphql only use grpc\nthat would make it scale a bit farther\nbut we were really in the territory of\nlike this is this is not a long-term\nsolution this is this is happening from\none workaround to another so we said we\nreally want a dedicated multi-tenancy\nsolution where the apis support tenants\nwhere uh the the architecture under the\nhood supports a lot of tenants and most\nimportantly where this also somehow\nscales linearly and we can talk a bit\nmore about about scaling\nyeah it's so interesting the um I\nremember when I first heard the question\nabout doing this kind of thing I heard\nuh you know I was at the Meetup in New\nYork City and someone said you guys\nsupport our back role-based access\ncontrol and you know the time I'm\nthinking also that you could just have\nthat filter through your class where you\nif you have like you know a document\nclass and you have content you have user\nand you know user edian user Connor to\nonly look at like Conor's emails instead\nof Connor seeing edian's emails and so\nhearing about the limitation of that as\nyou know if you connect to hsw graph the\nfilter it might not be connected still\nso you need to modify hsw itself um so\nas you were giving that explanation it\nreally helped something click for me is\nthe the difference between just kind of\nnaively using multiple classes in weave\nas we understand it you know like I have\nan Eddie in class I have a Conor class I\nhave a John class but so can you tell me\na little more about the design of\nmulti-tenancy and how you have native\nmulti-tenancy and I think really to uh\nseparate these two things hopefully it's\nnot a selfish question from my\nunderstanding but this the difference\nbetween just creating a bunch of classes\nand you know maybe at 1.19 compared to\nthe native multi-tenancy in 1.20\nabsolutely this is definitely not a\nselfish question I think our viewers and\nlisteners will will absolutely\nappreciate that as well so um the\nthis class-based workaround kind of work\nbecause one thing that a class already\ndoes is it creates something like some\nseparate space somewhere uh basically in\nthe class and internally in V8 this is a\nchart and within one class you could\nhave anywhere between so so this is in\nthe traditional mode without\nmulti-tenancy you could have anywhere\nbetween one and any number of shards and\nA Shard is essentially you can think of\neverything that's in the databases\ncontained in that one chart and whenever\nthere are two shards and you want to\nmaybe query across two shards then under\nthe hood this is split into two queries\neach Shard does their part on their own\nand then somehow aggregated again\num so this is why this workaround kind\nof work because by creating 10 classes\nyou also under the hood created 10\ncharts\num it was kind of doing a lot of\noverhead for essentially you just wanted\nto end up with 10 charts but you could\nkind of do it with with creating 10\nclasses the multi-tenancy feature the\nnative multi-tenancy feature in the\nsimplest terms you can think of it it's\na single class but within that class we\ncreate one chart per tenant so shards\nare now no longer this static thing but\nthey're completely Dynamic like you can\nadd them on the Fly you can delete them\non the Fly and\num the the kind of\ncluster association with a chart still\nholds true so I need to need to explain\na bit more for this probably so in a vv8\ncluster let's say you have a cluster of\nthree notes let's make it simple like\nmulti-10 so you can go hundreds of notes\nbut let's keep it simple three notes and\nlet's say that in our class example uh\nyou would have each class would just\nhave one shard\num A Shard is something that that can't\nbe split further basically like if you\nwant to split it you need you need more\ncharts so in that old setup this chart\nwith the specific configuration would\nlive on one of those exactly three notes\nso one way of Distributing this around\nthe the setup of it would be if you have\n30 tenons or sorry in the old set of 30\nclasses I mean you would have 30 tens\nbut you would model them with 30 classes\neach node could hold 10 of those\nbasically\nand then\nthat's that that would sort of be evenly\ndistributed\num but you'd have very little little\ncontrol now with multi-tenancy we keep\nthat idea of having one chart\num but the shards have become much more\nlightweight so you have essentially\nwe've run a couple of load tests and um\nwe could in one example this depends a\nbit on on sort of what you actually what\nkind of properties you have in your\nclass and these kind of things\num and essentially the bottleneck is\njust the file descriptor limit that that\nLinux systems have and one test we could\nreach 70 000 charts per note and per\nnode is now where this this sort of very\ninteresting part comes in because you\ncan just scale this by adding more and\nmore notes to your cluster so\num you would have a single class that\nclass potentially spans the entire\ncluster but a single tenant still is\nisolated to one node so this this and\nand to make this a bit more complex we\ncould then also add replication because\nthen you can make sure that this node\ndoesn't become a single point of failure\nbut for now let's just ignore ignore\nreplication so you could have you could\nstart with a three node cluster let's\nsay your tenons are really huge huge and\nyou would only fit 10 per per\nnote then you could fill up to 30\ntenants on that cluster now if you want\nto onboard more users you just add a new\nnote and per note you would again have\nroughly 10 tenants capacity and in\nreality it's more like 10 000 but for\nfor our example that\nthat um yeah makes it easier to to\nreason about\nand then vv8 under the hood make sure\nthat you hit the right note so of course\nas a user you don't know where that\nstuff is scheduled but will be able to\nsay like with every and this is the the\nonly real API change for multi-tenancy\nis that now you have to specify your\ntenant key so you don't have to use a\nfilter basically\num in your let's say you do a a get near\ntext search then you just have an\nadditional property that's the tenant\nand then you should specify that and vv8\nuses that under the hood to figure out\nsort of where in that three or five or\nfive hundred node cluster where does the\ntenant actually live\nso then um so so I'm curious now about\nkind of maybe the design decisions\nbehind the tenant key and kind of the\nthe reduction of the sharding the sorry\nlike taking down the size of each Shard\nmaybe we could step more into the\ntechnical details behind what it means\nto have a Shard be lighter weight and\ndynamic\nyeah yeah yeah so\num the the idea of splitting\nby Shard or by class or by index type\nfor multi-tenancy that is not new I know\nfor example that um for for if you need\nto be gdpr compliant and traditional\nsearch engines such as elasticsearch you\nwould also try to use that same pattern\nwhere you would create an index I think\nit's called an elasticsearch per 10 and\nthen to have that strict isolation so\nthat is kind of where also this is\nessentially our class-based workaround\num but this always yeah sort of comes\nwith with limits so what we said is we\nneed to make the chart more lightweight\nand lightweight making it more\nlightweight is essentially sort of a an\numbrella term for all the kind of things\nthat kept us from\num from having sort of a lot of charts\non the on the uh on OneNote one thing\nfor example is asynchronous processes so\nsince A Shard is its own contained unit\nin the past we would have lots of async\nprocesses so so for example for uh the\nagents W index for maintenance that\nwould be an async process for every\nproperty that's stored in an LSM store\none async process would be to switch uh\nfrom ment table to segment so when a mem\ntable is flushed that's essentially the\nmemory storage is then to disk then you\nhave compactions in the background so\nyou have all these kind of sort of\nbackground processes that tend to be\nrelatively lightweight but let's say you\nhave 10 per of them per Shard and now\nyou have 10 000 charts because you have\n10 000 tenants now all of a sudden you\nwould have a hundred thousand of those\nbackup processes and in our very first\ntest before adapting anything we could\nsee that essentially all CPU time was\nnow spent on just idle background\nprocesses doing nothing and then to make\nthis even worse\nsome of those doing nothing kind of\nprocesses uh for example the ones that\nwould uh check for whether a\nLSM segments need to be compacted they\ndid that using a discrete so a very\ninnocent sort of simple discrete of hey\nwhat is the state on disk right now but\nnow this happens a hundred thousand\ntimes in parallel and now you hit your\ndisk with all these unnecessary reads\njust to find out that you don't have to\ndo anything so very simple changes such\nas sort of yeah making sure can we cache\nsome of that information can we check\nless frequency less frequently if there\nhasn't been a change in that much time\nuh can we yeah so if all these these\nkind of kind of can we combine uh\ninternally I think this thing is called\nthe cycle manager and there's one\ndiscussion that we have like instead of\nhaving 10 per chart could we maybe have\njust one or could we have less than one\nbecause we take that outside of the\nchart so all these kind of optimizations\num that enable us to run more charts\nunder hood I'm essentially making them\nthem more lightweight\num another thing is also the memory\nfootprint so in our very first test\num we had a surprisingly large memory\nfootprint for an idle chart don't\nremember what it was but I think it was\nsomething around five megabytes or so\nwhere in a single class single chart\nsetup this would just never like you\nwouldn't even notice that that like you\nhave one class with one chart and now\nyou have five megabyte of memory usage\nnot a lot but now again times a hundred\nthousand or times ten thousand all of a\nsudden you have this like 50 gigabyte of\nidle memory for or not idle memory 50\ngigabyte of use memory for what is\nessentially idle classes so that that\nwas another optimization again sort of a\nvery simple optimization which is to to\nmake this more more lightweight just\nlook at what kind of memory how are we\nallocating memory how are we doing this\ndynamically are we sort of are we a bit\ntoo optimistic about where the the chart\nis going to grow to and just sort of\nsetting more reasonable defaults making\nsure it can still grow so there's\nessentially no no negative user impact\neverything can still grow but just the\ndefaults are more reasonable and more\naligned for for having many of them and\nthat's kind of what we mean by by making\nthem more lightweight\namazing that's such a clear explanation\nof it and um yeah it's it's really\ninteresting hearing about these\nbackground processes\num you know hearing about the Compact\nand merge I don't know too much about\nthe LSM myself but that explanation I\ncan understand how there would be you\nknow background processes that check on\nthe database and that kind of thing so\num yes you mentioned the um you know\nseeing the five megabytes of overhead\nper class and you know kind of the\ninsights that you gain by testing it I\nthink that transitions nicely into this\nnext question that I'm uh how have you\nbeen testing multi-tenancy and I guess\nit's kind of I think like some like what\nare the lessons from it like it sounds\nlike with the megabyte thing you you\nknow even though you have such a deep\nunderstanding of weeviate internals you\nstill learn from your tests yeah that\nmakes I'm curious like how this feedback\nprocess of testing it how exactly it's\ntested and then what has been\nilluminated from the tests oh yeah\nabsolutely this this feedback cycle goes\nsort of Beyond just the the test it's\nalso it's user feedback like I think the\nthe whole journey of getting into\nmulti-tenancy started out with user\nfeedback saying like hey I'm trying to\napply these workarounds but now I'm\nseeing large memory footprint or now I'm\nseeing seeing\nstuff slowing down or emptying my disk\nbeing hit even though I'm not querying\nand these kind of things and and that\nwas I think in multi titanically that\nwas the first thing that we that even\nmade us aware of hey there is a need for\nsomething new there's a need for for a\nrevolutionary change basically not just\nnot just extending the the workarounds\nand then\num I think they're in testing you could\nsay that there were two major faces so\nthe first one was initially when we\nstarted out this was in proposal phase\nand we just wanted to say like is this a\nviable idea we have the proposal out on\nbecause we do this out in the open of\ncourse we have that out on on GitHub and\njust it was sort of a mix of this is why\nwe think it's technically feasible this\nis what it would provide and uh just ask\na couple of of users for feedback and if\nthat would solve their cases and that\nwas overwhelmingly positive so that was\ngreat so then we just did a very simple\nload test essentially the the old setup\nwhich because we knew that the shark is\ngoing to be a Shard yes we're going to\nmake it more lightweight but essentially\nwe can already create a chart so I think\nthe first test we use the classwork\naround but then we also increase the\nnumber of shards per class because our\nend goal was having as many shards as\npossible and then we we um I don't\nremember the exact numbers but let's say\nit was something like 50 shards per\nclass and then we just kept on adding\nclass after class after class and at\nsome point we would say maybe hit 10 000\nand we would say like oh all of a sudden\nthe next query is now failing I'm going\nto investigate like why why are the\nqueries failing what is what is going on\nand\nthat that would be part CPU profiles for\nexample seeing what do CPUs spend time\non that is when that whole background\ncycle sort of thing became apparent\nmemory profiles just to see like where\nis the memory actually used right now\num simple queries or sending queries\nwhere we said like okay this query in\nisolation should be fast but now in this\nlarge setup it's slow and then sort of\ninvestigating working backward from that\nwhere where it is that that was the\ninitial phase where we saw\nokay what we have right now is not a\ngood sort of not not the final solution\nbut we're well aware of what these\nproblems are and how we can fix them\nsort of with the final solution so that\nwas essentially proof of concept that\nwe're on the right path and that was\nthat was sort of it it failed\nsuccessfully like it failed in the\nplaces where we expected it to fail\num but also it proved that if we fix\nthese kind of hurdles if we get rid of\nthose hurdles it would essentially work\nand that was before we rode any kind of\nline of production code so that was\nbasically back in you could say in the\n1.19 release cycle we kind of prepared\nfor the 1.20 really cycle where we've\nbuilt multi-tenancy uh then came sort of\na classical implementation phase of\ncourse Implement implementing that has\nlots of tests and everything and we\nwould sort of constantly try to evaluate\nbut also sometimes you just need to wait\nfor it to be sort of like for to reach a\ncertain level of majority that you can\ndo sort of these these\num Black Box end-to-end tests where sort\nof in unit tests and integration tests\nand and even to some degrees end to end\ntests you always have some kind of\nknowledge of the internals but with\nthese kind of end to end really\nend-to-end black box\num uh sort of API level tests you just\ndon't know anything about the system you\nonly get to use the same functionality\nthat your user would use and then you\njust try this just try to replicate how\nwould a user use it and and see where it\ngoes and there was a point I think about\ntwo weeks or so before before the\nrelease uh when red run from our team I\nasked him sort of like hey what what can\nwe do right now to to sort of help you\ngive you confidence in the release\nbecause he took a lead basically on the\nwhole Cloud orchestration side of\nmulti-tenancy and he said please break\nit for me try to break it and then we\ngot together and we tried to break it\nand only we found a couple of there was\nno no uh no fundamental issue but we did\nfind a couple of things and um it was\ngreat that we found them because I think\nsome of them like some of them were a\nbit on the edge case side that would\nhave probably taken a few weeks for\nusers to run into them some of them were\na bit more obvious we really could find\nthem out right away and then\num we built this\nmore or less elaborate I would say load\ntest setup where he just kept on\nimporting kept on querying measuring all\nthe the metrics like what so so the the\nlinear scaling was extremely important\nfor us we wanted to make sure that if we\nhave say a I'm trying to use the actual\nnumbers that we used but I think we\nstarted with a three node cluster and we\naim for 10 000 uh tenants per node so\nwe'd start with a three node cluster\nwith a total of 30 000 notes and then we\nsaid hey if we turn this into a nine\nnode cluster and instead of 30 000 we\nwould do 90 000 attendance then we still\nhave that same ratio of ten thousand per\nnode so everything should scale the same\nis that the case and then we tried and\nwe could see and this was this was like\none of those moments where it's like yes\nwe've reached linear scaling where we\ncould see that the the nine node cluster\nessentially is three times the three\nnode cluster and that is that is very\nvery comforting because then you know\nokay now most likely the 12-minute\ncluster is also going to be four times\nas large as the three note cluster and\nand all these kind of things and that\ngave us the the confidence that hey the\nthese claims that we make about if you\nwant to extend it just add more notes so\nthis is really true and there's no\nadditional overhead for say\num for for import time or something\nbecause the the orchestration needs to\nhappen in the cluster is minimal uh the\nnode that owns the chart basically does\nall the work so by adding more nodes to\nyour cluster you're not just are you\nadding a more more space in a sense like\nmore disk space and more memory but\nyou're also adding more compute power\nwhich can take off the load so um a very\nsimple way of scaling import throughput\nessentially just using a larger cluster\nnow is the the second testing phase\nwhere\num sorry that gave us the release\nconfidence and this is something of\ncourse like you would do testing and\nautomated and in manual and explorative\nand all these kind of stress testing\nchaos testing you do this before any\nrelease but I think\nfor for this release this was the most\namount of testing we've ever done so\nthis is the I would say the most\nconfident we've ever been about a\nfeature and this is mainly because we\nknew we have stakeholders that were\nreally waiting for this and they're\nreally saying like hey this is sort of\nnot having multi-tenancy or not having a\nmulti-tenancy solution that scales to\nmillions of tenants is what's keeping us\nfrom from reaching the next level with\nV8 or maybe for others keeping us from\nfrom using vv8 in the first place so we\nreally wanted to make sure that while we\nknew that there was a limit with the\nprevious solution like now we want to\nconfidently say these limits are gone\nyou can use it for the kind of scale you\nwant and now essentially the the only\nlimit that there are two limits one is\nthe number of file descriptors so that\nvaries a bit and that is is local to one\nnote so\num the safe estimate is around 50 000\ntenants per node most likely you're\ngoing to run out of um out of uh other\nresources before that\num so that's the the one limit\num so so upwards of fifty thousand per\nnode and the other is just resources so\nresources is something that um that that\nit's just Vector search in general has\nnothing to do with multi-tenancy and\nthat is also something that's easy to\nincrease you just add more notes so\nthese are the the two theoretical limits\nfor number of tenants and both are very\neasy to overcome because both linearly\nscale with the number of notes in a\ncluster\nyeah it's so interesting hearing about\nthe uh like try to break it test I think\nthat's been one of the like one of my\nfavorite stories following along with\nweaviate has been like you know the\nsphere tests and you know blog posts\nabout that and you know trying to get a\nbillion nodes into eviate and that whole\nlike load testing thing has always been\nso interesting like I remember it's kind\nof like a theme of all of AI like you\nknow like with language models it's like\nhow big of a language model can you\ntrain it's like oh we train a 50 billion\nit's like hot so impressive and this is\nkind of like our analog of that is like\nhow many vectors can we put into a into\none index and I think also kind of\nseeing it you know across indexes and\nyou know one kind of system across the\nnodes is so interesting um I have a\nquick kind of clarifying question\nselfishly for me hopefully there is a\nlistener with it too so when you're\nsharding a class\nso I understand that each class you know\nit's the um you know it's the document\nclass and I have a million documents in\nit and I have one vector index so when I\nShard this you mentioned like searching\neach chart separately and then\naggregating the results somehow can you\nmaybe take me a little through further\nhow you Shard a class so here we really\nneed to\num separate multi-tenancy from single\ntenant cases because in in multi-tenancy\nwe use the these shards as isolation\nunits because you only want to search\nthrough one like in in multi-tenancy\nlike your query is for a specific tenant\nso we'd say there are 100 charts each\ncorresponding to one Tenon we would pick\nexactly one of those hundred we would\nhave that chart serve the request return\nit to the user nothing nothing changed\nso this is kind of the the\num\nlike a small portion of the data on the\nnote is queried in isolation in the\nentire query is completely sort of\nself-contained in that chart if you\nShard a so so for that case it doesn't\nmatter how many charts you have like\neven if you have a million charts\nbecause you're always hitting exactly\none there's never any overhead for for\nnumber of shards like whether you have\nwhether you query one out of ten or one\nout of 100 or one of out of 100 000 it's\nalways the same you're always querying\none uh but in a single tenant case the\nmotivation for sharding is different so\nin a single tenant case you don't have\nthe tenant key so the results that you\nexpect are the entire Vector space so\nnow your motivation for sharding is kind\nof the uh the the other way around like\nyou don't want to have as many charts as\npossible you want to have as few shorts\nas possible so for this the question is\nbasically why do you want to so if you\nif it's better to have fewer shards why\ndo you want to have shards at all and\nthis is uh where where sort of\num\nthe the scale of a single index comes\ncomes into place with respect to the\nhardware that is scheduled on so that\nwas a very very complex way of\nessentially saying like you can only fit\nso much on one machine and if you need\nmore than that one machine you need to\nShard it across two or three or four so\nthat is the motivation in let's say we\nhave an index of a billion objects and\neach machine could only fit 250 million\nthen you would chart that across four\nmachines\nand now if the query comes in all four\nmachines would say hey okay I'll give\nyou the top 10 results out of my 250\nmillion so you would now and end up\nessentially with four lists of uh each\n10 top objects now you have to aggregate\nthat list again that's super easy to do\nbecause you have let's say the distance\nmetric so essentially you just Resort\nthat list of 40 and cut off the the\nbottom 30 again so you remember you have\nthe the top 10 remaining\nthat is easy to do but from a sort of\nscaling out perspective you need four\nnodes to serve your query whereas in the\nmulti-tenancy case because only a single\nnode hits sort of owns the the data for\nyour one tenant also only that one\ntenant needs to answer it which means\nall the other nodes and all the other\nCPUs on that node are idle to serve\nother tenants basically\nso when you when you have the hsw graph\nis there anything to how you partition\nso imagine like on layer zero I have\nlike these are clustered and so we're\ngoing to go Shard together is there\nanything\nuh the the chart is at a higher level so\nthe the hnsw index would be fully\ncontained within them and it wouldn't\neven know that there are other charts\nlike the the shards you can think of\nthis like class Shard and then in the\nchart you would have Vector index uh\nregular index Etc\nso the the agent's double Unix doesn't\neven know that that said this is\nsomething I think that that would be\ninteresting for for future research to\nsee like could we Shard instead of\nsharding by sort of an application Level\nattribute so right now we use a hash on\nthe ID you could also potentially Shard\nby Vector proximity so that would be the\ncase where sorry if a bit similar to how\nIVF based uh indexes worker you have\nthese kind of buckets so you could say\nif I know that my data set is going to\nhave 10 charts\ncould I try to sort of pre-select one of\nthose 10 shards based on proximity and\nmake sure that the vector sort of ends\nup in in The Shard that's closest to it\nproblem is what we know from from IVF\nbased indexes is with a\nmulti-dimensional distance it's not as\neasy like in a sort of it's not a binary\ndecision it belongs into bucket one or\nmaybe along so in bucket ten so in IVF\nwhat you need to do is you still need to\nquery multiple buckets like the let's\nsay out of 100 buckets you still need to\nquery the the top 25 closes so that's a\nlike interesting for for for research um\nbut currently that's that's not\nsomething so currently the the chart is\nat a higher level in a in a um single\ntenant sharded setup and the vector\nindex within that chart wouldn't even\nknow that other shards exist\nyeah it's really fascinating thinking\nabout the you know like the vector\nindexes and then how you distribute\nVector indexes across Cloud computers\nacross the world and this is all really\nexciting I think kind of we're\ntransitioning into this like you know\nfuture improvements Future Works uh\ntopic broadly I I hopefully we didn't go\ntoo into sharding particularly but like\nback into the multi-tenancy like do you\nsee you know what are some things that\nare top of mind now that rolling it out\nyeah there's one aspect that I haven't\neven mentioned at all yet that I'm super\nexcited about\num what what we wanted to achieve with\n1.20 was we wanted to have a stable API\nwe wanted to have uh sort of the main\nfunctionality which is scale so anyone\nwho was blocked by not being able to to\nonboard uh or or to to get started with\nVBA because of lack of multi-tenancy\nfeatures we wanted to say like you can\nget started with 1.20 but this is not\nwhere multi-tenancy ends basically this\nis basically just the beginning and one\none big sort of potential for\nimprovement that we have is cost\nreduction because something that if you\nhave a multi-tenancy case most likely\nnot all of your tenants are active at\nthe same time if they are all your\ntenants would need to sort of Be Active\nand this is I'm using this fuzzy term of\nactive versus versus inactive because\nthat's that sort of yeah just just as an\nabstraction we can go into what that\nthat means as well\num but if we say that we have active\ntenants who are potentially expensive\nlike let's say they need a lot of memory\nthey need a lot of compute and we would\nadd inactive tenants which are not\nexpensive because they don't need memory\nor or\nthey would need the memory at the moment\nthat they become active but in their\ninactive State they would use fewer\nresources now what that allows us to do\nis potentially size vv8 cluster for the\nnumber of active tenants as opposed to\nfor the number of total tenants so if\nyou have let's say\na hundred thousand tenants and you would\nsay that you're in for a bill for a\nhundred thousand tenants is one thousand\ndollars if you know that only ten\npercent of your tenants are active at\nthe same time and you have a way to\ndeactivate those other ninety percent\nyou could reduce your infer cost from a\nthousand dollars to a hundred dollars\nand this is where where it gets really\ninteresting because that's the the cases\ngrow like in Vector search right now\nalmost all cases are always sized for\ndata set like it's always the first\nquestions like how large is your data I\nsaid what's your dimensionality how many\nobjects do you have\num maybe can you use compression what\nare your query requirements Etc but kind\nof the cost is linear to the number of\nobjects but with multi-tenancy in the\nability and this is sort of what's\nwhat's coming next the the ability to\ndeactivate and then there's there's more\nthings sort of in the pipeline um Beyond\njust deactivating them but just with\ndeactivating them we would not need any\nmemory for for those tenants anymore so\nif you can say that even if it's just 50\nor so they're inactive you would\nessentially have twice the number of\ntens or could host twice the number of\ntenants on the same hardware and this is\nwhere I see a ton of potential for for\neven more cost saving because\nrealistically there's always this like\nlong tail distribution where you have a\nfew tenons that have a lot of queries\nand a lot of tenants that maybe query\nonce per minute or once per hour even or\nmaybe just once per day and you can\npotentially serve them for much much\ncheaper hardware and that makes it way\neasier to to reduce operating cost with\nmulti-tenancy\nhmm yeah that is it's really interesting\nhearing the hearing the uh yeah like the\nprioritization of certain resources and\nso on and I I just kind of was having\nthis Epiphany about how this could help\nlike if you know if I'm building a\nmachine learning app with weavi and I'm\ncollecting user data I'm thinking about\nyou know I just kind of unlocked my\nbrain how much this multi-tenancy\nenables like you know just as a random\nexample our last movie a podcast was\nwith Alexa gordich from ordis and you\nknow if I'm collecting the user data of\neveryone interacting with his uh YouTube\nchat bot I I have this new multi-tensity\nway of collecting that data and feeding\nit into my machine Learning System and\nyeah just incredibly exciting as I am\nunderstanding it better from hearing you\nspeaking about it yeah also makes it\neasier to get the data out for a\nspecific tenant because essentially like\na like a list all query is now specific\nto attendance so you're now listing all\nthe data for for that tenant so\nyeah I guess like when I was first\num thinking about this topic of\nmulti-tenancy I hadn't understood the\njust users with the same kind of class\nwell I hope I'm making this like I I\nthought about it like say I want to have\nall these different classes with\ndifferent properties like for whatever\nreason Conor has a age property but\nEddie and I don't know you don't have\nyou just have like a HomeTown like you\nknow what I'm trying to say it's like\ndifferent properties or like different\nuh I'm ageless\njust a good example yeah like I I guess\nI was thinking like you know especially\nwith the vectorization configuration I\nthink maybe you know like different\nclasses would vectorize a different\nproperty and stuff like that but yeah I\nthink just you know having you know\nmillions of users for the same kind of\nclass that application makes a ton of\nsense and yeah it's just really really\nexciting stuff uh awesome so I think\nthat's kind of a Roundup of\nmulti-tenancy edian's written this\nincredible blog post in addition with\nthe Alice Corp and Bob and car yeah like\nthe examples\nand uh so you know release post uh\nrelease notes all this cool stuff so\nawesome so pivoting to our next topic um\nproduct quantization I think this was\nfirst released in 118 or 119 uh you know\na super clever Vector compression\ntechnique I think the history of we V8\nyou first started exploring a vector\ncompression with binary passage\nretrieval where you're like hashing you\nknow positive negative of each Vector\nDimension but the problem with that uh\nproduct binary passage retrieval I think\nis you really need to train with it it's\nnot really something that you can do\nwith vectors that just come out of the\nbox from whatever model\nbut this product quantization thing\nwhere you uh cluster the values and then\nrepresent them with the centroid IDs you\ncan apply that to any kind of vector and\nso it's really Universal way to compress\nvectors and just overall really exciting\nway to kind of reduce the memory\nrequired in these Vector indexes um it\ncould maybe start broadly on like where\nis your head currently at with product\nquantization\nyeah yeah as you said we we initially\nreleased this uh I think two or so\nreleases ago one 1.18 and we put this we\nwe slapped this experimental label on it\nand that was not because we didn't trust\nour implementation that was because we\nsaid we're hey we're Gathering feedback\nwe want to see how do users use that\nwhat can we potentially do to improve it\nand and we were well aware that this\ncould come maybe with with some API\nchanges that in the sort of regular\nsemantic versioning API guarantees we\ncouldn't uphold so they said like hey\nthis is experimental this is probably\ngoing to change maybe not but this is\nkind of the the evaluation phase at the\nsame time we don't want to keep it from\nusers here you are here please please do\nuse it be aware that yeah either it\nwould change or maybe you don't use it\nin production yet unless XYZ and that\nalso in turn meant that at some point\nthat the experimental phase needs to end\nand it needs to become generally\navailable and that is now with 1.20 and\nthat's the the one big change just in\ngeneral we incorporated all that\nfeedback\num of sort of maybe it wouldn't work as\nexpected in some cases maybe there are\nways to to do things better like that is\nthat is one part of of uh where we're at\nwith PQ right now but then the other\npart is also we've improved it we've\nadded what we call re-scoring and this\nis a very simple change that has a\nmassive impact so in general PQ is a\nlossful compression so you reduce the\nnumber of let's say standard example you\nhave\num a 768 dimensional vector\nwith PQ you would represent four\ndimensions or this configurable then\nlet's say for example you represent four\ndimensions with a single byte so now\nwhat you have is previously for those\nfour dimensions you use float32s so that\nwas 16 bytes and now you not just do you\nreduce this or from float 32 into just a\nsingle byte but also you pack multiple\nDimensions into a single byte so in this\nsimple example you can reduce the memory\nfootprint for 786 Dimensions\num by a factor of 16. so that's that's\nextremely\nsort of yeah that's an extreme\nImprovement but at the same time if you\nif you do that you're recall would drop\nconsiderably likely you would you would\nyeah your memory requirement is is a\nfraction of what it was before but also\nyour result quality is bad now\nso what we said is well but we're still\nusing PQ in combination with this H and\nSW index and in hnsw essentially you\nhave a a top K Heap so you have like a\ntemporary list of what you think is the\ncurrent results and while traversing the\ngraph you sort of keep\num improving upon that list and the\nfinal result is essentially the result\nthat we pass to a user so that's a very\nsimplified explanation of of how search\nresults are gathered within hmsw and\nthat means that if we use that that\ncompressed PQ distance to make a\ndecision of whether we wanted something\nbasically whether we want to add it to\nthe list or whether we want to drop\nsomething because the list has a fixed\nsize so whenever we add something\nsomething that's a worse result needs to\ndrop off and if we do that based on the\ncompressed distance oftentimes that\ndecision is wrong and then this this\nsort of small compression error would\naccumulate more and more and more and in\nthe end you would have a lot less recall\nbut if instead if occasionally we\nactually load the real Vector from disk\nand use the real Vector to make that\nthat decision so this is the rescoring\npart we actually take the the original\nVector from disk so now you're trading a\nbit of performance you have a discrete\nnow where previously you wouldn't have a\ndiscrete but in exchange you now get way\nmore accurate results and I think we\nhave a blog post coming up in a week or\nso or maybe two where we have some some\ngraphs in there and it's super exciting\nbecause in some cases\nthe uh recall QPS trade-off sort of the\nthese these curves were top into the\nright is typically best it's actually\nnot any worse than it is without\ncompressed so it's like you you don't\nlose anything that's not true for every\ncase like in some cases there is still a\nbit of a trade-off where you can\nessentially say like am I optimizing\nmore for efficient storage or am I\noptimizing more for for efficient\nquerying but in in many cases it's\nactually not a draft anymore you just\nhave to turn it on and this is this is\nin turn a bit sort of depends a bit on\nhow much memory you have available in\ngeneral because just of the way that\noperating systems work like a disk hit\nonly makes it to disk if that that\nportion of the disk hasn't been cached\nin memory before if there is memory\navailable the operating systems and it's\nthe the page cache will start caching\nportions of the disk so what is\ntechnically a disk read might actually\nnot even be a discrete if you still have\nsome memory available so you get these\nlike buffers zones in in which the\nperformance is still great even though\nthe memory\num usage has dropped a lot and this is\nthe second feature for PQ that we said\nlike okay once we go for General\navailability we want to want to have\nthat in and now we're at a point where\npreviously it was like\nuse PQ sparingly in some cases because\nit reduces your Recall now it's more\nlike why would you use PQ please\neveryone go out and use it it's awesome\nyeah I think when I first heard PQ\nrescoring I thought of it in the\nre-ranking way that we'll talk about\nnext where it's like you know you get\nthe top 100 results with the compressed\ndistances and then you just kind of\nbring the full Precision to just re-rank\nI didn't realize how uh deep into the\nhsw traversal how say you know I I am\nstill like Curious so you're like\nexploring the neighbors of the center\nnode and you're going to kind of like\nResort the candidate I think maybe um\nyeah I don't know I think it's maybe\noutside of the scope of the podcast but\nlike just yeah understanding that hsw\nhas this like candidate list and dynamic\nnearest neighbors and you like rescore\nit to have a better search like deep\ninto the traversal rather than just at\nthe very end I think very important yeah\nyeah that's like the Elegance of it so I\nhave kind of two questions I want to ask\nyou about PQ the first is maybe a\nclarification of the in-memory on disk\npart of PQ so\nI'm curious if this is the correct\nunderstanding so you know we we load say\n200 vectors into Eva and now we trigger\nPQ we fit the k-mean centroids so from\nthen on do we a new Vector comes into\nweviate do we just send the full uh full\nPrecision Vector to disk and then the\ncompressed Vector that goes to memory is\nexactly exactly yeah so we always retain\nthe full Vector basically on on disk\nlike you store what the user provides\nwhich is the the the real Vector so to\nspeak but we also compress it and then\nonly keep the compressed one in memory\nadditionally we also need to store the\ncompressed one in memory so if the the\nserver restarts you don't want to\nrecompress everything so we we store\nboth actually but exactly as you say\ninitially we keep the the original on\ndisk and keep the compressed in memory\nfascinating I remember one of the my\ntakeaways from when we met with Matthias\ndouzay from meta who's one of the\npioneers of product quantization was\nthis um like online k-means kind of\nproblem and thinking about like you know\nhow many vectors do you need to load in\nto compute the centroids and then how\ndoes this scale with incremental updates\num like I guess the question would be\nlike what are you kind of learning about\nthat problem uh\nyeah you you need a certain so so it's\nvery hard to pick an exact number\nbecause you need your data or\nthat subset of your data needs to\nsomehow be representative of the entire\ndata set so in in the ideal case let's\nsay you would have five clusters in your\ndata naturally then those five clusters\nand then you want to train on ten\npercent then those five clusters need to\nbe represented in your original\num\na ten percent if they're not for\nwhatever reason like if let's say the\nfifth cluster didn't even exist and you\nonly have four clusters for training\nthen you're training on something where\nthe fifth cluster doesn't exist yet so\nthat doesn't mean the fifth cluster\ncan't be assigned anywhere but it means\nthe the sort of trained model isn't as\ngood as it would be\num uh if if that had print part of it\nthe good news is that there's a certain\nsize where it gets more and more likely\nthat the the distribution of the data\nset sort of doesn't change of course\nthat's no guarantee like it could\ncompletely change\num but often what you see so for example\non on\num these a n Benchmark data sets that\nare typically between 1 and 10 million I\nthink we typically didn't see any\nImprovement uh when using more than a\nhundred thousand vectors for for\num for training so\nthat that's still a very manageable\namount I think it takes 10 seconds or so\nto train with with a hundred thousand\nvectors\num yeah sort of a good a good trade-off\nyeah fascinating Yeah a hundred thousand\njust like getting a sense of that is\nsomething that I think is a really\ninteresting research question when\nyou're you know going into the details\nand yeah I think this whole like cluster\nanalysis is definitely a lot to it I\nremember you know earlier we did a\npodcast on Bert topic and we were\ndiscussing like what would cluster\nanalysis and weba look like I think it's\nstarting to see I don't want to give\naway I know that we have something\nplanned but like um you know we're\nseeing Partners Integrations other\ntechnology companies building further\ninto this like embedding space\nvisualization topic modeling cluster\nanalysis and you know definitely people\nlistening to be on the lookout because\nthere's going to be some really cool\nstuff around that coming soon on\nembedding drift detection and all this\ncool stuff\num yeah awesome so I think that's a\ngreat coverage of the PQ rescoring I\nthink probably for me the biggest\ntakeaway is um you know understanding\nHow Deeply that goes into the hsw\ntraversal and what you said I think is a\nhuge point about originally you were\nthinking I'm trading off memory for\naccuracy but now that you know you have\nthis restore ring there's no trade-offs\nso just use it that's kind of a funny\ntakeaway yeah yeah for for some cases I\nmean there there is it's engineering so\nthere's always trade-offs and it's kind\nof like not seeing our users data it's\nhard to make predictions but we're\ntrying to model real life data usage\nwith with as many different data sets as\npossible and I'm really surprised to see\nsort of how good it is in in some cases\nlike yes in some cases there is still a\nbit of a trade-off but in others it's\njust not\noh yeah yeah I think um having all these\ndiverse data sets is such an interesting\nthing of machine learning generally like\nthe very is still kind of no free lunch\nin a way like uh yeah\nawesome so I think kind of we have a\ntrio of new features I'd like to kind of\ncluster into one kind of category which\nis kind of like we V8 being feature Rich\nfor search you know like vector search\nobviously like search relevance is a\nhuge topic and I kind of just to give a\nquick tldr of each of these things uh\nthe first of which is re-ranking where\nso as I mentioned with my original\nunderstanding of PQ re-scoring\nre-ranking is this idea where you take\nyour top say k 100 results from Vector\nsearch and then you're going to take\neach of those candidates in the query\nand pass each of those as input to a\nhigh capacity scoring model so where do\nthese scoring models come from sentence\nTransformers has six of them that are\nopen source and so we've built the\ndocker image where you know similar to\ntext to back Transformers you you have\nthis like local image if you wanted to\nspin it up in your laptop host it\nyourself and all that good stuff we have\nthe you know the cross encoders that\nhave been trained from sentence\nTransformers are up there and then also\nreally interestingly we have a\nre-ranking endpoint from cohere and\nobviously our friendship with Mills\nrhymers you've seen Eddie and those\nrhymers together um you know this is a\nan API for a re-ranking model and I\nthink that's quite fascinating I don't\nthink there's another another company\ndoing that and it's pretty interesting\nto see that API and you know it comes\ninto like weeviate's module system how\nwe have these um this orchestration of\nAPI requests I think I think it's quite\nfascinating maybe actually adding we\ncould take a pause and I I really wanted\nto ask you this question about how um I\nthink an interesting thing about we v8's\nmodule system is how golang handles\nthese uh like concurrent HTTP requests\nand maybe talk a bit about like um what\ngoes behind that how golang is optimized\nfor\nscaling this kind of you know\norchestration to model inference because\nI think it's such a fascinating part if\nwe get in general go is a language as\nyou say sort of easy simple concurrency\nis one of the building blocks of why you\nwould use go so go has channels and it\nhas a simple synchronization Primitives\nsuch as mutexes and and these kind of\nthings that you need for easy\nconcurrency it has a race detector which\nis also a super important feature to to\nfigure out like are you doing your\nconcurrency safely are there any kind of\nrisks so in general sort of go is a good\nI would say it it makes it relatively\neasy and then\nwhat that allows us to do is if a user\njust gives us sort of a a bulk data\nthingy like like a batch of data objects\nessentially and our job is to say\nvectorize them or or also re-rank them\nI'm going to pass them to to the model\nif we would do that sequentially that\nwould also mean that whatever the\nlatency is for one of those requests\nwill be bounded by that but\nthe kind of cool thing about these\nserverless applications is that they\ntend to scale way better than that so if\nyou hit them concurrently that allows\nyou to to sort of not have to wait for\none thing to be finished but just do\nmultiple of them at a time that said\nagain there is a bit of a trade-off\nbecause\num often these models have rate limits\nso I think something that that often\npops up in our support forum is for\nexample with open AI there the the\ndefault rate limit is I don't know what\nit is but something like I think maybe\n60 per second or so that that has a 600\nper minute so if you want to vectorize\nmore than that you could potentially hit\nthe the rate limit so also concurrency\nis not a free lunch here but it it helps\nto at least sort of max out whatever the\nthe model provider can can use and this\nis the same for for if you self-host\nyour your model\num if you don't concurrently do\nsomething and it doesn't necessarily\nmean that it has to happen with with\nconcurrent HTTP requests that could also\nmean that you bolt this up into a single\nrequest and then maybe the concurrency\nhappens on the model side but in in\ngeneral you want to sort of not have\nidle capacity around like like if there\nare either gpus or CPUs that are used\nfor model inference you want to make\nsure that they're they're maxed down and\nthat's one of the motivations for using\nconcurrency in in that case and then\nsort of taking that back to to golang\nit's a language that makes concurrent\nprogramming rather easily I know that\nthat some of our team members sometimes\nhate go for it because it's it's not\nperfect in any way but no no language is\num but it's it's\nyeah a very good sort of fundamental\nconcurrency kind of model\nyeah fascinating I think I also like the\num you know like the Gen like our\ngenerate module with openai palm the\ncode here I the way that particularly\nthe way that it paralyzes doing the\nsingle result call I find a ton of use\nin that where you want to put a prompt\nof each of your search results and have\nthat all be parallelized it's so much\nfaster than looping through there yeah\nyou know I think it's pretty cool so\nokay cool so on this topic again of\nsearch results so we have re-ranking now\nwhere you can get better search results\nand you know kind of building it right\ninto weviate is why we took this tangent\ninto the more technical thing you don't\nhave to have you know manage some other\nservice of your own to do re-ranking\nit's built natively into alleviate and\nyou know taking that search result\nslices and either way so the next cool\nthing is auto cut so autocut has this\nhigh level motivation of you know how\nmany search results are relevant if you\nsir you know if you search for I I've\nstolen Bob's uh landmarks in France\nthing I use this all the time now so if\nyou search landmarks in front and you\nhave the Eiffel Tower in Paris and you\nhave some other things and you know say\nyou only have like eight thing eight\nlandmarks in France and your search\nresult cutting off that nine and only\nshowing eight results\nthis is incredibly interesting for the\nlanguage model retrieve blog went to\ngeneration thing as well because the\nlanguage model can be quite thrown off\nby these irrelevant results so like I\ndid see something about Spain it might\nwrite about that because it's in it says\nbased on the search results and it's\nlike well it's there so you know so I\nthink the interesting thing about\nautocut also is understanding uh how\nthat's done so it's like the um you know\nit's like calculating the delta in the\nslope from the point so you take each of\nthe distances and then you have to\ninterpolate a line from there and then\nyou measure the steepest change in the\nslope so what we've got users will see\nis a hyper parameter on which of the\nsteepest slopes because you know you\nhave like two to three as each reverse\nof the thing so it's like how extremely\ndo you want to be cutting it and then\nthe last thing is you know one of our\nfavorite topics is this kind of uh rank\nFusion algorithm weeviate has built-in\nBM 25 as well as Vector search so what\nthat means is you end up with two lists\nand so you know the first time we did\nthis we were just doing you know just\ncombining it based on the ranks so you\nknow if you're like first ranked and BM\n25 and like fourth rank in Vector search\nthat ends up you know just purely based\non the ranks and now we have like a\nrelative score Fusion so yeah I think\nall these things um just all this new uh\nbenefits in search I think the question\nI'd like to ask again is just kind of\nwhere your head's at on yeah like the\nsearch stuff I think you know maybe\nBerlin buzzwords there was some\ndiscussions on the search side of things\nI've been to the haystack conference\npersonally and seen that kind of like\nyou know world around search relevance\nyeah to to me this is a someone who's\nwho's very use case driven this is for\nme extremely important that we think of\nthese these use cases and search is a\nbig one it's not the only one there's\nrecommendations there's classifications\nthere's called the of course the whole a\ngenerative AI part but generative search\nthen also as a part in there\num and for me it's I I like being able\nto give our users something that they\ncan use end to end where they can say\nlike hey I want to spin up bb-8 I want\nto have the data management part so you\nwant to have the whole database I want\nto have the vector search obviously\nbecause that's the whole reason of of\nusing bv8 as opposed to say traditional\nsearch engine uh the Hybrid part that's\nsuch a big one just the the bm25 like\nnot having to spin up something else\nthat that does the pm25 for you but\nbeing able to to do that\num and then all these new additions and\nand re-ranking and auto cut these are\njust sort of the most recent ones but I\nthink there will be way more down the\nline and a essentially if it makes your\nsearch experience better then it's\nsomething that we could potentially\nextend deviate within because of the\nmodule system we're very flexible and\neven adding sort of things that that\nessentially the module system allows us\nto add things that not everyone needs\nwithout loading VBA because if you don't\nneed the module just don't turn it on so\num this this kind of yeah everything\nthat makes search\neasier to use better and I guess in a\nway also more accessible because there's\nlike information res information\nretrieval is like this whole research\ntopic out there and there's there's\nspecialized conferences you just\nmentioned Berlin Bus words uh which are\nsuper fun for us but for someone who\njust wants better search maybe they're\nnot maybe they don't want to get into\nthe nitty-gritty details maybe they just\nwant to have better search and there\nwe're we're always trying to find that\nbalance of making it easy for someone\nwho's new to the space without sort of\ntaking away the flexibility that someone\nwho's more experienced needs and I think\nthese these new re-ranking modules and\nthen AutoCAD these are great examples\nbecause if you have let's say you have\nyour own complex ranking pipeline that's\nway more complex than something that you\ncould fit into evade just don't use the\nre-ranking module but for someone else\nwho's already using bb8 maybe doesn't\nhave a complex pipeline says like hey\nthere's something that I can just turn\non with a single flag and it makes my\nsearch experience better by all means do\nit that's that's what we're here for\nyeah yeah yeah I I think there are so\nmany interesting tricks from the\ninformation retrieval uh Community we\ncan get out of to get better search I\nthink um I think with re-ranking to\nmaybe stay on it a little more I think\nmaybe some of the search zealots would\nbe a little bit saying you know I still\ncan't do symbolic re-ranking you know\nlike if you're searching for products\nyou might want to have the price and the\nrelevancy these as features that you\nwould put into re-ranking uh so you've\nseen we've had a podcast with meta rank\nwith Roman and Siva where you know we're\nthinking about this thing as well it'll\ninterface very similar to this\nre-ranking with coherent sentence\nTransformers we'll just send the\nproperty to the API as well and and\nit'll look super similar there's also a\npretty interesting discussion around\nlike multi-valued vector search like I\nhave a title Vector as well as a\ndocument Vector so you know we're\nlooking into all these things as well as\nlike the the large language model\nre-ranking thing is quite interesting\nwhere you prompt it with kind of the\nsymbolic rule so like please boost it if\nit has the if it's recent and then you\nand then you give it the result like\nJson dictionaries and that does work\npretty interestingly I think it's also\nreally exciting for Integrations with\nlike llama index and Lang chain about\nand Samantha Colonel and Pat make a list\nof these things but I'll just leave it\nthere about how you can you know just\nget better search results by just taking\nthese things on and yeah I think also\nmaybe just before uh graduating to our\nlast topic I also just want to thank\nDirk for this Dirk did an incredible job\nwith you know the rank Fusion and the\nauto cut I remember like the other guy\nwas trying to figure it out myself and\nit was such a headache trying to figure\nout how to do the um uh get the go the\ngo numb uh thing to do the slope\ninterfellation and I had showed it to\nDirk and he was just like the next day\nlike oh yeah here's the notebook to test\nit I was like that's so fast I've turned\nit around so really impressive stuff and\nuh yeah it's awesome so kind of our last\ntopic is the um the cloud monitoring\nmetrics with the Prometheus and I'm\ncurious you know I imagine as you do we\nmentioned like the trying to break your\ntesting with the multi-tenancy I'm\ncurious like where your head's at with\nthese you know how you you know\nbasically observe this massive thing\nrunning in the cloud right\nyeah it almost feels like one of those\nboring features compared to all the\nexciting features that we just talked\nabout but I think this is this is what's\nso important to to pay attention to the\nboring stuff because\neven something like multi-tenancy I mean\nwe're super excited about it but in the\nend you could argue like from a from a\nAI perspective this is a very boring\nfeature like it's not a New Concept but\nthese are the kind of features that you\nneed to run reliably in production for\nme anytime we we add something like that\nand we we sort of expand on on the\nobservability stack like this is a\nsimple change\num essentially it allows you to to track\nyour success query rates like previously\nyou could only monitor I think latency\nbut now you also get just a ratio of\nwhat queries have succeeded versus what\nqueries have failed and and the failure\nreason so was this a user initiated\nfailure so in HTTP status codes this\nwould be like a 400 uh plus a status\ncode or is this did something go wrong\nin vb8 so a 500 or so so internal server\nerror and these kind of things and the\nones from from out there\num so so adding the the monitoring for\nthis to get better observability\nbasically on these kind of things\nit's very boring but it's very essential\nand I I love seeing bva grow up and and\nhave our users ask for these boring\nthings because in the end boring is\nproduction and production makes our\ncustomers money and that makes us happy\nso\nalways always a place in my heart for\nthe boring features\nyeah I think well yeah I think um like\njust for me learning about like datadog\nand learning about these companies that\nhave built around that kind of thing is\nit's really fast yeah I mean it's it's\nto me it sounds like kind of like\nInsurance on your uh thing that you know\nlike your data is up in the cloud it's a\nsuper abstract concept of like a little\nbit of it is in Virginia a little bit of\nit is in Germany so yeah yeah oh that is\nyeah data Doc is a massive company so\nthat just goes to show like how much\nhow much need there is or how much how\nmuch room there is for for these kind of\nobservability topics exactly as you say\nit's like insurance it's\nearly warning it's just sort of looking\ninto that the the more complex that your\nsystem grows\nyou can only see so much from the\noutside and sort of getting these\ninsights and just seeing what's going on\ncan\nhelp sort of in in hindsight for for\ndebugging cases like what was going on\nbut ideally before something happens\nlike early warning systems like simplest\nthing uh monitoring for for memory usage\nor something like you see something goes\nup nicely in a linear line and you know\nthat there is a certain limit with your\ncapacity\nif it says 30 maybe that's fine 50 maybe\nthat's fine if it's approaching eighty\npercent maybe you want to change\nsomething about your infrastructure so\nyeah these kind of operational things\nthat you need um\nit's it's\nfor me it's it's just such a positive\nsign of like hey people people do need\nthem which means they're doing serious\nstuff with bb8 so I'm very happy to\nalways ask our team to prioritize some\nsome observability features\nyeah well eddian awesome thank you so\nmuch for another release podcast\nmulti-tenancy PQ with rescoring\nre-ranking autocut new hybrid rank\nfusion and then new updates to the\nPrometheus Cloud monitoring all so\nexciting um and also you know you can\njoin the community slack if you need\nhelp with any of these things everyone's\nalways around and checking that all the\ntime and uh yeah thanks so much for\nlistening\nthanks for having me had a blast thank\nyou"

Meeting Notes
Date: July 13, 2023
Attendees: Connor (C), Weaviate (W)

Agenda: Exploring Multi-Tenancy with Weaviate's Vector Database

Introduction and Goals of Meeting (B):

Connor explained that the primary goal of the meeting was to discuss how Weaviate's vector database could be integrated into Connor's platform, a photo sharing platform, with a specific focus on multi-tenancy.
Explanation of Vector Databases (W):

Weaviate provided an overview of vector databases, focusing on how they handle high-dimensional data points and efficiently perform similarity searches.
Benefits of Multi-Tenancy (W):

Weaviate explained the concept of multi-tenancy, highlighting its potential for resource sharing, cost reduction, and ease of maintenance. They mentioned that their vector database is capable of supporting multi-tenant architecture.
Connors's Interest in Multi-Tenancy (B):

Connor confirmed the importance of multi-tenancy for their platform. Given that they serve many different users and organizations, it's crucial for them to isolate each tenant's data while still using the same physical infrastructure.
Implementing Multi-Tenancy with Weaviate (W):

Weaviate laid out a preliminary plan for implementing multi-tenancy. This included isolating data on a per-tenant basis, setting up resource quotas and access controls, and providing tools for monitoring and management. They also discussed how the vector search capabilities could be applied on a per-tenant basis.
Concerns and Questions (B):

Connor raised questions about how to handle data migration, backup, and recovery in a multi-tenant setup. They also asked about scalability and performance considerations, including how to ensure one tenant's activities don't impact others'.
Addressing Concerns and Next Steps (B):

Weaviate addressed Connor's concerns, explaining how they handle data management and backups, and assuring them that their system is designed to prevent cross-tenant performance impacts. They proposed a follow-up meeting to delve into technical details and to start planning a potential proof-of-concept integration.
Next Steps: A follow-up meeting will be arranged to discuss technical details, potential challenges, and the planning of a proof-of-concept for the multi-tenant vector database integration.

INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/BlogPost "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema/BlogPost "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/nodes "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/nodes "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/nodes "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/nodes "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/nodes "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/nodes "HTTP/1.1 200 OK"

INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/PodCast "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema/PodCast "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"

INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/MeetingNotes "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema/MeetingNotes "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"

INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/BlogPost "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema/BlogPost "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/BlogPost "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema/BlogPost "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/PodCast "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema/PodCast "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/PodCast "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema/PodCast "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/MeetingNotes "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema/MeetingNotes "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/MeetingNotes "HTTP/1.1 200 OK"
HTTP Request: GET http://localhost:8079/v1/schema/MeetingNotes "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Multi-tenancy is the capability to serve multiple distinct users or user groups from a single application. In Connor's application, multi-tenancy is crucial because it allows for access isolation, speed in narrowing requests down to a single tenant, easy on and offboarding of tenants without affecting others, setting resource boundaries to prevent one tenant's activities from impacting others, ensuring cost-efficiency by not overpaying for idle resources, and accommodating a diverse range of tenants in terms of size and data volume.

> Source (Doc id: 52d45997-c5bb-4386-9269-99880ae8cb34): Multi-tenancy is the ability to serve multiple distinct users or user groups from a single applic...

> Source (Doc id: 285431a7-6bd2-4242-b9b6-45fd29f99804): Multi-tenancy is the capability to serve multiple distinct users or user groups from a single app...

> Source (Doc id: 1dff9708-e905-4df3-9487-df514ba45c0b): Multi-tenancy is the concept of hosting multiple separate instances or groups of users within a s...

> Source (Doc id: 62b0e1c6-a364-4d29-a3fa-69b8cc8921cf): ### Using filters
Another approach was to use a single class and use Weaviate’s built-in filterin...

> Source (Doc id: 5b56877b-0a7c-481f-a549-4841961087ea): title: Multi-Tenancy Vector Search with millions of tenants


Large-scale setups were always a gr...

> Source (Doc id: 4cc589ef-6ed8-4273-8810-0e9feb779ca9): ### Using filters
Another approach was to use a single class and use Weaviate’s built-in filterin...

> Source (Doc id: a837c5c6-62ae-4195-975d-11d909f58e90): title: Multi-Tenancy Vector Search with millions of tenants


Large-scale setups were always a gr...

> Source (Doc id: 37d5dd1b-1a49-4925-a370-add607fa4cf9): Meeting Notes
Date: July 13, 2023
Attendees: Connor (C), Weaviate (W)

Agenda: Exploring Multi-Te...