Notebooks
W
Weaviate Hybrid Search Databricks

Hybrid Search Databricks

vector-searchvector-databaseretrieval-augmented-generationllm-frameworksweaviate-featuresfunction-callingweaviate-recipesmodel-providersPythongenerative-aidatabricks
alph-notebooks/weaviate-recipes / hybrid_search_databricks.ipynb

Open In Colab

Hybrid Search with Databricks

This recipe will show you how to run hybrid search with embeddings from Databricks.

Requirements

  1. Weaviate cluster

    1. You can create a 14-day free sandbox on WCD
    2. Embedded Weaviate
    3. Local deployment
    4. Other options

  2. Databricks token

  3. Endpoint to an embedding model on Databricks

Import Dependencies, Libraries, and Keys

[ ]
[1]

Connect to Weaviate

Only choose one option from the below.

Weaviate Cloud Deployment

[13]
True

Embedded Weaviate

[ ]

Local Deployment

[ ]

Create a collection

Collection stores your data and vector embeddings.

[3]
Successfully created collection: JeopardyQuestion.

{"build_git_commit":"b7b77150ac","build_go_version":"go1.24.2","build_image_tag":"HEAD","build_wv_version":"1.30.0","level":"warning","msg":"prop len tracker file /Users/erikacardenas/.local/share/weaviate/jeopardyquestion/UNvR5jmmimUt/proplengths does not exist, creating new tracker","time":"2025-05-02T15:09:01-04:00"}
{"action":"hnsw_prefill_cache_async","build_git_commit":"b7b77150ac","build_go_version":"go1.24.2","build_image_tag":"HEAD","build_wv_version":"1.30.0","level":"info","msg":"not waiting for vector cache prefill, running in background","time":"2025-05-02T15:09:01-04:00","wait_for_cache_prefill":false}
{"build_git_commit":"b7b77150ac","build_go_version":"go1.24.2","build_image_tag":"HEAD","build_wv_version":"1.30.0","level":"info","msg":"Created shard jeopardyquestion_UNvR5jmmimUt in 10.411875ms","time":"2025-05-02T15:09:01-04:00"}
{"action":"hnsw_vector_cache_prefill","build_git_commit":"b7b77150ac","build_go_version":"go1.24.2","build_image_tag":"HEAD","build_wv_version":"1.30.0","count":1000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2025-05-02T15:09:01-04:00","took":143292}

Import the Data

[4]
Insert complete.

Hybrid Search

The alpha parameter determines the weight given to the sparse and dense search methods. alpha = 0 is pure sparse (bm25) search, whereas alpha = 1 is pure dense (vector) search.

Alpha is an optional parameter. The default is set to 0.75.

Hybrid Search only

The below query is finding Jeopardy questions about animals and is limiting the output to only two results. Notice alpha is set to 0.80, which means it is weighing the vector search results more than bm25. If you were to set alpha = 0.25, you would get different results.

[5]
ID: 2a35dc97-40c1-4ab1-bd4c-baaa63fd3b06
Data: {
  "answer": "species",
  "question": "2000 news: the Gunnison sage grouse isn't just another northern sage grouse, but a new one of this classification",
  "category": "SCIENCE"
} 

ID: e5aca3f1-1f00-435c-9d9b-c30ada64d128
Data: {
  "answer": "the diamondback rattler",
  "question": "Heaviest of all poisonous snakes is this North American rattlesnake",
  "category": "ANIMALS"
} 

ID: 392fbc4c-fecf-4ffe-873c-d87e91f66e4a
Data: {
  "answer": "Antelope",
  "question": "Weighing around a ton, the eland is the largest species of this animal in Africa",
  "category": "ANIMALS"
}

Hybrid Search with a where filter

Find Jeopardy questions about elephants, where the category is set to Animals.

[6]
ID: e5aca3f1-1f00-435c-9d9b-c30ada64d128
Data: {
  "answer": "the diamondback rattler",
  "question": "Heaviest of all poisonous snakes is this North American rattlesnake",
  "category": "ANIMALS"
} 

ID: 392fbc4c-fecf-4ffe-873c-d87e91f66e4a
Data: {
  "answer": "Antelope",
  "question": "Weighing around a ton, the eland is the largest species of this animal in Africa",
  "category": "ANIMALS"
} 

ID: f1163de8-4035-46e0-9f5b-6304c3dcaa50
Data: {
  "answer": "the nose or snout",
  "question": "The gavial looks very much like a crocodile except for this bodily feature",
  "category": "ANIMALS"
}