Elastic With Langchain Splitters

With Langchain Splitters

document-chunkingopenai-chatgptlangchain-pythonchatgptgenaielasticsearchelasticopenaiAIchatlogvectordatabasenotebooksPythonsearchgenaistackvectorelasticsearch-labslangchainapplications

alph-notebooks/elasticsearch-labs / with-langchain-splitters.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

Document Chunking With LangChain Document Splitters

Using Elasticsearch Nested Dense Vector Support

This interactive notebook will:

load the model "sentence-transformers__all-minilm-l6-v2" from Hugging Face and into Elasticsearch ML Node
Use LangChain splitters to chunk the passages into sentences and index them into Elasticsearch with nested dense vector
perform a search and return docs with the most relevant passages

Prefer the `semantic_text` field type

Elasticsearch version 8.15 introduced the semantic_text field type which handles the chunking process behind the scenes. Before continuing with this notebook, we highly recommend looking into this:

Dependencies

In this notebook, we're going to use Langchain and the Elasticsearch python client.

We will also require a running Elasticsearch instance with an ML node and model deployed to it.

[ ]

Connect to Elasticsearch

ℹ️ We're using an Elastic Cloud deployment of Elasticsearch for this notebook. If you don't have an Elastic Cloud deployment, sign up here for a free trial.

We'll use the Cloud ID to identify our deployment, because we are using Elastic Cloud deployment. To find the Cloud ID for your deployment, go to https://cloud.elastic.co/deployments and select your deployment.

[ ]

[3]

Download our example Dataset

We are going to use Langchain's tooling to ingest and split raw documents into smaller chunks. We are using our example workplace search dataset.

LangChain has a number of other loaders to ingest data from other sources. See their core loaders or loaders integration for more information.

[4]

[5]

Load Model from hugging face

The first thing you will need is a model to create the text embeddings out of the chunks, you can use whatever you would like, but this example will run end to end on the minilm-l6-v2 model. With an Elastic Cloud cluster created or another Elasticsearch cluster ready, we can upload the text embedding model using the eland library.

[ ]

Setting up our Elasticsearch Index

In this example we're going to use a pipeline to do the inference and store the embeddings in our index.

In this example, we are using the sentence transformers minilm-l6-v2 model, which you will need to is running on the ML node. With this model, we are setting up an index_pipeline to do the inference and store the embeddings in our index.

[11]

ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'nb_parent_retriever_index'})

Utils: Parent Child Splitter Function

This function will split a document into multiple passages, and return the parent document with the child passages.

It also has an option to chunk the parent document into smaller documents, meaning the parent document will be split into multiple index documents. We will use this in example 2.

[12]

Utils: Pretty Response

This function will print out the response from Elasticsearch in an easier to read format.

[13]

Full Document, nested passages

In this example we will split a document into passages, and store the full document as a parent document. We will then store the passages as nested documents, with a link back to the parent document.

Below we are using the parent child splitter to split the full documents into passages. The parent_child_splitter fn returns a list of documents, with an array of nested passages.

We then index these documents into Elasticsearch. This will index the full document and the passages will be stored in a nested field.

Our index pipeline processor will then run the inference on the passages, and store the embeddings in the index.

[14]

Indexed 15 documents with [] errors

Perform a Nested Search

We can now perform a nested search, to find the passages that match our query, which will be returned in inner_hits. In the example that follows only one passage per parent document is requested.

[15]

ID: 1XvjyowBidHK_OJxJozM
Doc Title: Work From Home Policy
parent text:

Passage Text:
Effective: March 2020
Purpose

The purpose of this full-time work-from-home policy is to provide guidelines and support for employees to conduct their work remotely, ensuring the continuity and productivity of business operations during the COVID-19 pandemic and beyond.
Scope

This policy applies to all employees who are eligible for remote work as determined by their role and responsibilities. It is designed to allow employees to work from home full time while maintaining the same level of performance and collaboration as they would in the office.
Eligibility

Score: 0.84830964

---

ID: 3HvjyowBidHK_OJxJozM
Doc Title: Intellectual Property Policy
parent text:

Passage Text:
Purpose
The purpose of this Intellectual Property Policy is to establish guidelines and procedures for the ownership, protection, and utilization of intellectual property generated by employees during their employment. This policy aims to encourage creativity and innovation while ensuring that the interests of both the company and its employees are protected.

Scope
This policy applies to all employees, including full-time, part-time, temporary, and contract employees.

Score: 0.7292882

---

ID: 2XvjyowBidHK_OJxJozM
Doc Title: Company Vacation Policy
parent text:

Passage Text:
Purpose

The purpose of this vacation policy is to outline the guidelines and procedures for requesting and taking time off from work for personal and leisure purposes. This policy aims to promote a healthy work-life balance and encourage employees to take time to rest and recharge.
Scope

This policy applies to all full-time and part-time employees who have completed their probationary period.
Vacation Accrual

Score: 0.7137784

---

ID: 13vjyowBidHK_OJxJozM
Doc Title: Wfh Policy Update May 2023
parent text:

Passage Text:
As we continue to prioritize the well-being of our employees, we are making a slight adjustment to our hybrid work policy. Starting May 1, 2023, employees will be required to work from the office three days a week, with two days designated for remote work. Please communicate with your supervisor and HR department to establish your updated in-office workdays.

Score: 0.70840263

---

With Langchain

We can also peform this search within Langchain with an adjustment to the query.

We also override the doc_builder to populate the site_content with the passages rather than the full document.

[18]

Doc title: Work From Home Policy
Text:
Effective: March 2020
Purpose

Doc title: Intellectual Property Policy
Text:
Purpose
The purpose of this Intellectual Property Policy is to establish guidelines and procedures for the ownership, protection, and utilization of intellectual property generated by employees during their employment. This policy aims to encourage creativity and innovation while ensuring that the interests of both the company and its employees are protected.