Weaviate Rag With Weaviate And Cleanlab

Rag With Weaviate And Cleanlab

vector-searchvector-databaseretrieval-augmented-generationllm-frameworksoperationsfunction-callingweaviate-recipesintegrationsPythongenerative-aicleanlab

alph-notebooks/weaviate-recipes / rag_with_weaviate_and_cleanlab.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

Deploy Trustworthy RAG with Weaviate and Cleanlab

Large Language Models (LLMs) occasionally hallucinate incorrect answers, especially for questions not well-supported within their training data. Retrieval Augmented Generation (RAG) mitigates this by supplying LLMs with context retrieved from knowledge databases like Weaviate. While organizations are rapidly adopting RAG to pair the power of LLMs with their own proprietary data, hallucinated/incorrect RAG responses remain a problem.

This tutorial demonstrates an easy solution: build trustworthy RAG applications using Weaviate and Cleanlab to mitigate hallucinated/incorrect responses.

Cleanlab boosts the reliability of any LLM application by scoring when LLM responses are untrusworthy. Trust scoring happens in real-time and does not require any data labeling or model training work. Cleanlab provides additional real-time Evals for other RAG components like the retrieved context, which help you diagnose why certain RAG responses and inaccurate/bad. Cleanlab's detection enables you to automatically flag/prevent inaccurate/bad responses from your RAG app, and avoid losing your users' trust.

Weaviate streamlines RAG application development, providing a scalable vector database to store your organization's knowledge, as well vectorizers and integrations with generative model providers (like OpenAI & Anthropic). By providing all of the necessary components via a simple developer experience, Weaviate enables you to quickly deploy a performant RAG application. Adding Cleanlab on top provides an additional layer of trust.

Setup

For this tutorial, we will need:

Weaviate Cloud Cluster: A managed vector database (DB) instance provided by Weaviate. Follow the quickstart to create one
Cleanlab API Key: Sign up at tlm.cleanlab.ai/ to get a free key
OpenAI API Key: To make completion requests to an LLM

Start by installing the required dependencies. We use the following versions for developing this notebook:

Weaviate Database Version == 1.30.1
weaviate-client==4.11.1
cleanlab_tlm==1.1.0

[ ]

[1]

Set the retrieved keys as environment variables

[ ]

[3]

True

Now, we initialize Cleanlab's object with default settings. You can achieve better results or lower latency by adjusting the configurations.

[ ]

Create collection in vector database

A collection in Weaviate is synonymous with an index in a database. Here, we will instantiate the collection by specifying two key configurations:

Vectorizer / Embedding Model - Weaviate offers in-house managed embedding service and third-party integrations (like text-embedding from OpenAI). We use Weaviate Embedding service here.
Generator Model - We use Weaviate's integration with OpenAI's API to seamlessly generate an answer based on the user query and retrieved context.

[5]

Read data

RAG is all about connecting LLMs to data, to better inform their answers. This tutorial uses Nvidia’s Q1 FY2024 earnings report as an example data source for populating the RAG application's knowledge base.

[ ]

# NVIDIA Announces Financial Results for First Quarter Fiscal 2024

NVIDIA (NASDAQ: NVDA) today reported revenue for the first quarter ended April 30, 2023, of $7.19 billion, down 13% from a year ago

Chunk documents

To control the size of each context embedded in our vector index and provided to the LLM, we split documents into smaller chunks. When a user submits a query to the RAG system, the relevant chunks are retrieved from the vector database and used as context for the LLM to generate a response. Embedding just a few words provides too little information in each context, while embedding entire documents makes it harder to retrieve specific snippets of useful information for accurate responses.
It's important to find the right chunk size for your use-case, depending on the types of documents and questions that are handled. Here we use a simple chunking strategy, splitting the text based on a fixed chunk_size. Additionally, chunks overlap based on a specified overlap_size.

[7]

[ ]

The document is divided into 8 chunks. Now, let's upload them to our vector index.

Upsert chunks & vectors to Weaviate

[ ]

Run a quick test to verify that all 8 chunks have been upserted and vectorized.

[ ]

Ask questions to our RAG application

Now that the Weaviate database is loaded with text chunks and their corresponding embeddings, we can start querying it to answer questions.

Easy

We first pose a straightforward question that can be directly answered by the provided data and easily located within a few lines of text.

[11]

[23]

# NVIDIA Announces Financial Results for First Quarter Fiscal 2024 NVIDIA (NASDAQ: NVDA) today reported revenue for the first quarter ended April 30, 2023, of $7.19 billion, down 13% from a year ago and up 19% from the previous quarter. - **Quarterly revenue** of $7.19 billion, up 19% from the previous quarter - **Record Data Center revenue** of $4.28 billion - **Second quarter fiscal 2024 revenue outlook** of $11.00 billion GAAP earnings per diluted share for the quarter were $0.82, up 28% from a year ago and up 44% from the previous quarter. Non-GAAP earnings per diluted share were $1.09, down 20% from a year ago and up 24% from the previous quarter. Jensen Huang, founder and CEO of NVIDIA, commented on the significant transitions the computer industry is undergoing, particularly accelerated computing and generative AI, emphasizing NVIDIA's role and advancements in these areas. During the first quarter of fiscal
Earnings Per Share | $1.09 | $0.88 | $1.36 | Up 24% | Down 20% | ## Outlook NVIDIA’s outlook for the second quarter of fiscal 2024 includes: - **Revenue**: Expected to be $11.00 billion, plus or minus 2%. - **Gross Margins**: GAAP and non-GAAP gross margins are expected to be 68.6% and 70.0%, respectively, plus or minus 50 basis points. - **Operating Expenses**: GAAP and non-GAAP operating expenses are expected to be approximately $2.71 billion and $1.90 billion, respectively. - **Tax Rates**: GAAP and non-GAAP tax rates are expected to be 14.0%, plus or minus 1%, excluding any discrete items. ## Highlights NVIDIA has made significant progress in various areas since its last earnings announcement: ### Data Center - **First-quarter revenue** was a record $4.28 billion, up 14% from a year ago and up 18% from the previous quarter. - Launched **four inference platforms** that combine the company’s full-stack inference software with the latest NVIDIA Ada, NVIDIA Hopper™, and NVIDIA Grace Hopper™ processors. - Announced that **Google Cloud** is the first cloud provider offering

Let's format these chunks into a prompt for the LLM, and call the OpenAI API to get the response.

[33]

NVIDIA's total revenue in the first quarter of fiscal 2024 was $7.19 billion.

We can easily verify the above response is correct as its evident in the retrieved chunks.

Let's automatically evaluate the response. TrustworthyRAG runs Cleanlab's state-of-the-art LLM uncertainty estimator, the Trustworthy Language Model, to provide a trustworthiness score indicating overall confidence that your RAG's response is correct.

TrustworthyRAG can simultaneously run additional evaluations to diagnose why responses are likely incorrect or other types of issues.
Let's see what Evals are run by default:

[47]

context_sufficiency
response_groundedness
response_helpfulness
query_ease

Each Eval returns a score between 0-1 (higher is better) that assesses a different aspect of your RAG system:

context_sufficiency: Evaluates whether the retrieved context contains sufficient information to completely answer the query. A low score indicates that key information is missing from the context (perhaps due to poor retrieval or missing documents).
response_groundedness: Evaluates whether claims/information stated in the response are explicitly supported by the provided context.
response_helpfulness: Evaluates whether the response effectively addresses the user query and appears helpful.
query_ease: Evaluates whether the user query seems easy for an AI system to properly handle. Complex, vague, tricky, or disgruntled-sounding queries receive lower scores.

To run TrustworthyRAG, we need the prompts sent to the LLM, which includes the system message, retrieved chunks, the user's query, and the LLM's response.

[34]

Evaluation results:
trustworthiness: 1.0
context_sufficiency: 0.9975123397108121
response_groundedness: 0.9975124375711611
response_helpfulness: 0.9975124251881636
query_ease: 0.9974677786997358

Analysis: The trustworthiness_score of 1.0 indicates that this response is reliable and correct i.e. non-hallucinated. Even the context is sufficient to respond to this easy query, which is captured by the high context_sufficiency and query_ease scores.

Let's define a function that runs the above workflow.

[83]

Now let’s run a out-of-data query that cannot be answered using the provided document.

[75]

Query:
How does the report explain why NVIDIA's Gaming revenue decreased year over year?

Context:
# NVIDIA Announces Financial Results for First Quarter Fiscal 2024 NVIDIA (NASDAQ: NVDA) today reported revenue for the first quarter ended April 30, 2023, of $7.19 billion, down 13% from a year ago and up 19% from the previous quarter. - **Quarterly revenue** of $7.19 billion, up 19% from the previous quarter - **Record Data Center revenue** of $4.28 billion - **Second quarter fiscal 2024 revenue outlook** of $11.00 billion GAAP earnings per diluted share for the quarter were $0.82, up 28% from a year ago and up 44% from the previous quarter. Non-GAAP earnings per diluted share were $1.09, down 20% from a year ago and up 24% from the previous quarter. Jensen Huang, founder and CEO of NVIDIA, commented on the significant transitions the computer industry is undergoing, particularly accelerated computing and generative AI, emphasizing NVIDIA's role and advancements in these areas. During the first quarter of fiscal
Up 70% | Up 15% | | Net Income | $2,043 | $1,414 | $1,618 | Up 44% | Up 26% | | Diluted Earnings Per Share | $0.82 | $0.57 | $0.64 | Up 44% | Up 28% | ### Non-GAAP | ($ in millions, except earnings per share) | Q1 FY24 | Q4 FY23 | Q1 FY23 | Q/Q Change | Y/Y Change | |---------------------------------------------|---------|---------|---------|------------|------------| | Revenue | $7,192 | $6,051 | $8,288 | Up 19% | Down 13% | | Gross Margin | 66.8% | 66.1% | 67.1% | Up 0.7 pts | Down 0.3 pts | | Operating Expenses | $1,750 | $1,775 | $1,608 | Down 1% | Up 9% | | Operating Income | $3,052 | $2,224 | $3,955 | Up 37% | Down 23% | | Net Income | $2,713 | $2,174 | $3,443 | Up 25% | Down 21% | | Diluted Earnings Per Share | $1.09 | $0.88 | $1.36 | Up 24% | Down 20% | ## Outlook NVIDIA’s outlook for the second quarter of

Generated response:
The provided context does not specify the reasons for the decrease in NVIDIA's Gaming revenue year over year. It mainly focuses on the overall financial results for the first quarter of fiscal 2024, highlighting various metrics such as quarterly revenue, net income, and earnings per share, but does not delve into the specifics of the Gaming division or the factors contributing to its decline. For information on the reasons behind the decrease in Gaming revenue, additional details or a separate report from NVIDIA would be necessary.

Evaluation results:
trustworthiness: 0.9347711807707823
context_sufficiency: 0.28861326055082803
response_groundedness: 0.9882697144951728
response_helpfulness: 0.24692539926446214
query_ease: 0.754295489462312

Analysis: The generator LLM avoids conjecture by providing a reliable response, as seen in the high trustworthiness_score. The low context_sufficiency score reflects that the retrieved context was lacking, and the response doesn’t actually answer the user’s query, as indicated by the low response_helpfulness.

Let’s see how our RAG system responds to challenging questions, which may be misleading.

[79]

Query:
How much did Nvidia's revenue decrease this quarter vs last quarter, in dollars?

Context:
# NVIDIA Announces Financial Results for First Quarter Fiscal 2024 NVIDIA (NASDAQ: NVDA) today reported revenue for the first quarter ended April 30, 2023, of $7.19 billion, down 13% from a year ago and up 19% from the previous quarter. - **Quarterly revenue** of $7.19 billion, up 19% from the previous quarter - **Record Data Center revenue** of $4.28 billion - **Second quarter fiscal 2024 revenue outlook** of $11.00 billion GAAP earnings per diluted share for the quarter were $0.82, up 28% from a year ago and up 44% from the previous quarter. Non-GAAP earnings per diluted share were $1.09, down 20% from a year ago and up 24% from the previous quarter. Jensen Huang, founder and CEO of NVIDIA, commented on the significant transitions the computer industry is undergoing, particularly accelerated computing and generative AI, emphasizing NVIDIA's role and advancements in these areas. During the first quarter of fiscal
Earnings Per Share | $1.09 | $0.88 | $1.36 | Up 24% | Down 20% | ## Outlook NVIDIA’s outlook for the second quarter of fiscal 2024 includes: - **Revenue**: Expected to be $11.00 billion, plus or minus 2%. - **Gross Margins**: GAAP and non-GAAP gross margins are expected to be 68.6% and 70.0%, respectively, plus or minus 50 basis points. - **Operating Expenses**: GAAP and non-GAAP operating expenses are expected to be approximately $2.71 billion and $1.90 billion, respectively. - **Tax Rates**: GAAP and non-GAAP tax rates are expected to be 14.0%, plus or minus 1%, excluding any discrete items. ## Highlights NVIDIA has made significant progress in various areas since its last earnings announcement: ### Data Center - **First-quarter revenue** was a record $4.28 billion, up 14% from a year ago and up 18% from the previous quarter. - Launched **four inference platforms** that combine the company’s full-stack inference software with the latest NVIDIA Ada, NVIDIA Hopper™, and NVIDIA Grace Hopper™ processors. - Announced that **Google Cloud** is the first cloud provider offering

Generated response:
NVIDIA's revenue for the first quarter ended April 30, 2023, was $7.19 billion. The report indicates that this was an increase of 19% from the previous quarter. To find the revenue for the previous quarter (fourth quarter of fiscal 2023), we can use the following calculation:

Let $ x $ be the revenue from the previous quarter. The equation based on the increase is:

$ x + (0.19x) = 7.19 $

This simplifies to:

$ 1.19x = 7.19 $

Now, solve for $ x $:

$ x = \frac{7.19}{1.19} \approx 6.04 $ billion.

Now to find the dollar amount of the decrease from the previous quarter to this quarter:

The revenue decreased by:

$ 7.19 - 6.04 = 1.15 $ billion dollars.

So, NVIDIA's revenue decreased by approximately **$1.15 billion** this quarter compared to the last quarter.

Evaluation results:
trustworthiness: 0.07622373797481402
context_sufficiency: 0.9868578610267666
response_groundedness: 0.0413584486687836
response_helpfulness: 0.9712986992618623
query_ease: 0.3784677848937411

Analysis: The generated response incorrectly states that NVIDIA's revenue decreased this quarter, when in fact the referenced report notes a 19% increase quarter-over-quarter.

This mismatch between the query and the context leads to a very low response_groundedness score of 0.04 and a trustworthiness_score of just 0.08, indicating that the answer not only fabricates financial trends but also misleads with confident but false arithmetic.

Let's try another one:

[87]

Query:
If NVIDIA's Data Center segment maintains its Q1 FY2024 quarter-over-quarter growth rate for the next four quarters, what would be its projected annual revenue?

Context:
Earnings Per Share | $1.09 | $0.88 | $1.36 | Up 24% | Down 20% | ## Outlook NVIDIA’s outlook for the second quarter of fiscal 2024 includes: - **Revenue**: Expected to be $11.00 billion, plus or minus 2%. - **Gross Margins**: GAAP and non-GAAP gross margins are expected to be 68.6% and 70.0%, respectively, plus or minus 50 basis points. - **Operating Expenses**: GAAP and non-GAAP operating expenses are expected to be approximately $2.71 billion and $1.90 billion, respectively. - **Tax Rates**: GAAP and non-GAAP tax rates are expected to be 14.0%, plus or minus 1%, excluding any discrete items. ## Highlights NVIDIA has made significant progress in various areas since its last earnings announcement: ### Data Center - **First-quarter revenue** was a record $4.28 billion, up 14% from a year ago and up 18% from the previous quarter. - Launched **four inference platforms** that combine the company’s full-stack inference software with the latest NVIDIA Ada, NVIDIA Hopper™, and NVIDIA Grace Hopper™ processors. - Announced that **Google Cloud** is the first cloud provider offering
# NVIDIA Announces Financial Results for First Quarter Fiscal 2024 NVIDIA (NASDAQ: NVDA) today reported revenue for the first quarter ended April 30, 2023, of $7.19 billion, down 13% from a year ago and up 19% from the previous quarter. - **Quarterly revenue** of $7.19 billion, up 19% from the previous quarter - **Record Data Center revenue** of $4.28 billion - **Second quarter fiscal 2024 revenue outlook** of $11.00 billion GAAP earnings per diluted share for the quarter were $0.82, up 28% from a year ago and up 44% from the previous quarter. Non-GAAP earnings per diluted share were $1.09, down 20% from a year ago and up 24% from the previous quarter. Jensen Huang, founder and CEO of NVIDIA, commented on the significant transitions the computer industry is undergoing, particularly accelerated computing and generative AI, emphasizing NVIDIA's role and advancements in these areas. During the first quarter of fiscal

Generated response:
NVIDIA's Data Center segment generated revenue of $4.28 billion in Q1 FY2024, which was up 18% from the previous quarter. To project its annual revenue while maintaining this growth rate for the next four quarters, we can first calculate the revenue for the upcoming quarters.

1. Calculate the Q2 revenue:
   - Q1 revenue = $4.28 billion
   - Q2 revenue = Q1 revenue × (1 + growth rate)
   - Q2 revenue = $4.28 billion × (1 + 0.18) = $4.28 billion × 1.18 = $5.06 billion

2. Calculate the Q3 revenue:
   - Q3 revenue = Q2 revenue × (1 + growth rate)
   - Q3 revenue = $5.06 billion × 1.18 = $5.96 billion

3. Calculate the Q4 revenue:
   - Q4 revenue = Q3 revenue × (1 + growth rate)
   - Q4 revenue = $5.96 billion × 1.18 = $7.03 billion

4. Calculate the Q5 revenue (which would be the revenue for the first quarter of FY2025):
   - Q5 revenue = Q4 revenue × (1 + growth rate)
   - Q5 revenue = $7.03 billion × 1.18 = $8.28 billion

Now, to find the projected annual revenue, we sum the revenues for four quarters:
- Annual projected revenue = Q1 + Q2 + Q3 + Q4 + Q5
- Annual projected revenue = $4.28 billion + $5.06 billion + $5.96 billion + $7.03 billion + $8.28 billion = $30.61 billion

Thus, if NVIDIA's Data Center segment maintains its Q1 FY2024 quarter-over-quarter growth rate for the next four quarters, its projected annual revenue would be approximately **$30.61 billion**.

Evaluation results:
trustworthiness: 0.268388115138118
context_sufficiency: 0.35389089942877244
response_groundedness: 0.4707081971638484
response_helpfulness: 0.9975071025542379
query_ease: 0.7518852806087112

Analysis: The generated response overstates (sums up the financials of Q1) the projected revenue and leads to a low trustworthiness_score of 0.27.

Custom Evals

You can also specify custom evaluations to assess specific criteria, and combine them with the default evaluations for comprehensive/tailored assessment of your RAG system.

For instance, here's how to create and run a custom eval that checks the conciseness of the generated response.

[89]

[101]

Query:
What significant transitions did Jensen comment on?

Context:
# NVIDIA Announces Financial Results for First Quarter Fiscal 2024 NVIDIA (NASDAQ: NVDA) today reported revenue for the first quarter ended April 30, 2023, of $7.19 billion, down 13% from a year ago and up 19% from the previous quarter. - **Quarterly revenue** of $7.19 billion, up 19% from the previous quarter - **Record Data Center revenue** of $4.28 billion - **Second quarter fiscal 2024 revenue outlook** of $11.00 billion GAAP earnings per diluted share for the quarter were $0.82, up 28% from a year ago and up 44% from the previous quarter. Non-GAAP earnings per diluted share were $1.09, down 20% from a year ago and up 24% from the previous quarter. Jensen Huang, founder and CEO of NVIDIA, commented on the significant transitions the computer industry is undergoing, particularly accelerated computing and generative AI, emphasizing NVIDIA's role and advancements in these areas. During the first quarter of fiscal
Cyberpunk 2077. - Expanded **GeForce NOW’s** game titles to more than 1,600, including the first Microsoft Xbox game, Gears 5. ### Professional Visualization - First-quarter revenue was $295 million, down 53% from a year ago and up 31% from the previous quarter. - Announced **NVIDIA Omniverse™ Cloud**, a fully managed service running in Microsoft Azure, for the development and deployment of industrial metaverse applications. - Expanded its collaboration with **Microsoft** to connect Microsoft 365 applications with Omniverse. - Announced **six new NVIDIA RTX™ GPUs** for mobile and desktop workstations based on the Ada architecture. ### Automotive - First-quarter revenue was a record $296 million, up 114% from a year ago and up 1% from the previous quarter. - Announced that its automotive design win pipeline has grown to **$14 billion over the next six years**, up from $11 billion a year ago. - Announced that the world’s leading electric vehicle maker **BYD** will extend its use of NVIDIA DRIVE Orin™ across new models. ## Financial Statements and Non-GAAP Measures Detailed financial statements and reconciliation of

Generated response:
Jensen Huang, the founder and CEO of NVIDIA, commented on the significant transitions the computer industry is undergoing, particularly in the areas of accelerated computing and generative AI.

Evaluation results:
trustworthiness: 0.9910890553199009
context_sufficiency: 0.9930264352424849
response_groundedness: 0.9975120964465403
response_helpfulness: 0.7485612752531596
query_ease: 0.3465444527839642
response_conciseness: 0.7458295840526216

Replace your LLM with Cleanlab's

Beyond evaluating responses already generated from your LLM, Cleanlab can also generate responses and evaluate them simultaneously (using one of many supported models).
This replaces your own LLM within your RAG system and can be more convenient/accurate/faster.

Let's replace OpenAI LLM with a line to call Cleanlab's endpoint:

[102]

Query:
How much did Nvidia's revenue decrease this quarter vs last quarter, in dollars?

Context:
# NVIDIA Announces Financial Results for First Quarter Fiscal 2024 NVIDIA (NASDAQ: NVDA) today reported revenue for the first quarter ended April 30, 2023, of $7.19 billion, down 13% from a year ago and up 19% from the previous quarter. - **Quarterly revenue** of $7.19 billion, up 19% from the previous quarter - **Record Data Center revenue** of $4.28 billion - **Second quarter fiscal 2024 revenue outlook** of $11.00 billion GAAP earnings per diluted share for the quarter were $0.82, up 28% from a year ago and up 44% from the previous quarter. Non-GAAP earnings per diluted share were $1.09, down 20% from a year ago and up 24% from the previous quarter. Jensen Huang, founder and CEO of NVIDIA, commented on the significant transitions the computer industry is undergoing, particularly accelerated computing and generative AI, emphasizing NVIDIA's role and advancements in these areas. During the first quarter of fiscal
Earnings Per Share | $1.09 | $0.88 | $1.36 | Up 24% | Down 20% | ## Outlook NVIDIA’s outlook for the second quarter of fiscal 2024 includes: - **Revenue**: Expected to be $11.00 billion, plus or minus 2%. - **Gross Margins**: GAAP and non-GAAP gross margins are expected to be 68.6% and 70.0%, respectively, plus or minus 50 basis points. - **Operating Expenses**: GAAP and non-GAAP operating expenses are expected to be approximately $2.71 billion and $1.90 billion, respectively. - **Tax Rates**: GAAP and non-GAAP tax rates are expected to be 14.0%, plus or minus 1%, excluding any discrete items. ## Highlights NVIDIA has made significant progress in various areas since its last earnings announcement: ### Data Center - **First-quarter revenue** was a record $4.28 billion, up 14% from a year ago and up 18% from the previous quarter. - Launched **four inference platforms** that combine the company’s full-stack inference software with the latest NVIDIA Ada, NVIDIA Hopper™, and NVIDIA Grace Hopper™ processors. - Announced that **Google Cloud** is the first cloud provider offering

[103]

Generated Response:
NVIDIA's revenue for the first quarter was $7.19 billion, which is up 19% from the previous quarter. To find the revenue for the previous quarter, we can calculate it as follows:

Let \( x \) be the revenue for the previous quarter. According to the information provided:

\[ x + 0.19x = 7.19 \]
\[ 1.19x = 7.19 \]
\[ x = \frac{7.19}{1.19} \]
\[ x \approx 6.04 \text{ billion} \]

Now, to find the decrease in revenue from the previous quarter to this quarter, we can calculate:

\[ 7.19 - 6.04 = 1.15 \text{ billion} \]

Therefore, NVIDIA's revenue did not decrease this quarter compared to the last quarter; instead, it increased by approximately $1.15 billion.

Evaluation Scores:
trustworthiness: 0.7047853317728329
context_sufficiency: 0.9793883585825877
response_groundedness: 0.16683115793235678
response_helpfulness: 0.2743783166467349
query_ease: 0.41290746161713915

While it remains hard to achieve a RAG application that will accurately answer any possible question, you can easily use Weaviate and Cleanlab to deploy a trustworthy RAG application which at least flags answers that are likely inaccurate.