Watsonx Example
agentsllmsvector-databaselancedbgptopenaiAImultimodal-aimachine-learningembeddingsRAG-with-watsonxfine-tuningexamplesdeep-learninggpt-4-visionllama-indexragmultimodallangchainlancedb-recipes
Export
Retrieval-Augmented Generation(RAG) using IBM Watsonx
In this notebook, we will build a Retrieval-Augmented Generation(RAG) using IBM Watsonx embeddings and GPT 3.5 LLM .
RAG is a technique that retrieves related documents to the user's question, combines them with LLM-base prompt, and sends them to LLMs like GPT to produce more factually accurate generation.
In this example, we'll use DeadPool and Wolverine marvel page to build RAG around it.
Install Dependencies
[ ]
[ ]
Load documents from webpage
[ ]
[ ]
1
Chunking
[ ]
'# Marvel Cinematic Universe Wiki\n\nWith the release of _[Deadpool& Wolverine](/wiki/Deadpool_%26_Wolverine\n"Deadpool & Wolverine")_ we\'d like to remind the community of some things.\nPlease DO NOT add info from the Fox X-Men movies when it comes to pages. Only\ninformation stated within the movie itself is eligible to be added. In\naddition, do not upload pirated images from cam copies. We\'d also like to ask\nthat universe pages NOT be created if the universe name has a Placeholder\ntemplate on it, as that indicates it is not the intended final name. Thank\nyou.\n\nREAD MORE\n\n[ \n](//marvelcinematicuniverse.fandom.com) [Marvel Cinematic Universe\nWiki](//marvelcinematicuniverse.fandom.com)\n\n * Explore\n\n * [ Main Page ](https://marvelcinematicuniverse.fandom.com/wiki/Marvel_Cinematic_Universe_Wiki)\n * [ Discuss ](/f)\n * [ All Pages ](https://marvelcinematicuniverse.fandom.com/wiki/Special:AllPages)\n * [ Community ](https://marvelcinematicuniverse.fandom.com/wiki/Special:Community)\n * [ Interactive Maps ](https://marvelcinematicuniverse.fandom.com/wiki/Special:AllMaps)\n * [ Recent Blog Posts ](/Blog:Recent_posts)\n\n * [ Universe ](https://marvelcinematicuniverse.fandom.com/wiki/Marvel_Cinematic_Universe)\n\n * [ Organizations ](https://marvelcinematicuniverse.fandom.com/wiki/Category:Organizations)\n\n * [ S.H.I.E.L.D. ](https://marvelcinematicuniverse.fandom.com/wiki/S.H.I.E.L.D.)\n * [ HYDRA ](https://marvelcinematicuniverse.fandom.com/wiki/HYDRA)\n * [ Stark Industries ](https://marvelcinematicuniverse.fandom.com/wiki/Stark_Industries)\n * [ United States Armed Forces ](https://marvelcinematicuniverse.fandom.com/wiki/United_States_Armed_Forces)\n * [ Strategic Scientific Reserve ](https://marvelcinematicuniverse.fandom.com/wiki/Strategic_Scientific_Reserve)\n * [ Roxxon Oil Corporation ](https://marvelcinematicuniverse.fandom.com/wiki/Roxxon_Oil_Corporation)\n * [ New York City Police Department ](https://marvelcinematicuniverse.fandom.com/wiki/New_York_City_Police_Department)\n * [ more. ](https://marvelcinematicuniverse.fandom.com/wiki/Category:Organizations)\n\n * [ Teams ](https://marvelcinematicuniverse.fandom.com/wiki/Category:Teams)\n\n * [ Avengers ](https://marvelcinematicuniverse.fandom.com/wiki/Avengers)\n * [ Guardians of the Galaxy ](https://marvelcinematicuniverse.fandom.com/wiki/Guardians_of_the_Galaxy)\n * [ Secret Warriors ](https://marvelcinematicuniverse.fandom.com/wiki/Secret_Warriors)\n * [ Defenders ](https://marvelcinematicuniverse.fandom.com/wiki/Defenders)\n * [ Runaways ](https://marvelcinematicuniverse.fandom.com/wiki/Runaways)\n * [ Thunderbolts ](https://marvelcinematicuniverse.fandom.com/wiki/Thunderbolts)\n * [ more. ](https://marvelcinematicuniverse.fandom.com/wiki/Category:Teams)\n\n * [ Characters ](https://marvelcinematicuniverse.fandom.com/wiki/Category:Characters)\n\n * [ Iron Man ](https://marvelcinematicuniverse.fandom.com/wiki/Iron_Man)\n * [ Captain America ](https://marvelcinematicuniverse.fandom.com/wiki/Captain_America)\n * [ Thor ](https://marvelcinematicuniverse.fandom.com/wiki/Thor)\n * [ Hulk ](https://marvelcinematicuniverse.fandom.com/wiki/Hulk)\n * [ Ant-Man ](https://marvelcinematicuniverse.fandom.com/wiki/Ant-Man)\n * [ Spider-Man ](https://marvelcinematicuniverse.'
Set Watson and OpenAI creds as env variable
[ ]
[ ]
OPENAI_API_KEY is already set WATSONX_API_KEY is already set WATSONX_PROJECT_ID is already set
Define custom Embedding function for watsonx model
[ ]
Create Table
[ ]
Query
Query ingested webpage
[ ]
[ ]
['Despite\nDeadpool\'s antagonism towards Nicepool, the latter gives them directions to\nthe Void\'s borderlands and provides them with a car to make it there. While\ndriving, Deadpool accidentally slips up that he is not certain that the TVA\nwill fix Wolverine\'s reality, admitting that he made an "educated wish".\nWolverine is enraged and lashes out at Deadpool, which results in both of them\nfighting each other until both are too exhausted, and someone drives their car\nto the Resistance hideout.\n\n[](https://static.wikia.nocookie.net/marvelcinematicuniverse/images/1/13/Deadpool_%26_Wolverine_Trailer_%2864%29.png/revision/latest?cb=20240422182040)\n[](https://static.wikia.nocookie.net/marvelcinematicuniverse/images/1/13/Deadpool_%26_Wolverine_Trailer_%2864%29.png/revision/latest?cb=20240422182040)\n[](/wiki/File:Deadpool_%26_Wolverine_Trailer_\\(64\\).png)\n\n[Wolverine](/wiki/Wolverine "Wolverine") reflects on his past\n\nThere, they meet [Elektra](/wiki/Elektra/Forgotten_Elektra "Elektra/Forgotten\nElektra"), [Blade](/wiki/Blade/Universe_Defender_Blade "Blade/Universe\nDefender Blade"), and [Gambit](/wiki/Gambit "Gambit") variants, led by X-23\nwho had been banished to the Void by the TVA. Deadpool convinces the\nResistance to team up and take on Nova and avenge Storm\'s death, while leaving\nout the reason why he died. Wolverine scoffs that their plan is doomed to fail\nand refuses to join them, so X-23 goes to talk to him. Wolverine admits that\nin his universe, he abandoned his fellow X-Men, which resulted in them being\nkilled by the anti-mutant vigilantes, much to his horror and despair, so he\nkeeps wearing the X-Man suit since then. X-23 tells Wolverine that regardless\nof what he thinks about himself, he is still a good person at heart and always\nwill be. Meanwhile, Pyro follows Deadpool and Wolverine\'s tracks and strikes a\ndeal with Paradox, who decides that Nova is too dangerous and must be\neliminated.\n\n[](https://static.wikia.nocookie.net/marvelcinematicuniverse/images/2/23/Cassandra_Nova_-\n_Boys_Are_So_Silly.jpg/revision/latest?cb=20240422202104) [](https://static.wikia.nocookie.net/marvelcinematicuniverse/images/2/23/Cassandra_Nova_-\n_Boys_Are_So_Silly.jpg/revision/latest?cb=20240422202104)\n[](/wiki/File:Cassandra_Nova_-_Boys_Are_So_Silly.jpg)\n\n[Nova](/wiki/Cassandra_Nova "Cassandra Nova") meets [Deadpool](/wiki/Deadpool\n"Deadpool") and [Wolverine](/wiki/Wolverine "Wolverine") again\n\nWith Wolverine joining them, the Resistance storms Nova\'s base, planning to\nacquire [Juggernaut](/wiki/Juggernaut "Juggernaut")\'s\n[helmet](/wiki/Juggernaut%27s_Helmet "Juggernaut\'s Helmet") to neutralize\nNova\'s psychic ability.', '[Deadpool & Wolverine - Official Teaser - In\nTheaters July\n26](data:image/gif;base64,R0lGODlhAQABAIABAAAAAP///yH5BAEAAAEALAAAAAABAAEAQAICTAEAOw%3D%3D)](/wiki/File:Deadpool_%26_Wolverine_-\n_Official_Teaser_-_In_Theaters_July_26 "Deadpool & Wolverine - Official Teaser\n- In Theaters July 26 \\(19 KB\\)")\n\nDeadpool & Wolverine - Official Teaser - In Theaters July 26\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Official_Trailer_-_In_Theaters_July_26 "Deadpool & Wolverine - Official\nTrailer - In Theaters July 26 \\(33 KB\\)")\n\nDeadpool & Wolverine - Official Trailer - In Theaters July 26\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Final_Trailer_-_In_Theaters_July_26 "Deadpool & Wolverine - Final Trailer -\nIn Theaters July 26 \\(27 KB\\)")\n\nDeadpool & Wolverine - Final Trailer - In Theaters July 26\n\n### TV Spots[]\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Nice_-_In_Theaters_July_26 "Deadpool & Wolverine - Nice - In Theaters July 26\n\\(23 KB\\)")\n\nDeadpool & Wolverine - Nice - In Theaters July 26\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_LFG_-_In_Theaters_July_26 "Deadpool & Wolverine - LFG - In Theaters July 26\n\\(25 KB\\)")\n\nDeadpool & Wolverine - LFG - In Theaters July 26\n\n[!', '[Deadpool & Wolverine - Gear Up - In Theaters July\n26](data:image/gif;base64,R0lGODlhAQABAIABAAAAAP///yH5BAEAAAEALAAAAAABAAEAQAICTAEAOw%3D%3D)](/wiki/File:Deadpool_%26_Wolverine_-\n_Gear_Up_-_In_Theaters_July_26 "Deadpool & Wolverine - Gear Up - In Theaters\nJuly 26 \\(26 KB\\)")\n\nDeadpool & Wolverine - Gear Up - In Theaters July 26\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Pray_-_In_Theaters_July_26 "Deadpool & Wolverine - Pray - In Theaters July 26\n\\(21 KB\\)")\n\nDeadpool & Wolverine - Pray - In Theaters July 26\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Go_-_In_Theaters_July_26 "Deadpool & Wolverine - Go - In Theaters July 26\n\\(25 KB\\)")\n\nDeadpool & Wolverine - Go - In Theaters July 26\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Legendary_-_In_Theaters_July_26 "Deadpool & Wolverine - Legendary - In\nTheaters July 26 \\(23 KB\\)")\n\nDeadpool & Wolverine - Legendary - In Theaters July 26\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Smash_-_Now_Playing_In_Theaters "Deadpool & Wolverine - Smash - Now Playing\nIn Theaters \\(32 KB\\)")\n\nDeadpool & Wolverine - Smash - Now Playing In Theaters\n\n[!', '[Deadpool & Wolverine - Crying - Now Playing In\nTheaters](data:image/gif;base64,R0lGODlhAQABAIABAAAAAP///yH5BAEAAAEALAAAAAABAAEAQAICTAEAOw%3D%3D)](/wiki/File:Deadpool_%26_Wolverine_-\n_Crying_-_Now_Playing_In_Theaters "Deadpool & Wolverine - Crying - Now Playing\nIn Theaters \\(21 KB\\)")\n\nDeadpool & Wolverine - Crying - Now Playing In Theaters\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Smashing_-_Now_Playing_In_Theaters "Deadpool & Wolverine - Smashing - Now\nPlaying In Theaters \\(29 KB\\)")\n\nDeadpool & Wolverine - Smashing - Now Playing In Theaters\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Portal_-_Now_Playing_In_Theaters "Deadpool & Wolverine - Portal - Now Playing\nIn Theaters \\(27 KB\\)")\n\nDeadpool & Wolverine - Portal - Now Playing In Theaters\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Flame_On_-_Now_Playing_In_Theaters "Deadpool & Wolverine - Flame On - Now\nPlaying In Theaters \\(20 KB\\)")\n\nDeadpool & Wolverine - Flame On - Now Playing In Theaters\n\n[](/wiki/File:Deadpool_%26_Wolverine_-\n_Billion_with_a_B "Deadpool & Wolverine - Billion with a B \\(23 KB\\)")\n\nDeadpool & Wolverine - Billion with a B\n\n### Other[]\n\n[](/wiki/File:Deadpool_Update\n"Deadpool Update \\(28 KB\\)")\n\nDeadpool Update\n\n[!', '[Deadpool & Wolverine Trailer\n\\(73\\)](https://static.wikia.nocookie.net/marvelcinematicuniverse/images/5/5b/Deadpool_%26_Wolverine_Trailer_%2873%29.png/revision/latest/scale-\nto-width-\ndown/250?cb=20240422182041)](https://static.wikia.nocookie.net/marvelcinematicuniverse/images/5/5b/Deadpool_%26_Wolverine_Trailer_%2873%29.png/revision/latest?cb=20240422182041)\n[](/wiki/File:Deadpool_%26_Wolverine_Trailer_\\(73\\).png)\n\n[Deadpool](/wiki/Deadpool "Deadpool") and [Wolverine](/wiki/Wolverine\n"Wolverine") confront the [Deadpool Corps](/wiki/Deadpool_Corps "Deadpool\nCorps")\n\nDeadpool and Wolverine fight through the hordes of Deadpool variants, who keep\ncoming back due to their regeneration, however, they all cease fighting once\nthey see [Peter Wisdom](/wiki/Peter_Wisdom "Peter Wisdom"), who is deeply\nloved by all Deadpools. With them distracted, Deadpool and Wolverine make it\nto Paradox who tells them that Nova already took control over the Time Ripper\nand is preparing to unravel the entire Multiverse. He tells them that the only\nway to stop this is to short-circuit the chambers of matter and anti-matter\nthat power the device, however, it is all but guaranteed to kill them even\nwith their regeneration. Wolverine says he should be the one to do it,\nclaiming that he has nothing to fight for anymore, however, Deadpool locks him\nout of the bunker and prepares to sacrifice himself instead.\n\nDespite Wolverine\'s protests, Deadpool decides that he must do it himself to\nsave everyone he loves, however, Wolverine breaks in and connects the chamber\nwith him, resulting in the destruction of the Time Ripper and Nova\'s death.\n[B-15](/wiki/B-15 "B-15") and the TVA come to investigate the incident, with\nParadox claiming that Nova is solely responsible while mourning the deaths of\nDeadpool and Wolverine, however, they manage to survive by sharing the energy\nburst between each other. Paradox gets taken under custody and Deadpool asks\nB-15 to do good by their Resistance allies who were left in the Void and fix\nWolverine\'s home reality. B-15 promises to consider the former but notes that\nWolverine\'s unfortunate circumstances were what led him to save the\nMultiverse.\n\n[](https://static.wikia.nocookie.net/marvelcinematicuniverse/images/3/36/Wade_smiling_at_Vanessa.png/revision/latest?cb=20240723211721)\n[](https://static.wikia.nocookie.net/marvelcinematicuniverse/images/3/36/Wade_smiling_at_Vanessa.png/revision/latest?cb=20240723211721)\n[](/wiki/File:Wade_smiling_at_Vanessa.png)\n\n[Wade](/wiki/Wade_Wilson "Wade Wilson") and [Vanessa](/wiki/Vanessa_Carlysle\n"Vanessa Carlysle") rekinlde their relationship\n\nWilson and Logan then go on to have some shawarma and get reunited with\nDogpool, now that Nicepool got killed in the battle. Wilson takes both of them\nto his place with [Blind Al](/wiki/Blind_Al "Blind Al"), inviting their\nfriends and [X-23](/wiki/X-23 "X-23") to hang out together, and Wilson\nrekindles his relationship with Carlysle. Later on, Deadpool pulls up the\nfootage of Storm actually cursing out Nova, proving that he did not lie or set\nhim up to be killed.']
LLM using Chain
Set up a process where you use a prompt and OpenAI's language model to generate responses based on the content of the documents you provide.
[ ]
[ ]
The role of Deadpool in the events described involves teaming up with Wolverine and other characters to take on Nova and avenge Storm's death. Deadpool plays a crucial part in convincing the Resistance to join forces and confront Nova, ultimately leading to the destruction of the Time Ripper and Nova's demise. Despite Deadpool's initial intention to sacrifice himself, Wolverine also plays a significant role in the final act, resulting in both characters surviving the ordeal and being commended for their actions by B-15 and the TVA [1].
Conclusion
By the end of this example, we’ve created a Retrieval-Augmented Generation (RAG) system for webpages. We showed how it works with pages about Deadpool and Wolverine from Marvel. Here’s how we did it in 4 simple steps:
- Load data from the webpage.
- Create a custom embedding function with IBM Watsonx.
- Add the loaded documents into a table with their embeddings.
- Use ChatChain to generate answers based on the retrieved documents.