Main

agentsllmsvector-databaselancedbgptopenaiAImultimodal-aimachine-learningembeddingsAdvance-RAG-with-HyDEfine-tuningexamplesdeep-learninggpt-4-visionllama-indexragmultimodallangchainlancedb-recipes

HyDE

Hypothetical Document Embeddings (HyDE), an innovative approach detailed in the paper titled Precise Zero-Shot Dense Retrieval without Relevance Labels. The core hypothesis of HyDE is simple yet profound: when conducting a document search, using hypothetical answers might yield superior results compared to using the question itself.

Installing Libraries

[10]
[2]
Enter Your OpenAI API Key:··········

Download the data (you can change the pdf as you like)

[3]
--2024-09-10 10:36:00--  https://ncert.nic.in/textbook/pdf/kehe103.pdf
Resolving ncert.nic.in (ncert.nic.in)... 164.100.166.133
Connecting to ncert.nic.in (ncert.nic.in)|164.100.166.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1705618 (1.6M) [application/pdf]
Saving to: ‘kehe103.pdf’

kehe103.pdf         100%[===================>]   1.63M   871KB/s    in 1.9s    

2024-09-10 10:36:04 (871 KB/s) - ‘kehe103.pdf’ saved [1705618/1705618]

Importing the neccessary libraries

[11]

Instantiate llm and embeddings

[32]
[33]

Generated Embeddings using HYDE

[62]
[-0.02528518708247117,
, 0.00831749535867708,
, 0.005965764392187786,
, -0.01159230572621031,
, 0.018957366397482973,
, 0.0017434122730279422,
, 0.0008382731126336937,
, -0.005045947413990321,
, -0.0017417814108506582,
, -0.003525965703971974,
, 0.03152167669156384,
, -0.03068666594356896,
, -0.011487929615541584,
, -0.03838442855260409,
, -0.039558662358764235,
, -0.003100305669842161,
, 0.04649969374166905,
, 0.005714608590087773,
, 0.011807581861918166,
, -0.049317857111627485,
, -0.011957623510534657,
, -0.00332862916862956,
, -0.0056787293893568866,
, 0.011403123385339,
, -0.020353400253721378,
, 0.013236234358686061,
, 0.008376207794043116,
, -0.02193209439910576,
, -0.0010135928921674534,
, -0.01484754528195486,
, -0.005655897086044274,
, -0.004511018100583344,
, -0.01492582759778704,
, -0.013816828278718265,
, 0.00905465391037364,
, -0.013340610469354787,
, -0.01672631993060461,
, -0.010757294046892891,
, 0.017991884812660282,
, -0.014469180600212241,
, 0.023236799042093713,
, 0.0013047052250645386,
, -0.004807838509308582,
, -0.003920639240318069,
, -0.0129557228045643,
, -0.0010559957744381112,
, -0.0007889389205904317,
, -0.02231045908084838,
, -0.014482227497630515,
, 0.036114240462148374,
, 0.04344668435553662,
, 0.011637970332835536,
, -0.015408568390198385,
, -0.0015379211939684733,
, 0.011448787991964226,
, -0.028259915084093958,
, -0.012153328834453834,
, 0.000999730389037562,
, -0.007365060671272662,
, 0.016334909282766256,
, 0.02648551468346786,
, 0.011566211931373763,
, 0.008800236151088426,
, 0.00687579643015218,
, 0.009589583223780618,
, -0.018592049544481164,
, -0.026133245659206862,
, 0.029668996769073176,
, 0.00451754154929248,
, 0.01899650802106033,
, 0.04148962552840962,
, -0.00010203202277525801,
, 0.020144648032383927,
, 0.002588208892578431,
, 0.020757858730300546,
, -0.00884590075771365,
, -0.04616046882085872,
, -0.012016335014578156,
, -0.02883398415843322,
, 0.008871995483872736,
, -0.014312615968547883,
, -0.030008219827238437,
, -0.013660263647053908,
, -0.0013870647545475864,
, -0.0025017721494445143,
, -0.0014539308023081413,
, -0.026707314733546123,
, -0.0004982342741338383,
, 0.00532319701092688,
, -0.0005895636620072663,
, -0.00274151179988822,
, 0.02666817497261384,
, 0.0047654358598685585,
, 0.00041526318588622365,
, -0.038854120584952086,
, 0.011409647299709405,
, -0.015995686224600993,
, 0.04850894015846915,
, 0.02343250529733543,
, -0.03731456992579014,
, 0.0006609147262744992,
, 0.042350730071241044,
, -0.0186050955105769,
, -0.01165101723025381,
, -0.01716992003076114,
, -0.014182145131720074,
, -0.006804037563029138,
, 0.010196270938649367,
, -0.0012859500189874775,
, 0.006764896405113048,
, -0.000991575961735824,
, 0.00590052943943515,
, -0.005130753178531637,
, -0.0065072171543038996,
, -0.024724163974227647,
, -0.013764639757722633,
, -0.010829053379677203,
, -0.023628211552577143,
, 0.000717995600971349,
, -0.015695602927368015,
, 0.020770906559041354,
, 0.011109564933798964,
, -0.0004509387180198401,
, -0.005913576336853423,
, 0.010802958653518118,
, 0.002069588666605565,
, -0.016308815487929708,
, -0.02776412739426434,
, -0.004732818150661606,
, -0.00011477328633039456,
, 0.016282721693093163,
, 0.023549928305422428,
, 0.01924440186597514,
, 0.008337066170465758,
, 0.02129278804370752,
, -0.0039858741930707065,
, -0.01577388617452273,
, 0.009870094777902379,
, -0.002937217589468667,
, -0.040498050148750374,
, 0.0009190017217317986,
, 0.013438463596975645,
, -0.005045947413990321,
, -0.009172077849783177,
, 0.010294124066270225,
, 0.014690980650290503,
, 0.05041380767063291,
, 0.020040270990392663,
, 0.0058842205848316734,
, 0.010574635620391987,
, 0.020379495911203,
, 0.0043772857722316,
, 0.006608331773448692,
, 0.006618117179343031,
, -0.001039686919834635,
, -0.0010078848163393003,
, 0.03963694560591895,
, -0.01199024028841907,
, 0.0006079110652285185,
, -0.013542839707644371,
, -0.007195449142190031,
, -0.02797287961560179,
, 0.0010299016303556127,
, 0.00634739056545434,
, 0.004957879692263803,
, 0.022401788294098834,
, 0.0034215893604726136,
, -0.006102758677724733,
, 0.0020304475086894754,
, -0.018761660142241256,
, 0.005117706281113363,
, 0.04185494238141142,
, -0.04357715519236443,
, 0.0327220061552056,
, -0.01936182487406214,
, -0.0004448229266473658,
, 0.03770597871098342,
, -0.000571216258786014,
, -0.03206965197106656,
, 0.009472159284371081,
, 0.013712451236727001,
, 0.04172447154458361,
, 0.005336244374006423,
, 0.025624410140636433,
, -0.015604273714117564,
, -0.0184876725024899,
, 0.03504438183533442,
, -0.006637687525470441,
, -0.0021038371215744843,
, -0.012583882037192086,
, 0.02439798688215812,
, 0.012799158172899942,
, -0.025193857869220716,
, -0.026537704135786027,
, -0.6371136645187176,
, -0.030347442885403697,
, 0.02302804682075626,
, -7.93525800499574e-05,
, 0.021188412864361333,
, 0.04041976690159566,
, 0.012857869676943441,
, 0.008141360846546581,
, 0.004031539265357201,
, 0.007534672666016561,
, -0.0035487982401152213,
, 0.02344555312607624,
, -0.01655670933284452,
, -0.016817649143855064,
, -0.013249281256104336,
, -0.030138690664066246,
, -0.008989418491959736,
, -0.029538525932245364,
, 0.015017156811037493,
, -0.02665512714387303,
, -0.03700144066246142,
, 0.005646111680149934,
, -0.023380317707662336,
, -0.014560509813462694,
, -0.006177779036371709,
, -0.00968091243703107,
, -0.000623404430535694,
, 0.009139460140576225,
, -0.005065517760117731,
, 0.019061743439474234,
, -0.047934869221484815,
, 0.02229741125210757,
, -0.014834498384536588,
, 0.0012508860747222571,
, 0.049865832391130196,
, 0.0164131906672759,
, -0.032539347728704704,
, 0.012212040338497335,
, 0.0032731791561099945,
, 0.054171358830577485,
, -0.030660572148732412,
, -0.0023435767720181903,
, 0.018579001715740354,
, -0.0017352579621415212,
, -0.00647786140228215,
, 0.03248716013903161,
, 0.01542161528761666,
, -0.014482227497630515,
, -0.008911136176127557,
, -0.0050035442988890285,
, 0.0023272679174147137,
, 0.011018235720548513,
, 0.009576536326362343,
, 0.005398217835235124,
, -0.02585925615681043,
, -0.027320527294107815,
, -0.009759194752863248,
, -0.013451510494393918,
, 0.009693959334449343,
, 0.013288421948359157,
, 0.009628723916035439,
, 0.019505343539630758,
, -0.009472159284371081,
, -0.007064978771023489,
, -0.020901377395869166,
, 0.03439202765119537,
, -0.00924383625124495,
, -0.00911336541441714,
, 0.034809535819160416,
, -0.029408055095417555,
, -0.01775703693384121,
, 0.012016335014578156,
, 0.011624923435417262,
, -0.015747792379686185,
, 0.009172077849783177,
, -0.016308815487929708,
, 0.022727965386168357,
, -0.024293609840166857,
, 0.001784184409536633,
, 0.02279319894193719,
, -0.017287343038848137,
, -0.033504831176172464,
, 0.0015558609107492337,
, 0.019544483300563045,
, 0.010150606332024142,
, -0.0038651892277985042,
, -0.01748304929408985,
, 0.014612698334458324,
, -0.01484754528195486,
, -0.010861671088884155,
, 0.034522498488023176,
, -0.005877696670461268,
, 0.01889213097906907,
, -0.018004930778756017,
, 0.0209144233619649,
, 0.016648038546094972,
, -0.0021543944311468804,
, -0.01689593239100978,
, 0.02578097477230079,
, -0.01011798862281719,
, 0.018004930778756017,
, 0.005502593945903851,
, 0.03060838455905932,
, -0.01886603718423252,
, -0.004025015350986796,
, 0.023471646920912786,
, -0.008624101638957927,
, 0.011246558753674642,
, 0.014038628328796527,
, 0.006197349848160388,
, -0.01604787381427409,
, -0.004462091536772915,
, -0.014103862815887895,
, 0.002242462152873398,
, 0.006030999810601691,
, -0.023497740715749334,
, -0.004002183047674183,
, 0.022362646670521475,
, -0.031782618365219464,
, 0.0007881234895017895,
, 0.03139120585473603,
, 0.004426212336042028,
, -0.0030954129668949912,
, -0.020040270990392663,
, -0.023315082289248428,
, 0.011096518036380692,
, 0.01688288456226897,
, -0.0172742970727524,
, -0.004243553443879855,
, 0.005646111680149934,
, -0.011370505676132047,
, -0.0012786110809820401,
, 0.025089480827229456,
, 0.0037281951750921917,
, 0.029616807316755006,
, -0.012342511175325144,
, 0.005248176652279904,
, -0.0164131906672759,
, 0.021984281988778855,
, -0.0350182861778528,
, -0.006001644058579941,
, 0.00151998159360303,
, -0.007991319197930091,
, -0.000837457623337393,
, -0.022179988244020572,
, -0.002821425094313,
, -0.03483562775135189,
, -3.2413771399261474e-05,
, -0.0522403956609321,
, -0.011070423310221606,
, -0.007528148751646157,
, -0.012316417380488597,
, -0.009674389453983202,
, 0.02473720994032338,
, 0.013855968970973086,
, -0.007952178505675271,
, -0.0048959062310351,
, 0.0034411599394306583,
, -0.0017825534309440316,
, -0.02812944424726615,
, 0.021436306709276143,
, 0.02083614197745526,
, -0.05605013441054977,
, -0.012114188142199013,
, 0.02565050393547298,
, -0.004794791611890309,
, 0.031104172248888937,
, -0.004644750428935088,
, 0.028651325731932315,
, -0.035409698688336225,
, 0.012505599721359907,
, -0.000837457623337393,
, 0.01882689556065516,
, 0.009439542506486666,
, 0.018344155699566354,
, -0.0066703052346773935,
, -0.003946733035154617,
, 0.006494170256885626,
, 0.02992993658008372,
, 0.018918224773905614,
, -0.000651944867884119,
, -0.004504494651874207,
, -0.008160930727012723,
, -0.022532259130926643,
, 0.03238278123439527,
, -0.009628723916035439,
, -0.020105506408806567,
, 0.03629689888864928,
, -0.002128300403479699,
, 0.004070680423273291,
, -0.018187591067901996,
, 0.024841586982314645,
, 0.006441981735889995,
, 0.003129661654694545,
, 0.0021299311492416657,
, 0.002938848335230634,
, 0.006680090640571733,
, -0.0001044783451450136,
, 0.0050557328198846604,
, 0.009817906256906749,
, 0.007926083779516186,
, -0.012140281937035561,
, 0.0495787987852831,
, -0.029016642584934124,
, -0.022153894449184024,
, -0.023745634560664145,
, 0.020301212664048284,
, -0.026576843896718313,
, -0.0009842370237303866,
, 0.002519711982640592,
, -0.018774707970982067,
, -0.029877748990410628,
, -0.007104119928939579,
, -0.016165296822361088,
, 0.017952743189082923,
, 0.003956518441048956,
, -0.00793913067693446,
, 0.029747278153582815,
, -0.013066622829603431,
, -0.015747792379686185,
, 0.01350369901538955,
, 0.017456955499253305,
, 0.015969592429764447,
, 0.004171795042418082,
, 0.0031785879856743395,
, 0.019153072652724687,
, 0.005714608590087773,
, -0.011148705626053785,
, 0.018905178807809876,
, 0.005903790930959083,
, 0.005743964807770792,
, 0.019009553987156067,
, -0.012075046518621655,
, 0.017678755549331567,
, 0.006673567191862596,
, -0.02167115272545014,
, 0.00533298241682122,
, -0.021892952775528402,
, 0.04890034894366243,
, -0.01269478113090868,
, 0.013986439807800895,
, 0.0028997071773145446,
, 0.005590661667630369,
, 0.009230789353826676,
, 0.035409698688336225,
, -0.015095439126869671,
, 0.028077256657593055,
, 0.01749609712283066,
, 0.009093795533950998,
, 0.006265846758098227,
, -0.00973962394107457,
, -0.019400966497639498,
, 0.0011652648208846411,
, 0.00022546935900992005,
, 0.015056297503292314,
, -0.021775529767441404,
, -0.012805681155947811,
, 0.0186050955105769,
, 0.019061743439474234,
, 0.02995603037492027,
, 0.03243497254935851,
, 0.006967125643402633,
, 0.014351756660802704,
, -0.011285700377252,
, 0.016830696972595875,
, -0.006993219903900448,
, 0.016491473914430614,
, -0.005160108930553386,
, 0.0001715483452651743,
, -0.01551294450086711,
, -0.002147870982437743,
, -0.03154777234904546,
, 0.026981304235942555,
, 0.009635247830405843,
, 0.026446374922535577,
, 0.013673310544472182,
, -0.024384940916062383,
, 0.026589891725459124,
, -0.004520803506477683,
, 0.0033384143416932653,
, -0.002754558930137128,
, -0.05054427850746072,
, 0.034444215240868464,
, 0.0046871535440363805,
, -0.005127491221346434,
, -0.012492551892619096,
, 0.01073772416642675,
, -0.013764639757722633,
, -0.019792377145477852,
, 0.04370762230390209,
, -0.002180488458814062,
, 0.008989418491959736,
, -0.008891566295661414,
, 0.01625662603561154,
, -0.005995120144209535,
, 0.0013846184030740015,
, 0.016439286324757517,
, -0.015747792379686185,
, 0.012857869676943441,
, -0.03045181992739496,
, -0.0004085358066453531,
, 0.016478426085689803,
, -0.001637405067351298,
, 0.008363159965302305,
, 0.022388740465358024,
, 0.0012084831924515996,
, -0.01613920302752454,
, -0.001307966949419107,
, 0.014782309863540956,
, -0.005372123574737309,
, 0.028155538042102697,
, -0.03175652270773784,
, -0.011442265008916357,
, -9.219577249117421e-05,
, 0.00820007235059008,
, -0.009524347805366712,
, 0.00014871601284873194,
, 0.0024218590878503686,
, 0.02301499899201545,
, 0.0008399039748109778,
, -0.006552881760929126,
, -0.020014177195556114,
, -0.0327220061552056,
, -0.010489829390189404,
, 0.14330881252404454,
, 0.011311794172088548,
, -0.002493617722142776,
, 0.03060838455905932,
, -0.013725499065467812,
, 0.004374024280707665,
, -0.007025837613107401,
, -0.007717331092517466,
, 0.009367783173702356,
, -0.016687180169672328,
, -0.006066879011332578,
, -0.0008978002805964697,
, -0.02899054879009758,
, 0.007749948801724419,
, 0.013829875176136538,
, -0.005414526689838601,
, 0.013014434308607799,
, -0.01251864661877818,
, -0.01844853087891254,
, -0.012779587361111263,
, 0.008761095458833605,
, -0.0033922336084508636,
, 0.013542839707644371,
, 0.016830696972595875,
, 0.007691237297680919,
, -0.0036760068869271943,
, 0.0217102943490275,
, 0.0046186561684372724,
, 0.027190056457280006,
, -0.014234333652715706,
, 0.014704027547708777,
, 0.0038619272706133015,
, 0.012609975832028632,
, 0.004331621165606373,
, -0.021788577596182215,
, 0.004002183047674183,
, -0.007436819538395704,
, -0.03580111119881966,
, 0.02431970549764848,
, -0.017078590817510685,
, 0.008956800782752784,
, 0.009256883148663225,
, 0.013673310544472182,
, -0.02058824813254045,
, 0.04206369646539395,
, -0.02431970549764848,
, -0.003020392608248015,
, 0.008904613193079688,
, -0.02146240050411269,
, 0.007926083779516186,
, 0.020523012714126546,
, -0.0026550750567543033,
, -0.01083557636272507,
, -0.015734744550945375,
, 0.0003806069571653244,
, 0.010913858678557249,
, -0.016595850956421875,
, -0.0011269391522648524,
, 0.0002564561041761859,
, 0.019348778907966404,
, -0.0014131587822147679,
, -0.004374024280707665,
, -0.03953256670128261,
, 0.0025001414036825475,
, -0.01787446180457328,
, 0.0024838325490790713,
, -0.02232350504694412,
, -0.011500976512959858,
, -0.0039956595989650455,
, -0.004083727320691564,
, 0.011931528784375572,
, -0.012675211250442537,
, -0.028286008878930506,
, 0.035566265182645655,
, 0.018435484912816807,
, 0.012655640438653859,
, 0.03047791372223151,
, -0.011566211931373763,
, 0.0028948147071980092,
, -0.00943301859211626,
, -0.0025996250442347378,
, -0.010052753204403284,
, -0.005251438609465107,
, -0.0411764962650809,
, 0.02167115272545014,
, -0.004778482757286833,
, -0.005743964807770792,
, -0.007365060671272662,
, -0.02763365655743653,
, 0.024228376284398025,
, 0.019048695610733423,
, 0.003959780398234159,
, 0.0031541247037691253,
, -0.010913858678557249,
, -0.016713273964508876,
, 0.004064156508902885,
, 0.03214793521822127,
, 0.0059168378283773575,
, 6.141287782111158e-05,
, -0.02339336367375807,
, -0.004915477042823779,
, -0.026446374922535577,
, -0.005306888156323404,
, -0.0016439285160604346,
, -0.0004285141011477188,
, -0.0033400453202858667,
, -0.0006417518919646049,
, 0.020679577345790904,
, -0.003033439505666289,
, -0.010111465639769321,
, 0.01784836614709166,
, -0.020666529517050093,
, 0.00846753700729357,
, 0.024254470079234574,
, -0.00960263012119889,
, 0.025585268517059074,
, 0.003129661654694545,
, 0.012303369551747786,
, 0.001718949107538045,
, -0.006213658702763864,
, 0.0024300135151521067,
, -0.02527213925373036,
, 0.04811752764798572,
, -0.0003852957586845897,
, 0.007769519613513098,
, 0.0028687204467001933,
, -0.010183224041231094,
, -0.010046230221355416,
, -0.01576083834578192,
, 0.003072580663582378,
, 0.006415887941053447,
, 0.029668996769073176,
, -0.010339788672895452,
, -0.026955208578460933,
, -0.02032730645888483,
, -0.003943471543630683,
, -0.00477522080010163,
, 0.018187591067901996,
, -0.007195449142190031,
, 0.011788011050129488,
, -0.016648038546094972,
, -0.02995603037492027,
, -0.011448787991964226,
, -0.037392849447654704,
, -0.0077042841950991925,
, -0.02417618683207986,
, -0.01994894177714221,
, 0.006008167507289078,
, -0.015656463166435732,
, -0.00666704374315346,
, -0.021018800403956164,
, -0.013340610469354787,
, -0.01788750777066902,
, 0.034522498488023176,
, -0.0011774964618372482,
, 0.000987498748084955,
, 0.027659750352273076,
, 0.006474599445096947,
, 0.02812944424726615,
, 0.011337887966925095,
, 0.010711629440267666,
, -0.005346029314239493,
, -0.01000056561473019,
, -0.00577005860260734,
, 0.010007088597778057,
, -0.0018575740224216419,
, -0.02441103471089893,
, -0.0032160983978284617,
, 0.009191647730249318,
, 0.0043707623235224626,
, 0.002395764827352553,
, 0.005104658918033821,
, -0.010215841750438047,
, -0.021057942027533524,
, -0.02441103471089893,
, 0.02193209439910576,
, -0.011031282617966785,
, -0.00509487397780075,
, -0.010789911756099845,
, -0.03971522512778352,
, -0.009217742456408403,
, 0.02454150554772674,
, -0.016465380119594065,
, -0.011292223360299869,
, -0.03376576912453794,
, 0.01115522954042419,
, 0.03992397921176604,
, 0.007834754566265735,
, 0.018083214025910732,
, -0.004951356243554666,
, 0.02279319894193719,
, -0.022010375783615404,
, -0.00410003617529504,
, -0.012035905826366834,
, -0.026903020988787836,
, -0.019179166447561236,
, -0.020875283601032617,
, 0.00746291379889352,
, 0.01640014470118016,
, 0.012016335014578156,
, 0.013921204389386991,
, 0.008630624622005794,
, 0.0011766809725409476,
, 0.0003883536689227415,
, 0.007919560796468317,
, 0.007280254906731347,
, -0.023915247021069314,
, -0.015486850706030564,
, 0.016543661504103708,
, -0.015656463166435732,
, -0.0002012100076364394,
, -0.016608896922517612,
, -0.01886603718423252,
, -0.017783132591322828,
, 0.01200981110020775,
, 0.0021005753972199157,
, 0.0010013612512148463,
, 0.007547719563434836,
, 0.020014177195556114,
, -0.012609975832028632,
, -0.005212297451549017,
, -0.01104432951538506,
, 0.00903508309858496,
, -0.007580337272641788,
, 0.011063900327173738,
, 0.022245223662434477,
, 8.98004109264801e-05,
, -0.000976897969309632,
, -0.003858665546258733,
, 0.004908953128453374,
, -0.007645572225394424,
, 0.005179679742342065,
, 0.01788750777066902,
, -0.0002900930585881971,
, -0.003848880373195028,
, 0.009674389453983202,
, 0.022010375783615404,
, -0.020509964885385736,
, -0.01227075277386337,
, 0.007926083779516186,
, 0.0141690982343018,
, -0.01379073355255918,
, -0.005440620484675148,
, 0.005202512045654678,
, 0.0016039719852633618,
, 0.03266981856553251,
, -0.01602178001943754,
, -0.002754558930137128,
, -0.010391977193891082,
, -0.019922847982305664,
, -0.026981304235942555,
, 0.009817906256906749,
, -0.00022567323133399522,
, 0.0029845131746864943,
, 0.0057635351538982025,
, -0.03376576912453794,
, -0.012818728984688622,
, 0.006572452572717805,
, 0.004977450504052482,
, 0.01182062875933644,
, -0.0041326538845019925,
, 0.009570012411991938,
, -0.010907335695509382,
, 0.012929628078405215,
, -0.022780152975841454,
, 0.006021214404707351,
, -0.018044072402333376,
, 0.0030089764565917085,
, 0.0013984809062038929,
, 0.02623762083855305,
, -0.02168420055419095,
, -0.005316673562217744,
, 0.014338709763384432,
, -0.0033987570571600008,
, -0.0030872587724238873,
, -0.017822272352255114,
, 0.005551520509714279,
, -0.036192523709303086,
, 0.004726294701952469,
, 0.007945654591304866,
, -0.001246808861071388,
, -0.007906513899050045,
, -0.01886603718423252,
, 0.0034998716763047924,
, -0.00761295498184874,
, -0.013921204389386991,
, -0.013634168920894824,
, -0.024815493187478097,
, 0.013855968970973086,
, 0.003773859781717418,
, -0.01957057709539959,
, -0.03269591049772398,
, 0.0020467563632929515,
, 0.0035944628467404474,
, 0.013673310544472182,
, -0.02538956226181736,
, 0.022727965386168357,
, 0.008630624622005794,
, -0.01547380380861229,
, 0.030791042985560225,
, -0.027659750352273076,
, 0.012414269576786917,
, -0.028807890363596673,
, -0.00941344778032758,
, -0.016752415588086232,
, -0.0011228619386139834,
, 0.01945315408731259,
, -0.020783954387782164,
, -0.01851376629732645,
, -0.03564454470451023,
, 0.021136223412043163,
, 0.004390333135311141,
, 0.010757294046892891,
, -0.01236860497016169,
, -0.001345477245157912,
, 0.005505855903089054,
, 0.014469180600212241,
, -0.015186768340120123,
, -0.028677419526768864,
, 0.024932916195565098,
, -0.011226987941885964,
, -0.01324275734173393,
, 0.009870094777902379,
, -0.033844052371692655,
, -0.03376576912453794,
, -0.001402558119854762,
, 0.011442265008916357,
, 0.00215928713409405,
, -0.06445243693075198,
, -0.020888329567128355,
, -0.011644494247205942,
, 0.021618965135777046,
, -0.028155538042102697,
, -0.007469437247602657,
, -0.00394999499233982,
, 0.013855968970973086,
, -0.009282977874822308,
, 0.03491391099850661,
, 0.022414836122839645,
, -0.012499075806989501,
, 0.050674749344288526,
, 0.027607562762599982,
, 0.00911336541441714,
, 0.03358311069803704,
, -0.013542839707644371,
, -0.024606739103495572,
, -0.014090815918469621,
, -0.03713191149928923,
, -0.020353400253721378,
, -0.019087837234310782,
, 0.00017898924853647862,
, 0.012212040338497335,
, 0.01388206369713217,
, 0.0013201985903717143,
, -0.020183789655961282,
, 0.010463735595352856,
, -0.03293076023918813,
, -0.02070567114062745,
, -0.006203873296869524,
, 0.014912780700368767,
, -0.0004309604235174744,
, -0.016608896922517612,
, 0.019087837234310782,
, 0.005189464682575137,
, -0.0006670305467508028,
, 0.02870351332160541,
, -0.012681734233490406,
, -0.022610540515436286,
, -0.0013740176242986785,
, -0.03415718163502137,
, 0.03405280645567518,
, -0.007058455322314353,
, -0.043107459434726283,
, 0.0024642619701210266,
, 0.018344155699566354,
, 0.0009402031628671275,
, 0.014103862815887895,
, 0.0012337618472377974,
, -0.0005630618314842759,
, 1.2575625477277462e-05,
, 0.009061177824744046,
, 0.040967742181098374,
, 0.020405589706039545,
, 0.011005187891807702,
, -0.010561588722973714,
, -0.029877748990410628,
, -0.008911136176127557,
, 0.008656719348164879,
, -0.021605917307036236,
, -0.010802958653518118,
, 0.006406102535159108,
, 0.01827892028115245,
, 0.0007290040079795052,
, -0.015891309182609732,
, -0.034000618866002086,
, 0.011435741094545952,
, 0.023615163723836333,
, -0.007893466070309234,
, 0.04323793027155409,
, 0.008324019273047486,
, -0.0114618348893825,
, 0.026211527043716504,
, -0.0063376056252212685,
, -0.018004930778756017,
, -0.0011538486692283345,
, -0.01936182487406214,
, 0.0036727451625726258,
, 0.0010519185607872422,
, 0.020627389756117807,
, -0.01711773244108804,
, 0.020053318819133473,
, -0.0020108769297314307,
, 0.0040119684535685225,
, -0.02416314086598412,
, -0.001016854616522022,
, 0.0030383322086134585,
, -0.018931272602646425,
, -0.01581302779810009,
, 0.005111182366742958,
, 0.009485207113111892,
, -0.008630624622005794,
, -0.023628211552577143,
, 0.006693137537990007,
, -0.006070140968517781,
, 0.0020108769297314307,
, -0.0028246868186675685,
, 0.006262584800913024,
, -0.01475621606870441,
, -0.0019277019109520824,
, -0.0014115279200374836,
, -0.01663499071735416,
, -0.006650734888549984,
, -0.0038619272706133015,
, 0.009876618692272784,
, -0.012048952723785108,
, 0.010450688697934583,
, 0.20687404925448344,
, 0.006109282126433869,
, 0.009335165464495402,
, 0.01559122681669929,
, -0.026094104035629503,
, 0.007012790250027858,
, 0.03167824318587327,
, 0.012596928934610358,
, 0.0032568703015065184,
, -0.0003732679900560577,
, 0.0002277118236075151,
, 0.0021919046104703684,
, -0.0061223290238521435,
, 0.0041228684786076535,
, 0.024815493187478097,
, 0.009961423991152832,
, -0.023497740715749334,
, -0.039454283454127895,
, 0.0038619272706133015,
, -0.017678755549331567,
, 0.008160930727012723,
, 0.003724933217906989,
, 0.016700226135768065,
, 0.011729299546085988,
, 0.021423258880535333,
, 0.0182528264863159,
, 0.049839736733648575,
, 0.015617321542858375,
, 0.00515358548184425,
, 0.01170972873429731,
, -0.021057942027533524,
, 0.006086449823121257,
, 0.00197010479322274,
, -0.0026730146571197467,
, 0.002320744468705577,
, -0.02464588072707293,
, -0.007802136857058782,
, 0.022910623812669263,
, -0.00028214250361053417,
, 0.004445783147830707,
, -0.004067418466088088,
, 0.026746456357123482,
, -0.022949763573601546,
, -0.022010375783615404,
, 0.004135915376025927,
, 0.040941650248906905,
, ...]

Multiple generations

We can also generate multiple documents and then combine the embeddings for those. By default, we combine those by taking the average. We can do this by changing the LLM we use to generate documents to return multiple things.

[35]
[60]
[45]

The HypotheticalDocumentEmbedder does not actually create full hypothetical documents. It only generates an embedding vector representing a hypothetical document. This is used to generate dummy embeddings that can be inserted into a vectorstore index. This allows you to reserve space for documents that don't exist yet, so that you can incrementally add new real documents later. But the embedder itself does not generate any actual text content for these hypothetical documents. It simply generates an embedding vector using a strategy like sampling from a normal distribution.

Making Your Own Prompts

You can also make and use your own prompts when creating documents with LLMChain. This is helpful if you know what topic you're asking about. With a custom prompt, you can get text that fits your topic better.

Let's try this out. We'll make a prompt about a state of the union address, which we'll use in the next example.

[56]
[63]

Loading data from pdf

[39]

Initialize the Vectorstore

[64]
[66]
[67]
[Document(metadata={'page': 11, 'source': '/content/kehe103.pdf'}, page_content='Calcium, riboflavin, vitamin A: These essential nutrients are low unless milk or a milkshake is ordered.Folic acid, fibre:  There are few fast food sources of these key factors.\nFat: The percentage of energy from fat is high in many meal combinations.Sodium: The sodium content of fast food meals is high, which is not desirable.Energy: Common meal combinations contain excessive energy when compared with the amounts of other nutrients provided.'),
, Document(metadata={'page': 11, 'source': '/content/kehe103.pdf'}, page_content='39population. \tIf\tthis\tis\tnot\tmain tained,\t80\tper\tcent\tof\tthem\twill\tstay\t \noverweight \tas\tadul ts.\tThis\tcan\tput\tthem\tat\trisk\tfor\tmany\tmedi cal\tprobl ems,\t\nincluding \tdia betes,\thigh\tblo od\tpre ssure,\thigh\tcho lesterol\tand\tsle ep\tapn ea\t\n(a\tsleep\tdisorder) .\nTable 2: Nutritional Limitations of Fast Foods\nThe following factors appear to be the major nutritional limitations of fast-food meals.'),
, Document(metadata={'page': 11, 'source': '/content/kehe103.pdf'}, page_content='the nutritional needs of teenagers. Both adolescents and health professionals should be aware that fast foods are acceptable nutritionally when they are consumed judiciously and as a part of a well-balanced diet. But when they become the mainstay of the diet there is cause for concern. A nutrient imbalance may not appear to be a problem until a number of years have gone by, unless some specific problem such as a chronic disease exists. However, evidence is accumulating to show that food intake'),
, Document(metadata={'page': 11, 'source': '/content/kehe103.pdf'}, page_content='Although fast foods can contribute nutrients to the diet, they cannot completely meet')]
[68]
<ipython-input-68-106fbd6a4823>:1: LangChainDeprecationWarning: The method `Chain.run` was deprecated in langchain 0.1.0 and will be removed in 1.0. Use invoke instead.
  llm_chain.run(
"The major nutritional limitations of fast-food meals can vary depending on the specific meal and location, but some common factors include high levels of saturated fat, sodium, and added sugars. These ingredients can contribute to an increased risk of heart disease, obesity, and other health issues. Additionally, fast-food meals often lack important nutrients such as fiber, vitamins, and minerals. Limited options for fresh fruits and vegetables and the use of processed meats are also common limitations. However, it's important to note that not all fast-food meals are nutritionally limited, and some options may offer healthier choices."
[69]
'\nPulses and legumes are excellent sources of plant-based protein and dietary fiber. They also contain a variety of essential vitamins and minerals, including iron, potassium, magnesium, and folate. Additionally, they are low in fat and cholesterol, making them a healthy choice for a balanced diet.'
[70]
'It is difficult to determine which specific groups serve come from without more context or information. However, it is likely that serve come from a variety of food groups, including grains, vegetables, fruits, dairy, and protein sources. The amount of serve come from each group may vary depending on the specific recipe or dish being served. It is best to consult a nutritionist or refer to a dietary guideline for more specific information on serving sizes and food groups. '

Thanks