Notebooks
M
Microsoft
MSConceptGraph

MSConceptGraph

artificial-intelligencernnganmicrosoft-for-beginnerslessonsAImicrosoft-AI-For-Beginnersmachine-learningdeep-learningcomputer-visioncnn2-SymbolicNLP

Concept Graph using ConceptNet

Note: The original Microsoft Concept Graph API is no longer available. This notebook has been updated to use ConceptNet as a replacement, which is a freely available open knowledge graph with similar is-a relations between concepts.

ConceptNet is a large semantic network of concepts with relationships such as IsA, PartOf, UsedFor, and more. It is available as:

  • A downloadable data file
  • A REST API (no API key required)

ConceptNet statistics:

  • Over 8 million nodes
  • 21+ million edges across 83 languages

Using ConceptNet Web Service

ConceptNet provides a REST API to explore is-a (IsA) relationships between concepts. No API key is required. Here is the sample URL to call: https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10

[ ]

Let's try to categorize the news titles using parent concepts. To get news titles, we will use NewsApi.org service. You need to obtain your own API key in order to use the service - go to the web site and register for free developer plan.

[20]
[21]
['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',
, 'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',
, 'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',
, 'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',
, "Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN",
, '2 teachers killed at Swedish high school, student arrested - ABC News',
, 'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',
, 'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',
, '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',
, 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',
, "US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business",
, "'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO",
, "Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail",
, 'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',
, 'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',
, 'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',
, 'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',
, 'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',
, "UConn vs UCF - NCAA women's tournament second-round highlights - March Madness",
, 'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',
, "Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror",
, 'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',
, 'Daniel Morgan murder: damning report condemns Met police - The Guardian',
, 'What to expect from Rishi Sunak’s Spring Statement - BBC.com',
, 'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',
, "Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror",
, 'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',
, "Vladimir Putin threatens civilians to break Mariupol's spirit - The Times",
, 'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',
, 'St Helens dog attack: Girl aged 17 months killed at home - BBC',
, "PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME",
, '‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',
, 'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',
, 'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',
, 'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',
, 'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',
, 'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',
, 'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',
, "UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider",
, 'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']

First of all, we want to be able to extract nouns from news titles. We will use TextBlob library to do this, which simplifies a lot of typical NLP tasks like this.

[15]
Requirement already satisfied: textblob in c:\winapp\miniconda3\lib\site-packages (0.17.1)
Requirement already satisfied: nltk>=3.1 in c:\winapp\miniconda3\lib\site-packages (from textblob) (3.5)
Requirement already satisfied: joblib in c:\winapp\miniconda3\lib\site-packages (from nltk>=3.1->textblob) (1.0.1)
Requirement already satisfied: regex in c:\winapp\miniconda3\lib\site-packages (from nltk>=3.1->textblob) (2021.11.10)
Requirement already satisfied: tqdm in c:\winapp\miniconda3\lib\site-packages (from nltk>=3.1->textblob) (4.61.2)
Requirement already satisfied: click in c:\winapp\miniconda3\lib\site-packages (from nltk>=3.1->textblob) (8.0.3)
Requirement already satisfied: colorama in c:\winapp\miniconda3\lib\site-packages (from click->nltk>=3.1->textblob) (0.4.4)
Finished.
[nltk_data] Downloading package brown to
[nltk_data]     C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data]   Package brown is already up-to-date!
[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package wordnet to
[nltk_data]     C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package conll2000 to
[nltk_data]     C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data]   Package conll2000 is already up-to-date!
[nltk_data] Downloading package movie_reviews to
[nltk_data]     C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data]   Package movie_reviews is already up-to-date!
[22]
{'covid-19 live updates': 1,
, 'vaccines': 1,
, 'boosters': 1,
, 'york': 4,
, 'ukrainians flee mariupol': 1,
, 'forces push': 1,
, 'port city': 1,
, 'wall street journal': 3,
, 'bond yields': 1,
, 'futures rise': 1,
, 'powell says fed': 1,
, 'ready': 1,
, 'be': 1,
, 'aggressive': 1,
, 'putin': 3,
, 'alexei navalny': 2,
, 'russian': 2,
, 'supreme court nominee': 1,
, 'ketanji brown jackson': 1,
, "confirmation hearing 's": 1,
, 'cnn': 1,
, 'swedish': 1,
, 'high school': 1,
, 'abc': 1,
, 'clues': 1,
, 'covid-19': 1,
, '’ s': 2,
, 'moves': 1,
, 'sewers': 1,
, 'roll dice': 1,
, 'jackson': 1,
, 'decisions |': 1,
, 'thehill': 1,
, 'clear': 2,
, 'chemical weapons': 2,
, 'ukraine': 3,
, 'claims president': 2,
, 'biden': 2,
, 'nasa': 2,
, 'solar system': 2,
, 'daily mail': 3,
, 'us stocks': 1,
, 'fed chair powell': 1,
, "'s remarks": 1,
, 'fox': 1,
, "'we 've": 1,
, 'tests': 1,
, 'covid': 1,
, 'politico': 1,
, 'duchess': 1,
, 'cambridge': 1,
, 'swaps khaki jungle gear': 1,
, 'vampire': 1,
, 'wife': 1,
, 'belize': 1,
, 'china': 2,
, 'flight recorders': 1,
, 'plane crash': 1,
, 'reuters': 2,
, 'russian oligarch': 1,
, 'abramovich': 1,
, 'live': 1,
, 'russia': 2,
, 'stops talks': 1,
, 'japan': 1,
, 'español': 1,
, 'powers remain': 1,
, 'threats lurk': 1,
, 'set': 1,
, 'webb': 1,
, 'telescope begins multi-instrument alignment': 1,
, 'scitechdaily': 1,
, 'uconn': 1,
, 'ucf': 1,
, 'ncaa': 1,
, "women 's tournament second-round highlights": 1,
, 'march madness': 1,
, 'bucking republican trend': 1,
, 'indiana': 1,
, 'vetoes transgender': 1,
, 'bill': 1,
, 'maggie fox': 1,
, 'coronation': 1,
, 'shameless': 1,
, "'sudden accident": 1,
, 'mirror online': 1,
, 'mirror': 2,
, 'plane crash –': 1,
, 'search': 1,
, 'moment flight': 1,
, 'daniel morgan': 1,
, 'report condemns': 1,
, 'met': 1,
, 'guardian': 6,
, 'rishi sunak': 1,
, '’ s spring': 1,
, 'statement': 1,
, 'bbc.com': 1,
, 'uk': 3,
, 'ireland': 1,
, 'euro': 1,
, 'vladimir putin': 2,
, "'s 'lover": 1,
, 'brass eye': 1,
, '’ s outtakes': 1,
, 'brutal tv comedy': 1,
, 'threatens civilians': 1,
, 'mariupol': 1,
, "'s spirit": 1,
, 'shell u-turn': 1,
, 'cambo': 1,
, 'green targets': 1,
, 'st helens': 1,
, 'dog attack': 1,
, 'girl': 1,
, 'bbc': 1,
, 'playstation': 1,
, "'assassin 's": 1,
, 'creed': 1,
, 'jade raymond': 1,
, 'haven studios': 1,
, 'nme': 1,
, 'nintendo switch': 1,
, 'folders •': 1,
, 'eurogamer.net': 2,
, 'fa': 1,
, 'solution ”': 1,
, 'liverpool': 1,
, 'fan group blasts “ shambolic ”': 1,
, 'wembley': 1,
, 'anfield': 1,
, 'manchester': 1,
, 'live erik': 1,
, 'hag': 1,
, 'utd': 1,
, 'manager updates': 1,
, 'manchester evening': 1,
, 'inflation': 1,
, 'government borrowing': 1,
, 'february': 1,
, 'crude oil': 1,
, '– business': 1,
, 'kremlin': 1,
, 'large-scale fraud': 1,
, 'sky': 1,
, 'natural gas': 1,
, 'gazprom': 1,
, 'retail unit': 1,
, 'insider': 1,
, 'zaghari-ratcliffe': 1,
, 'hunt': 1,
, 'iran': 1,
, 'debt payment': 1}

We can see that nouns do not give us large thematic groups. Let's substitute nouns by more general terms obtained from the concept graph. This will take some time, because we are doing REST call for each noun phrase.

[23]
[24]
{'city': 9,
, 'brand': 4,
, 'place': 9,
, 'town': 4,
, 'factor': 4,
, 'film': 4,
, 'nation': 11,
, 'state': 5,
, 'person': 4,
, 'organization': 5,
, 'publication': 10,
, 'market': 5,
, 'economy': 4,
, 'company': 6,
, 'newspaper': 6,
, 'relationship': 6}
[27]

ECONOMY:
China searches for victims, flight recorders after first plane crash in 12 years - Reuters
Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español
China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent
UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider

NATION:
‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent
Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail
China searches for victims, flight recorders after first plane crash in 12 years - Reuters
Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español
Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español
China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent
UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian
Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror
‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent
UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider
Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian

PERSON:
‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent
Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail
Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters
‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent