Notebook
Challenge: Analyzing Text about Data Science
In this example, let's do a simple exercise that covers all steps of a traditional data science process. You do not have to write any code, you can just click on the cells below to execute them and observe the result. As a challenge, you are encouraged to try this code out with different data.
Goal
In this lesson, we have been discussing different concepts related to Data Science. Let's try to discover more related concepts by doing some text mining. We will start with a text about Data Science, extract keywords from it, and then try to visualize the result.
As a text, I will use the page on Data Science from Wikipedia:
Step 1: Getting the Data
First step in every data science process is getting the data. We will use requests library to do that:
<!DOCTYPE html>
<html class="client-nojs" lang="en" dir="ltr">
<head>
<meta charset="UTF-8"/>
<title>Data science - Wikipedia</title>
<script>document.documentElement.className="client-js";RLCONF={"wgBreakFrames":!1,"wgSeparatorTransformTable":["",""],"wgDigitTransformTable":["",""],"wgDefaultDateFormat":"dmy","wgMonthNames":["","January","February","March","April","May","June","July","August","September","October","November","December"],"wgRequestId":"1a104647-90de-485a-b88a-1406e889a5d1","wgCSPNonce":!1,"wgCanonicalNamespace":"","wgCanonicalSpecialPageName":!1,"wgNamespaceNumber":0,"wgPageName":"Data_science","wgTitle":"Data science","wgCurRevisionId":1038046078,"wgRevisionId":1038046078,"wgArticleId":35458904,"wgIsArticle":!0,"wgIsRedirect":!1,"wgAction":"view","wgUserName":null,"wgUserGroups":["*"],"wgCategories":["CS1 maint: others","Articles with short description","Short description matches Wikidata","Use dmy dates from December 2012","Information science","Computer occupations"
Step 2: Transforming the Data
The next step is to convert the data into the form suitable for processing. In our case, we have downloaded HTML source code from the page, and we need to convert it into plain text.
There are many ways this can be done. We will use the simplest built-in HTMLParser object from Python. We need to subclass the HTMLParser class and define the code that will collect all text inside HTML tags, except <script> and <style> tags.
Data science - Wikipedia Data science From Wikipedia, the free encyclopedia Jump to navigation Jump to search Interdisciplinary field of study focused on deriving knowledge and insights from data Not to be confused with information science . The existence of Comet NEOWISE (here depicted as a series of red dots) was discovered by analyzing astronomical survey data acquired by a space telescope , the Wide-field Infrared Survey Explorer . Part of a series on Machine learning and data mining Problems Classification Clustering Regression Anomaly detection AutoML Association rules Reinforcement learning Structured prediction Feature engineering Feature learning Online learning Semi-supervised learning Unsupervised learning Learning to rank Grammar induction Supervised learning ( classification  • regression ) Decision trees Ensembles Bagging Boosting Random forest k -NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine
Step 3: Getting Insights
The most important step is to turn our data into some form from which we can draw insights. In our case, we want to extract keywords from the text, and see which keywords are more meaningful.
We will use Python library called RAKE for keyword extraction. First, let's install this library in case it is not present:
Requirement already satisfied: nlp_rake in c:\winapp\miniconda3\lib\site-packages (0.0.2) Requirement already satisfied: numpy>=1.14.4 in c:\winapp\miniconda3\lib\site-packages (from nlp_rake) (1.19.5) Requirement already satisfied: pyrsistent>=0.14.2 in c:\winapp\miniconda3\lib\site-packages (from nlp_rake) (0.17.3) Requirement already satisfied: regex>=2018.6.6 in c:\winapp\miniconda3\lib\site-packages (from nlp_rake) (2021.8.3) Requirement already satisfied: langdetect>=1.0.8 in c:\winapp\miniconda3\lib\site-packages (from nlp_rake) (1.0.9) Requirement already satisfied: six in c:\winapp\miniconda3\lib\site-packages (from langdetect>=1.0.8->nlp_rake) (1.16.0)
C:\winapp\Miniconda3\lib\site-packages\secretstorage\dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead from cryptography.utils import int_from_bytes C:\winapp\Miniconda3\lib\site-packages\secretstorage\util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead from cryptography.utils import int_from_bytes WARNING: Ignoring invalid distribution -umpy (c:\winapp\miniconda3\lib\site-packages) WARNING: Ignoring invalid distribution -umpy (c:\winapp\miniconda3\lib\site-packages) WARNING: Ignoring invalid distribution -umpy (c:\winapp\miniconda3\lib\site-packages) WARNING: Ignoring invalid distribution -umpy (c:\winapp\miniconda3\lib\site-packages) WARNING: Ignoring invalid distribution -umpy (c:\winapp\miniconda3\lib\site-packages)
The main functionality is available from Rake object, which we can customize using some parameters. In our case, we will set the minimum length of a keyword to 5 characters, minimum frequency of a keyword in the document to 3, and maximum number of words in a keyword - to 2. Feel free to play around with other values and observe the result.
[('machine learning', 4.0),
, ('big data', 4.0),
, ('data scientist', 4.0),
, ('21st century', 4.0),
, ('data science', 3.909090909090909),
, ('computer science', 3.909090909090909),
, ('information science', 3.797979797979798),
, ('data analysis', 3.666666666666667),
, ('application domains', 3.6),
, ('science', 1.9090909090909092),
, ('field', 1.25),
, ('statistics', 1.2272727272727273),
, ('classification', 1.2),
, ('techniques', 1.1666666666666667),
, ('datasets', 1.0),
, ('education', 1.0),
, ('archived', 1.0),
, ('original', 1.0),
, ('chikio', 1.0),
, ('forbes', 1.0)] We obtained a list terms together with associated degree of importance. As you can see, the most relevant disciplines, such as machine learning and big data, are present in the list at top positions.
Step 4: Visualizing the Result
People can interpret the data best in the visual form. Thus it often makes sense to visualize the data in order to draw some insights. We can use matplotlib library in Python to plot simple distribution of the keywords with their relevance:
There is, however, even better way to visualize word frequencies - using Word Cloud. We will need to install another library to plot the word cloud from our keyword list.
WordCloud object is responsible for taking in either original text, or pre-computed list of words with their frequencies, and returns and image, which can then be displayed using matplotlib:
<matplotlib.image.AxesImage at 0x224b8677400>
We can also pass in the original text to WordCloud - let's see if we are able to get similar result:
<matplotlib.image.AxesImage at 0x224b9e5fbb0>
<wordcloud.wordcloud.WordCloud at 0x224b99d76a0>
You can see that word cloud now looks more impressive, but it also contains a lot of noise (eg. unrelated words such as Retrieved on). Also, we get fewer keywords that consist of two words, such as data scientist, or computer science. This is because RAKE algorithm does much better job at selecting good keywords from text. This example illustrates the importance of data pre-processing and cleaning, because clear picture at the end will allow us to make better decisions.
In this exercise we have gone through a simple process of extracting some meaning from Wikipedia text, in the form of keywords and word cloud. This example is quite simple, but it demonstrates well all typical steps a data scientist will take when working with data, starting from data acquisition, up to visualization.
In our course we will discuss all those steps in detail.