Microsoft CBoW TF

CBoW TF

15-LanguageModelingartificial-intelligencernnganmicrosoft-for-beginnerslessonsAImicrosoft-AI-For-Beginnersmachine-learning5-NLPdeep-learningcomputer-visioncnnNLP

alph-notebooks/microsoft-AI-For-Beginners / CBoW-TF.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

Training CBoW Model

This notebooks is a part of AI for Beginners Curriculum

In this example, we will look at training CBoW language model to get our own Word2Vec embedding space. We will use AG News dataset as the source of text.

[30]

We will start by loading the dateset:

[1]

CBoW Model

CBoW learns to predict a word based on the $2N$ neighboring words. For example, when $N=1$ , we will get the following pairs from the sentence I like to train networks: (like,I), (I, like), (to, like), (like,to), (train,to), (to, train), (networks, train), (train,networks). Here, first word is the neighboring word used as an input, and second word is the one we are predicting.

To build a network to predict next word, we will need to supply neighboring word as input, and get word number as output. The architecture of CBoW network is the following:

Input word is passed through the embedding layer. This very embedding layer would be our Word2Vec embedding, thus we will define it separately as embedder variable. We will use embedding size = 30 in this example, even though you might want to experiment with higher dimensions (real word2vec has 300)
Embedding vector would then be passed to a dense layer that will predict output word. Thus it has the vocab_size neurons.

Embedding layer in Keras automatically knows how to convert numeric input into one-hot encoding, so that we do not have to one-hot-encode input word separately. We specify input_length=1 to indicate that we want just one word in the input sequence - normally embedding layer is designed to work with longer sequences.

For the output, if we use sparse_categorical_crossentropy as loss function, we would also have to provide just word numbers as expected results, without one-hot encoding.

We will set vocab_size to 5000 to limit computations a bit. We will also define a vectorizer which we will use later.

[68]

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 embedding_1 (Embedding)     (None, 1, 30)             150000    
                                                                 
 dense_1 (Dense)             (None, 1, 5000)           155000    
                                                                 
=================================================================
Total params: 305,000
Trainable params: 305,000
Non-trainable params: 0
_________________________________________________________________

Let's initialize the vectorizer and get out the vocabulary:

[69]

Preparing Training Data

Now let's program the main function that will compute CBoW word pairs from text. This function will allow us to specify window size, and will return a set of pairs - input and output word. Note that this function can be used on words, as well as on vectors/tensors - which will allow us to encode the text, before passing it to to_cbow function.

[70]

[['like', 'I'], ['to', 'I'], ['I', 'like'], ['to', 'like'], ['train', 'like'], ['I', 'to'], ['like', 'to'], ['train', 'to'], ['networks', 'to'], ['like', 'train'], ['to', 'train'], ['networks', 'train'], ['to', 'networks'], ['train', 'networks']]
[[<tf.Tensor: shape=(), dtype=int64, numpy=376>, <tf.Tensor: shape=(), dtype=int64, numpy=771>], [<tf.Tensor: shape=(), dtype=int64, numpy=3>, <tf.Tensor: shape=(), dtype=int64, numpy=771>], [<tf.Tensor: shape=(), dtype=int64, numpy=771>, <tf.Tensor: shape=(), dtype=int64, numpy=376>], [<tf.Tensor: shape=(), dtype=int64, numpy=3>, <tf.Tensor: shape=(), dtype=int64, numpy=376>], [<tf.Tensor: shape=(), dtype=int64, numpy=1>, <tf.Tensor: shape=(), dtype=int64, numpy=376>], [<tf.Tensor: shape=(), dtype=int64, numpy=771>, <tf.Tensor: shape=(), dtype=int64, numpy=3>], [<tf.Tensor: shape=(), dtype=int64, numpy=376>, <tf.Tensor: shape=(), dtype=int64, numpy=3>], [<tf.Tensor: shape=(), dtype=int64, numpy=1>, <tf.Tensor: shape=(), dtype=int64, numpy=3>], [<tf.Tensor: shape=(), dtype=int64, numpy=1045>, <tf.Tensor: shape=(), dtype=int64, numpy=3>], [<tf.Tensor: shape=(), dtype=int64, numpy=376>, <tf.Tensor: shape=(), dtype=int64, numpy=1>], [<tf.Tensor: shape=(), dtype=int64, numpy=3>, <tf.Tensor: shape=(), dtype=int64, numpy=1>], [<tf.Tensor: shape=(), dtype=int64, numpy=1045>, <tf.Tensor: shape=(), dtype=int64, numpy=1>], [<tf.Tensor: shape=(), dtype=int64, numpy=3>, <tf.Tensor: shape=(), dtype=int64, numpy=1045>], [<tf.Tensor: shape=(), dtype=int64, numpy=1>, <tf.Tensor: shape=(), dtype=int64, numpy=1045>]]

Let's prepare the training dataset. We will go through all news, call to_cbow to get the list of word pairs, and add those pairs to X and Y. For the sake of time, we will only consider first 10k news items - you can easily remove the limitation in case you have more time to wait, and want to get better embeddings :)

[100]

We will also convert that data to one dataset, and batch it for training:

[101]

Now let's do the actual training. We will use SGD optimizer with pretty high learning rate. You can also try playing around with other optimizers, such as Adam. We will train for 200 epochs to begin with - and you can re-run this cell if you want even lower loss.

[102]

Epoch 1/200

/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/gradient_descent.py:102: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(SGD, self).__init__(name, **kwargs)

2156/2156 [==============================] - 7s 3ms/step - loss: 5.6134
Epoch 2/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.5431
Epoch 3/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.5029
Epoch 4/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.4754
Epoch 5/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.4548
Epoch 6/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.4382
Epoch 7/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.4243
Epoch 8/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.4123
Epoch 9/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.4019
Epoch 10/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3926
Epoch 11/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3843
Epoch 12/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3767
Epoch 13/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3697
Epoch 14/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3632
Epoch 15/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3571
Epoch 16/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3513
Epoch 17/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3459
Epoch 18/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3408
Epoch 19/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3359
Epoch 20/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3312
Epoch 21/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3266
Epoch 22/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3223
Epoch 23/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3181
Epoch 24/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3140
Epoch 25/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3101
Epoch 26/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3062
Epoch 27/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.3025
Epoch 28/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2989
Epoch 29/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2953
Epoch 30/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2919
Epoch 31/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2885
Epoch 32/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2852
Epoch 33/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2819
Epoch 34/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2787
Epoch 35/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2756
Epoch 36/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2725
Epoch 37/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2695
Epoch 38/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2665
Epoch 39/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2636
Epoch 40/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2607
Epoch 41/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2578
Epoch 42/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2550
Epoch 43/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2523
Epoch 44/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2495
Epoch 45/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2468
Epoch 46/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2442
Epoch 47/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2416
Epoch 48/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2390
Epoch 49/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2364
Epoch 50/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2339
Epoch 51/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2314
Epoch 52/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2290
Epoch 53/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2266
Epoch 54/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2242
Epoch 55/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2218
Epoch 56/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2195
Epoch 57/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2172
Epoch 58/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2149
Epoch 59/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2126
Epoch 60/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2104
Epoch 61/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2082
Epoch 62/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2060
Epoch 63/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2038
Epoch 64/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.2017
Epoch 65/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1996
Epoch 66/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1975
Epoch 67/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1954
Epoch 68/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1933
Epoch 69/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1913
Epoch 70/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1893
Epoch 71/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1873
Epoch 72/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1853
Epoch 73/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1833
Epoch 74/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1814
Epoch 75/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1795
Epoch 76/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1775
Epoch 77/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1756
Epoch 78/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1737
Epoch 79/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1719
Epoch 80/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1700
Epoch 81/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1682
Epoch 82/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1663
Epoch 83/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1645
Epoch 84/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1627
Epoch 85/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1609
Epoch 86/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1592
Epoch 87/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1574
Epoch 88/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1557
Epoch 89/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1539
Epoch 90/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1522
Epoch 91/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1505
Epoch 92/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1488
Epoch 93/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1471
Epoch 94/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1454
Epoch 95/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1438
Epoch 96/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1421
Epoch 97/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1405
Epoch 98/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1389
Epoch 99/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1372
Epoch 100/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1356
Epoch 101/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1341
Epoch 102/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1325
Epoch 103/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1309
Epoch 104/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1293
Epoch 105/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1278
Epoch 106/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1263
Epoch 107/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1247
Epoch 108/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1232
Epoch 109/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1217
Epoch 110/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1202
Epoch 111/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1187
Epoch 112/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1173
Epoch 113/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1158
Epoch 114/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1144
Epoch 115/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1129
Epoch 116/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1115
Epoch 117/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1101
Epoch 118/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1086
Epoch 119/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1072
Epoch 120/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1058
Epoch 121/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1045
Epoch 122/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1031
Epoch 123/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1017
Epoch 124/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.1004
Epoch 125/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0990
Epoch 126/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0977
Epoch 127/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0963
Epoch 128/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0950
Epoch 129/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0937
Epoch 130/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0924
Epoch 131/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0911
Epoch 132/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0898
Epoch 133/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0885
Epoch 134/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0873
Epoch 135/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0860
Epoch 136/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0848
Epoch 137/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0835
Epoch 138/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0823
Epoch 139/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0810
Epoch 140/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0798
Epoch 141/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0786
Epoch 142/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0774
Epoch 143/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0762
Epoch 144/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0750
Epoch 145/200
2156/2156 [==============================] - 8s 4ms/step - loss: 5.0739
Epoch 146/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0727
Epoch 147/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0715
Epoch 148/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0704
Epoch 149/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0692
Epoch 150/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0681
Epoch 151/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0670
Epoch 152/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0658
Epoch 153/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0647
Epoch 154/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0636
Epoch 155/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0625
Epoch 156/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0614
Epoch 157/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0603
Epoch 158/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0593
Epoch 159/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0582
Epoch 160/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0571
Epoch 161/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0561
Epoch 162/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0550
Epoch 163/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0539
Epoch 164/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0529
Epoch 165/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0519
Epoch 166/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0508
Epoch 167/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0498
Epoch 168/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0488
Epoch 169/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0478
Epoch 170/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0468
Epoch 171/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0458
Epoch 172/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0448
Epoch 173/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0438
Epoch 174/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0428
Epoch 175/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0418
Epoch 176/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0409
Epoch 177/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0399
Epoch 178/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0389
Epoch 179/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0380
Epoch 180/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0370
Epoch 181/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0361
Epoch 182/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0351
Epoch 183/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0342
Epoch 184/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0333
Epoch 185/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0323
Epoch 186/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0314
Epoch 187/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0305
Epoch 188/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0296
Epoch 189/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0287
Epoch 190/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0278
Epoch 191/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0269
Epoch 192/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0260
Epoch 193/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0251
Epoch 194/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0242
Epoch 195/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0233
Epoch 196/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0225
Epoch 197/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0216
Epoch 198/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0207
Epoch 199/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0199
Epoch 200/200
2156/2156 [==============================] - 7s 3ms/step - loss: 5.0190

<keras.callbacks.History at 0x7ff7e52572d0>

Trying out Word2Vec

To use Word2Vec, let's extract vectors corresponding to all words in our vocabulary:

[103]

Let's see, for example, how the word Paris is encoded into a vector:

[104]

tf.Tensor(
[-0.13308628  0.50972325  0.00344684  0.185389   -0.03176536  0.22262476
 -0.3856765  -0.6854793   0.5185803  -0.7215402  -0.16101503  0.15622072
  0.00653811 -0.14954254  0.03379822 -0.01243829  0.27907634 -0.32538188
  0.21718933  0.31112966 -0.24142407  0.15589055  0.2915561   0.19029242
  0.08425518 -0.0941902  -0.54313695 -0.24854654  0.26196313  0.18027727], shape=(30,), dtype=float32)

It is interesting to use Word2Vec to look for synonyms. The following function will return n closest words to a given input. To find them, we compute the norm of $|w_i - v|$ , where $v$ is the vector corresponding to our input word, and $w_i$ is the encoding of $i$ -th word in the vocabulary. We then sort the array and return corresponding indices using argsort, and take first n elements of the list, which encode positions of closest words in the vocabulary.

[105]

['paris', 'philippines', 'seoul', 'jakarta', 'zoo']

[112]

['china', 'russia', 'pakistan', 'israel', 'turkey']

[113]

['official', 'military', 'office', 'police', 'sources']

Takeaway

Using clever techniques such as CBoW, we can train Word2Vec model. You may also try to train skip-gram model that is trained to predict the neighboring word given the central one, and see how well it performs.