Notebook
artificial-intelligencernn22-DeepRLganmicrosoft-for-beginnerslessonsAImicrosoft-AI-For-Beginnersmachine-learningdeep-learningcomputer-vision6-OthercnnNLP
Export
CartPole Skating
Problem: If Peter wants to escape from the wolf, he needs to be able to move faster than him. We will see how Peter can learn to skate, in particular, to keep balance, using Q-Learning.
First, let's install the gym and import required libraries:
[1]
Defaulting to user installation because normal site-packages is not writeable
Collecting gym
Downloading gym-0.25.0.tar.gz (720 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 720.4/720.4 KB 3.3 MB/s eta 0:00:0000:0100:01
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.18.0 in /usr/lib/python3/dist-packages (from gym) (1.21.5)
Collecting gym-notices>=0.0.4
Downloading gym_notices-0.0.7-py3-none-any.whl (2.7 kB)
Collecting cloudpickle>=1.2.0
Downloading cloudpickle-2.1.0-py3-none-any.whl (25 kB)
Building wheels for collected packages: gym
Building wheel for gym (pyproject.toml) ... done
Created wheel for gym: filename=gym-0.25.0-py3-none-any.whl size=824430 sha256=3f4ed647f1d12814bb457f7d83a7ccd0f682d12a0259ca07b7fab0db5100fc6e
Stored in directory: /home/leo/.cache/pip/wheels/c0/3c/33/32d86254a5bd554f5f07759ae1794646e490dd5fa81ebdcda3
Successfully built gym
Installing collected packages: gym-notices, cloudpickle, gym
Successfully installed cloudpickle-2.1.0 gym-0.25.0 gym-notices-0.0.7
Create a cartpole environment
[2]
Discrete(2) Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32) 1
/home/leo/.local/lib/python3.10/site-packages/gym/core.py:329: DeprecationWarning: WARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future. deprecation( /home/leo/.local/lib/python3.10/site-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: WARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future. deprecation(
To see how the environment works, let's run a short simulation for 100 steps.
[3]
/home/leo/.local/lib/python3.10/site-packages/gym/core.py:57: DeprecationWarning: WARN: You are calling render method, but you didn't specified the argument render_mode at environment initialization. To maintain backward compatibility, the environment will render in human mode.
If you want to render in human mode, initialize the environment in this way: gym.make('EnvName', render_mode='human') and don't call the render method.
See here for more information: https://www.gymlibrary.ml/content/api/
deprecation(
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) ~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py in _render(self, mode) 221 try: --> 222 import pygame 223 from pygame import gfxdraw ModuleNotFoundError: No module named 'pygame' During handling of the above exception, another exception occurred: DependencyNotInstalled Traceback (most recent call last) /tmp/ipykernel_32716/4123126963.py in <module> 2 3 for i in range(100): ----> 4 env.render() 5 env.step(env.action_space.sample()) 6 env.close() ~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs) 64 ) 65 ---> 66 return render_func(self, *args, **kwargs) 67 68 return render ~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs) 429 def render(self, *args, **kwargs): 430 """Renders the environment.""" --> 431 return self.env.render(*args, **kwargs) 432 433 def close(self): ~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs) 64 ) 65 ---> 66 return render_func(self, *args, **kwargs) 67 68 return render ~/.local/lib/python3.10/site-packages/gym/wrappers/order_enforcing.py in render(self, *args, **kwargs) 49 "set `disable_render_order_enforcing=True` on the OrderEnforcer wrapper." 50 ) ---> 51 return self.env.render(*args, **kwargs) 52 53 @property ~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs) 64 ) 65 ---> 66 return render_func(self, *args, **kwargs) 67 68 return render ~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs) 429 def render(self, *args, **kwargs): 430 """Renders the environment.""" --> 431 return self.env.render(*args, **kwargs) 432 433 def close(self): ~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs) 64 ) 65 ---> 66 return render_func(self, *args, **kwargs) 67 68 return render ~/.local/lib/python3.10/site-packages/gym/wrappers/env_checker.py in render(self, *args, **kwargs) 51 if self.checked_render is False: 52 self.checked_render = True ---> 53 return env_render_passive_checker(self.env, *args, **kwargs) 54 else: 55 return self.env.render(*args, **kwargs) ~/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py in env_render_passive_checker(env, *args, **kwargs) 322 ) 323 --> 324 result = env.render(*args, **kwargs) 325 326 # TODO: Check that the result is correct ~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs) 64 ) 65 ---> 66 return render_func(self, *args, **kwargs) 67 68 return render ~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py in render(self, mode) 215 return self.renderer.get_renders() 216 else: --> 217 return self._render(mode) 218 219 def _render(self, mode="human"): ~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py in _render(self, mode) 223 from pygame import gfxdraw 224 except ImportError: --> 225 raise DependencyNotInstalled( 226 "pygame is not installed, run `pip install gym[classic_control]`" 227 ) DependencyNotInstalled: pygame is not installed, run `pip install gym[classic_control]`
During simulation, we need to get observations in order to decide how to act. In fact, step function returns us back current observations, reward function, and the done flag that indicates whether it makes sense to continue the simulation or not:
[4]
[ 0.03044442 -0.19543914 -0.04496216 0.28125618] -> 1.0 [ 0.02653564 -0.38989186 -0.03933704 0.55942606] -> 1.0 [ 0.0187378 -0.19424049 -0.02814852 0.25461393] -> 1.0 [ 0.01485299 -0.38894946 -0.02305624 0.53828712] -> 1.0 [ 0.007074 -0.19351108 -0.0122905 0.23842953] -> 1.0 [ 0.00320378 0.00178427 -0.00752191 -0.05810469] -> 1.0 [ 0.00323946 0.19701326 -0.008684 -0.35315131] -> 1.0 [ 0.00717973 0.00201587 -0.01574703 -0.06321931] -> 1.0 [ 0.00722005 0.19736001 -0.01701141 -0.36082863] -> 1.0 [ 0.01116725 0.39271958 -0.02422798 -0.65882671] -> 1.0 [ 0.01902164 0.19794307 -0.03740452 -0.37387001] -> 1.0 [ 0.0229805 0.39357584 -0.04488192 -0.67810827] -> 1.0 [ 0.03085202 0.58929164 -0.05844408 -0.98457719] -> 1.0 [ 0.04263785 0.78514572 -0.07813563 -1.2950295 ] -> 1.0 [ 0.05834076 0.98116859 -0.10403622 -1.61111521] -> 1.0 [ 0.07796413 0.78741784 -0.13625852 -1.35259196] -> 1.0 [ 0.09371249 0.98396202 -0.16331036 -1.68461179] -> 1.0 [ 0.11339173 0.79106371 -0.1970026 -1.44691436] -> 1.0 [ 0.12921301 0.59883361 -0.22594088 -1.22169133] -> 1.0
We can get min and max value of those numbers:
[5]
[-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38] [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38]
State Discretization
[6]
Let's also explore other discretization method using bins:
[7]
Sample bins for interval (-5,5) with 10 bins [-5. -4. -3. -2. -1. 0. 1. 2. 3. 4. 5.]
Let's now run a short simulation and observe those discrete environment values.
[8]
(0, 0, -1, -3) (0, 0, -2, 0) (0, 0, -2, -3) (0, 1, -3, -6) (0, 2, -4, -9) (0, 3, -6, -12) (0, 2, -8, -9) (0, 3, -10, -13) (0, 4, -13, -16) (0, 4, -16, -19) (0, 4, -20, -17) (0, 4, -24, -20)
Q-Table Structure
[9]
Let's Start Q-Learning!
[10]
[11]
0: 108.0, alpha=0.3, epsilon=0.9
Plotting Training Progress
[20]
[<matplotlib.lines.Line2D at 0x2814bb79788>]
From this graph, it is not possible to tell anything, because due to the nature of stochastic training process the length of training sessions varies greatly. To make more sense of this graph, we can calculate running average over series of experiments, let's say 100. This can be done conveniently using np.convolve:
[22]
[<matplotlib.lines.Line2D at 0x2814cc63488>]
Varying Hyperparameters and Seeing the Result in Action
Now it would be interesting to actually see how the trained model behaves. Let's run the simulation, and we will be following the same action selection strategy as during training: sampling according to the probability distribution in Q-Table:
[23]
Saving result to an animated GIF
If you want to impress your friends, you may want to send them the animated GIF picture of the balancing pole. To do this, we can invoke env.render to produce an image frame, and then save those to animated GIF using PIL library:
[26]
360