diff --git a/README.md b/README.md index d23cc6851..f68ebdd06 100644 --- a/README.md +++ b/README.md @@ -142,7 +142,7 @@ Here is a table of algorithms, the figure, name of the algorithm in the book and | 19.3 | Version-Space-Learning | `version_space_learning` | [`knowledge.py`](knowledge.py) | Done | Included | | 19.8 | Minimal-Consistent-Det | `minimal_consistent_det` | [`knowledge.py`](knowledge.py) | Done | | | 19.12 | FOIL | `FOIL_container` | [`knowledge.py`](knowledge.py) | Done | | -| 21.2 | Passive-ADP-Agent | `PassiveADPAgent` | [`rl.py`][rl] | Done | | +| 21.2 | Passive-ADP-Agent | `PassiveADPAgent` | [`rl.py`][rl] | Done | Included | | 21.4 | Passive-TD-Agent | `PassiveTDAgent` | [`rl.py`][rl] | Done | Included | | 21.8 | Q-Learning-Agent | `QLearningAgent` | [`rl.py`][rl] | Done | Included | | 22.1 | HITS | `HITS` | [`nlp.py`][nlp] | Done | Included | diff --git a/rl.ipynb b/rl.ipynb index f05613ddd..a8f6adc2c 100644 --- a/rl.ipynb +++ b/rl.ipynb @@ -11,7 +11,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": { "collapsed": true }, @@ -28,7 +28,11 @@ "\n", "* Overview\n", "* Passive Reinforcement Learning\n", - "* Active Reinforcement Learning" + " - Direct Utility Estimation\n", + " - Adaptive Dynamic Programming\n", + " - Temporal-Difference Agent\n", + "* Active Reinforcement Learning\n", + " - Q learning" ] }, { @@ -56,171 +60,331 @@ "source": [ "## PASSIVE REINFORCEMENT LEARNING\n", "\n", - "In passive Reinforcement Learning the agent follows a fixed policy and tries to learn the Reward function and the Transition model (if it is not aware of these)." + "In passive Reinforcement Learning the agent follows a fixed policy $\\pi$. Passive learning attempts to evaluate the given policy $pi$ - without any knowledge of the Reward function $R(s)$ and the Transition model $P(s'\\ |\\ s, a)$.\n", + "\n", + "This is usually done by some method of **utility estimation**. The agent attempts to directly learn the utility of each state that would result from following the policy. Note that at each step, it has to *perceive* the reward and the state - it has no global knowledge of these. Thus, if a certain the entire set of actions offers a very low probability of attaining some state $s_+$ - the agent may never perceive the reward $R(s_+)$.\n", + "\n", + "Consider a situation where an agent is given a policy to follow. Thus, at any point it knows only its current state and current reward, and the action it must take next. This action may lead it to more than one state, with different probabilities.\n", + "\n", + "For a series of actions given by $\\pi$, the estimated utility $U$:\n", + "$$U^{\\pi}(s) = E(\\sum_{t=0}^\\inf \\gamma^t R^t(s')$$)\n", + "Or the expected value of summed discounted rewards until termination.\n", + "\n", + "Based on this concept, we discuss three methods of estimating utility:\n", + "\n", + "1. **Direct Utility Estimation (DUE)**\n", + " \n", + " The first, most naive method of estimating utility comes from the simplest interpretation of the above definition. We construct an agent that follows the policy until it reaches the terminal state. At each step, it logs its current state, reward. Once it reaches the terminal state, it can estimate the utility for each state for *that* iteration, by simply summing the discounted rewards from that state to the terminal one.\n", + "\n", + " It can now run this 'simulation' $n$ times, and calculate the average utility of each state. If a state occurs more than once in a simulation, both its utility values are counted separately.\n", + " \n", + " Note that this method may be prohibitively slow for very large statespaces. Besides, **it pays no attention to the transition probability $P(s'\\ |\\ s, a)$.** It misses out on information that it is capable of collecting (say, by recording the number of times an action from one state led to another state). The next method addresses this issue.\n", + " \n", + "2. **Adaptive Dynamic Programming (ADP)**\n", + " \n", + " This method makes use of knowledge of the past state $s$, the action $a$, and the new perceived state $s'$ to estimate the transition probability $P(s'\\ |\\ s,a)$. It does this by the simple counting of new states resulting from previous states and actions.
\n", + " The program runs through the policy a number of times, keeping track of:\n", + " - each occurrence of state $s$ and the policy-recommended action $a$ in $N_{sa}$\n", + " - each occurrence of $s'$ resulting from $a$ on $s$ in $N_{s'|sa}$.\n", + " \n", + " It can thus estimate $P(s'\\ |\\ s,a)$ as $N_{s'|sa}/N_{sa}$, which in the limit of infinite trials, will converge to the true value.
\n", + " Using the transition probabilities thus estimated, it can apply `POLICY-EVALUATION` to estimate the utilities $U(s)$ using properties of convergence of the Bellman functions.\n", + "\n", + "3. **Temporal-difference learning (TD)**\n", + " \n", + " Instead of explicitly building the transition model $P$, the temporal-difference model makes use of the expected closeness between the utilities of two consecutive states $s$ and $s'$.\n", + " For the transition $s$ to $s'$, the update is written as:\n", + "$$U^{\\pi}(s) \\leftarrow U^{\\pi}(s) + \\alpha \\left( R(s) + \\gamma U^{\\pi}(s') - U^{\\pi}(s) \\right)$$\n", + " This model implicitly incorporates the transition probabilities by being weighed for each state by the number of times it is achieved from the current state. Thus, over a number of iterations, it converges similarly to the Bellman equations.\n", + " The advantage of the TD learning model is its relatively simple computation at each step, rather than having to keep track of various counts.\n", + " For $n_s$ states and $n_a$ actions the ADP model would have $n_s \\times n_a$ numbers $N_{sa}$ and $n_s^2 \\times n_a$ numbers $N_{s'|sa}$ to keep track of. The TD model must only keep track of a utility $U(s)$ for each state." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Passive Temporal Difference Agent\n", + "#### Demonstrating Passive agents\n", "\n", - "The PassiveTDAgent class in the rl module implements the Agent Program (notice the usage of word Program) described in **Fig 21.4** of the AIMA Book. PassiveTDAgent uses temporal differences to learn utility estimates. In simple terms we learn the difference between the states and backup the values to previous states while following a fixed policy. Let us look into the source before we see some usage examples." + "Passive agents are implemented in `rl.py` as various `Agent-Class`es.\n", + "\n", + "To demonstrate these agents, we make use of the `GridMDP` object from the `MDP` module. `sequential_decision_environment` is similar to that used for the `MDP` notebook but has discounting with $\\gamma = 0.9$.\n", + "\n", + "The `Agent-Program` can be obtained by creating an instance of the relevant `Agent-Class`. The `__call__` method allows the `Agent-Class` to be called as a function. The class needs to be instantiated with a policy ($\\pi$) and an `MDP` whose utility of states will be estimated." ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "%psource PassiveTDAgent" + "from mdp import sequential_decision_environment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The Agent Program can be obtained by creating the instance of the class by passing the appropriate parameters. Because of the __ call __ method the object that is created behaves like a callable and returns an appropriate action as most Agent Programs do. To instantiate the object we need a policy ($\\pi$) and a mdp whose utility of states will be estimated. Let us import a `GridMDP` object from the `MDP` module. **Figure 17.1 (sequential_decision_environment)** is similar to **Figure 21.1** but has some discounting as **gamma = 0.9**." + "The `sequential_decision_environment` is a GridMDP object as shown below. The rewards are **+1** and **-1** in the terminal states, and **-0.04** in the rest. Now we define actions and a policy similar to **Fig 21.1** in the book." ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "from mdp import sequential_decision_environment" + "# Action Directions\n", + "north = (0, 1)\n", + "south = (0,-1)\n", + "west = (-1, 0)\n", + "east = (1, 0)\n", + "\n", + "policy = {\n", + " (0, 2): east, (1, 2): east, (2, 2): east, (3, 2): None,\n", + " (0, 1): north, (2, 1): north, (3, 1): None,\n", + " (0, 0): north, (1, 0): west, (2, 0): west, (3, 0): west, \n", + "}\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "**Figure 17.1 (sequential_decision_environment)** is a GridMDP object and is similar to the grid shown in **Figure 21.1**. The rewards in the terminal states are **+1** and **-1** and **-0.04** in rest of the states. Now we define a policy similar to **Fig 21.1** in the book." + "### Direction Utility Estimation Agent\n", + "\n", + "The `PassiveDEUAgent` class in the `rl` module implements the Agent Program described in **Fig 21.2** of the AIMA Book. `PassiveDEUAgent` sums over rewards to find the estimated utility for each state. It thus requires the running of a number of iterations." ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "# Action Directions\n", - "north = (0, 1)\n", - "south = (0,-1)\n", - "west = (-1, 0)\n", - "east = (1, 0)\n", - "\n", - "policy = {\n", - " (0, 2): east, (1, 2): east, (2, 2): east, (3, 2): None,\n", - " (0, 1): north, (2, 1): north, (3, 1): None,\n", - " (0, 0): north, (1, 0): west, (2, 0): west, (3, 0): west, \n", - "}\n" + "%psource PassiveDUEAgent" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "DUEagent = PassiveDUEAgent(policy, sequential_decision_environment)\n", + "for i in range(200):\n", + " run_single_trial(DUEagent, sequential_decision_environment)\n", + " DUEagent.estimate_U()\n", + "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Let us create our object now. We also use the **same alpha** as given in the footnote of the book on **page 837**." + "The calculated utilities are:" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print('\\n'.join([str(k)+':'+str(v) for k, v in DUEagent.U.items()]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Adaptive Dynamic Programming Agent\n", + "\n", + "The `PassiveADPAgent` class in the `rl` module implements the Agent Program described in **Fig 21.2** of the AIMA Book. `PassiveADPAgent` uses state transition and occurrence counts to estimate $P$, and then $U$. Go through the source below to understand the agent." + ] + }, + { + "cell_type": "code", + "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "our_agent = PassiveTDAgent(policy, sequential_decision_environment, alpha=lambda n: 60./(59+n))" + "%psource PassiveADPAgent" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We instantiate a `PassiveADPAgent` below with the `GridMDP` shown and train it over 200 iterations. The `rl` module has a simple implementation to simulate iterations. The function is called **run_single_trial**." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "ADPagent = PassiveADPAgent(policy, sequential_decision_environment)\n", + "for i in range(200):\n", + " run_single_trial(ADPagent, sequential_decision_environment)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The rl module also has a simple implementation to simulate iterations. The function is called **run_single_trial**. Now we can try our implementation. We can also compare the utility estimates learned by our agent to those obtained via **value iteration**.\n" + "The calculated utilities are:" ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "print('\\n'.join([str(k)+':'+str(v) for k, v in ADPagent.U.items()]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Passive Temporal Difference Agent\n", + "\n", + "`PassiveTDAgent` uses temporal differences to learn utility estimates. We learn the difference between the states and backup the values to previous states. Let us look into the source before we see some usage examples." + ] + }, + { + "cell_type": "code", + "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "from mdp import value_iteration" + "%psource PassiveTDAgent" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The values calculated by value iteration:" + "In creating the `TDAgent`, we use the **same learning rate** $\\alpha$ as given in the footnote of the book on **page 837**." ] }, { "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{(0, 1): 0.3984432178350045, (1, 2): 0.649585681261095, (3, 2): 1.0, (0, 0): 0.2962883154554812, (3, 0): 0.12987274656746342, (3, 1): -1.0, (2, 1): 0.48644001739269643, (2, 0): 0.3447542300124158, (2, 2): 0.7953620878466678, (1, 0): 0.25386699846479516, (0, 2): 0.5093943765842497}\n" - ] - } - ], + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], "source": [ - "print(value_iteration(sequential_decision_environment))" + "TDagent = PassiveTDAgent(policy, sequential_decision_environment, alpha = lambda n: 60./(59+n))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Now the values estimated by our agent after **200 trials**." + "Now we run **200 trials** for the agent to estimate Utilities." ] }, { "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{(0, 1): 0.4431282384930237, (1, 2): 0.6719826603921873, (3, 2): 1, (0, 0): 0.32008510559157544, (3, 0): 0.0, (3, 1): -1, (2, 1): 0.6258841793121656, (2, 0): 0.0, (2, 2): 0.7626863051408717, (1, 0): 0.19543350078456248, (0, 2): 0.550838599140139}\n" - ] - } - ], + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], "source": [ "for i in range(200):\n", - " run_single_trial(our_agent,sequential_decision_environment)\n", - "print(our_agent.U)" + " run_single_trial(TDagent,sequential_decision_environment)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The calculated utilities are:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print('\\n'.join([str(k)+':'+str(v) for k, v in TDagent.U.items()]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Comparison with value iteration method\n", + "\n", + "We can also compare the utility estimates learned by our agent to those obtained via **value iteration**.\n", + "\n", + "**Note that value iteration has a priori knowledge of the transition table $P$, the rewards $R$, and all the states $s$.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from mdp import value_iteration" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The values calculated by value iteration:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "U_values = value_iteration(sequential_decision_environment)\n", + "print('\\n'.join([str(k)+':'+str(v) for k, v in U_values.items()]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "We can also explore how these estimates vary with time by using plots similar to **Fig 21.5a**. To do so we define a function to help us with the same. We will first enable matplotlib using the inline backend." + "## Evolution of utility estimates over iterations\n", + "\n", + "We can explore how these estimates vary with time by using plots similar to **Fig 21.5a**. We will first enable matplotlib using the inline backend. We also define a function to collect the values of utilities at each iteration." ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": null, "metadata": { "collapsed": true }, @@ -248,25 +412,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Here is a plot of state (2,2)." + "Here is a plot of state $(2,2)$." ] }, { "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xd4HNW5wOHft7vqlmWrudtyxQ03\nhAummRbTe0JPAsFAAgkhgUBuQgqQhJBw0yghQCimd8MFDAbTMbbce5ObXCVZvWv33D92ZrRarcrK\nWsnWfu/z+LF2djQ6I82e7/QjxhiUUkopAFdXJ0AppdThQ4OCUkophwYFpZRSDg0KSimlHBoUlFJK\nOTQoKKWUckQsKIjIkyJyQETWNPP+lSKyyvr3lYhMjFRalFJKtU0kawpPAbNbeH8bcJIxZgJwD/BY\nBNOilFKqDTyRurAx5jMRyWrh/a8CXi4CBkYqLUoppdomYkEhTNcB7zX3pojMAeYAJCUlHTN69OjO\nSpdSSnULS5cuLTDGZLR2XpcHBRGZhT8oHN/cOcaYx7Cal7Kzs01OTk4npU4ppboHEdnRlvO6NCiI\nyATgceBMY0xhV6ZFKaVUFw5JFZHBwOvA1caYTV2VDqWUUg0iVlMQkReAk4F0EckDfgPEABhjHgXu\nBtKAh0UEoN4Ykx2p9CillGpdJEcfXd7K+z8AfhCpn6+UUip8OqNZKaWUQ4OCUkophwYFpZRSDg0K\nSimlHBoUlFJKOTQoKKWUcmhQUEop5dCgoJRSyqFBQSmllEODglJKKYcGBaWUUg4NCkoppRwaFJRS\nSjk0KCillHJoUFBKKeXQoKCUUsqhQUEppZRDg4JSSimHBgWllFIODQpKKaUcGhSUUko5NCgopZRy\naFBQSinl0KCglFLKoUFBKaWUQ4OCUkophwYFpZRSjogFBRF5UkQOiMiaZt4XEfmHiGwRkVUiMiVS\naVFKKdU2kawpPAXMbuH9M4GR1r85wCMRTItSSqk2iFhQMMZ8Bhxs4ZTzgWeM3yKgl4j0i1R6lFJK\nta4r+xQGALsCXudZx5RSSnWRrgwKEuKYCXmiyBwRyRGRnPz8/AgnSymloldXBoU8YFDA64HAnlAn\nGmMeM8ZkG2OyMzIyOiVxSikVjboyKMwDrrFGIU0HSowxe7swPUopFfU8kbqwiLwAnAyki0ge8Bsg\nBsAY8yjwLnAWsAWoBL4fqbQopZRqm4gFBWPM5a28b4AfRernK6WUCp/OaFZKKeXQoKCUUsqhQUEp\npZRDg4JSSimHBgWllFIODQpKKaUcGhSUUko5NCgopZRyaFBQSinl0KCglFLKoUFBKaWUQ4OCUkop\nhwYFpZRSDg0KSimlHBoUlFJKOTQoKKWUcmhQUEop5dCgoJRSyqFBQSmllCNiezQfbr7cUsAD8zcC\nMCQtkR8cP4zHv8jlhyePICs9kTiP2zn3/TV7ydlexF1njcHtEue4z2d4c8VuEmPdnD62Ly8t2cXX\nuYXkl1Xj9Rl8BnzGMHtcX244aXib0lXn9fHJxnw27ivlhpOGE+NuGqeNMSzZXsSnmw6QmhTHdccP\nBaCoopYVu4o5+agMRARjDF/nFrJkWxFTh6YyY3ham9JQWF7DF1sK2Ly/nIOVtfz4lJH0TYlv9vyK\nmnoW5RayaX85u4oqKamq46RRGXw7e1DIc5fvLGZIWiKDUhOd49V1XpbtKGLzgXLiY1x859jBbUqr\nUs3x+gy7i6rYX1bNwYpaiipqOVhZS0VNPYmxHm48aXijz7MKLWqCgscl9EyIoaSqjrdW7OGtFXsA\nqPca/m/1Xl6cM52x/XuyYmcxN85dBsDl0wYzPKOHc41/LdzCgx9uAuA72YN4KWcXA3olMKBXAjFu\nFy4RNuwr5Y3lu9sUFEoq67j6yW9YlVcCwLFZqUwbltbknBvnLuXr3ELn2DUzhvDM1zt4YP4Gqut8\nvHLjDFISYrj1xRWs21sKwAkj01sNCuU19fz1g408+/UO6n0Gl4DPwFF9kvnucVlNzq+oqefBDzfx\n3Dc7qK7zAZCaFEtVrZdt+RWNgsLBiloemL+RN5bnUV3n44SR6Tx73TT2lVTz9482O8dtp43pQ1qP\nuCY/0+szPL94J4tyC/nzxRNIiouaR7aRqlovCzceYMG6/Zw2tg9nHd2vq5PUpYwxbNxfxqKthSzZ\nXsTG/WXsLKyk1utrcq4IGAPHDOnN9GFtKyhFs6j5hE0blsa0YWlU1tYz9u75zvH5a/cB8OG6/Vz2\n2KJG32NMw9fVdV4e/XSr8/qlnF1cO3Movz5nDCINpY+fvrSCnB0H8fkMLqtUYozho/UHmDU6s1FJ\n5ddvrWH93lJu/9ZRPDB/I2+v2sNry/K4/+IJTsn/1peWs3RHEfecP45ar+Ged9bx0MIt/G3BZqZm\npbJ4+0GWbD/Ik19sRwQeuGQCLyzeSU190w9HoOo6L1c9/g2r8or5zrGDuXzqIMb068nRv51Pbn45\nVz6+iD7J8Tz4nUkAFJTXcPlji9h8oJyLpwzk4mMGMK5/CikJMfz4heWszCt2rr12TwnXPZXDwYpa\nLpoygO2FFWw5UM6ynUX84OkcyqrruOSYgZwxri8Hy2v52SsryS2oIK1HHHuKq/jLBxv5xezRuF3C\n9c/ksHyn/9rfzh7ESaMy2vT3bsmS7QfpnRjDiMzkQ75Wa/YUV7Fk+0HOm9i/0XMS6PPN+fRLiQ+Z\nnjqvj/9+uY1HP83lYEUtAIUVtVEbFEoq65j7zQ5eXZrHtoIKAAb0SmD8gJ6cNqYPQ9MT6ZeSQGpS\nLKlJsfROjKXe52PKPR/ys5dXMq5/Tx67JruL7+LwFjVBwZYY6+Gvl07k/vc3cKCshnqfP+cvLK8J\ncXZDVPhm20Eqa718a1wf5q/dD8BNJw9v8kHvEeehvLqeYb98l0uOGchfLp3IC4t38cs3VvPAJRO4\n1CpN7yys5O1Ve7jhxOF8f2YWD8zfyNxFOwH4zbnjSIrz8NXWQhZuzOdXZ4/h6hlZfL45H4C/LdhM\n9pDePH3tVMbc/T5/fn8jsR4X79xyPKP6JDNv5R7Ka+pb/D38Zf5GVuwq5tGrpjB7fEMGMzg1kae/\n3uG8fvA7kzDG8NOXVrCrqJK5103j+JHpja6VFOehwvp5heU1fO+/S/C4hNd/eBzjB6Twz482syh3\nExc/8hWDeifyyo0znBrYroOVAGw9UM7wjB5c/cQ3bM2vYHTfZN5cvofcgnJ+e+5Yfvv2Oh78cBOP\nfrKV56+f1mwGG6ym3ovXZ0iM9T/qLy/ZxR2vrWJEZg9+fsZRPPrpVp65bio942PadL1w5Gw/yCWP\nfg3AmH49GdWncaZfW+/jztdX8fqy3UzNSuXlG2c0en9PcRVzns1hze5SThqVwZwTh/Gfz3PZXxrq\nWe3e6rw+Hvssl0c/3UpZdT3Th6Vyw4nDOH5kOgN7J7by3W6+Na4v763ZR37Iz3l4jDF8uikfEemQ\nQsrhJio7mi8+ZiDf/PJUYgPa73N2FDlf28cDawpLdxQhAhdNGegcy0hu2tzRI95DUWUdAK8uzQP8\n/RkABeW1bDlQxvaCCt5bsxdj4OoZQ5wMy1bv9f/gFxbvJCUhhqumDwGgf68E55zbv3UUCbFup+Zx\nxdTBTqYT53FR20JNYW9JFc98vYNvZw9sFBAA4mMa+lYGWD9v/tr9fL65gP85a0yTgADQI87tBKE/\nvreB4spanvjusYwfkALA0IwkwP/7fOmG6Y2a5Pr3SiDO42Jrfjm/enM1uw5WEetx8Yd3N7Bpfxn/\nvjqb780cSkZyHCt3FfN1biElVXXN3hv4a0F/fHc9OworyL53Ad/+tz9j/mprAb98YzUAWw6Uc+Pc\npazYVcyNzy7l6ie+afGa4Vq6o4grH2+45u7iqkbv19b7+OFzS3l92W4AKmobB/HtBRVc9PBX7Cio\n5JErp/D0tVOZOSKdfinx5Jf5M7aaem+HpvlQfbmlgDP//jlbDpR16HXziiq56OGveGD+RqYNTeW9\nn5zAi3NmcNnUwW0ICH7/umIKN500nLoQzUvhKKms48cvruB7/13Cd59cfEjXOlxFZVAAEBFiPQ23\nn1fU8KG1O1l9AUFhb3EVmclxDEtPavG6PQLavAel+jPVDfv87fz7S6s57cHPOPkvn7Aot5BhGUlO\nxhuozufD6/OXRs4c39fJqPunNJw7dWgq4G9zB7hsakN7fozb1eLD//qy3dR6fdw8a2ST9zKtQHfi\nqAwno3rss60MTU/i8qmhO4N7xMVQXecjN7+c15flcc2MLMb27+m8P76/Pzj8z1lj6JfS+H7dLmFE\nZg+e/2Yn767ex82njMBn3dONJw13SmLjAq5X0Epp7+mvtvPvz3K55snFlFXXs2Z3KTX1Xn7x2ioG\npyXyu/PGAZAY6/+9frW1kM83F7R4zXAcKKvmprlL6dMznnduOR6AvcXVjc65//0NLFh/gHsvGM/F\nUwZSZDUNGWMoq67juqeXUFPv5eUbZ3BmQFNRRnI8hRU1PDB/AxN/9wEHyvzX9fkMb6/cQ0llHZ9t\nyscElmg6weeb87ny8W9Yv7eUDfs6Lihs2FfKhQ9/xfbCCh65cgqPf/dYxvTr2fo3hhDjdmFMw2cm\nXJv2l3HWPz7nvdV7iXH7C2MtBWafzzhNfkeSqA0KQKOgYLtw8gB+cqo/szQBzUf7Sqvpm5JAPysT\nP3V0ZshrJsc3BIWj+vSk3utjR6HVRJJf7ryXs72IaUMbOr08AX0NXp9hw75Sq5rccE6ClYmdG9A+\nPeuoDOtnNTRN+INC8w/+Wyv8zRWD05qWsv540QRemjOdsf16UlnjZWt+Oct2FnPF1MF4QoyMAkiK\n86fryS+3ATDnxGGN3s9KT2Ll3WdwfdBx24SBKVTUeumdGMN1xw/loikDALjx5IbO+j9fMoEHLpkA\nQH5Z8x+0oopa/rVwC4DzexeBJ77Yxq6DVfz23HGcPrYPfXrG8dAVUxjbzgymJX/4v/UUV9Xx2DXH\nMLpvMi6BfSUNhY7PNuXzxBfb+O6MIVw1fQjpybEUVNTycs4uJvzuA259cQXbCyt56MopTTLAjOQ4\njIGHFm6lus7Hx+sPAPCfz3O55YXlzPrrJ1zz5GLueHWVE2iC7Sup5kBpQ5Dy+Qx3vb6at1bsbtf9\nbtxXxk1zl5FkPZ81dYdWGrftOljJ1U8sxi3Cazcd1yg4tofHysjbU1tYu6eESx/9mjqvj1dvOo77\nLjgagP0loQso1XVeZv31E6bc82G7gxD4WxsufPhLqus6r1YY3UEhRCZ3+7eOckqQgYWtvSXV9E+J\np0ech7dvPp5/XjE55DUDawoxbiGvqMrptwgsjZbV1DO6b0NGvvDnJ/MDa6hpndfHMqs5Kzurd6Pr\nb77vTP5udf4CPHZNNut/P7tRG3uMu/nmowOl1WzaX84pY0IHtYzkOKYNSyMx1k2t18f7a/wd8edO\n7B/y/MB7fu6bnZw4KoM+PZsOZ01JbL7Nfli6vzlpxvA0kuI83HPBeFbefUaj32VmcjwTB/UCYOfB\nimav9dw3OyirrndqUhdNHoAx8I+PNnPCyHROHJVB/14JfPPL05g1OpMHvzOR2eP6Ajg1lEOxYlcx\nb67Yw/UnDGV035543C4yk+PZW+LPhL0+w33/t56stETuOmsMAOlJcdTW+7jj1VWUVdfz0YYDXH/C\nMI4b3rSpLj0p1vk6JSGGTzfls6+kmr8t2AzglExfWZrnBGnwZ1Iv5+zixy8s5+JHvuJnr6zE6zMs\n3VHEs4t28MLinfzx3Q1h329NvZdbXlhGQqyb566fDkBVKxmYPwit4vHPc53fyTur9lBV2/B91XVe\n5jy7lNp6H89eN7VJf0x72J/3cINCXlEl3/vvEpJi3bx203FMGtTLacrdU1LV5Hyvz3DbyyucQkll\nbcv9e815OWcXP39lJct3FrNhXxl/fG89K3YVt/6NhyjqOpoDhaopJMa6nQzWDgrGGPYWV3GC1Z5+\n9MCUZq8ZmJF5fYZthf4MbFz/nqzdU9ro3CEBJfVBqYlOk0u917A1v4IecZ4mzUvB8xhi3C4CugGc\n+wo1NA9whrYe18pwVTswvrdmL6P7Jrc4b6GHVTsyBs4Y27fF64ZyzsR+fLhuP3ed6c8k4zzuRvNG\nbOnWkNVfvLaaEZnJHDOkccA0xvDikl3MHJHGHy+cQM6Og/SMj+H15buprvPxvRDDbEf37cnRA1N4\nf+0+6nw+4lxNf244/vHRZtJ7xHLTySOcY31T4tlnlczfXrmHjfvL+Oflk51mwfTk2EbX6NsznltO\nGUEods3h75dN4p1Ve9lyoJz739+A1xhuPW0kCzccYGSfZF5dmuc8v//9chu/e3tdk2v95YONPPJJ\nw4i6Xi0E7uY89PEWNu0v58nvZTPUalptrVT74pJdvLB4FwDfPnYQ//ksl39+vIX7Lz7ama/ywPyN\nrN9bypPfy2ZkBwQEaKgp1LdQiw5WW+/jR88to6bOy/M3HefMtenXy/952FPcNCj8/aPNvLt6H0PT\nk9hWUEFVrZfkMAcyfLWlgF++vtrJN347by0rdhUT73EzySocRUpEawoiMltENorIFhG5M8T7g0Vk\noYgsF5FVInJWJNMTzG4XDBTrcWEXun3Wp6qspp6KWi99Q5SAg/WIbxwUdljD5qYM7t3k3CFpjfsn\n7OaZep+P3IIKhqYntXmUTaN7cEuzpaFVeSXEx7habTax5wOs2V3a6nyHwLkDJ4ToiG5Nv5QEXr5x\nRqPJbaH0Smj4YL2zak+T91fllZBXVMUFkwYwOC2Ri6YMJLOnP5CkJcVyYjMjRezO+kOp5oO/M3nh\nxgNcPnVwo8JBeo9YCsv9Jfh/f5bL6L7JnB3QFJJi3VdynIcFt53Ec9dPa3Y+RlZ6Elv/cBbnTxpA\nVloi2woqeHPFbq6dOZRbTxvFWzcfz18unUhyvIfymnqq67whA8Lekiqe+KKhJjF9WGqTzvCWFFfW\nsnJXMY9+lsv5k/pzyug+xMf4n9+WhkOXVNXxlw82Op+9V3Py+OfHWxp93/q9pfz3y21cNX0wp4zu\n0+Y0tcbTjprC3z/axMq8Eu6/eEKj4GT37wX2RQIsyi3knx9v5qIpA5zAXlnbcpB8f80+p0YO/j6p\nHz2/jKHpSTz3g2nEelys2FXM2Uf349bTmvYDdrSIBQURcQMPAWcCY4HLRWRs0Gm/Al42xkwGLgMe\njlR6QrGzALtUDP4qZnA2XF7tr/61pSSVHNdwTr3PUFhRi0tgRGaPJuc2qQW47DZPw7aCcqfkFa6W\nmo827CtlVJ/kZvsHbIG/k4kDWy6ZNO5cb9tokPZwBfS7LN52sMn7H6zbh9slnD62ISOxA/m5E/uH\nnC0ODf059WEEhXvfWcdv3lrT6NhLS6zSb9DM7l6JsRRX1rJ2Twnr95Zy5bTBje7l6AG9GJHZg6eu\nncqIzB6NRmeFYgexwWlJ1PsMbhG+PzOr0Tkp1kTNedYkzay0RH548nA++fnJ3HP+OHzGXwq+/oSh\n3HvBeGYdlUlZdT2l1S2P7LKd/9CXnP/Ql9TW+/j5GUcB/s+OS1quKTy8cAtFlbW8dMMM3C7hz/Mb\nmqxKKuswxnDv/62jZ0IMt58xuk1paatYu0+hjX/nrfnl/PvTXC6aMqBJf0ZCrJsBvRL4ZOMBLn7k\nK5btLKKm3ssvX1/N4NRE7jl/vPMZaiko2KPgbpy7FPDXdn/x6ioqa708ctUUeiXGMrZfT4amJ/Gn\ni49uVyExXJFsPpoKbDHG5AKIyIvA+UBgscUAdpE1BWha/Isk69lIjvc4fziPNTMZGpqP7Pfig9tp\nQmi0LIbxjz7olRjrNH0AvHLjDFbsLG7SfGVn1FV1XnYXVXHh5IG0R4yn+dFHG/aWcWoz/QmBAofJ\nttRcBo2DQqQtuO1E/nfBZj7flN/kvS+2FDJ5UC96JTY0x2T2jOdfV0zm+BHN12DsoOBtY7NCbb2P\nF5fscmoh4P8wv74sj+NHpDcJjL0SYiiqrOPVpXnEul1N+mcykuNYcNtJbfrZgYZYP+eso/s16cfp\nGe8PCk9/vZ1RfXow/9YTnQzFbgefOjSV/znbX06za167i6ro2a/lws/SHUVOe/kZY/s49ysixMe4\nG/UNBCqpqmPuoh2cN7E/Uwb3Zmh6ElsOlHPFtMG8sWw3pdV1LNlexJdbCvn1OWNb7IdqD4/Lqom3\nsaZw7zvrSIhxO82awUb3TeajDf6O/p+9vJJLjhlIbkEFT33/WJLiPM5nqKoudJ+CMYbfvb3WeV3v\n9fHBuv0s3JjPr88Z60xmfOyaY4h1u8JugmqvSDYfDQB2BbzOs44F+i1wlYjkAe8Ct4S6kIjMEZEc\nEcnJz2+aGbSXnQUE/7KDm4/skk/wfIJQBqclEuexHz5DcWUdvRJj6J3U8DOOzUoNORLHbvPcX1KN\nz9Cm5qpQYq3RR8HDEg9W1FJYUdumTrukgJrC0LSWaywJVrBsKePtKCMykxnbryel1fW8t3qvc7y0\nuo7VecUh+0rOmdC/UaAI5rabFXxtyyxydhykvKaegrKGkSe5BRXkFVXxrXFN+1R6J8VSVefljeW7\nOW1sZotpCcfEgb2YNjSVH85quqRKSkIMX2wuYO2eUq6ekdWohDmmb08GpSZw66kNTRF2rXV3UetN\nSC8u3klSrJuHr5zCX789sdF78TFuqpsZpvni4p1U1Hq5/gT/sz+6bzIel3DTScOdms3jn+fSKzGG\nK5oZ/nwowhl9tHxnEQs35vPDWSNCzkcCGGUNFEmKdbOtoIKHF27htDF9OPkof6ErVE0hcDj1xxsO\n8PnmAud3P/P+j/nhc8sY3TeZ784Y4pyXmRzfYc9MW0QyKISq5wQXxS4HnjLGDATOAp4VkSZpMsY8\nZozJNsZkZ2R03AxCO9MMHEYKDUHBTqw9miKhDTWFHnEeNt57JtOHpeL1GYoqa+md6J9y35oYqyRj\nj1RJ79G+B8GugdR6fVTW1jujH+zZw8F9GaEkBpT+Xa0sIjYkLZE/XzyBh66c0q70hssOljc9t8z5\nkK3YWYzPwNSh4a9tExNmn8KnG/0Fk9LqeqfAYB8LNcPVbnYsrqzjtDEd10aekhjDSzfMYHTfpv1D\nKQkx1Hp9eFzCeRMa10x6J8Xy+R2ncFxAEB/Q2woKLfQrGOOfC/HK0jzOmdCfs47u16RAlRDjbrSm\nlc3nMzzz9Q5mDEtzJjX+9PRR/OeabAalJtIzwcOqvBI+XL+fq6YNcYZfd6SG0Uct/53fXL6bCx/+\nipSEGK4OyJyDnX10P84+uh//sp77ilovPz29IdAmBASF4spabnlhOdn3LuDzzf55JA9+uImh6Un8\n4kx/M5k9U/3uc8e22rwbSZGs9+cBgY2rA2naPHQdMBvAGPO1iMQD6cCBCKbLYecBwc0fQsOaRdAQ\n6RNi2/6H8rhcVHm9lFXWM6BXPKltiPR2SWavVb1Pb6aE0poYd0PfxNT7FlBT72PrH85yOsUG9m46\nYS5YYhgfShHh28c2XSE1UgJHQu0uqiK9Rxyrd/sXFWytqSsUu8mvraNSFm5seDwLK2qprKnnT+9t\nYFhGUsg+lV4JDX/7zqhNQUPn9YzhaW1qhklPiiPW42oxKKzYVcwtLywH4NvHhm7ajItxhRySmrOj\niN3FVdwx+yjn2PCMhv6TlIQYlmwvwiU4M/g7mjOQo4W/c0llHbe+tALwLzzZUtPo+AEpPHTlFCpq\n6vG4hJOPymRc/4bnz2k+qvXy+3fW8fZKf/b3Te5BvD7D2j2l/PmSCU5NPHtIb+b+YFqbmqkjKZLh\naAkwUkSGikgs/o7keUHn7AROBRCRMUA80HHtQ62wm4eC171pUlOwg0JM22Oo2yXU+wzFlf4+Bbv6\n19wIGGjIzO2aQkaIVUPbwikR1fuorPU6JeC8In9NYUAbgoJ9jT4925eGSApsP7czsZW7ihmanuRk\nhuFwhipav6fymnruf39DyA7Tkso6Nu0vZ8pgf+d7QVkNNzy7lFqvr9kVOHtbmfJRfZLJbGeTYLjs\nNX7OCNGcFYrLJQzoldBi89EH6/xrfl05bXDI0XQA8R43NSF+b2+t2E18jKvZmpL9dztueHqLw58P\nhf35am64NsDry/1L0/zw5OH8aFboYcHBkuI8zP3BNO6/+OhGx+2C1e7iKt5dvddpVt6wr4x/f5pL\n/5R4Lpg0gJF9ejDrqAx+dc7YLg8IEMGagjGmXkRuBuYDbuBJY8xaEfk9kGOMmQf8DPiPiPwUfx78\nPdOJ8/NNQEdzoIZ5Co37FMKp0rpdgtfns5qPYoj1uPjgpyeGXNbCZneE7XOaj9pZUwhoPgqUV1RF\nSkJMmxZ/65sSz5CAJSEOJ/0CMg070K3fV8qEVkZJNcf+vXutPoUH3t/A01/vYFSfHk06+9fs8ddI\nTh3Th2U7i9lbUs0Oq1nux6eEHi5oFwhCrRsVKXbgbG7mfSgDeiWQ10JN4cN1+5k5Io37Ljy62XPi\nY1xNmo/qvD7eXb2X08f2bXao7Xar4/q8Sc1PkjxUMe7WO5rfXL6bcf17csfs8EY+hSoQ2PnFU19t\np7rOx/u3nsCjn2zlTWtE2O3fOspp6v3v96eG9fMiKaINV8aYd40xo4wxw40x91nH7rYCAsaYdcaY\nmcaYicaYScaYDyKZnhDpA0IEBed9///h9CnY3C6hssZLdZ3PyRRG9UlucT8AT0BNISnW3e52Vfvh\nD1wpdXdxFW8u393mZR3iY9x8evssp9PscJIU52HZr0+nR5yH3UVVVNbWk1dUxah2LoUdPCR1ldUU\nFapzz26mmmX9Xm6cuxSvz/DPyyc3W8Idmp7EiaMyuOSY9o0ma49fnT2Gd245vtEiiq1JTYqltJnF\nBrcV+Jc/P72VPpGEWHeTGtbyncUUVdZx1vjmay12k2aojvqO4gkY8h1Kbn45K/NKuHBy8HiY9km0\n8ov8shqOzerN6L49GwWPSzvxeQhHVM9obm70kTMk1Xrd0KfQ9kza45KGZqA29g3Ymfnu4iqGZbRv\njgI0NP0EzrZctqOIspp67jyzY8d+d5XUpFgG9EpgX2k1ufkVGAOj+rQ8vr85gX0KNfVeZ/+GUJXW\n1XklDEpNaLRECcBJRzXfLJhSLpeCAAAbOElEQVQQ6+aZazu3JJgU53E6dNuqpfkt9rLtrU0mi/e4\nKa5sHFg+35yP2yWNOraD/e07k9hbUt2u5r+2smvQzY0ye2vFHkRaXtIlHIGdxRdYgcauLfaM93Ra\nU2K4ojsoNNfRbA9J9QU1H4VRU3C5xKlhtHVoaeAch/H9w+8wtdlV0sD2YbuDsC39CUeKXokxFFfW\n8XKOf+TzyHYGhcA+hcBZvrX1IYLC7hKOHpDSaETW49dkR2Q/hs7W0vIoS7YX0S8l3ln5tznxMU1r\nCp9tLmDSoF4tZviB/W6REuMK3dG8v7SaTzYe4MN1+zl2SGrItbsO1VnWEvUDeydyzwXjW11mpitF\n9YJ4vtaaj6z/q2q9uF0SclmM5gSuetrWh8x+aMG/cmh72TWO4Cn4AL07cbxzpPVKjGGPtTfEqaMz\nW50J3JzAPoUt+xtWsg0ez15SWcfOg5UcPcDfd2EXHgKXCT+SNbc8ijGGJdsOcmxWaqszauODhqQW\nV9ayKq+4XcufdLQYT8M8hY37ypydFOc8u5RfvLaadXtLmRVGH0w4egcMSb96+pB2P6udIbprCtb/\nwUHBjgqBM5oTY9xhTTF3NwoKbWs+8gQEnfauGQ8NoyzstfabS9eRrldCLLsO+gPf+ZMHtHsJAE9A\n89GOg5X+UTjFVU2aUjZbm8eM7udvOpp73TTeWL67Ucf3kSy2mc2Z8oqq2FdazbFZoUccBfJ3NDfU\nFBblFmJM+9bE6mh28Ld3cXttWR7fOy6LlQErj84a3bE7qS247cROnXjWEaI7KFiZfkLQTOWGPgX/\nCVV1XuLD7PR1W9eI9bja3E4aGBQOpTnCbj7KL+ve2zYGrkU16BCaxdwBHc07CisY1z+F3cVVjUrN\nheU1bDngr0VkWePKZ45IZ2YnzTvoDM1tzrRku3+dqeys1FavkRjrbjRPYfnOYmLdrrD7NyIhJmCV\n1EXWasH239R2VAetyGrrjH3AO1pUB4XTx/bhhcU7ndU37YJm8Oij6jpvWP0J0JDBZybHtbkEG9h8\nlBjX/vHKdkdzR+xHezgLnJTV1m0ZQ7E7BEuq6igor2VkZg8+3ZTvZJB1Xh/H3LsA8AeQloYVH8li\nPQ3LowQ+s8t2FpEc52nT8iiJsf51xHzW5KxlO4sY079nyKXQO5vdrLqtoMKZ3/LeGv9SKUcPSOGs\no/t1yoJzh7uoDgq/P38cPzl1pFNlth+HwP0ULnr4S5btLA67BGGXPsNZLC6wphDOjOJg9sNf0MIO\nZd1B4Ezh9i4JAg3NR7nWznh2h3Wt1SEZuJVm/17xIffh6A7s56bW62uUiW/YW8aYfj3b1PRo78K3\ndGcRlz7q3xs71D4WXcH+fNkjqQDeW72PXokxvPmjmd2qafVQdM+nu41i3C76psQ7HVA2+9kwGJZZ\nwxPjYsL7VdnNR3FhZCCBSzu3ZfG91q5TWNG4pnDOhEPbzvBwYzcfJcWG198TzM4Mtub7976wlzmv\nrffx0fr9vBmwTWVWG9aNOlKFWhvI5zNs2Ffm9KO0xn5uc7YXOccmDur6piNoqImvzCtxjuUWVJA9\nJFUDQoCorinY7A+DnbE0rJLacI4nzIfGbT2A4ZQqA3/GodQUBqUmEOMW6rwGl/jvY3TfZP55eegt\nRI9Udt/Poc4UtoOovYf2iAx/Bljn9XHd0zmNzh0cwf0iupqzkGK9D6yxEbuLqyivqQ+56F4o9nNr\n90MATBrUegd1Z4gJ+CyOzOzBZqs/YfLgyO5kdqSJ6pqCzX5YGrLkxstcAGGvWmhXVcMJCoGlleY2\nhGmL5PgYZ+ak3XwV53F1u/bSE0amc/nUQS0uu9AWTk3hQDlpSbGkJMY0u1lMd64pxLgbRufY1u/1\nbyE7Jsyagh0U+qfEk5V2eATSwELXCSMbRhlFenvLI40GBQJrCjT6P3CKSzhzFKChFBsbRubekZm2\nvfuYvdn9dSc03b/hSJcU5+GPF01o9xpRNjuzqKj1MtCqCTS3Ymhrk7eOZI1qCpYN+8oQoU2dzNDQ\np1BWXc/lUwfz1V2nHjaFkcCC1swR/kKTyKHNCeqONCjgz7jdLuHX5/h3oXKFiAoeV5g1BVf4NYWO\ndKq1Rk1WWhLb/3Q253XQ1P3uKLCGlmktSRLjdrHd2l870KEGoMNZqFVE1+8tZUhqYotrdgUK7Asb\nGWIL2q4U+HeePLg3SbFuRmb26LQdzY4U2qeAf0mKrX84y3ltPzq+gOajcGsKbicodM1QvAG9Evjl\nWaOZ1o5NZ6JNYAnSXqcq1u1yVu4EeObaqazfW9rsktHdgT0oIrD5aMuB8kYb1rcmsC8s1L7kh4vU\npFiy0pOaXe48mmlQCCF0R3OYo4+soBBmLOlQc05suk2jaiqwBGnvYRHjdlFYUY0IrPntt0iK87S4\nF0Z34AxJtZqPfD7/DO9wln5ICqwptHMtqs7y2k3H6aijEDQohGA3H3kDVlP0tLOm4A4zmKjOF9gB\nadcU7GHKwzN6tLnp5EgX3NG8v6ya2npfWCOu7EmXPeI87d5jPJJ+dfYYZ7TR4bChzeEoOp72dgoc\nrx3uaCCPExQ6NEkqAgIDfkZAnwLAxHZu3HMksvu/aqyawvYCf/NZOCOu7OajEZk9DpsO5kA/6IYD\nLjqaZlkh2M9y4Ebu4c9T8J/vascH4zD8LHVrgU2DdlCorPEPR23rUMzuICZo8trOg/6O9iFhDCmN\n97gROfw6mVXbaU0hBKHxTlwQ/jyFhuaj8HL4j392Ej2CV21VERWqT6HE2oGsrRskdQdOR7NVU9hR\nWInHJWGtAutyCT86eUSLmw6pw5vmPiHYBcfAvVzDHX1kCzcoDDuM11nvrkL1KdgrfWZ04yGowQLX\nPgJ/UBiUmhh2gejn3zqqw9OmOo82H4Vg1xQCh+aFO/rIbnpqT/OR6lyBu6gFdz6mR1FNwS742M/9\njoMV3XpZDxWaBoUQ7Hz812+tdY6FW1Owg0K4fRHq8JKWdGRtkHIoAjuajTHsKKwMqz9BdQ8aFEII\nlY+HOyTVa01803HQR7butH1pa2IDhqSW1dRTVl3PwG60p7dqGw0KITXNyMNtPvLZzUcaFI5o0fT3\niw3oaN5f4t9DIhKb2KvDm3Y0hxCqGyDc5qN6bT464lw4eYDz9W/OHcvaPaVdmJrOF9jRvK/UHxQO\nxwloKrI0KIQQKhsPdwSGTzuajyjb/3R2o6XSvz9zaBempms4NQWvYZ9VU+iXos1H0Uabj0IIlZGH\nW+LXPoUjz+E4A7cz2c94Tb2P/VZNIbNn9Iy+Un5aUwghdPNRuENS/f9rUFBHChEh1u2izuvjYEUd\nvRNjdH2gKBTRmoKIzBaRjSKyRUTubOacb4vIOhFZKyLPRzI9bSWhOprD7FPwaU1BHYFiPS5q633s\nK6nWTuYoFbGgICJu4CHgTGAscLmIjA06ZyRwFzDTGDMOuDVS6QlHqJqCO8ymhUuPGUis28XZR/fr\noFQpFXn+vb39Hc19w1jeQnUfLTYfichtQYcMUAB8YYzZ1sq1pwJbjDG51rVeBM4H1gWccz3wkDGm\nCMAYcyCMtEdMqPw/3ObmkX2S2XTfmR2TIKU6SazH33y0r6SG8f11m8po1FpNITnoX08gG3hPRC5r\n5XsHALsCXudZxwKNAkaJyJciskhEZoe6kIjMEZEcEcnJz89v5cceumjvcFTRK8btoqLGS2FFjTYf\nRakWawrGmN+FOi4iqcAC4MUWvj1UzmqCXnuAkcDJwEDgcxEZb4wpDkrHY8BjANnZ2cHX6HAaElS0\ninW72F1chTFo81GUalefgjHmIK3nnXnAoIDXA4E9Ic55yxhTZzVHbcQfJLqUzi1Q0SrW4yKvyL+5\nTh8djhqV2hUUROQUoKiV05YAI0VkqIjEApcB84LOeROYZV0zHX9zUm570tSRNCaoaBXjdnGgrAaA\n9ChaNlw1aK2jeTVNm3xS8Zf4r2npe40x9SJyMzAfcANPGmPWisjvgRxjzDzrvTNEZB3gBW43xhS2\n71Y6jsYEFa1iPS7sid3RtBigatDa5LVzgl4boNAYU9GWixtj3gXeDTp2d8DXBrjN+nfY0I5mFa0C\n1/hK66FBIRq11tG8o7MScjjRmKCiVazHP4M5PsZFYqwueBCNdO2jEDQmqGgVa9UUUrXpKGppUAgh\nuPloeEYS507s30WpUarz2Gt8pWrTUdTSoBBC8HJFj1x1jFalVVSwl89OTdKRR9FKg0IIwQvi6bwF\nFS2cmkJiTBenRHUVDQqhBMUAXelURQutKSgNCiEExwDdUlNFi1irpqDDUaOXBoUQgjuao2nzdhXd\n7JqCTlyLXhoUQggOAeHupaDUkcqevJaapEEhWmlQCCE4Brj0t6SiRIw2H0U9ze5CCB5t5NGooKKE\nNh8pze3aQJuPVLRIjvPgEsjQFVKjls7ICkGbj1S0unDKQEb360mKzlOIWprdhaDNRypa9YjzcGxW\nalcnQ3Uhze1CCG4s0piglIoWmt2FEDxPQfsUlFLRQoNCCE3mKejkNaVUlNCgEEJwxUB3YlNKRQsN\nCiFoEFBKRSsNCq14/vppXZ0EpZTqNBoUWjEsvUdXJ0EppTqNBoVWaB+zUiqaaFBohfYvKKWiiQaF\nVmhNQSkVTTQotEL3Z1ZKRRMNCq3QoKCUiiYaFFqjMUEpFUUiGhREZLaIbBSRLSJyZwvnXSIiRkSy\nI5me9tA+BaVUNIlYUBARN/AQcCYwFrhcRMaGOC8Z+DHwTaTScii0+UgpFU0iWVOYCmwxxuQaY2qB\nF4HzQ5x3D/BnoDqCaWk3DQpKqWgSyaAwANgV8DrPOuYQkcnAIGPMOy1dSETmiEiOiOTk5+d3fEpb\n/Nmd+uOUUqpLRTIohMpOjfOmiAv4X+BnrV3IGPOYMSbbGJOdkZHRgUlsndYUlFLRJJJBIQ8YFPB6\nILAn4HUyMB74RES2A9OBeYdbZ7N2NCulokkkg8ISYKSIDBWRWOAyYJ79pjGmxBiTbozJMsZkAYuA\n84wxORFMU9h0mQulVDSJWFAwxtQDNwPzgfXAy8aYtSLyexE5L1I/t6NpTUEpFU08kby4MeZd4N2g\nY3c3c+7JkUxLe2lNQSkVTXRGs1JKKYcGBaWUUg4NCkoppRwaFJRSSjk0KCillHJoUFBKKeXQoKCU\nUsqhQUEppZRDg4JSSimHBgWllFIODQpKKaUcGhSUUko5NCgopZRyaFBQSinl0KCglFLKoUFBKaWU\nQ4OCUkophwYFpZRSDg0KSimlHBoUlFJKOTQoKKWUcmhQUEop5dCgoJRSyqFBQSmllEODglJKKYcG\nBaWUUg4NCkoppRwRDQoiMltENorIFhG5M8T7t4nIOhFZJSIficiQSKZHKaVUyyIWFETEDTwEnAmM\nBS4XkbFBpy0Hso0xE4BXgT9HKj1KKaVaF8mawlRgizEm1xhTC7wInB94gjFmoTGm0nq5CBgYwfQo\npZRqRSSDwgBgV8DrPOtYc64D3otgepRSSrXCE8FrS4hjJuSJIlcB2cBJzbw/B5gDMHjw4I5Kn1JK\nqSCRrCnkAYMCXg8E9gSfJCKnAf8DnGeMqQl1IWPMY8aYbGNMdkZGRkQSGywjOY7UpNhO+VlKKXW4\niGRNYQkwUkSGAruBy4ArAk8QkcnAv4HZxpgDEUxL2L6569SuToJSSnW6iNUUjDH1wM3AfGA98LIx\nZq2I/F5EzrNOewDoAbwiIitEZF6k0hMul0twuUK1gCmlVPcVyZoCxph3gXeDjt0d8PVpkfz5Siml\nwqMzmpVSSjk0KCillHJoUFBKKeXQoKCUUsqhQUEppZRDg4JSSimHBgWllFIODQpKKaUcEZ28ppRS\nXaWuro68vDyqq6u7OimdKj4+noEDBxITE9Ou79egoJTqlvLy8khOTiYrKwuR6FiyxhhDYWEheXl5\nDB06tF3X0OYjpVS3VF1dTVpaWtQEBAARIS0t7ZBqRxoUlFLdVjQFBNuh3rMGBaWUUg4NCkopFSFV\nVVWcdNJJeL1eVqxYwYwZMxg3bhwTJkzgpZdeavX7H3zwQcaOHcuECRM49dRT2bFjBwD5+fnMnj07\nImnWoKCUUhHy5JNPctFFF+F2u0lMTOSZZ55h7dq1vP/++9x6660UFxe3+P2TJ08mJyeHVatWcckl\nl3DHHXcAkJGRQb9+/fjyyy87PM06+kgp1e397u21rNtT2qHXHNu/J785d1yL5zz33HM8//zzAIwa\nNco53r9/fzIzM8nPz6dXr17Nfv+sWbOcr6dPn87cuXOd1xdccAHPPfccM2fObO8thKQ1BaWUioDa\n2lpyc3PJyspq8t7ixYupra1l+PDhbb7eE088wZlnnum8zs7O5vPPP++IpDaiNQWlVLfXWok+EgoK\nCkLWAvbu3cvVV1/N008/jcvVtnL53LlzycnJ4dNPP3WOZWZmsmfPng5Lr02DglJKRUBCQkKT+QKl\npaWcffbZ3HvvvUyfPr1N11mwYAH33Xcfn376KXFxcc7x6upqEhISOjTNoM1HSikVEb1798br9TqB\noba2lgsvvJBrrrmGSy+9tNG5d911F2+88UaTayxfvpwbbriBefPmkZmZ2ei9TZs2MX78+A5PtwYF\npZSKkDPOOIMvvvgCgJdffpnPPvuMp556ikmTJjFp0iRWrFgBwOrVq+nbt2+T77/99tspLy/n0ksv\nZdKkSZx33nnOewsXLuTss8/u8DRr85FSSkXIzTffzIMPPshpp53GVVddxVVXXRXyvLq6OmbMmNHk\n+IIFC5q99rx583jrrbc6LK02rSkopVSETJ48mVmzZuH1els8b/78+WFdNz8/n9tuu43evXsfSvJC\n0pqCUkpF0LXXXtvh18zIyOCCCy7o8OuC1hSUUt2YMaark9DpDvWeNSgopbql+Ph4CgsLoyow2Psp\nxMfHt/sa2nyklOqWBg4cSF5eHvn5+V2dlE5l77zWXhoUlFLdUkxMTLt3H4tmEW0+EpHZIrJRRLaI\nyJ0h3o8TkZes978RkaxIpkcppVTLIhYURMQNPAScCYwFLheRsUGnXQcUGWNGAP8L3B+p9CillGpd\nJGsKU4EtxphcY0wt8CJwftA55wNPW1+/Cpwq0bh/nlJKHSYi2acwANgV8DoPmNbcOcaYehEpAdKA\ngsCTRGQOMMd6WS4iG9uZpvTga0cBvefooPccHQ7lnoe05aRIBoVQJf7gsWFtOQdjzGPAY4ecIJEc\nY0z2oV7nSKL3HB30nqNDZ9xzJJuP8oBBAa8HAsGLfzvniIgHSAEORjBNSimlWhDJoLAEGCkiQ0Uk\nFrgMmBd0zjzgu9bXlwAfm2iaaaKUUoeZiDUfWX0ENwPzATfwpDFmrYj8HsgxxswDngCeFZEt+GsI\nl0UqPZZDboI6Auk9Rwe95+gQ8XsWLZgrpZSy6dpHSimlHBoUlFJKOaIiKLS23MaRSkSeFJEDIrIm\n4FiqiHwoIput/3tbx0VE/mH9DlaJyJSuS3n7icggEVkoIutFZK2I/MQ63m3vW0TiRWSxiKy07vl3\n1vGh1vIwm63lYmKt491m+RgRcYvIchF5x3rdre9ZRLaLyGoRWSEiOdaxTn22u31QaONyG0eqp4DZ\nQcfuBD4yxowEPrJeg//+R1r/5gCPdFIaO1o98DNjzBhgOvAj6+/Zne+7BjjFGDMRmATMFpHp+JeF\n+V/rnovwLxsD3Wv5mJ8A6wNeR8M9zzLGTAqYj9C5z7Yxplv/A2YA8wNe3wXc1dXp6sD7ywLWBLze\nCPSzvu4HbLS+/jdweajzjuR/wFvA6dFy30AisAz/6gAFgMc67jzn+Ef8zbC+9ljnSVenvR33OhB/\nJngK8A7+ya7d/Z63A+lBxzr12e72NQVCL7cxoIvS0hn6GGP2Alj/Z1rHu93vwWoimAx8Qze/b6sZ\nZQVwAPgQ2AoUG2PqrVMC76vR8jGAvXzMkeZvwB2Az3qdRve/ZwN8ICJLreV9oJOf7WjYT6FNS2lE\ngW71exCRHsBrwK3GmNIW1lHsFvdtjPECk0SkF/AGMCbUadb/R/w9i8g5wAFjzFIROdk+HOLUbnPP\nlpnGmD0ikgl8KCIbWjg3IvccDTWFtiy30Z3sF5F+ANb/B6zj3eb3ICIx+APCc8aY163D3f6+AYwx\nxcAn+PtTelnLw0Dj++oOy8fMBM4Tke34V1g+BX/NoTvfM8aYPdb/B/AH/6l08rMdDUGhLcttdCeB\nS4d8F3+bu338GmvEwnSgxK6SHknEXyV4AlhvjHkw4K1ue98ikmHVEBCRBOA0/J2vC/EvDwNN7/mI\nXj7GGHOXMWagMSYL/2f2Y2PMlXTjexaRJBFJtr8GzgDW0NnPdld3rHRS581ZwCb87bD/09Xp6cD7\negHYC9ThLzVch78d9SNgs/V/qnWu4B+FtRVYDWR3dfrbec/H468irwJWWP/O6s73DUwAllv3vAa4\n2zo+DFgMbAFeAeKs4/HW6y3W+8O6+h4O8f5PBt7p7vds3dtK699aO6/q7Gdbl7lQSinliIbmI6WU\nUm2kQUEppZRDg4JSSimHBgWllFIODQpKKaUcGhRU1BGRcuv/LBG5ooOv/cug11915PWVijQNCiqa\nZQFhBQVr1d2WNAoKxpjjwkyTUl1Kg4KKZn8CTrDWrv+ptejcAyKyxFqf/gYAETlZ/Hs4PI9/khAi\n8qa1aNlae+EyEfkTkGBd7znrmF0rEevaa6z18r8TcO1PRORVEdkgIs9Zs7YRkT+JyDorLX/p9N+O\nikrRsCCeUs25E/i5MeYcACtzLzHGHCsiccCXIvKBde5UYLwxZpv1+lpjzEFr2YklIvKaMeZOEbnZ\nGDMpxM+6CP9eCBOBdOt7PrPemwyMw79uzZfATBFZB1wIjDbGGHuZC6UiTWsKSjU4A/9aMivwL8ed\nhn8DE4DFAQEB4McishJYhH9RspG07HjgBWOM1xizH/gUODbg2nnGGB/+ZTuygFKgGnhcRC4CKg/5\n7pRqAw0KSjUQ4Bbj3/VqkjFmqDHGrilUOCf5l3I+Df+mLhPxr0sU34ZrN6cm4Gsv/k1k6vHXTl4D\nLgDeD+tOlGonDQoqmpUByQGv5wM3WUtzIyKjrNUqg6Xg3/qxUkRG41/G2lZnf3+Qz4DvWP0WGcCJ\n+BduC8naLyLFGPMucCv+pielIk77FFQ0WwXUW81ATwF/x990s8zq7M3HX0oP9j5wo4iswr8F4qKA\n9x4DVonIMuNf6tn2Bv7tI1fiX+X1DmPMPiuohJIMvCUi8fhrGT9t3y0qFR5dJVUppZRDm4+UUko5\nNCgopZRyaFBQSinl0KCglFLKoUFBKaWUQ4OCUkophwYFpZRSjv8HCYQC9uLbcJsAAAAASUVORK5C\nYII=\n", - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "agent = PassiveTDAgent(policy, sequential_decision_environment, alpha=lambda n: 60./(59+n))\n", "graph_utility_estimates(agent, sequential_decision_environment, 500, [(2,2)])" @@ -276,25 +429,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "It is also possible to plot multiple states on the same plot." + "It is also possible to plot multiple states on the same plot. As expected, the utility of the finite state $(3,2)$ stays constant and is equal to $R((3,2)) = 1$." ] }, { "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xd8VfX9x/HXJ3tAAoQwwybsESAo\niAsVxL1nXW0VbdUOq1at1dYOaWut2vqzUkVpRcUttSoqoiLICAhhb0LCTBghZI/v7497c8giA3KJ\nhPfz8cgj957zved+z83NeZ/v95zzPeacQ0REBCCoqSsgIiLfHQoFERHxKBRERMSjUBAREY9CQURE\nPAoFERHxBCwUzGyKme02sxWHmf89M0v1/8wzs6GBqouIiNRPIFsKLwMTapm/GTjDOTcE+B0wOYB1\nERGReggJ1IKdc1+ZWfda5s+r8HQ+kBCouoiISP0ELBQa6IfAR4ebaWYTgYkA0dHRI/r163es6iUi\n0iwsXrw4yzkXX1e5Jg8FMxuLLxROPVwZ59xk/N1LycnJLiUl5RjVTkSkeTCztPqUa9JQMLMhwAvA\nec65PU1ZFxERacJTUs2sK/AOcKNzbl1T1UNERA4JWEvBzF4DzgTamlkG8CgQCuCc+yfwCBAH/J+Z\nAZQ455IDVR8REalbIM8+uq6O+bcCtwbq/UVEpOF0RbOIiHgUCiIi4lEoiIiIR6EgIiIehYKIiHgU\nCiIi4lEoiIiIR6EgIiIehYKIiHgUCiIi4lEoiIiIR6EgIiIehYKIiHgUCiIi4lEoiIiIR6EgIiIe\nhYKIiHgUCiIi4lEoiIiIR6EgIiIehYKIiHgUCiIi4lEoiIiIR6EgIiIehYKIiHgUCiIi4lEoiIiI\nJ2ChYGZTzGy3ma04zHwzs2fMbIOZpZrZ8EDVRURE6ieQLYWXgQm1zD8PSPT/TASeC2BdRESkHgIW\nCs65r4C9tRS5BPi385kPtDKzjoGqj4iI1C2kCd+7M5Be4XmGf9qOgLzbRw/AzuUBWbSIyDHRYTCc\nNymgb9GUB5qthmmuxoJmE80sxcxSMjMzA1wtEZETV1O2FDKALhWeJwDbayronJsMTAZITk6uMTjq\nFOB0FRFpDpqypTADuMl/FtIoINs5F5iuIxERqZeAtRTM7DXgTKCtmWUAjwKhAM65fwIfAucDG4A8\n4PuBqouIiNRPwELBOXddHfMdcGeg3l9ERBpOVzSLiIhHoSAiIh6FgoiIeBQKIiLiUSiIiIhHoSAi\nIh6FgoiIeBQKIiLiUSiIiIhHoSAiIh6FgoiIeBQKIiLiUSiIiIhHoSAiIh6FgoiIeBQKIiLiUSiI\niIhHoSAiIh6FgoiIeBQKIiLiUSiIiIhHoSCNpqikjM1ZuU1dDRE5CiFNXQE5/q3ecYBpC9KYsXQ7\nBwpK+ODuUxnUOfaw5bftz2dfblGtZU4UzjmWpu/ng9QdtIkO486xvRu8jIOFJaTtyWVgp/p/ngXF\npeQXldI6OqzB7yfNm0IB2JdbxNRvtnBJUmd6tI1u6uqQmVNIYUkpCa2jAv5ehSWlzFmXxRl94wkN\nbljDccGmPfz103Us3LyX8JAgxvZtx8crd7IsY3+NG/z1u3J4etZ6Plqxk+AgY9kj44kMCyY1Yz/P\nzFpPm+gw/nzl0MZatSa3JSuXg4UlNX4WeUUlvLEondcWprN2Vw4A0WHBDQqFFduymbYgjbcXb6Oo\ntIyxfeN56tphxEaGAr6wCAsOIizk0N919Y4D/PubLXywbAdhIUHMf+jsBv/dJTD25hYRHR5MeEhw\nk9bjhA+Fb7fu48fTlrAju4CSUse95/Ztsro453gjJZ3ffbCazq0imfnz0wP6fnM3ZPHweyvYnJXL\nveP7cNdZifV63a4DBTz83go+XbWL9jHh/Or8/lyVnEBsZCiDHp3JrNW7iQ4L4dJhnQHIzitm0sdr\nmL5oK9FhIZye2JbZazP5Yu1uPli+g/+l7vCW/cB5/Zny9WaKy8p48Lz+3vTt+/Mxg46xkY37IRyB\nxWn7+Osna+nXIYZHLhpQbX763jyenrWed5ZkEBUWwtJHxhHi3/DmF5Xyyvw0/vnlRvbkFjEkIZbH\nLx/Mtn35/GP2BnILS4gOP/RvuTe3iJfnbmZ4t9bsOlDA8m3ZXDYsgWdmrefLdZlEhAZxwZCOvPvt\nNmavzWTagjSuGtGFZ2atZ3pKOlcM78zjlw8hZcte/v75Br5cl0lkaDCDO8eycMtevt26n5N6tDlm\nn93xKn1vHm2iwyr9bcDXZRpkUOZgZ3YBXeMatiNXVuaYtWY3/5mfxpz1mUw8vWel731TOGFCwTnH\nPz7fQFR4CD88tQcA736bwf1vpdI+JgKAotKyJqtfYUkp97+VyvtLtxMWEsSunIKAvVdpmePJT9fy\n7OyNdI+LYkhCLFO/Sas1FAqKS4kIDeaD1O089M5yikrLuH9CX34wpgcRoYf2bPp0aMnna3bz+Zrd\njOjWmrQ9edz75jIyDxZy0+ju/PTsRIpLyzjpj7P40bQlhIcE8ZOzE0ls14K7X/uWcU9+yZ7cIgDu\nHd+XkCBj6rwt/Oa/q+jfMYaPfnpavdbROYeZHd0HVUV2fjGTPlrNawvTAV9LoGIoFJWU8fyXG/n7\n5xvAYFTPOOZt3MPkOZv4ZOUuxg9sz7T5W9m2P5/TEtvys3MSGdHNt0F+e3EG4GslRoeHUFrmeHXh\nVp6YuZbs/OJK9Xhl/lZaRYXywHn9uG5kV2KjQnn0ogGc9dcveeWbNP5v9kYKiktp1zKcmSt3kVOw\nhA9Sd9C2RRj3ju/DDaO6YWYMe+wTPlqxo1oobNh9kNjIUOJbhjfaZ1dW5igqLav0XTnWysocQUEN\n+05s2J3DX2auZebKXVx3Uhcev3wI4Av3F7/exAtfb6Zzq0j25xWzIzufz+45g25x0RjU+l6lZY73\nvt3Gc19uZMPug3SMjaBleAhpWXlHs4qN4oQJhemL0vnrp+sA+OGpPXgzJZ37305lVI84nrthOGOf\n+IKC4tImqduBgmJunZrCws17uXd8Hw4WlvLCnE0B2bDlF5VyxyuL+XJdJteO7MJvLh7IK/PT+P3/\nVrMvt6jGPuYPl+/gnjeWMqhTLClp+xjetRV/vTqpxq62m0d3p13LHcxcuYtH3l/BF+sy6dk2msk3\nncKQhFZeuSEJsUSEBDPpisH0jG/B/rwiwoKDiAgN5qoRCby5OINV2w/w7OwNfLJqF+Dr+sgvKiUy\nrPYNy0tzN/Pb/67i1VtP5pTebY/q89qfV8SbKRn06dCSB99OZeeBAiae3pMgM/755UYOFBSzevsB\n2sdEcOerS1i5/QAXDunIwxcMoKSsjFP/NJs/f7wWM1iavp/e7Vrw2m2jGN0rrtL7tIvxbYB35xQS\nFhLEz6cvZcHmvYzuGcf3x3TnjZR0rhiewOY9uRSXOL5/andiIkK917eKCuOqEQk8/9UmzunfnofO\n78fS9P3c88YyPl21i5+encgdZ/Sq9NldMTyBqfO2cNPo7vRoG01BcSl/+ngNL8/bwumJ8fxyQj/m\nrM/k1tN6ElzDBi47r5iYyJA6v6Nrd+Zw/9uppO/NY94DZwU0GIpLy0jNyGZ411ZevXILS/jbp+uY\n+s0Wpn7/pHp9J3yt29VMX5ROVJhvM/nh8p3szC6gdXQY32zcw47sAkZ2b01K2j4GdYpl2/58Hnl/\nJUvT93PraT342Tl9vGU9NWsdn6zcxaiecZzepy3PfbGRNTtz6NehJU9fm8QFgzty/QsL2JtXFLDP\npr5OmFC4fHgCT89az+6cQmav3c0D7yxnTK+2vHBzMhGhwUSGBpNfdOxDoaC4lNumpvDt1n08c90w\nLh7aiee+2EhJmaOguKzODWBD5BaW8MOpi1iweS9/vGww15/cFcDbuG/KymVElVAoD8+IkGBS0vZx\ndXICv790cKV+6oouHdaZC4Z0ZNCjM5m9NpPzBnXgyauTqq3H+3eOqbQxaRUVxoc/PY0OsRFsyjzI\nm4szuPmlheQUlPDwBf3p2iaKif9ZzKod2d7edVVlZY4/fbyG57/aBMCyjOyjCoWNmQf5/kuL2LrX\nt/fWuVUk7/x4DEldWvHfZdsBuOzZuWzM9J1xFRsZyvM3juDcgR0AX2vlyhEJdGoVyQWDO5KStpcr\nRyTU2Gdcvlf+32XbmbFsO8WlZfz5iiFclZyAmTHev8za/PScRC4fnkDfDi0B6NQqkr25RUwY1KHG\n41M/G9eHNxdn8LPXvyVjXz5BQUZmTiHd46KYuyGLS5+dS1FpGYM7x1b6HEvLHM/O3sBTn63jsUsG\nccOobt688h2ZrXt8XWihwcY7327DOUdxqWPdrpxKOwcVFRSXEh4SdMQ7Qut35fCz6UtZuf0Ar952\nMqf0asuSrfv4+fSlbN2bhwFfrMus8zvxycqd/Oq9FezNLeLmU7pz91mJfLZ6F/e/lcrstZkADOoc\nw9PXDuOkHm3Yl1tEq6hQzv7rl3y9IYuI0CBmLNvO7af34ptNWTzw9nJ25xQC8PaSDN5ekkFC60ie\nvX445w/u4K1vm6gwNmUdPKJ1b0wBDQUzmwA8DQQDLzjnJlWZ3xWYCrTyl3nAOfdhIOoSFhLEPeP6\ncN9bqdzxn8X0jm/BczcM9/ZaIkKDyT/GLYWyMsc9b/j2CJ++NomLh3YCoEWE78+SU1jcaKFQUlrG\nHa8sZuHmvTx1TRKXJHX25pWHwhXPzWPVY+d6e0b/S93BL99O5dTebfn7dcNYvSOHUT3b1PlPGxoc\nxJ1jexNk8OMze9fYjK5pGb3btQCgZ7zvd0FxKZNvHMHZ/duzbX8+AGt25tQYCs45HnxnOdNT0rlx\nVDfeSEkn62BhnZ+Lc46UtH0MTWhV7YDsDS8swAy+d3JXCkvKePiC/rSKCqv0maXvzScqLJgBHWP4\n2zVJdGlzaONrZjxx1aED5+Ub65q0a+nrwvzP/DQS27Vg8k3JDT7pISospNJ7RIQGc+tpPQ9bvnOr\nSPq0b8GyjGwAWoaH8NItI4mJDOWK5+ZxVr945m7IYsay7d6GNK+ohJ+9vpRPVu3CDB5+bwXR4cEM\n79qa2/+zmBHdWjNhUAd+PG0JOQUlAJzVrx0/OTuRS5+dy4ptB2oMhS/W7uaWlxZx37l9j+gMrPeX\nbuPBd5YT6f9/XpaezeIt+3hq1no6xEQwfeJoJn20msVp+0jbk0u3uOqfbWFJKb+ZsYrXFm6lf8cY\nXrplpHeSwGXDOhMV5lvPtbtyOCMx3vtel7eun7luGAfyi1mydR9PfLKO/o98DEDf9i2ZcstI2rYI\nZ/m2bLbvz+eakV2qtZhaR4eyN61yV2FTCFgomFkw8CwwDsgAFpnZDOfcqgrFHgbecM49Z2YDgA+B\n7oGqU/k/WXCQ8fyNI2hZofkdERpcrfuorMzxm/+u5LTEeMYNaN/o9Zk8ZxMfLt/Jr87vX2kjHVMe\nCgUltDv8dqRWzjkKSw714U76aA1z1mfxpysGV3ovoNKGbPWOA4zo1oYlW/fxs+nfMqJba56/cQRR\nYSHVujxq85Oz63fQuiYtwkN44qqh9GnfwtuAxPn/8fbnVf+ncc7x+EdrmJ6Szt1n9eaecX2Ysz7T\n2zurzVOfrefpWev585VDuDq5C+Dr7rjuX/OJDA1m2q0neyFVUb8OLZl4ek/GDWjPyO5Hf6C2VWQo\nbaLDGNAxhme/N9w7gyjQHr98CFkHC0ls14KYyFDatvC1WL7+5Vg6t4rkV++t4NUFW+nXoSWXJHXm\n5pcWsmJbNo9eNIDCkjImfbSG+99KJTYyjKyDhazZmcP0Ren0bteCRy8ayN7cIs4f7GvlxESE8NC7\ny3no3eX0io/m/743gr4dWvLawq08/N4KwBf6DeGc44lPfMfHTurehn9cP4zL/m8ez8xaT35xKZck\ndeJ3lw4iJiKUk3vG8dwXGznjL1/wxb1n0r1C6O7OKeC2fy9mWfp+7jijF78Y36fSWVmhwUFcOMS3\n09apVc0nO5QHSNe4KGat2c3mrFyuP6krPz0n0WsddoiNOOy6tI4KY39eUaVu4/S9eaSk7eWyYQkN\n+lyORiBbCicBG5xzmwDM7HXgEqBiKDggxv84FtgewPrQp0NLOreK5N5z+1T6QgBEhlVvKby2aCv/\n/iaNL9dlNnoopGbs54mZa7lgcEduPa1HpXkt/Gc4HPTvaR2Jm19axOodB1j40Nn8N3UHL3y9mZtH\nd+OakV2rlQ0NDuLJq4dyzxvLWLHtAIXFZdznPwD/wk0jvZbDsXTliMr/BBGhwYSHBFU76Aq+vevJ\nX23i5tHduGdcH8yM+Jbh7MouYHHaXoZ3bV1jy2T6oq08PWs9AMvS93N1chd25xTwg5cXERYcxBu3\nj64UmBWFBAfx0PmNd5ZIUJDx1f1jiQ4LbvTjSLUZ0a11jdPLu5t+e/FAdh8o4Pf/W83ri9LZlJXL\n5BuTOWdAe8rKHAXFpTz1ma+b6HeXDuLX761gZPc2PH/TiErHPADuO7cvv35/JQAbM3P53QerOL1P\nW/744RrO6BPPmp0HKHOu3nUvLXM85G8dXndSFx67ZBChwUEM7hzLzFU7eeC8ftx+ek/v87xrbG9m\nrtjJpqxc0vfleduArXvyuHHKAnYfKOSfNwxnwqCODf4cq3527/54TINf1zoqjBJ/19y/5mzmYGEJ\npWW+z2NYl9bVtlmBEsj/9s5AeoXnGcDJVcr8BvjEzO4GooFzalqQmU0EJgJ07Vp9o1ZfMRGhzH3g\nrBrnRYYGk1d0aCOcnVfMEzPXevMaU0lpGQ++s5w20WH88bLB1TYC5S2YnCMMhU9W7uSrdb6+z3W7\nDvLr91YwvGsrHr6w+umT5S4c0ol731zGozNWetPev3MMsVHHZo+1PmIjQ8mu0FIoLi1j1fYD/O6D\nVZzdrx2PXjTQ+yzjW4bz4fKdXPHcNzVeTLc8I5tfv7+SMb3jKC5xrNiWTWFJKbf9ezF7c4tqDYRA\naRH+3TvEFxocxC8n9OOz1bvZlJnLv25O5ow+8YAvyG47rSelZY5rT+pK51aRDOoUw4BOMTUeN7lx\ndHeuGdmVMuf46ydrefHrzXy9IYsLhnTkqWuSuOK5eeQW1u8775zjV+/6AuEnZ/Xm5/6dAYBHLx7A\nj8f2qtZNFR0ewou3jGTsE1+Q6W9FbsnK5ernv6GotIxXbzuZYV1rDsljobwb6olP1nnT4luGk5lT\nyOasXNbuyiG5W2viWjTeWWE1CeRVKzXt7lTdDbgOeNk5lwCcD/zHzKrVyTk32TmX7JxLjo+PD0BV\ny48pHDol9e+fr2d/fjGje8aRticP14A9mLq8Mj+NldsP8OhFA2vc6HothcKG9S8WlZRx85SFTPzP\nYm/aj6YtpqC4lCeuGlrrRUphIUF0bn2oWfzUNUkM7VLzAcGmEhsZ6rUUCopLufCZr7nk2bm0axnB\nX68eWunYRfkeFviucagoO7+YO15ZTNvoMJ65dhjDurZi9Y4c/vi/1SxL38/frklicIKuti6X2L4l\nv75wAC//YKQXCOWiw0P4xfi+dPZ3qQzr2rrWi6/CQnxnmJ3Rpx1lDsYNaM9T1yQRGhxEdFhIvVvH\nf/zQ13L5yVm9uWd830o7Vh1jIw97MLv8gH7WwUJ2Zhdww4sLKClzvHH76CYNBIAO/lPjr05OYO3v\nJ7D58fP52H8K9pS5m/nRK4t58tN1tS2iUQQyFDKALhWeJ1C9e+iHwBsAzrlvgAjg6M4hPEKRYcEU\n+ruPMnMK+c/8NK4YnsD5QzqSX1zKrgN1908fjnOOOeszKStz5BaW8PfPN3BKrzivr7Wqlv5jCj+a\ntqTaBq020xb4uroA/nyF73zqTZm53H1W7xr7xasa06stZ/drx6Y/nu9dePZd0irqUCj84/MN3pXA\n/7h+mHcAuNwNo7pxkr+vP7PKAedJH61mR3Y+z35vOHEtwhnVK46i0jKmfpPG9Sd3ZcKgus/0OdH8\n8NQenNKr8f41x/SO443bR/OP64d5OystIkI4WI+WwusLt/KvOZu55ZTu/Hxcnwa9b3RYMBGhQWzO\nyuPmKQvZn1fM1O+fRJ/2R3jwrhGN7hXH+3eO4U9XDCE8xNeN2CY6jJiIEOasz2J419b86oLAX9gW\nyFBYBCSaWQ8zCwOuBWZUKbMVOBvAzPrjC4XMANbpsCJDg8gvLmXD7hwmPPUVhSVl/PjMXnTyHxja\ndeDILyb715xN3PjiQj5csYOX5m5mT24R953b97B9x+Wh4Bw8/tGaer1Hdn4xT89az5jecSx++Bwu\nTuqEme8Mk9rOQKlo0hVDeOHm5AZf4HOsxEaGsiM7n1++lco/Zm/g8uGd2TLpghr38E5LjGfabb7e\nyqycQ+d+z9+0h9cWpnPbaT29153cow1hwUH0aBvNw8fgn058Z2ad1KNNpVZFy/CaQ2FHdj73TF/K\nnoOFLNm6j0feX8npfeL59YUDGnz8pfx40+uLtrJ+dw7/vGHEd6ZVGBxkDO3SqtI6mfmmDUmIZcr3\nj83xvYC9g3OuxMzuAmbiO910inNupZk9BqQ452YAvwD+ZWY/x9e1dItrzH6aBig/JfXeN1PZk1vE\nhUM60jO+BTuzfWFwpBe2lZU5pny9BYBV2w/w6sKtnN2vXa1N1ZYRoQzqHMOKbQfIqscZNAD/nreF\n/XnFPHhef6/P8d7xfUnu1rpBFwsdy4OcDRUTGcqWPXls2ZNHTEQID19w+GMk4OsPbx0VSuZB39+w\nqKSMh95ZTtc2Ud6FReA7lXPyTSPoHhfdJAfVxSe6Qihk5xfz/tJtjOoZxy/eWMbybdkM6hzLC3M2\n0SE2gmeuTarxgrr6iG8RTvrefB46vz+nJjZJx0SDvHjzSIIMb6iUQAvof4D/moMPq0x7pMLjVUDD\nD9MHQGRoMPvzilmat5+Lh3biD5cNAiDcv0EtKDmyITDmbsxip7+V8X9fbARg4um177kHBxkf3H0a\nN764gAP16GPNKyrhpXlbOKtfu0oHVI/kfO/vsvKLC8f0juMvVw6lTT1G+AwLCeKV+Vs5b1BHNmfl\nsikrl5duGVnt+o8z+7YLSJ2l/lpEhJBbWIJzjnumL2XWmt3ePDOY9PEaSsscb//olGrdhQ1x0dBO\nDOvautpZf99Vh7tQNFA0PKJf+d50cJDxm4sHemcARYT6PqIjbSlMnbeFti3CONV/8c+gzjH1HoAs\nKiyY/KK6Q+GtxRnszS3ix2f2OqI6Hi/KL8yadPmQw54rXlX5saCnZ63nmVnrGdm9NWf2DczJCnJ0\nWoSHUFzqmLlyV6VAmHT5YIZ1aUWRv0s36ShPgPj+mB5H1PV0olBb2a88jU/pFVdpD7Q8LI4kFHZk\n5zNrzW7uPLM3I/1BcPsZPev9ZYwKCyGvjqE3nHO8Mj+NIQmxJDfCRVTfZXeO7c31J3f1rv6tj6ev\nTeKnry9l4ea9APz9umHaGHxHlZ91d++by+gZH81/7zqVIDMiw4I56B899u56juQrR06h4Fd+ls9F\n/qsWy9UVCh+v2MG9b6Yy78Gzql2s8/7S7TjnuxCre9voaqfz1SUy7NB4TIcbHG9x2j7W7TrIpMsH\nN2jZx6PQ4KAGBQLAJUmdycwp5Pf/W80ZfeI5uWf9r8qWY+vQqdglTL5kRKVhqm89rWe9T5iQo6NQ\n8Lv99F5Eh4dw2fDKp2JGhJR3H9V8TOGOV5YAvlM/qzZr3/t2G8O6tjriKxGjw4K9lsKdry4hKiyk\n0lg6AK8tTKdFeAgXDe1U0yIEuOWU7ozqGfeduIGSHN7QLq0Y1DmGs/q1P+rRbeXIKRT8usZF1Ths\nQW0thclfbfQe78utPOTtmp0HWLMzh8cuGXjEdYoMCyG/uJTdOQV8vGIn/TrEVJpfUFzKzJU7OX9w\nh2o3/5BDQoKDdOvP40Dvdi344O763S9DAkcHmutwKBQqtxScc0xbsJWW/o1xZpVTRz9Z6RtF8ryj\nGEclyn+GzHvfbqPMwb4qY61/sTaTg4UlaiWISKNRKNQhOMgIDTYKSiq3FNbvPkjanjx+5r+i8h+z\nN1S68GbW6l0MTWh1VHevKg+FN1N8d+Xam1vEl+sySfeP7//f1O3ERYcxWv3kItJIFAr1EBFSfVjt\n8gHnzvMPibB1bx4vztkMwO4DBSzLyOac/kd37nv5QHzrdx8kJiKEwpIyfvDyIv41ZxMFxaXMXrOb\nCYM6HLOLWkSk+dPWpB7CQ4OrdR/NWZ9Fz/joSufLl1/T8P5S3xBP5xzlcNsVr6690N9FVFrm2Jtb\nxILNe8krKuXsowweEZGKFAr1EBEa5A2WB747NC3YvIfT/GdItPffX7fM+VoJf565hlN6xdH3KAfZ\niqpw1W3F01mz84uZvWY34SFBjO6pszREpPEoFOohIjS40jGFxVv2UVBcxmmJvg31V/ePBXz3QJ67\nMYviUsdD5/c/6oukwv0tj/Yx4bRtceiCuuz8Yj5fs5tTesU16j2cRUQUCvVQVFLGh8t3MnPlTsA3\n0maQwck9fVcQh4cEE+Mf9vebjXuIjQxlQMeY2hZZLy3DfRfD3TS6O51bRREcZMREhLBuVw5b9+Yx\ntp+6jkSkcSkU6mGr/2yfqfO2ALB46z76d4ypdI/nFv4RHhdt2cfI7m0aZfjpwQmxfPiT0/jxmb3o\nEBvB178cyyVJnb3jG405vr2ICCgUGqRbXBQlpWUs3bq/2r1tW0SEsHVPHpuzcknu3nh3cBrQKcbr\nhuoYG0kr/53a2rYIp1e8rtAVkcalUKiHZH8AFBaXsXZXDrlFpdVCITo8hIVbfIOuHe5m6I0hNtIX\nCqN6ttHAbiLS6BQK9fD6xFH0bd+SvXlFLEnbB8DwKjfJKR/MKzTYGBzAIRUOhYIuWBORxqdQqIeQ\n4CDax0awL7eIJVv3065lOAmtK4/nXx4KAzvFNuhOZw3Vt0NLWkaENHjEVRGR+tAoavXUJiqUzVkH\nyduWzeDOsdW6bor8d2ar7w10jtSQhFakPjpeXUciEhAKhXpqHR3G9v0FOOe8oS0q2pXju+XmlSMS\nAl4XBYKIBIpCoZ7iosMoLXMADOhU/ZjBX64cyqIte+lzlFcxi4g0JYVCPfWKb+E9Htip+oVp/TvG\n0L8RLlgTEWlKOtBcT/0qbPD8dr6JAAARh0lEQVSrHmQWEWkuFAr11LVNlPdYffoi0lyp+6iegoOM\n+87tS0/d51dEmjGFQgPcObZ3U1dBRCSg1H0kIiIehYKIiHgCGgpmNsHM1prZBjN74DBlrjazVWa2\n0sxeDWR9RESkdgE7pmBmwcCzwDggA1hkZjOcc6sqlEkEHgTGOOf2mZnuGiMi0oRqDQUzu6fKJAdk\nAV875zbXseyTgA3OuU3+Zb0OXAKsqlDmNuBZ59w+AOfc7gbUXUREGlld3Uctq/zEAMnAR2Z2bR2v\n7QykV3ie4Z9WUR+gj5nNNbP5ZjahpgWZ2UQzSzGzlMzMzDreVkREjlStLQXn3G9rmm5mbYDPgNdr\neXlNV3i5Gt4/ETgTSADmmNkg59z+KvWYDEwGSE5OrroMERFpJEd0oNk5t5eaN/oVZQBdKjxPALbX\nUOZ951yxvztqLb6QEBGRJnBEoWBmZwH76ii2CEg0sx5mFgZcC8yoUuY9YKx/mW3xdSdtOpI6iYjI\n0avrQPNyqnf5tMG3x39Tba91zpWY2V3ATCAYmOKcW2lmjwEpzrkZ/nnjzWwVUArc55zbc2SrIiIi\nR8ucO3wXvZl1qzLJAXucc7kBrVUtkpOTXUpKSlO9vYjIccnMFjvnkusqV9eB5rTGq5KIiHzXaZgL\nERHxKBRERMSjUBAREY9CQUREPAoFERHxKBRERMSjUBAREY9CQUREPAoFERHxKBRERMSjUBAREY9C\nQUREPAoFERHxKBRERMSjUBAREY9CQUREPAoFERHxKBRERMSjUBAREY9CQUREPAoFERHxKBRERMSj\nUBAREY9CQUREPAoFERHxKBRERMQT0FAwswlmttbMNpjZA7WUu9LMnJklB7I+IiJSu4CFgpkFA88C\n5wEDgOvMbEAN5VoCPwEWBKouIiJSP4FsKZwEbHDObXLOFQGvA5fUUO53wJ+BggDWRURE6iGQodAZ\nSK/wPMM/zWNmw4AuzrkPaluQmU00sxQzS8nMzGz8moqICBDYULAapjlvplkQ8DfgF3UtyDk32TmX\n7JxLjo+Pb8QqiohIRYEMhQygS4XnCcD2Cs9bAoOAL8xsCzAKmKGDzSIiTSeQobAISDSzHmYWBlwL\nzCif6ZzLds61dc51d851B+YDFzvnUgJYJxERqUXAQsE5VwLcBcwEVgNvOOdWmtljZnZxoN5XRESO\nXEggF+6c+xD4sMq0Rw5T9sxA1kVEROqmK5pFRMSjUBAREY9CQUREPAoFERHxKBRERMSjUBAREY9C\nQUREPAoFERHxKBRERMSjUBAREY9CQUREPAoFERHxKBRERMSjUBAREY9CQUREPAoFERHxKBRERMSj\nUBAREY9CQUREPAoFERHxKBRERMSjUBAREY9CQUREPAoFERHxKBRERMSjUBAREY9CQUREPAENBTOb\nYGZrzWyDmT1Qw/x7zGyVmaWa2Swz6xbI+oiISO0CFgpmFgw8C5wHDACuM7MBVYp9CyQ754YAbwF/\nDlR9RESkbiEBXPZJwAbn3CYAM3sduARYVV7AOTe7Qvn5wA0BrI+InECKi4vJyMigoKCgqatyTEVE\nRJCQkEBoaOgRvT6QodAZSK/wPAM4uZbyPwQ+CmB9ROQEkpGRQcuWLenevTtm1tTVOSacc+zZs4eM\njAx69OhxRMsI5DGFmv4KrsaCZjcAycBfDjN/opmlmFlKZmZmI1ZRRJqrgoIC4uLiTphAADAz4uLi\njqp1FMhQyAC6VHieAGyvWsjMzgF+BVzsnCusaUHOucnOuWTnXHJ8fHxAKisizc+JFAjljnadAxkK\ni4BEM+thZmHAtcCMigXMbBjwPL5A2B3AuoiISD0ELBSccyXAXcBMYDXwhnNupZk9ZmYX+4v9BWgB\nvGlmS81sxmEWJyJy3MnPz+eMM86gtLSUpUuXMnr0aAYOHMiQIUOYPn16na9/8sknGTBgAEOGDOHs\ns88mLS0NgMzMTCZMmBCQOgfyQDPOuQ+BD6tMe6TC43MC+f4iIk1pypQpXH755QQHBxMVFcW///1v\nEhMT2b59OyNGjODcc8+lVatWh339sGHDSElJISoqiueee47777+f6dOnEx8fT8eOHZk7dy5jxoxp\n1DoHNBRERL4LfvvflazafqBRlzmgUwyPXjSw1jLTpk3j1VdfBaBPnz7e9E6dOtGuXTsyMzNrDYWx\nY8d6j0eNGsUrr7ziPb/00kuZNm1ao4eChrkQEQmAoqIiNm3aRPfu3avNW7hwIUVFRfTq1avey3vx\nxRc577zzvOfJycnMmTOnMapaiVoKItLs1bVHHwhZWVk1tgJ27NjBjTfeyNSpUwkKqt9++SuvvEJK\nSgpffvmlN61du3Zs317thM6jplAQEQmAyMjIatcLHDhwgAsuuIDf//73jBo1ql7L+eyzz/jDH/7A\nl19+SXh4uDe9oKCAyMjIRq0zqPtIRCQgWrduTWlpqRcMRUVFXHbZZdx0001cddVVlco++OCDvPvu\nu9WW8e2333L77bczY8YM2rVrV2neunXrGDRoUKPXW6EgIhIg48eP5+uvvwbgjTfe4KuvvuLll18m\nKSmJpKQkli5dCsDy5cvp0KFDtdffd999HDx4kKuuuoqkpCQuvvhib97s2bO54IILGr3O6j4SEQmQ\nu+66iyeffJJzzjmHG264gRtuqHnMz+LiYkaPHl1t+meffXbYZc+YMYP333+/0epaTi0FEZEAGTZs\nGGPHjqW0tLTWcjNnzmzQcjMzM7nnnnto3br10VSvRmopiIgE0A9+8INGX2Z8fDyXXnppoy8X1FIQ\nEZEKFAoiIuJRKIiIiEehICIiHoWCiEiAVBw6Oy0tjREjRpCUlMTAgQP55z//Wefr77vvPvr168eQ\nIUO47LLL2L9/P+C7ruGWW24JSJ0VCiIiAVJx6OyOHTsyb948li5dyoIFC5g0aVKdYxeNGzeOFStW\nkJqaSp8+fXj88ccBGDx4MBkZGWzdurXR66xTUkWk+fvoAdi5vHGX2WEwnDep1iIVh84OCwvzphcW\nFlJWVlbnW4wfP957PGrUKN566y3v+UUXXcTrr7/O/fff39Ca10otBRGRAKhp6Oz09HSGDBlCly5d\n+OUvf0mnTp3qvbwpU6Zo6GwRkUZRxx59INQ0dHaXLl1ITU1l+/btXHrppVx55ZW0b9++zmX94Q9/\nICQkhO9973vetEANna2WgohIANQ0dHa5Tp06MXDgwHrt6U+dOpUPPviAadOmYWbedA2dLSJyHKk6\ndHZGRgb5+fkA7Nu3j7lz59K3b18AbrrpJhYuXFhtGR9//DF/+tOfmDFjBlFRUZXmaehsEZHjTMWh\ns1evXs3JJ5/M0KFDOeOMM7j33nsZPHgwAKmpqXTs2LHa6++66y5ycnIYN24cSUlJ3HHHHd48DZ0t\nInKcqTh09rhx40hNTa1W5sCBAyQmJtKlS5dq8zZs2FDjcgsLC0lJSeGpp55q9DqrpSAiEiD1GTo7\nJiaGN998s0HL3bp1K5MmTSIkpPH369VSEBEJoEAMnZ2YmEhiYmKjLxfUUhCRZsw519RVOOaOdp0V\nCiLSLEVERLBnz54TKhicc+zZs4eIiIgjXoa6j0SkWUpISCAjI4PMzMymrsoxFRERQUJCwhG/XqEg\nIs1SaGgoPXr0aOpqHHcC2n1kZhPMbK2ZbTCzB2qYH25m0/3zF5hZ90DWR0REahewUDCzYOBZ4Dxg\nAHCdmQ2oUuyHwD7nXG/gb8CfAlUfERGpWyBbCicBG5xzm5xzRcDrwCVVylwCTPU/fgs42yoO7iEi\nIsdUII8pdAbSKzzPAE4+XBnnXImZZQNxQFbFQmY2EZjof3rQzNYeYZ3aVl32CUDrfGLQOp8Yjmad\nu9WnUCBDoaY9/qrnhtWnDM65ycDko66QWYpzLvlol3M80TqfGLTOJ4Zjsc6B7D7KACoO5pEAVB38\n2ytjZiFALLA3gHUSEZFaBDIUFgGJZtbDzMKAa4EZVcrMAG72P74S+NydSFeaiIh8xwSs+8h/jOAu\nYCYQDExxzq00s8eAFOfcDOBF4D9mtgFfC+HaQNXH76i7oI5DWucTg9b5xBDwdTbtmIuISDmNfSQi\nIh6FgoiIeE6IUKhruI3jlZlNMbPdZraiwrQ2Zvapma33/27tn25m9oz/M0g1s+FNV/MjZ2ZdzGy2\nma02s5Vm9lP/9Ga73mYWYWYLzWyZf51/65/ewz88zHr/cDFh/unNZvgYMws2s2/N7AP/82a9zma2\nxcyWm9lSM0vxTzum3+1mHwr1HG7jePUyMKHKtAeAWc65RGCW/zn41j/R/zMReO4Y1bGxlQC/cM71\nB0YBd/r/ns15vQuBs5xzQ4EkYIKZjcI3LMzf/Ou8D9+wMdC8ho/5KbC6wvMTYZ3HOueSKlyPcGy/\n2865Zv0DjAZmVnj+IPBgU9erEdevO7CiwvO1QEf/447AWv/j54Hraip3PP8A7wPjTpT1BqKAJfhG\nB8gCQvzTve85vjP+Rvsfh/jLWVPX/QjWNQHfRvAs4AN8F7s293XeArStMu2YfrebfUuBmofb6NxE\ndTkW2jvndgD4f7fzT292n4O/i2AYsIBmvt7+bpSlwG7gU2AjsN85V+IvUnG9Kg0fA5QPH3O8eQq4\nHyjzP4+j+a+zAz4xs8X+4X3gGH+3T4T7KdRrKI0TQLP6HMysBfA28DPn3IFaxlFsFuvtnCsFksys\nFfAu0L+mYv7fx/06m9mFwG7n3GIzO7N8cg1Fm806+41xzm03s3bAp2a2ppayAVnnE6GlUJ/hNpqT\nXWbWEcD/e7d/erP5HMwsFF8gTHPOveOf3OzXG8A5tx/4At/xlFb+4WGg8no1h+FjxgAXm9kWfCMs\nn4Wv5dCc1xnn3Hb/7934wv8kjvF3+0QIhfoMt9GcVBw65GZ8fe7l02/yn7EwCsgub5IeT8zXJHgR\nWO2ce7LCrGa73mYW728hYGaRwDn4Dr7Oxjc8DFRf5+N6+Bjn3IPOuQTnXHd8/7OfO+e+RzNeZzOL\nNrOW5Y+B8cAKjvV3u6kPrByjgzfnA+vw9cP+qqnr04jr9RqwAyjGt9fwQ3z9qLOA9f7fbfxlDd9Z\nWBuB5UByU9f/CNf5VHxN5FRgqf/n/Oa83sAQ4Fv/Oq8AHvFP7wksBDYAbwLh/ukR/ucb/PN7NvU6\nHOX6nwl80NzX2b9uy/w/K8u3Vcf6u61hLkRExHMidB+JiEg9KRRERMSjUBAREY9CQUREPAoFERHx\nKBTkhGNmB/2/u5vZ9Y287IeqPJ/XmMsXCTSFgpzIugMNCgX/qLu1qRQKzrlTGlgnkSalUJAT2STg\nNP/Y9T/3Dzr3FzNb5B+f/nYAMzvTfPdweBXfRUKY2Xv+QctWlg9cZmaTgEj/8qb5p5W3Ssy/7BX+\n8fKvqbDsL8zsLTNbY2bT/FdtY2aTzGyVvy5PHPNPR05IJ8KAeCKH8wBwr3PuQgD/xj3bOTfSzMKB\nuWb2ib/sScAg59xm//MfOOf2+oedWGRmbzvnHjCzu5xzSTW81+X47oUwFGjrf81X/nnDgIH4xq2Z\nC4wxs1XAZUA/55wrH+ZCJNDUUhA5ZDy+sWSW4huOOw7fDUwAFlYIBICfmNkyYD6+QckSqd2pwGvO\nuVLn3C7gS2BkhWVnOOfK8A3b0R04ABQAL5jZ5UDeUa+dSD0oFEQOMeBu57vrVZJzrodzrrylkOsV\n8g3lfA6+m7oMxTcuUUQ9ln04hRUel+K7iUwJvtbJ28ClwMcNWhORI6RQkBNZDtCywvOZwI/8Q3Nj\nZn38o1VWFYvv1o95ZtYP3zDW5YrLX1/FV8A1/uMW8cDp+AZuq5H/fhGxzrkPgZ/h63oSCTgdU5AT\nWSpQ4u8Gehl4Gl/XzRL/wd5MfHvpVX0M3GFmqfhugTi/wrzJQKqZLXG+oZ7LvYvv9pHL8I3yer9z\nbqc/VGrSEnjfzCLwtTJ+fmSrKNIwGiVVREQ86j4SERGPQkFERDwKBRER8SgURETEo1AQERGPQkFE\nRDwKBRER8fw/mBIlJRttB04AAAAASUVORK5CYII=\n", - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "graph_utility_estimates(agent, sequential_decision_environment, 500, [(2,2), (3,2)])" ] @@ -321,7 +463,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": null, "metadata": { "collapsed": true }, @@ -348,7 +490,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": null, "metadata": { "collapsed": true }, @@ -367,7 +509,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": null, "metadata": { "collapsed": true }, @@ -391,58 +533,9 @@ }, { "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "defaultdict(float,\n", - " {((0, 0), (-1, 0)): -0.10293706293706295,\n", - " ((0, 0), (0, -1)): -0.10590764087842354,\n", - " ((0, 0), (0, 1)): 0.05460040868097919,\n", - " ((0, 0), (1, 0)): -0.09867203219315898,\n", - " ((0, 1), (-1, 0)): 0.07177237857105365,\n", - " ((0, 1), (0, -1)): 0.060286786739471215,\n", - " ((0, 1), (0, 1)): 0.10374209705939107,\n", - " ((0, 1), (1, 0)): -0.04,\n", - " ((0, 2), (-1, 0)): 0.09308553784444584,\n", - " ((0, 2), (0, -1)): 0.09710376713758972,\n", - " ((0, 2), (0, 1)): 0.12895703412485182,\n", - " ((0, 2), (1, 0)): 0.1325347830202934,\n", - " ((1, 0), (-1, 0)): -0.07589625670469141,\n", - " ((1, 0), (0, -1)): -0.0759999433406361,\n", - " ((1, 0), (0, 1)): -0.07323076923076924,\n", - " ((1, 0), (1, 0)): 0.07539875443960498,\n", - " ((1, 2), (-1, 0)): 0.09841555812424703,\n", - " ((1, 2), (0, -1)): 0.1713989451054505,\n", - " ((1, 2), (0, 1)): 0.16142640572251182,\n", - " ((1, 2), (1, 0)): 0.19259892322613212,\n", - " ((2, 0), (-1, 0)): -0.0759999433406361,\n", - " ((2, 0), (0, -1)): -0.0759999433406361,\n", - " ((2, 0), (0, 1)): -0.08367037404281108,\n", - " ((2, 0), (1, 0)): -0.0437928007023705,\n", - " ((2, 1), (-1, 0)): -0.009680447057460156,\n", - " ((2, 1), (0, -1)): -0.6618548845169473,\n", - " ((2, 1), (0, 1)): -0.4333323454834963,\n", - " ((2, 1), (1, 0)): -0.8872940082892214,\n", - " ((2, 2), (-1, 0)): 0.1483330033351123,\n", - " ((2, 2), (0, -1)): 0.04473676319907405,\n", - " ((2, 2), (0, 1)): 0.13217540013336543,\n", - " ((2, 2), (1, 0)): 0.30829164610044535,\n", - " ((3, 0), (-1, 0)): -0.6432395354845424,\n", - " ((3, 0), (0, -1)): 0.0,\n", - " ((3, 0), (0, 1)): -0.787040488208054,\n", - " ((3, 0), (1, 0)): -0.04,\n", - " ((3, 1), None): -0.7641890167582844,\n", - " ((3, 2), None): 0.4106787728880888})" - ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" - } - ], + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "q_agent.Q" ] @@ -461,7 +554,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": null, "metadata": { "collapsed": true }, @@ -476,31 +569,9 @@ }, { "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "defaultdict(>,\n", - " {(0, 0): 0.05460040868097919,\n", - " (0, 1): 0.10374209705939107,\n", - " (0, 2): 0.1325347830202934,\n", - " (1, 0): 0.07539875443960498,\n", - " (1, 2): 0.19259892322613212,\n", - " (2, 0): -0.0437928007023705,\n", - " (2, 1): -0.009680447057460156,\n", - " (2, 2): 0.30829164610044535,\n", - " (3, 0): 0.0,\n", - " (3, 1): -0.7641890167582844,\n", - " (3, 2): 0.4106787728880888})" - ] - }, - "execution_count": 17, - "metadata": {}, - "output_type": "execute_result" - } - ], + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "U" ] @@ -514,17 +585,9 @@ }, { "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{(0, 1): 0.3984432178350045, (1, 2): 0.649585681261095, (3, 2): 1.0, (0, 0): 0.2962883154554812, (3, 0): 0.12987274656746342, (3, 1): -1.0, (2, 1): 0.48644001739269643, (2, 0): 0.3447542300124158, (2, 2): 0.7953620878466678, (1, 0): 0.25386699846479516, (0, 2): 0.5093943765842497}\n" - ] - } - ], + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "print(value_iteration(sequential_decision_environment))" ] @@ -564,7 +627,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.1" + "version": "3.6.3" } }, "nbformat": 4, diff --git a/rl.py b/rl.py index 94664b130..1b7e20c33 100644 --- a/rl.py +++ b/rl.py @@ -7,6 +7,61 @@ import random +class PassiveDUEAgent: + + """Passive (non-learning) agent that uses direct utility estimation + on a given MDP and policy.""" + def __init__(self, pi, mdp): + self.pi = pi + self.mdp = mdp + self.U = {} + self.s = None + self.a = None + self.s_history = [] + self.r_history = [] + self.init = mdp.init + + def __call__(self, percept): + s1, r1 = percept + self.s_history.append(s1) + self.r_history.append(r1) + ## + ## + if s1 in self.mdp.terminals: + self.s = self.a = None + else: + self.s, self.a = s1, self.pi[s1] + return self.a + + def estimate_U(self): + # this function can be called only if the MDP has reached a terminal state + # it will also reset the mdp history + assert self.a is None, 'MDP is not in terminal state' + assert len(self.s_history) == len(self.r_history) + # calculating the utilities based on the current iteration + U2 = {s : [] for s in set(self.s_history)} + for i in range(len(self.s_history)): + s = self.s_history[i] + U2[s] += [sum(self.r_history[i:])] + U2 = {k : sum(v)/max(len(v), 1) for k, v in U2.items()} + # resetting history + self.s_history, self.r_history = [], [] + # setting the new utilities to the average of the previous + # iteration and this one + for k in U2.keys(): + if k in self.U.keys(): + self.U[k] = (self.U[k] + U2[k]) /2 + else: + self.U[k] = U2[k] + return self.U + + def update_state(self, percept): + '''To be overridden in most cases. The default case + assumes the percept to be of type (state, reward)''' + return percept + + + class PassiveADPAgent: """Passive (non-learning) agent that uses adaptive dynamic programming diff --git a/tests/test_rl.py b/tests/test_rl.py index 932b34ae5..95a0e2224 100644 --- a/tests/test_rl.py +++ b/tests/test_rl.py @@ -15,7 +15,17 @@ (0, 0): north, (1, 0): west, (2, 0): west, (3, 0): west, } - +def test_PassiveDUEAgent(): + agent = PassiveDUEAgent(policy, sequential_decision_environment) + for i in range(200): + run_single_trial(agent,sequential_decision_environment) + agent.estimate_U() + # Agent does not always produce same results. + # Check if results are good enough. + #print(agent.U[(0, 0)], agent.U[(0,1)], agent.U[(1,0)]) + assert agent.U[(0, 0)] > 0.15 # In reality around 0.3 + assert agent.U[(0, 1)] > 0.15 # In reality around 0.4 + assert agent.U[(1, 0)] > 0 # In reality around 0.2 def test_PassiveADPAgent(): agent = PassiveADPAgent(policy, sequential_decision_environment) pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy