Project 2: Continuous Control

This is a solution to this Udacity project: https://github.com/udacity/deep-reinforcement-learning/tree/master/p2_continuous-control

Introduction

In this project I worked with the (by-now-deprecated) Reacher environment.

(run example with the model I trained in 109 episodes)

In this environment, a double-jointed arm can move to target locations. A reward of +0.1 is provided for each step that the agent's hand is in the goal location. Thus, the goal of the agent is to maintain its position at the target location for as many time steps as possible. The target rotates around the arm and a randomized speed and direction of rotation.

The observation space consists of 33 variables corresponding to position, rotation, velocity, and angular velocities of the arm. Each action is a vector with four numbers, corresponding to torque applicable to two joints. Every entry in the action vector should be a number between -1 and 1.

Solving the Multi-Agent environment (Option 2)

For this project, we will use the evironment that supports multiple agents.

This version contains 20 identical agents, each with its own copy of the environment.

My agents must get an average score of +30 (over 100 consecutive episodes, and over all agents). Specifically,

• After each episode, we add up the rewards that each agent received (without discounting), to get a score for each agent. This yields 20 (potentially different) scores. We then take the average of these 20 scores.
• This yields an average score for each episode (where the average is over all 20 agents).

The environment is considered solved, when the average (over 100 episodes) of those average scores is 30 or more.

Getting Started

0. Prepare the environment as described in The Udacity repo README. Note that this repo uses pytorch 0.4.0 (!) which required me to have my machine support cuda 9:

conda install pytorch=0.4.0 cuda90 -c pytorch
conda install -c anaconda cudatoolkit==9.0

1. Download the environment from one of the links below. You need only select the environment that matches your operating system:

   • Version 2: Twenty (20) Agents
     • Linux: click here
     • Mac OSX: click here
     • Windows (32-bit): click here
     • Windows (64-bit): click here

   (For Windows users) Check out this link if you need help with determining if your computer is running a 32-bit version or 64-bit version of the Windows operating system.

2. unzip the zip file in the repository root.

Instructions

Run the P2_continuous_control notebook.

The notebook starts the environment, trains a model and saves the checkpoints and final models.

See the Report.md for more information