Dyna Class Reference

#include <Dyna.hh>

Inheritance diagram for Dyna:

List of all members.

Classes
struct	dynaExperience
Public Member Functions
	Dyna (int numactions, float gamma, float initialvalue, float alpha, int k, float epsilon, Random rng=Random())
	Dyna (const Dyna &)
virtual int	first_action (const std::vector< float > &s)
int	getBestAction (const std::vector< float > &s)
double	getSeconds ()
float	getValue (std::vector< float > state)
virtual void	last_action (float r)
void	logValues (ofstream *of, int xmin, int xmax, int ymin, int ymax)
virtual int	next_action (float r, const std::vector< float > &s)
void	printState (const std::vector< float > &s)
std::vector< float >::iterator	random_max_element (std::vector< float >::iterator start, std::vector< float >::iterator end)
virtual void	savePolicy (const char *filename)
virtual void	seedExp (std::vector< experience >)
virtual void	setDebug (bool d)
virtual	~Dyna ()
Public Attributes
float	epsilon
Protected Types
typedef const std::vector < float > *	state_t
Protected Member Functions
void	addExperience (float r, state_t s, bool term)
state_t	canonicalize (const std::vector< float > &s)
Private Attributes
bool	ACTDEBUG
const float	alpha
float *	currentq
std::vector< dynaExperience >	experiences
const float	gamma
const float	initialvalue
const int	k
int	lastact
state_t	laststate
const int	numactions
std::map< state_t, std::vector < float > >	Q
Random	rng
std::set< std::vector< float > >	statespace

---

Detailed Description

Agent that uses straight Q-learning, with no generalization and epsilon-greedy exploration.

Definition at line 17 of file Dyna.hh.

---

Member Typedef Documentation

typedef const std::vector<float>* Dyna::state_t [protected]

The implementation maps all sensations to a set of canonical pointers, which serve as the internal representation of environment state.

Definition at line 61 of file Dyna.hh.

---

Constructor & Destructor Documentation

Dyna::Dyna	(	int	numactions,
		float	gamma,
		float	initialvalue,
		float	alpha,
		int	k,
		float	epsilon,
		Random	rng = Random()
	)

Standard constructor

Parameters:

numactions	The number of possible actions
gamma	The discount factor
initialvalue	The initial value of each Q(s,a)
alpha	The learning rate
epsilon	The probability of taking a random action
rng	Initial state of the random number generator to use

Definition at line 7 of file Dyna.cc.

Dyna::Dyna

(

const Dyna &

)

Unimplemented copy constructor: internal state cannot be simply copied.

Dyna::~Dyna

(

)

[virtual]

Definition at line 21 of file Dyna.cc.

---

Member Function Documentation

void Dyna::addExperience	(	float	r,
		state_t	s,
		bool	term
	)		[protected]

Definition at line 93 of file Dyna.cc.

Dyna::state_t Dyna::canonicalize

(

const std::vector< float > &

s

)

[protected]

Produces a canonical representation of the given sensation.

Parameters:

s	The current sensation from the environment.

Returns:

A pointer to an equivalent state in statespace.

Definition at line 146 of file Dyna.cc.

int Dyna::first_action

(

const std::vector< float > &

s

)

[virtual]

Implements Agent.

Definition at line 23 of file Dyna.cc.

int Dyna::getBestAction

(

const std::vector< float > &

s

)

Definition at line 35 of file Dyna.cc.

double Dyna::getSeconds

(

)

Definition at line 317 of file Dyna.cc.

float Dyna::getValue

(

std::vector< float >

state

)

Definition at line 251 of file Dyna.cc.

void Dyna::last_action

(

float

r

)

[virtual]

Implements Agent.

Definition at line 133 of file Dyna.cc.

void Dyna::logValues	(	ofstream *	of,
		int	xmin,
		int	xmax,
		int	ymin,
		int	ymax
	)

Definition at line 235 of file Dyna.cc.

int Dyna::next_action	(	float	r,
		const std::vector< float > &	s
	)		[virtual]

Implements Agent.

Definition at line 106 of file Dyna.cc.

void Dyna::printState

(

const std::vector< float > &

s

)

Definition at line 185 of file Dyna.cc.

std::vector< float >::iterator Dyna::random_max_element	(	std::vector< float >::iterator	start,
		std::vector< float >::iterator	end
	)

Definition at line 160 of file Dyna.cc.

void Dyna::savePolicy

(

const char *

filename

)

[virtual]

Reimplemented from Agent.

Definition at line 285 of file Dyna.cc.

void Dyna::seedExp

(

std::vector< experience >

seeds

)

[virtual]

Reimplemented from Agent.

Definition at line 193 of file Dyna.cc.

void Dyna::setDebug

(

bool

d

)

[virtual]

Implements Agent.

Definition at line 180 of file Dyna.cc.

---

Member Data Documentation

bool Dyna::ACTDEBUG [private]

Definition at line 103 of file Dyna.hh.

const float Dyna::alpha [private]

Definition at line 95 of file Dyna.hh.

float* Dyna::currentq [private]

Definition at line 99 of file Dyna.hh.

float Dyna::epsilon

Definition at line 55 of file Dyna.hh.

std::vector<dynaExperience> Dyna::experiences [private]

Definition at line 89 of file Dyna.hh.

const float Dyna::gamma [private]

Definition at line 92 of file Dyna.hh.

const float Dyna::initialvalue [private]

Definition at line 94 of file Dyna.hh.

const int Dyna::k [private]

Definition at line 96 of file Dyna.hh.

int Dyna::lastact [private]

Definition at line 101 of file Dyna.hh.

state_t Dyna::laststate [private]

Definition at line 100 of file Dyna.hh.

const int Dyna::numactions [private]

Definition at line 91 of file Dyna.hh.

std::map<state_t, std::vector<float> > Dyna::Q [private]

The primary data structure of the learning algorithm, the value function Q. For state_t s and int a, Q[s][a] gives the learned maximum expected future discounted reward conditional on executing action a in state s.

Definition at line 87 of file Dyna.hh.

Random Dyna::rng [private]

Definition at line 98 of file Dyna.hh.

std::set<std::vector<float> > Dyna::statespace [private]

Set of all distinct sensations seen. Pointers to elements of this set serve as the internal representation of the environment state.

Definition at line 81 of file Dyna.hh.

---

The documentation for this class was generated from the following files:

• Dyna.hh
• Dyna.cc