aGrUM 3.3.2
a C++ library for (probabilistic) graphical models
gum::StructuredPlaner< GUM_ELEMENT > Class Template Referenceabstract

<agrum/FMDP/planning/structuredPlaner.h> More...

#include <structuredPlaner.h>

Inheritance diagram for gum::StructuredPlaner< GUM_ELEMENT >:
[legend]
Collaboration diagram for gum::StructuredPlaner< GUM_ELEMENT >:
[legend]

Public Member Functions

Datastructure access methods
const FMDP< GUM_ELEMENT > * fmdp ()
 Returns a const ptr on the Factored Markov Decision Process on which we're planning.
const MultiDimFunctionGraph< GUM_ELEMENT > * vFunction ()
 Returns a const ptr on the value function computed so far.
Size vFunctionSize () override
 Returns vFunction computed so far current size.
MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > * optimalPolicy () override
 Returns the best policy obtained so far.
Size optimalPolicySize () override
 Returns optimalPolicy computed so far current size.
std::string optimalPolicy2String () override
 Provide a better toDot for the optimal policy where the leaves have the action name instead of its id.
Planning Methods
void initialize (const FMDP< GUM_ELEMENT > *fmdp) override
 Initializes data structure needed for making the planning.
void makePlanning (Idx nbStep=1000000) override
 Performs a value iteration.
Initialization
virtual void initialize (const FMDP< GUM_SCALAR > *fmdp)=0
 Initializes the learner.

Static Public Member Functions

static StructuredPlaner< GUM_ELEMENT > * spumddInstance (GUM_ELEMENT discountFactor=0.9, GUM_ELEMENT epsilon=0.00001, bool verbose=true)
static StructuredPlaner< GUM_ELEMENT > * sviInstance (GUM_ELEMENT discountFactor=0.9, GUM_ELEMENT epsilon=0.00001, bool verbose=true)

Protected Member Functions

Value Iteration Methods
virtual void initVFunction_ ()
 Performs a single step of value iteration.
virtual MultiDimFunctionGraph< GUM_ELEMENT > * valueIteration_ ()
 Performs a single step of value iteration.
virtual MultiDimFunctionGraph< GUM_ELEMENT > * evalQaction_ (const MultiDimFunctionGraph< GUM_ELEMENT > *, Idx)
 Performs the P(s'|s,a).V^{t-1}(s') part of the value itération.
virtual MultiDimFunctionGraph< GUM_ELEMENT > * maximiseQactions_ (std::vector< MultiDimFunctionGraph< GUM_ELEMENT > * > &)
 Performs max_a Q(s,a).
virtual MultiDimFunctionGraph< GUM_ELEMENT > * minimiseFunctions_ (std::vector< MultiDimFunctionGraph< GUM_ELEMENT > * > &)
 Performs min_i F_i.
virtual MultiDimFunctionGraph< GUM_ELEMENT > * addReward_ (MultiDimFunctionGraph< GUM_ELEMENT > *function, Idx actionId=0)
 Perform the R(s) + gamma . function.

Protected Attributes

const FMDP< GUM_ELEMENT > * fmdp_
 The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions ).
MultiDimFunctionGraph< GUM_ELEMENT > * vFunction_
 The Value Function computed iteratively.
MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > * optimalPolicy_
 The associated optimal policy.
gum::VariableSet elVarSeq_
 A Set to eleminate primed variables.
GUM_ELEMENT discountFactor_
 Discount Factor used for infinite horizon planning.
IOperatorStrategy< GUM_ELEMENT > * operator_
bool verbose_
 Boolean used to indcates whether or not iteration informations should be displayed on terminal.

Private Attributes

GUM_ELEMENT _threshold_
 The threshold value Whenever | V^{n} - V^{n+1} | < threshold, we consider that V ~ V*.
bool _firstTime_

Constructor & destructor.

 StructuredPlaner (IOperatorStrategy< GUM_ELEMENT > *opi, GUM_ELEMENT discountFactor, GUM_ELEMENT epsilon, bool verbose)
 Default constructor.
 ~StructuredPlaner () override
 Default destructor.

Optimal policy extraction methods

virtual void evalPolicy_ ()
 Perform the required tasks to extract an optimal policy.
MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * makeArgMax_ (const MultiDimFunctionGraph< GUM_ELEMENT > *Qaction, Idx actionId)
 Creates a copy of given Qaction that can be exploit by a Argmax.
virtual MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * argmaximiseQactions_ (std::vector< MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * > &)
 Performs argmax_a Q(s,a).
void extractOptimalPolicy_ (const MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > *optimalValueFunction)
 From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet.
NodeId _recurArgMaxCopy_ (NodeId, Idx, const MultiDimFunctionGraph< GUM_ELEMENT > *, MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > *, HashTable< NodeId, NodeId > &)
 Recursion part for the createArgMaxCopy.
NodeId _recurExtractOptPol_ (NodeId, const MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > *, HashTable< NodeId, NodeId > &)
 Recursion part for the createArgMaxCopy.
void _transferActionIds_ (const ArgMaxSet< GUM_ELEMENT, Idx > &, ActionSet &)
 Extract from an ArgMaxSet the associated ActionSet.

Detailed Description

template<typename GUM_ELEMENT>
class gum::StructuredPlaner< GUM_ELEMENT >

<agrum/FMDP/planning/structuredPlaner.h>

A class to find optimal policy for a given FMDP.

Perform a structure value iteration planning

Pure virtual functions : regress_, maximize_, argmaximize_, add_ and subtract_ are a priorthe ones to be respecified according to the used datastructure (MDDs, DTs, BNs, ...)

Definition at line 82 of file structuredPlaner.h.

Constructor & Destructor Documentation

◆ StructuredPlaner()

template<typename GUM_ELEMENT>
gum::StructuredPlaner< GUM_ELEMENT >::StructuredPlaner ( IOperatorStrategy< GUM_ELEMENT > * opi,
GUM_ELEMENT discountFactor,
GUM_ELEMENT epsilon,
bool verbose )
protected

Default constructor.

Definition at line 87 of file structuredPlaner_tpl.h.

90 :
93
95 vFunction_ = nullptr;
96 optimalPolicy_ = nullptr;
97 }
<agrum/FMDP/planning/structuredPlaner.h>
GUM_ELEMENT _threshold_
The threshold value Whenever | V^{n} - V^{n+1} | < threshold, we consider that V ~ V*.
GUM_ELEMENT discountFactor_
Discount Factor used for infinite horizon planning.
IOperatorStrategy< GUM_ELEMENT > * operator_
bool verbose_
Boolean used to indcates whether or not iteration informations should be displayed on terminal.
StructuredPlaner(IOperatorStrategy< GUM_ELEMENT > *opi, GUM_ELEMENT discountFactor, GUM_ELEMENT epsilon, bool verbose)
Default constructor.
MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > * optimalPolicy_
The associated optimal policy.
MultiDimFunctionGraph< GUM_ELEMENT > * vFunction_
The Value Function computed iteratively.

References discountFactor_, operator_, and verbose_.

Referenced by ~StructuredPlaner(), spumddInstance(), and sviInstance().

Here is the caller graph for this function:

◆ ~StructuredPlaner()

template<typename GUM_ELEMENT>
gum::StructuredPlaner< GUM_ELEMENT >::~StructuredPlaner ( )
override

Default destructor.

Definition at line 103 of file structuredPlaner_tpl.h.

103 {
105
106 if (vFunction_) { delete vFunction_; }
107
108 if (optimalPolicy_) delete optimalPolicy_;
109
110 delete operator_;
111 }

References StructuredPlaner(), operator_, optimalPolicy_, and vFunction_.

Here is the call graph for this function:

Member Function Documentation

◆ _recurArgMaxCopy_()

template<typename GUM_ELEMENT>
NodeId gum::StructuredPlaner< GUM_ELEMENT >::_recurArgMaxCopy_ ( NodeId currentNodeId,
Idx actionId,
const MultiDimFunctionGraph< GUM_ELEMENT > * src,
MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * argMaxCpy,
HashTable< NodeId, NodeId > & visitedNodes )
private

Recursion part for the createArgMaxCopy.

Definition at line 505 of file structuredPlaner_tpl.h.

510 {
512
513 NodeId nody;
514 if (src->isTerminalNode(currentNodeId)) {
516 nody = argMaxCpy->manager()->addTerminalNode(leaf);
517 } else {
519 NodeId* sonsMap = static_cast< NodeId* >(
520 SOA_ALLOCATE(sizeof(NodeId) * currentNode->nodeVar()->domainSize()));
521 for (Idx moda = 0; moda < currentNode->nodeVar()->domainSize(); ++moda)
524 nody = argMaxCpy->manager()->addInternalNode(currentNode->nodeVar(), sonsMap);
525 }
527 return nody;
528 }
NodeId _recurArgMaxCopy_(NodeId, Idx, const MultiDimFunctionGraph< GUM_ELEMENT > *, MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > *, HashTable< NodeId, NodeId > &)
Recursion part for the createArgMaxCopy.

References _recurArgMaxCopy_(), gum::DiscreteVariable::domainSize(), gum::HashTable< Key, Val >::exists(), gum::HashTable< Key, Val >::insert(), gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::isTerminalNode(), gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::node(), gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::nodeValue(), gum::InternalNode::nodeVar(), SOA_ALLOCATE, and gum::InternalNode::son().

Referenced by _recurArgMaxCopy_(), and makeArgMax_().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ _recurExtractOptPol_()

template<typename GUM_ELEMENT>
NodeId gum::StructuredPlaner< GUM_ELEMENT >::_recurExtractOptPol_ ( NodeId currentNodeId,
const MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * argMaxOptVFunc,
HashTable< NodeId, NodeId > & visitedNodes )
private

Recursion part for the createArgMaxCopy.

Definition at line 582 of file structuredPlaner_tpl.h.

586 {
588
589 NodeId nody;
590 if (argMaxOptVFunc->isTerminalNode(currentNodeId)) {
593 nody = optimalPolicy_->manager()->addTerminalNode(leaf);
594 } else {
596 NodeId* sonsMap = static_cast< NodeId* >(
597 SOA_ALLOCATE(sizeof(NodeId) * currentNode->nodeVar()->domainSize()));
598 for (Idx moda = 0; moda < currentNode->nodeVar()->domainSize(); ++moda)
600 nody = optimalPolicy_->manager()->addInternalNode(currentNode->nodeVar(), sonsMap);
601 }
603 return nody;
604 }
NodeId _recurExtractOptPol_(NodeId, const MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > *, HashTable< NodeId, NodeId > &)
Recursion part for the createArgMaxCopy.
void _transferActionIds_(const ArgMaxSet< GUM_ELEMENT, Idx > &, ActionSet &)
Extract from an ArgMaxSet the associated ActionSet.

References _recurExtractOptPol_(), _transferActionIds_(), gum::DiscreteVariable::domainSize(), gum::HashTable< Key, Val >::exists(), gum::HashTable< Key, Val >::insert(), gum::InternalNode::nodeVar(), optimalPolicy_, SOA_ALLOCATE, and gum::InternalNode::son().

Referenced by _recurExtractOptPol_(), and extractOptimalPolicy_().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ _transferActionIds_()

template<typename GUM_ELEMENT>
void gum::StructuredPlaner< GUM_ELEMENT >::_transferActionIds_ ( const ArgMaxSet< GUM_ELEMENT, Idx > & src,
ActionSet & dest )
private

Extract from an ArgMaxSet the associated ActionSet.

Definition at line 611 of file structuredPlaner_tpl.h.

612 {
613 for (auto idi = src.beginSafe(); idi != src.endSafe(); ++idi)
614 dest += *idi;
615 }

References gum::ArgMaxSet< GUM_SCALAR_VAL, GUM_SCALAR_SEQ >::beginSafe(), and gum::ArgMaxSet< GUM_SCALAR_VAL, GUM_SCALAR_SEQ >::endSafe().

Referenced by _recurExtractOptPol_().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ addReward_()

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< GUM_ELEMENT > * gum::StructuredPlaner< GUM_ELEMENT >::addReward_ ( MultiDimFunctionGraph< GUM_ELEMENT > * function,
Idx actionId = 0 )
protectedvirtual

Perform the R(s) + gamma . function.

Warning
function is deleted, new one is returned

Definition at line 414 of file structuredPlaner_tpl.h.

415 {
416 // *****************************************************************************************
417 // ... we multiply the result by the discount factor, ...
419 newVFunction->copyAndMultiplyByScalar(*Vold, this->discountFactor_);
420 delete Vold;
421
422 // *****************************************************************************************
423 // ... and finally add reward
425
426 return newVFunction;
427 }
const FMDP< GUM_ELEMENT > * fmdp_
The Factored Markov Decision Process describing our planning situation (NB : this one must have funct...

References gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::add(), gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::copyAndMultiplyByScalar(), discountFactor_, fmdp_, operator_, and RECAST.

Referenced by evalPolicy_(), and valueIteration_().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ argmaximiseQactions_()

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< GUM_ELEMENT >::argmaximiseQactions_ ( std::vector< MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * > & qActionsSet)
protectedvirtual

Performs argmax_a Q(s,a).

Warning
Performs also the deallocation of the QActions

Definition at line 535 of file structuredPlaner_tpl.h.

References operator_.

Referenced by evalPolicy_().

Here is the caller graph for this function:

◆ evalPolicy_()

template<typename GUM_ELEMENT>
void gum::StructuredPlaner< GUM_ELEMENT >::evalPolicy_ ( )
protectedvirtual

Perform the required tasks to extract an optimal policy.

Reimplemented in gum::AdaptiveRMaxPlaner.

Definition at line 441 of file structuredPlaner_tpl.h.

441 {
442 // *****************************************************************************************
443 // Loop reset
445 newVFunction->copyAndReassign(*vFunction_, fmdp_->mapMainPrime());
446
449 // *****************************************************************************************
450 // For each action
451 for (auto actionIter = fmdp_->beginActions(); actionIter != fmdp_->endActions(); ++actionIter) {
453
455
457 }
458 delete newVFunction;
459
460
461 // *****************************************************************************************
462 // Next to evaluate main value function, we take maximise over all action
463 // value, ...
466
467 // *****************************************************************************************
468 // Next to evaluate main value function, we take maximise over all action
469 // value, ...
471 }
virtual MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * argmaximiseQactions_(std::vector< MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * > &)
Performs argmax_a Q(s,a).
void extractOptimalPolicy_(const MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > *optimalValueFunction)
From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracti...
virtual MultiDimFunctionGraph< GUM_ELEMENT > * addReward_(MultiDimFunctionGraph< GUM_ELEMENT > *function, Idx actionId=0)
Perform the R(s) + gamma . function.
MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * makeArgMax_(const MultiDimFunctionGraph< GUM_ELEMENT > *Qaction, Idx actionId)
Creates a copy of given Qaction that can be exploit by a Argmax.
virtual MultiDimFunctionGraph< GUM_ELEMENT > * evalQaction_(const MultiDimFunctionGraph< GUM_ELEMENT > *, Idx)
Performs the P(s'|s,a).V^{t-1}(s') part of the value itération.

References addReward_(), argmaximiseQactions_(), gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::copyAndReassign(), evalQaction_(), extractOptimalPolicy_(), fmdp_, makeArgMax_(), operator_, and vFunction_.

Here is the call graph for this function:

◆ evalQaction_()

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< GUM_ELEMENT > * gum::StructuredPlaner< GUM_ELEMENT >::evalQaction_ ( const MultiDimFunctionGraph< GUM_ELEMENT > * Vold,
Idx actionId )
protectedvirtual

Performs the P(s'|s,a).V^{t-1}(s') part of the value itération.

Definition at line 362 of file structuredPlaner_tpl.h.

364 {
365 // ******************************************************************************
366 // Initialisation :
367 // Creating a copy of last Vfunction to deduce from the new Qaction
368 // And finding the first var to eleminate (the one at the end)
369
370 return operator_->regress(Vold, actionId, this->fmdp_, this->elVarSeq_);
371 }
gum::VariableSet elVarSeq_
A Set to eleminate primed variables.

References elVarSeq_, fmdp_, and operator_.

Referenced by evalPolicy_(), and valueIteration_().

Here is the caller graph for this function:

◆ extractOptimalPolicy_()

template<typename GUM_ELEMENT>
void gum::StructuredPlaner< GUM_ELEMENT >::extractOptimalPolicy_ ( const MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * optimalValueFunction)
protected

From V(s)* = argmax_a Q*(s,a), this function extract pi*(s) This function mainly consists in extracting from each ArgMaxSet presents at the leaves the associated ActionSet.

Warning
deallocate the argmax optimal value function

Definition at line 558 of file structuredPlaner_tpl.h.

560 {
561 optimalPolicy_->clear();
562
563 // Insertion des nouvelles variables
565 = argMaxOptimalValueFunction->variablesSequence().beginSafe();
566 varIter != argMaxOptimalValueFunction->variablesSequence().endSafe();
567 ++varIter)
568 optimalPolicy_->add(**varIter);
569
571 optimalPolicy_->manager()->setRootNode(_recurExtractOptPol_(argMaxOptimalValueFunction->root(),
573 src2dest));
574
576 }

References _recurExtractOptPol_(), gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::clear(), and optimalPolicy_.

Referenced by evalPolicy_().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ fmdp()

template<typename GUM_ELEMENT>
const FMDP< GUM_ELEMENT > * gum::StructuredPlaner< GUM_ELEMENT >::fmdp ( )

Returns a const ptr on the Factored Markov Decision Process on which we're planning.

Definition at line 640 of file structuredPlaner_tpl.h.

640 {
641 return fmdp_;
642 }

References fmdp_.

Referenced by initialize().

Here is the caller graph for this function:

◆ initialize() [1/2]

virtual void gum::IPlanningStrategy< GUM_SCALAR >::initialize ( const FMDP< GUM_SCALAR > * fmdp)
pure virtualinherited

Initializes the learner.

◆ initialize() [2/2]

template<typename GUM_ELEMENT>
void gum::StructuredPlaner< GUM_ELEMENT >::initialize ( const FMDP< GUM_ELEMENT > * fmdp)
override

Initializes data structure needed for making the planning.

Warning
No calling this methods before starting the first makePlaninng will surely and definitely result in a crash

Definition at line 243 of file structuredPlaner_tpl.h.

243 {
244 fmdp_ = fmdp;
245
246 // Determination of the threshold value
248
249 // Establishement of sequence of variable elemination
250 for (auto varIter = fmdp_->beginVariables(); varIter != fmdp_->endVariables(); ++varIter)
251 elVarSeq_ << fmdp_->main2prime(*varIter);
252
253 // Initialisation of the value function
254 vFunction_ = operator_->getFunctionInstance();
255 optimalPolicy_ = operator_->getAggregatorInstance();
256 _firstTime_ = true;
257 }
const FMDP< GUM_ELEMENT > * fmdp()
Returns a const ptr on the Factored Markov Decision Process on which we're planning.

References _threshold_, discountFactor_, elVarSeq_, fmdp(), fmdp_, gum::FMDP< GUM_ELEMENT >::main2prime(), operator_, optimalPolicy_, and vFunction_.

Referenced by gum::AdaptiveRMaxPlaner::initialize().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ initVFunction_()

template<typename GUM_ELEMENT>
void gum::StructuredPlaner< GUM_ELEMENT >::initVFunction_ ( )
protectedvirtual

Performs a single step of value iteration.

Reimplemented in gum::AdaptiveRMaxPlaner.

Definition at line 308 of file structuredPlaner_tpl.h.

308 {
309 // V0(s) = max_a R(s,a) (i.e. R(s) when the reward does not depend on the action)
311 for (auto actionIter = fmdp_->beginActions(); actionIter != fmdp_->endActions(); ++actionIter) {
312 MultiDimFunctionGraph< GUM_ELEMENT >* reward = operator_->getFunctionInstance();
313 reward->copy(*(RECAST(fmdp_->reward(*actionIter))));
314 rewards.push_back(reward);
315 }
316 delete vFunction_;
318 }
virtual MultiDimFunctionGraph< GUM_ELEMENT > * maximiseQactions_(std::vector< MultiDimFunctionGraph< GUM_ELEMENT > * > &)
Performs max_a Q(s,a).

References gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::copy(), fmdp_, operator_, and RECAST.

Referenced by makePlanning().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ makeArgMax_()

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< ArgMaxSet< GUM_ELEMENT, Idx >, SetTerminalNodePolicy > * gum::StructuredPlaner< GUM_ELEMENT >::makeArgMax_ ( const MultiDimFunctionGraph< GUM_ELEMENT > * Qaction,
Idx actionId )
protected

Creates a copy of given Qaction that can be exploit by a Argmax.

Hence, this step consists in replacing each lea by an ArgMaxSet containing the value of the leaf and the actionId of the Qaction

Parameters
Qaction: the function graph we want to transform
actionId: the action Id associated to that graph
Warning
delete the original Qaction, returns its conversion

Definition at line 480 of file structuredPlaner_tpl.h.

482 {
484 = operator_->getArgMaxFunctionInstance();
485
486 // Insertion des nouvelles variables
488 = qAction->variablesSequence().beginSafe();
489 varIter != qAction->variablesSequence().endSafe();
490 ++varIter)
491 amcpy->add(**varIter);
492
494 amcpy->manager()->setRootNode(
496
497 delete qAction;
498 return amcpy;
499 }

References _recurArgMaxCopy_(), gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::add(), gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::manager(), operator_, gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::root(), and gum::MultiDimImplementation< GUM_ELEMENT >::variablesSequence().

Referenced by evalPolicy_().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ makePlanning()

template<typename GUM_ELEMENT>
void gum::StructuredPlaner< GUM_ELEMENT >::makePlanning ( Idx nbStep = 1000000)
overridevirtual

Performs a value iteration.

Parameters
nbStep: enables you to specify how many value iterations you wish to do. makePlanning will then stop whether when optimal value function is reach or when nbStep have been performed

Implements gum::IPlanningStrategy< GUM_ELEMENT >.

Definition at line 263 of file structuredPlaner_tpl.h.

263 {
264 if (_firstTime_) {
265 this->initVFunction_();
266 _firstTime_ = false;
267 }
268
269 // *****************************************************************************************
270 // Main loop
271 // *****************************************************************************************
272 Idx nbIte = 0;
274 while ((gap > _threshold_) && (nbIte < nbStep)) {
275 ++nbIte;
276
278
279 // *****************************************************************************************
280 // Then we compare new value function and the old one
282 gap = 0;
283
284 for (deltaV->beginValues(); deltaV->hasValue(); deltaV->nextValue())
285 if (gap < fabs(deltaV->value())) gap = fabs(deltaV->value());
286 delete deltaV;
287
288 if (verbose_)
289 std::cout << " ------------------- Fin itération n° " << nbIte << std::endl
290 << " Gap : " << gap << " - " << _threshold_ << std::endl;
291
292 // *****************************************************************************************
293 // And eventually we update pointers for next loop
294 delete vFunction_;
296 }
297
298 // *****************************************************************************************
299 // Policy matching value function research
300 // *****************************************************************************************
301 this->evalPolicy_();
302 }
virtual void evalPolicy_()
Perform the required tasks to extract an optimal policy.
virtual void initVFunction_()
Performs a single step of value iteration.
virtual MultiDimFunctionGraph< GUM_ELEMENT > * valueIteration_()
Performs a single step of value iteration.

References _firstTime_, and initVFunction_().

Referenced by gum::AdaptiveRMaxPlaner::makePlanning().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ maximiseQactions_()

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< GUM_ELEMENT > * gum::StructuredPlaner< GUM_ELEMENT >::maximiseQactions_ ( std::vector< MultiDimFunctionGraph< GUM_ELEMENT > * > & qActionsSet)
protectedvirtual

Performs max_a Q(s,a).

Warning
Performs also the deallocation of the QActions

Definition at line 377 of file structuredPlaner_tpl.h.

378 {
380 qActionsSet.pop_back();
381
382 while (!qActionsSet.empty()) {
384 qActionsSet.pop_back();
386 }
387
388 return newVFunction;
389 }

References operator_.

Referenced by valueIteration_().

Here is the caller graph for this function:

◆ minimiseFunctions_()

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< GUM_ELEMENT > * gum::StructuredPlaner< GUM_ELEMENT >::minimiseFunctions_ ( std::vector< MultiDimFunctionGraph< GUM_ELEMENT > * > & qActionsSet)
protectedvirtual

Performs min_i F_i.

Warning
Performs also the deallocation of the F_i

Definition at line 395 of file structuredPlaner_tpl.h.

396 {
398 qActionsSet.pop_back();
399
400 while (!qActionsSet.empty()) {
402 qActionsSet.pop_back();
404 }
405
406 return newVFunction;
407 }

References operator_.

◆ optimalPolicy()

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy > * gum::StructuredPlaner< GUM_ELEMENT >::optimalPolicy ( )
overridevirtual

Returns the best policy obtained so far.

Implements gum::IPlanningStrategy< GUM_ELEMENT >.

Definition at line 656 of file structuredPlaner_tpl.h.

656 {
657 return optimalPolicy_;
658 }

References optimalPolicy_.

◆ optimalPolicy2String()

template<typename GUM_ELEMENT>
std::string gum::StructuredPlaner< GUM_ELEMENT >::optimalPolicy2String ( )
overridevirtual

Provide a better toDot for the optimal policy where the leaves have the action name instead of its id.

Implements gum::IPlanningStrategy< GUM_ELEMENT >.

Definition at line 125 of file structuredPlaner_tpl.h.

125 {
126 // ************************************************************************
127 // Discarding the case where no \pi* have been computed
128 if (!optimalPolicy_ || optimalPolicy_->root() == 0) return "NO OPTIMAL POLICY CALCULATED YET";
129
130 // ************************************************************************
131 // Initialisation
132
133 // Declaration of the needed string stream
138
139 // First line for the toDot
140 output += "\ndigraph \" OPTIMAL POLICY \" {\n";
141
142 // Form line for the internal node stream en the terminal node stream
143 terminalStream += "node [shape = box];\n";
144 nonTerminalStream += "node [shape = ellipse];\n";
145
146 // For somme clarity in the final string
147 std::string tab = "\t";
148
149 // To know if we already checked a node or not
151
152 // FIFO of nodes to visit
154
155 // Loading the FIFO
156 fifo.push(optimalPolicy_->root());
157 visited << optimalPolicy_->root();
158
159
160 // ************************************************************************
161 // Main loop
162 while (!fifo.empty()) {
163 // Node to visit
164 NodeId currentNodeId = fifo.front();
165 fifo.pop();
166
167 // Checking if it is terminal
168 if (optimalPolicy_->isTerminalNode(currentNodeId)) {
169 // Get back the associated ActionSet
171
172 // Creating a line for this node
173 terminalStream += std::format("{0}{1};{0}{1} [label=\"{1} - ", tab, currentNodeId);
174
175 // Enumerating and adding to the line the associated optimal actions
176 for (SequenceIteratorSafe< Idx > valIter = ase.beginSafe(); valIter != ase.endSafe();
177 ++valIter) {
178 terminalStream += fmdp_->actionName(*valIter);
179 terminalStream += ' ';
180 }
181
182 // Terminating line
183 terminalStream += "\"];\n";
184 continue;
185 }
186
187 // Either wise
188 {
189 // Geting back the associated internal node
191
192 // Creating a line in internalnode stream for this node
193 nonTerminalStream += std::format("{0}{1};{0}{1} [label=\"{1} - {2}\"];\n",
194 tab,
196 currentNode->nodeVar()->name());
197
198 // Going through the sons and agregating them according the the sons Ids
200 for (Idx sonIter = 0; sonIter < currentNode->nbSons(); ++sonIter) {
201 if (!visited.exists(currentNode->son(sonIter))) {
202 fifo.push(currentNode->son(sonIter));
203 visited << currentNode->son(sonIter);
204 }
205 if (!sonMap.exists(currentNode->son(sonIter)))
206 sonMap.insert(currentNode->son(sonIter), new LinkedList< Idx >());
207 sonMap[currentNode->son(sonIter)]->addLink(sonIter);
208 }
209
210 // Adding to the arc stram
211 for (auto sonIter = sonMap.beginSafe(); sonIter != sonMap.endSafe(); ++sonIter) {
212 arcstream += std::format("{}{} -> {} [label=\" ", tab, currentNodeId, sonIter.key());
213 Link< Idx >* modaIter = sonIter.val()->list();
214 while (modaIter) {
215 arcstream += currentNode->nodeVar()->label(modaIter->element());
216 if (modaIter->nextLink()) arcstream += ", ";
217 modaIter = modaIter->nextLink();
218 }
219 arcstream += "\",color=\"#00ff00\"];\n";
220 delete sonIter.val();
221 }
222 }
223 }
224
225 // Terminating
226 output += terminalStream + '\n' + nonTerminalStream + '\n' + arcstream + "\n}\n";
227
228 return output;
229 }

References optimalPolicy_.

◆ optimalPolicySize()

template<typename GUM_ELEMENT>
Size gum::StructuredPlaner< GUM_ELEMENT >::optimalPolicySize ( )
overridevirtual

Returns optimalPolicy computed so far current size.

Implements gum::IPlanningStrategy< GUM_ELEMENT >.

Definition at line 661 of file structuredPlaner_tpl.h.

661 {
662 return optimalPolicy_ != nullptr ? optimalPolicy_->realSize() : 0;
663 }

References optimalPolicy_.

◆ spumddInstance()

template<typename GUM_ELEMENT>
StructuredPlaner< GUM_ELEMENT > * gum::StructuredPlaner< GUM_ELEMENT >::spumddInstance ( GUM_ELEMENT discountFactor = 0.9,
GUM_ELEMENT epsilon = 0.00001,
bool verbose = true )
static

Definition at line 619 of file structuredPlaner_tpl.h.

References StructuredPlaner().

Referenced by gum::SDYNA::RandomMDDInstance(), and gum::SDYNA::spimddiInstance().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ sviInstance()

template<typename GUM_ELEMENT>
StructuredPlaner< GUM_ELEMENT > * gum::StructuredPlaner< GUM_ELEMENT >::sviInstance ( GUM_ELEMENT discountFactor = 0.9,
GUM_ELEMENT epsilon = 0.00001,
bool verbose = true )
static

Definition at line 630 of file structuredPlaner_tpl.h.

References StructuredPlaner().

Referenced by gum::SDYNA::RandomTreeInstance(), and gum::SDYNA::spitiInstance().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ valueIteration_()

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< GUM_ELEMENT > * gum::StructuredPlaner< GUM_ELEMENT >::valueIteration_ ( )
protectedvirtual

Performs a single step of value iteration.

Reimplemented in gum::AdaptiveRMaxPlaner.

Definition at line 332 of file structuredPlaner_tpl.h.

332 {
333 // *****************************************************************************************
334 // Loop reset
336 newVFunction->copyAndReassign(*vFunction_, fmdp_->mapMainPrime());
337
338 // *****************************************************************************************
339 // For each action
341 for (auto actionIter = fmdp_->beginActions(); actionIter != fmdp_->endActions(); ++actionIter) {
343 // Q(s,a) = R(s,a) + gamma * E[V(s')|s,a]: the reward depends on the action, hence it must
344 // be added before maximising over the actions
346 qActionsSet.push_back(qAction);
347 }
348 delete newVFunction;
349
350 // *****************************************************************************************
351 // Next to evaluate main value function, we take maximise over all action
352 // value, ...
354
355 return newVFunction;
356 }

References addReward_(), gum::MultiDimFunctionGraph< GUM_ELEMENT, TerminalNodePolicy >::copyAndReassign(), evalQaction_(), fmdp_, maximiseQactions_(), operator_, and vFunction_.

Here is the call graph for this function:

◆ vFunction()

template<typename GUM_ELEMENT>
const MultiDimFunctionGraph< GUM_ELEMENT > * gum::StructuredPlaner< GUM_ELEMENT >::vFunction ( )

Returns a const ptr on the value function computed so far.

Definition at line 645 of file structuredPlaner_tpl.h.

645 {
646 return vFunction_;
647 }

References vFunction_.

◆ vFunctionSize()

template<typename GUM_ELEMENT>
Size gum::StructuredPlaner< GUM_ELEMENT >::vFunctionSize ( )
overridevirtual

Returns vFunction computed so far current size.

Implements gum::IPlanningStrategy< GUM_ELEMENT >.

Definition at line 650 of file structuredPlaner_tpl.h.

650 {
651 return vFunction_ != nullptr ? vFunction_->realSize() : 0;
652 }

References vFunction_.

Member Data Documentation

◆ _firstTime_

template<typename GUM_ELEMENT>
bool gum::StructuredPlaner< GUM_ELEMENT >::_firstTime_
private

Definition at line 368 of file structuredPlaner.h.

Referenced by makePlanning().

◆ _threshold_

template<typename GUM_ELEMENT>
GUM_ELEMENT gum::StructuredPlaner< GUM_ELEMENT >::_threshold_
private

The threshold value Whenever | V^{n} - V^{n+1} | < threshold, we consider that V ~ V*.

Definition at line 367 of file structuredPlaner.h.

Referenced by initialize().

◆ discountFactor_

template<typename GUM_ELEMENT>
GUM_ELEMENT gum::StructuredPlaner< GUM_ELEMENT >::discountFactor_
protected

Discount Factor used for infinite horizon planning.

Definition at line 351 of file structuredPlaner.h.

Referenced by StructuredPlaner(), addReward_(), and initialize().

◆ elVarSeq_

template<typename GUM_ELEMENT>
gum::VariableSet gum::StructuredPlaner< GUM_ELEMENT >::elVarSeq_
protected

A Set to eleminate primed variables.

Definition at line 346 of file structuredPlaner.h.

Referenced by evalQaction_(), and initialize().

◆ fmdp_

template<typename GUM_ELEMENT>
const FMDP< GUM_ELEMENT >* gum::StructuredPlaner< GUM_ELEMENT >::fmdp_
protected

The Factored Markov Decision Process describing our planning situation (NB : this one must have function graph as transitions and reward functions ).

Definition at line 326 of file structuredPlaner.h.

Referenced by addReward_(), evalPolicy_(), evalQaction_(), fmdp(), initialize(), initVFunction_(), and valueIteration_().

◆ operator_

◆ optimalPolicy_

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< ActionSet, SetTerminalNodePolicy >* gum::StructuredPlaner< GUM_ELEMENT >::optimalPolicy_
protected

The associated optimal policy.

Warning
Leaves are ActionSet which contains the ids of the best actions While this is sufficient to be exploited, to be understood by a human somme translation from the fmdp_ is required. optimalPolicy2String do this job.

Definition at line 341 of file structuredPlaner.h.

Referenced by ~StructuredPlaner(), _recurExtractOptPol_(), extractOptimalPolicy_(), initialize(), optimalPolicy(), optimalPolicy2String(), and optimalPolicySize().

◆ verbose_

template<typename GUM_ELEMENT>
bool gum::StructuredPlaner< GUM_ELEMENT >::verbose_
protected

Boolean used to indcates whether or not iteration informations should be displayed on terminal.

Definition at line 359 of file structuredPlaner.h.

Referenced by StructuredPlaner().

◆ vFunction_

template<typename GUM_ELEMENT>
MultiDimFunctionGraph< GUM_ELEMENT >* gum::StructuredPlaner< GUM_ELEMENT >::vFunction_
protected

The Value Function computed iteratively.

Definition at line 331 of file structuredPlaner.h.

Referenced by ~StructuredPlaner(), evalPolicy_(), initialize(), valueIteration_(), vFunction(), and vFunctionSize().


The documentation for this class was generated from the following files: