ARl7115
Second Exam (continued)
July 2022
Question 1 ..................................................................
[25 points]
(a) Consider the blocks world. The blocks can be on a table or in a box. Consider three
[15]
generic actions: a0, a1, and a2 described as follows:
a0 : when applied to a block, will keep it in the box;
a1 : when applied to a block, will move it on the table;
a2: when applied to two blocks, will move the first one on top ofthe
second one.
Consider the following four states in the system:
S0 : all blocks are in the box, no block is on the table;
S1: only block Bis on the table; all other blocks are in the box;
S2: both blocks B and Care on the table, with Con top of B;
S3: blocks B, C and Dare on the table, with Don top of C and Con top of B.
Furthermore, additional information is provided in Table 1, where each state has a re-
ward, possible actions and a transition model for each action. Note that for a given ac-
tion, the probability values indicated in its transition model all sum up to 1.
Table 1: Additional information
State Reward Action
Transition Model
So
ro
aob
(1, So)
a1b
(Po,So); (Pi, S1)
r1
S1
aoc
(1, S1)
a1c (p5,S1);(pf,S4);(P½,S2)
a2c
(p5, S1); (Pi, S2);
r2
S2
aod
(l,S2)
a1d (p5,S2);(Pi,Ss);(p~,S3)
a2d
(P6,S2); (p1, S3);
S3
100
Assuming we model this problem as Markov Decision Process (MVP) and consider a
discount value CY, provide the utility of each of the states S0, S1 and S2 for the first three
iterations using the value iteration algorithm. Note that although the states S4 and S5
have not been defined, they should be assumed in the system.
(b) Consider the following policy, n0 = {So H aob,S1 H a1c, S2 H a2d}- Is n0 optimal?
[10]
Explain.
Question 2 ..................................................................
The diagram in Figure 1 represents the extensive form of a sequential game
[15 points]
1. Provide the strategic form associated with the game;
2. Does any player have a dominant strategy?
3. Is there a dominant strategy equilibrium?
Page 1 of 3
Please turn over to the next page ...