Bellman Equation

What is Bellman Equation

The Bellman Equation expresses the value of a decision problem at a certain point in time in terms of the payoff from initial choices and the value of the remaining decision problem resulting from those initial choices.

The Formula of the Bellman Equation

$ V(s)\ =\ max(R(s,a)\ +\ \gamma V(s')) $

• s : Current state or given state.
• s’ : The following state. = The state that ends after this state.
• a : Action.
• R(s, a) : Reward.

Since few things are 100% certain in the real world, there is a more advanced formula based on the MDP.