Initalize all to zero For every state s and every action a in this state, perform update: Repeat step 2