During the Mind Camp episode in Season Four, Peter has enough and tells Stewart, "let's do away with you,", before listing the reasons why Stewart should be fired. Among the people seated in the circle is Mary Drake, who looks up very seriously at Peter. In the finale, who should show up but Mary Drake, to announce that the PM wants Stewart out and she is taking over his job.
I always took it that he was made a Life Peer, which is alluded to when Malcolm and the DOSA gang are trying to spread rumors about him becoming Foreign Secretary in Series 2, because he wasn't a member of the House of Commons. Either way it counts because Malcolm would have called it well before it actually happened.