Possibly. But not necessarily — it sounds like he'd left the Earth long before the events of the first book. Possibly all the humans the Vogons wanted back on Earth were the ones who'd dodged just out of the Earth's original destruction, the ones who by all rights should have perished then.
