Now we finally come back to their perspective, picking up with Valjean’s decision to leave the convent. Basically, he was so happy there that it troubled his conscience (sort of like the first Matrix being paradise, but humans wouldn’t accept it), and it came down to whether he would be right to steer Cosette into the life of a nun without letting her experience the world outside and all its possibility first so that she could make an informed choice.
Well, not quite. If comes down to the fact that he’s afraid if he did that, she’d come to resent him, and that’s the final straw.
So he finds that house in the Rue Plumet, set well back from the street behind a huge, overgrown garden, with a secret entrance to the grounds running between the neighborhood’s walls that lets out in another part of town entirely (it was built for someone who wanted to visit his mistress in secret). “Only the birds had observed this curiosity, which doubtless was the subject of much interested speculation among the sparrows and finches of a century ago.”
He also rents two apartments elsewhere in Paris, to cover his tracks and provide a bolt-hole. That explains how they move so quickly when Marius follows them home: they’re at one of the decoy apartments at the time.