I never changed the locks....but after the sh@t really hit the fan the OP only came here once. That said....the house is our security nest and I don't think that bringing the OW to your house is showing your home respect. He can pull up with her to get the kids etc....but being in your house when you aren't home....I don't see it.
Another way at looking at it....would you just let the cable guy in your house to do repairs when you aren't home? The mail man? Your neighbors?