If he were truly just a colleague, you would ask about his time off, but that is not the case. So, do not bring it up at all. If he was with the OW on his night off, he will of course not want to talk about it in any way.
If by some chance he was not with her, then he will want to tell you every bit of his day so that you will know he was not with her.
Remember that a large part of the excitement of an affair is that it is 'verboten.' When it is clear that you know about it but do not scream and yell and cry about it, it becomes less interesting. Then, once the newness of the relationship wears off, he may find that she is not so wonderful after all.
