Optimist....do you have the number he called you from? You may need to lovingly confront him....no accusations, just ask outright...where did he call you from? Tell him you saw the number on caller id and are confused and a bit upset. You don't want to make a big thing out of it..but you need to know. Tell him you'd like to work on some solutions to make you feel more at ease when he gets "called out" to work at odd times like that...make it seem to be all about YOU and YOUR feelings...not him and his behaviour. It's hard, but you can do it.
