All the better for your case in securing sole custody. The history of his lack of visitation will work against him if he decides to contest the custody arrangement you propose.