I would start with the BIL's expectations. Does he drink alcoholic beverages and you do not? Tell him you keep a dry boat. You get to be finicky. If he uses weed and you don't want it on the boat at all, you get to declare it off limits. If he has
health issues for which you don't want to be responsible, don't accept him aboard. It is harsh, but you're the
skipper. It is your duty to take skippering seriously. Are the BiL's expectations relative to some sort of yachting fantasy? or does the poor guy just want to get to know you better?
Ask yourself if it possible you might get to like him if he toned down a certain kind of behavior? [ask him to change it for the day] Or are you firmly set against him? [don't let him on board] What will the
family repercussions be if you are truly unwilling to take him? How would your Mom handle it? Would he be easier to take if your sister came, too? [Maybe she could bring lunch and snacks for everyone?] The "I want to be alone" response at least doesn't blame him, you're taking responsibility for what you want.
You don't say how much room there is on the boat, what kind is it? How big is it? Big enough for three for a day sail. Leave at 10, rtb by 4?
I don't think there IS a diplomatic way to say to someone that they are unwelcome.
El Pinguino's suggestion/manipulation could
work, take him out and get him seasick, and then make him stand watch, and he might not want to come back. Maybe think about what he would have to agree to do, and then hold him to it for
safety reasons, since you are responsible for welfare of boat and crew.
Or just don't care. Accept that there may be unknown fallout from refusing point blank to take him because of whatever it is that you find unacceptable, and just say no, anyway. Just don't think it is diplomatic.
Ann