Basically, he took advantage of you. There a lot of men
like that....and some women, as well. As the owner/skipper you should have made sure that the crew understood there would be instructions, which must be obeyed, and that failure to do so was reason enough to return to port with him...or her, 'cause it can happen both ways.
It will be hard, but what will be growthful for you if you choose to do it is to have a really close look at what hidden benefits there are for you in playing the compliance game
. Once you understand that, it is easier to take a hard line.
Lots of women are afraid of men
saying something they do is "unfeminine." If you want to be liked, and perceived as "feminine" someone else can exploit that aspect of your character. I'm not saying that that particular shoe fits your foot, just that it might be so---and might not.
FWIW, knew a skipper
once that made sure he held $$ in the amount of return flight to port of origin that all the crew had to supply before they were accepted aboard, that he kept in a locked box aboard, along with their passports..
These days, you cannot do anything to crew that can kill them, like walking the plank or keelhauling.
All for now, gotta work above decks.