Enlist help from your HAM friend. 99% something is wrong with either the coax, connectors, or antenna.
RG8x is terrible coax. It is fine for short runs, but usually on a sailboat I recommend against it, since even with a small sailboat the run is fairly long to the top of the
mast. Depending on length, you might be loosing 50%-75% of your signal.
Your HAM friend hopefully knows this, but with a lossy cable like RG8x, you can read a good VSWR on the meter, and still have serious problems. Hidden by the cable losses. So, just because the VSWR tests good, does not mean it is good. (but bad VSWR always means something is wrong)
It might be worth cutting off a connector, and seeing if there is any
corrosion of the wire itself. If there is any sign of corrosion or
water inside the coax, the whole run needs replaced. If you are not experienced putting on a new connector, know that it is very easy to mess up, and have it look fine, and
work fine for 6 months before it stops working.
1%, Something is wrong (damaged) with the VHF radio. I can't think of any settings that would do this, but perhaps the internal switch or splitter burned up. It does happen that a radio fails, but
reception issues are almost always an issue with the coax or antenna.
Sharing an antenna with both the VHF and AIS signals is fine. With a VHF that has a built in AIS receiver, I don't think it can be done any other way.