The "click" is a contactor as mentioned. It effectively "plugs in" the
motor. So you hit the switch, the switch activates the contactor, and then
current is flowing to the
motor, so the motor turns.
So for the motor to turn itself, it sounds like a problem with the contactor.
What you're describing is strange, though. The contactor is magnetic, activated by the switch, and I'm not sure that it can be partially on.
One way to check is to bypass the contactor and just see if the breaker will turn the motor on and off. I wouldn't suggest playing with this yourself if you're not familiar with working on the high
current systems on your
boat.
The windlass contactors are bloody expensive. I replaced mine with a 500a relay from Albright on amazon (meant for jeep winches). It's the same thing but 1/4 of the
price. Have to pay attention if your contactor is the 3 post type or 4 post type.