You only have to worry about hurricanes if one is heading your way.
Most have "plans" on what they will do if one is forecast
to be in the area and depending on the threat level.
I totally disagree with the statement that in the Virgin Islands
"lots of boats still get damaged or sunk". The historical charts
show lesser probability for storms than many other areas. In the past few years, only St. Croix has had boat damage and most of that was attributed to poorly moored/ anchored commercial
boats drawing down on other boats. St.Croix is Not a big Cruiser destination
... for it size it actually has very few good boating
locations for the typical cruiser and has not yet developed the infrastructure for the industry.
Going South is always an option.... if you want to go. If not, make alternative plans and keep a weather
eye and sail where you like. You do not need to be a turtle if you stay prepared and informed and take the necessary actions when a threat presents itself. These typically do not occur instantly.
I know many full time liveaboards who have lived and worked here many years and have never left their boats. Some may just be the luck of the path of the recent storms but most is they stayed inform of the development and movement and had plans and procedures of what they would do in various threat conditions... one of which is secure the boat as best as possible and move ashore for the duration of the event. You just have to have several plans ready to execute.
Really I do not find it much different from living in Florida
for the past 35 or so years. Some people worry too much about minor threats and some know when to start worrying and what to do about it.