We just spent 2 months there - with 4'6' and this was OK. I would say we have been able to approach 98% of the places we desired. We have seen deeper drafts (6 and a bit more) and the Owner said its OK but you already much more limited (or at least very stressed).
I personally would say max. 5ft for the Bahamas
- and if you ever intend to sell the boat and the new owner would take it back and using the ICW
5ft is the real limit to travel comfortable (and Mast
below 63ft) - Can be a big thing when reselling!