Sure, people start this trip into the beginning of December. It just gets colder and colder. November still carries the risk of tropical systems, especially as you go south.
Insurance companies as a whole don't like anyone going south of Cape Hatteras before Nov 1, so there's a pileup of
boats in Hampton and Norfolk waiting to jump to the
Bahamas shortly after. Early November is the traditional big yacht exodus from
Newport, so your timeline is right when lots of folk are doing it.