Been there many times. The marinas
tend to be $$$ for a week-long stay, but anchoring
out is cerainly practical.
I've a got a story on my blog:
Sail Delmarva: Trip Report - 2010 Delmarva Circumnavigation
There's a fairly obvious small boat anchorage in the harbor, toward the west end. The moorings are all private, but the holding is good. There are a number of unsupervised dingy landings. It can get rough in a strong east wind
. Some anchor
in Cold Spring Creek; the protection is better, but the holding is not as good, it's buggy, and it's tight.
Utsch's marina is as good as any. A nice compromise.
Utsch's Marina Cape May New Jersey
Where you sailing from?