My guess is motor
mounts. If you have a shaft problem, I believe vibrations will be worse at higher RPMs.
Getting back to motor
mounts and I do not know what you have on your boat. I have seen a collection of junk engineering over the years for mounts. There is no one motor mount for all situations. For example, those mounts with adjustable heights change the fulcrum point via the set heights. Remember, the forward & reverse loading created by the engine is transferred to the hull
via the mounts.
A better solution to the height adjustment problem is to shim the mount itself rather than to have a threaded rod type adjustment.
Some more info on mounts. They are designed for both horizontal and vertical forces along with vibration isolation. In fact, mounts have their own frequency resonances. This means if a vibration frequency of say 500 Hz at 2Gs is applied to the mount that has a resonance at that frequency, instead of damping the applied vibrations, the vibrations will be amplified making it much worse.
Enough for motor mount engineering. If your engine vibrations were acceptable before the prop got tangled, you might be wise to just replace them with parts
that have similar ratings. When the boat is out of the water
, you can if needed have the shaft checked.