I think there are several things that help determine the shaft size....if its unsupported for some distance a thicker shaft would help eliminate "whip" in its length.
With just the info you have supplied, and with no personal knowledge of the Passport 41 I would think the end being turned down to 1" would be OK.
Originally my 48ft boat had a 72 hp engine
with a 23" three bladed prop and an 1-1/4" shaft....this lasted for 30 years and about 30,000 miles.
What did the surveyor
have to say about it?