Good question. It depends on the installation
. Normally the limiting factor is the appture the prop fits into and normally this is small enough to restrict the ability to fit too big a prop. So some detail on what surrounds the prop would be helpful.
Diameter does not have any affect on the condition we often name around here as "over-proping". It is pitch
that is critical to that issue. Diameter and No. of blades equate to surface area. More surface area, the greater the friction, thus the more energy that goes into just turning the prop and not being put into thrust. However, more surface area equates to less slippage and thus more efficient at producing thrust. So it comes down to experience in a lot of situations and I have found not all experiences have the same outcome.
Diameter also predicts speed of rotation to some point. A ruff rule
of thumb in most of our common boat applications (in other words not powered racing
craft) the tips of the blades should be traveling at a max of about 160Km/hr. So a large blade turning slower should have a tip speed ruffly equal to a smaller blade turning at a higher speed, if that makes sense. However, the reality of this can be far from it in most of our make do type applications.
So as you see, there are several variables and nothing really nailed down.
If Aparture dictates prop size enough, then work from there and work backward as i suggested earlier. And once you settle on a Prop diameter, then the prop manufacturer should be able to help you with the rest.
There is also a spec sheet available (I think) I put it in the Study Hall above.
Gord I am sure can place a link to it. It shows sizes in relation to hull and skeg distance.