I've opined this on many a prop thread on this forum.
Get ahold of the "Propellor Handbook" authored by Dave Gerr. I believe it is available online on this forum.
The book is a technical read and gives a step by step way to determine what
engine, prop, etc, is best for your
boat, etc, etc.
It's a bit of a complicated and long winded exercise to explain here, but most people expect little
diesel engines to perform far better than they really are able to.
Diesel engines are designed to run optimally at a certain rpm, but a correctly propped diesel should be able to attain it's max rated rpm while underway.
If your diesel is rated at say...3,000 rpm max rpm. it should be able to attain that underway, if it doesn't and say only manages 2500 rpm, you are over propped. In the above example a 10x12 prop is a tiny little thing and one should not expect it to perform miracles. In the same breath, if the engine spools up to 3,000 rpm without a hiccup, you are likely under propped.
You can get a prop re-pitched, but you can't increase the diameter. a 10x12 prop is already oversquare so to speak, in other words the
pitch exceeds the diameter. There comes a point where you can only expect to
pitch up a prop so much. Attaining a certain speed does not necessarily mean the prop is pitched correctly if it can't hold that speed in an adverse wind/wave condition.
Before I get too long winded on the topic, 5 knots for a 30' boat is not bad. I would imagine the
hull speed of the Tartan is likely around 6 knots or so, but to get that extra
knot requires one to do some homework first.
The engine,
transmission, prop size, etc, must all
work in harmony with each other.
Get the book and spend some time reading and studying it and you will be able to solve your own problem.