I'm not sure you are getting the "big picture" or I am not being sufficiently clear.
Try this: Stop thinking of transom shape as an independant shape entity on the hull
. The transom shape is a pure reflection of the boat's sectional shape aft.
Changing the transom shape changes the hull
shape. You can vary the rake angle from positive to negative. You can vary the radius of the transom. But the hull shape will remain the same.
You design a hull carrying the stern well beyond the aft end of LOA
You cut a transom thru that extended shape using whatever angle and radius suits your eye. The exact location of the cut may depend upon a number of factors, i.e. rating rules, LOA
target and actual sectional shape.
The famous British designer
Douglas Phillips-Birt once said, "If you are being overtaken by a breaking wave four feet higher than you transom, then you are going to get wet regardless of the shape of your stern." Or something quite like that.