I assume you have wound a bit of weather helm
into the main rudder by adjusting the tiller lines. Also it can sometimes be worth moving the lines down the tiller slightly to increase the rudder movements
. It's a tricky balance sometimes as downwind you want the movememts low when you are on course, but if she starts to round up you suddenly need lots of rudder to drive her back down.
Ultimately the sailplan with just one headsail is unbalanced, I've normally used a main to leeward and headsail poled out to windward. In stronger winds I reef both down.
She might balance better with the staysail set, either sheeted in tight to drive the bow down in a roundup or winged out opposite the jib
. I'd probably run the staysail to leeward and the jib
Maybe you have a touch to much sail up. What happens if you slow down a bit? Also it can sometime be worth coming up a touch, maybe 15 degrees rather than Setting the vane to DDW.
Good luck, and happy experimentation. It should be able to be made to work
well in those conditions.