Another thing to try, is to heave to on the main alone.
With out first Insatiable, we put a snatch block on the toe rail, and led the sheet through that (obviously not self-tacking). However, with a snatch block on your toe rail, you's have a place to haul your self tacker's clew towards.
If your self tacker has a track, then installing stops that you can pre-set might be a way to go, but you'll then have to be going on the foredeck to set up to heave to.
Another option, is to drop your main, or go to the 3rd reef: don't heave to, just fore-reach slowly, down to about 2 knots boat
speed, enough for steerage way. We've done many miles that way, and it is nice to not give up distance to windward that you have made, even if there is little progress, at least you're not going backwards.
For us it was a process of trial and error, till we found what worked in which conditions.