With the wind on the stern quarter any force will be directed largely forward and not result in heeling. I am a bit confused about your desiring to raise the boom to let it out. With the main halyard
tight and the sail fully hoisted the boom should swing clear of any obstruction. You then let it out to about forty five degrees or more to even near 70 if the apparent wind changes that is moves abeam with with an increase in boat speed.
You let it out first by letting the traveller out to leeward, then the mainsheet, and help keep the boom down and the sail flatter by tightening the boom vang
The pulley on the end of the boom could be anything. Normally there would be an attachment for the topping lift
which keeps the boom up when the sail is down and is released when the sail is up, to allow the tension on the sail to be adjusted by the mainsheet and traveller position.
You may not have a topping lift
if you have a rigid boom vang
and the pulley you refer to may be a remnant of an old system.
You may get some heeling with the sails
at fortyfive degrees as half the force is directed forward and half to the side.
Normally it would not be major, and rather less than going to windward. However it depends on the windspeed. You were going well with jib and mizzen alone so probably doubled the sail area and may have been overcanvassed. It also sounds like the main was too full, ie not flattened and very likely oversheeted.