sounds like you were simply over-trimmed in a de-powered setting.
you were set up to de-power a sail that theoretically was too large for the given wind conditions. ie: open slot with increased twist
not sure what you were trying to achieve otherwise.
also basically your lower telltail by luffing was the same as if you were sailing too high an angle to the wind. cracking the sheets
should give you a better angle of attack and the lower telltail should start flying again.
btw you can sometimes increase halyard
tension and this will give you a wider groove and remedy this situation also. this is good to know for times the helmsman is having difficulty keeping the boat moving through a lumpy sea.