If you didn't have a jib boom until now, that could likely be the key issue right there. Because anything that's attached to a sail when it starts to luff is going to want to slam back & forth too.
The boom on a
mainsail would do the same thing when a main luffs too, except that it's attached to the
mast. And even attached as it is, if you luff violently enough, the boom will wrack & slam regardless.
I don't know what, if anything your jib boom is attached to, but if it's a new piece of
gear, you might want to start your diagnostics there. Like say, go up forward, & physically restrain the forward end of the jib boom. Maybe even by sitting in the pullpit & facing aft, if you can manage it. And then hold it firmly, while your helmsman intentionally luffs up, to varying degrees.
You might even go so far as to stand up, next to the headstay, & put your foot on your jib boom, so that you can exert a lot of pressure on it, & then, again, have your helmsman luff up.
Plus, assuming you can do so safely, try restraining the jib boom @ it's aft end, when intentionally luffing. Even if it's via a line tied on to it's aft end, that's run through a block, say, on the leeward rail, & then belayed to something solid, to tension it.
You can also try sailing without the jib boom, luffing up to varying amounts, in different wind speeds, & see what kind of results that produces, also.
Can you describe, & or post pics of the jib boom setup please. It'd help with diagnosing things.
Also, if it's not too big a deal, check everything on the headstay, from the stem, all the way up to the masthead. And check everything else on the spar while you're at it. If naught else, it'll give you peace of mind that nothing's awry, or has shaken itself loose up there.
On that, I always take/tell folks to take a
camera up the rig with them, & snap multiple shots of everything (from different perspectives & angles). That way, you don't have to
recall from memory what you saw, plus you can always show the various suspect components/areas to others, afterwards.
Plus, & this is key, it lets you take pictures of/view remotely, areas of the masthead which are nigh on impossible to view from a bosun's chair/climbing harness. Because you can position the
camera at all kinds of heights & angles, above, & along side the masthead & hounds. Recording all of the details of components that are normally only usually viewable when the
mast is pulled out of the boat.
One other idea which just came to mind. Take a small block & tackle up forward, when you're under sail. And hook it up to the tack cringle, & to a hard point on
deck near the stem. Then use it to add varying degrees of extra tension to the luff of the sail, while your helmsman intentionally luffs up, to see if that'll stop the
noise.
Also, try the same thing, only on the jib boom's forward end too.