I went through a similar situation trying to fit a fishing
rod holder clamp on my rail. I experimented with electrical tape, duct tape, sailmaker
tapes, shims of different nature, but nothing worked.
I think success depends on the use case. If the clamp needs to provide strong holding and not allow the clamp to rotate at all (like in the case of a fishing
rod holder, which needs to resist the pull from a fish
biting the lure), it may be difficult to achieve. If the clamp simply needs to hold something without wobbling (e.g. holding a rail-mounted flag pole) a combination of electrical tape with some tape with a rougher texture like duct tape or sailmaker
tape should work well.