I went through a similar situation trying to fit a
fishing rod holder clamp on my rail. I experimented with electrical tape, duct tape,
sailmaker tapes, shims of different nature, but nothing worked.
I think success depends on the use case. If the clamp needs to provide strong holding and not allow the clamp to rotate at all (like in the case of a
fishing rod holder, which needs to resist the pull from a
fish biting the lure), it may be difficult to achieve. If the clamp simply needs to hold something without wobbling (e.g. holding a rail-mounted flag pole) a combination of electrical tape with some tape with a rougher texture like duct tape or
sailmaker tape should work well.