If you want it to be tighter you Can do one of two things.
1. Remove the set screw and drill a slight detent into the tube. This will let the set screw sit in the detent.
2. Better yet. Drill out the hole and tap it and insert a bolt. The screw head
will show. make sure to use a 316 bolt perhaps only slightly bigger. I would use a round headed bolt. A Robertson head
might look best but a Phillips is OK too. You don't want a hex head with edges to catch things on.
You should not need a nut on the other side and don't put any junk on it as it just makes it harder when it has to come apart. A threaded bolt can do a better job than the set screw. I have some of those and they have been there 18 years and 40,000 miles. They sort of work OK.
Anytime you say "Oh I'll never take that off" it means you will and probably soon. It really does not require that extra step to make it totally impossible to get off.