If you don't want to make holes, mount the extinguisher on a 1/2" thick piece of plywood or plastic, same size as extinguisher. Now use industrial velcro, 2-3" wide and a foot long, to stick that to your bulkhead. If you apply it properly, isn't going to come off. Of course if it is at floor level it can't fall very far.
Better than industrial velcro is the stuff made by 3M that looks like "plastic" velcro. I forget the name of it. A couple of 1" tabs of that stuff and you need a pry bar to remove the bracket, but the product itself is removable from the bulkhead if needed.
Are the extinguishers with the metal clamps getting hard to find? That's all I have. I get mine professionally serviced once per year, so it's not like I have let an old metal clamp extinguisher sit there for ten years.