Yes, it’s a dumb idea for several reasons:
The gasket is a specific thickness to give the impeller the correct endplay. You can easily duplicate this with a similar gasket material or paper but you can’t do it with stuff that has no crush.
Also, when you use a gasket, you don’t get sticky gunk all over everything, including the impeller. Make a couple of gaskets and keep a spare handy, it’s a lot easier than scraping old sealer off the machined surfaces.
If the mating surfaces are pitted beyond hope, then disregard the above. In that case you may get a little more service
out of the pump if you use silicone. (Sparingly)