It's actually what is called "water hammer." When the pump is running, it is sucking water out of the tank and that water is flowing through the pipes. when that water flow is suddenly shut off, the water's velocity is converted to pressure, which pushes through the foot pump and out the faucet.
If you only use the foot pump as a backup the simplest solution would be to install a shutoff valve. Or a three way valve at the junction.
If you do NOT have an accumulator in the pressure side of the system that can fix it also, although its effectiveness will depend on exactly what kind of
electric water pump you have.