Inverter battery charger and shore power wiring question

[Mg451]

I've got an inverter and a battery charger. The battery charger is plugged into the house ac pannel. When I'm at sea I've just been unplugging the battery charger and plugging the inverter into the the shore power socket. I once forgot to unplug the battery charger and blew destroyed the inverter. I replaced it with an Amazon jobbie it's been doing fine for over a year ( as long as I'm not plugging the battery charger in) so I have a change over switch I'm looking for a more automated install. My plan is to combine the shore power and battery charger to a selector switch then to the house pannel. When I switch it over to inverter the shore power and the inverter will be isolated however the batter charger will still be charging the batteries because it's tied into the shore power side before the switch. I assume thos will work? Since the inverter isn't powering the charger enen though it's still powerd on by the shore power?