Since you are always seeing voltage on your meter when switching banks, sounds like you have a 'make before break' switch. In other words, the
current battery bank is not switched off before the new battery bank is on line. If you have that type, switch you can switch at will, just don't turn it off with the
engine above idle.
One way to be sure that you don't damage the alternator is only switch between
batteries with the
engine at IDLE. If you switch at higher
rpm, the alternator can produce enough juice to fry itself.
The change in voltage probably results because of different loads from the
batteries because of different size, charge and/or condition of the different banks.
Aloha
Peter O.