There are several possibilities, but I'm betting that you have a bad connection somewhere, probably a bad ground. CHECK ALL YOUR BATTERY CABLE CONNECTIONS CAREFULLY to be sure they're clean and tight. Checking means removing and cleaning
, then replacing with proper torque.
A start battery should not need to "recover"; it should start your engine instantly. If it does not, then suspect the cable connections first, the battery capacity second (expecially if you can start the engine easily using the house batteries).
If it's not connections or battery capacity, it could be a faulty battery switch (I recently ran across one on a client's boat which was really bad, causing a serious drop in voltage). It could be a faulty solenoid or, possibly, a faulty starter. Could even be an intermittent in the wiring
harness to the engine, or a badly corroded or otherwise deficient starting switch.
Try the easy things first. That's usually the path to finding the fault fast.
Let us know what you discover.