Boat Size - How big is too big?

[Thames 4 Blood]

Hi All!
Havent posted for a while. My wife and I have just passed our RYA Day skipper and Coastal skippers respectively.
Currently looking for a yacht to live aboard. In the Med for at least a year, maybe more and then possibly Atlantic crossing.
We are completely convinced that a Jeanneau will serve our needs well.
We have sailed, among others, a Jeanneau 45.2 and a Beneteau 473.
My Q is how big is too big for 2?
We have found that with a yacht set up for easy handling ( in mast reefing, electric winches, bow thruster for occasional ease etc ) the size does not seem to be much of a challenge. We are both 40ish and fit.
I see a 45ft Jeanneau Sun Oddysey as being suitable but could run to a 50ft if warranted...
The advantages of the larger boat being more space ( duh ) but also a far better bed in the forecabbin ( this matters as I am not short and we need to have some creature comforts ), better sea keeping, a fab lazarette in the bow etc etc....
Now I cant see why this should be disproportionately harder to handle than a 45ft???

Any thoughts?