Everything that follows is written in a friendly tone of voice, and is a humble attempt to share another's point of view.

I am writing in some detail to answer your questions.
I don't claim to be a expert on boats, and others will have different experiences and desires. I am only sharing mine to help you with a different POV.
I understand solo travel and have spent years doing it. There can be some appeal to going alone on a journey. I have done this several times. But each time, I highly valued my time with others I met while traveling.
And I have lived alone on a
remote cattle ranch for months, with only cows, a bull, coyotes, scorpions, buzzards, and snakes for company. So I understand the appeal and value of solitude in nature, like you do. I have also spent weeks and months in close quarters and traveling with strangers who became close friends.
What size
boat?
That is a million dollar question!
If one had a million dollars to buy any
boat, it could be equipped to be easy enough for two to handle, or one with more difficulty.
Electric winches,
furling mainsails, thrusters, and other things can help. But I would not want to Singlehand a 60 or 70 or large boat. I have no desire for that. The Dashews double handed an 80 foot
ketch, Beowulf,, but they are very experienced as a team.
But, most do not buy million dollar boats to Singlehand around the world.
If forced to sail alone, my skill level and
budget would keep me to boats 40 foot or much less. I would look for things to make it easier for one or two to handle the
sails, reefing,
anchoring, etc. When alone, a boat of 30 feet can be big if the
weather turns ugly and the
sails are not reefed in a squall, etc.
If one were more skilled and had more
money and support and desire to go faster, then one could go larger, as the Open class racingboats (40, 50,60) show highly skilled racers can manage. That is not for me either. It is possible for some, but that is not for me. I am not that kind of sailor and have no desire to be that kind.
But I also see how a smaller boat could be an adventure for a while. A few days or a few weeks at most for me. But a 24 foot ultralight or a very slow Flicka 20 to sail around the world is not my idea of fun and comfort either. Fine for short trips or
day sailing or coastal cruising, but not what I would want to have to live on for 3 years while voyaging.
Even Robin Lee Graham traded up to a larger boat from Dove, while on his RTW trip.
A few months ago I saw a video clip of two young French sailors who were quickly sailing around the world in a small, Spartan, ultralight boat. It was impressive, and fast, but not my idea of fun and comfort.
Back to me?
Solo?
I would rather plan and make it possible to have crew for the passages, even if they are only temporary. I think that is smarter, safer, less stressful and more fun.
So, to have 3 others on board, a boat 35 (a little cramped) 40 to 45 looks appealing to me. Not too big for two to handle, but big enough to take two more adults for temporary crew. I feel I could happily live with a mate or SO on 40 and occasionally take on two more for passages. The question becomes, can you stand having two or three strangers in a small shared space with you for three or four weeks? I can and have. So I will.
I hope that helps.