Regarding the wire, #12 will be fine for your 40-ft round-trip panel/controller connection. #12 is OK for the controller/battery run if the distance is short. This all assumes that you have a 12V system.
Here's how you figure it:
The maximum current
from your panel is around 3A (50W / 17V = 2.94A)
#12 wire has a resistance of 1.588 milliOhms / ft, so 40 ft = 0.0635 Ohms
0.0635 Ohms x 3 Amps = 0.19V voltage drop in the wires, which is 1.11% of 17V
We generally figure that a 2% voltage drop is good enough. Less is better, but 2% is probably near the point of diminishing returns. I used 17V in my calculations, rather than 12V, since 17V is approximately the Maximum Power Point (MPPT) of your panel, and the Genasun controller will be loading the panel to this voltage.
between the controller and the battery will be carrying slightly more than the raw panel current
(the controller boosts the current and drops the voltage), but it's going to be under 5A. If you have (for example) 10 ft round-trip of #12 wire between the controller and battery, that gives you a 0.0795V drop at 5A, which is an additional 0.66% at 12V. This gives you a total percentage voltage drop of 1.77%, which is still pretty good. You will have connector losses, but these should be manageable at these power levels.
There isn't a compelling reason to put a fuse between the panel and the controller, since the short-circuit current will be limited to the maximum panel output which is not much more than 3A. I have a breaker in my panel wiring
, but this is more of a convenient disconnect point should I need to work on the charging system. If your panel were to be feeding the battery directly (not using a controller) then a fuse would be essential.
will probably never feed more than 5A into the batteries. 50W at 10V (a very dead battery) is 5A. You generally fuse to protect the wiring, and for #12 wires a 15A fuse should be fine.
As has been mentioned, if you plan to increase your solar panel capacity you might as well run heavier wires now.