I'm not sure what you mean by a modem in this context. But if your cell phone
has the ability to use an external antenna
, yes, that can radically increase the performance. They range from 6-9dbi omnidirectional antennas (which are vertical rods) to sometimes more powerful omnis, to directional antennas (typical "yagi" or "barbeque" parabolic dish) that can provide 20-20 dbi of gain. Every 3dbi of gain doubles your signal, so you can see there's quite a lot to be had.
The problem is that antenna CABLES
for microwave use tend to be very inefficient. You will start at 1/4" thick and move rapidly to 1/2" thick in order to get a low-loss cable. If, for instance, you install a 9dbi gain antenna, but add in 50' of cable that has a 7dbi loss....you've only gained 2dbi. Not much.
WIlson or AntennaWorld or a half dozen of the long-established marine
antenna and cellular folks can help you figure out bang for your buck. Generally, go for the shortest thickest cable you can come to terms with (quality counts) and make sure they install the right connections on it. Then go for a high-gain omnidirectional antenna, unless you plan to stand next to the antenna pole and literally swing the directional antenna till it points at a tower to pick it up and work that call.