The old sleeve would be the best guide as to the length and also to the number of fasteners.
If the section you have found is same size as current
mast you need to take some action to fit it inside, either by cutting out a strip and then bending the remaining section and push it inside. Another method is to cut the section vertically/diagonally at two opposite sides of section, than slide one of the pieces up against the other until you'll get required dimension.
Be careful of galvanic action between the fasteners and mast. Best will be identical material, i.e. if your mast is aluminium, use aluminium rivets - the role of the fasteners in your case is only to keep the internal section in place, so no real stress if the rig is positioned properly.
S/S or Monel rivets are difficult to install, and as your sleeve is below deck it is in danger
of being in contact with salt water
- the combination of S/S salt water
and aluminium is not very happy one.
Ensure that the internal end of any fastener you use is not sharp (very important) and does not protrude too much - lines and cables
should pass unhindered (do you have a conduit?).
Bolts are problematic - first of all it is difficult to close the nuts inside mast profile. Second problem is to ensure the nuts remain in place as the mast 'works' - you should use some peening/punching on the bolt end or Locktite. Pop rivets will be better and if you are worried add more rivets.
If you are going to use existing holes for new fasteners, check that the size fits exactly and that the holes are perfectly round (the tend get elongated when working).