Try removing the compass and manually rotate it horizontally in a clockwise then a counter clockwise through 2 double circle. Pause at each 45 degree increment and have someone read the display. Using this method you can determine a potential problem in the compass. The fluxgate needs to compute the quadrants properly and it alone does that computation. If it fails this test the compass is bad or the location is bad. Wires could be bad too but it may get you some information that can lead to the answer.
If you can read the values and not have them screw up it generally means the location is bad and who knows why. When I went through this we never found the reason but found a location that was perfect. I would not assume the location is good you really need to pretend like was a totally new installation
. If you could make a new set of wires so you could reconnect it to the system and then be able to try other locations that too could rule
out the wires.
The ugly party of all this if you start replacing parts you can't send them back for a refund if they are not bad. Main boards are not cheap
but they can be worked on even if Raymarine
won't. I know a guy<g>. Fluxgates may not be that bad but are easier to install. The wires and playing with the compass you can all do for free. You are at a point where nothing can be assumed to be good. You really need to get very systematic about tracking this down.
You may want to review the on the water
and dockside calibration procedures. Recalibration can be a something that may help. The calibration attempts to build a stored table of corrections to be applied to the 360 degrees of the compass. It is set up so the magnetic variations caused by the boat can be computed so that the computer can create a nice smooth circle from the fluxgate readings. If a table value went haywire it could show up as some points of the compass being goofy. This is where the manual test above might lead you to find what the symptoms of your disease really is and may help you try some things that would make a repair of one part likely to be a success. If the table is bad then it could be rebuilt through calibration and if not then it may be a compass issue or a computer issue. The details of how it screws up may point you in the better direction.
Troubleshooting this stuff can consume a lot of time. You could expect to pay for 8 to 10 hours labor to install a new system by someone that really does them often and well. Add the cost of the new system and of course it's now larger numbers. All the steps I mentioned were all done on my boat by someone that really knows this stuff. It took about 6 hours worth of his time to find the problem on my own I never would have found it. You sometimes can't believe what is happening is possible until you get the whole answer and then you just slap the forehead and wonder how you missed it.