I'd find that fast. If it reaches the float switch fines might be your biggest problem.
You can buy an absorbent that only absorbs petroleum products, it won't get water logged, put one or two of these things in your bilge and they will absorb most of the oil, they look like rags in a sock, sort of.
The fines can be enormous.
Boat: None,build the one shown of glass, had many from 6' to 48'.
Re: Source of oily bilge?
a64pilot just beat me to the punch on absorbents, I was googling for a source. What perplexes me is only rising to below the float switch. I might suggest you are just catching it between pump outs. Someone made a good suggestion about the mast. If you are going on the hard check it then. Could be a leaky deck cleat. Putting a rap of absorbent cloth around all fuel fitting might find or eliminate them. For what it might or might not be worth.