You're right, in that nothing wants to stick to silicone, including silicone.
The process for removing silicone caulk (residue) takes some time and effort.
1. Tear off the bulk of the caulk. (As youíve done)
2. Use a razor blade to scrape off as much of the residue as possible without scratching the surface you are cleaning
3. Apply mineral spirits or denatured alcohol
to the residue (silicone 'oil').
4. Brush away any bits that "pill" and become detached.
5. Rinse & repeat.
6. Dry, before applying new caulk.