Figured it out.
I have a really old computer, and the speakers weren't plugged in, and for some reason it wasn't detecting my headphones, so it thought that there was no sound device plugged in, and told the VVVVVV program to not make sound. So when I pressed the mute button, it tried to mute sound that didn't exist, and flipped out.