Could be a number of things:
1. VC rub. remove the speaker from the enclosure an place it on a table. gently move the cone in/out with your fingertips gently, and listen for any scraping noise.
2. loose lead connection. take a 1.5V C D or AA cell and touch it to the terminals with some wire- watch/listen as the come moves in and out. don't hold in for too long or the DC will heat up the VC. Sometimes a 9V transistor battery's terminals are spaced right so you can just touch it without any wires... You should hear a pop or thump as the cone accelerates in or out if you hear a scratching noise then the leads may be loose or damaged. look to see if the leads appear frayed or loose, may be fixable.
3. Intermittent short: Same test as #2, but the leads may be touching, or there is a scrape or other damage to the VC that causes it to be shorted out on the pole pieces or elsewhere in the motor.
If the speaker makes scratchy noises with both the battery and manually (fingertips) I would bet that the VC is loose and rubbing in the gap. If that is so, the speaker needs to be reconed and the VC replaced. It's tricky to do it yourself, but there are kits and instructions available for some of the more popular models. The parts have to be right or the parameters can change drastically. heavier cone and or coil will be more bassy and less efficient but handle more power... lighter cone & coil will have less bass and sound 'tinny' in comparason. The resistance (ohms) has to be right also.