Cut out the bottom half of the baffle, so it only covers the top. That's what I do.

As for not using sound damping material unless you have multiple amps is bullcrap. It helps, even if you don't have a car stereo. Also if you only have 15 watts (what most head units put out) going to a speaker (a device that is only 2%-4% efficient at best) you would want all that power being made into sound and not being disipated as movement of metal panels.