Impedance is not going to be 4ohms when you play a tone through the speaker.

Use an oscilloscope for true RMS.

True, I goofed there and posted without thinking. Measuring open circuit voltage is not so bad, it's the lack of being able to tell when clipping occurs. If the amp has a regulated supply the voltage rails would not sag much (if at all) under load and so the open circuit voltage could be used for calculations. Even an unregulated amp would give measurements that are fine for what the OP wants to know. I doubt the he needs a high degree of accuracy.

The rated load (4 ohms) is fine for calculations. A scope is the ideal but anyone who asks how to determine power output isn't likely to have access to one.

This should have been the first comment in this thread.
Unless you can measure current you'll never know what your amp is outputting. At least the scope will tell you if you're clipping.
Speaker impedance varies too much with frequency, which you can ignore if you know the current, and voltage. Without that, you're just wasting your time. You're not going to get any useful numbers, just numbers that you think are useful.

Depends on the degree of accuracy required. The generic shape of a (moving coil) speaker impedance curve is well known. If you have the published Rs you can make a reasonable guess at the optimum test tone frequency to avoid impedance issues. There is a region above resonance where the inductance is not yet significant.

If you only want to know if you are getting close to the advertised output of your amp then a scope is not required.

bunch o' nerds in this thread

LOL... guilty:)