Stain will highlight swirls. You need to completely remove all sanding marks from the previous grit before proceeding to the next grit. Finishing with hand sanding in the direction of the grain will also help.

Wiping with a damp (wet but not dripping) sponge between grits and allowing to dry helps to 'raise the grain' i.e. get the fibers that are lying on the surface to 'stand up' so they can be sanded off with the next finer grit. The sponge also will help to collect any loose bits of abrasive that may have come off the paper. There's nothing like picking up a piece of coarse grit when fine sanding to really 'make your day'. :-( Wetting the surface will often reveal swirl marks and problems before you commit to stain/finish.
Better quality abrasives and collecting the sanding dust as you work (if possible) can minimize this problem, IME.