: Can a handheld GPS eliminate the need for a compass?

Absolutely not. A compass and chart should always be carried, whether you have a GPS or not. While they have their shortcomings, they work in all conditions and don't need batteries.

: What are it`s shortcomings?

Susceptibility to moisture and battery life.

: What kind do you like?

I don't have one yet, but will probably spring for one eventually. I'd rather learn more about manual navigation, first.

: Any opinions on small, reliable, handheld VHF radios?

I've got a Horizon HX-460S. It lives unprotected in a PFD pocket where it gets soaked every time I paddle. It has proven to live up to its submersible rating. The small package combined with a very flexible antenna makes it very unobtrusive to carry. The battery life is outstanding, effectively eliminating the need for alkaline backup, except perhaps on a long trip. It comes with both wall and car chargers. Throw the belt clip away! It releases the radio when you invert it; just what you want when rolling your kayak, eh? I keep mine tethered to my PFD.