maybe something to consider is one tiny button on the UI, to quickly flip through the choices of selections methods. Similar to Rhino's CPlane/World toggle button in the lower left corner of screen. But in Moi, could flip through 4 basic choices:

O = Object mode
S = Selection Set mode
G = Group mode
L = Layer mode

This way you only need a dialog when managing the sets/layers/groups. Other times you just click the little button, or use a hotkey to flip through them. Holding the button down for a second (or using RMB) could open the dialog...

jonah