>How can you tell if you are choosing the wrong one?

If you're using the latest beta version, look at that object properties panel, or whatever Michael calls it.

The one that says "closed crv" is just a line, and wont fillet



but the one that says "edge", is the edge of a surface that you want to fillet.



wish they had that when i started with Moi and was doing the same thing you were. :-)