It seems to be the 3DX driver does not handle the buttons with any XQuartz application which is what MoI on OSX currently uses to run.

Seems to be a similar problem reported on their forum here:
http://www.3dconnexion.com/forum/viewtopic.php?f=23&t=7314

I don't know what I can do myself to fix this currently, it's probably something that would need to be tuned up either in the driver or in XQuartz. In the future at some point I'd like to try and get MoI to not use XQuartz and that would probably make this work at that point but for now I don't think it will be possible to use the device buttons with MoI, sorry.

- Michael