Thanks for the comments pior!

It is possible to turn off the numeric input panel using a setting in moi.ini, it's in the [UI] section:

[UI]
<...>
DisableNumericInputFlyout=n

Set that to DisableNumericInputFlyout=y and it shouldn't show up.

- Michael