"The ulimit() function will get and set process limits. Currently, this is limited to the maximum file size."
It appears that it may be possible to limit max process memory size for all processes by setting some system initialization settings, since that's a system wide thing it doesn't really make sense for MoI to try and alter that on its own.
So as far as I can tell there isn't any way set up on the Mac for an individual process to limit the amount of memory that the OS will give it.