For the domain's setSOADomainEnv.cmd file, try:
set DEFAULT_MEM_ARGS=-Xms512m -Xmx1024m
set PORT_MEM_ARGS=-Xms768m -Xmx3072m
this is the value I am using
set DEFAULT_MEM_ARGS=-Xms768m -Xmx2048m
set PORT_MEM_ARGS=-Xms768m -Xmx2560m
what am worried is in my local I created only few instances (I have deployed and undeployed the process so many times as well), even then the DEV_MDS growth rate is large. how do we handle in a production environment where we have 8-10 applications with lots of transactions.