You can limit the maximum memory used via command line options to the VM.
Noting of course that typically if you are in fact actually using that memory that reducing it is going to have some negative impact on your application including catastrophic failure.
Conversely if the application is not in fact actively using that memory then limiting it just because some tool reports that the OS has reserved it then limiting it is pointless.
You must looking for in Sybase Documentation for better performance for your OS and Machine. I believe that it has a configuration file to set JVM Options.
However you can use -Xmx, -Xms and another options parameters you need when to run Sybase executable file.
Look at :