can I create an in memory application without using set_catchsize?
set_cachesize has nothing to do with creating an in-memory application or not. It configures the size of the cache ( http://docs.oracle.com/cd/E17076_03/html/programmer_reference/general_am_conf.html#am_conf_cachesize ), which is in memory. In order to learn how to create an in-memory application, please read: http://docs.oracle.com/cd/E17076_03/html/programmer_reference/program_ram.html
does it possible to control the cache size at runtime?
Yes, by using
DB_ENV->set_cachesize(). The supplied size will be rounded to the nearest multiple of the region size and may not be larger than the maximum size configured with DB_ENV->set_cache_max().
DB_ENV->set_cachesize - http://docs.oracle.com/cd/E17076_03/html/api_reference/C/envset_cachesize.html