This discussion is archived
5 Replies Latest reply: Jan 11, 2007 12:08 PM by 807573 RSS

Directory 5.2 P4 crashes with virtual memory error.

807573 Oracle ACE
Currently Being Moderated
One of my master (dual master repl) crashed last night with the following errors (sorry it is going to be a little long...):

[05/Jan/2007:02:04:44 -0500] - ERROR<5135> - Resource Limit - conn=-1 op=-1 msgId=-1 - Memory allocation er
ror calloc of 878636 bytes failed; errno 12
The server has probably allocated all available virtual memory. To solve this problem, make more virtual m
emory available to your server, or reduce the size of the server's `Maximum Entries in Cache' (cachesize) o
r `Maximum DB Cache Size' (dbcachesize) parameters.
can't recover; calling exit(1)

[05/Jan/2007:02:04:44 -0500] - ERROR<5135> - Resource Limit - conn=-1 op=-1 msgId=-1 - Memory allocation er
ror calloc of 878636 bytes failed; errno 12
The server has probably allocated all available virtual memory. To solve this problem, make more virtual m
emory available to your server, or reduce the size of the server's `Maximum Entries in Cache' (cachesize) o
r `Maximum DB Cache Size' (dbcachesize) parameters.
can't recover; calling exit(1)

[05/Jan/2007:02:04:44 -0500] - ERROR<5130> - Resource Limit - conn=-1 op=-1 msgId=-1 - Memory allocation er
ror malloc of 1083535 bytes failed; errno 12
The server has probably allocated all available virtual memory. To solve this problem, make more virtual m
emory available to your server, or reduce the size of the server's `Maximum Entries in Cache' (cachesize) o
r `Maximum DB Cache Size' (dbcachesize) parameters.
can't recover; calling exit(1)

[05/Jan/2007:02:04:44 -0500] - ERROR<5130> - Resource Limit - conn=-1 op=-1 msgId=-1 - Memory allocation er
ror malloc of 1083535 bytes failed; errno 12
The server has probably allocated all available virtual memory. To solve this problem, make more virtual m
emory available to your server, or reduce the size of the server's `Maximum Entries in Cache' (cachesize) o
r `Maximum DB Cache Size' (dbcachesize) parameters.
can't recover; calling exit(1)
-----------------------
First it is trying to allocate 870K, then it tries to allocate 1MB.

1. Why does it try more than once? Is that by design?
2. This server runs on an AIX 5.2 server with 4CPUs,8GB od RAM, and 6GB of paging space. Why did this happened? We did not run out of paging space because the whole server would have crashed instead. Is it fair to asume that the available paging space went below 1mb, or 800k from 6GB?

These are my cache configuration values:

dn: cn=changelog,cn=ldbm database,cn=plugins,cn=config
nsslapd-cachesize: -1
nsslapd-cachememsize: 2097152
...

dn: cn=config,cn=ldbm database,cn=plugins,cn=config
nsslapd-dbcachesize: 268435456
nsslapd-import-cachesize: 268435456
nsslapd-search-tune: 57
nsslapd-disk-low-threshold: 100
nsslapd-disk-full-threshold: 10
...

dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
nsslapd-cachesize: -1
nsslapd-cachememsize: 10485760
...

The DS has about 400,000 entries.
The whole database uses about 2.1GB, and id2entry.db3 is about
1.9GB.

Am I looking at a memory leak issue? Is this a documented problem?

The error says to reduce the cache sizes, but they do not seem to be significantly bigger, do they?

I would appreciate any help on this (before it happens again...)

(Does anybody have any word on DS 6???)
  • 1. Re: Directory 5.2 P4 crashes with virtual memory error.
    807573 Oracle ACE
    Currently Being Moderated
    This happens for a simple reason - and it's not that your system is out of memory or out of swap space. The reason is that a 32-bit process has a limited amount of address space (2GB on your AIX system), and all that address space has been allocated.

    There are many things that eat up address space inside a process - program binaries, libraries, stack, heap, memory mapped in files - and in a memory hungry application like the directory server, all the available address space can be chewed up. There's a couple of items you can configure that will control how much static areas the application will allocate - those are the DB cache and entry cache. Otherwise, you can't control how much memory the process will use - that will depend on how many binaries get mapped in, and how much it wants to use in heap and stack space.

    There are two general solutions to your problem - first, reduce the static areas the directory server allocates for itself. Second, upgrade to a 64-bit OS and a 64-bit version of the directory server. I recommend Directory 5.2 on Solaris 10, this will fix your problem today. I have no info on what other OSes 64-bit versions of Directory will ship.
  • 2. Re: Directory 5.2 P4 crashes with virtual memory error.
    807573 Oracle ACE
    Currently Being Moderated
    Directory Server Enterprise Edition 6 will be available as 64bit applications on Solaris 9, 10 Sparc, Solaris 10 x64 (AMD processors) and HP-UX 11i.
    It will be available as 32bit applications on RH Linux 3.something and 4.something, as well as Windows servers.

    Regards,

    Ludovic.
  • 3. Re: Directory 5.2 P4 crashes with virtual memory error.
    807573 Oracle ACE
    Currently Being Moderated
    Thanks Ludovic, I see no support for AIX there... Sun LDAP may be doomed in my company then.
  • 4. Re: Directory 5.2 P4 crashes with virtual memory error.
    807573 Oracle ACE
    Currently Being Moderated
    If I understand what you are saying, Ihave to reduce the db cache, entry caches, etc. Would not that hurt performace of the DS?
    Also, my database is growing, the whole database directory occupies +/- 1.5GB. If that database has to be cached for the DS to run, that will leave me with only 500MB for everything else. Not quite enough, probably.
    Also, what is going to happen when the database reaches 2 or more GB? Will I be able to run the DS in a 32 bit OS?
  • 5. Re: Directory 5.2 P4 crashes with virtual memory error.
    807573 Oracle ACE
    Currently Being Moderated
    We have done some research and found what sems to be a 'memory leak' attributable to our main LDAP instance. We can see that our DS crashed twice approx. 60 days apart, with the infamous virtual memeory error. The memory utilization increased linearly, and consistently during that period. In fact we are sure to predict it is going to happen again in about 10 days --we have bounced the DS two days ago, so now we have another 60 days of grace.

    Meanwhile, does anybody have any experience with this issue on AIX 5.2 - DS 5.2 P4? Is this a documented problem?