This discussion is archived
9 Replies Latest reply: Mar 22, 2010 12:40 PM by 807559 RSS

Problem with "swap space limit exceeded"

807559 Newbie
Currently Being Moderated
Hi, I'm having a problem with swap memory space.

A have a box with Solaris 10 8/07 s10x_u4wos_12b X86, and there, there is a zone running a Web Server Apache.

But in a non-specific cicle time, my solaris zone, and sometimes global zone too, stop, and i got the following messages in dmesg:

Apr 7 21:42:18 aga253distp209 genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 18718 (httpd)
Apr 7 21:42:18 aga253distp209 last message repeated 1 time
Apr 8 07:54:28 aga253distp209 tmpfs: [ID 518458 kern.warning] WARNING: /zonas/sitesoutros/root/etc/svc/volatile: File system full, swap space limit exceeded
Apr 8 07:54:28 aga253distp209 last message repeated 1 time
Apr 8 07:59:28 aga253distp209 tmpfs: [ID 518458 kern.warning] WARNING: /zonas/sitesoutros/root/etc/svc/volatile: File system full, swap space limit exceeded
Apr 8 07:59:28 aga253distp209 last message repeated 1 time
Apr 8 08:04:28 aga253distp209 tmpfs: [ID 518458 kern.warning] WARNING: /zonas/sitesoutros/root/etc/svc/volatile: File system full, swap

I cant understand why this happen if i have much memory available.

[root@aga253distp209:/] # swap -l
swapfile dev swaplo blocks free
/dev/dsk/c1t0d0s1 59,65 8 16787912 16123384

[root@aga253distp209:/] # df -h | grep swap
swap 6,8G 992K 6,8G 1% /etc/svc/volatile
swap 6,8G 44K 6,8G 1% /tmp
swap 6,8G 36K 6,8G 1% /var/run

So, how can i find out the root cause of the problem or use dtrace for that ? I dont know very much of dtrace.

Regards.
Kleyson Rios.
  • 1. Re: Problem with "swap space limit exceeded"
    807559 Newbie
    Currently Being Moderated
    Are you writing logs to /tmp ?

    This is a common cause of 'out of swap' errors as when /tmp fills up, so does swap, when you reboot the system, it fixes it as /tmp is cleared by default.

    Alternatively if something is writing to /tmp and it fails due to filesystem full the file will be removed and so you won't see any evidence that it was there.

    Try keeping a track of the contents of /tmp and especially any files that are large and growing - that might point you in the right direction.
  • 2. Re: Problem with "swap space limit exceeded"
    807559 Newbie
    Currently Being Moderated
    smurf4568

    But how my /tmp or swap area fills up if i have a lot of swap space free ? Look.

    root@aga253distp209:/ # swap -l
    swapfile dev swaplo blocks free
    /dev/dsk/c1t0d0s1 59,65 8 16787912 *16123384*

    root@aga253distp209:/ # df -h | grep swap
    swap 6,8G 992K 6,8G *1%* /etc/svc/volatile
    swap 6,8G 44K 6,8G *1%* /tmp
    swap 6,8G 36K 6,8G *1%* /var/run

    1% of memory used !!!
  • 3. Re: Problem with "swap space limit exceeded"
    user4994457 Newbie
    Currently Being Moderated
    Presumably it's a temporary error and the memory is returned shortly afterward. Unless you're inspecting the system when the error occurs (or simply inspecting it constantly), then noting that swap space is available right now doesn't help.

    You may want to set up a script to capture 'ps -o pid,vsz,args' or 'swap -s' output periodically. Then you could compare that output to the time when the limit error occurs.

    --
    Darren
  • 4. Re: Problem with "swap space limit exceeded"
    807559 Newbie
    Currently Being Moderated
    Darren

    The informations about swap that i showed, i got it at the moment of the problem.
  • 5. Re: Problem with "swap space limit exceeded"
    user4994457 Newbie
    Currently Being Moderated
    I can only imagine that the size request is happening so quickly that it is difficult to catch by polling.

    Perhaps a probe to watch all memory allocation/deallocations would be a good start.

    --
    Darren
  • 6. Re: Problem with "swap space limit exceeded"
    807559 Newbie
    Currently Being Moderated
    Darren

    If the problem occurs so quickly, why i need reboot my server for it get working again ?

    And what's the bestter way to watch all memory allocation and deallocation ?

    Regards.
    Kleyson Rios.
  • 7. Re: Problem with "swap space limit exceeded"
    user4994457 Newbie
    Currently Being Moderated
    KleysonRios wrote:
    Darren

    If the problem occurs so quickly, why i need reboot my server for it get working again ?
    I didn't see from your post that the memory issue continues. What is the behavior that you see? Does it continue to log limit exceeded messages to log files? What does 'swap -s' show when the log messages are being generated?

    If you run 'vmstat 5' simultaneously, do you see the page scanner kick in? Is there any fluctuation in memory over that time?
    And what's the bestter way to watch all memory allocation and deallocation ?
    That's a good question. The 'vminfo' provider has a lot of probes that fire on page allocation/deallocation, but that might not be the best level to view things. If this is because of a user process and not the kernel, you could probe process allocations at either the 'malloc' or 'sbrk'

    You might also want to ask on the dtrace discussion board. That tends to be an active area.

    --
    Darren
  • 8. Re: Problem with "swap space limit exceeded"
    807559 Newbie
    Currently Being Moderated
    Hi,

    Still you have same problem ? What is the solution ?

    -Jay
  • 9. Re: Problem with "swap space limit exceeded"
    807559 Newbie
    Currently Being Moderated
    I recently had the same errors with Apache - turned out we run a 32bit version (1.3.41) and it blows whenever a log file exceeds 2Gb.