9 Replies Latest reply: Mar 22, 2010 2:40 PM by 807559 RSS

    Problem with "swap space limit exceeded"

    807559
      Hi, I'm having a problem with swap memory space.

      A have a box with Solaris 10 8/07 s10x_u4wos_12b X86, and there, there is a zone running a Web Server Apache.

      But in a non-specific cicle time, my solaris zone, and sometimes global zone too, stop, and i got the following messages in dmesg:

      Apr 7 21:42:18 aga253distp209 genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 18718 (httpd)
      Apr 7 21:42:18 aga253distp209 last message repeated 1 time
      Apr 8 07:54:28 aga253distp209 tmpfs: [ID 518458 kern.warning] WARNING: /zonas/sitesoutros/root/etc/svc/volatile: File system full, swap space limit exceeded
      Apr 8 07:54:28 aga253distp209 last message repeated 1 time
      Apr 8 07:59:28 aga253distp209 tmpfs: [ID 518458 kern.warning] WARNING: /zonas/sitesoutros/root/etc/svc/volatile: File system full, swap space limit exceeded
      Apr 8 07:59:28 aga253distp209 last message repeated 1 time
      Apr 8 08:04:28 aga253distp209 tmpfs: [ID 518458 kern.warning] WARNING: /zonas/sitesoutros/root/etc/svc/volatile: File system full, swap

      I cant understand why this happen if i have much memory available.

      [root@aga253distp209:/] # swap -l
      swapfile dev swaplo blocks free
      /dev/dsk/c1t0d0s1 59,65 8 16787912 16123384

      [root@aga253distp209:/] # df -h | grep swap
      swap 6,8G 992K 6,8G 1% /etc/svc/volatile
      swap 6,8G 44K 6,8G 1% /tmp
      swap 6,8G 36K 6,8G 1% /var/run

      So, how can i find out the root cause of the problem or use dtrace for that ? I dont know very much of dtrace.

      Regards.
      Kleyson Rios.
        • 1. Re: Problem with "swap space limit exceeded"
          807559
          Are you writing logs to /tmp ?

          This is a common cause of 'out of swap' errors as when /tmp fills up, so does swap, when you reboot the system, it fixes it as /tmp is cleared by default.

          Alternatively if something is writing to /tmp and it fails due to filesystem full the file will be removed and so you won't see any evidence that it was there.

          Try keeping a track of the contents of /tmp and especially any files that are large and growing - that might point you in the right direction.
          • 2. Re: Problem with "swap space limit exceeded"
            807559
            smurf4568

            But how my /tmp or swap area fills up if i have a lot of swap space free ? Look.

            root@aga253distp209:/ # swap -l
            swapfile dev swaplo blocks free
            /dev/dsk/c1t0d0s1 59,65 8 16787912 *16123384*

            root@aga253distp209:/ # df -h | grep swap
            swap 6,8G 992K 6,8G *1%* /etc/svc/volatile
            swap 6,8G 44K 6,8G *1%* /tmp
            swap 6,8G 36K 6,8G *1%* /var/run

            1% of memory used !!!
            • 3. Re: Problem with "swap space limit exceeded"
              user4994457
              Presumably it's a temporary error and the memory is returned shortly afterward. Unless you're inspecting the system when the error occurs (or simply inspecting it constantly), then noting that swap space is available right now doesn't help.

              You may want to set up a script to capture 'ps -o pid,vsz,args' or 'swap -s' output periodically. Then you could compare that output to the time when the limit error occurs.

              --
              Darren
              • 4. Re: Problem with "swap space limit exceeded"
                807559
                Darren

                The informations about swap that i showed, i got it at the moment of the problem.
                • 5. Re: Problem with "swap space limit exceeded"
                  user4994457
                  I can only imagine that the size request is happening so quickly that it is difficult to catch by polling.

                  Perhaps a probe to watch all memory allocation/deallocations would be a good start.

                  --
                  Darren
                  • 6. Re: Problem with "swap space limit exceeded"
                    807559
                    Darren

                    If the problem occurs so quickly, why i need reboot my server for it get working again ?

                    And what's the bestter way to watch all memory allocation and deallocation ?

                    Regards.
                    Kleyson Rios.
                    • 7. Re: Problem with "swap space limit exceeded"
                      user4994457
                      KleysonRios wrote:
                      Darren

                      If the problem occurs so quickly, why i need reboot my server for it get working again ?
                      I didn't see from your post that the memory issue continues. What is the behavior that you see? Does it continue to log limit exceeded messages to log files? What does 'swap -s' show when the log messages are being generated?

                      If you run 'vmstat 5' simultaneously, do you see the page scanner kick in? Is there any fluctuation in memory over that time?
                      And what's the bestter way to watch all memory allocation and deallocation ?
                      That's a good question. The 'vminfo' provider has a lot of probes that fire on page allocation/deallocation, but that might not be the best level to view things. If this is because of a user process and not the kernel, you could probe process allocations at either the 'malloc' or 'sbrk'

                      You might also want to ask on the dtrace discussion board. That tends to be an active area.

                      --
                      Darren
                      • 8. Re: Problem with "swap space limit exceeded"
                        807559
                        Hi,

                        Still you have same problem ? What is the solution ?

                        -Jay
                        • 9. Re: Problem with "swap space limit exceeded"
                          807559
                          I recently had the same errors with Apache - turned out we run a 32bit version (1.3.41) and it blows whenever a log file exceeds 2Gb.