4 Replies Latest reply: Mar 31, 2010 4:39 AM by 807567 RSS

    smpatch: IOException: Not enough space??

    807567
      One of my favorite things about the Sun UC software is that I never know what error I'm going to get next. The endless variety really tweaks my sense of adventure!

      I just did a patch run on 12/01/06 on five Solaris 10 systems via our LPS. True to form, even that run had its problems, but ultimately completed successfully. This week I run a smpatch analyze, which dutifully produces a list of about a half dozen new patches on 4 of the systems, and fails on the 5th, with the following error spewed out to the screen:
      bash-3.00# smpatch analyze
      java.io.IOException: Not enough space
              at java.lang.UNIXProcess.forkAndExec(Native Method)
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
              at java.lang.Runtime.exec(Runtime.java:591)
              at java.lang.Runtime.exec(Runtime.java:429)
              at java.lang.Runtime.exec(Runtime.java:326)
              at com.sun.patchpro.util.CCRCmdLine.exec(CCRCmdLine.java:196)
              at com.sun.patchpro.util.CCRCmdLine.get(CCRCmdLine.java:86)
              at com.sun.patchpro.util.Downloader.connectToURL(Downloader.java:299)
              at com.sun.patchpro.util.CachingDownloader.establishConnection(CachingDownloader.java:584)
              at com.sun.patchpro.util.CachingDownloader.setSourceURL(CachingDownloader.java:274)
              at com.sun.patchpro.util.CachingDownloader.setupCache(CachingDownloader.java:200)
              at com.sun.patchpro.util.CachingDownloader.<init>(CachingDownloader.java:185)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadFile(ServerPatchServiceProvider.java:947)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadEntitlementFile(ServerPatchServiceProvider.java:1019)
              at com.sun.patchpro.server.PatchServerProxy.downloadEntitlementFile(PatchServerProxy.java:210)
              at com.sun.patchpro.cli.PatchServices.getEntitlementList(PatchServices.java:1931)
              at com.sun.patchpro.cli.PatchServices.getEntitledPatchList(PatchServices.java:2064)
              at com.sun.patchpro.cli.PatchServices.displayPatchList(PatchServices.java:783)
              at com.sun.patchpro.cli.PatchServices.main(PatchServices.java:611)
      Thu Dec 07 10:04:43 EST 2006(ERROR)=> com.sun.patchpro.log.SunOSSystemLogger@7e5619 <=Error: Unable to write to syslog file
      Thu Dec 07 10:04:43 EST 2006(ERROR)=> com.sun.patchpro.log.SunOSSystemLogger@7e5619 <=java.io.IOException: Not enough space
              at java.lang.UNIXProcess.forkAndExec(Native Method)
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
              at java.lang.Runtime.exec(Runtime.java:591)
              at java.lang.Runtime.exec(Runtime.java:464)
              at com.sun.patchpro.log.SunOSSystemLogger.print(SunOSSystemLogger.java:93)
              at com.sun.patchpro.log.ApplicationLog.print(ApplicationLog.java:379)
              at com.sun.patchpro.log.PatchProLog.printStackTrace(PatchProLog.java:139)
              at com.sun.patchpro.util.CCRCmdLine.get(CCRCmdLine.java:106)
              at com.sun.patchpro.util.Downloader.connectToURL(Downloader.java:299)
              at com.sun.patchpro.util.CachingDownloader.establishConnection(CachingDownloader.java:584)
              at com.sun.patchpro.util.CachingDownloader.setSourceURL(CachingDownloader.java:274)
              at com.sun.patchpro.util.CachingDownloader.setupCache(CachingDownloader.java:200)
              at com.sun.patchpro.util.CachingDownloader.<init>(CachingDownloader.java:185)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadFile(ServerPatchServiceProvider.java:947)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadEntitlementFile(ServerPatchServiceProvider.java:1019)
              at com.sun.patchpro.server.PatchServerProxy.downloadEntitlementFile(PatchServerProxy.java:210)
              at com.sun.patchpro.cli.PatchServices.getEntitlementList(PatchServices.java:1931)
              at com.sun.patchpro.cli.PatchServices.getEntitledPatchList(PatchServices.java:2064)
              at com.sun.patchpro.cli.PatchServices.displayPatchList(PatchServices.java:783)
              at com.sun.patchpro.cli.PatchServices.main(PatchServices.java:611)
      Thu Dec 07 10:04:43 EST 2006(ERROR)=> com.sun.patchpro.log.SunOSSystemLogger@7e5619 <=Error: Unable to write to syslog file
      Thu Dec 07 10:04:43 EST 2006(ERROR)=> com.sun.patchpro.log.SunOSSystemLogger@7e5619 <=java.io.IOException: Not enough space
              at java.lang.UNIXProcess.forkAndExec(Native Method)
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
              at java.lang.Runtime.exec(Runtime.java:591)
              at java.lang.Runtime.exec(Runtime.java:464)
              at com.sun.patchpro.log.SunOSSystemLogger.print(SunOSSystemLogger.java:93)
              at com.sun.patchpro.log.ApplicationLog.print(ApplicationLog.java:379)
              at com.sun.patchpro.log.PatchProLog.printStackTrace(PatchProLog.java:139)
              at com.sun.patchpro.util.CCRCmdLine.get(CCRCmdLine.java:106)
              at com.sun.patchpro.util.Downloader.connectToURL(Downloader.java:299)
              at com.sun.patchpro.util.CachingDownloader.establishConnection(CachingDownloader.java:584)
              at com.sun.patchpro.util.CachingDownloader.setSourceURL(CachingDownloader.java:274)
              at com.sun.patchpro.util.CachingDownloader.setupCache(CachingDownloader.java:200)
              at com.sun.patchpro.util.CachingDownloader.<init>(CachingDownloader.java:185)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadFile(ServerPatchServiceProvider.java:947)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadEntitlementFile(ServerPatchServiceProvider.java:1019)
              at com.sun.patchpro.server.PatchServerProxy.downloadEntitlementFile(PatchServerProxy.java:210)
              at com.sun.patchpro.cli.PatchServices.getEntitlementList(PatchServices.java:1931)
              at com.sun.patchpro.cli.PatchServices.getEntitledPatchList(PatchServices.java:2064)
              at com.sun.patchpro.cli.PatchServices.displayPatchList(PatchServices.java:783)
              at com.sun.patchpro.cli.PatchServices.main(PatchServices.java:611)
      Thu Dec 07 10:04:43 EST 2006(ERROR)=> com.sun.patchpro.log.SunOSSystemLogger@7e5619 <=Error: Unable to write to syslog file
      Thu Dec 07 10:04:43 EST 2006(ERROR)=> com.sun.patchpro.log.SunOSSystemLogger@7e5619 <=java.io.IOException: Not enough space
              at java.lang.UNIXProcess.forkAndExec(Native Method)
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
              at java.lang.Runtime.exec(Runtime.java:591)
              at java.lang.Runtime.exec(Runtime.java:464)
              at com.sun.patchpro.log.SunOSSystemLogger.print(SunOSSystemLogger.java:93)
              at com.sun.patchpro.log.ApplicationLog.print(ApplicationLog.java:379)
              at com.sun.patchpro.log.PatchProLog.printStackTrace(PatchProLog.java:139)
              at com.sun.patchpro.util.CCRCmdLine.get(CCRCmdLine.java:106)
              at com.sun.patchpro.util.Downloader.connectToURL(Downloader.java:299)
              at com.sun.patchpro.util.CachingDownloader.establishConnection(CachingDownloader.java:584)
              at com.sun.patchpro.util.CachingDownloader.setSourceURL(CachingDownloader.java:274)
              at com.sun.patchpro.util.CachingDownloader.setupCache(CachingDownloader.java:200)
              at com.sun.patchpro.util.CachingDownloader.<init>(CachingDownloader.java:185)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadFile(ServerPatchServiceProvider.java:947)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadEntitlementFile(ServerPatchServiceProvider.java:1019)
              at com.sun.patchpro.server.PatchServerProxy.downloadEntitlementFile(PatchServerProxy.java:210)
              at com.sun.patchpro.cli.PatchServices.getEntitlementList(PatchServices.java:1931)
              at com.sun.patchpro.cli.PatchServices.getEntitledPatchList(PatchServices.java:2064)
              at com.sun.patchpro.cli.PatchServices.displayPatchList(PatchServices.java:783)
              at com.sun.patchpro.cli.PatchServices.main(PatchServices.java:611)
      Thu Dec 07 10:04:43 EST 2006(ERROR)=> com.sun.patchpro.util.CachingDownloader@19ce47a <=java.lang.RuntimeException: Cannot exec /usr/lib/cc-ccr/bin/ccr -g cns.assetid: java.io.IOException: Not enough space
              at com.sun.patchpro.util.CCRCmdLine.exec(CCRCmdLine.java:236)
              at com.sun.patchpro.util.CCRCmdLine.get(CCRCmdLine.java:86)
              at com.sun.patchpro.util.Downloader.connectToURL(Downloader.java:299)
              at com.sun.patchpro.util.CachingDownloader.establishConnection(CachingDownloader.java:584)
              at com.sun.patchpro.util.CachingDownloader.setSourceURL(CachingDownloader.java:274)
              at com.sun.patchpro.util.CachingDownloader.setupCache(CachingDownloader.java:200)
              at com.sun.patchpro.util.CachingDownloader.<init>(CachingDownloader.java:185)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadFile(ServerPatchServiceProvider.java:947)
              at com.sun.patchpro.server.ServerPatchServiceProvider.downloadEntitlementFile(ServerPatchServiceProvider.java:1019)
              at com.sun.patchpro.server.PatchServerProxy.downloadEntitlementFile(PatchServerProxy.java:210)
              at com.sun.patchpro.cli.PatchServices.getEntitlementList(PatchServices.java:1931)
              at com.sun.patchpro.cli.PatchServices.getEntitledPatchList(PatchServices.java:2064)
              at com.sun.patchpro.cli.PatchServices.displayPatchList(PatchServices.java:783)
              at com.sun.patchpro.cli.PatchServices.main(PatchServices.java:611)
      It's not a simple matter of "no space":
      bash-3.00# df -h
      Filesystem             size   used  avail capacity  Mounted on
      /dev/dsk/c1t0d0s0       11G   4.8G   5.7G    46%    / 
      /devices                 0K     0K     0K     0%    /devices
      ctfs                     0K     0K     0K     0%    /system/contract
      proc                     0K     0K     0K     0%    /proc
      mnttab                   0K     0K     0K     0%    /etc/mnttab
      swap                   513M   1.0M   512M     1%    /etc/svc/volatile
      objfs                    0K     0K     0K     0%    /system/object
      fd                       0K     0K     0K     0%    /dev/fd
      /dev/dsk/c1t0d0s5      5.8G   1.6G   4.1G    29%    /var
      swap                   514M   2.4M   512M     1%    /tmp
      swap                   512M    56K   512M     1%    /var/run
      /dev/dsk/c1t1d0s0       30G   5.4G    24G    19%    /home
      /dev/dsk/c1t1d0s1       30G   5.4G    24G    19%    /opt/local
      /dev/md/dsk/d10        269G   7.0G   260G     3%    /oracle
      Looks like plenty of space to me. I did a "touch /var/foo" just to make sure the filesystem was writable. It worked as expected. So I'm kindof at a loss here.

      Here's the "smpatch get" output:
      bash-3.00# smpatch get
      patchpro.backout.directory      -               ""
      patchpro.baseline.directory     -               /var/sadm/spool
      patchpro.download.directory     -               /var/sadm/spool
      patchpro.install.types          -               rebootafter:reconfigafter:standard
      patchpro.patch.source           http://OBFUSCATED:3816/solaris/       https://getupdates1.sun.com/
      patchpro.patchset               -               current
      patchpro.proxy.host             -               ""
      patchpro.proxy.passwd           ****            ****
      patchpro.proxy.port             -               8080
      patchpro.proxy.user             -               ""
      More config stuff...
      bash-3.00# /usr/lib/cc-ccr/bin/ccr -k | xargs -t -n 1 /usr/lib/cc-ccr/bin/ccr -g
      /usr/lib/cc-ccr/bin/ccr -g 17
      Property not defined: 17
      /usr/lib/cc-ccr/bin/ccr -g cns.assetid
      
      /usr/lib/cc-ccr/bin/ccr -g cns.ccr.keyGenPath
      /usr/lib/cc-ccr/bin/ccrKeyGen
      /usr/lib/cc-ccr/bin/ccr -g cns.clientid
      
      /usr/lib/cc-ccr/bin/ccr -g cns.httpproxy.auth
      
      /usr/lib/cc-ccr/bin/ccr -g cns.httpproxy.ipaddr
      
      /usr/lib/cc-ccr/bin/ccr -g cns.httpproxy.port
      
      /usr/lib/cc-ccr/bin/ccr -g cns.regtoken
      
      /usr/lib/cc-ccr/bin/ccr -g cns.security.password
      
      /usr/lib/cc-ccr/bin/ccr -g cns.security.privatekey
      
      /usr/lib/cc-ccr/bin/ccr -g cns.security.publickey
      
      /usr/lib/cc-ccr/bin/ccr -g cns.swup.UMautolaunch
      false
      /usr/lib/cc-ccr/bin/ccr -g cns.swup.autoAnalysis.enabled
      true
      /usr/lib/cc-ccr/bin/ccr -g cns.swup.checkinInterval
      2
      /usr/lib/cc-ccr/bin/ccr -g cns.swup.lastCheckin
      0
      /usr/lib/cc-ccr/bin/ccr -g cns.swup.patchbaseline
      current
      /usr/lib/cc-ccr/bin/ccr -g cns.swup.regRequired
      true
      /usr/lib/cc-ccr/bin/ccr -g cns.transport.serverurl
      ...and...
      bash-3.00# java -version
      java version "1.5.0_09"
      Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
      Java HotSpot(TM) Server VM (build 1.5.0_09-b03, mixed mode)
      Help please?!?

      -cw-
        • 1. Re: smpatch: IOException: Not enough space??
          807567
          Hi

          Please have a check in the cache dir on the client and on the Update Connection Proxy. Check if it is also writeable. Try freeing up some of that particular filesystem, rather than just /var/foo in case there is a cache size limit reached somewhere.

          Let us know.
          • 2. Re: smpatch: IOException: Not enough space??
            807567
            I don't think this is an updatemanager problem. Quite simply UM tried to exec another process and that failed. The OS execvp method returns ENOMEM and that translates to "Not enough space".

            In other words there was not enough memory available on the system to the exec the new process.
            • 3. Re: smpatch: IOException: Not enough space??
              807567
              I swear I disabled it in crontab but days
              after an update to S10U8, its back produding
              those perplexing messages including the mysterious
              "no space" one at one this morning.

              Mar 31 01:06:40 hostnamename root: => com.sun.patchpro.util.CCRCmdLine@1318b <=java.l
              ang.RuntimeException: Cannot exec /usr/lib/cc-ccr/bin/ccr -g cns.assetid: java.i
              o.IOException: Cannot run program "/usr/lib/cc-ccr/bin/ccr": error=12, Not enoug
              h space
              Mar 31 01:06:40 hostname at com.sun.patchpro.util.CCRCmdLine.exec(CCRCmdLine.java
              :236)
              Mar 31 01:06:40 hostname at com.sun.patchpro.util.CCRCmdLine.get(CCRCmdLine.java:
              86)
              Mar 31 01:06:40 hostname at com.sun.patchpro.util.Downloader.connectToURL(Downloa
              der.java:303)
              Mar 31 01:06:40 hostname at com.sun.patchpro.util.CachingDownloader.establishConn
              ection(CachingDownloader.java:618)
              Mar 31 01:06:40 hostnamename root: => com.sun.patchpro.util.CCRCmdLine@1318b <=
              at com.sun.patchpro.util.CachingDownloader.setSourceURL(CachingDownloader.java:2
              82)























































              # df -k
              Filesystem kbytes used avail capacity Mounted on
              rpool/ROOT/s10s_u8wos_08a
              70189056 15822749 47084061 26% /
              /devices 0 0 0 0% /devices
              ctfs 0 0 0 0% /system/contract
              proc 0 0 0 0% /proc
              mnttab 0 0 0 0% /etc/mnttab
              swap 1012096 448 1011648 1% /etc/svc/volatile
              objfs 0 0 0 0% /system/object
              sharefs 0 0 0 0% /etc/dfs/sharetab
              /platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1
              62906810 15822749 47084061 26% /platform/sun4u-us3/lib/libc_psr.so.1
              /platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
              62906810 15822749 47084061 26% /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
              fd 0 0 0 0% /dev/fd
              swap 1013248 1600 1011648 1% /tmp
              swap 1011680 32 1011648 1% /var/run
              rpool/export 70189056 22 47084061 1% /export
              rpool/export/home 70189056 415306 47084061 1% /export/home
              rpool 70189056 23 47084061 1% /rpool
              rpool/ROOT 70189056 18 47084061 1% /rpool/ROOT
              /vol/dev/dsk/c0t0d0/sol_10_1009_sparc
              2645420 2645420 0 100% /cdrom/sol_10_1009_sparc
              [ snipped /export home remote server details ]


              So, like the OP, don't believe "no space" to be disk.


              # vmstat
              kthr memory page disk faults cpu
              r b w swap free re mf pi po fr de sr s1 s1 s1 -- in sy cs us sy id
              0 0 0 1009232 1474480 5 20 20 0 0 0 0 2 2 2 0 841 1287 675 1 1 98
              # vmstat -S
              kthr memory page disk faults cpu
              r b w swap free si so pi po fr de sr s1 s1 s1 -- in sy cs us sy id
              0 0 0 1009232 1474480 0 0 20 0 0 0 0 2 2 2 0 841 1287 675 1 1 98
              # vmstat -s
              0 swap ins
              0 swap outs
              0 pages swapped in
              0 pages swapped out
              4718185 total address trans. faults taken
              172379 page ins
              12 page outs
              579625 pages paged in
              96 pages paged out
              1284338 total reclaims
              1284338 reclaims from free list
              0 micro (hat) faults
              4718185 minor (as) faults
              172379 major faults
              1278653 copy-on-write faults
              1782666 zero fill page faults
              51196 pages examined by the clock daemon
              0 revolutions of the clock hand
              96 pages freed by the clock daemon
              36062 forks
              1640 vforks
              35414 execs
              157653006 cpu context switches
              196398551 device interrupts
              5933003 traps
              300665895 system calls
              25760529 total name lookups (cache hits 100%)
              435886 user cpu
              446240 system cpu
              45843369 idle cpu
              0 wait cpu


              # prstat
              PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
              7784 alan 188M 111M sleep 59 0 0:00:05 6.2% soffice-9.0.bin/7
              7053 michael 223M 140M sleep 49 0 0:26:01 0.5% acroread/1
              1696 michael 115M 71M sleep 59 0 0:14:54 0.2% soffice-5.2.bin/179
              7772 michael 3856K 3208K cpu1 59 0 0:00:00 0.1% prstat/1
              5415 alan 67M 38M sleep 59 0 0:03:30 0.1% soffice-5.2.bin/4
              2834 daemon 6528K 3536K sleep 59 0 0:00:12 0.1% fetchmail/1

              [ snipped < 0.1% CPU ]

              # top
              load averages: 0.11, 0.05, 0.04; up 2+16:49:24 09:39:41
              119 processes: 117 sleeping, 1 zombie, 1 on cpu
              CPU states: 97.5% idle, 1.0% user, 1.5% kernel, 0.0% iowait, 0.0% swap
              Memory: 4096M phys mem, 1436M free mem

              PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
              7789 michael 1 59 0 3448K 1968K cpu/1 0:00 0.89% top
              7053 michael 1 59 0 223M 140M sleep 26:01 0.51% acroread
              1696 michael 13 59 0 115M 71M sleep 14:54 0.17% soffice-5.2.bin
              5415 alan 4 59 0 67M 38M sleep 3:30 0.14% soffice-5.2.bin
              7784 alan 7 59 0 188M 111M sleep 0:05 0.09% soffice-9.0.bin
              2077 alan 1 59 0 12M 2128K sleep 4:11 0.07% xclock
              1594 alan 1 59 0 12M 2400K sleep 2:37 0.04% xclock
              557 root 1 59 0 9928K 1280K sleep 0:02 0.03% snmpd
              1029 root 14 0 19 77M 29M sleep 2:12 0.02% java
              746 root 42 29 10 77M 30M sleep 2:03 0.02% java
              936 noaccess 18 59 0 210M 112M sleep 4:04 0.01% java
              1595 alan 1 59 0 4808K 1680K sleep 0:04 0.01% xload
              7 root 13 59 0 11M 3152K sleep 0:06 0.01% svc.startd
              561 root 1 59 0 8616K 1632K sleep 0:05 0.01% sendmail
              488 root 1 100 -20 3192K 1248K sleep 0:34 0.01% xntpd

              [ snipped < 0.1% CPU ]

              So there's nothing interesting running continuously
              that might have running at the time.
              Here's the crontab:

              # The root crontab should be used to perform accounting data collection.
              #
              #
              # min hour day month w/day Command
              # 0-59 0-23 1-31 1-12 0-6
              #
              # As Distributed
              #
              10 3 * * * /usr/sbin/logadm
              15 3 * * 0 /usr/lib/fs/nfs/nfsfind
              30 3 * * * [ -x /usr/lib/gss/gsscred_clean ] && /us
              r/lib/gss/gsscred_clean
              #06 1 * * * /usr/lib/patch/swupAuto > /dev/null 2>&1
              #10 3 * * * /usr/lib/krb5/kprop_script ___slave_kdcs
              ___
              #
              # Local Additions
              11 03 * * 0,4 [ -x /usr/local/bin/respawnwindowlog ] &
              & /usr/local/bin/respawnwindowlog 2>&1 > /dev/null

              0 02 1 * * /opt/SUNWexplo/bin/explorer -q -e -P # SUNWexplo
              :q!
              The crontab file was not changed.

              Can anyone please explain what SUC is on about with it's unable
              to exec - "no space".

              One last thing it was trying to exec is
              executable:

              # ls -l /usr/lib/cc-ccr/bin/ccr
              -rwxr-xr-x 1 root root 58196 Nov 18 2005 /usr/lib/cc-ccr/bin/ccr

              TIA,

              Suntan.
              • 4. Re: smpatch: IOException: Not enough space??
                807567
                Check for patch 121081-08 (sparc) or 121082-08 (x86) - if it isn't installed then install it. Check for and kill any ccr processes that are still running:

                # ps -ef | grep ccr

                Use `# kill -9 <pid>` to get rid of them.

                Having said that, I don't think that the issue you are currently seeing is related to the original issue in this thread so would suggest that you start a new thread if you are still seeing problems. I would ask that you run our [data collection script|https://supportuploads.sun.com/download?directory=downloads&file=suc%2esh] and upload the output to our [Support Files|https://supportuploads.sun.com/] site when you do. Remember to reference this thread if you start a new one.