0 Replies Latest reply: Jul 7, 2011 9:24 AM by 847261 RSS

    rpool full on one server but not on two other identical servers

    847261
      Hi,
      Tester reported the issue below.
      The question is, why is it not happening on the other two servers, which are at different sites, but running the same tasks with the same software.
      Any help appreciated ....
      Thanks!
      /Enda.

      ======

      The lack of updates in the Solaris Patches could cause some problems in the running systems.

      I.e.: In the live server we run into a situation in which the rpool got full. This was caused because the /var/fm/fmd directory was full of logs. As the rpool was 100% full no actions could be taken (snapshots cleanup, files removal, etc). The only way of solving this situation was to move the files from this directory to a new directory in the zpool.

      root@akr1e1nav/var/fm/fmd> ll -lrth
      total 55394206
      drwxr-xr-x 3 root sys 3 Mar 23 17:00 ..
      drwx------ 2 root sys 2 Mar 23 17:00 ckpt
      drwx------ 2 root sys 2 Mar 23 17:00 rsrc
      drwx------ 2 root sys 2 Mar 23 17:16 xprt
      -rw-r--r-- 1 root root 341 Mar 23 17:32 fltlog
      -rw-r--r-- 1 root root 457M May 23 03:09 errlog.10
      -rw-r--r-- 1 root root 457M May 24 03:09 errlog.9
      -rw-r--r-- 1 root root 460M May 25 03:09 errlog.8
      -rw-r--r-- 1 root root 462M May 26 03:09 errlog.7
      -rw-r--r-- 1 root root 459M May 27 03:10 errlog.6
      -rw-r--r-- 1 root root 466M May 28 03:09 errlog.5
      -rw-r--r-- 1 root root 470M May 29 03:09 errlog.4
      -rw-r--r-- 1 root root 469M May 30 03:09 errlog.3
      -rw-r--r-- 1 root root 457M May 31 03:09 errlog.2
      -rw-r--r-- 1 root root 19G Jun 7 03:09 errlog.1
      drwxr-xr-x 5 root sys 17 Jun 8 03:10 .
      -rw-r--r-- 1 root root 3.2G Jun 13 13:49 errlog

      root@akr1e1nav/var/fm/fmd> du -sh .
      26G .

      root@akr1e1nav/var/fm/fmd> fmstat
      module ev_recv ev_acpt wait svc_t %w %b open solve memsz bufsz
      cpumem-retire 0 0 0.0 1.8 0 0 0 0 0 0
      disk-transport 0 0 1.0 3290721.0 100 0 0 0 32b 0
      eft 0 0 0.0 1.7 0 0 0 0 1.1M 0
      fabric-xlate 4683 0 0.0 15.4 0 0 0 0 0 0
      fmd-self-diagnosis 670129816 0 82.2 2.5 37 0 0 0 0 0
      io-retire 0 0 0.0 1.7 0 0 0 0 0 0
      snmp-trapgen 0 0 0.0 1.8 0 0 0 0 32b 0
      sp-monitor 0 0 0.0 3.0 0 0 0 0 20b 0
      sysevent-transport 0 0 0.0 1021.0 0 0 0 0 0 0
      syslog-msgs 0 0 0.0 1.8 0 0 0 0 0 0
      zfs-diagnosis 0 0 0.0 2.2 0 0 0 0 0 0
      zfs-retire 0 0 0.0 1.8 0 0 0 0 0 0


      The problems reported (more detailed at the end of the message):

      root@akr1e1nav/var/fm/fmd> fmdump -e -v
      TIME CLASS ENA
      May 22 03:10:04.2516 ereport.io.ddi.fm-capability 0x00b7ad1f74700001
      May 22 03:10:04.2517 ereport.io.ddi.fm-capability 0x00b7ad3a50904801
      May 22 03:10:04.2527 ereport.io.ddi.fm-capability 0x00b7ae3605b00001
      May 22 03:10:04.2528 ereport.io.ddi.fm-capability 0x00b7ae3fc3b00001
      May 22 03:10:13.0374 ereport.io.ddi.fm-capability 0x00d867e651301401
      May 22 03:10:13.0375 ereport.io.ddi.fm-capability 0x00d8680725b07801
      May 22 03:10:13.0386 ereport.io.ddi.fm-capability 0x00d86902f2000001

      [...]

      Jun 13 14:14:03.8668 ereport.io.ddi.fm-capability 0x518e39497c001401
      Jun 13 14:14:03.8824 ereport.io.ddi.fm-capability 0x518e4828af609401
      Jun 13 14:14:03.8958 ereport.io.ddi.fm-capability 0x518e54fc57206c01
      Jun 13 14:14:03.9031 ereport.io.ddi.fm-capability 0x518e5beb6bb01401
      Jun 13 14:14:03.9196 ereport.io.ddi.fm-capability 0x518e6b9df850ac01
      Jun 13 14:14:03.9311 ereport.io.ddi.fm-capability 0x518e7698a1002c01
      Jun 13 14:14:03.9425 ereport.io.ddi.fm-capability 0x518e817ef2301401
      Jun 13 14:14:03.9539 ereport.io.ddi.fm-capability 0x518e8c50df801801
      Jun 13 14:14:03.9660 ereport.io.ddi.fm-capability 0x518e97ee41506401
      Jun 13 14:14:03.9775 ereport.io.ddi.fm-capability 0x518ea2d758b02001

      Checking on the internet I have seen it is a popular problem that is fixed with a patch:

      http://appsdbaworkshop.blogspot.com/2011/03/root-file-system-full-ereportioddifm.html