3 Replies Latest reply: Nov 9, 2012 9:48 AM by Sbarber.Uk-Oracle RSS

    EM12c Agent getting DISK FULL status

    David Ooi
      Hi Everyone
      I been getting collection DISK FULL in one of our agent even though the agent path still have 90% of freespace.

      Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
      Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
      ---------------------------------------------------------------
      Agent Version : 12.1.0.1.0
      OMS Version : 12.1.0.1.0
      Protocol Version : 12.1.0.1.0
      Agent Home : /opt/oracle/agent/agent12c/agent_inst
      Agent Binaries : /opt/oracle/agent/agent12c/core/12.1.0.1.0
      Agent Process ID : 585
      Parent Process ID : 431
      Agent URL : https://server:3880/emd/main/
      Repository URL : https://OMSserver:4901/empbs/upload
      Started at : 2012-06-28 16:30:18
      Started by user : processoemagent
      Last Reload : (none)
      Last successful upload : 2012-11-07 04:16:59
      Last attempted upload : 2012-11-07 04:16:59
      Total Megabytes of XML files uploaded so far : 2,425.38
      Number of XML files pending upload : 4,901
      Size of XML files pending upload(MB) : 2.94
      Available disk space on upload filesystem    : 90.09%
      *Collection Status                            : [DISK_FULL]*
      Last attempted heartbeat to OMS : 2012-11-08 19:52:30
      Last successful heartbeat to OMS : 2012-11-08 19:52:30

      ---------------------------------------------------------------
      Agent is Running and Ready

      When i edit the gcagent.log for more detail , i found the warning message as below
      *2012-11-07 04:19:36,179 [233:E22D17A1:GC.Executor.0 (rac_database:cmhta.uat.amer.com:Temporary File Status)] WARN - Upload system can't accept inbound message; can't purge*
      *2012-11-07 04:19:36,205 [233:E22D17A1] WARN - Upload failure rac_database.cmhta.uat.amer.com::Temporary File Status*
      oracle.sysman.gcagent.upload.UploadException: Upload system full - can't accept more data
      at oracle.sysman.gcagent.upload.UploadManager.send(UploadManager.java:1828)
      at oracle.sysman.gcagent.upload.UploadManager.send(UploadManager.java:1940)
      at oracle.sysman.gcagent.upload.UploadManager.send(UploadManager.java:1898)
      at oracle.sysman.gcagent.upload.UploadManager.uploadToDataChannel(UploadManager.java:1999)
      at oracle.sysman.gcagent.metadata.impl.collection.CollectionItem$CollectionItemTask.uploadIndividually(CollectionItem.java:1622)
      at oracle.sysman.gcagent.metadata.impl.collection.CollectionItem$CollectionItemTask.run(CollectionItem.java:1422)
      at oracle.sysman.gcagent.task.AbstractTemplateTask.call(AbstractTemplateTask.java:185)
      at oracle.sysman.gcagent.task.AbstractTemplateTask.call(AbstractTemplateTask.java:47)
      at oracle.sysman.gcagent.task.scheduler.DispatchingTaskScheduler$ReschedulingHelper$ReschedulingTask.call(DispatchingTaskScheduler.java:386)
      at oracle.sysman.gcagent.task.scheduler.DispatchingTaskScheduler$ReschedulingHelper$ReschedulingTask.call(DispatchingTaskScheduler.java:348)
      at oracle.sysman.gcagent.task.executor.DiagWrappedTask.call(DiagWrappedTask.java:60)
      at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:548)
      at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:581)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at oracle.sysman.gcagent.task.TaskFutureImpl.run1(TaskFutureImpl.java:340)
      at oracle.sysman.gcagent.task.TaskFutureImpl.run(TaskFutureImpl.java:305)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:139)
      at java.lang.Thread.run(Thread.java:662)

      How do i clear the DISK FULL status?

      Thanks
        • 1. Re: EM12c Agent getting DISK FULL status
          Mnazim-Oracle
          First Please check the space in your machine
          • 2. Re: EM12c Agent getting DISK FULL status
            David Ooi
            Here is the diskspace

            Filesystem Size Used Avail Use% Mounted on
            /dev/mapper/osvg-root.fs
            8.6G 743M 7.4G 9% /
            /dev/mapper/osvg-usr.fs
            5.7G 2.0G 3.4G 38% /usr
            /dev/mapper/osvg-var.fs
            4.8G 419M 4.1G 10% /var
            /dev/mapper/osvg-varlog.fs
            4.8G 768M 3.8G 17% /var/log
            /dev/mapper/osvg-tmp.fs
            3.8G 164M 3.5G 5% /tmp
            /dev/mapper/osvg-opt.fs
            3.8G 2.5G 1.2G 68% /opt
            /dev/mapper/osvg-home.fs
            3.8G 694M 3.0G 19% /home
            /dev/sdb1 99M 33M 61M 35% /boot
            tmpfs 37G 268M 37G 1% /dev/shm
            */dev/mapper/osvg-emagent*
            *9.9G 1000M 8.9G 10% /opt/oracle/agent*

            10% used 8.9G still available

            Edited by: David Ooi on 09-Nov-2012 04:31
            • 3. Re: EM12c Agent getting DISK FULL status
              Sbarber.Uk-Oracle
              I have had this issue before, it might not be [DISK FULL]. It is a generic message that has been fixed in EM12cR2.

              We limit the maximum size of the upload area.

              UploadMaxDiskUsedPct default is 98%
              UploadMaxNumberXML default = 5000

              2% of 5000 is 100 or 5000 - 100 = 4900 (hence at 4901 your in violation of the number of files)