1 Reply Latest reply on Feb 20, 2012 9:31 PM by lrp1

    File verification errors applying GI PSU 11.2.0.3.1 for acfs on linux

    lrp1
      (Hi guys. i'm filing an SR at the same time, but I find I get more helpful responses from the forums, and we end up sharing technical knowledge in a more effective way.)

      Platform: Proof of concept Virtualbox VM running Oracle Linux 5u7
      Installed Grid Infrastructure 11.2.0.3 straight (didn't start with 11.2.0.1 and upgraded out-of-place)
      My Linux kernel was apparently not supporting
      root@oel4::/root] uname -r
      2.6.32-200.13.1.el5uek
      root@oel4::/root] uname -a
      Linux oel4 2.6.32-200.13.1.el5uek #1 SMP Wed Jul 27 21:02:33 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
      
      oracle@oel4 [+ASM] Mon Feb 13:/orafs1/app/grid/product/11.2.0.3/grid/bin
      22:51:14] oracleasm configure
      ORACLEASM_ENABLED=true
      ORACLEASM_UID=oracle
      ORACLEASM_GID=dba
      ORACLEASM_SCANBOOT=true
      ORACLEASM_SCANORDER=""
      ORACLEASM_SCANEXCLUDE=""
      In the ASMCA utility, I do NOT see a valid tab for the ADVM drive or ACFS filesystem. I could ONLY create ASM diskgroups. In addition, when trying to check whether the ACFS / ADVM drivers are even installed, tI see they are NOT:
      --
      Error encountered:
      oracle@oel4 [+ASM] Mon Feb 13:/orafs1/app/grid/product/11.2.0.3/grid/bin
      23:05:08] ./acfsdriverstate installed
      ACFS-9204: false
      
      oracle@oel4 [+ASM] Mon Feb 13:/orafs1/app/grid/product/11.2.0.3/grid/bin> ./acfsdriverstate supported
      ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.32-200.13.1.el5uek'
      ACFS-9201: Not Supported
      -- I have already checked the following articles:
      +ACFS tools fail with ADVM/ACFS is not supported on [ID 1085146.1]+
      +IS ACFS/ADVM SUPPORTED/CERTIFIED ON SOLARIS SPARC 64 PLATFORM? [ID 973387.1]+
      +How to Verify that the ACFS Driver is installed and loaded [ID 1319263.1]+

      - out of curiosity, would I be able to boot the GRUB-loader WITHOUT the linux UEK kernel, and instead boot with the
      *2.6.18-274.el5 kernel*? I tried it and ACFS now worked under 11.2.0.3 (when loading via ./acfsroot install)I wasn't sure of the impact to other application, and I assume there ARE risks in doing this.

      -- I had thought that as of *11.2.0.3*, ACFS WAS supported under the Linux UEK. In addition, the MOS note 1369107.1 that 11.2.0.3.1 patch set update 1 would provide support for ACFS on the UEK2.6.32-200
      "Support for "2.6.32-200" kernels will start in the first GI PSU 11.2.0.3.1 (11.2.0.3 GIPSU #1), (RFI Bug:13241736 & Bug:12825835). "

      To test that, I tried applying the 11.2.0.3.1 grid control PSU 1:
      ## get existing opatch config:
           /orafs1/app/grid/product/11.2.0/grid/OPatch/opatch lsinventory
                Oracle Home       : /orafs1/app/grid/product/11.2.0/grid
                Central Inventory : /orafs1/app/oraInventory
                   from           : /orafs1/app/grid/product/11.2.0/grid/oraInst.loc
                OPatch version    : 11.2.0.1.9
                OUI version       : 11.2.0.3.0
                Log file location : /orafs1/app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2012-02-18_12-20-11PM.log
                Lsinventory Output file location : /orafs1/app/grid/product/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2012-02-18_12-20-11PM.txt
                --------------------------------------------------------------------------------
                Installed Top-level Products (1):
                Oracle Grid Infrastructure                                           11.2.0.3.0
                There are 1 products installed in this Oracle Home.
                There are no Interim patches installed in this Oracle Home.
                --------------------------------------------------------------------------------
                OPatch succeeded.
      
      ## run conflict resolution checks:
      
      $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 13343438 -oh /orafs1/app/grid/product/11.2.0/grid
      
           PREREQ session
           Oracle Home       : /orafs1/app/grid/product/11.2.0/grid
           Central Inventory : /orafs1/app/oraInventory
              from           : /orafs1/app/grid/product/11.2.0/grid/oraInst.loc
           OPatch version    : 11.2.0.1.9
           OUI version       : 11.2.0.3.0
           Log file location : /orafs1/app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2012-02-18_12-17-54PM.log
           Invoking prereq "checkconflictagainstohwithdetail"
           Prereq "checkConflictAgainstOHWithDetail" passed.
           OPatch succeeded.
      
      $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 13348650 -oh /orafs1/app/grid/product/11.2.0/grid
           PREREQ session
           Oracle Home       : /orafs1/app/grid/product/11.2.0/grid
           Central Inventory : /orafs1/app/oraInventory
              from           : /orafs1/app/grid/product/11.2.0/grid/oraInst.loc
           OPatch version    : 11.2.0.1.9
           OUI version       : 11.2.0.3.0
           Log file location : /orafs1/app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2012-02-18_12-18-06PM.log
           Invoking prereq "checkconflictagainstohwithdetail"
           Prereq "checkConflictAgainstOHWithDetail" passed.
           OPatch succeeded.
      I had checked the MOS Oracle Database Patch Set Update 11.2.0.3.1 Known Issues [ID 1374706.1] article, which didn't mention any show stoppers related to me.
      # copy patch to tmp dir instead and run from there
           su - oracle
           cd /tmp
           mkdir p13348650
           unzip /mnt/share/100g/downloads/oracle/gi_11.2.0.3.1_psu1/p13348650_112030_Linux-x86-64.zip -d p13348650
           cp -p /mnt/share/100g/downloads/oracle/gi_11.2.0.3.1_psu1/ocm.rsp .
           ls -l
      
                -rwxrwxrwx  1 root    michael       1069 Feb 16 16:55 ocm.rsp
                drwxr-xr-x  4 oracle  oinstall      4096 Feb 18 08:30 p13348650
           su -
           . oraenv
                +ASM
           cd /tmp
           $ORACLE_HOME/OPatch/opatch auto  p13348650  -oh $ORACLE_HOME  -ocmrf ocm.rsp
      
                Executing /usr/bin/perl /orafs1/app/grid/product/11.2.0/grid/OPatch/crs/patch112.pl -patchdir . -patchn p13348650 -oh /orafs1/app/grid/product/11.2.0/grid -ocmrf ocm.rsp -paramfile /orafs1/app/grid/product/11.2.0/grid/crs/install/crsconfig_params
                opatch auto log file location is /orafs1/app/grid/product/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-02-18_08-31-32.log
                Detected Oracle Restart install
                Using configuration parameter file: /orafs1/app/grid/product/11.2.0/grid/crs/install/crsconfig_params
                Successfully unlock /orafs1/app/grid/product/11.2.0/grid
                patch ./p13348650/13348650  apply  failed  for home  /orafs1/app/grid/product/11.2.0/grid
                ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.32-200.13.1.el5uek'
                CRS-4123: Oracle High Availability Services has been started.
                
      # it STILL does not support ACFS on my kernel! ngng.
      
      capture log output:
           cat /orafs1/app/grid/product/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-02-18_08-31-32.log
      
                 Is the local system ready for patching? [y|n]
                 Y (auto-answered by -silent)
                 User Responded with: Y
                 Backing up files...
                 Applying interim patch '13348650' to OH '/orafs1/app/grid/product/11.2.0/grid'
                
                 Patching component oracle.crs, 11.2.0.3.0...
                 Copying file to "/orafs1/app/grid/product/11.2.0/grid/crs/install/crsconfig_lib.pm"
                 Copying file to "/orafs1/app/grid/product/11.2.0/grid/crs/install/crspatch.pm"
                 Copying file to "/orafs1/app/grid/product/11.2.0/grid/crs/install/s_crsconfig_lib.pm"
                
                 Patching component oracle.usm, 11.2.0.3.0...
                 There are 2 copy files under ORACLE_HOME that are not patched.
                 Files check failed: Some files under ORACLE_HOME are not patched. Please see log file for details.
                 ApplySession failed in system modification phase... 'Verification of patch failed: Files are not updated completely.'
                
                 Restoring "/orafs1/app/grid/product/11.2.0/grid" to the state prior to running NApply...
                 Checking if OPatch needs to invoke 'make' to restore some binaries...
                 OPatch was able to restore your system. Look at log file and timestamp of each file to make sure your system is in the state prior to applying the patch.
                
                 NApply restored the home. Please check your ORACLE_HOME to make sure:
                   - files are restored properly.
                   - binaries are re-linked correctly.
                 (use restore.[sh,bat] and make.txt (Unix only) as a reference. They are located under
                 "/orafs1/app/grid/product/11.2.0/grid/.patch_storage/NApply/2012-02-18_08-33-44AM"
                
                 UtilSession failed: ApplySession failed in system modification phase... 'Verification of patch failed: Files are not updated completely.'
                 Log file location: /orafs1/app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2012-02-18_08-33-44AM.log
                
                 OPatch failed with error code 73
                
                2012-02-18 08:39:08: patch ./p13348650/13348650  apply  failed  for home  /orafs1/app/grid/product/11.2.0/grid
      
      cat /orafs1/app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2012-02-18_08-33-44AM.log
      
           [Feb 18, 2012 8:37:49 AM]    verifying 921 copy files.
           [Feb 18, 2012 8:37:56 AM]    Failed file pair information (copy)::
           [Feb 18, 2012 8:37:56 AM]    Source file name is : /tmp/p13348650/13348650/files/bin/oraagent.bin,  size is : 28054745
           [Feb 18, 2012 8:37:56 AM]    Destination file name is : /orafs1/app/grid/product/11.2.0/grid/bin/oraagent.bin,  size is : 28054745
           [Feb 18, 2012 8:38:04 AM]    Failed file pair information (copy)::
           [Feb 18, 2012 8:38:04 AM]    Source file name is : /tmp/p13348650/13348650/files/install/usm/SLES10/x86_64/2.6.16.21-0.8/default/bin/oracleacfs.ko,  size is : 19649565
           [Feb 18, 2012 8:38:04 AM]    Destination file name is : /orafs1/app/grid/product/11.2.0/grid/install/usm/SLES10/x86_64/2.6.16.21-0.8/default/bin/oracleacfs.ko,  size is : 19649565
           [Feb 18, 2012 8:38:07 AM]    verifying 3 plugin actions.
           [Feb 18, 2012 8:38:07 AM]    There are 2 copy files under ORACLE_HOME that are not patched.
           [Feb 18, 2012 8:38:07 AM]    OUI-67124:Files check failed: Some files under ORACLE_HOME are not patched. Please see log file for details.
           [Feb 18, 2012 8:38:07 AM]    OUI-67124:ApplySession failed in system modification phase... 'Verification of patch failed: Files are not updated completely.'
           [Feb 18, 2012 8:38:07 AM]    Restoring "/orafs1/app/grid/product/11.2.0/grid" to the state prior to running NApply...
           [Feb 18, 2012 8:39:08 AM]    Checking if OPatch needs to invoke 'make' to restore some binaries...
           [Feb 18, 2012 8:39:08 AM]    OPatch was able to restore your system. Look at log file and timestamp of each file to make sure your system is in the state prior to applying the patch.
           [Feb 18, 2012 8:39:08 AM]    OUI-67124:
                                        NApply restored the home. Please check your ORACLE_HOME to make sure:
                                          - files are restored properly.
                                          - binaries are re-linked correctly.
                                        (use restore.[sh,bat] and make.txt (Unix only) as a reference. They are located under
                                        "/orafs1/app/grid/product/11.2.0/grid/.patch_storage/NApply/2012-02-18_08-33-44AM"
           [Feb 18, 2012 8:39:08 AM]    OUI-67073:UtilSession failed: ApplySession failed in system modification phase... 'Verification of patch failed: Files are not updated completely.'
           [Feb 18, 2012 8:39:08 AM]    --------------------------------------------------------------------------------
           [Feb 18, 2012 8:39:08 AM]    The following warnings have occurred during OPatch execution:
           [Feb 18, 2012 8:39:08 AM]    1) OUI-67124:Files check failed: Some files under ORACLE_HOME are not patched. Please see log file for details.
           [Feb 18, 2012 8:39:08 AM]    2) OUI-67124:ApplySession failed in system modification phase... 'Verification of patch failed: Files are not updated completely.'
           [Feb 18, 2012 8:39:08 AM]    3) OUI-67124:
                                        NApply restored the home. Please check your ORACLE_HOME to make sure:
                                          - files are restored properly.
                                          - binaries are re-linked correctly.
                                        (use restore.[sh,bat] and make.txt (Unix only) as a reference. They are located under
                                        "/orafs1/app/grid/product/11.2.0/grid/.patch_storage/NApply/2012-02-18_08-33-44AM"
           [Feb 18, 2012 8:39:08 AM]    --------------------------------------------------------------------------------
      There were errors with the file verification after the copy , for some reason. I don't know why it would have coughed on the two files oraagent.bin and oracleacfs.ko, but Oracle Support and google seaches didn't come up with anything. I plan to manually try an opatch apply instead of the oaptch auto, but while I do so, does anybody have clues for correctly applying this PSU?
        • 1. Re: File verification errors applying GI PSU 11.2.0.3.1 for acfs on linux
          lrp1
          To update this thread, I ended up speaking to Oracle Support on this. It turned out that one of the prerequisites in the README was the culprit:
          2.1.4 Unzipping the Patch
          The patch application requires explicit user actions to run 'opatch auto' command on each node of Oracle clusterware. So, it is recommended that you download and unzip the patch in a shared location to be able to access it from any node in the cluster and then as the Grid home owner execute the unzip command. To prevent installation failures, this location should be an empty directory.

          Note:
          Do not unzip the patch in the top level /tmp directory.

          The unzipped patch location should have read permission for ORA_INSTALL group in order to patch Oracle homes owned by different owners. The ORA_INSTALL group is the primary group of the user who owns the GI home or the group owner of the Oracle central inventory.>
          In my example, I had unzipped to /tmp. However, the resulting error message stack didn't really point that out.

          When I double-checked the README and tried again from /home/oracle/patches/p13348650 instead, it worked. It doesn't leave me with a confident feeling, more just relief. On a related note (and the entire reason for me patching to 11.2.0.3.1), ACFS filesystems ARE supported now under Oracle 5 update 7 with the 2.6.32-200.13.1.el5uek kernel (you had to really dig through the matrix in 1369107.1). Had this not worked, I would had to fail back to the non-UEK kernel to achieve ACFS.