1 2 Previous Next 17 Replies Latest reply: May 16, 2012 5:13 PM by Avi Miller-Oracle RSS

    autofs broken between base 6.2 and latest update?

    846491
      Having a problem working with an automount file system on latest update. I have two identical systems, one with 6.2 then latest updates applied. The other has 6.2 only, no updates. The results of attempting to run a script on the two systems are below. First is the updated (failing) system, second the non-updated (working) system. Machines setup with same anaconda script, so they are identical except for the yum -y update step. The working system will fail if updated.
      # cat /etc/auto.atsls 
      tools   fugitive:/export/vol02/$CPU
      
      # ls -l /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh
      -r-xr-xr-x 1 nobody nobody 174 May 14 12:48 /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh
      # uname -r
      2.6.32-300.21.1.el6uek.x86_64
      
      # strace /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh 
      execve("/auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh", ["/auto_atsls/tools/OracleDb_vers/"...], [/* 32 vars */]) = -1 ENOTDIR (Not a directory)
      dup(2)                                  = 3
      fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
      fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
      mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbc14660000
      lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
      write(3, "strace: exec: Not a directory\n", 30strace: exec: Not a directory
      ) = 30
      close(3)                                = 0
      munmap(0x7fbc14660000, 4096)            = 0
      exit_group(1)                           = ?
      # uname -r
      2.6.32-300.3.1.el6uek.x86_64
      
      # strace /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh 
      execve("/auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh", ["/auto_atsls/tools/OracleDb_vers/"...], [/* 32 vars */]) = 0
      brk(0)                                  = 0x1a21000
      mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe711596000
      access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
      open("/etc/ld.so.cache", O_RDONLY)      = 3
      fstat(3, {st_mode=S_IFREG|0644, st_size=94039, ...}) = 0
      mmap(NULL, 94039, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe71157f000
      close(3)                                = 0
      Any thoughts? Real bug? Missing something? I can examine the full directory hierarchy on either system with no problems.

      Thanks,
      Rob
        • 1. Re: autofs broken between base 6.2 and latest update?
          Catch-22
          What is the difference or problem you are experiencing?
          • 2. Re: autofs broken between base 6.2 and latest update?
            846491
            If you look at the first line after the strace in each of the above examples, you'll see the problem. More compact shown here:
            # /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh
            -bash: /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh: Not a directory
            • 3. Re: autofs broken between base 6.2 and latest update?
              Catch-22
              The strace output can be useful when troubleshooting a particular problem, but not necessarily when comparing different software versions. Perhaps the two systems are not as identical as it seems. What is the output of the following commands on each machine:

              mount
              id
              sestatus
              stat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh
              • 4. Re: autofs broken between base 6.2 and latest update?
                846491
                Working machine:
                # mount
                /dev/mapper/vg_root-lv_root on / type ext4 (rw)
                proc on /proc type proc (rw)
                sysfs on /sys type sysfs (rw)
                devpts on /dev/pts type devpts (rw,gid=5,mode=620)
                tmpfs on /dev/shm type tmpfs (rw)
                /dev/sda1 on /boot type ext4 (rw)
                none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
                sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
                nfsd on /proc/fs/nfsd type nfsd (rw)
                fugitive:/export/vol02/x86_64 on /auto_atsls/tools type nfs (rw,vers=4,addr=10.80.174.130,clientaddr=10.135.77.151)
                
                # id
                uid=1000(oracle) gid=1004(oinstall) groups=1004(oinstall),1000(asmdba),1002(asmadmin),1003(dba),1005(oper)
                
                # sestatus
                SELinux status:                 disabled
                
                # stat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh
                  File: `/auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh'
                  Size: 174            Blocks: 3          IO Block: 1048576 regular file
                Device: 73h/115d     Inode: 31023       Links: 1
                Access: (0555/-r-xr-xr-x)  Uid: (   99/  nobody)   Gid: (   99/  nobody)
                Access: 2012-05-14 14:44:06.948754993 -0600
                Modify: 2012-05-14 12:48:40.042642155 -0600
                Change: 2012-05-14 12:50:06.651966484 -0600
                Broken machine:
                # mount
                /dev/mapper/vg_root-lv_root on / type ext4 (rw)
                proc on /proc type proc (rw)
                sysfs on /sys type sysfs (rw)
                devpts on /dev/pts type devpts (rw,gid=5,mode=620)
                tmpfs on /dev/shm type tmpfs (rw)
                /dev/sda1 on /boot type ext4 (rw)
                none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
                sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
                nfsd on /proc/fs/nfsd type nfsd (rw)
                fugitive:/export/vol02/x86_64 on /auto_atsls/tools type nfs (rw,vers=4,addr=10.80.174.130,clientaddr=10.135.77.138)
                
                # id
                uid=1000(oracle) gid=1004(oinstall) groups=1004(oinstall),1000(asmdba),1002(asmadmin),1003(dba),1005(oper)
                
                # sestatus
                SELinux status:                 disabled
                
                # stat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh
                  File: `/auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh'
                  Size: 174            Blocks: 3          IO Block: 1048576 regular file
                Device: 73h/115d     Inode: 31023       Links: 1
                Access: (0555/-r-xr-xr-x)  Uid: (   99/  nobody)   Gid: (   99/  nobody)
                Access: 2012-05-14 14:44:06.948754993 -0600
                Modify: 2012-05-14 12:48:40.042642155 -0600
                Change: 2012-05-14 12:50:06.651966484 -0600
                • 5. Re: autofs broken between base 6.2 and latest update?
                  Catch-22
                  What's the output of +cat -e /etc/auto.atsls+ on each machine? Could there be any illegal character, line feed or space, etc.? What happens if you replace $CPU with x86_64?
                  • 6. Re: autofs broken between base 6.2 and latest update?
                    846491
                    Same on both machines - no wonky characters.
                    # cat -e /etc/auto.atsls 
                    tools   fugitive:/export/vol02/$CPU$
                    
                    # xxd /etc/auto.atsls
                    0000000: 746f 6f6c 7320 2020 6675 6769 7469 7665  tools   fugitive
                    0000010: 3a2f 6578 706f 7274 2f76 6f6c 3032 2f24  :/export/vol02/$
                    0000020: 4350 550a                                CPU.
                    Changing /etc/auto.atsls makes no difference (on either machine)
                    # cat -e /etc/auto.atsls 
                    tools   fugitive:/export/vol02/x86_64$
                    
                    # service restart autofs
                    Stopping automount:                                        [  OK  ]
                    Starting automount:                                        [  OK  ]
                    
                    # cat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh
                    cat: /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh: Not a directory
                    • 7. Re: autofs broken between base 6.2 and latest update?
                      Catch-22
                      Perhaps you can try to isolate the problem.

                      Are there any clues in /var/log/messages?

                      Can you verify the directory path? For instance, copy a shells script into each of directory and see if you can execute it?

                      What is the content of /etc/auto.master, /etc/nsswitch.conf and /etc/sysconfig/autofs on each host?

                      Have you tried to recreate the /auto_atsls directory?

                      And perhaps most important: What happens if you stop autofs and mount the NFS share manually? Can you execute the script then?
                      • 8. Re: autofs broken between base 6.2 and latest update?
                        846491
                        Dude wrote:
                        Perhaps you can try to isolate the problem.
                        I have been :-)
                        Are there any clues in /var/log/messages?
                        Nope. Nothing logged on either client or server during this activity.
                        Can you verify the directory path? For instance, copy a shells script into each of directory and see if you can execute it?
                        On working machine
                        # echo hey > /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/pass
                        root@dhcp-brm-bl5-204-2e-east-10-135-77-151:/root[]
                        # cat !$
                        cat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/pass
                        hey
                        On failing machine - note that the file fail was created, but is empty
                        # echo "hey" > /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/fail
                        -bash: /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/fail: Not a directory
                        # cat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/fail
                        cat: /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/fail: Not a directory
                        # cat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/pass 
                        cat: /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/pass: Not a directory
                        # rm -f !$
                        rm -f /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/pass
                        # cat !$
                        cat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/pass
                        cat: /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/pass: No such file or directory
                        What is the content of /etc/auto.master, /etc/nsswitch.conf and /etc/sysconfig/autofs on each host?
                        Same on both hosts
                        # cat /etc/auto.master | grep -v ^#
                        /misc     /etc/auto.misc
                        /net     -hosts
                        +auto_master
                        /auto_atsls     auto.atsls
                        /apps     auto.apps
                        /home     auto_home -nobrowse
                        /clearcase/vobs     /etc/auto_tape_vobs -rw
                        /clearcase/views     /etc/auto_tape_views -rw
                        +auto_master_tape
                        +auto_tape_vobs
                        # cat /etc/nsswitch.conf | egrep -v "(^#|^$)"
                        passwd:     files nis
                        shadow:     files nis
                        group:      files nis
                        hosts:      files dns nis
                        ipnodes:    files
                        networks:   nisplus [NOTFOUND=return] files
                        protocols:  nisplus [NOTFOUND=return] files
                        rpc:        files nis
                        ethers:     nisplus [NOTFOUND=return] files
                        netmasks:   nisplus [NOTFOUND=return] files
                        bootparams: nisplus [NOTFOUND=return] files
                        publickey:  nisplus [NOTFOUND=return] files
                        netgroup:   nisplus
                        automount:  files nis
                        aliases:    files nisplus
                        services:   files nis
                        sendmailvars:   files
                        printers:       user nis files
                        auth_attr:  files nis
                        prof_attr:  files nis
                        exec_attr:  files nis
                        user_attr:  files nis
                        project:    files nis
                        # cat /etc/sysconfig/autofs | grep -v ^#
                        TIMEOUT=300
                        BROWSE_MODE="yes"
                        MOUNT_NFS_DEFAULT_PROTOCOL=3
                        USE_MISC_DEVICE="yes"
                        >
                        Have you tried to recreate the /auto_atsls directory?
                        Directory isn't there - automount. Just for giggles, I stopped autofs, created /auto_atsls, then /auto_atsls/tools. Both still failed as before
                        And perhaps most important: What happens if you stop autofs and mount the NFS share manually? Can you execute the script then?
                        Nope same failure.

                        The only difference I can find is the autofs package level
                        Working machine
                        # yum list installed "nfs*" "autofs*"
                        Loaded plugins: refresh-packagekit, security
                        Installed Packages
                        autofs.x86_64                                              1:5.0.5-39.el6                                      @anaconda-OracleLinuxServer-201112121319.x86_64/6.2
                        nfs-utils.x86_64                                           1:1.2.3-15.el6                                      @anaconda-OracleLinuxServer-201112121319.x86_64/6.2
                        nfs-utils-lib.x86_64                                       1.1.5-4.el6                                         @anaconda-OracleLinuxServer-201112121319.x86_64/6.2
                        nfs4-acl-tools.x86_64                                      0.3.3-5.el6                                         @anaconda-OracleLinuxServer-201112121319.x86_64/6.2
                        Failing machine
                        # yum list installed "nfs*" "autofs*"
                        Loaded plugins: refresh-packagekit, security
                        Installed Packages
                        autofs.x86_64                                            1:5.0.5-39.el6_2.1                                    @ol6_latest                                        
                        nfs-utils.x86_64                                         1:1.2.3-15.el6                                        @anaconda-OracleLinuxServer-201112121319.x86_64/6.2
                        nfs-utils-lib.x86_64                                     1.1.5-4.el6                                           @anaconda-OracleLinuxServer-201112121319.x86_64/6.2
                        nfs4-acl-tools.x86_64                                    0.3.3-5.el6                                           @anaconda-OracleLinuxServer-201112121319.x86_64/6.2
                        Although downgrading autofs didn't help.
                        I downgraded the kernel next, didn't help.
                        I guess I'll start from a fresh system, and work individual upgrades until it fails, unless you have other suggestions? I tried rolling back the big update, and yum urped...
                        • 9. Re: autofs broken between base 6.2 and latest update?
                          Avi Miller-Oracle
                          LogicalChaos wrote:
                          Having a problem working with an automount file system on latest update. I have two identical systems, one with 6.2 then latest updates applied. The other has 6.2 only, no updates. The results of attempting to run a script on the two systems are below.
                          Have you logged an SR for this? Oracle Development don't look at the forums. :)
                          • 10. Re: autofs broken between base 6.2 and latest update?
                            Catch-22
                            And perhaps most important: What happens if you stop autofs and mount the NFS share manually? Can you execute the script then?
                            Nope same failure.
                            This tells me that the problem is not autofs, but related to the configuration or compatibility of NFS.

                            How is the NFS share configured on the "fugitive" host? Is it NFS v4? What is the content of the /etc/exports file on the remote host? How about DNS and /etc/hosts file on all machines?
                            • 11. Re: autofs broken between base 6.2 and latest update?
                              846491
                              Avi Miller wrote:
                              Have you logged an SR for this? Oracle Development don't look at the forums. :)
                              Not yet. I wanted to figure out if it was me or real first. Besides, I haven't figured out how to do that yet, despite working for Oracle (from Sun from StorageTek) :-)
                              • 12. Re: autofs broken between base 6.2 and latest update?
                                846491
                                Dude wrote:
                                How is the NFS share configured on the "fugitive" host? Is it NFS v4? What is the content of the /etc/exports file on the remote host? How about DNS and /etc/hosts file on all machines?
                                Intel Solaris, zfs shared. Clients were using v4 first round, then switched it to v3, no change.
                                % zfs get all | grep x86_64
                                pool02/vol02/x86_64   type             filesystem             -
                                pool02/vol02/x86_64   creation         Tue Aug 31 12:01 2010  -
                                pool02/vol02/x86_64   used             6.93G                  -
                                pool02/vol02/x86_64   available        93.1G                  -
                                pool02/vol02/x86_64   referenced       6.93G                  -
                                pool02/vol02/x86_64   compressratio    1.00x                  -
                                pool02/vol02/x86_64   mounted          yes                    -
                                pool02/vol02/x86_64   quota            100G                   local
                                pool02/vol02/x86_64   reservation      none                   default
                                pool02/vol02/x86_64   recordsize       128K                   default
                                pool02/vol02/x86_64   mountpoint       /export/vol02/x86_64   local
                                pool02/vol02/x86_64   sharenfs         on                     inherited from pool02/vol02
                                pool02/vol02/x86_64   checksum         on                     default
                                pool02/vol02/x86_64   compression      off                    default
                                pool02/vol02/x86_64   atime            on                     default
                                pool02/vol02/x86_64   devices          on                     default
                                pool02/vol02/x86_64   exec             on                     default
                                pool02/vol02/x86_64   setuid           on                     default
                                pool02/vol02/x86_64   readonly         off                    default
                                pool02/vol02/x86_64   zoned            off                    default
                                pool02/vol02/x86_64   snapdir          hidden                 default
                                pool02/vol02/x86_64   aclmode          groupmask              default
                                pool02/vol02/x86_64   aclinherit       restricted             default
                                pool02/vol02/x86_64   canmount         on                     default
                                pool02/vol02/x86_64   shareiscsi       off                    default
                                pool02/vol02/x86_64   xattr            on                     default
                                pool02/vol02/x86_64   copies           1                      default
                                pool02/vol02/x86_64   version          3                      -
                                pool02/vol02/x86_64   utf8only         off                    -
                                pool02/vol02/x86_64   normalization    none                   -
                                pool02/vol02/x86_64   casesensitivity  sensitive              -
                                pool02/vol02/x86_64   vscan            off                    default
                                pool02/vol02/x86_64   nbmand           off                    default
                                pool02/vol02/x86_64   sharesmb         off                    default
                                pool02/vol02/x86_64   refquota         none                   default
                                pool02/vol02/x86_64   refreservation   none                   default
                                On clients.
                                # cat /etc/hosts
                                127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
                                ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
                                10.80.2.1     cabin     stengnis2
                                10.80.2.71     morris     stengnis1
                                DNS setup by dhcp, same on both (obviously 3 different DNS servers, obfuscated).
                                # cat /etc/resolv.conf 
                                ; generated by /sbin/dhclient-script
                                search us.oracle.com usdhcp.oraclecorp.com
                                nameserver ww.xx.yy.xx
                                nameserver ww.xx.yy.xx
                                nameserver ww.xx.yy.xx
                                • 13. Re: autofs broken between base 6.2 and latest update?
                                  Catch-22
                                  What about the /etc/exports file which defines access to the NFS clients?
                                  • 14. Re: autofs broken between base 6.2 and latest update?
                                    846491
                                    Dude wrote:
                                    What about the /etc/exports file which defines access to the NFS clients?
                                    Not there. I talked with the admin of that system, and it's a native zfs capability.
                                    But, guess what?
                                    It's the kernel. Same machine, a reboot between. Only difference is selecting different kernel version on boot. That was also the only update yum -y update kernel-uek done from the base 6.2 install.
                                    # uname -a
                                    Linux dhcp-brm-bl5-204-2e-east-10-135-77-151.usdhcp.oraclecorp.com 2.6.32-300.3.1.el6uek.x86_64 #1 SMP Fri Dec 9 18:57:35 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
                                    # cat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh 
                                    #!/bin/sh
                                    
                                    /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/database/runInstaller -ignorePrereq -silent -force -responseFile /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.rsp
                                    # uname -a
                                    Linux dhcp-brm-bl5-204-2e-east-10-135-77-151.usdhcp.oraclecorp.com 2.6.32-300.24.1.el6uek.x86_64 #1 SMP Fri May 4 15:24:10 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
                                    # cat /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh 
                                    cat: /auto_atsls/tools/OracleDb_vers/11.2.0.3.0/plainDB.sh: Not a directory
                                    Now I can submit a SR, if I can figure out how...
                                    1 2 Previous Next