1 2 Previous Next 16 Replies Latest reply: Sep 25, 2012 8:15 AM by FullCarbonBike RSS

    Post-installation configuration interface not running

    fchagnon
      I've just installed Ops Center 12.1 on a new SPARC T4-1b running Solaris 11.
      We are using a remote database hosted inside an Oracle Database Appliance. The installation went through entirely, and I'm at the point where I need to configure the application.

      The problem is that the web GUI that is supposed to be running on port 9443 isn't running. (doesn't appear in netstat -an).
      The instances bound to port 80 (the redirect) and port 443 exist -- but the rewrite sends me to the non-existent service running on port 9443, so I am stuck.

      root@host:/opt/SUNWxvmoc/bin# ./ecadm status
      online

      OCDoctor reports no issues.

      root@host:/opt/SUNWxvmoc/bin# /var/opt/sun/xvm/OCDoctor/OCDoctor.sh --troubleshoot
      Ops Center Doctor 4.05 [OC 12.1.1.2076,SunOS11] [Read only] [21-Aug-2012 14:46PM EDT]
      ======================== Checking Enterprise Controller... ==============================
      OK: Total number of agents: 0
      OK: Number of agents with inventories: 0
      OK: Memory is (65024), ZFS is used and zfs_arc_max is set to 8589934592 in /etc/system
      OK: /etc/netboot folder exists
      OK: User 'uce-sds' home folder points to the right location
      OK: User 'oracleoc' home folder points to the right location
      OK: User 'allstart' home folder points to the right location
      OK: Apache logs are smaller than 2 GB
      OK: n1gc folder has the right permissions
      OK: All Enterprise Controller packages are installed properly
      OK: All agent packages are installed properly
      OK: Enterprise Controller status is online
      OK: the version is the latest one (12.1.1.2076)
      OK: satadm timeouts were increased
      OK: tar command was properly adjusted in satadm
      OK: Found the server .uce.rc
      OK: Server .uce.rc has the correct file permissions
      OK: Server .uce.rc has the correct ownership
      OK: Connectivity to the KB public servers works properly
      OK: Grouplock file doesn't exist
      OK: Locales are OK ("C")
      OK: No partially installed packages
      OK: No old jobs rerun (CR 6990675)
      OK: No NCP encryption is used
      OK: No credential issues foundWhere is this instance
      =========== Proxy controller is installed but not configured, skipping ==================
      =========== Agent controller is installed but not configured, skipping ==================


      How can I start this instance on port 9443 so I can begin configuring the platform?

      Edited by: fchagnon on Aug 21, 2012 11:50 AM
        • 1. Re: Post-installation configuration interface not running
          Eran_Steiner
          Try running "OCDoctor.sh --update" and see if the newer version (4.10) finds anything.

          Also, can you copy/paste the output of those?
          # netstat -na | grep 443 | grep LISTEN
          # ps -ef | grep java
          # svcs '*scn*' '*common'

          Last, make sure the output of:
          # getent hosts `hostname`
          matches the output of:
          # getent hosts `getent hosts \`hostname\` | awk '{print $1}'`
          • 2. Re: Post-installation configuration interface not running
            fchagnon
            Thanks for your reply.

            root@manager1:~# netstat -an | grep 443
            *.443 *.* 0 0 128000 0 LISTEN
            *.443 *.* 0 0 128000 0 LISTEN

            root@manager1:~# ps -ef | grep java
            root 12584 11237 0 20:40:15 pts/1 0:00 grep java
            root 13185 13184 0 14:22:01 ? 2:14 /usr/jdk/instances/jdk1.7.0/bin/sparcv9/java -Xms200M -Xmx8192M -server -XX:Str

            root@manager1:~# svcs scn common
            STATE STIME FMRI
            disabled Aug_20 svc:/application/management/common-agent-container-1:default
            disabled Aug_20 svc:/application/management/common-agent-container-1:scn-proxy
            disabled Aug_20 svc:/application/scn/db:local
            disabled Aug_20 svc:/application/scn/oracle:default
            disabled Aug_20 svc:/application/scn/oralistener:default
            disabled Aug_20 svc:/application/scn/proxy-enable:default
            disabled Aug_20 svc:/application/scn/ec-server-splashpage:default
            disabled Aug_20 svc:/application/scn/dhcpd:default
            online 14:21:59 svc:/application/scn/satellite-enable:default
            online 14:22:00 svc:/application/scn/db:remote
            online 14:22:00 svc:/application/scn/uce-scheduler:default
            online 14:22:00 svc:/application/scn/db:default
            online 14:22:00 svc:/application/scn/uce-server:default
            online 14:22:05 svc:/application/scn/ajaxterm:default
            online 14:22:40 svc:/application/management/common-agent-container-1:oem-ec
            online 14:22:40 svc:/application/scn/satellite-available:default
            offline Aug_20 svc:/application/scn/proxy-available:default

            root@manager1:~# getent hosts `hostname`
            172.19.233.26 manager1
            root@manager1:~# getent hosts `getent hosts \`hostname\` | awk '{print $1}'
            172.19.233.26 manager1

            The new version of OCDoctor has some issues with accessing the KB. However the 'check-connectivity' function works fine.

            root@manager1:~# /var/opt/sun/xvm/OCDoctor/OCDoctor.sh --troubleshoot
            Ops Center Doctor 4.10 [OC 12.1.1.2076,SunOS11] [Read only] [21-Aug-2012 20:43PM EDT]
            ======================== Checking Enterprise Controller... ==============================
            OK: Total number of agents: 0
            OK: Number of agents with inventories: 0
            OK: Memory is 65024 MB, ZFS is used and zfs_arc_max is set to 8589934592 in /etc/system
            OK: All Solaris 11 agent bundles in /var/opt/sun/xvm/images/agent/ are imported properly to the repository
            OK: Files in /var/opt/sun/xvm/images/agent/ have the right permissions
            OK: Files in /var/opt/sun/xvm/osp/web/pub/pkgs/ have the right permissions
            OK: auth.cgi returns XML
            OK: User 'oracleoc' home folder points to the right location
            OK: User 'allstart' home folder points to the right location
            OK: Apache logs are smaller than 2 GB
            OK: n1gc folder has the right permissions
            OK: All Enterprise Controller packages are installed properly
            OK: All agent packages are installed properly
            OK: Enterprise Controller status is online
            OK: the version is the latest one (12.1.1.2076)
            OK: satadm timeouts were increased
            OK: tar command was properly adjusted in satadm
            OK: no 'conn_properties_file_name' value in .uce.rc
            OK: Found the server .uce.rc
            OK: Server .uce.rc has the correct file permissions
            OK: Server .uce.rc has the correct ownership
            ERROR: Could not access the KB public servers
            Check the username and password and proxy configuration in "Administration" ->
            "Edit Authentications" and "Change HTTP proxy"

            Also, you may try the following command to test connectivity:
            /var/opt/sun/xvm/OCDoctor/OCDoctor.sh --check-connectivity

            OK: Grouplock file doesn't exist
            OK: uce-server .uce.rc file points to a proper JAVA
            OK: Disconnected mode is not configured
            OK: Locales are OK ("C")
            OK: Found only one Solaris 11 agent bundle
            OK: No partially installed packages
            OK: No http_proxy is set in the user profile files
            OK: 'public' folder has the right ownership
            OK: 'public' folder is writable for uce-sds
            OK: 'private' folder has the right ownership
            OK: 'private' folder is writable for uce-sds
            OK: No old jobs rerun (CR 6990675)
            OK: No NCP encryption is used
            OK: 'update-saved-state' folder exists and has the right permissions
            OK: No credential issues found
            =========== Proxy controller is installed but not configured, skipping ==================
            =========== Agent controller is installed but not configured, skipping ==================

            root@manager1:~# /var/opt/sun/xvm/OCDoctor/OCDoctor.sh --check-connectivity
            Ops Center Doctor 4.10 [OC 12.1.1.2076,SunOS11]OCDoctor.sh --check-connectivity
            ======================== Checking Network Connectivity ==============================
            Please enter the My Oracle Support (MOS) Username: xxxxxxxxxxxxxxxx
            Password:
            Is there a proxy? Type the proxy server here, for example 'www-proxy.us.oracle.com' (hit Enter if there is no proxy):

            Test 1/4: Checking CC service reachability (without using user/password yet)
            ==============================================================================
            OK: CC status is OK

            Test 2/4: Checking GUUS service reachability (without using user/password yet)
            ==============================================================================
            OK: GUUS status is OK

            Test 3/4: Downloading Knowledge Channels file from updates.oracle.com
            ===========================================================================
            OK: Knowledge download was successful!

            Test 4/4: Downloading patch 108437-06 from updates.oracle.com
            ===========================================================================
            OK: Patch download was successful!
            • 3. Re: Post-installation configuration interface not running
              fchagnon
              Perhaps if someone could show me what this instance looks like in the output of a ps -ef, it would help me know where I should be looking. As it is right now it's hard for me to troubleshoot why something isn't starting when I have no idea where it should be running from.
              • 4. Re: Post-installation configuration interface not running
                FullCarbonBike
                I'm seeing a similar problem, on two separate systems: on a new installation, same hardware and OS as you describe, the other on what was an existing system (also smae HW and OS), that was working. The problem on both systems may have been related to doing an Solaris 11 SRU update, we went from SRU 5.5 to SRU 10.5. The Ops Center 12c U1 application had been configured and working on the existing system, but the symptoms that you describe are the same. I can get the initial http splash page on the new installation system, but as sone as it re-directs, the page isn't available. I'll keep this thread posted if I find a solution.
                • 5. Re: Post-installation configuration interface not running
                  Eran_Steiner
                  I don't see anything wrong in the output you submitted, including the 'ps -ef' output. I would suggest opening up a service request with Oracle so the logs can be examined.

                  If you want to try and look through the logs yourself, you can try:
                  Installation log:
                  /var/tmp/installer.log.latest

                  Java log:
                  /var/cacao/instances/oem-ec/logs/
                  newest log is cacao.0, cacao.1 is older etc.

                  BUI log:
                  /var/opt/sun/xvm/logs/emoc.log

                  Good luck!
                  • 6. Re: Post-installation configuration interface not running
                    fchagnon
                    Which service or process is it that should be bound to the port? IDoes it sppear in my output? I can begin troublshooting if I know more about the process itself.
                    But in the meantime, doesn't hurt to open a case either.
                    • 7. Re: Post-installation configuration interface not running
                      FullCarbonBike
                      I had an odd occurance, after several hours of poking around, but making NO changes to either system, I decided to cycle the ECs which I had done before, with no success. So I did an /opt/SUNWxvmoc/bin/ecadm stop -v -w, let the systems quiesce a bit (10-15 minutes) and then did a /opt/SUNWxvmoc/bin/ecadm start -v -w, and now the 9443 ports are back.
                      netstat -na | egrep 9443
                      *.9443 *.* 0 0 128000 0 LISTEN
                      *.9443 *.* 0 0 128000 0 LISTEN

                      I did that yesterday afternoon ~3:45PM) and the 9443 LISTEN ports, on both systems, are still there this morning. Opening a case sounds like a good idea, I'll be doing that later this morning as well.
                      • 8. Re: Post-installation configuration interface not running
                        fchagnon
                        No luck for me.

                        root@manager1:~# /opt/SUNWxvmoc/bin/ecadm stop -v -w
                        ecadm: Shutting down Enterprise Controller using SMF...
                        ecadm: ... milestone "satellite-enable" successfully disabled
                        ecadm: ... waiting for services to go "offline"
                        ecadm: Enterprise Controller services have stopped

                        <time passes>

                        root@manager1:~# /opt/SUNWxvmoc/bin/ecadm start -v -w
                        ecadm: Enable HTML redirect
                        ecadm: Disabling svc:/application/scn/ec-server-splashpage:default
                        ecadm: svc:/application/scn/ec-server-splashpage:default successfully stopped
                        ecadm: Starting Enterprise Controller with SMF...
                        ecadm: ... milestone "satellite-enable" successfully enabled
                        ecadm: ... Waiting for Enterprise Controller services to go "online"
                        ecadm: Enterprise Controller services have started

                        root@manager1:~# netstat -an | grep 443
                        *.443 *.* 0 0 128000 0 LISTEN
                        *.443 *.* 0 0 128000 0 LISTEN


                        I've opened an SR with Oracle for this.

                        Strangely they don't list Ops Center in the "Software SR" list, presumably because it's not licensed so I had to go with Enterprise Manager Cloud Control. I figured that misdrect would get resolved sooner than opening a hardware SR and explaining this issue to a SPARC T4 expert. :)
                        • 9. Re: Post-installation configuration interface not running
                          FullCarbonBike
                          I have an SR open as well, the probelm has come back on a different system, our "POC/test" EC, on a T-5xxx box.
                          • 10. Re: Post-installation configuration interface not running
                            fchagnon
                            Cool. My SR, which was open last Friday, has been sent to backline engineering. Good luck.
                            • 11. Re: Post-installation configuration interface not running
                              FullCarbonBike
                              I was able to collect some logs while the issue was present, but after a few ecadm stop / starts, this system also restarted the LISTEN 9443 ports. No explaination as to what or why it's happening.
                              • 12. Re: Post-installation configuration interface not running
                                FullCarbonBike
                                Just a note, after another occurrence where I had to stop the EC, it took something like 10 ecadm stop/starts to finally get one where the 9443 ports came up.
                                • 13. Re: Post-installation configuration interface not running
                                  FullCarbonBike
                                  There's a workaround, have you received it from Oracle yet?
                                  • 14. Re: Post-installation configuration interface not running
                                    fchagnon
                                    Yes. Just yesterday. Increased the size of the cacao heap but doing the following:

                                    # cacaoadm set-param java-flags="-d64 -Xms200M -Xmx8192M -server -XX:StringTableSize=27001 -XX:PermSize=128m -XX:MaxPermSize=384m -Xss256k -XX:+UseParallelOldGC -Dsun.security.pkcs11.enable-solaris=false -Djava.endorsed.dirs=/usr/lib/cacao/lib/endorsed -Dxvmserver=false -Dcom.sun.xml.ws.monitoring.endpoint=false -Dcom.sun.xml.ws.monitoring.client=false" -i oem-ec

                                    This had to be followed by a reboot.

                                    I'm in business now.

                                    Was this the same resolution for you?
                                    1 2 Previous Next