8 Replies Latest reply: Jun 3, 2010 9:26 AM by psadminMD RSS

    Suddenly not able to start up the appserver? SQL library error

    psadminMD
      Our datacenter went down this past weekend and so my Template 9.1 environments also went down, unfortuantely I have yet to have time to put in start up scripts on those servers. I am trying to bring all 3 tiers up manually and am encountering errors I've never had before. I am assuming that I am doing something wrong because this environment has not been touched and I've never had this problem before.

      Templates have not been changed, we accepted all the defaults. We have 3 different servers, 1 for DB, 1 App, 1 Webserver
      I have started the database and listener and can reach them via sqlplus no problem. However, when starting up my appserver I am getting the following error: GenMessageBox(200, 0, M): PS General SQL Routines: Missing or invalid version of SQL library libpsora

      I have set my Oracle environment (. oraenv) to my database name. And my Oracle home to: /opt/oracle/psft/appbatch/oracle-client/10.2.0.4-64bit. Is this correct? I am starting the appserver as psadm2.

      What am I missing? Why am I suddently having library file version errors?

      Thanks!!
        • 1. Re: Suddenly not able to start up the appserver? SQL library error
          Nicolas.Gasparotto
          GenMessageBox(200, 0, M): PS General SQL Routines: Missing or invalid version of SQL library libpsora
          Please, provide some more input of the psadmin command error and appserver log file entries.
          Are people and sysadm users' password not expired (even if I would expect an other error) ?
          Verify it from sqlplus :
          select username,account_status,expiry_date,profile
          from   dba_users
          where username in ('PEOPLE','SYSADM');
          From the App/Batch server, within psadm2 user, are you able to connect onto the database ?
          I have set my Oracle environment (. oraenv) to my database name.
          Where are you running it from ? I've never ran it to get it work.

          Nicolas.
          • 2. Re: Suddenly not able to start up the appserver? SQL library error
            psadminMD
            Ran your sql statement, here is the output:

            SQL> select username,account_status,expiry_date,profile
            2 from dba_users
            3 where username in ('PEOPLE','SYSADM');

            USERNAME ACCOUNT_STATUS EXPIRY_DA
            ------------------------------ -------------------------------- ---------
            PROFILE
            ------------------------------
            PEOPLE OPEN 08-JUL-10
            DEFAULT

            SYSADM OPEN 29-JUL-10
            DEFAULT

            Here is more of the appserver error:
            Do you wish to see the error messages in the APPSRV.LOG file? (y/n) [n] :y

            PSADMIN.1656 (0) [06/02/10 14:24:58](0) Begin boot attempt on domain APPDOM
            PSAPPSRV.1671 (0) [06/02/10 14:25:12](0) PeopleTools Release 8.50.02 (Linux) sta
            rting. Tuxedo server is APPSRV(99)/1
            PSAPPSRV.1671 (0) [06/02/10 14:25:12](0) Cache Directory being used: /home/psadm
            2/ps/pt/8.50/appserv/APPDOM/CACHE/PSAPPSRV_1/
            PSAPPSRV.1671 (0) [06/02/10 14:25:12](1) GenMessageBox(200, 0, M): PS General SQ
            L Routines: Missing or invalid version of SQL library libpsora (200,0)
            PSAPPSRV.1671 (0) [06/02/10 14:25:12](1) GenMessageBox(0, 0, M): Database Signon
            : Could not sign on to database PS91DEMO with user PS.
            PSAPPSRV.1671 (0) [06/02/10 14:25:12](0) Server failed to start
            PSADMIN.1656 (0) [06/02/10 14:25:19](0) End boot attempt on domain APPDOM

            Yes I can connect to sqlplus from the app/batch server with the psadm2 user:
            [psadm2@ovm002 appserv]$ sqlplus SYSADM@psdmo

            SQL*Plus: Release 10.2.0.1.0 - Production on Wed Jun 2 14:28:57 2010

            Copyright (c) 1982, 2005, Oracle. All rights reserved.

            Enter password:

            Connected to:
            Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
            With the Partitioning, OLAP, Data Mining and Real Application Testing options

            SQL>

            Thanks.
            • 3. Re: Suddenly not able to start up the appserver? SQL library error
              psadminMD
              Also,
              Here is how I am setting the environment.

              [psadm2@ovm002 ~]$ . oraenv
              ORACLE_SID = [psadm2] ? psdmo
              Cannot locate ORACLE_HOME.
              ORACLE_HOME = [] ? /opt/oracle/psft/appbatch/oracle-client/10.2.0.4-64bit
              [psadm2@ovm002 ~]$
              • 4. Re: Suddenly not able to start up the appserver? SQL library error
                Nicolas.Gasparotto
                pspeon2.0 wrote:
                Also,
                Here is how I am setting the environment.

                [psadm2@ovm002 ~]$ . oraenv
                Why are you doing this ? You don't need it, did you try to disconnect/reconnect and start the appserver without this ?

                Nicolas.
                • 5. Re: Suddenly not able to start up the appserver? SQL library error
                  psadminMD
                  Out of habit I guess...we do this on our other environments because we sometimes have different Oracle Homes.

                  I got completely out and back in and tried it again without setting any Oracle params and I'm still getting the same issue. I've encountered this before and had to add some symbolic links, but that was on my other environments and only after an Oracle version upgrade. Not sure why this is suddenly having a problem with the SQL library files. I'm playing around with symbolic link as per the Oracle document to see if maybe that will fix it, but so far it doesn't look like it is.
                  • 6. Re: Suddenly not able to start up the appserver? SQL library error
                    Nicolas.Gasparotto
                    Here what I have :
                    [psadm2@psovmab 10.2.0.4-64bit]$ pwd
                    /opt/oracle/psft/appbatch/oracle-client/10.2.0.4-64bit
                    [psadm2@psovmab 10.2.0.4-64bit]$ cd lib
                    [psadm2@psovmab lib]$ ls -lrt
                    ...
                    lrwxrwxrwx 1 root root       15 Oct  5  2009 libagtsh.so -> libagtsh.so.1.0
                    lrwxrwxrwx 1 root root       15 Oct  5  2009 libocci.so -> libocci.so.10.1
                    drwxrwxr-x 2 root root     4096 Oct  5  2009 stubs
                    lrwxrwxrwx 1 root root       76 Oct  5  2009 libclntsh.so.9.0 -> /opt/oracle/psft/appbatch/oracle-client/10.2.0.4-64bit/lib/libclntsh.so.10.1
                    lrwxrwxrwx 1 root root       76 Oct  5  2009 libclntsh.so -> /opt/oracle/psft/appbatch/oracle-client/10.2.0.4-64bit/lib/libclntsh.so.10.1
                    [psadm2@psovmab lib]$ cd ../lib32
                    [psadm2@psovmab lib32]$ ls -lrt
                    ...
                    lrwxrwxrwx 1 root root       17 Oct  5  2009 libclntsh.so -> libclntsh.so.10.1
                    lrwxrwxrwx 1 root root       15 Oct  5  2009 libagtsh.so -> libagtsh.so.1.0
                    lrwxrwxrwx 1 root root       50 Oct  5  2009 ldflags -> /products/oracle_client/10.2.0.4-64bit/lib/ldflags
                    lrwxrwxrwx 1 root root       15 Oct  5  2009 libocci.so -> libocci.so.10.1
                    drwxrwxr-x 2 root root     4096 Oct  5  2009 stubs
                    Nicolas.
                    • 7. Re: Suddenly not able to start up the appserver? SQL library error
                      psadminMD
                      What do you have under your $PS_HOME/bin directory for links? In the past they had to have matching links between the $PS_HOME/bin and $ORACLE_HOME/lib directories.

                      Thanks!
                      • 8. Re: Suddenly not able to start up the appserver? SQL library error
                        psadminMD
                        FIXED IT!
                        Found a hint in one of the solutions on My Oracle Support at the very bottom. ID 650409.1.

                        I did add the links to $PS_HOME/bin, but what finally fixed it was the path to my $ORACLE_HOME/lib was incorrect in the $PS_HOME/appserv/DomainName/psappsrv.env file for my LD_LIBRARY_PATH. It had my path as /opt/oracle/psft/appbatch/oracle-client/lib instead of /opt/oracle/psft/appbatch/oracle-client/*10.2.0.4-64bit*/lib. Funny how it was fine everywhere else, except this file

                        Now as to how in the world that got changed I have no clue. I am the only one that touches these servers and I haven't touched them in months.

                        Does this file get rebuilt? Would a crash cause it not to be populated correctly? So weird.

                        t.