1 2 Previous Next 26 Replies Latest reply: Jan 10, 2013 6:13 AM by Michel RSS

    Application Engine initiated: Tuxedo cannot find RunAeAsync

    Honza
      Hello,

      running PeopleSoft Financials 9.1 on PeopleTools 8.52.07 (on OEL 5 64bit) I came across the following problem when running any kind of application engine
      PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)    Server: PSUNX checking status...
      PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)    Server action mode: Ok (looking for requests)
      PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)    Server: PSUNX looking for work
      PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)    Checking Process cancels...
      PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)      Requests found in Process Request table         2
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](3)    Server: PSUNX checking status...
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](3)    Server action mode: Submitting request
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](3)    Number of New Process Request(s) To Start: 2
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync1.  Make sure the application server advertising this service is booted.
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync2.  Make sure the application server advertising this service is booted.
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync3.  Make sure the application server advertising this service is booted.
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync1.  Make sure the application server advertising this service is booted.
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync2.  Make sure the application server advertising this service is booted.
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
      PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync3.  Make sure the application server advertising this service is booted.
      Here is an example of process parameters:
      psae -CT ORACLE -CD FC91DB -CO "VP1" -CP OPRPSWD -R 1 -I 47592 -AI AEMINITEST -OT 6 -FP "/home/fc91/psft/pt/8.52/appserv/prcs/FC91DB/log_output/AE_AEMINITEST_47592/" -OF 14 -TRACE 387 -TOOLSTRACEPC 3596
      I've tried deleting the application and scheduler domain and recreating with the same configuration file as well as rebooting the scheduler as well as setting the Recycle Count to 0, but there is very limited information on this on metalink.

      I'm looking for any input or ideas on this. Thank you.
      Best regards.

      Edited by: Honza on Jan 9, 2013 7:28 AM

      Edited by: Honza on Jan 9, 2013 7:52 AM
      Corrected Tools and OS versions.
        • 1. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
          Nicolas.Gasparotto
          Wondering why you have trace turned on. Is it also the case when you run the AE without trace ? Try :
          1. remove the trace option from the AE definition
          2. append the option instead of override
          Case described in the note *Application Engine Fails with Service RunAeProgAsync failed [ID 618643.1]*+

          Nicolas.
          • 2. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
            Michel
            First of all PeopleSoft Financials 9.1 is not certified on PeopleTools 8.49. Did you recently upgrade to this version?

            I would try creating a new process scheduler domain with a new configuration file. This rules out any issues with the configuration file itself. Another option is setting the App Eng Server in the quick configuration to No. This option creates a separate Tuxedo process for Application Engines, which isn't required to run Application Engines. This process seems non existent in your Tuxedo server, which is giving the error messages seen in the process scheduler log.

            The TUXLOG might give insights as to why this error is occuring. Could you paste the contents of this file here?
            • 3. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
              HakanBiroglu
              Could you check if you have other processes with a status Error/No Success?
              If this is a non-production environment, try cleaning the processes with an Error/No Success status and try again.
              • 4. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                Nicolas.Gasparotto
                Michel wrote:
                First of all PeopleSoft Financials 9.1 is not certified on PeopleTools 8.49. Did you recently upgrade to this version?
                That's a good one.

                Nicolas.
                • 5. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                  Honza
                  >
                  Wondering why you have trace turned on. Is it also the case when you run the AE without trace ? Try :
                  1. remove the trace option from the AE definition
                  2. append the option instead of override
                  Case described in the note Application Engine Fails with Service RunAeProgAsync failed [ID 618643.1]
                  >

                  The trace (-TRACE 387 -TOOLSTRACEPC 3596) was appended to the parameter list of the process. Removing this does not change anything, I also checked definitions between process scheduler servers where it runs (e.g. a PSNT definition), it's the same as far as override options go.

                  >
                  First of all PeopleSoft Financials 9.1 is not certified on PeopleTools 8.49. Did you recently upgrade to this version?
                  >
                  My mistake, I will correct that 8.49 to 8.52 in the original post.

                  >
                  I would try creating a new process scheduler domain with a new configuration file. This rules out any issues with the configuration file itself. Another option is setting the App Eng Server in the quick configuration to No. This option creates a separate Tuxedo process for Application Engines, which isn't required to run Application Engines. This process seems non existent in your Tuxedo server, which is giving the error messages seen in the process scheduler log.
                  >
                  I did create a new scheduler domain with no progress, I will try recreating the application domain as well.

                  >
                  The TUXLOG might give insights as to why this error is occuring. Could you paste the contents of this file here?
                  >
                  Here is the tuxlog:
                  162825.vm-psdemo-app1!tmloadcf.4304.405370112.-2: 01-09-2013: client high water (0), total client (0)
                  162825.vm-psdemo-app1!tmloadcf.4304.405370112.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  162825.vm-psdemo-app1!tmloadcf.4304.405370112.-2: CMDTUX_CAT:879: INFO: A new file system has been created. (size = 980 512-byte blocks)
                  162825.vm-psdemo-app1!tmloadcf.4304.405370112.-2: CMDTUX_CAT:871: INFO: TUXCONFIG file /home/fc91/psft/pt/8.52/appserv/prcs/FC91DB/PSTUXCFG has been created
                  162825.vm-psdemo-app1!PSADMIN.2715: Begin  attempt on domain FC91DB
                  162829.vm-psdemo-app1!tmadmin.4309.2222861424.-2: TMADMIN_CAT:1330: INFO: Command: boot -A
                  162833.vm-psdemo-app1!BBL.4311.452553824.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit, Patch Level (none)
                  162833.vm-psdemo-app1!BBL.4311.452553824.0: LIBTUX_CAT:262: INFO: Standard main starting
                  162837.vm-psdemo-app1!PSMSTPRC.4315.3348190624.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  162837.vm-psdemo-app1!PSMSTPRC.4315.3348190624.-2: LIBTUX_CAT:262: INFO: Standard main starting
                  162839.vm-psdemo-app1!PSDSTSRV.4323.3595146656.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  162839.vm-psdemo-app1!PSDSTSRV.4323.3595146656.0: LIBTUX_CAT:262: INFO: Standard main starting
                  162840.vm-psdemo-app1!PSPRCSRV.4328.3671094688.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  162840.vm-psdemo-app1!PSPRCSRV.4328.3671094688.-2: LIBTUX_CAT:262: INFO: Standard main starting
                  162841.vm-psdemo-app1!PSMONITORSRV.4336.1247655952.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  162841.vm-psdemo-app1!PSMONITORSRV.4336.1247655952.-2: LIBTUX_CAT:262: INFO: Standard main starting
                  162842.vm-psdemo-app1!PSADMIN.2715: End boot attempt on domain FC91DB
                  163047.vm-psdemo-app1!tmadmin.4389.483019888.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  163047.vm-psdemo-app1!tmadmin.4389.483019888.-2: TMADMIN_CAT:1330: INFO: Command: boot -A
                  163049.vm-psdemo-app1!BBL.4394.3046009952.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit, Patch Level (none)
                  163049.vm-psdemo-app1!BBL.4394.3046009952.0: LIBTUX_CAT:577: ERROR: Unable to register because the slot is already owned by another process
                  163049.vm-psdemo-app1!BBL.4394.3046009952.0: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr =
                  163049.vm-psdemo-app1!BBL.4394.3046009952.0: CMDTUX_CAT:26: INFO: The BBL is exiting system
                  163054.vm-psdemo-app1!BBL.4311.452553824.0: CMDTUX_CAT:26: INFO: The BBL is exiting system
                  163126.vm-psdemo-app1!tmloadcf.4403.3192333568.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  163126.vm-psdemo-app1!tmloadcf.4403.3192333568.-2: CMDTUX_CAT:879: INFO: A new file system has been created. (size = 980 512-byte blocks)
                  163126.vm-psdemo-app1!tmloadcf.4403.3192333568.-2: CMDTUX_CAT:871: INFO: TUXCONFIG file /home/fc91/psft/pt/8.52/appserv/prcs/FC91DB/PSTUXCFG has been created
                  163127.vm-psdemo-app1!PSADMIN.2715: Begin  attempt on domain FC91DB
                  163131.vm-psdemo-app1!tmadmin.4410.1259068528.-2: TMADMIN_CAT:1330: INFO: Command: boot -A
                  163135.vm-psdemo-app1!BBL.4412.3280346208.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit, Patch Level (none)
                  163135.vm-psdemo-app1!BBL.4412.3280346208.0: LIBTUX_CAT:262: INFO: Standard main starting
                  163137.vm-psdemo-app1!PSMSTPRC.4416.2734544288.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  163137.vm-psdemo-app1!PSMSTPRC.4416.2734544288.-2: LIBTUX_CAT:262: INFO: Standard main starting
                  163138.vm-psdemo-app1!PSDSTSRV.4421.1773737376.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  163138.vm-psdemo-app1!PSDSTSRV.4421.1773737376.0: LIBTUX_CAT:262: INFO: Standard main starting
                  163139.vm-psdemo-app1!PSPRCSRV.4426.1229747616.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  163139.vm-psdemo-app1!PSPRCSRV.4426.1229747616.-2: LIBTUX_CAT:262: INFO: Standard main starting
                  163140.vm-psdemo-app1!PSMONITORSRV.4434.264923152.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
                  163140.vm-psdemo-app1!PSMONITORSRV.4434.264923152.-2: LIBTUX_CAT:262: INFO: Standard main starting
                  163141.vm-psdemo-app1!PSADMIN.2715: End boot attempt on domain FC91DB
                  Could you check if you have other processes with a status Error/No Success?
                  If this is a non-production environment, try cleaning the processes with an Error/No Success status and try again.
                  >
                  Yes, I cleared the queue.

                  Updated tuxlog
                  • 6. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                    Honza
                    >
                    Another option is setting the App Eng Server in the quick configuration to No. This option creates a separate Tuxedo process for Application Engines, which isn't required to run Application Engines. This process seems non existent in your Tuxedo server, which is giving the error messages seen in the process scheduler log.
                    >

                    Interesing about setting App Eng Server to No, I will definitely try some testing with this. The processes then run into an error.
                    Process Request shows status of 'INITIATED' or 'PROCESSING' but no longer running
                    4:34:44PM
                    Published message with ID 1750597a-5a72-11e2-b91a-fd04306aa2b2 to create entry in folder GENERAL.
                    • 7. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                      Honza
                      I've updated the tuxlog this line seems like the problem when booting:
                      163049.vm-psdemo-app1!BBL.4394.3046009952.0: LIBTUX_CAT:577: ERROR: Unable to register because the slot is already owned by another process
                      163049.vm-psdemo-app1!BBL.4394.3046009952.0: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr =
                      Is it possible there is a conflict between application server and process scheduler? I'm not aware of any other processes running on this server.

                      Edit:
                      This only appears with the No option for Application Engine.

                      Edited by: Honza on Jan 9, 2013 7:48 AM
                      • 8. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                        Nicolas.Gasparotto
                        Honza wrote:
                        ...
                        >
                        First of all PeopleSoft Financials 9.1 is not certified on PeopleTools 8.49. Did you recently upgrade to this version?
                        >
                        My mistake, I will correct that 8.49 to 8.52 in the original post.
                        Since we are talking about certification and you are updating your first post, bear in mind that PeopleSoft PeopleTools 8.52 is not certified on Linux x86 Oracle Linux 5.

                        Nicolas.
                        • 9. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                          Honza
                          Nicolas, nice catch, did not realize x86 was not certified, however this is on x86_64 which is.
                          • 10. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                            Nicolas.Gasparotto
                            Looks like your server is not properly configured. You may have some kernel parameters to adjust properly.
                            I'm not a fan of providing only links to the notes, but you should really have a look to the differents following notes :
                            PeopleSoft Servers Die With: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr = : msgget: No such file or directory [ID 1298855.1]
                            TUX.ALL - Possible causes of error LIBTUX_CAT:577: Unable to register because the slot is already owned by another process [ID 764563.1]
                            They can explain your lastest errors.

                            By the way, has it aever worked before ? Or is it the very first try ?

                            Nicolas.
                            • 11. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                              Honza
                              >
                              Looks like your server is not properly configured. You may have some kernel parameters to adjust properly.
                              I'm not a fan of providing only links to the notes, but you should really have a look to the differents following notes :
                              PeopleSoft Servers Die With: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr = : msgget: No such file or directory [ID 1298855.1]
                              TUX.ALL - Possible causes of error LIBTUX_CAT:577: Unable to register because the slot is already owned by another process [ID 764563.1]
                              They can explain your lastest errors.

                              By the way, has it aever worked before ? Or is it the very first try ?
                              >

                              Those errors only appear when the scheduler is set without applications engines to be configured, but I havn't been through those documents in too much detail yet.

                              And yes, it has worked before, I can see successful completion of application engines on this server less than a month ago, but I'm not aware of any changes after that date, there shouldn't have been any activity during the holidays .. I might consider looking for backups, but I would prefer to find the resolution.
                              • 12. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                                Michel
                                That App Eng server setting creates a separate Tuxedo process for Application Engines when turned on. When off all Application Engines will be run in the regular PSPRCS Tuxedo process.
                                Disabling the Application Engine Server (PSAESRV) from the Process Scheduler is a very common solution, and it will not affect the existing PeopleTools functionality on the process scheduler to be able to execute any Application Engine program as usual. There are two supported method in process scheduler to run Application Engine program, PSAE or PSAESRV, enabling or disabling Application Engine Server in the process scheduler config toggle the two method around.
                                The impact of disabling it is minimal.

                                1. The AE will take a bit longer to start, time delay may be range from millisecond to seconds, it is variant depending on your hardware and software configuration, and it will start the psae directly, rather then through psaesrv.

                                The net gain is that even when the Process Scheduler is shutting down, the launched psae processes (corresponding to the running AE processes) will keep running until finish (as they are independent processes.). And the other advantage is that you don't have to worry about the accumulated memory usage of the PSAESRV anymore, as it will be cleaned after each execution.

                                2. The AE will get better performance with PSAESRV, only in the situation of large amount of small AE program is required to run in a short period of time on the process scheduler, for example, 1000 or more small AE program is required to run in an hour. Please note, the above number of AE and time period may change according to your hardware and software configuration.
                                Back to the problem however; the error message about the slot already being in use. Are there duplicate process scheduler servers running?
                                • 13. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                                  Honza
                                  Michael, thanks for that quote, i assume its from the PeopleBooks so I will look it up to get more info.
                                  However, there are no other schedulers running on this server, just the single application domain and process scheduler.

                                  I'm still trying to figure out why this suddenly occurred, but I can't find anything about any recent changes, couldn't have been just out of the blue.

                                  Since recreating the scheduler domain doesn't seem to have an impact, this suggests a tuxedo problem, but I never had problems with tuxedo AFTER initial configuration and correct environment settings.
                                  • 14. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
                                    Michel
                                    It's actually an article on Oracle support (ID 651970.1).

                                    The weird thing about the error in the TUXLOG is that it doesn't specify the error message. Normally you'd get some sort of reason after the "Uunixerr = ". The only case I can find on Oracle support where there isn't an error message displayed refers to Tuxedo 7 and a rolling patch where an error is fixed. That doesn't apply to your situation since this isn't Tuxedo 7. It does say that the BB is corrupt when such an error message is displayed. One last thing I can suggest; can you clean the IPC resources for the process scheduler domain? Should be option 10 in the process scheduler domain configuration in PSADMIN.
                                    1 2 Previous Next