This discussion is archived
1 2 Previous Next 26 Replies Latest reply: Jan 10, 2013 4:13 AM by Michel RSS

Application Engine initiated: Tuxedo cannot find RunAeAsync

Honza Newbie
Currently Being Moderated
Hello,

running PeopleSoft Financials 9.1 on PeopleTools 8.52.07 (on OEL 5 64bit) I came across the following problem when running any kind of application engine
PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)    Server: PSUNX checking status...
PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)    Server action mode: Ok (looking for requests)
PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)    Server: PSUNX looking for work
PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)    Checking Process cancels...
PSPRCSRV.2899 (0) [01/09/13 15:37:15](3)      Requests found in Process Request table         2
PSPRCSRV.2899 (0) [01/09/13 15:37:16](3)    Server: PSUNX checking status...
PSPRCSRV.2899 (0) [01/09/13 15:37:16](3)    Server action mode: Submitting request
PSPRCSRV.2899 (0) [01/09/13 15:37:16](3)    Number of New Process Request(s) To Start: 2
PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync1.  Make sure the application server advertising this service is booted.
PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync2.  Make sure the application server advertising this service is booted.
PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync3.  Make sure the application server advertising this service is booted.
PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync1.  Make sure the application server advertising this service is booted.
PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync2.  Make sure the application server advertising this service is booted.
PSPRCSRV.2899 (0) [01/09/13 15:37:16](2)        Process Scheduler encountered an error sending the request to the AE Tuxedo Server
PSPRCSRV.2899 (0) [01/09/13 15:37:16](1) (NET.334): Tuxedo cannot find the service RunAeAsync3.  Make sure the application server advertising this service is booted.
Here is an example of process parameters:
psae -CT ORACLE -CD FC91DB -CO "VP1" -CP OPRPSWD -R 1 -I 47592 -AI AEMINITEST -OT 6 -FP "/home/fc91/psft/pt/8.52/appserv/prcs/FC91DB/log_output/AE_AEMINITEST_47592/" -OF 14 -TRACE 387 -TOOLSTRACEPC 3596
I've tried deleting the application and scheduler domain and recreating with the same configuration file as well as rebooting the scheduler as well as setting the Recycle Count to 0, but there is very limited information on this on metalink.

I'm looking for any input or ideas on this. Thank you.
Best regards.

Edited by: Honza on Jan 9, 2013 7:28 AM

Edited by: Honza on Jan 9, 2013 7:52 AM
Corrected Tools and OS versions.
  • 1. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Nicolas.Gasparotto Oracle ACE
    Currently Being Moderated
    Wondering why you have trace turned on. Is it also the case when you run the AE without trace ? Try :
    1. remove the trace option from the AE definition
    2. append the option instead of override
    Case described in the note *Application Engine Fails with Service RunAeProgAsync failed [ID 618643.1]*+

    Nicolas.
  • 2. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Michel Journeyer
    Currently Being Moderated
    First of all PeopleSoft Financials 9.1 is not certified on PeopleTools 8.49. Did you recently upgrade to this version?

    I would try creating a new process scheduler domain with a new configuration file. This rules out any issues with the configuration file itself. Another option is setting the App Eng Server in the quick configuration to No. This option creates a separate Tuxedo process for Application Engines, which isn't required to run Application Engines. This process seems non existent in your Tuxedo server, which is giving the error messages seen in the process scheduler log.

    The TUXLOG might give insights as to why this error is occuring. Could you paste the contents of this file here?
  • 3. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    HakanBiroglu Oracle ACE
    Currently Being Moderated
    Could you check if you have other processes with a status Error/No Success?
    If this is a non-production environment, try cleaning the processes with an Error/No Success status and try again.
  • 4. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Nicolas.Gasparotto Oracle ACE
    Currently Being Moderated
    Michel wrote:
    First of all PeopleSoft Financials 9.1 is not certified on PeopleTools 8.49. Did you recently upgrade to this version?
    That's a good one.

    Nicolas.
  • 5. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Honza Newbie
    Currently Being Moderated
    >
    Wondering why you have trace turned on. Is it also the case when you run the AE without trace ? Try :
    1. remove the trace option from the AE definition
    2. append the option instead of override
    Case described in the note Application Engine Fails with Service RunAeProgAsync failed [ID 618643.1]
    >

    The trace (-TRACE 387 -TOOLSTRACEPC 3596) was appended to the parameter list of the process. Removing this does not change anything, I also checked definitions between process scheduler servers where it runs (e.g. a PSNT definition), it's the same as far as override options go.

    >
    First of all PeopleSoft Financials 9.1 is not certified on PeopleTools 8.49. Did you recently upgrade to this version?
    >
    My mistake, I will correct that 8.49 to 8.52 in the original post.

    >
    I would try creating a new process scheduler domain with a new configuration file. This rules out any issues with the configuration file itself. Another option is setting the App Eng Server in the quick configuration to No. This option creates a separate Tuxedo process for Application Engines, which isn't required to run Application Engines. This process seems non existent in your Tuxedo server, which is giving the error messages seen in the process scheduler log.
    >
    I did create a new scheduler domain with no progress, I will try recreating the application domain as well.

    >
    The TUXLOG might give insights as to why this error is occuring. Could you paste the contents of this file here?
    >
    Here is the tuxlog:
    162825.vm-psdemo-app1!tmloadcf.4304.405370112.-2: 01-09-2013: client high water (0), total client (0)
    162825.vm-psdemo-app1!tmloadcf.4304.405370112.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    162825.vm-psdemo-app1!tmloadcf.4304.405370112.-2: CMDTUX_CAT:879: INFO: A new file system has been created. (size = 980 512-byte blocks)
    162825.vm-psdemo-app1!tmloadcf.4304.405370112.-2: CMDTUX_CAT:871: INFO: TUXCONFIG file /home/fc91/psft/pt/8.52/appserv/prcs/FC91DB/PSTUXCFG has been created
    162825.vm-psdemo-app1!PSADMIN.2715: Begin  attempt on domain FC91DB
    162829.vm-psdemo-app1!tmadmin.4309.2222861424.-2: TMADMIN_CAT:1330: INFO: Command: boot -A
    162833.vm-psdemo-app1!BBL.4311.452553824.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit, Patch Level (none)
    162833.vm-psdemo-app1!BBL.4311.452553824.0: LIBTUX_CAT:262: INFO: Standard main starting
    162837.vm-psdemo-app1!PSMSTPRC.4315.3348190624.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    162837.vm-psdemo-app1!PSMSTPRC.4315.3348190624.-2: LIBTUX_CAT:262: INFO: Standard main starting
    162839.vm-psdemo-app1!PSDSTSRV.4323.3595146656.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    162839.vm-psdemo-app1!PSDSTSRV.4323.3595146656.0: LIBTUX_CAT:262: INFO: Standard main starting
    162840.vm-psdemo-app1!PSPRCSRV.4328.3671094688.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    162840.vm-psdemo-app1!PSPRCSRV.4328.3671094688.-2: LIBTUX_CAT:262: INFO: Standard main starting
    162841.vm-psdemo-app1!PSMONITORSRV.4336.1247655952.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    162841.vm-psdemo-app1!PSMONITORSRV.4336.1247655952.-2: LIBTUX_CAT:262: INFO: Standard main starting
    162842.vm-psdemo-app1!PSADMIN.2715: End boot attempt on domain FC91DB
    163047.vm-psdemo-app1!tmadmin.4389.483019888.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    163047.vm-psdemo-app1!tmadmin.4389.483019888.-2: TMADMIN_CAT:1330: INFO: Command: boot -A
    163049.vm-psdemo-app1!BBL.4394.3046009952.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit, Patch Level (none)
    163049.vm-psdemo-app1!BBL.4394.3046009952.0: LIBTUX_CAT:577: ERROR: Unable to register because the slot is already owned by another process
    163049.vm-psdemo-app1!BBL.4394.3046009952.0: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr =
    163049.vm-psdemo-app1!BBL.4394.3046009952.0: CMDTUX_CAT:26: INFO: The BBL is exiting system
    163054.vm-psdemo-app1!BBL.4311.452553824.0: CMDTUX_CAT:26: INFO: The BBL is exiting system
    163126.vm-psdemo-app1!tmloadcf.4403.3192333568.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    163126.vm-psdemo-app1!tmloadcf.4403.3192333568.-2: CMDTUX_CAT:879: INFO: A new file system has been created. (size = 980 512-byte blocks)
    163126.vm-psdemo-app1!tmloadcf.4403.3192333568.-2: CMDTUX_CAT:871: INFO: TUXCONFIG file /home/fc91/psft/pt/8.52/appserv/prcs/FC91DB/PSTUXCFG has been created
    163127.vm-psdemo-app1!PSADMIN.2715: Begin  attempt on domain FC91DB
    163131.vm-psdemo-app1!tmadmin.4410.1259068528.-2: TMADMIN_CAT:1330: INFO: Command: boot -A
    163135.vm-psdemo-app1!BBL.4412.3280346208.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit, Patch Level (none)
    163135.vm-psdemo-app1!BBL.4412.3280346208.0: LIBTUX_CAT:262: INFO: Standard main starting
    163137.vm-psdemo-app1!PSMSTPRC.4416.2734544288.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    163137.vm-psdemo-app1!PSMSTPRC.4416.2734544288.-2: LIBTUX_CAT:262: INFO: Standard main starting
    163138.vm-psdemo-app1!PSDSTSRV.4421.1773737376.0: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    163138.vm-psdemo-app1!PSDSTSRV.4421.1773737376.0: LIBTUX_CAT:262: INFO: Standard main starting
    163139.vm-psdemo-app1!PSPRCSRV.4426.1229747616.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    163139.vm-psdemo-app1!PSPRCSRV.4426.1229747616.-2: LIBTUX_CAT:262: INFO: Standard main starting
    163140.vm-psdemo-app1!PSMONITORSRV.4434.264923152.-2: 01-09-2013: Tuxedo Version 10.3.0.0, 64-bit
    163140.vm-psdemo-app1!PSMONITORSRV.4434.264923152.-2: LIBTUX_CAT:262: INFO: Standard main starting
    163141.vm-psdemo-app1!PSADMIN.2715: End boot attempt on domain FC91DB
    Could you check if you have other processes with a status Error/No Success?
    If this is a non-production environment, try cleaning the processes with an Error/No Success status and try again.
    >
    Yes, I cleared the queue.

    Updated tuxlog
  • 6. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Honza Newbie
    Currently Being Moderated
    >
    Another option is setting the App Eng Server in the quick configuration to No. This option creates a separate Tuxedo process for Application Engines, which isn't required to run Application Engines. This process seems non existent in your Tuxedo server, which is giving the error messages seen in the process scheduler log.
    >

    Interesing about setting App Eng Server to No, I will definitely try some testing with this. The processes then run into an error.
    Process Request shows status of 'INITIATED' or 'PROCESSING' but no longer running
    4:34:44PM
    Published message with ID 1750597a-5a72-11e2-b91a-fd04306aa2b2 to create entry in folder GENERAL.
  • 7. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Honza Newbie
    Currently Being Moderated
    I've updated the tuxlog this line seems like the problem when booting:
    163049.vm-psdemo-app1!BBL.4394.3046009952.0: LIBTUX_CAT:577: ERROR: Unable to register because the slot is already owned by another process
    163049.vm-psdemo-app1!BBL.4394.3046009952.0: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr =
    Is it possible there is a conflict between application server and process scheduler? I'm not aware of any other processes running on this server.

    Edit:
    This only appears with the No option for Application Engine.

    Edited by: Honza on Jan 9, 2013 7:48 AM
  • 8. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Nicolas.Gasparotto Oracle ACE
    Currently Being Moderated
    Honza wrote:
    ...
    >
    First of all PeopleSoft Financials 9.1 is not certified on PeopleTools 8.49. Did you recently upgrade to this version?
    >
    My mistake, I will correct that 8.49 to 8.52 in the original post.
    Since we are talking about certification and you are updating your first post, bear in mind that PeopleSoft PeopleTools 8.52 is not certified on Linux x86 Oracle Linux 5.

    Nicolas.
  • 9. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Honza Newbie
    Currently Being Moderated
    Nicolas, nice catch, did not realize x86 was not certified, however this is on x86_64 which is.
  • 10. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Nicolas.Gasparotto Oracle ACE
    Currently Being Moderated
    Looks like your server is not properly configured. You may have some kernel parameters to adjust properly.
    I'm not a fan of providing only links to the notes, but you should really have a look to the differents following notes :
    PeopleSoft Servers Die With: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr = : msgget: No such file or directory [ID 1298855.1]
    TUX.ALL - Possible causes of error LIBTUX_CAT:577: Unable to register because the slot is already owned by another process [ID 764563.1]
    They can explain your lastest errors.

    By the way, has it aever worked before ? Or is it the very first try ?

    Nicolas.
  • 11. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Honza Newbie
    Currently Being Moderated
    >
    Looks like your server is not properly configured. You may have some kernel parameters to adjust properly.
    I'm not a fan of providing only links to the notes, but you should really have a look to the differents following notes :
    PeopleSoft Servers Die With: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr = : msgget: No such file or directory [ID 1298855.1]
    TUX.ALL - Possible causes of error LIBTUX_CAT:577: Unable to register because the slot is already owned by another process [ID 764563.1]
    They can explain your lastest errors.

    By the way, has it aever worked before ? Or is it the very first try ?
    >

    Those errors only appear when the scheduler is set without applications engines to be configured, but I havn't been through those documents in too much detail yet.

    And yes, it has worked before, I can see successful completion of application engines on this server less than a month ago, but I'm not aware of any changes after that date, there shouldn't have been any activity during the holidays .. I might consider looking for backups, but I would prefer to find the resolution.
  • 12. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Michel Journeyer
    Currently Being Moderated
    That App Eng server setting creates a separate Tuxedo process for Application Engines when turned on. When off all Application Engines will be run in the regular PSPRCS Tuxedo process.
    Disabling the Application Engine Server (PSAESRV) from the Process Scheduler is a very common solution, and it will not affect the existing PeopleTools functionality on the process scheduler to be able to execute any Application Engine program as usual. There are two supported method in process scheduler to run Application Engine program, PSAE or PSAESRV, enabling or disabling Application Engine Server in the process scheduler config toggle the two method around.
    The impact of disabling it is minimal.

    1. The AE will take a bit longer to start, time delay may be range from millisecond to seconds, it is variant depending on your hardware and software configuration, and it will start the psae directly, rather then through psaesrv.

    The net gain is that even when the Process Scheduler is shutting down, the launched psae processes (corresponding to the running AE processes) will keep running until finish (as they are independent processes.). And the other advantage is that you don't have to worry about the accumulated memory usage of the PSAESRV anymore, as it will be cleaned after each execution.

    2. The AE will get better performance with PSAESRV, only in the situation of large amount of small AE program is required to run in a short period of time on the process scheduler, for example, 1000 or more small AE program is required to run in an hour. Please note, the above number of AE and time period may change according to your hardware and software configuration.
    Back to the problem however; the error message about the slot already being in use. Are there duplicate process scheduler servers running?
  • 13. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Honza Newbie
    Currently Being Moderated
    Michael, thanks for that quote, i assume its from the PeopleBooks so I will look it up to get more info.
    However, there are no other schedulers running on this server, just the single application domain and process scheduler.

    I'm still trying to figure out why this suddenly occurred, but I can't find anything about any recent changes, couldn't have been just out of the blue.

    Since recreating the scheduler domain doesn't seem to have an impact, this suggests a tuxedo problem, but I never had problems with tuxedo AFTER initial configuration and correct environment settings.
  • 14. Re: Application Engine initiated: Tuxedo cannot find RunAeAsync
    Michel Journeyer
    Currently Being Moderated
    It's actually an article on Oracle support (ID 651970.1).

    The weird thing about the error in the TUXLOG is that it doesn't specify the error message. Normally you'd get some sort of reason after the "Uunixerr = ". The only case I can find on Oracle support where there isn't an error message displayed refers to Tuxedo 7 and a rolling patch where an error is fixed. That doesn't apply to your situation since this isn't Tuxedo 7. It does say that the BB is corrupt when such an error message is displayed. One last thing I can suggest; can you clean the IPC resources for the process scheduler domain? Should be option 10 in the process scheduler domain configuration in PSADMIN.
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points