8 Replies Latest reply: Dec 21, 2010 9:06 PM by Pranilesh Chand. RSS

    Answers to "Why are my jobs not running?"


      This is one of the most common Scheduler questions asked.
      Here we list some of the common problems and their solutions.

      1) job_queue_processes may be too low (this is the most common problem)
      The value of job_queue_processes limits the total number of dbms_scheduler
      and dbms_job jobs that can be running at a given time.
      To check whether this is the case check the current value of
      job_queue_processes with
      SQL> select value from v$parameter where name='job_queue_processes';
      Then check the number of running jobs
      SQL> select count(*) from dba_scheduler_running_jobs;
      SQL> select count(*) from dba_jobs_running;

      If this is the problem you can increase the parameter using
      SQL> alter system set job_queue_processes=1000;

      2) max_job_slave_processes may be too low
      If this parameter is not NULL then it limits how many dbms_scheduler jobs can
      be running at a time. To check whether this is the problem, check the current
      value using
      SQL> select value from dba_scheduler_global_attribute
      where attribute_name='MAX_JOB_SLAVE_PROCESSES';
      Then check the number of running jobs
      SQL> select count(*) from dba_scheduler_running_jobs;

      If this is the problem you can increase the number or just NULL it out using
      SQL> exec dbms_scheduler.set_scheduler_attribute('max_job_slave_processes',null)

      3) sessions may be too low
      This parameter limits the number of sessions at any time. Every Scheduler job
      requires 2 sessions. To check whether this is the problem, check the current
      valule using
      SQL> select value from v$parameter where name='sessions';
      Then check the current number of sessions using
      SQL> select count(*) from v$session ;

      If the numbers are too close you can increase the maximum using
      SQL> alter system set job_queue_processes=200;

      4) Have you recently applied a timezone update patch or upgraded the database
      to a version with newer timezone information ? If you skipped any steps when
      updating the timezone information, jobs may not run. To check whether this
      is the case try doing
      SQL> select * from sys.scheduler$_job;
      SQL> select * from sys.scheduler$_window;
      and make sure they finish without errors.

      If it throws a timezone warning, reapply the upgrade or
      timezone patch making sure to follow all the steps.

      5) Is the database running in restricted mode ?
      If the database is running in restricted mode then no jobs will run (unless
      you are using 11g and use the ALLOW_RUNS_IN_RESTRICTED_MODE attribute).
      To check this use
      SQL> select logins from v$instance ;

      If logins is restricted you can disable the restricted mode using

      6) Is the job scheduled to run on an instance which is down ?

      You can check this by seeing whether instance_id is set for the job (check the dba_scheduler_jobs view), and if so you should check whether that instance is up.

      7) Is the job scheduled to run on a service which has not been started on any instances ?

      You can check this by checking what job_class a job points to and then checking whether that class points to a service. If it does, make sure the service has been started on at least one running instance. You can start a service on an instance using dbms_service.start_service.

      8) Is the Resource Manager in effect with a restrictive resource plan ?

      If a restrictive resource plan is in effect, scheduler jobs may not have sufficient resources allocated so they may not run. You can check what resource plan is in effect by doing

      SQL> select name from V$RSRC_PLAN ;

      If no plan is in effect or the plan in effect is INTERNAL_PLAN then the resource manager is not in effect. If the resource manager is in effect you can disable it by doing

      SQL>alter system set resource_manager_plan = '';

      9) Has the Scheduler been disabled ? This is not a supported action
      but it is possible that someone has done it anyway. To check this do
      SQL> select value from dba_scheduler_global_attribute where attribute_name='SCHEDULER_DISABLED'

      If this query returns TRUE then you can fix this using
      SQL> exec dbms_scheduler.set_scheduler_attribute('scheduler_disabled','false');

      Reasons why jobs may run late

      1) The first thing to check is the timezone that the job is scheduled with
      SQL> select owner, job_name, next_run_date from dba_scheduler_jobs ;

      If the jobs are in the wrong timezone they may not run at the expected
      time. If the next_run_date is using an absolute timezone offset (like
      +08:00) instead of a named timezone (like US/PACIFIC) then the jobs may not
      run as expected if daylight savings is in effect - they may run an hour
      early or late.

      2) It may be that at the time the job was scheduled to run, one of the several
      limits above may have been temporarily reached causing the job to be delayed.
      Check if the limits above are high enough and if possible check them during
      the time that the job is being delayed.

      3) One possible reason that one of the above limits may be hit is that a
      maintenance window may have come into effect. Maintenance windows are Oracle
      Scheduler windows that belong to the window group named
      MAINTENANCE_WINDOW_GROUP. During a scheduled maintenance window, several
      maintenance tasks are run using jobs. This may cause one of the limits listed
      above to be hit and user jobs to be delayed. See the admin guide for more info
      about this (chapter 24).

      To get a list of maintenance windows use
      SQL> select * from dba_scheduler_wingroup_members;

      To see when the windows run use
      SQL> select * from dba_scheduler_windows;

      To fix this you can either increase the limits or reschedule the maintenance
      windows to run at more convenient times.

      Diagnosing other Problems

      If none of this works, here are some further steps you can take to try to
      figure out what is going on.

      1) Check whether there are any errors in the alert log. If the database is
      having trouble allocating memory or has run out of disk space or any other
      catastrophic errors have occurred, you should resolve those first. You can
      find the location of the alert log by using
      SQL> select value from v$parameter where name = 'background_dump_dest';
      The alert log will be in this directory with a name starting with "alert".

      2) Check whether if a job coordinator trace file and if it does, check if it
      contains any errors. If this exists, it will be located in the
      'background_dump_dest' directory which you can find as above and will look
      something like SID-cjq0_nnnn.trc . If there are any errors here they may
      hint at why jobs are not running.

      3) If either of the above indicates that the SYSAUX tablespace (where the scheduler stores its logging tables) is full, you can use the dbms_scheduler.purge_log procedure to clear out old log entries.

      4) See if there is a window currently open. If there is, you can try closing it to see if that helps .
      SQL> select * from DBA_SCHEDULER_GLOBAL_ATTRIBUTE where
      SQL> exec DBMS_SCHEDULER.close_window ('WEEKNIGHT_WINDOW');

      5)try running a simple run-once job and see if it runs
      dbms_scheduler.create_job (
      job_name => 'test_job',
      job_type => 'plsql_block',
      job_action => 'null;',
      enabled => true);
      SQL> -- wait a while
      SQL> select * from user_scheduler_job_run_details where job_name='TEST_JOB';

      6) If a simple run-once job doesn't run, you can try restarting the scheduler as follows.

      SQL> exec dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED', 'TRUE');
      SQL> alter system set job_queue_processes=0;
      SQL> exec dbms_ijob.set_enabled(FALSE);
      SQL> alter system flush shared_pool;
      SQL> alter system flush shared_pool;
      SQL> exec dbms_ijob.set_enabled(TRUE);
      SQL> alter system set job_queue_processes=99;
      SQL> exec dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED', 'FALSE');
        • 1. Re: Answers to "Why are my jobs not running?"
          Very good info. Thanks. One more thing, though. How about jobs that hang in the "running" state? We have a job that does an export of a user's partitioned tables by invoking a shell script on the local file system. When we run the script from the OS command line, it completes fine. When we run the exact same script from the scheduler, it hangs. I'd love to see any troubleshooting pointers for jobs that don't complete you may have put together. It's almost like it times out...


          • 2. Re: Answers to "Why are my jobs not running?"

            Hanging PL/SQL jobs and hanging external jobs need to be treated differently.

            - hanging pl/sql jobs are typically associated with a real background session which is hanging and issues can be tracked down using normal tools/methods to track down oracle hangs (see what sql statement is being run, see what locks are being held/waited on etc).

            - hanging external jobs may be the result of a hanging script or may be hanging because the external job never returned results. Both of these issues are covered in the post on running external jobs here
            Guide to External Jobs on 10g with dbms_scheduler e.g. scripts,batch files

            First check whether the script is running (look through the process list for the name of the script or the commands that the script runs).

            If it is, then you need to debug the script itself or you may be running into the bug where the scheduler hangs on large amounts of stderr (fixed in version in which case you should redirect stderr to a file.

            If nothing is running then you should check your permissions, in particular, the user that the job runs as must have execute access on $ORACLE_HOME/bin and all parent directories. So for e.g. if your $ORACLE_HOME is /opt/oracle/db then you would have to make sure that

            chmod a+rx /opt
            chmod a+rx /opt/oracle
            chmod a+rx /opt/oracle/db
            chmod a+rx /opt/oracle/db/bin

            Hope this helps,
            • 3. Re: Answers to "Why are my jobs not running?"
              You nailed it, Ravi. The script was calling the exp command. The command had "> /dev/null" appended to it. Once I changed it to ">& /dev/null", it all worked fine.

              Thanks a lot

              • 4. Re: Answers to "Why are my jobs not running?"
                Ronald Rood
                A job that has auto_drop = false and end_time < sysdate also won't run. From what I see, resetting the end_date to null makes the job look like able to run but the job log keeps showing 'end time reached'.

                • 5. Re: Answers to "Why are my jobs not running?"
                  Hi Ronald,

                  As you probably know. Jobs should not be running if end_date < sysdate since this is the purpose of the end_date attribute as documented.

                  However, it looks like you have run into a bug in that resetting the end_date and then reenabling the job still doesn't force the job to start running again. This is the first time that this bug has been reported.

                  I will need some info from you to be able to better track this as a bug and get it fixed in the next release (and possibly backported). I will try to contact you directly.

                  • 6. Re: Answers to "Why are my jobs not running?"

                    The bug Ronald found (resetting end_date for a completed job does not work) has been filed internally as bug #7607696 and is being investigated.

                    • 7. Re: Answers to "Why are my jobs not running?"
                      Ronald Rood
                      Hi Ravi,

                      an other reason for a job not to run is that the database is, has plenty resources available and is using resource manager. I see this happen on systems that have lots of cpu idle, a resource manager plan that has no limits on cpu. What I see is that the job just keeps status scheduled. In the resource manager monitors there is nothing to see what is waiting for resources. It all starts running as soon as we switch to the INTERNAL_PLAN, effectively switching off resource manager.
                      Strange is there are no waiters for resources ...

                      • 8. Re: Answers to "Why are my jobs not running?"
                        Pranilesh Chand.

                        Just want to add that if you are using dbms_job to submit your job, your job will be running in accordance of how your database timezone is set.

                        For example,
                        My dbtimezone is set to a time which is 2 hours behind than my local server date.
                        Hence, if i will submit a job via dbms_job procedure to run now, the job will actually run 2 hours later.