Step by Step Instructions for Detection, Diagnosis & Resolution of ORA-600 (ORA-00600 Internal error)

 

As a DBA it is not uncommon to find ORA-00600 errors in your alert log.

 

Because there is no other context to the ORA-600 it can be difficult to know if it's something you need to act on.

 

This post guides you through what the error means, how to use Oracle Trace File Analyzer (TFA) to detect it, how to diagnose the cause and then how to resolve.

Here's what's covered below:

 

 

This is the overall process for detection, diagnosis & resolution:

ORA-600 detection diagnosis and resolution process.jpg

 

What is an ORA-600 (ORA-00600 internal error)?

ORA-00600 internal error, also referred to as ORA-600 is a generic error code. The Oracle Database uses ORA-00600 when it encounters an unexpected internal condition.

The error will look like this:

Thu Sep 13 08:09:49 2018

Errors in file /scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/trace/ogg112041_ora_8881.trc  (incident=99577):

ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], []

Incident details in: /scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/incident/incdir_99577/ogg112041_ora_8881_i99577.trc

The arguments in square brackets will contain different values. These depend on where in the code the problem occurred and what it was doing.

 

The value of the first argument shows where in the code the error occurred. Later arguments contain other runtime values such as variable values.

 

The first argument and database version is often enough to identify a known problem. But as functions get invoked in many different ways, Oracle Support may also need to use the error stack trace for more specific analysis. This shows the code path taken leading up to the error.

 

 

 

1) Detecting ORA-00600 Errors

The first step is to detect when an ORA-00600 error occurs. For this you should make sure the latest Oracle Trace File Analyzer (TFA) is installed and email notifications are enabled.

 

You may already have TFA installed. It’s included with Oracle Grid Infrastructure 11.2.0.4+ and 12.1.0.2+ and with the 12.2 Database. For more details on TFA, take a look at the TFA Overview Guide.

 

Oracle releases new versions of TFA every 3 months. New versions include bug fixes and enhancements to the data collected.  So you should upgrade to the latest if possible.

 

For the purposes of ORA-00600 error detection, the earliest version of TFA you should have is 18.3.0.

 

TFA installation

Perform the TFA installation as the root user.

 

Installation will:

Install on all nodes if clustered

Auto discover relevant Oracle Software & Exadata Storage Servers

Start monitoring all discovered items for significant events

Automatically collect all required diagnostics when significant errors occur and send you email notification

 

To install TFA follow these steps:

 

1) Download latest version from My Oracle Support Document 1513912.1, transfer to required server & unzip

Execute the installer (if the database is clustered the installer takes care of the remote nodes as well):

 

./installTFA<platform>

 

2) Configure the email(s) to use for notification when TFA detects significant errors

To set the notification email to use for a specific ORACLE_HOME, include the operating system owner in the command:

 

tfactl set notificationAddress=os_user:email

 

For example:

 

tfactl set notificationAddress=oracle:some.body@example.com

 

To set the notification email to use for any ORACLE_HOME:

 

tfactl set notificationAddress=email

 

For example:

 

tfactl set notificationAddress=another.body@example.com

 

 

3) Configure the SMTP server for TFA to use when sending email notifications.

Set the SMTP parameters when prompted:

 

tfactl set smtp

 

You can verify the smtp configuration using:

 

tfactl sendmail email_address

 

When TFA detects an ORA-00600 error has occurred (or other events), you will get an email notification like this:

ORA-600 Notification.jpg

 

 

 

What to do if you can’t install TFA as root?

If you are unable to install as root it is still useful to use TFA for diagnostic collection.

 

You will not be able to detect errors or collect from remote hosts unless installed as root.

 

1) Download the latest version from My Oracle Support Document 1513912.1, transfer to each required server & unzip:

2) Execute the installer:

 

./installTFA<platform> -extractto <dir> -javahome <jre_home>
Use the JRE 1.8 or higher, already available in the RDBMS home.

 

 

If you install as a non-root user, TFA will not be able to automatically detect ORA-00600, so to collect diagnostic packages you will need to follow the steps in the next section, where you will perform an on-demand diagnostic collection.

 

 

What to do if the ORA-00600 error occurred before you had TFA installed

If the ORA-00600 error occurred before you had TFA 18.3.0+ installed you can still use TFA to collect all necessary diagnostics.

 

As the oracle user or equivalent run:

 

tfactl diagcollect –srdc ORA-00600

 

 

This will prompt for time of the event and database, then show you all corresponding errors found and let you collect diagnostics for the appropriate one.

 

Then it will collect all diagnostics and package them in a single zip file, ready to move onto the next step.

 

For example:

bash-4.1$ /scratch/app/11.2.0.4/grid/bin/tfactl diagcollect -srdc ORA-00600

Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] : 2018-09-13

Enter the Database Name [<RETURN>=ALL] :

No events matching the timestamp Sep/12/2018 18:00:00-Sep/13/2018 06:00:00.

The timestamp must be between Sep/13/2018 07:08:40 and Sep/13/2018 10:17:18.

  1. 1. Sep/13/2018 10:17:18 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], []
  2. 2. Sep/13/2018 09:41:24 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], []
  3. 3. Sep/13/2018 09:31:29 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], []
  4. 4. Sep/13/2018 08:38:43 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], []
  5. 5. Sep/13/2018 08:09:49 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], []

 

Please choose the event : 1-5 [1] 1

Selected value is : ( 1. Sep/13/2018 10:17:18 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], [] )

Scripts to be run by this srdc: ipspack rdahcve1210 rdahcve1120 rdahcve1110

Components included in this srdc: OS CRS DATABASE NOCHMOS

Collecting data for local node(s)

Scanning files from Sep/13/2018 04:17:18 to Sep/13/2018 16:17:18

Collection Id : 20180914032048myserver69

Detailed Logging at : /scratch/tfa_repository/srdc_ora600_collection_Fri_Sep_14_03_20_48_PDT_2018_node_local/diagcollect_20180914032048_myserver69.log

2018/09/14 03:20:53 PDT : NOTE : Any file or directory name containing the string .com will be renamed to replace .com with dotcom

2018/09/14 03:20:53 PDT : Collection Name : tfa_srdc_ora600_Fri_Sep_14_03_20_48_PDT_2018.zip

2018/09/14 03:20:54 PDT : Scanning of files for Collection in progress...

2018/09/14 03:20:54 PDT : Collecting additional diagnostic information...

2018/09/14 03:21:04 PDT : Getting list of files satisfying time range [09/13/2018 04:17:18 PDT, 09/13/2018 16:17:18 PDT]

2018/09/14 03:21:21 PDT : Collecting ADR incident files...

2018/09/14 03:22:27 PDT : Completed collection of additional diagnostic information...

2018/09/14 03:22:31 PDT : Completed Local Collection

.--------------------------------------.

|          Collection Summary          |

+------------+-----------+------+------+

| Host       | Status    | Size | Time |

+------------+-----------+------+------+

| myserver69 | Completed | 20MB |  97s |

'------------+-----------+------+------'

Logs are being collected to: /scratch/tfa_repository/srdc_ora600_collection_Fri_Sep_14_03_20_48_PDT_2018_node_local

/scratch/tfa_repository/srdc_ora600_collection_Fri_Sep_14_03_20_48_PDT_2018_node_local/myserver69.tfa_srdc_ora600_Fri_Sep_14_03_20_48_PDT_2018.zip

 

 

2. Compare diagnostics to known problems

ORA-00600 diagnosis can be done automatically using the My Oracle Support ORA-00600 Troubleshooting Tool.

 

The ORA-00600 Troubleshooting Tool performs the same actions as an older tool you may have heard of the “ORA-600 lookup tool”, but it does it automatically and can also be used to log SRs.

 

 

On the first screen choose to “Troubleshoot a new issue” and then click next

ORA-600 Troubleshooting Tool Step 1-highlight.jpg

 

Now choose the zip file TFA created, choose upload and then next

ORA-600 Troubleshooting Tool Step 2-highlight.jpg

 

The ORA-00600 Troubleshooting tool will now analyze the contents of the diagnostics in the TFA collected zip file and compare it to known problems.

Follow the instructions on these to understand more about the problem and how to fix it.

ORA-600 Troubleshooting Tool Step 3.jpg

 

 

3. Create a new SR

If the troubleshooting tool is unable to find a known problem, or you need more help then scroll down and choose Create SR

ORA-600 Troubleshooting Tool Step 4-highlight.jpg

 

On the SR creation screen, complete all fields and choose Create SR

ORA-600 Troubleshooting Tool Step 5.jpg

 

This will create a new SR and automatically attach the TFA diagnostic collection you previously provided.

Oracle Support will now complete further analysis on the diagnostics and work with you to help resolve the problem.

 

 

Summary

In summary

ORA-00600 is a generic error code. The Oracle Database uses ORA-00600 when it encounters an unexpected internal condition.

Use Oracle Trace File Analyzer (TFA) to automatically detect ORA-00600 errors.

Alternatively TFA can be used on-demand after an ORA-00600 to collect and package all required diagnostics.

Use the My Oracle Support ORA-00600 Troubleshooting Tool to compare the TFA collected diagnostics to known problems

If you need further help continue with the Troubleshooting Tool to log an SR.

This will automatically upload the TFA diagnostic collection, providing Oracle Support everything they need to help you resolve the problem.

 

For more information and to download Oracle Trace File Analyzer see Document 1513912.1.