Step by Step Instructions for Detection, Diagnosis & Resolution of ORA-600 (ORA-00600 Internal error)
As a DBA it is not uncommon to find ORA-00600 errors in your alert log.
Because there is no other context to the ORA-600 it can be difficult to know if it's something you need to act on.
This post guides you through what the error means, how to use Oracle Trace File Analyzer (TFA) to detect it, how to diagnose the cause and then how to resolve.
Here's what's covered below:
This is the overall process for detection, diagnosis & resolution:
What is an ORA-600 (ORA-00600 internal error)?
ORA-00600 internal error, also referred to as ORA-600 is a generic error code. The Oracle Database uses ORA-00600 when it encounters an unexpected internal condition.
The error will look like this:
Fri Mar 13 08:09:49 2020
Errors in file /scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/trace/ogg112041_ora_8881.trc (incident=99577):
ORA-00600: internal error code, arguments: [ktfbtgex-7], , , , , , , , , , , 
Incident details in: /scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/incident/incdir_99577/ogg112041_ora_8881_i99577.trc
The arguments in square brackets will contain different values. These depend on where in the code the problem occurred and what it was doing.
The value of the first argument shows where in the code the error occurred. Later arguments contain other runtime values such as variable values.
The first argument and database version is often enough to identify a known problem. But as functions get invoked in many different ways, Oracle Support may also need to use the error stack trace for more specific analysis. This shows the code path taken leading up to the error.
1) Detecting ORA-00600 Errors
The first step is to detect when an ORA-00600 error occurs. For this you should make sure the latest Oracle Trace File Analyzer (TFA) is installed and email notifications are enabled.
You may already have TFA installed. It’s included with Oracle Grid Infrastructure 220.127.116.11+ and 18.104.22.168+ and with the 12.2 Database. For more details on TFA, take a look at the TFA Overview Guide.
Oracle releases new versions of TFA every 3 months. New versions include bug fixes and enhancements to the data collected. So you should upgrade to the latest if possible.
For the purposes of ORA-00600 error detection, the earliest version of TFA you should have is 18.3.0.
Perform the TFA installation as the root user.
Install on all nodes if clustered
Auto discover relevant Oracle Software & Exadata Storage Servers
Start monitoring all discovered items for significant events
Automatically collect all required diagnostics when significant errors occur and send you email notification
To install TFA follow these steps:
1) Download latest version from My Oracle Support Document 2550798.1, transfer to required server & unzip
Execute the installer (if the database is clustered the installer takes care of the remote nodes as well):
2) Configure the email(s) to use for notification when TFA detects significant errors
The installation will prompt you: "Do you want to add AHF Notification Email IDs ?". Provide your email or a series of comma separated notification email addresses here.
Alternatively you can set the notification after installation for a specific ORACLE_HOME, include the operating system owner in the command:
tfactl set notificationAddress=os_user:email
tfactl set notificationAddress=oracle:email@example.com
To set the notification email to use for any ORACLE_HOME:
tfactl set notificationAddress=email
tfactl set notificationAddressfirstname.lastname@example.org
3) Configure the SMTP server for TFA to use when sending email notifications.
Set the SMTP parameters when prompted:
tfactl set smtp
You can verify the smtp configuration using:
tfactl sendmail email_address
When TFA detects an ORA-00600 error has occurred (or other events), you will get an email notification like this:
What to do if you can’t install TFA as root?
If you are unable to install as root, then you can install as the Oracle home owner.
Oracle Autonomous Health Framework has reduced capabilities when you install it as the non-root user in non-daemon mode. Therefore, you cannot complete the following tasks:
- Automate diagnostic collections
- Collect diagnostics from remote hosts
- Collect files that are not readable by the Oracle home owner, for example, /var/log/messages, or certain Oracle Grid Infrastructure logs
1) Download the latest version from My Oracle Support Document 2550798.1, transfer to each required server & unzip:
2) Execute the installer:
./ahf_setup -ahf_loc install_dir
What to do if the ORA-00600 error occurred before you had TFA installed
If the ORA-00600 error occurred before you had TFA 18.3.0+ installed you can still use TFA to collect all necessary diagnostics.
As the oracle user or equivalent run:
tfactl diagcollect –srdc ORA-00600
This will prompt for time of the event and database, then show you all corresponding errors found and let you collect diagnostics for the appropriate one.
Then it will collect all diagnostics and package them in a single zip file, ready to move onto the next step.
bash-4.1$ /scratch/app/22.214.171.124/grid/bin/tfactl diagcollect -srdc ORA-00600
Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] : 2018-09-13
Enter the Database Name [<RETURN>=ALL] :
No events matching the timestamp Sep/12/2018 18:00:00-Sep/13/2018 06:00:00.
The timestamp must be between Sep/13/2018 07:08:40 and Sep/13/2018 10:17:18.
- 1. Sep/13/2018 10:17:18 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], , , , , , , , , , , 
- 2. Sep/13/2018 09:41:24 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], , , , , , , , , , , 
- 3. Sep/13/2018 09:31:29 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], , , , , , , , , , , 
- 4. Sep/13/2018 08:38:43 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], , , , , , , , , , , 
- 5. Sep/13/2018 08:09:49 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], , , , , , , , , , , 
Please choose the event : 1-5  1
Selected value is : ( 1. Sep/13/2018 10:17:18 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], , , , , , , , , , ,  )
Scripts to be run by this srdc: ipspack rdahcve1210 rdahcve1120 rdahcve1110
Components included in this srdc: OS CRS DATABASE NOCHMOS
Collecting data for local node(s)
Scanning files from Sep/13/2018 04:17:18 to Sep/13/2018 16:17:18
Collection Id : 20180914032048myserver69
Detailed Logging at : /scratch/tfa_repository/srdc_ora600_collection_Fri_Sep_14_03_20_48_PDT_2018_node_local/diagcollect_20180914032048_myserver69.log
2018/09/14 03:20:53 PDT : NOTE : Any file or directory name containing the string .com will be renamed to replace .com with dotcom
2018/09/14 03:20:53 PDT : Collection Name : tfa_srdc_ora600_Fri_Sep_14_03_20_48_PDT_2018.zip
2018/09/14 03:20:54 PDT : Scanning of files for Collection in progress...
2018/09/14 03:20:54 PDT : Collecting additional diagnostic information...
2018/09/14 03:21:04 PDT : Getting list of files satisfying time range [09/13/2018 04:17:18 PDT, 09/13/2018 16:17:18 PDT]
2018/09/14 03:21:21 PDT : Collecting ADR incident files...
2018/09/14 03:22:27 PDT : Completed collection of additional diagnostic information...
2018/09/14 03:22:31 PDT : Completed Local Collection
| Collection Summary |
| Host | Status | Size | Time |
| myserver69 | Completed | 20MB | 97s |
Logs are being collected to: /scratch/tfa_repository/srdc_ora600_collection_Fri_Sep_14_03_20_48_PDT_2018_node_local
2. Compare diagnostics to known problems
ORA-00600 diagnosis can be done automatically using the My Oracle Support ORA-00600 Troubleshooting Tool.
On the first screen choose to “Troubleshoot a new issue” and then click next
Now choose the zip file TFA created, choose upload and then next
The ORA-00600 Troubleshooting tool will now analyze the contents of the diagnostics in the TFA collected zip file and compare it to known problems.
Follow the instructions on these to understand more about the problem and how to fix it.
3. Create a new SR
If the troubleshooting tool is unable to find a known problem, or you need more help then scroll down and choose Create SR
On the SR creation screen, complete all fields and choose Create SR
This will create a new SR and automatically attach the TFA diagnostic collection you previously provided.
Oracle Support will now complete further analysis on the diagnostics and work with you to help resolve the problem.
ORA-00600 is a generic error code. The Oracle Database uses ORA-00600 when it encounters an unexpected internal condition.
Use Oracle Trace File Analyzer (TFA) to automatically detect ORA-00600 errors.
Alternatively TFA can be used on-demand after an ORA-00600 to collect and package all required diagnostics.
Use the My Oracle Support ORA-00600 Troubleshooting Tool to compare the TFA collected diagnostics to known problems
If you need further help continue with the Troubleshooting Tool to log an SR.
This will automatically upload the TFA diagnostic collection, providing Oracle Support everything they need to help you resolve the problem.
For more information and to download Oracle Trace File Analyzer see Document 2550798.1.