This discussion is archived
2 Replies Latest reply: Apr 5, 2013 12:14 PM by mbobak RSS

Kernel panic on Oracle Linux 6.3 with kernel 2.6.39-200.24.1.el6uek.x86_64

mbobak Oracle ACE
Currently Being Moderated
Hi all,

Running OL6.3 with the subject UEK, on Cisco UCS hardware. When doing a fabric failover at the hardware level, we see the following kernel panic:
Apr  2 17:23:15 pqmdb101 kernel: host1: libfc: Link down on port (ce0ea1)
Apr  2 17:23:37 pqmdb101 gpagentd[3519]: Error at lwignome.c:296. Error code [0x       5] (CENTERROR_ACCESS_DENIED)
Apr  2 17:23:37 pqmdb101 gpagentd[3519]: Error at gpauserthread.c:178. Error code [0x       5] (CENTERROR_ACCESS_DENIED)
Apr  2 17:23:45 pqmdb101 kernel: rport-1:0-0: blocked FC remote port time out: removing rport
Apr  2 17:23:45 pqmdb101 kernel: rport-1:0-1: blocked FC remote port time out: removing target and saving binding
Apr  2 17:23:45 pqmdb101 kernel: rport-1:0-2: blocked FC remote port time out: removing target and saving binding
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 11 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 37 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 29 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 23 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 13 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 2 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 12 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 6 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 5 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 3 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 4 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 1 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 26 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 40 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 19 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 36 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 25 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 39 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 28 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 32 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 30 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 8 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 31 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 38 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 35 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 7 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 15 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 34 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 16 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 9 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 21 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 10 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 18 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 20 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 22 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 41 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 33 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 24 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 27 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 14 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 17 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Bus 1 to Symmetrix 000192604666 port 7eB is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 17 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 29 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 23 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 13 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 5 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 12 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 6 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 2 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 37 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 3 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 4 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 1 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 19 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 26 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 39 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 35 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 22 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 25 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 28 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 27 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 8 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 32 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 31 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 16 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 36 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 11 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 18 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 41 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 38 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 7 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 21 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 14 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 15 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 33 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 9 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 30 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 34 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 20 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 40 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 24 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Path Bus 1 Tgt 1 Lun 10 to 000192604666 is dead.
Apr  2 17:23:45 pqmdb101 kernel: Error:Mpx:Bus 1 to Symmetrix 000192604666 port 5eB is dead.
Apr  2 17:23:49 pqmdb101 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
Apr  2 17:23:49 pqmdb101 kernel: IP: [<ffffffffa027c996>] sd_release+0x26/0xe0 [sd_mod]
Apr  2 17:23:49 pqmdb101 kernel: PGD f2adf5067 PUD 2f59067067 PMD 0
Apr  2 17:23:49 pqmdb101 kernel: Oops: 0000 [#1] SMP
Apr  2 17:23:49 pqmdb101 kernel: CPU 9
Apr  2 17:23:49 pqmdb101 kernel: Modules linked in: oracleasm nfs lockd fscache auth_rpcgss nfs_acl autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf microcode pcspkr ghes hed i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support sg enic ext4 mbcache jbd2 sd_mod crc_t10dif fnic libfcoe libfc scsi_transport_fc scsi_tgt megaraid_sas wmi dm_multipath emcpdm(P)(U) emcpgpx(P)(U) emcpmpx(P)(U) emcp(P)(U) dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Apr  2 17:23:49 pqmdb101 kernel:
Apr  2 17:23:49 pqmdb101 kernel: Pid: 14598, comm: oracle Tainted: P            2.6.39-200.24.1.el6uek.x86_64 #1 Cisco Systems Inc UCSB-B200-M3/UCSB-B200-M3
Apr  2 17:23:49 pqmdb101 kernel: RIP: 0010:[<ffffffffa027c996>]  [<ffffffffa027c996>] sd_release+0x26/0xe0 [sd_mod]
Apr  2 17:23:49 pqmdb101 kernel: RSP: 0018:ffff88106f73dc08  EFLAGS: 00010246
Apr  2 17:23:49 pqmdb101 kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Can anyone offer any insight as to what's going on here? Known bug, by any chance? I tried MOS searching and Google searching, to no avail.

We have a case opened w/ Cisco, and they're supposed to be opening an Oracle SR that can be tied to it, but, I just thought I'd post here, see if anyone had anything useful/constructive/informative to say.

Thanks,

-Mark

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points