1 Reply Latest reply: Apr 21, 2014 7:48 PM by User12869221-Oracle RSS

    BAD TRAP: type=31 rp=2a1038db1d0 addr=2a76a463fc4 mmu_fsr=0

    1061610

      Hi Team, My server is rebooting again and again with below Panic errors i didnt get any solution for that please help Apr 20 05:15:14 bovs1a unix: [ID 836849 kern.notice] Apr 20 05:15:14 bovs1a ^Mpanic[cpu1]/thread=3000d29fa60: Apr 20 05:15:14 bovs1a unix: [ID 799565 kern.notice] BAD TRAP: type=31 rp=2a103aa11d0 addr=2a759bbbfc4 mmu_fsr=0 Apr 20 05:15:15 bovs1a unix: [ID 100000 kern.notice] Apr 20 05:15:15 bovs1a unix: [ID 839527 kern.notice] oracle: Apr 20 05:15:15 bovs1a unix: [ID 520581 kern.notice] trap type = 0x31 Apr 20 05:15:15 bovs1a unix: [ID 381800 kern.notice] addr=0x2a759bbbfc4 Apr 20 05:15:15 bovs1a unix: [ID 101969 kern.notice] pid=5340, pc=0x12d6378, sp=0x2a103aa0a71, tstate=0x4480001601, context=0x1889 Apr 20 05:15:15 bovs1a unix: [ID 743441 kern.notice] g1-g7: 1fc0, ffffffff00001fff, ffffe000, 60030b62000, 60012134100, 1889, 3000d29fa60 Apr 20 05:15:15 bovs1a unix: [ID 100000 kern.notice] Apr 20 05:15:15 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa0ef0 unix:die+9c (31, 2a103aa11d0, 2a759bbbfc4, 0, 2a103aa0fb0, f4162004) Apr 20 05:15:15 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000001fff 0000000000000031 0000000001000000 0000000000002000 Apr 20 05:15:15 bovs1a  %l4-7: 0000000000100000 000006001ac61a38 0000000000000000 00000000010a5c00 Apr 20 05:15:15 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa0fd0 unix:trap+9e0 (2a103aa11d0, 0, 1fff, 503, 2a759bba000, 1) Apr 20 05:15:15 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000000 000006001ac61a38 0000000000000031 0000000000001c00 Apr 20 05:15:15 bovs1a  %l4-7: 0000000000000000 0000000000000001 ffffffffffffe000 0000000000000003 Apr 20 05:15:15 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1120 unix:ktl0+48 (6002c2a38c0, 2a759bac000, ffc0, 3000ef50af8, 69, 1) Apr 20 05:15:15 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000002 0000000000001400 0000004480001601 000000000101c23c Apr 20 05:15:15 bovs1a  %l4-7: 0000060030b62000 00000000000667f9 0000000000000000 000002a103aa11d0 Apr 20 05:15:15 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1270 ufs:ufs_dirlook+870 (2a759bbbfc0, 2a103aa1680, 3000db3d880, 36, ffc0, ffc0) Apr 20 05:15:16 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000000 0000000080000000 0000000000000001 000003000eb9c940 Apr 20 05:15:16 bovs1a  %l4-7: 0000000000000000 0000000000000001 0000000000000000 000000000000000c Apr 20 05:15:16 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa13b0 ufs:ufs_lookup+290 (3000eb9c940, 2a103aa1680, 2a103aa1678, 0, 0, 0) Apr 20 05:15:16 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000038 000006002b4bb290 0000000000000056 000003000db3d880 Apr 20 05:15:16 bovs1a  %l4-7: 0000000080000000 0000000000000002 0000000000003006 0000000000000000 Apr 20 05:15:16 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1490 genunix:fop_lookup+28 (3000eb9c940, 2a103aa1680, 2a103aa1678, 12e6710, 0, 3000163e000) Apr 20 05:15:16 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000000 000006002fae5540 0000000000000004 0000000000000003 Apr 20 05:15:16 bovs1a  %l4-7: 0000000006ecbf59 0000000006ecbf58 000006002f3291b8 000003000004cb80 Apr 20 05:15:16 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1550 genunix:lookuppnvp+354 (2a103aa1940, 0, 0, 2a103aa1678, 2a103aa1680, 600104b9a40) Apr 20 05:15:16 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 00000000018b14a0 000003000eb9c940 0000000000000001 0000000000000000 Apr 20 05:15:16 bovs1a  %l4-7: 000006002b4bb290 00000600104b9a40 0000000000000000 0000000000000002 Apr 20 05:15:17 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1790 genunix:lookuppnat+120 (600104b9a40, 0, 1, 0, 2a103aa1ad8, 0) Apr 20 05:15:17 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000095 000000000000002f 00000600104b9a40 0000000000000094 Apr 20 05:15:17 bovs1a  %l4-7: 000000000000005d 0000000000000076 000006002b4bb290 000002a103aa1940 Apr 20 05:15:17 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1850 genunix:lookupnameat+a8 (0, 0, 1, 0, 2a103aa1ad8, 0) Apr 20 05:15:17 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 000000000000000a 0000000109a2ce98 0000000000000000 00000001086ac000 Apr 20 05:15:17 bovs1a  %l4-7: ffffffff7fff637c 000002a103aa1940 ffffffff7fff7e30 00000000018b1400 Apr 20 05:15:17 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1960 genunix:cstatat_getvp+198 (ffd19400, ffffffff7fff637c, 1, 1, 2a103aa1ad8, 0) Apr 20 05:15:17 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: ffffffffffd19553 0000000000000000 0000000000000000 0000000040000004 Apr 20 05:15:17 bovs1a  %l4-7: 0000000000000000 00000000018b1400 000000001c000000 000006002b4bb290 Apr 20 05:15:17 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1a20 genunix:cstatat+40 (ffffffffffd19553, ffffffff7fff637c, 1000, ffffffff7fff62f0, 0, 10) Apr 20 05:15:17 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000000 000002a103aa1ad0 0000000000000001 0000000000000001 Apr 20 05:15:17 bovs1a  %l4-7: 0000000000000000 000002a103aa1ad8 0000000000000000 0000000000000000 Apr 20 05:15:18 bovs1a unix: [ID 100000 kern.notice] Apr 20 05:15:18 bovs1a genunix: [ID 672855 kern.notice] syncing file systems... Apr 20 05:15:18 bovs1a genunix: [ID 733762 kern.notice]  36 Apr 20 05:15:20 bovs1a genunix: [ID 733762 kern.notice]  23 Apr 20 05:15:22 bovs1a genunix: [ID 733762 kern.notice]  1 Apr 20 05:15:50 bovs1a last message repeated 20 times Apr 20 05:15:51 bovs1a genunix: [ID 622722 kern.notice]  done (not all i/o completed) Apr 20 05:15:52 bovs1a genunix: [ID 111219 kern.notice] dumping to /dev/md/dsk/d31, offset 859701248, content: kernel Apr 20 05:16:31 bovs1a genunix: [ID 100000 kern.notice] Apr 20 05:16:31 bovs1a genunix: [ID 665016 kern.notice] ^M100% done: 125520 pages dumped, Apr 20 05:16:31 bovs1a genunix: [ID 851671 kern.notice] dump succeeded Apr 20 05:18:23 bovs1a genunix: [ID 540533 kern.notice] ^MSunOS Release 5.10 Version Generic_147440-12 64-bit Apr 20 05:18:23 bovs1a genunix: [ID 459285 kern.notice] Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved. Apr 20 05:18:23 bovs1a genunix: [ID 678236 kern.info] Ethernet address = 0:14:4f:37:eb:a5 Apr 20 05:18:23 bovs1a unix: [ID 673563 kern.info] NOTICE: Kernel Cage is ENABLED Apr 20 05:18:23 bovs1a unix: [ID 389951 kern.info] mem = 8388608K (0x200000000) Apr 20 05:18:23 bovs1a unix: [ID 930857 kern.info] avail mem = 8362508288 Apr 20 05:18:23 bovs1a rootnex: [ID 466748 kern.info] root nexus = Sun Fire V440 Apr 20 05:18:23 bovs1a rootnex: [ID 349649 kern.info] pseudo0 at root Apr 20 05:18:23 bovs1a genunix: [ID 936769 kern.info] pseudo0 is /pseudo Apr 20 05:18:23 bovs1a rootnex: [ID 349649 kern.info] scsi_vhci0 at root Apr 20 05:18:23 bovs1a genunix: [ID 936769 kern.info] scsi_vhci0 is /scsi_vhci Apr 20 05:18:23 bovs1a rootnex: [ID 349649 kern.info] pcisch3 at root: SAFARI 0x1f 0x700000 Apr 20 05:18:23 bovs1a genunix: [ID 936769 kern.info] pcisch3 is /pci@1f,700000 Apr 20 05:18:23 bovs1a scsi: [ID 365881 kern.info] /pci@1f,700000/scsi@2 (mpt0): Apr 20 05:18:23 bovs1a  initiator SCSI ID now 7 Apr 20 05:18:23 bovs1a scsi: [ID 365881 kern.info] /pci@1f,700000/scsi@2 (mpt0): Apr 20 05:18:23 bovs1a  Rev. 7 LSI, Inc. 1030 found. Apr 20 05:18:23 bovs1a scsi: [ID 365881 kern.info] /pci@1f,700000/scsi@2 (mpt0): Apr 20 05:18:23 bovs1a  mpt0 supports power management. Apr 20 05:18:32 bovs1a scsi: [ID 365881 kern.info] /pci@1f,700000/scsi@2 (mpt0): Apr 20 05:18:32 bovs1a  mpt0 Firmware version v1.3.30.22 (IM/IME) Apr 20 05:18:32 bovs1a scsi: [ID 365881 kern.info] /pci@1f,700000/scsi@2 (mpt0): Apr 20 05:18:32 bovs1a  mpt0: IOC Operational. Apr 20 05:18:50 bovs1a pcisch: [ID 370704 kern.info] PCI-device: scsi@2, mpt0 Apr 20 05:18:50 bovs1a genunix: [ID 936769 kern.info] mpt0 is /pci@1f,700000/scsi@2 Apr 20 05:18:50 bovs1a scsi: [ID 583861 kern.info] sd1 at mpt0: unit-address 0,0: target 0 lun 0 Apr 20 05:18:50 bovs1a genunix: [ID 936769 kern.info] sd1 is /pci@1f,700000/scsi@2/sd@0,0 Apr 20 05:18:50 bovs1a genunix: [ID 408114 kern.info] /pci@1f,700000/scsi@2/sd@0,0 (sd1) online Apr 20 05:18:55 bovs1a swapgeneric: [ID 308332 kern.info] root on /pci@1f,700000/scsi@2/disk@0,0:a fstype ufs

        • 1. Re: BAD TRAP: type=31 rp=2a1038db1d0 addr=2a76a463fc4 mmu_fsr=0
          User12869221-Oracle

          Please format the error messages properly next time so that it is more readable.

           

          Apr 20 05:15:14 bovs1a unix: [ID 836849 kern.notice]

          Apr 20 05:15:14 bovs1a ^Mpanic[cpu1]/thread=3000d29fa60:

          Apr 20 05:15:14 bovs1a unix: [ID 799565 kern.notice] BAD TRAP: type=31 rp=2a103aa11d0 addr=2a759bbbfc4 mmu_fsr=0

          Apr 20 05:15:15 bovs1a unix: [ID 100000 kern.notice] Apr 20 05:15:15 bovs1a unix: [ID 839527 kern.notice] oracle:

          Apr 20 05:15:15 bovs1a unix: [ID 520581 kern.notice] trap type = 0x31

          Apr 20 05:15:15 bovs1a unix: [ID 381800 kern.notice] addr=0x2a759bbbfc4

          Apr 20 05:15:15 bovs1a unix: [ID 101969 kern.notice] pid=5340, pc=0x12d6378, sp=0x2a103aa0a71, tstate=0x4480001601, context=0x1889

          Apr 20 05:15:15 bovs1a unix: [ID 743441 kern.notice] g1-g7: 1fc0, ffffffff00001fff, ffffe000, 60030b62000, 60012134100, 1889, 3000d29fa60

          Apr 20 05:15:15 bovs1a unix: [ID 100000 kern.notice]

          Apr 20 05:15:15 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa0ef0 unix:die+9c (31, 2a103aa11d0, 2a759bbbfc4, 0, 2a103aa0fb0, f4162004)

          Apr 20 05:15:15 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000001fff 0000000000000031 0000000001000000 0000000000002000

          Apr 20 05:15:15 bovs1a  %l4-7: 0000000000100000 000006001ac61a38 0000000000000000 00000000010a5c00

          Apr 20 05:15:15 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa0fd0 unix:trap+9e0 (2a103aa11d0, 0, 1fff, 503, 2a759bba000, 1)

          Apr 20 05:15:15 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000000 000006001ac61a38 0000000000000031 0000000000001c00

          Apr 20 05:15:15 bovs1a  %l4-7: 0000000000000000 0000000000000001 ffffffffffffe000 0000000000000003

          Apr 20 05:15:15 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1120 unix:ktl0+48 (6002c2a38c0, 2a759bac000, ffc0, 3000ef50af8, 69, 1)

          Apr 20 05:15:15 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000002 0000000000001400 0000004480001601 000000000101c23c

          Apr 20 05:15:15 bovs1a  %l4-7: 0000060030b62000 00000000000667f9 0000000000000000 000002a103aa11d0

          Apr 20 05:15:15 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1270 ufs:ufs_dirlook+870 (2a759bbbfc0, 2a103aa1680, 3000db3d880, 36, ffc0, ffc0)

          Apr 20 05:15:16 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000000 0000000080000000 0000000000000001 000003000eb9c940

          Apr 20 05:15:16 bovs1a  %l4-7: 0000000000000000 0000000000000001 0000000000000000 000000000000000c

          Apr 20 05:15:16 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa13b0 ufs:ufs_lookup+290 (3000eb9c940, 2a103aa1680, 2a103aa1678, 0, 0, 0)

          Apr 20 05:15:16 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000038 000006002b4bb290 0000000000000056 000003000db3d880

          Apr 20 05:15:16 bovs1a  %l4-7: 0000000080000000 0000000000000002 0000000000003006 0000000000000000

          Apr 20 05:15:16 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1490 genunix:fop_lookup+28 (3000eb9c940, 2a103aa1680, 2a103aa1678, 12e6710, 0, 3000163e000)

          Apr 20 05:15:16 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000000 000006002fae5540 0000000000000004 0000000000000003

          Apr 20 05:15:16 bovs1a  %l4-7: 0000000006ecbf59 0000000006ecbf58 000006002f3291b8 000003000004cb80

          Apr 20 05:15:16 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1550 genunix:lookuppnvp+354 (2a103aa1940, 0, 0, 2a103aa1678, 2a103aa1680, 600104b9a40)

          Apr 20 05:15:16 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 00000000018b14a0 000003000eb9c940 0000000000000001 0000000000000000

          Apr 20 05:15:16 bovs1a  %l4-7: 000006002b4bb290 00000600104b9a40 0000000000000000 0000000000000002

          Apr 20 05:15:17 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1790 genunix:lookuppnat+120 (600104b9a40, 0, 1, 0, 2a103aa1ad8, 0)

          Apr 20 05:15:17 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000095 000000000000002f 00000600104b9a40 0000000000000094

          Apr 20 05:15:17 bovs1a  %l4-7: 000000000000005d 0000000000000076 000006002b4bb290 000002a103aa1940

          Apr 20 05:15:17 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1850 genunix:lookupnameat+a8 (0, 0, 1, 0, 2a103aa1ad8, 0)

          Apr 20 05:15:17 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 000000000000000a 0000000109a2ce98 0000000000000000 00000001086ac000

          Apr 20 05:15:17 bovs1a  %l4-7: ffffffff7fff637c 000002a103aa1940 ffffffff7fff7e30 00000000018b1400

          Apr 20 05:15:17 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1960 genunix:cstatat_getvp+198 (ffd19400, ffffffff7fff637c, 1, 1, 2a103aa1ad8, 0)

          Apr 20 05:15:17 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: ffffffffffd19553 0000000000000000 0000000000000000 0000000040000004

          Apr 20 05:15:17 bovs1a  %l4-7: 0000000000000000 00000000018b1400 000000001c000000 000006002b4bb290

          Apr 20 05:15:17 bovs1a genunix: [ID 723222 kern.notice] 000002a103aa1a20 genunix:cstatat+40 (ffffffffffd19553, ffffffff7fff637c, 1000, ffffffff7fff62f0, 0, 10)

          Apr 20 05:15:17 bovs1a genunix: [ID 179002 kern.notice]  %l0-3: 0000000000000000 000002a103aa1ad0 0000000000000001 0000000000000001

          Apr 20 05:15:17 bovs1a  %l4-7: 0000000000000000 000002a103aa1ad8 0000000000000000 0000000000000000

          Apr 20 05:15:18 bovs1a unix: [ID 100000 kern.notice]

          Apr 20 05:15:18 bovs1a genunix: [ID 672855 kern.notice] syncing file systems... Apr 20 05:15:18 bovs1a genunix: [ID 733762 kern.notice]  36

          Apr 20 05:15:20 bovs1a genunix: [ID 733762 kern.notice]  23

          Apr 20 05:15:22 bovs1a genunix: [ID 733762 kern.notice]  1 Apr 20 05:15:50 bovs1a last message repeated 20 times

          Apr 20 05:15:51 bovs1a genunix: [ID 622722 kern.notice]  done (not all i/o completed)

          Apr 20 05:15:52 bovs1a genunix: [ID 111219 kern.notice] dumping to /dev/md/dsk/d31, offset 859701248, content: kernel

          Apr 20 05:16:31 bovs1a genunix: [ID 100000 kern.notice] Apr 20 05:16:31 bovs1a genunix: [ID 665016 kern.notice] ^M100% done: 125520 pages dumped,

          Apr 20 05:16:31 bovs1a genunix: [ID 851671 kern.notice] dump succeeded

           

           

          Apr 20 05:18:23 bovs1a genunix: [ID 540533 kern.notice] ^MSunOS Release 5.10 Version Generic_147440-12 64-bit

          Apr 20 05:18:23 bovs1a genunix: [ID 459285 kern.notice] Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved.

          Apr 20 05:18:23 bovs1a genunix: [ID 678236 kern.info] Ethernet address = 0:14:4f:37:eb:a5 Apr 20 05:18:23 bovs1a unix: [ID 673563 kern.info] NOTICE: Kernel Cage is ENABLED

          Apr 20 05:18:23 bovs1a unix: [ID 389951 kern.info] mem = 8388608K (0x200000000)

          Apr 20 05:18:23 bovs1a unix: [ID 930857 kern.info] avail mem = 8362508288

          Apr 20 05:18:23 bovs1a rootnex: [ID 466748 kern.info] root nexus = Sun Fire V440

           

           

          This look like a panic in UFS ufs_dirlook() in accessing fbuf structure.

          There is a recent bug that was closed as not reproduceable whicvh is similar to this panic.

           

          Bug 16818353 : PANIC IN UFS_DIRLOOK(): BAD TRAP 0X31 WRONG VALUE IN FBUF->FB_ADDR  

           

          I will recommend that you open a new Service Request with Oracle Support and provide explorer result along with

          the crash dump to  Oracle Support for further investigation. Thanks.