This discussion is archived
6 Replies Latest reply: Feb 27, 2013 4:30 AM by 422096 RSS

Irreparable cylinder group header problem.  Program terminated.

422096 Newbie
Currently Being Moderated
Good afternoon,
I am trying to get my machine (SUN Fire T5120) booting again.
This seems not to work due to some bad filesystem (I have been doing some modifications).

In order to be sure that everything works fine, I have decided to run a file-system check, with following results (the machine is booted using a CD-ROM):
# fsck /dev/dsk/c1t0d0s0
** /dev/rdsk/c1t0d0s0
BAD SUPERBLOCK AT BLOCK 16: NUMBER OF DIRECTORIES OUT OF RANGE

LOOK FOR ALTERNATE SUPERBLOCKS WITH MKFS? y


LOOK FOR ALTERNATE SUPERBLOCKS WITH NEWFS? y

SEARCH FOR ALTERNATE SUPERBLOCKS FAILED.

USE GENERIC SUPERBLOCK FROM MKFS? y

CALCULATED GENERIC SUPERBLOCK WITH MKFS
If filesystem was created with manually-specified geometry, using
auto-discovered superblock may result in irrecoverable damage to
filesystem and user data.

CANCEL FILESYSTEM CHECK? n

** Last Mounted on
** Phase 1 - Check Blocks and Sizes
FRAGMENT 12352 OUT OF RANGE I=309 LFN 24
FRAGMENT 12360 OUT OF RANGE I=310 LFN 0
FRAGMENT 12368 OUT OF RANGE I=310 LFN 8
FRAGMENT 12355 OUT OF RANGE I=310 LFN 16
FRAGMENT 12376 OUT OF RANGE I=311 LFN 0
FRAGMENT 12384 OUT OF RANGE I=312 LFN 0
FRAGMENT 12392 OUT OF RANGE I=312 LFN 8
FRAGMENT 12400 OUT OF RANGE I=312 LFN 16
FRAGMENT 12408 OUT OF RANGE I=313 LFN 0
FRAGMENT 12358 OUT OF RANGE I=313 LFN 8
FRAGMENT 12416 OUT OF RANGE I=314 LFN 0
FRAGMENT 12403 OUT OF RANGE I=314 LFN 8
FRAGMENT 12424 OUT OF RANGE I=315 LFN 0
FRAGMENT 12432 OUT OF RANGE I=315 LFN 8
FRAGMENT 12440 OUT OF RANGE I=315 LFN 16
FRAGMENT 12405 OUT OF RANGE I=315 LFN 24
FRAGMENT 12496 OUT OF RANGE I=407 LFN 0
FRAGMENT 12472 OUT OF RANGE I=484 LFN 8
FRAGMENT 12504 OUT OF RANGE I=484 LFN 80
FRAGMENT 12512 OUT OF RANGE I=484 LFN 88
FRAGMENT 5488984 OUT OF RANGE I=657 LFN 104
FRAGMENT 5488992 OUT OF RANGE I=657 LFN 112
FRAGMENT 5489000 OUT OF RANGE I=657 LFN 120
FRAGMENT 5489008 OUT OF RANGE I=657 LFN 128
FRAGMENT 5489016 OUT OF RANGE I=657 LFN 136
FRAGMENT 5489024 OUT OF RANGE I=657 LFN 144
FRAGMENT 5489032 OUT OF RANGE I=657 LFN 152
FRAGMENT 5489040 OUT OF RANGE I=657 LFN 160
FRAGMENT 5489048 OUT OF RANGE I=657 LFN 168
FRAGMENT 5489056 OUT OF RANGE I=657 LFN 176
EXCESSIVE BAD FRAGMENTS I=657
CONTINUE? y

FRAGMENT 33080 OUT OF RANGE I=730 LFN 56
FRAGMENT 45320 OUT OF RANGE I=730 LFN 88
FRAGMENT 7692400 OUT OF RANGE I=730 LFN 3592
FRAGMENT 7692408 OUT OF RANGE I=730 LFN 3600
FRAGMENT 7692416 OUT OF RANGE I=730 LFN 3608
FRAGMENT 7692424 OUT OF RANGE I=730 LFN 3616
FRAGMENT 7692432 OUT OF RANGE I=730 LFN 3624
FRAGMENT 7692440 OUT OF RANGE I=730 LFN 3632
FRAGMENT 7692448 OUT OF RANGE I=730 LFN 3640
FRAGMENT 7692456 OUT OF RANGE I=730 LFN 3648
EXCESSIVE BAD FRAGMENTS I=730
CONTINUE? y

FRAGMENT 20752 OUT OF RANGE I=758 LFN 0
FRAGMENT 20664 OUT OF RANGE I=759 LFN 32
FRAGMENT 20672 OUT OF RANGE I=759 LFN 40
FRAGMENT 20680 OUT OF RANGE I=760 LFN 0
FRAGMENT 20688 OUT OF RANGE I=760 LFN 8
FRAGMENT 20696 OUT OF RANGE I=760 LFN 16
FRAGMENT 20768 OUT OF RANGE I=760 LFN 24
FRAGMENT 20776 OUT OF RANGE I=760 LFN 32
FRAGMENT 20760 OUT OF RANGE I=761 LFN 0
FRAGMENT 5259400 OUT OF RANGE I=1088 LFN 88
FRAGMENT 29016 OUT OF RANGE I=1578 LFN 0
FRAGMENT 29024 OUT OF RANGE I=1578 LFN 8
FRAGMENT 29032 OUT OF RANGE I=1579 LFN 24
FRAGMENT 33024 OUT OF RANGE I=1579 LFN 32
FRAGMENT 33032 OUT OF RANGE I=1579 LFN 40
FRAGMENT 33040 OUT OF RANGE I=1579 LFN 48
FRAGMENT 33048 OUT OF RANGE I=1579 LFN 56
FRAGMENT 33056 OUT OF RANGE I=1579 LFN 64
FRAGMENT 33064 OUT OF RANGE I=1579 LFN 72
FRAGMENT 33072 OUT OF RANGE I=1579 LFN 80
FRAGMENT 45328 OUT OF RANGE I=1585 LFN 8
FRAGMENT 45336 OUT OF RANGE I=1585 LFN 16
FRAGMENT 45344 OUT OF RANGE I=1585 LFN 24
FRAGMENT 45352 OUT OF RANGE I=1585 LFN 32
FRAGMENT 45464 OUT OF RANGE I=1585 LFN 56
FRAGMENT 45472 OUT OF RANGE I=1585 LFN 64
FRAGMENT 7008512 OUT OF RANGE I=1585 LFN 1112
FRAGMENT 7008520 OUT OF RANGE I=1585 LFN 1120
FRAGMENT 7008528 OUT OF RANGE I=1585 LFN 1128
FRAGMENT 7008536 OUT OF RANGE I=1585 LFN 1136
EXCESSIVE BAD FRAGMENTS I=1585
CONTINUE? y

FRAGMENT -16843010 OUT OF RANGE I=2939 LFN 0
NEGATIVE SIZE 4528613516951552 I=2963
CLEAR? y

CG 6: BAD CG MAGIC NUMBER (0x0 should be 0x90255)
WRONG CG NUMBER (0 should be 6)
IMPOSSIBLE NUMBER OF CYLINDERS IN GROUP (0 is less than 1)
INCORRECT NUMBER OF INODES IN GROUP (0 should be 1920)
INCORRECT NUMBER OF DATA BLOCKS IN GROUP (5 should be 4096)
IMPOSSIBLE FRAGMENT ALLOCATION ROTOR POSITION (-891582800 should be at least 0 and less than 4096)
IMPOSSIBLE INODE ALLOCATION ROTOR POSITION (-872058600 should be at least 0 and less than 1920)
INCORRECT BLOCK TOTALS OFFSET (1476397312 should be 168)
BAD FREE BLOCK POSITIONS TABLE OFFSET (5529600 should 232)
INCORRECT USED INODE MAP OFFSET (218103892 should be 488)
INCORRECT FREE FRAGMENT MAP OFFSET (1610617416 should be 728)
END OF HEADER POSITION INCORRECT (1297350722 should be 1240)


UPDATE STANDARD SUPERBLOCK? y

Irreparable cylinder group header problem.  Program terminated.
Do you have any idea what I can do? (it's a machine that needs to be reconfigured completely, can I continue without re-installing Solaris again?)
  • 1. Re: Irreparable cylinder group header problem.  Program terminated.
    cindys Pro
    Currently Being Moderated
    Can you provide format (format-->disk-->partition-->print) output for this disk?


    Thanks, Cindy
  • 2. Re: Irreparable cylinder group header problem.  Program terminated.
    422096 Newbie
    Currently Being Moderated
    Of course, here it is (I have removed the parts that don't contain interesting information):
    # format
    ...
    AVAILABLE DISK SELECTIONS:
           0. c1t0d0 <LSILOGIC-LogicalVolume-3000 cyl 65533 alt 2 hd 16 sec 273>
              /pci@0/pci@0/pci@2/scsi@0/sd@0,0
           1. c1t2d0 <LSILOGIC-LogicalVolume-3000 cyl 65533 alt 2 hd 16 sec 273>
              /pci@0/pci@0/pci@2/scsi@0/sd@2,0
    Specify disk (enter its number): 0
    selecting c1t0d0
    [disk formatted]
    Error occurred with device in use checking: Permission denied
    
    format> p
    
    partition> p
    
    Current partition table (original):
    Total disk cylinders available: 65533 + 2 (reserved cylinders)
    
    Part      Tag    Flag     Cylinders         Size            Blocks
      0       root    wm       0 -  4688        9.77GB    (4689/0/0)   20481552
      1        var    wm    4689 -  6564        3.91GB    (1876/0/0)    8194368
      2     backup    wm       0 - 65532      136.49GB    (65533/0/0) 286248144
      3       swap    wm    6565 - 11564       10.41GB    (5000/0/0)   21840000
      4 unassigned    wm       0                0         (0/0/0)             0
      5 unassigned    wm       0                0         (0/0/0)             0
      6       home    wm   11565 - 65521      112.38GB    (53957/0/0) 235684176
      7 unassigned    wm   65522 - 65532       23.46MB    (11/0/0)        48048
    By the way, thanks for the fast reply.
  • 3. Re: Irreparable cylinder group header problem.  Program terminated.
    cindys Pro
    Currently Being Moderated
    Well, that's a good sign.

    I haven't worked with UFS for a long time but a quick search says these messages are either related to a disk going bad or corrupted data. I wonder if your modifications trampled some important info.

    Were there any error messages in /var/adm/messages that lead up to this problem or iostat -En errors?

    Either way, I don't see how this is recoverable.

    You might try mounting the file system read-only to see if it is accessible read-only.

    Thanks, Cindy
  • 4. Re: Irreparable cylinder group header problem.  Program terminated.
    422096 Newbie
    Currently Being Moderated
    The partitioning looks fine indeed, but there seems to be a problem, which is in my opinion indeed a problem of misconfiguration (no hardware issues).

    I can tell you the history of this issue:
    A collegue had configured a stripe over two disks, and the metadb was included in that stripe (on slice 6).
    As I needed to create space for some "swap" space, I have modified the partitioning of the disks, by re-creating a metadb on slice 7, hence:
    6       home    wm   11565 - 65521      112.38GB    (53957/0/0) 235684176
    7 unassigned    wm   65522 - 65532       23.46MB    (11/0/0)        48048
    However it seems that I have messed up some things while labeling the partitions or while saving the disk configurations (there were quite some errors, like stale databases on "metastat").

    After some time I had the impression that there were no problems anymore, and a collegue has rebooted the machine, which failed, and from that moment on, nothing is working anymore (in fact now the machine is booted using a CD-ROM).

    As far as your questions are concerned: /var/adm/messages, that file does not even exist: most probably it has been erased during the file-system check (I have answered "Yes" on all questions, including the ones that implied clearing information).
    And for the "iostat" results, that doesn't look too good:
    # iostat -En
    c1t0d0           Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
    Vendor: LSILOGIC Product: Logical Volume   Revision: 3000 Serial No:
    Size: 146.56GB <146561286144 bytes>
    Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
    Illegal Request: 2 Predictive Failure Analysis: 0
    c0t0d0           Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
    Vendor: TSSTcorp Product: CD/DVDW TS-T632A Revision: SR03 Serial No:
    Size: 0.67GB <671088640 bytes>
    Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
    Illegal Request: 5 Predictive Failure Analysis: 0
    c1t2d0           Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
    Vendor: LSILOGIC Product: Logical Volume   Revision: 3000 Serial No:
    Size: 146.56GB <146561286144 bytes>
    Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
    Illegal Request: 2 Predictive Failure Analysis: 0
    As this machine seems to miss some important files (like /var/adm/messages), I'm starting to think that it might be better simply to re-install Solaris 10. Do you agree or do you see a way out of this?
  • 5. Re: Irreparable cylinder group header problem.  Program terminated.
    cindys Pro
    Currently Being Moderated
    Yes, I checked for overlapping slices but the partitioning looks good.

    Other than reinstalling, I think you're only alternate is to restore from backup.

    Unless you can restore the original meta* device configuration, but I don't see how
    that could resolve the apparent data corruption.

    Thanks, Cindy
  • 6. Re: Irreparable cylinder group header problem.  Program terminated.
    422096 Newbie
    Currently Being Moderated
    Hello again Cindy,
    As I don't have a backup of this machine I will re-install Solaris on it.

    Thanks for your support
    Dominique

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points