0 Replies Latest reply on Nov 11, 2019 3:42 AM by 26b0e9bc-5a1d-423f-a8e9-1a45173162d9

    vdbench report data validation error,when we do storage test

    26b0e9bc-5a1d-423f-a8e9-1a45173162d9

      When we did the storage data validation test, we found the following problems(Notethe following data are captured by FC protocol analyzer):

      (1) we read the disk LBA: 00BC3D20, and the LBA in the response data is: 00BC3D20

      (2) when we perform the write operation and execute the write SCSI command, the write LBA address changes to: 00BC3C40, and in the data to be written, LBA is: 00BC3D20

      (3) based on the above two steps, vdbench reported data validation error

       

       

       

      (1)The following is our physical connection topology:

       

      (2)OS and vdbench version

      OS Version: Ubuntu17.04

      Vdbench version: vdbench50406

       

      (3)our vdbench config file

      root@ubuntu:/vdbench# cat zhixing-fc.sh

      monitor=/vdbench/shutdown

      data_errors=01

      sd=default,openflags=o_direct

      sd=sd1,lun=/dev/mapper/mpathgo

      sd=sd2,lun=/dev/mapper/mpathgn

      sd=sd3,lun=/dev/mapper/mpathgm

      sd=sd4,lun=/dev/mapper/mpathgp

      wd=wd1,sd=sd*,xfersize=4096,seekpct=0,rdpct=50

      rd=default,iorate=max,interval=1,forthreads=32,elapsed=720h

      rd=run1,wd=wd*

       

      (4) Our test command

      When we do storage test ,we use the following command:

      ./vdbench -f zhixing-fc.sh -o test_1108 -vr

       

      (5)Our vdbench errorlog:

      root@ubuntu:/vdbench# cat test_1108/errorlog.html

      <title>Vdbench test_1108/errorlog.html</title><pre>

      Vdbench error log, created 04:29:14 Nov 08 2019 CST

      Error log. If there are no error messages beyond this line then there were no Data Validation or I/O errors.

       

       

      04:29:15.758 localhost-0: Created new mmap file: /tmp/vdbench.pid4505.sd1.mmap.0 mmap size: 7.500m

      04:29:15.759 localhost-0: Allocating Data Validation map: 7,864,320 one-byte entries for each 4,096-byte block.

      04:29:15.760 localhost-0: Created new DV map: sd1 size: 32,212,254,720 bytes; key block size: 4096; entries: 7,864,320

      04:29:15.768 localhost-1: Created new mmap file: /tmp/vdbench.pid4524.sd2.mmap.0 mmap size: 2.500m

      04:29:15.768 localhost-1: Allocating Data Validation map: 2,621,440 one-byte entries for each 4,096-byte block.

      04:29:15.769 localhost-1: Created new DV map: sd2 size: 10,737,418,240 bytes; key block size: 4096; entries: 2,621,440

      04:29:15.774 localhost-2: Created new mmap file: /tmp/vdbench.pid4544.sd3.mmap.0 mmap size: 5m

      04:29:15.776 localhost-2: Allocating Data Validation map: 5,242,880 one-byte entries for each 4,096-byte block.

      04:29:15.777 localhost-2: Created new DV map: sd3 size: 21,474,836,480 bytes; key block size: 4096; entries: 5,242,880

      04:29:15.789 localhost-3: Created new mmap file: /tmp/vdbench.pid4575.sd4.mmap.0 mmap size: 10m

      04:29:15.789 localhost-3: Allocating Data Validation map: 10,485,760 one-byte entries for each 4,096-byte block.

      04:29:15.790 localhost-3: Created new DV map: sd4 size: 42,949,672,960 bytes; key block size: 4096; entries: 10,485,760

      05:39:52.761 localhost-2: 05:39:52.719 05:39:52.716 op: read   lun: /dev/mapper/mpathgm            lba:   6316523520 0x1787E8000 xfer:     4096 errno: BAD_READ_RETURN: 'Read was successful, but data buffer contents not changed'

      05:39:52.762 localhost-0: 05:39:52.744

      05:39:52.762 localhost-0: 05:39:52.744 Time of first corruption: Fri Nov 08 2019 05:39:52.744 CST

      05:39:52.762 localhost-0: 05:39:52.744

      05:39:52.762 localhost-0: 05:39:52.744 At least one Data Validation error has been detected.                              

      05:39:52.763 localhost-0: 05:39:52.744

      05:39:52.763 localhost-0: 05:39:52.744 Terminology:                                                                       

      05:39:52.763 localhost-0: 05:39:52.744 - Data block: a block of xfersize= bytes.                                          

      05:39:52.763 localhost-0: 05:39:52.744 - Key block:  the smallest xfersize specified by the user which is the unit of     

      05:39:52.763 localhost-0: 05:39:52.745               data that Data Validation keeps track of.                            

      05:39:52.763 localhost-0: 05:39:52.745 - Sector:     512 bytes of disk storage, regardless of actual storage sector size. 

      05:39:52.763 localhost-0: 05:39:52.745 - Lba:        Logical Byte Address, not to be confused with Logical Block Address. 

      05:39:52.763 localhost-0: 05:39:52.745

      05:39:52.764 localhost-0: 05:39:52.745

      05:39:52.764 localhost-0: 05:39:52.745 The output starts with a summary of a data block, followed by a summary of each    

      05:39:52.764 localhost-0: 05:39:52.745 key block. If all sectors in a key block show a similar type of data corruption    

      05:39:52.764 localhost-0: 05:39:52.745 only the FIRST sector of the key block will be reported.                           

      05:39:52.764 localhost-0: 05:39:52.745 For all other cases, ALL sectors will be reported.                                 

      05:39:52.764 localhost-0: 05:39:52.745

      05:39:52.764 localhost-0: 05:39:52.745 Contents of the first 32 bytes of each sector:                                     

      05:39:52.764 localhost-0: 05:39:52.745

      05:39:52.765 localhost-0: 05:39:52.745 Byte 0x00 -  0x07: Byte offset of this disk block

      05:39:52.765 localhost-0: 05:39:52.745 Byte 0x08 -  0x0f: Timestamp: number of milliseconds since 1/1/1970

      05:39:52.765 localhost-0: 05:39:52.745 Byte 0x10        : Data Validation key from 1 - 126

      05:39:52.765 localhost-0: 05:39:52.745 Byte 0x11        : Checksum of timestamp

      05:39:52.765 localhost-0: 05:39:52.745 Byte 0x12 -  0x13: Reserved

      05:39:52.765 localhost-0: 05:39:52.745 Byte 0x14 -  0x1b: SD or FSD name in ASCII hexadecimal

      05:39:52.765 localhost-0: 05:39:52.745 Byte 0x1c -  0x1f: Process-id when written

      05:39:52.765 localhost-0: 05:39:52.745 Byte 0x20 - 0x1ff: 480 bytes of compression data pattern

      05:39:52.766 localhost-0: 05:39:52.745

      05:39:52.766 localhost-0: 05:39:52.746 On the left:  the data that was expected ('.' marks unknown value).

      05:39:52.766 localhost-0: 05:39:52.746 On the right: the data that was found.

      05:39:52.766 localhost-0: 05:39:52.746

      05:39:52.767 localhost-1: 05:39:52.739

      05:39:52.767 localhost-1: 05:39:52.739 Time of first corruption: Fri Nov 08 2019 05:39:52.739 CST

      05:39:52.767 localhost-1: 05:39:52.739

      05:39:52.767 localhost-1: 05:39:52.739 At least one Data Validation error has been detected.                              

      05:39:52.767 localhost-1: 05:39:52.739

      05:39:52.767 localhost-1: 05:39:52.739 Terminology:                                                                       

      05:39:52.768 localhost-1: 05:39:52.739 - Data block: a block of xfersize= bytes.                                          

      05:39:52.768 localhost-1: 05:39:52.739 - Key block:  the smallest xfersize specified by the user which is the unit of     

      05:39:52.768 localhost-1: 05:39:52.739               data that Data Validation keeps track of.                            

      05:39:52.768 localhost-1: 05:39:52.739 - Sector:     512 bytes of disk storage, regardless of actual storage sector size. 

      05:39:52.768 localhost-1: 05:39:52.739 - Lba:        Logical Byte Address, not to be confused with Logical Block Address. 

      05:39:52.768 localhost-1: 05:39:52.740

      05:39:52.768 localhost-1: 05:39:52.740

      05:39:52.769 localhost-1: 05:39:52.740 The output starts with a summary of a data block, followed by a summary of each    

      05:39:52.769 localhost-1: 05:39:52.740 key block. If all sectors in a key block show a similar type of data corruption    

      05:39:52.769 localhost-1: 05:39:52.740 only the FIRST sector of the key block will be reported.                           

      05:39:52.769 localhost-1: 05:39:52.740 For all other cases, ALL sectors will be reported.                                 

      05:39:52.769 localhost-1: 05:39:52.740

      05:39:52.769 localhost-1: 05:39:52.740 Contents of the first 32 bytes of each sector:                                     

      05:39:52.770 localhost-1: 05:39:52.740

      05:39:52.770 localhost-1: 05:39:52.740 Byte 0x00 -  0x07: Byte offset of this disk block

      05:39:52.770 localhost-1: 05:39:52.740 Byte 0x08 -  0x0f: Timestamp: number of milliseconds since 1/1/1970

      05:39:52.770 localhost-1: 05:39:52.740 Byte 0x10        : Data Validation key from 1 - 126

      05:39:52.770 localhost-1: 05:39:52.740 Byte 0x11        : Checksum of timestamp

      05:39:52.770 localhost-1: 05:39:52.740 Byte 0x12 -  0x13: Reserved

      05:39:52.770 localhost-1: 05:39:52.740 Byte 0x14 -  0x1b: SD or FSD name in ASCII hexadecimal

      05:39:52.771 localhost-1: 05:39:52.740 Byte 0x1c -  0x1f: Process-id when written

      05:39:52.771 localhost-1: 05:39:52.740 Byte 0x20 - 0x1ff: 480 bytes of compression data pattern

      05:39:52.771 localhost-1: 05:39:52.740

      05:39:52.771 localhost-1: 05:39:52.740 On the left:  the data that was expected ('.' marks unknown value).

      05:39:52.771 localhost-1: 05:39:52.740 On the right: the data that was found.

      05:39:52.771 localhost-1: 05:39:52.740

      05:39:52.771 localhost-1: 05:39:52.744

      05:39:52.772 localhost-1: 05:39:52.744

      05:39:52.772 localhost-1: 05:39:52.744

      05:39:52.772 localhost-1: 05:39:52.744 Corrupted data block for sd=sd2,lun=/dev/mapper/mpathgn; lba: 7,525,941,248 (0x1c094c000) xfersize=4096

      05:39:52.772 localhost-1: 05:39:52.744

      05:39:52.772 localhost-1: 05:39:52.744 Data block has 1 key block(s) of 4096 bytes each.

      05:39:52.772 localhost-1: 05:39:52.744 All key blocks are corrupted.

      05:39:52.773 localhost-1: 05:39:52.744 Key block lba: 0x1c094c000

      05:39:52.773 localhost-1: 05:39:52.744    Key block of 4,096 bytes has 8 512-byte sectors.

      05:39:52.773 localhost-1: 05:39:52.745    Timeline:

      05:39:52.773 localhost-1: 05:39:52.745    Fri Nov 08 2019 05:22:27.470 CST Sector last written. (As found in the first corrupted sector, timestamp is taken just BEFORE the actual write).

      05:39:52.773 localhost-1: 05:39:52.746    Fri Nov 08 2019 05:39:52.736 CST Key block first found to be corrupted during a read-after-write.

      05:39:52.773 localhost-1: 05:39:52.746

      05:39:52.774 localhost-1: 05:39:52.746    All 8 sectors in this key block are corrupted.

      05:39:52.774 localhost-1: 05:39:52.746    Corruption types do not match. Reporting all sectors:

      05:39:52.774 localhost-1: 05:39:52.746    Error type counts:

      05:39:52.774 localhost-1: 05:39:52.746    ===> Data Validation Key miscompare:       8

      05:39:52.774 localhost-1: 05:39:52.746    ===> Logical byte address miscompare:      8

      05:39:52.774 localhost-1: 05:39:52.746    ===> SD or FSD name miscompare:            7

      05:39:52.775 localhost-1: 05:39:52.746    ===> Compression pattern miscompare:       8

      05:39:52.775 localhost-1: 05:39:52.747

      05:39:52.775 localhost-1: 05:39:52.747         Data Validation error for sd=sd2,lun=/dev/mapper/mpathgn

      05:39:52.805 localhost-1: 05:39:52.747         Block lba: 0x1c094c000; sector lba: 0x1c094c000; Key block size: 4096; relative sector in data block: 0x00 ( 0); current pid: 4524 (0x11ac)

      05:39:52.805 localhost-1: 05:39:52.748         ===> Data Validation Key miscompare. Expecting key=0x0b received key=0x04 (11/4)

      05:39:52.805 localhost-1: 05:39:52.748         ===> Logical byte address miscompare. Expecting 0x1c094c000, receiving 0x1c08ea000

      05:39:52.805 localhost-1: 05:39:52.748         ===> Compression pattern miscompare.

      05:39:52.805 localhost-1: 05:39:52.748         Timestamp found in sector: Fri Nov 08 2019 05:22:27.470 CST

      05:39:52.805 localhost-1: 05:39:52.748 0x000*  00000001 c094c000 ........ ........   00000001 c08ea000 0000016e 4ac04f0e

      05:39:52.806 localhost-1: 05:39:52.749 0x010   0b..0000 20326473 20202020 00000000   04d60000 20316473 20202020 00001199

      05:39:52.806 localhost-1: 05:39:52.749 0x020*  3d98e971 5cd50ee6 3b5026b8 71f843c0   68503889 0d109bb5 36b98050 39f396c0

      05:39:52.806 localhost-1: 05:39:52.750 0x030*  6b0ae7c0 6f418bb3 39a697bb 51796142   7caab94e 5391a0cd 716fe98e 0023fc1a

      05:39:52.806 localhost-1: 05:39:52.750 0x040*  44a65640 7a82dc28 607fd9a6 3c64f249   699904e3 06137fa0 7df24cc2 705958c3

      05:39:52.806 localhost-1: 05:39:52.751 0x050*  5ebaed8c 57f8a6c4 1ff9761d 7f09d891   21875044 347cba74 6e466f0c 517b2aa1

      05:39:52.807 localhost-1: 05:39:52.751 0x060*  3639f5d3 53a181b4 6784bb4f 698b5954   6016df62 40d24354 7a115b3c 163cd2ee

      05:39:52.807 localhost-1: 05:39:52.752 0x070*  18d3f4f6 6115ebd0 46891058 61ab5920   412bb45a 34a27559 4d29b3e5 2657d23e

      05:39:52.807 localhost-1: 05:39:52.752 0x080*  6f45e4f3 71520cfa 15a64dba 3e87e105   2829c50c 553ed440 2cca0ded 6fb94d26

      05:39:52.807 localhost-1: 05:39:52.752 0x090*  5d3ef216 595f3de8 499020f9 2e6ce9fe   2acab9b8 26d594e0 545c52a1 3fc3d91b

      05:39:52.807 localhost-1: 05:39:52.753 0x0a0*  4e67c0f5 4b0e58f3 569b7596 69fb70f6   571532e9 1f9bed77 30f729b7 67e68f29

      05:39:52.807 localhost-1: 05:39:52.753 0x0b0*  10343d31 7613d1a6 7636e874 3f06d48b   341e987f 615a2d7f 0cc66bcd 77ea6994

      05:39:52.808 localhost-1: 05:39:52.753 0x0c0*  0448ef3d 00f69595 2da74176 08a97773   49606ebe 68ca9ed6 49b79cfc 6376e733

      05:39:52.808 localhost-1: 05:39:52.754 0x0d0*  475cdab3 2662b44c 2780f5ad 7931ae22   488a3540 3f5c2ca3 7ce36e36 376ef875

      05:39:52.808 localhost-1: 05:39:52.754 0x0e0*  5ca0e520 6490cffe 4777b237 2da7f234   29d8cdaa 5d09d52c 124d4897 16e732e6

      05:39:52.808 localhost-1: 05:39:52.754 0x0f0*  05d5c354 519354ac 5a3ced64 59cf5ef4   231c68b7 0d17e0ff 1ab1de87 0e964e15

      05:39:52.808 localhost-1: 05:39:52.755 0x100*  6c59706c 507a7a09 61468292 76a175da   5db5fd85 51a7e31a 033dc75a 68c0e712

      05:39:52.808 localhost-0: 05:39:52.750

      05:39:52.808 localhost-1: 05:39:52.755 0x110*  1f952bb8 10600a79 3c7652a7 3c662a22   2ff54745 6f34821c 39251fed 0ac18d8b

      05:39:52.809 localhost-0: 05:39:52.750

      05:39:52.809 localhost-1: 05:39:52.756 0x120*  3154fece 6f8bf368 59bbefae 54a93602   62604bfe 7a26078f 21353656 086e179c

      05:39:52.809 localhost-0: 05:39:52.750

      05:39:52.809 localhost-1: 05:39:52.756 0x130*  60b1cc60 12029e6b 1ce94e63 176db629   318d69c1 75d0b8ee 55993b37 00bb3d9c

      05:39:52.809 localhost-0: 05:39:52.751 Corrupted data block for sd=sd1,lun=/dev/mapper/mpathgo; lba: 6,804,574,208 (0x195959000) xfersize=4096

      05:39:52.809 localhost-0: 05:39:52.751

      05:39:52.809 localhost-1: 05:39:52.756 0x140*  5056e624 33ee1068 6013809f 52210e36   4958eeaf 4d4df757 4a2afc8e 021f7f36

      05:39:52.809 localhost-0: 05:39:52.751 Data block has 1 key block(s) of 4096 bytes each.

      05:39:52.809 localhost-1: 05:39:52.756 0x150*  350c2aa0 7d72d2a0 3ef877c4 05ee14bc   665f3556 2bc127b0 49c8a7cf 24564f51

      05:39:52.809 localhost-0: 05:39:52.751 All key blocks are corrupted.

      05:39:52.810 localhost-1: 05:39:52.757 0x160*  3d8dac5f 5ea1a283 5602cec3 3e318804   1ec44bf0 7ab963a9 3548939d 38448805

      05:39:52.810 localhost-0: 05:39:52.751 Key block lba: 0x195959000

      05:39:52.810 localhost-0: 05:39:52.751    Key block of 4,096 bytes has 8 512-byte sectors.

      05:39:52.810 localhost-1: 05:39:52.757 0x170*  2cd9df75 3f56ea47 37a49c5a 16f0ca32   0b7222ce 1280f137 06c412e0 65df2780

      05:39:52.810 localhost-0: 05:39:52.751    Timeline:

      05:39:52.810 localhost-1: 05:39:52.757 0x180*  3c3f4fa0 17f900a1 18e24d6a 4653e505   271641ce 714b837a 5a124e96 4b15539b

      05:39:52.810 localhost-0: 05:39:52.752    Fri Nov 08 2019 05:39:14.683 CST Sector last written. (As found in the first corrupted sector, timestamp is taken just BEFORE the actual write).

      05:39:52.810 localhost-1: 05:39:52.758 0x190*  5a92dfcf 55263193 1806874e 7b43d68d   38480758 1b5441ee 1d6c2813 2ddea635

      05:39:52.810 localhost-0: 05:39:52.752    Fri Nov 08 2019 05:39:52.730 CST Key block first found to be corrupted during a workload requested read.

      05:39:52.810 localhost-1: 05:39:52.759 0x1a0*  42178e55 39838e74 0698120f 266e2548   00606e47 171c9a50 13522332 16aec1e6

      05:39:52.810 localhost-0: 05:39:52.752

      05:39:52.810 localhost-0: 05:39:52.752    All 8 sectors in this key block are corrupted.

      05:39:52.810 localhost-1: 05:39:52.759 0x1b0*  26c0c502 45fd85f4 49caca60 4a0ebc98   30ac5c4c 4e1cd733 78ed58ca 2faf2e05

      05:39:52.811 localhost-0: 05:39:52.752    All corruptions are of the same type:

      05:39:52.811 localhost-1: 05:39:52.759 0x1c0*  6a982e98 09eb44d2 32c49c6a 38011098   53c2781b 73578171 724470e7 10407aac

      05:39:52.811 localhost-0: 05:39:52.753    ===> Data Validation Key miscompare. Expecting key=0x05 received key=0x08 (5/8)

      05:39:52.811 localhost-0: 05:39:52.753    ===> Logical byte address miscompare. Expecting 0x195959000, receiving 0x1958e2000

      05:39:52.811 localhost-1: 05:39:52.760 0x1d0*  563014b6 19a5ddcc 222f8568 34b98a68   68e5875e 6e608791 4fbda4c3 2b63ad39

      05:39:52.811 localhost-0: 05:39:52.753    ===> SD or FSD name miscompare. Expecting 'sd1     ', receiving 'sd2     ' (0x2032647320202020)

      05:39:52.811 localhost-1: 05:39:52.760 0x1e0*  472b46e3 4b4439d0 60b19e44 48e83d37   637c5565 2a892eb4 5686c99b 6accf46d

      05:39:52.811 localhost-0: 05:39:52.753    ===> Compression pattern miscompare.

      05:39:52.811 localhost-1: 05:39:52.760 0x1f0*  1f038834 3e4cd0d0 50042df4 22a5e137   18c8219b 0f01ca66 6f9c61f8 1780a4dc

      05:39:52.811 localhost-0: 05:39:52.753    Only the FIRST sector will be reported:

      05:39:52.811 localhost-1: 05:39:52.768

      05:39:52.812 localhost-1: 05:39:52.768         Data Validation error for sd=sd2,lun=/dev/mapper/mpathgn

      05:39:52.812 localhost-1: 05:39:52.768         Block lba: 0x1c094c000; sector lba: 0x1c094c200; Key block size: 4096; relative sector in data block: 0x01 ( 1); current pid: 4524 (0x11ac)

      05:39:52.812 localhost-1: 05:39:52.769         ===> Data Validation Key miscompare. Expecting key=0x0b received key=0x04 (11/4)

      05:39:52.812 localhost-1: 05:39:52.769         ===> Logical byte address miscompare. Expecting 0x1c094c200, receiving 0x1c08ea200

      05:39:52.812 localhost-1: 05:39:52.769         ===> SD or FSD name miscompare. Expecting 'sd2     ', receiving 'sd1     ' (0x2031647320202020)

      05:39:52.812 localhost-1: 05:39:52.769         ===> Compression pattern miscompare.

      05:39:52.813 localhost-1: 05:39:52.769         SD or FSD name in block expected: 'sd2'; received: 'sd1     '.

      05:39:52.813 localhost-1: 05:39:52.770         Timestamp found in sector: Fri Nov 08 2019 05:22:27.470 CST

      05:39:52.813 localhost-1: 05:39:52.770 0x000*  00000001 c094c200 ........ ........   00000001 c08ea200 0000016e 4ac04f0e

      05:39:52.813 localhost-1: 05:39:52.770 0x010   0b..0000 20326473 20202020 00000000   04d60000 20316473 20202020 00001199

      05:39:52.813 localhost-1: 05:39:52.770 0x020*  1c967a69 4d27fa00 3e2879e9 4765f0d2   1f74a57a 1dea75b1 6a3e8d62 797c40a4

      05:39:52.814 localhost-1: 05:39:52.771 0x030*  76b275ee 40753c57 7c1e7dfd 14bdbdac   6f44bf38 4ae73c70 3efde5b0 60521161

      05:39:52.814 localhost-1: 05:39:52.771 0x040*  414cdb52 67f54ac9 6894b9f0 1735f8ae   37551dd4 01265521 74d61ec2 077e5410

      05:39:52.814 localhost-1: 05:39:52.771 0x050*  508befdf 4581d1dd 29f3c788 6689ad65   6d58824d 01f8484f 5e880e61 2944b06b

      05:39:52.814 localhost-1: 05:39:52.772 0x060*  2264cf85 4df5f5ee 67b1cdd6 154b538d   0f64051b 0f2e9671 542d9785 30b7ba13

      05:39:52.814 localhost-1: 05:39:52.772 0x070*  741a1cfc 04f24c38 6d65d361 2d0809f2   285dc423 674fb09f 443cfce2 2d4c2bd3

      05:39:52.814 localhost-1: 05:39:52.772 0x080*  250a363d 698f9ac5 539b199f 067e1bd6   0fbb1a24 30b9b918 184c3bcc 207f6cc0

      05:39:52.815 localhost-1: 05:39:52.772 0x090*  1e35f3be 081160cd 535d7de8 6207cc12   5bf75b25 67c50a84 528d6d8e 49b25c2b

      05:39:52.815 localhost-0: 05:39:52.753

      05:39:52.815 localhost-1: 05:39:52.773 0x0a0*  1a08ee6d 15c92677 2f97e441 277b6aa8   234c1c6f 751789cb 0f1d324e 177965b4

      05:39:52.815 localhost-0: 05:39:52.753         Data Validation error for sd=sd1,lun=/dev/mapper/mpathgo

      05:39:52.815 localhost-0: 05:39:52.754         Block lba: 0x195959000; sector lba: 0x195959000; Key block size: 4096; relative sector in data block: 0x00 ( 0); current pid: 4505 (0x1199)

      05:39:52.815 localhost-1: 05:39:52.773 0x0b0*  300c4e35 117b95fe 097da9d9 7be1afa4   2e54018a 2679bca2 7e5c2311 2cc2ddb5

      05:39:52.815 localhost-0: 05:39:52.754         ===> Data Validation Key miscompare. Expecting key=0x05 received key=0x08 (5/8)

      05:39:52.815 localhost-1: 05:39:52.773 0x0c0*  744d1fdc 7bb85ad9 03581e52 668360a2   2c611334 5d9eaada 7bac4bb4 7354a551

      05:39:52.815 localhost-0: 05:39:52.754         ===> Logical byte address miscompare. Expecting 0x195959000, receiving 0x1958e2000

      05:39:52.815 localhost-1: 05:39:52.774 0x0d0*  4a5f3312 57b5cb11 47310769 0def3bcb   496f9d9e 656f861f 160badc2 04421702

      05:39:52.815 localhost-0: 05:39:52.754         ===> SD or FSD name miscompare. Expecting 'sd1     ', receiving 'sd2     ' (0x2032647320202020)

      05:39:52.815 localhost-0: 05:39:52.754         ===> Compression pattern miscompare.

      05:39:52.815 localhost-1: 05:39:52.774 0x0e0*  4005819d 466156ce 5cf32a6e 28b92d58   39be9614 7f8c1af7 3624ca98 421b3b01

      05:39:52.816 localhost-0: 05:39:52.755         SD or FSD name in block expected: 'sd1'; received: 'sd2     '.

      05:39:52.816 localhost-1: 05:39:52.774 0x0f0*  5ba91caf 3275c0c0 35b9dde7 14d3caa9   673689f6 60d107e9 078e34fc 57b642b4

      05:39:52.816 localhost-0: 05:39:52.755         Timestamp found in sector: Fri Nov 08 2019 05:39:14.683 CST

      05:39:52.816 localhost-1: 05:39:52.775 0x100*  46e7a396 2d881ea3 7028a575 23851790   68e8145f 7cd4be9d 69122c06 69944dc1

      05:39:52.817 localhost-1: 05:39:52.775 0x110*  1da241d2 4b1ca0ff 1bc3d22c 07179194   4d0add5e 2d6c4b98 69f73d5b 270275c9

      05:39:52.817 localhost-0: 05:39:52.755 0x000*  00000001 95959000 ........ ........   00000001 958e2000 0000016e 4acfad7b

      05:39:52.819 localhost-1: 05:39:52.775 0x120*  6e163553 77f423c0 27632e6e 728884e3   02061e40 7468e6e1 4062eefe 0b38234d

      05:39:52.820 localhost-0: 05:39:52.755 0x010   05..0000 20316473 20202020 00000000   08b00000 20326473 20202020 000011ac

      05:39:52.820 localhost-0: 05:39:52.756 0x020*  1b8b615c 7316ea86 53d2b1fa 308b8b97   65c5a809 06ab454c 0ffd6672 391e6cb4

      05:39:52.820 localhost-0: 05:39:52.756 0x030*  6e63dace 11ca1df4 003cb7d3 45560fd6   6f0b30c4 1e6268c3 4995db6d 74fafa0f

      05:39:52.820 localhost-1: 05:39:52.776 0x130*  1ea7dd33 00e38684 30f7af2c 6294a8b5   532bb247 4adf4c1b 2f0f71f1 05d3a8dc

      05:39:52.821 localhost-1: 05:39:52.776 0x140*  7a294ef9 35a5ccb0 1afd2ea3 32a402b5   79eeab41 2a69da7e 7ead5c7c 6d362881

      05:39:52.820 localhost-0: 05:39:52.756 0x040*  120eaea1 69c15fe6 6914ce37 3f811f6a   13a70809 6d055583 0cb185cd 2d56e67c

      05:39:52.821 localhost-1: 05:39:52.776 0x150*  138575a8 513a056e 1e112e5a 0f3613d0   347e703e 39f67d4c 016a6c1d 0c5fbf42

      05:39:52.822 localhost-1: 05:39:52.776 0x160*  05d8b83e 127dfca7 0347a145 0a8f4caf   26cf6387 48d15c8a 0d364ff0 510cf990

      05:39:52.822 localhost-1: 05:39:52.777 0x170*  4e20268e 34d6b2a5 4df6beb2 0f88c993   033a11bc 07bd546c 737e7b00 15f68fc3

      05:39:52.822 localhost-1: 05:39:52.777 0x180*  17c83130 41d79c8c 59d2dba9 29835651   13f9e70e 390b3638 3b39fea4 403469d3

      05:39:52.822 localhost-1: 05:39:52.777 0x190*  5583e6d4 01f9ac93 54534dcc 179ebb80   3ef46e3c 5dc0cfc5 443e6c76 65e4ec1b

      05:39:52.822 localhost-1: 05:39:52.778 0x1a0*  63f5ba52 23932239 7e79557a 647651fe   0152163f 3700bd0c 00f6c04e 77598b6a

      05:39:52.823 localhost-1: 05:39:52.778 0x1b0*  3bd2d1b1 04612296 4e320f79 42c3deca   289180fa 4803fec8 1db5538a 6b29a931

      05:39:52.823 localhost-1: 05:39:52.778 0x1c0*  6549dbce 63141892 54ac7010 3a69b8cd   3ade593b 5e165c09 3af9d66a 38acb7fb

      05:39:52.823 localhost-1: 05:39:52.778 0x1d0*  6e966ecd 1b6fdd36 411c450a 40853365   1094b149 012689de 52245242 7b7ca70c

      05:39:52.823 localhost-1: 05:39:52.779 0x1e0*  1bc9ee5c 70b0a795 6d833f73 284a53c5   14f9e8a0 109b1b55 722e34ba 6dc197f8

      05:39:52.823 localhost-1: 05:39:52.779 0x1f0*  5df2f9ab 4f05c5d8 1d390688 7189cc93   7c541ac0 353d3a68 3c22609e 7e27d8f7

      05:39:52.823 localhost-1: 05:39:52.785

      05:39:52.823 localhost-1: 05:39:52.785         Data Validation error for sd=sd2,lun=/dev/mapper/mpathgn

      05:39:52.824 localhost-1: 05:39:52.785         Block lba: 0x1c094c000; sector lba: 0x1c094c400; Key block size: 4096; relative sector in data block: 0x02 ( 2); current pid: 4524 (0x11ac)

      05:39:52.824 localhost-1: 05:39:52.786         ===> Data Validation Key miscompare. Expecting key=0x0b received key=0x04 (11/4)

      05:39:52.822 localhost-0: 05:39:52.756 0x050*  27108f82 27dae0db 6ec31801 23f8c630   6f82525d 6818fd8f 49b21ace 742f1f7b

      05:39:52.824 localhost-1: 05:39:52.786         ===> Logical byte address miscompare. Expecting 0x1c094c400, receiving 0x1c08ea400

      05:39:52.824 localhost-0: 05:39:52.757 0x060*  4674ffbc 651b00cc 11fcbe31 3e38b32b   606943ed 229ec348 2c269557 6a8a7624

      05:39:52.824 localhost-0: 05:39:52.757 0x070*  0953ff18 4d4f48e5 424e4150 163e7525   44d8ced3 703f71cb 4d1d87b9 7373db85

      05:39:52.824 localhost-1: 05:39:52.786         ===> SD or FSD name miscompare. Expecting 'sd2     ', receiving 'sd1     ' (0x2031647320202020)

      05:39:52.824 localhost-0: 05:39:52.757 0x080*  778741b1 73802ad9 0a45d45b 0d6efedf   42fbd90c 3f44fc91 741776ff 26e7e459

      05:39:52.824 localhost-1: 05:39:52.786         ===> Compression pattern miscompare.

      05:39:52.824 localhost-0: 05:39:52.758 0x090*  70f7eda3 6442894b 5a36be33 0290737e   4b40fab1 5974088f 47923c67 04d69546

      05:39:52.824 localhost-0: 05:39:52.758 0x0a0*  490de1df 171a08b8 78aec51e 60193a17   2e11c46a 4cf63beb 14466fa1 6b7a77f9

      05:39:52.824 localhost-1: 05:39:52.786         SD or FSD name in block expected: 'sd2'; received: 'sd1     '.

      05:39:52.824 localhost-0: 05:39:52.758 0x0b0*  53e98397 753d268c 09d69f69 3f355ac5   24548de1 53a65663 3c4fd98f 1b65f675

      05:39:52.824 localhost-0: 05:39:52.758 0x0c0*  11311573 452c1a6e 2d166f07 550322e3   1681cb9f 282f117a 6d99edf8 1ec7ffb3

      05:39:52.825 localhost-1: 05:39:52.787         Timestamp found in sector: Fri Nov 08 2019 05:22:27.470 CST

      05:39:52.825 localhost-0: 05:39:52.759 0x0d0*  705a951c 67ab7deb 62774376 2ea81092   3e35ed3f 558dc8d4 71094df1 3888f40b

      05:39:52.825 localhost-0: 05:39:52.759 0x0e0*  5e7b21e8 73d3bd0d 3f5aa327 53ff075e   7248c415 2ea9ea2d 39fb5262 1300d925

      05:39:52.825 localhost-1: 05:39:52.787 0x000*  00000001 c094c400 ........ ........   00000001 c08ea400 0000016e 4ac04f0e

      05:39:52.825 localhost-0: 05:39:52.759 0x0f0*  65c4eaf6 65064e4a 74411f2a 3fc0ccbb   05937106 40d22e24 57d49e55 3d27d66e

      05:39:52.825 localhost-1: 05:39:52.787 0x010   0b..0000 20326473 20202020 00000000   04d60000 20316473 20202020 00001199

      05:39:52.825 localhost-0: 05:39:52.759 0x100*  22fca521 36b9b2fa 4378ec5e 3727cbe1   1caddb51 3d10983b 0977b821 2041f917

      05:39:52.825 localhost-0: 05:39:52.760 0x110*  354f89c2 772d3f59 4aad85bf 6a30228f   3f0cae22 01663e89 749cad96 772740bc

      05:39:52.825 localhost-1: 05:39:52.787 0x020*  3b8f0878 060d71c2 295b7635 3d5d0396   138f8f4d 0d7e5c51 2751c3a9 7aadd054

      05:39:52.825 localhost-0: 05:39:52.760 0x120*  1112dca2 0b35ab39 6771574e 68b4f464   2c08df35 1c1fdaf2 6922bc2a 7368f336

      05:39:52.825 localhost-0: 05:39:52.760 0x130*  4615fa44 127369cb 08fa6ff8 5fcc087e   2479bd35 589422b7 229504c8 23b53933

      05:39:52.825 localhost-1: 05:39:52.787 0x030*  0f6efb2b 55a99e1e 2c886b6d 054fa838   3fdf3c52 69d508f7 3ea16917 435d34c3

      05:39:52.825 localhost-0: 05:39:52.760 0x140*  564fee40 1463ca42 6f6b3b8e 3de0ac13   65aa8599 1dbce6ff 283b457f 59fa026e

      05:39:52.825 localhost-1: 05:39:52.788 0x040*  490f59d3 685ed6a1 79e887a5 2add7d52   4308b00c 7e37c7f9 4a0c5e7d 7f794d9e

      05:39:52.825 localhost-0: 05:39:52.761 0x150*  29d19198 5bdddfd5 2b20453d 4b2edfef   02948311 243bfcb5 3e2f9b53 3ee831b1

      05:39:52.825 localhost-1: 05:39:52.788 0x050*  7a05f589 6641ce56 0c5874f5 09feb8ca   539e37ea 7b2227b3 3b3789a2 1572ecb9

      05:39:52.825 localhost-0: 05:39:52.761 0x160*  01053038 27ac31c0 249c0206 775fcc82   1c66b448 4a7ced6e 50124d78 73981138

      05:39:52.826 localhost-1: 05:39:52.788 0x060*  5904bf3b 01eb37c7 1d9c3232 59a6048c   29e1c090 5713373a 24faa5fc 0aacd7f7

      05:39:52.826 localhost-0: 05:39:52.761 0x170*  7e64b893 1c14346d 0fc9a2c9 14287dc9   3383f90c 12c46f65 58149e6a 19300ec5

      05:39:52.826 localhost-1: 05:39:52.788 0x070*  28b97505 3f04be9b 2fe366e9 163cb9c4   15948454 358a7dda 72d4964d 25680462

      05:39:52.826 localhost-0: 05:39:52.761 0x180*  7cef7073 2a3c8888 6b6fee47 6febe760   6640c996 201ea577 0333a76f 518ec070

      05:39:52.826 localhost-1: 05:39:52.788 0x080*  571cd9d4 39e4c6cb 1a2dbc22 5a5265d8   3a3641bb 6d13bbe9 713fc024 7ae31bba

      05:39:52.826 localhost-0: 05:39:52.762 0x190*  68c8345d 6f83041a 0fcd241b 428131b7   72c9740f 543d2759 359c3981 4b19301e

      05:39:52.826 localhost-1: 05:39:52.789 0x090*  509a71e8 2849f610 40883fcd 2da095e1   2d1cf97f 247272b7 0fdf6d5f 31ce46fb

      05:39:52.826 localhost-0: 05:39:52.762 0x1a0*  33853d17 5ff3823e 5574e5ef 215f4d02   35b9e38d 0175d284 1850c8e7 2cee707b

      05:39:52.826 localhost-1: 05:39:52.789 0x0a0*  2699caf1 5c6782a0 673cce1f 55a3ec06   54ee34a1 101e34fd 0c9f360c 44544416

      05:39:52.826 localhost-0: 05:39:52.763 0x1b0*  45da6e47 64307be1 07a92add 378ece82   270fed9a 793361de 034a5e57 3fe1789f

      05:39:52.826 localhost-1: 05:39:52.789 0x0b0*  3112487b 28934d1a 7a2c8450 7a2b7b1b   7222d399 6e60beb0 20fd736b 5419c1c6

      05:39:52.826 localhost-0: 05:39:52.763 0x1c0*  7c672cbb 6c3e67c6 23873887 48e314ae   034128b9 1848cdd7 656d919c 7bba48f4

      05:39:52.827 localhost-1: 05:39:52.789 0x0c0*  3ef27bd3 1b06bf21 2fae3ab6 7b5f6a00   245e6305 7986d614 58d9b9f7 2db4038d

      05:39:52.827 localhost-0: 05:39:52.763 0x1d0*  4d90bbf5 55c2e297 5781a69c 171bb28f   5e5d4ae1 4fb87554 3dcc8c59 2f5f5699

      05:39:52.827 localhost-1: 05:39:52.789 0x0d0*  39440106 2643750e 206a0701 25dfe235   1082fb63 24b4f33b 6c405b31 6837ce5a

      05:39:52.827 localhost-0: 05:39:52.764 0x1e0*  2a7225ad 65689f29 4cc067b6 12af251d   53460f87 09a00d51 4fcf134c 4da8404e

      05:39:52.827 localhost-1: 05:39:52.790 0x0e0*  5114e5e2 6bccad69 43cac69b 04bd4158   7f8aa643 1910b069 4bcbf0b5 20a19bf2

      05:39:52.827 localhost-0: 05:39:52.764 0x1f0*  050aa6d3 78391f94 7e1fdd18 581fce95   5925a5b1 2f1a5967 75f9996d 7fbacad2

      05:39:52.827 localhost-0: 05:39:52.774 05:39:52.773 op: read   lun: /dev/mapper/mpathgo            lba:   6804574208 0x195959000 xfer:     4096 errno: 60003: '60003            A Data Validation error was discovered'

      05:39:52.827 localhost-1: 05:39:52.790 0x0f0*  684f980a 040680eb 3abeb0cf 682a979a   664f4d2c 0e1002fa 0955ed9d 27f81878

      05:39:52.827 localhost-1: 05:39:52.790 0x100*  61ca5b15 472be38b 709ff0d5 52eb6a88   428a2fde 40df3a50 64cf394f 1629e5bb

      05:39:52.827 localhost-1: 05:39:52.790 0x110*  64f7467e 486c6c3a 5c3e488f 5f9dc264   1973e1dc 7beec983 0739d165 12c7a7b1

      05:39:52.828 localhost-1: 05:39:52.790 0x120*  65bdcccf 6562e4e2 1122ba4f 52217da3   288dd18d 50d9d6f0 5bdc7497 47db58d7

      05:39:52.828 localhost-1: 05:39:52.791 0x130*  6926c8f7 0135a3b0 5b463b44 707ded54   013e5605 5bf6940a 01e953e1 5a3de65f

      05:39:52.828 localhost-1: 05:39:52.791 0x140*  0ba414ee 3bd1aa04 4d946e36 69c6b97d   54084541 22015eb9 7ef0cdd1 6dec142e

      05:39:52.828 localhost-1: 05:39:52.791 0x150*  3ddca85b 32d18d87 7db36460 00a10d9d   204712bc 5573a87c 2c993022 2bfba9cb

      05:39:52.828 localhost-1: 05:39:52.791 0x160*  01063dbc 1ac1286a 1da969a7 622a0c33   517874fe 1237a72d 52348253 72f1e5ee

      05:39:52.828 localhost-1: 05:39:52.791 0x170*  23f581a6 448f10e9 3e19d522 091f8be3   0b3c94d0 6389cdae 74ed5797 515c3aea

      05:39:52.829 localhost-1: 05:39:52.791 0x180*  1e2dc90b 6c1cdaa7 41f1e470 484c1b01   0155c2c3 30f90b2c 2763a95b 25ac5350

      05:39:52.829 localhost-1: 05:39:52.792 0x190*  26594e6e 79ff1e0f 6e7e9d17 16574c27   3910b5ae 7c96d425 03b2b992 3d785d85

      05:39:52.829 localhost-1: 05:39:52.792 0x1a0*  05b2f93f 339e48d1 0904b7ae 783b4a5d   759e50ef 2d87f268 00f8fc7a 1160a0a5

      05:39:52.829 localhost-1: 05:39:52.792 0x1b0*  720aab8a 2a093636 1d139182 6ec0b533   5907f296 77471d52 7f9b0b85 41fb5865

      05:39:52.829 localhost-1: 05:39:52.792 0x1c0*  642d1883 10ea8f8a 54ec273d 1681c57a   6f8749d6 470a59f3 359a9f7a 4547d962

      05:39:52.829 localhost-1: 05:39:52.792 0x1d0*  7098228d 0431f3ed 3f751662 3915b90a   6b2ab4e0 7ddc2360 167a919c 4349fd9d

      05:39:52.829 localhost-1: 05:39:52.793 0x1e0*  1c84c197 2421e90f 7554a8b6 61258739   49cf27c2 1f195b98 6848e7c7 738ed822

      05:39:52.830 localhost-1: 05:39:52.793 0x1f0*  016f712c 549392fb 48e9e566 1a4adcff   77989a27 00250245 7187e542 647bb225

      05:39:52.830 localhost-1: 05:39:52.800

      05:39:52.830 localhost-1: 05:39:52.800         Data Validation error for sd=sd2,lun=/dev/mapper/mpathgn

      05:39:52.830 localhost-1: 05:39:52.800         Block lba: 0x1c094c000; sector lba: 0x1c094c600; Key block size: 4096; relative sector in data block: 0x03 ( 3); current pid: 4524 (0x11ac)

      05:39:52.830 localhost-1: 05:39:52.800         ===> Data Validation Key miscompare. Expecting key=0x0b received key=0x04 (11/4)

      05:39:52.830 localhost-1: 05:39:52.800         ===> Logical byte address miscompare. Expecting 0x1c094c600, receiving 0x1c08ea600

      05:39:52.831 localhost-1: 05:39:52.800         ===> SD or FSD name miscompare. Expecting 'sd2     ', receiving 'sd1     ' (0x2031647320202020)

      05:39:52.831 localhost-1: 05:39:52.800         ===> Compression pattern miscompare.

      05:39:52.831 localhost-1: 05:39:52.800         SD or FSD name in block expected: 'sd2'; received: 'sd1     '.

      05:39:52.831 localhost-1: 05:39:52.800         Timestamp found in sector: Fri Nov 08 2019 05:22:27.470 CST

      05:39:52.831 localhost-1: 05:39:52.800 0x000*  00000001 c094c600 ........ ........   00000001 c08ea600 0000016e 4ac04f0e

      05:39:52.831 localhost-1: 05:39:52.800 0x010   0b..0000 20326473 20202020 00000000   04d60000 20316473 20202020 00001199

      05:39:52.831 localhost-1: 05:39:52.800 0x020*  3c615b59 1fb57e41 48ec6388 2206042f   047fbe10 41fc57b5 37f66ada 39b0cdda

      05:39:52.832 localhost-1: 05:39:52.800 0x030*  59303f7a 4a07b8bc 0908a803 0ae0a8ca   7069f884 59840e74 6c7ebbb9 7ef6ee55

      05:39:52.832 localhost-1: 05:39:52.801 0x040*  530e9988 200187ba 07e08ae5 40b6082f   21d483c0 6f89dfc0 6efa539a 4fa4cd74

      05:39:52.832 localhost-1: 05:39:52.801 0x050*  7a9ac6ec 11b3a45a 76edc5f9 408c82d8   51ca3959 0575606a 721b289f 5b29679b

      05:39:52.832 localhost-1: 05:39:52.801 0x060*  7afc8cbc 48554f6b 408b30bd 2ba7f471   6e72d9fb 3f542daa 21bfce29 4728b4bb

      05:39:52.832 localhost-1: 05:39:52.801 0x070*  3625c579 5b9a4abf 5bea129c 245ef0a6   6643bc79 799fe50b 64d8c7f1 03c0e451

      05:39:52.832 localhost-1: 05:39:52.801 0x080*  44a297ce 38379913 21077d2d 2b434700   04c00398 2e32e509 6d4de30b 1e22e1cd

      05:39:52.832 localhost-1: 05:39:52.801 0x090*  586234e6 53a805bd 499aae74 28becf56   60315c36 047cd55f 02dc99d8 7d9f2108

      05:39:52.833 localhost-1: 05:39:52.801 0x0a0*  66011e2f 5a617593 12157ae2 52657d6f   7be243fd 7a27f6ba 3c087cab 7a67b234

      05:39:52.833 localhost-1: 05:39:52.801 0x0b0*  003df3dd 1acbff3e 57efc031 625dbe41   4a82d6c7 26803bae 4256a507 04689e13

      05:39:52.833 localhost-1: 05:39:52.801 0x0c0*  3d61cb20 086bcaae 2096de9a 22601bfc   28812640 340d2896 4d2d2fa5 1bb78a5a

      05:39:52.833 localhost-1: 05:39:52.801 0x0d0*  4e850c9a 4fcebabb 267a3c48 1aef2913   363e16db 48ef7b8e 30cfbe73 2b3ba606

      05:39:52.833 localhost-1: 05:39:52.801 0x0e0*  24b9da3e 14eedba9 5fcdcec6 4a88b639   2e27c659 37b39da3 351202df 694edda1

      05:39:52.833 localhost-1: 05:39:52.801 0x0f0*  1844fdb7 1ada9cfd 6cbbadca 1fb14d77   6ae27aaa 7769d9c0 61795091 39395743

      05:39:52.834 localhost-1: 05:39:52.801 0x100*  022e5eef 3c93d081 78ddacb8 0bdaf6f4   4dc91796 0af55e21 0aa63704 238836f5

      05:39:52.834 localhost-1: 05:39:52.802 0x110*  189201d1 4c36739f 5df7fd7f 644844e4   162e1d13 2ca303f5 2eff2471 5a60ab06

      05:39:52.834 localhost-1: 05:39:52.802 0x120*  223a8d0c 1e983eda 3c0ddb10 692ca831   7de62dbd 4438df92 16b50f48 62104034

      05:39:52.834 localhost-1: 05:39:52.802 0x130*  3e2e57c4 3eb1fdde 45093aa5 326b0be3   17c51ccd 62cf990a 355b2939 5d3b7dd9

      05:39:52.834 localhost-1: 05:39:52.802 0x140*  07f80064 7d43b09f 284e8735 6c73ba88   78d684a5 38e68c2f 796a98ad 272bca3c

      05:39:52.834 localhost-1: 05:39:52.802 0x150*  51938a9e 4e44734f 5cb56272 3ee289c3   253ae4d5 3843b0e2 082b3c15 55dd96e3

      05:39:52.835 localhost-1: 05:39:52.802 0x160*  00090536 26cf2dc6 7d7f7021 699e4e78   0db24847 54504b9a 5a9a72a9 708fd50b

      05:39:52.835 localhost-1: 05:39:52.802 0x170*  4fddb91e 535d0d4d 09062801 1c5a994a   22fd740d 18c364d6 4a08f055 1eb5b151

      05:39:52.835 localhost-1: 05:39:52.802 0x180*  62a4df2d 073ec308 0df8afaa 477cbafa   205e9cfc 178b0a76 59489670 4e4b983b

      05:39:52.835 localhost-1: 05:39:52.802 0x190*  7718deb9 33658de6 3622bd09 24851050   2ea2a5ce 7c0556ea 69635777 4fb08294

      05:39:52.835 localhost-1: 05:39:52.802 0x1a0*  114612e9 03ad0a08 23d580ff 773d9665   653be697 62ba429c 0ef41fb8 284489bc

      05:39:52.835 localhost-1: 05:39:52.802 0x1b0*  20701a6a 36f6c8ca 412b98e1 100ec798   7717797d 69e73aa5 675ac8fe 642dc3b7

      05:39:52.835 localhost-1: 05:39:52.802 0x1c0*  5c7aac8a 5f88b1a3 6e8109d4 62fbbe79   04f611b1 484d82e7 1b2413e4 7ac46723

      05:39:52.835 localhost-1: 05:39:52.802 0x1d0*  24bef7fc 763f2a36 6f984198 15e6a327   1f315a28 14d45c9c 2d1eaab3 68473890

      05:39:52.835 localhost-1: 05:39:52.803 0x1e0*  1e5688cc 0c440637 0d0521e3 3bde5f75   74f6daf6 4aaff786 4bb62ae1 72993d54

      05:39:52.835 localhost-1: 05:39:52.803 0x1f0*  2804b6d4 4c1b3ff2 199712b7 0a569a51   072167b0 7ade2a31 144d3848 25e9b890

      05:39:52.835 localhost-1: 05:39:52.808

      05:39:52.835 localhost-1: 05:39:52.808         Data Validation error for sd=sd2,lun=/dev/mapper/mpathgn

      05:39:52.835 localhost-1: 05:39:52.808         Block lba: 0x1c094c000; sector lba: 0x1c094c800; Key block size: 4096; relative sector in data block: 0x04 ( 4); current pid: 4524 (0x11ac)

      05:39:52.836 localhost-1: 05:39:52.808         ===> Data Validation Key miscompare. Expecting key=0x0b received key=0x04 (11/4)

      05:39:52.836 localhost-1: 05:39:52.808         ===> Logical byte address miscompare. Expecting 0x1c094c800, receiving 0x1c08ea800

      05:39:52.836 localhost-1: 05:39:52.808         ===> SD or FSD name miscompare. Expecting 'sd2     ', receiving 'sd1     ' (0x2031647320202020)

      05:39:52.836 localhost-1: 05:39:52.809         ===> Compression pattern miscompare.

      05:39:52.836 localhost-1: 05:39:52.809         SD or FSD name in block expected: 'sd2'; received: 'sd1     '.

      05:39:52.836 localhost-1: 05:39:52.809         Timestamp found in sector: Fri Nov 08 2019 05:22:27.470 CST

      05:39:52.836 localhost-1: 05:39:52.809 0x000*  00000001 c094c800 ........ ........   00000001 c08ea800 0000016e 4ac04f0e

      05:39:52.836 localhost-1: 05:39:52.809 0x010   0b..0000 20326473 20202020 00000000   04d60000 20316473 20202020 00001199

      05:39:52.836 localhost-1: 05:39:52.809 0x020*  310c3b0b 6f702763 3afe89dc 32a97a86   2243f98f 5eb46fb4 384fcafa 21cdc102

      05:39:52.836 localhost-1: 05:39:52.809 0x030*  4c060aeb 79d89423 45e37c1e 40424762   56f4bbdc 043d54d7 3ada258e 1bf0c5d3

      05:39:52.836 localhost-1: 05:39:52.809 0x040*  0e8b62bd 783f65e6 20020ba7 193a2182   20f960cd 0c7ea4d9 6f25463d 667b5b8d

      05:39:52.837 localhost-1: 05:39:52.809 0x050*  0ddc2c15 68725baa 579a02db 1c769365   016e4e2a 4f8cfab1 2f193357 7aaba91c

      05:39:52.837 localhost-1: 05:39:52.810 0x060*  294f0006 472844ae 69e61138 7f7dab33   1c1a192b 3be581bf 61e45894 0857d85f

      05:39:52.837 localhost-1: 05:39:52.810 0x070*  7ff2d5dd 781ff91b 45821e84 21a436bd   5bff34b7 5efcee29 2c51d9ac 008c5319

      05:39:52.837 localhost-1: 05:39:52.810 0x080*  74e0383b 0f8e1986 49f1a4b5 71af47ae   149d27db 6d1d3c4c 6c3fec60 309d4740

      05:39:52.837 localhost-1: 05:39:52.810 0x090*  45a30489 7cea97cc 753f11c8 360a0028   634a4ba1 08a33a50 702f3aff 7dcc72e4

      05:39:52.837 localhost-1: 05:39:52.810 0x0a0*  5a45b07c 284f072b 36cd32cf 0ad0a68f   382f11ec 19ccd70f 22044e7e 54c4383e

      05:39:52.837 localhost-1: 05:39:52.810 0x0b0*  7ea7185c 28b6b435 4893a53c 301201d5   768bd33a 1d693b82 46340000 2748fa49

      05:39:52.837 localhost-1: 05:39:52.810 0x0c0*  20e3d5f4 52918578 7e1f51f5 4dc7fe05   081224e9 69dbaa25 7eb3f4df 5da1c14e

      05:39:52.837 localhost-1: 05:39:52.811 0x0d0*  00bc1dd2 7b3aa3b1 2acfef47 22289878   0f3ab7eb 07022770 73281f48 7059264f

      05:39:52.837 localhost-1: 05:39:52.811 0x0e0*  6fff2659 6f03e9b0 16eb8b1e 221013c3   18a0be4f 56b0ea78 55e648ee 7217883b

      05:39:52.837 localhost-1: 05:39:52.811 0x0f0*  5c251564 3ca71ce5 75411d05 6a657468   438bda45 709394a7 7788a5a1 28778735

      05:39:52.837 localhost-1: 05:39:52.811 0x100*  5560772b 220dedcd 69332118 7c7a44bc   55f193a2 3d653244 38e86d48 6dd5c99c

      05:39:52.837 localhost-1: 05:39:52.811 0x110*  27903bc5 1381bfba 6f233952 1e86a11a   105756e0 0a9002d4 6d797e0d 753d07e5

      05:39:52.838 localhost-1: 05:39:52.811 0x120*  5d9b3e3e 07743950 5f57d8ce 5c828c76   5a1dfabd 006608e1 262006f8 2caf614e

      05:39:52.838 localhost-1: 05:39:52.811 0x130*  2fde51c1 66319ccb 4d94f55a 0cbd8c91   06dfcead 1a4362f4 66b9399f 612df74f

      05:39:52.838 localhost-1: 05:39:52.811 0x140*  6a75d959 35ede878 35c0c1bc 66b58e1b   01aa315c 790b6ac5 76b0053a 72ffd293

      05:39:52.838 localhost-1: 05:39:52.812 0x150*  484be470 58bdbe6f 380d7026 29ce1094   1afbae8b 25919ee4 4f16d782 57d90eb0

      05:39:52.838 localhost-1: 05:39:52.812 0x160*  13f3d66e 732c150e 1d40fca4 04a89b8d   0a8fa592 199f5192 1edf6909 5ba34f1b

      05:39:52.838 localhost-1: 05:39:52.812 0x170*  177c9482 573daf6d 75d3ff2c 64ff79d0   6e207760 2b6721fc 77e98d30 07c87ab7

      05:39:52.838 localhost-1: 05:39:52.812 0x180*  00823bc6 18d35d90 64c0855f 1303be46   2a693d57 40573bc8 75c20e15 5400c045

      05:39:52.838 localhost-1: 05:39:52.812 0x190*  5de85f87 3d7c0450 08f9f5ca 7a5f8fe8   13d0068e 795b602a 110a8e09 42c4e32d

      05:39:52.838 localhost-1: 05:39:52.812 0x1a0*  40c5cf96 4ae76def 7ea6f94b 461dbe73   28419f0d 5bbfb54b 58a371cc 4ea5ceb2

      05:39:52.838 localhost-1: 05:39:52.812 0x1b0*  522ae685 4c4ae25a 1b566db9 7bd79e13   6be7dd47 4f055f27 73d0d36e 14ea72fb

      05:39:52.838 localhost-1: 05:39:52.812 0x1c0*  5b8b6017 4028870c 56885fe7 0daa2bbe   26837911 2119df21 1eb37be4 74f4e975

      05:39:52.838 localhost-1: 05:39:52.813 0x1d0*  4fb4b6ff 3d0a87eb 42040e7c 699379fc   4f52690c 1f823d23 448ee571 6b0fe03c

      05:39:52.838 localhost-1: 05:39:52.813 0x1e0*  565a0bdc 7ce30704 0b93f32b 5ff96467   698bc9d0 352af744 71754603 40654f23

      05:39:52.839 localhost-1: 05:39:52.813 0x1f0*  345e9292 63e1d4e6 3be0d614 523a8cef   6b9a4b47 61adba14 0f12a181 40ff73f3

      05:39:52.839 localhost-1: 05:39:52.818

      05:39:52.839 localhost-1: 05:39:52.818         Data Validation error for sd=sd2,lun=/dev/mapper/mpathgn

      05:39:52.839 localhost-1: 05:39:52.819         Block lba: 0x1c094c000; sector lba: 0x1c094ca00; Key block size: 4096; relative sector in data block: 0x05 ( 5); current pid: 4524 (0x11ac)

      05:39:52.839 localhost-1: 05:39:52.819         ===> Data Validation Key miscompare. Expecting key=0x0b received key=0x04 (11/4)

      05:39:52.839 localhost-1: 05:39:52.819         ===> Logical byte address miscompare. Expecting 0x1c094ca00, receiving 0x1c08eaa00

      05:39:52.839 localhost-1: 05:39:52.819         ===> SD or FSD name miscompare. Expecting 'sd2     ', receiving 'sd1     ' (0x2031647320202020)

      05:39:52.839 localhost-1: 05:39:52.819         ===> Compression pattern miscompare.

      05:39:52.840 localhost-1: 05:39:52.819         SD or FSD name in block expected: 'sd2'; received: 'sd1     '.

      05:39:52.841 localhost-1: 05:39:52.819         Timestamp found in sector: Fri Nov 08 2019 05:22:27.470 CST

      05:39:52.841 localhost-1: 05:39:52.819 0x000*  00000001 c094ca00 ........ ........   00000001 c08eaa00 0000016e 4ac04f0e

      05:39:52.841 localhost-1: 05:39:52.819 0x010   0b..0000 20326473 20202020 00000000   04d60000 20316473 20202020 00001199

      05:39:52.841 localhost-1: 05:39:52.819 0x020*  1bae6f87 07ad7543 6fd53139 1bafeec9   0cfb0a1a 4416ac72 16a12c39 0d6d3220

      05:39:52.841 localhost-1: 05:39:52.819 0x030*  3420258a 028538a5 0d7d2f42 73660c27   1daf4e4a 1569e42c 1217eeb2 563c4371

      05:39:52.841 localhost-1: 05:39:52.819 0x040*  62e67d46 1f9a7949 69f251fb 6c045122   45d80f3a 51981f2e 70327e7e 2997800c

      05:39:52.841 localhost-1: 05:39:52.820 0x050*  0b7becbe 5438fc62 5a63736a 6b207280   183c3eda 5123fe63 5c37f1d3 2f5d38e2

      05:39:52.841 localhost-1: 05:39:52.820 0x060*  051ee144 71781fa6 15341ba5 4873b0a0   71fa465c 0ddb3ba0 5eef8ce7 6f86cab3

      05:39:52.841 localhost-1: 05:39:52.820 0x070*  4dd46e6c 0322d159 46d3d2db 1b6213a2   1c7ab52b 622ea139 616813d4 171fd908

      05:39:52.841 localhost-1: 05:39:52.820 0x080*  373a830a 400e5028 20d57abf 2d14efbb   57327665 77f8c9c6 77ff246f 5fd0d42e

      05:39:52.841 localhost-1: 05:39:52.820 0x090*  5492a8a7 6ff0b460 183fb1f9 0349b0a2   509d8fb4 0ac4a9ac 6aa1991d 4e1dc44b

      05:39:52.842 localhost-1: 05:39:52.820 0x0a0*  158e4976 3be83f7c 4e2f3d82 7b15ef7a   39fb66fa 32c4de5c 355df307 7d9a5e10

      05:39:52.842 localhost-1: 05:39:52.820 0x0b0*  01857def 74047430 08047b9c 3201cd59   297590c7 02ccc67b 2681cc5b 79745ea6

      05:39:52.842 localhost-1: 05:39:52.820 0x0c0*  72e1640d 6d41f76e 2a74dcdf 06f99878   6a7a270f 5cbc62fb 4d9b51a6 2ad5308b

      05:39:52.843 localhost-1: 05:39:52.820 0x0d0*  02a2fca4 388a380d 5cf96819 4bb7b87c   2c32a6ad 7ceffed8 20d7c5e7 35bbd723

      05:39:52.843 localhost-1: 05:39:52.820 0x0e0*  080f925a 1467df53 5133435e 5267e25a   3220563a 5e649f07 14580b07 3010236b

      05:39:52.843 localhost-1: 05:39:52.820 0x0f0*  28aba728 204108ca 23ff468c 5a649498   430733f1 3e623b73 59343510 75d0300c

      05:39:52.843 localhost-1: 05:39:52.821 0x100*  70cd6bda 0108433f 6c1195e4 7a0fdba4   0e706c24 2f9cbe98 18072413 7859257b

      05:39:52.843 localhost-1: 05:39:52.821 0x110*  4d2fbc61 5475580a 4c1243e6 02e85ea1   212d576c 59dcce3d 3cfb2694 45ec4680

      05:39:52.843 localhost-1: 05:39:52.821 0x120*  020ea831 00f6dc73 4453fba5 001bb28c   456400a4 34615b16 5170a3ab 29b14446

      05:39:52.844 localhost-1: 05:39:52.821 0x130*  64767f0d 25ad88c6 0a5db322 56f6f75e   52ce33a2 3e4af9d2 6977cd6b 2b96daf8

      05:39:52.844 localhost-1: 05:39:52.821 0x140*  268e6785 26e2595f 5aa065b5 3bb6bbc9   7ff4136b 718202db 59765b0a 6292b530

      05:39:52.844 localhost-1: 05:39:52.821 0x150*  77c77df0 108876fe 0ad1d5ae 1c572a06   354b37fe 69a87a54 3a724af8 7ae198df

      05:39:52.844 localhost-1: 05:39:52.821 0x160*  0df979a9 497be5e8 19855729 56257b65   37435496 79c8c146 399aad78 0508dbfa

      05:39:52.845 localhost-1: 05:39:52.821 0x170*  6495dc25 574dff88 11bba28a 01f3b569   346966e0 30920cfe 49c77667 197a1f23

      05:39:52.845 localhost-1: 05:39:52.821 0x180*  1b3aa6cb 3b90b251 5742adb5 5defacee   60ea6c3e 1413a762 0bc9585a 17da53de

      05:39:52.845 localhost-1: 05:39:52.821 0x190*  5d0d9919 40b18903 52de8f53 5b3e535b   48dea010 2b07f7d8 2482a57b 480d074f

      05:39:52.845 localhost-1: 05:39:52.821 0x1a0*  1e68f702 5d957c88 7b546898 189c4a3f   66e6427e 3ae052c0 3de23b06 6644f745

      05:39:52.845 localhost-1: 05:39:52.821 0x1b0*  4682d7b3 2c468ac9 629057a3 1a64c09e   54c0e66b 76e29277 19f9734e 2a7aee7c

      05:39:52.846 localhost-1: 05:39:52.821 0x1c0*  06d7fae1 492417b8 3c3f719a 5b7f954f   17a847de 35c97680 6d861f67 26cbe7c7

      05:39:52.846 localhost-1: 05:39:52.821 0x1d0*  324327aa 4d27150e 4556c520 61d7c520   1a57a9bd 2078cd3c 296989e2 675f7c4d

      05:39:52.846 localhost-1: 05:39:52.821 0x1e0*  59ca12b8 76eaf38c 0a206479 541b1e54   5ac8bc8f 2d7662b3 70a58124 119795f8

      05:39:52.846 localhost-1: 05:39:52.821 0x1f0*  4d48cc64 7b4c59c5 428777a4 25a43c6b   29cf0d30 21f8b9dc 3298692d 576a6c76

      05:39:52.846 localhost-1: 05:39:52.828

      05:39:52.846 localhost-1: 05:39:52.828         Data Validation error for sd=sd2,lun=/dev/mapper/mpathgn

      05:39:52.846 localhost-1: 05:39:52.828         Block lba: 0x1c094c000; sector lba: 0x1c094cc00; Key block size: 4096; relative sector in data block: 0x06 ( 6); current pid: 4524 (0x11ac)

      05:39:52.846 localhost-1: 05:39:52.829         ===> Data Validation Key miscompare. Expecting key=0x0b received key=0x04 (11/4)

      05:39:52.846 localhost-1: 05:39:52.829         ===> Logical byte address miscompare. Expecting 0x1c094cc00, receiving 0x1c08eac00

      05:39:52.847 localhost-1: 05:39:52.829         ===> SD or FSD name miscompare. Expecting 'sd2     ', receiving 'sd1     ' (0x2031647320202020)

      05:39:52.847 localhost-1: 05:39:52.829         ===> Compression pattern miscompare.

      05:39:52.847 localhost-1: 05:39:52.829         SD or FSD name in block expected: 'sd2'; received: 'sd1     '.

      05:39:52.847 localhost-1: 05:39:52.829         Timestamp found in sector: Fri Nov 08 2019 05:22:27.470 CST

      05:39:52.847 localhost-1: 05:39:52.829 0x000*  00000001 c094cc00 ........ ........   00000001 c08eac00 0000016e 4ac04f0e

      05:39:52.847 localhost-1: 05:39:52.829 0x010   0b..0000 20326473 20202020 00000000   04d60000 20316473 20202020 00001199

      05:39:52.848 localhost-1: 05:39:52.829 0x020*  6e86c10a 37fd6fc7 29d3a175 78a1e8c6   54e3b77c 0fb315c7 134dd676 4617a908

      05:39:52.849 localhost-1: 05:39:52.829 0x030*  31ce574c 4296ae2f 005a09f1 255d7efc   42e977e2 7992c481 50bc5eff 1ceaef0f

      05:39:52.849 localhost-1: 05:39:52.829 0x040*  6fa0b125 09b4c9eb 2776a5ca 67cf1f08   4df1571b 007856bc 31e74410 75b3c2c9

      05:39:52.849 localhost-1: 05:39:52.829 0x050*  674bd113 07e28ed7 69705fc4 350da835   6805d2f5 4b157392 219dac0c 6fc19f8c

      05:39:52.849 localhost-1: 05:39:52.829 0x060*  2faef89b 0778e873 201c9824 78f68d1d   2f562947 43696355 7488b36e 1d2213c7

      05:39:52.849 localhost-1: 05:39:52.830 0x070*  4b9e5719 3c4fdb74 40277756 620e0fc3   318a05a9 50e20636 2263be38 05f0fe04

      05:39:52.849 localhost-1: 05:39:52.830 0x080*  23364025 1efe44e2 5bbc4757 6412c6ed   0204b795 720b958d 4494d2d0 605c1046

      05:39:52.850 localhost-1: 05:39:52.830 0x090*  6d86e998 51b96369 5586d6d4 096568db   6e80f069 3de02b81 531dfbda 557a9d2a

      05:39:52.850 localhost-1: 05:39:52.830 0x0a0*  3a21b17f 68052678 4326e36f 2e85e036   418e0abe 65e81400 5f00b2c3 2e3aabb4

      05:39:52.850 localhost-1: 05:39:52.830 0x0b0*  5230ecd0 7f864709 684e8b06 0a06a8de   0a97d76c 677be42f 734c51a7 0ac4530e

      05:39:52.850 localhost-1: 05:39:52.830 0x0c0*  14e33d82 3167288e 41e4c74d 6e777332   4f41f47a 33995b06 54308e1a 51d46034

      05:39:52.850 localhost-1: 05:39:52.830 0x0d0*  43137120 00e07fe9 4aa4eeb3 02e8111c   59ffab10 31dc0980 058cfa7c 54af40b1

      05:39:52.850 localhost-1: 05:39:52.830 0x0e0*  6235e63d 0c96c4af 78d43f89 41c4a983   0df155ee 244ac340 58969137 376d37bb

      05:39:52.850 localhost-1: 05:39:52.830 0x0f0*  76b47b32 6d9d68ad 6ec67269 64ec35aa   40304fc0 16cad625 3a4c46a4 0480da03

      05:39:52.851 localhost-1: 05:39:52.830 0x100*  520204d0 5810d8ce 760a535e 01024409   12d2691f 2e2a0b01 1a93a351 6d78d2b5

      05:39:52.851 localhost-1: 05:39:52.830 0x110*  10ce4bb7 3e5844e4 1f376543 711d05ee   2e0de68a 57d06de4 05f665a7 1a1dee81

      05:39:52.851 localhost-1: 05:39:52.830 0x120*  29e39339 69403051 46758b8c 5710a281   7807077a 0c4addce 721a2d43 0a2e7131

      05:39:52.851 localhost-1: 05:39:52.831 0x130*  1656a759 2c3ec9a5 46f7bc4e 5bb8d43e   13f013e2 0bff659c 472b2c6a 7517b0b9

      05:39:52.851 localhost-1: 05:39:52.831 0x140*  27d27337 16530ba4 55c2bb2f 05c1cbf5   7d44f2f7 367c5bcb 5e92e275 3e2efa17

      05:39:52.851 localhost-1: 05:39:52.831 0x150*  11e81f1f 3d0fa4f6 4e38dafa 6c915de0   040b48ee 59f34b2e 0173de01 030abdcd

      05:39:52.851 localhost-1: 05:39:52.831 0x160*  7f6cb6cd 0082a870 3103c7a1 40117622   43201d8c 1990a2a3 678387d0 7cbd038b

      05:39:52.852 localhost-1: 05:39:52.831 0x170*  450d57e4 6bcb05a0 30155a23 153cd406   61bc0a86 4e812dfa 50faf3fe 63d026bd

      05:39:52.852 localhost-1: 05:39:52.831 0x180*  5e62e824 1f4cc92e 6098706b 226f0ec2   0d76f16a 10965534 1477bce5 4206dab7

      05:39:52.852 localhost-1: 05:39:52.831 0x190*  62ee5324 7e952452 0dcad192 1598e235   1a343a0d 609a25e4 5bc5e592 1c00771c

      05:39:52.852 localhost-1: 05:39:52.831 0x1a0*  04865161 2d1f3dfb 2dd916ed 7199c1fe   198098b0 3f842311 50abc358 20028bef

      05:39:52.852 localhost-1: 05:39:52.831 0x1b0*  70dfb600 3a4aca2c 23f59edb 671fb723   030a5cb9 52dfdca9 24f0f06b 0e48bddf

      05:39:52.852 localhost-1: 05:39:52.831 0x1c0*  1bf9451e 35f56b7a 490386bf 688e832d   33fd45e1 0fd65128 2ef94669 5a5beabc

      05:39:52.852 localhost-1: 05:39:52.831 0x1d0*  095411f1 4447d9ba 264ead9e 478f0d1b   1b2ae3d4 436b14ba 466ce057 14119500

      05:39:52.853 localhost-1: 05:39:52.831 0x1e0*  1e01659a 2867d3c1 63e9bdb3 7e08150c   5c087b40 2f9e41d4 22862451 79f49987

      05:39:52.853 localhost-1: 05:39:52.832 0x1f0*  5daf2c9a 22dfd68d 266b3fbe 5b613124   0aab751a 5a44317f 35bed6fd 2df829ef

      05:39:52.853 localhost-1: 05:39:52.837

      05:39:52.853 localhost-1: 05:39:52.837         Data Validation error for sd=sd2,lun=/dev/mapper/mpathgn

      05:39:52.853 localhost-1: 05:39:52.837         Block lba: 0x1c094c000; sector lba: 0x1c094ce00; Key block size: 4096; relative sector in data block: 0x07 ( 7); current pid: 4524 (0x11ac)

      05:39:52.853 localhost-1: 05:39:52.838         ===> Data Validation Key miscompare. Expecting key=0x0b received key=0x04 (11/4)

      05:39:52.854 localhost-1: 05:39:52.838         ===> Logical byte address miscompare. Expecting 0x1c094ce00, receiving 0x1c08eae00

      05:39:52.854 localhost-1: 05:39:52.838         ===> SD or FSD name miscompare. Expecting 'sd2     ', receiving 'sd1     ' (0x2031647320202020)

      05:39:52.854 localhost-1: 05:39:52.838         ===> Compression pattern miscompare.

      05:39:52.854 localhost-1: 05:39:52.839         SD or FSD name in block expected: 'sd2'; received: 'sd1     '.

      05:39:52.854 localhost-1: 05:39:52.839         Timestamp found in sector: Fri Nov 08 2019 05:22:27.470 CST

      05:39:52.854 localhost-1: 05:39:52.839 0x000*  00000001 c094ce00 ........ ........   00000001 c08eae00 0000016e 4ac04f0e

      05:39:52.854 localhost-1: 05:39:52.839 0x010   0b..0000 20326473 20202020 00000000   04d60000 20316473 20202020 00001199

      05:39:52.854 localhost-1: 05:39:52.839 0x020*  0bf3f76e 0d101f0d 7d7d22d3 5427f0b5   7a5cc9a1 1c39b3fc 40d9117f 0475add8

      05:39:52.854 localhost-1: 05:39:52.839 0x030*  3980682a 59b5fc90 351e53b1 0a5a27c9   191300a0 5e60fdc3 4b6bbeba 122e50c5

      05:39:52.854 localhost-1: 05:39:52.839 0x040*  0c5ac682 6f505fc5 34744f13 32751348   2ee60027 1fe15346 0e28df3b 1eaaabe1

      05:39:52.855 localhost-1: 05:39:52.839 0x050*  313da10e 7f6a1a74 2d070ff5 3de1bc6a   5ebcd2d6 475c6228 6590aa04 6d7c64b6

      05:39:52.855 localhost-1: 05:39:52.839 0x060*  4a620d9f 167ea6d4 4a66cedb 0292c82c   13908a4f 37e40091 607713d8 30b63bfc

      05:39:52.855 localhost-1: 05:39:52.839 0x070*  094457f1 34741f9d 17e55400 093db305   0920ee61 49e4252c 13ad20dc 4e954a04

      05:39:52.855 localhost-1: 05:39:52.839 0x080*  187837b7 56c3fff9 5acf5257 2467559f   12b8b2e3 53bba773 302a3fa5 6dfd83c7

      05:39:52.855 localhost-1: 05:39:52.839 0x090*  24f58f16 2063ace6 1e1ec889 0c64b079   2f6a35dd 2e14c7ea 58aeab7e 45ea85de

      05:39:52.855 localhost-1: 05:39:52.839 0x0a0*  7a66b043 5c9dc407 72bf6c3c 3f9100d0   1f4dc573 312e80b0 79f7d565 2f15a91c

      05:39:52.855 localhost-1: 05:39:52.839 0x0b0*  2e212cac 2f2d34fd 519e1bfc 2d1a1c79   356a6f34 3d679cfb 52bfd87f 3e325f8c

      05:39:52.855 localhost-1: 05:39:52.839 0x0c0*  40922a74 5d0b20eb 1adc5937 3be4165a   0e12553e 7a7c9a25 66e0f1da 3841d835

      05:39:52.855 localhost-1: 05:39:52.839 0x0d0*  6d07432d 36808341 5fa0caf9 0e252a33   219b8d1e 16094fc0 4b160485 019eeaae

      05:39:52.855 localhost-1: 05:39:52.840 0x0e0*  13dce9e2 4c2ca1d1 7a1dc7ef 757af1a4   5f7e8597 6aff5f3a 0cf12358 3b834cb7

      05:39:52.856 localhost-1: 05:39:52.840 0x0f0*  433b5958 3dd1448c 7186e865 6259dfe0   1602f5ca 20e26cf5 74c122bd 1ae70d30

      05:39:52.856 localhost-1: 05:39:52.840 0x100*  5eab0a26 1ad5b6bb 45cea15c 34d805c3   66c4525e 7abb1fcc 7d3f3301 1ebb5947

      05:39:52.856 localhost-1: 05:39:52.840 0x110*  45e9b182 79918e52 0124e55a 33f41eaf   6876cc56 3ad1ea0f 1efd83af 2aa18848

      05:39:52.856 localhost-1: 05:39:52.840 0x120*  1f88c6fc 1b903cfb 534fd05b 1399e45a   5a75d737 31629947 73afec01 2e5f6fd6

      05:39:52.856 localhost-1: 05:39:52.840 0x130*  13fe92cd 2a1e6795 051758c5 58c4ab13   76c53733 4199ae59 3f879eb7 6972006d

      05:39:52.856 localhost-1: 05:39:52.840 0x140*  51f2c438 4f92070e 401d0a43 1641464e   7b4d97da 614c7e0c 1cfae35e 053f2960

      05:39:52.856 localhost-1: 05:39:52.840 0x150*  24af8fd5 2ede507d 7998c7d9 6bb03491   733da997 54fd1992 5971d8e2 2f88053c

      05:39:52.856 localhost-1: 05:39:52.840 0x160*  39c055c9 7c2464b6 4493964e 638913c8   1d98c873 2adafdc6 0771403e 11dc4dfd

      05:39:52.856 localhost-1: 05:39:52.840 0x170*  6ae6cfc3 3e11c9db 6a596e0b 44005dc3   061c2a5d 3c918cd4 64fc4da1 79f01958

      05:39:52.857 localhost-1: 05:39:52.840 0x180*  7dafc7e5 08fdaa5a 65fb15de 21d06c0e   01c3951f 48d54d3b 7306841d 41d4dcc6

      05:39:52.857 localhost-1: 05:39:52.840 0x190*  4a105610 51d5de0e 41d9049f 0306c49f   40569cab 02c0f29e 7cee9662 0636ba54

      05:39:52.857 localhost-1: 05:39:52.840 0x1a0*  1d94a6dd 480cb9f8 5c504c58 2316ad63   088769e0 46332dfe 551b5279 7bdf1426

      05:39:52.857 localhost-1: 05:39:52.840 0x1b0*  78c9499c 7ad8a892 42c28b50 709209cf   7c4c07df 757e45e8 35f392b4 3cdd6960

      05:39:52.857 localhost-1: 05:39:52.840 0x1c0*  70a806a8 67368a4d 2051e77a 28097d6c   6f3b3b44 5dda771e 048a38f7 30d779cf

      05:39:52.857 localhost-1: 05:39:52.840 0x1d0*  0df13df1 693fddd6 2fca0fd2 7eb4d988   68d5df8c 5d2c1ba4 24773049 4321b265

      05:39:52.857 localhost-1: 05:39:52.840 0x1e0*  787acc44 6c85afb2 364f4718 22a4d093   60c5cdf9 64ce9c83 22767772 6c60e1fb

      05:39:52.857 localhost-1: 05:39:52.841 0x1f0*  149d7ab3 1c415363 468c75c5 6d5ef2e8   1b3b4b06 5a352923 358632ef 2c96349e

      05:39:52.858 localhost-1: 05:39:52.846 05:39:52.845 op: read   lun: /dev/mapper/mpathgn            lba:   7525941248 0x1C094C000 xfer:     4096 errno: 60003: '60003            A Data Validation error was discovered'

      root@ubuntu:/vdbench#