Help me locate the VM owner of (corrupt) VDI - also 'primary footer invalid: invalid cookie' error

Discussion:

France

2015-12-07 22:14:35 UTC

Hi Guys,

after battling with cloudstack (4.3.2+) for the whole day with another issue (unable to create snapshots for one VM, which I also failed to fix) I ran into message on XenServer 6.0.2+Hotfixes SMlog (copy pasted at the bottom).

Even thou i kinda understand XenServers relations of VM -> VBD -> VDI -> SR -> PBD, i failed to find out to which VM does this corrupted .vhd snapshot belong.
Can you please help me find it? Afterwards, we can find the appropriate action in order to remove the error and make coalesce work.

The physical file with corruption is:
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba

This is the VDI it supposedly belongs to:
[***@x3 log]# xe vdi-list uuid=02e8b56b-279c-4d5b-8870-b3c5f2255dc7 params=all
uuid ( RO) : 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
name-label ( RW): base copy
name-description ( RW):
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshots ( RO):
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): generate_config; update; resize; destroy; clone; copy; snapshot
current-operations (SRO):
sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
sr-name-label ( RO): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
vbd-uuids (SRO):
crashdump-uuids (SRO):
virtual-size ( RO): 21474836480
physical-utilisation ( RO): 12582912
location ( RO): 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
type ( RO): User
sharable ( RO): false
read-only ( RO): true
storage-lock ( RO): false
managed ( RO): false
parent ( RO): <not in database>
missing ( RO): false
other-config (MRW):
xenstore-data (MRO):
sm-config (MRO): vhd-parent: 555b38bf-fe63-41b2-9609-e16f4e26b274; vdi_type: vhd; vhd-blocks: eJxjYBgFo2AUjIKRCQAFAAAB
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
tags (SRW):

(It supposedly resides on this SR:
uuid ( RO) : 3b6e386d-8736-6b6b-7006-f3f4df9bd586
name-label ( RW): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
name-description ( RW): 208 - censored
host ( RO): <shared>
type ( RO): lvmoiscsi
content-type ( RO): user

Which is on one of many HA iscsi targets mounted on the XenServer cluster.
uuid ( RO) : f471ef16-4670-2f5e-967e-7e98d4721fe7
host-uuid ( RO): f6d699aa-f58f-449e-9a51-6e965562f178
sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
device-config (MRO): targetIQN: censoredi:storage.censored; target: censored; SCSIid: 1target2lun1
currently-attached ( RO): true)

Cleaned exempt from SMlog on cluster master:

[8756] 2015-12-07 21:50:45.118758 ['/usr/sbin/vhd-util', 'check', '--debug', '-n', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
[8756] 2015-12-07 21:50:45.131975 FAILED: (rc 22) stdout: 'primary footer invalid: invalid cookie
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba appears invalid; dumping metadata
VHD Footer Summary:
-------------------
Cookie : conectix
Features : (0x00000002) <RESV>
File format version : Major: 1, Minor: 0
Data offset : 512
Timestamp : Mon Dec 7 18:19:46 2015
Creator Application : 'tap'
Creator version : Major: 1, Minor: 3
Creator OS : Unknown!
Original disk size : 20480 MB (21474836480 Bytes)
Current disk size : 20480 MB (21474836480 Bytes)
Geometry : Cyl: 41610, Hds: 16, Sctrs: 63
: = 20479 MB (21474754560 Bytes)
Disk type : Differencing hard disk
Checksum : 0xffffef85|0xffffef85 (Good!)
UUID : 8f25299d-d7cc-44fe-a42b-95b4e9a31a47
Saved state : No
Hidden : 1

VHD Header Summary:
-------------------
Cookie : cxsparse
Data offset (unusd) : 18446744073709
Table offset : 1536
Header version : 0x00010000
Max BAT size : 1048576
Block size : 2097152 (2 MB)
Parent name : VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
Parent UUID : 8cece4c9-8c94-47b9-9f7a-cb6023da9b72
Parent timestamp : Mon Dec 7 18:19:46 2015
Checksum : 0xffffc6ed|0xffffc6ed (Good!)

VHD Parent Locators:
--------------------
locator: : 0
code : PLAT_CODE_MACX
data_space : 512
data_length : 110
data_offset : 4327424
decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7

locator: : 1
code : PLAT_CODE_W2KU
data_space : 512
data_length : 206
data_offset : 4327936
decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7

locator: : 2
code : PLAT_CODE_W2RU
data_space : 512
data_length : 206
data_offset : 4328448
decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7

VHD Batmap Summary:
-------------------
Batmap offset : 4196352
Batmap size (secs) : 256
Batmap version : 0x00010002
Checksum : 0xffffff7f|0xffffff7f (Good!)

', stderr: ''
[8756] 2015-12-07 21:50:45.132293 ['/usr/sbin/vhd-util', 'query', '--debug', '-s', '-n', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
[8756] 2015-12-07 21:50:45.145222 SUCCESS
[8756] 2015-12-07 21:50:45.145502 lock: tried lock /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr, acquired: True (exists: True)
[8756] 2015-12-07 21:50:45.145635 lock: released /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr
[8756] 2015-12-07 21:50:45.145752 ['/usr/sbin/lvchange', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba', '-p', 'r']
<8756> 2015-12-07 21:50:45.229864 *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
<8756> 2015-12-07 21:50:45.229919 ***********************
<8756> 2015-12-07 21:50:45.229969 * E X C E P T I O N *
<8756> 2015-12-07 21:50:45.230017 ***********************
<8756> 2015-12-07 21:50:45.230077 coalesce: EXCEPTION util.SMException, VHD *1a240d45[VHD](20.000G//136.000M|n) corrupted
<8756> 2015-12-07 21:50:45.230129 File "/opt/xensource/sm/cleanup.py", line 1397, in coalesce
self._coalesce(vdi)
File "/opt/xensource/sm/cleanup.py", line 1587, in _coalesce
vdi._doCoalesce()
File "/opt/xensource/sm/cleanup.py", line 1050, in _doCoalesce
self.parent.validate()
File "/opt/xensource/sm/cleanup.py", line 1043, in validate
VDI.validate(self, fast)
File "/opt/xensource/sm/cleanup.py", line 633, in validate
raise util.SMException("VHD %s corrupted" % self)

<8756> 2015-12-07 21:50:45.230181 *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
<8756> 2015-12-07 21:50:45.230234 Coalesce failed, skipping

Sorry for possible mistakes, but after working for 13 hours, my concentration has dropped and thank you for your help.

Regards,
F.

Nick Brody

2015-12-08 06:43:14 UTC

Permalink

Thank you

Post by France
Hi Guys,
after battling with cloudstack (4.3.2+) for the whole day with another
issue (unable to create snapshotâs for one VM, which I also failed to fix)
I ran into message on XenServer 6.0.2+Hotfixes SMlog (copy pasted at the
bottom).
Even thou i kinda understand XenServers relations of VM -> VBD -> VDI ->
SR -> PBD, i failed to find out to which VM does this corrupted .vhd
snapshot belong.
Can you please help me find it? Afterwards, we can find the appropriate
action in order to remove the error and make coalesce work.
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
uuid ( RO) : 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
name-label ( RW): base copy
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): generate_config; update; resize; destroy;
clone; copy; snapshot
sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
sr-name-label ( RO): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
virtual-size ( RO): 21474836480
physical-utilisation ( RO): 12582912
location ( RO): 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
type ( RO): User
sharable ( RO): false
read-only ( RO): true
storage-lock ( RO): false
managed ( RO): false
parent ( RO): <not in database>
missing ( RO): false
eJxjYBgFo2AUjIKRCQAFAAAB
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
uuid ( RO) : 3b6e386d-8736-6b6b-7006-f3f4df9bd586
name-label ( RW): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
name-description ( RW): 208 - censored
host ( RO): <shared>
type ( RO): lvmoiscsi
content-type ( RO): user
Which is on one of many HA iscsi targets mounted on the XenServer cluster.
uuid ( RO) : f471ef16-4670-2f5e-967e-7e98d4721fe7
host-uuid ( RO): f6d699aa-f58f-449e-9a51-6e965562f178
sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
device-config (MRO): targetIQN: censoredi:storage.censored;
target: censored; SCSIid: 1target2lun1
currently-attached ( RO): true)
[8756] 2015-12-07 21:50:45.118758 ['/usr/sbin/vhd-util', 'check',
'--debug', '-n',
'/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
[8756] 2015-12-07 21:50:45.131975 FAILED: (rc 22) stdout: 'primary
footer invalid: invalid cookie
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
appears invalid; dumping metadata
-------------------
Cookie : conectix
Features : (0x00000002) <RESV>
File format version : Major: 1, Minor: 0
Data offset : 512
Timestamp : Mon Dec 7 18:19:46 2015
Creator Application : 'tap'
Creator version : Major: 1, Minor: 3
Creator OS : Unknown!
Original disk size : 20480 MB (21474836480 Bytes)
Current disk size : 20480 MB (21474836480 Bytes)
Geometry : Cyl: 41610, Hds: 16, Sctrs: 63
: = 20479 MB (21474754560 Bytes)
Disk type : Differencing hard disk
Checksum : 0xffffef85|0xffffef85 (Good!)
UUID : 8f25299d-d7cc-44fe-a42b-95b4e9a31a47
Saved state : No
Hidden : 1
-------------------
Cookie : cxsparse
Data offset (unusd) : 18446744073709
Table offset : 1536
Header version : 0x00010000
Max BAT size : 1048576
Block size : 2097152 (2 MB)
VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
Parent UUID : 8cece4c9-8c94-47b9-9f7a-cb6023da9b72
Parent timestamp : Mon Dec 7 18:19:46 2015
Checksum : 0xffffc6ed|0xffffc6ed (Good!)
--------------------
locator: : 0
code : PLAT_CODE_MACX
data_space : 512
data_length : 110
data_offset : 4327424
./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
locator: : 1
code : PLAT_CODE_W2KU
data_space : 512
data_length : 206
data_offset : 4327936
./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
locator: : 2
code : PLAT_CODE_W2RU
data_space : 512
data_length : 206
data_offset : 4328448
./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
-------------------
Batmap offset : 4196352
Batmap size (secs) : 256
Batmap version : 0x00010002
Checksum : 0xffffff7f|0xffffff7f (Good!)
', stderr: ''
[8756] 2015-12-07 21:50:45.132293 ['/usr/sbin/vhd-util', 'query',
'--debug', '-s', '-n',
'/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
[8756] 2015-12-07 21:50:45.145222 SUCCESS
[8756] 2015-12-07 21:50:45.145502 lock: tried lock
/var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr, acquired: True
(exists: True)
[8756] 2015-12-07 21:50:45.145635 lock: released
/var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr
[8756] 2015-12-07 21:50:45.145752 ['/usr/sbin/lvchange',
'/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba',
'-p', 'r']
<8756> 2015-12-07 21:50:45.229864
*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
<8756> 2015-12-07 21:50:45.229919 ***********************
<8756> 2015-12-07 21:50:45.229969 * E X C E P T I O N *
<8756> 2015-12-07 21:50:45.230017 ***********************
<8756> 2015-12-07 21:50:45.230077 coalesce: EXCEPTION
util.SMException, VHD *1a240d45[VHD](20.000G//136.000M|n) corrupted
<8756> 2015-12-07 21:50:45.230129 File
"/opt/xensource/sm/cleanup.py", line 1397, in coalesce
self._coalesce(vdi)
File "/opt/xensource/sm/cleanup.py", line 1587, in _coalesce
vdi._doCoalesce()
File "/opt/xensource/sm/cleanup.py", line 1050, in _doCoalesce
self.parent.validate()
File "/opt/xensource/sm/cleanup.py", line 1043, in validate
VDI.validate(self, fast)
File "/opt/xensource/sm/cleanup.py", line 633, in validate
raise util.SMException("VHD %s corrupted" % self)
<8756> 2015-12-07 21:50:45.230181
*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
<8756> 2015-12-07 21:50:45.230234 Coalesce failed, skipping
Sorry for possible mistakes, but after working for 13 hours, my
concentration has dropped and thank you for your help.
Regards,
F.

Dag Sonstebo

2015-12-08 08:46:58 UTC

Permalink

Hi France,

have a look at the following writeup I did a few weeks back - http://www.shapeblue.com/recovery-of-vms-to-new-cloudstack-instance/

It should give you an idea how to troubleshoot the VHD chain and how to tie it back to DB entries and potential VM owner.

Regards,

Dag Sonstebo
Cloud Architect
ShapeBlue

Post by France
Hi Guys,
after battling with cloudstack (4.3.2+) for the whole day with another issue (unable to create snapshot’s for one VM, which I also failed to fix) I ran into message on XenServer 6.0.2+Hotfixes SMlog (copy pasted at the bottom).
Even thou i kinda understand XenServers relations of VM -> VBD -> VDI -> SR -> PBD, i failed to find out to which VM does this corrupted .vhd snapshot belong.
Can you please help me find it? Afterwards, we can find the appropriate action in order to remove the error and make coalesce work.
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
uuid ( RO) : 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
name-label ( RW): base copy
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): generate_config; update; resize; destroy; clone; copy; snapshot
sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
sr-name-label ( RO): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
virtual-size ( RO): 21474836480
physical-utilisation ( RO): 12582912
location ( RO): 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
type ( RO): User
sharable ( RO): false
read-only ( RO): true
storage-lock ( RO): false
managed ( RO): false
parent ( RO): <not in database>
missing ( RO): false
sm-config (MRO): vhd-parent: 555b38bf-fe63-41b2-9609-e16f4e26b274; vdi_type: vhd; vhd-blocks: eJxjYBgFo2AUjIKRCQAFAAAB
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
uuid ( RO) : 3b6e386d-8736-6b6b-7006-f3f4df9bd586
name-label ( RW): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
name-description ( RW): 208 - censored
host ( RO): <shared>
type ( RO): lvmoiscsi
content-type ( RO): user
Which is on one of many HA iscsi targets mounted on the XenServer cluster.
uuid ( RO) : f471ef16-4670-2f5e-967e-7e98d4721fe7
host-uuid ( RO): f6d699aa-f58f-449e-9a51-6e965562f178
sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
device-config (MRO): targetIQN: censoredi:storage.censored; target: censored; SCSIid: 1target2lun1
currently-attached ( RO): true)
[8756] 2015-12-07 21:50:45.118758 ['/usr/sbin/vhd-util', 'check', '--debug', '-n', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
[8756] 2015-12-07 21:50:45.131975 FAILED: (rc 22) stdout: 'primary footer invalid: invalid cookie
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba appears invalid; dumping metadata
-------------------
Cookie : conectix
Features : (0x00000002) <RESV>
File format version : Major: 1, Minor: 0
Data offset : 512
Timestamp : Mon Dec 7 18:19:46 2015
Creator Application : 'tap'
Creator version : Major: 1, Minor: 3
Creator OS : Unknown!
Original disk size : 20480 MB (21474836480 Bytes)
Current disk size : 20480 MB (21474836480 Bytes)
Geometry : Cyl: 41610, Hds: 16, Sctrs: 63
: = 20479 MB (21474754560 Bytes)
Disk type : Differencing hard disk
Checksum : 0xffffef85|0xffffef85 (Good!)
UUID : 8f25299d-d7cc-44fe-a42b-95b4e9a31a47
Saved state : No
Hidden : 1
-------------------
Cookie : cxsparse
Data offset (unusd) : 18446744073709
Table offset : 1536
Header version : 0x00010000
Max BAT size : 1048576
Block size : 2097152 (2 MB)
Parent name : VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
Parent UUID : 8cece4c9-8c94-47b9-9f7a-cb6023da9b72
Parent timestamp : Mon Dec 7 18:19:46 2015
Checksum : 0xffffc6ed|0xffffc6ed (Good!)
--------------------
locator: : 0
code : PLAT_CODE_MACX
data_space : 512
data_length : 110
data_offset : 4327424
decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
locator: : 1
code : PLAT_CODE_W2KU
data_space : 512
data_length : 206
data_offset : 4327936
decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
locator: : 2
code : PLAT_CODE_W2RU
data_space : 512
data_length : 206
data_offset : 4328448
decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
-------------------
Batmap offset : 4196352
Batmap size (secs) : 256
Batmap version : 0x00010002
Checksum : 0xffffff7f|0xffffff7f (Good!)
', stderr: ''
[8756] 2015-12-07 21:50:45.132293 ['/usr/sbin/vhd-util', 'query', '--debug', '-s', '-n', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
[8756] 2015-12-07 21:50:45.145222 SUCCESS
[8756] 2015-12-07 21:50:45.145502 lock: tried lock /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr, acquired: True (exists: True)
[8756] 2015-12-07 21:50:45.145635 lock: released /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr
[8756] 2015-12-07 21:50:45.145752 ['/usr/sbin/lvchange', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba', '-p', 'r']
<8756> 2015-12-07 21:50:45.229864 *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
<8756> 2015-12-07 21:50:45.229919 ***********************
<8756> 2015-12-07 21:50:45.229969 * E X C E P T I O N *
<8756> 2015-12-07 21:50:45.230017 ***********************
<8756> 2015-12-07 21:50:45.230077 coalesce: EXCEPTION util.SMException, VHD *1a240d45[VHD](20.000G//136.000M|n) corrupted
<8756> 2015-12-07 21:50:45.230129 File "/opt/xensource/sm/cleanup.py", line 1397, in coalesce
self._coalesce(vdi)
File "/opt/xensource/sm/cleanup.py", line 1587, in _coalesce
vdi._doCoalesce()
File "/opt/xensource/sm/cleanup.py", line 1050, in _doCoalesce
self.parent.validate()
File "/opt/xensource/sm/cleanup.py", line 1043, in validate
VDI.validate(self, fast)
File "/opt/xensource/sm/cleanup.py", line 633, in validate
raise util.SMException("VHD %s corrupted" % self)
<8756> 2015-12-07 21:50:45.230181 *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
<8756> 2015-12-07 21:50:45.230234 Coalesce failed, skipping
Sorry for possible mistakes, but after working for 13 hours, my concentration has dropped and thank you for your help.
Regards,
F.

Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBl

France

2015-12-08 16:39:21 UTC

Permalink

Thank you for your response Dag.
I read your article (even before you linked it :-).

After executing the SQL querry, I matched the results from it with all possible UUIDs i could think of. There were no matches. :-(
BTW Dag, I guess you should correct the column naming in your SQL statement, cause its voltype and not volpath (..cloud.volumes.volume_type as volpath,) :-)

I read the article to the end and tried reading the VHD directly with vhd-util, but noticed that it is is actually missing:
[***@x3 ~]# ls -la /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
ls: /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba: No such file or directory
[***@x3 ~]# ls -la /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
ls: /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba: No such file or directory
[***@x3 ~]# ls -la /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/
total 24
drwxr-xr-x 2 root root 180 Dec 8 15:12 .
drwxr-xr-x 22 root root 17200 Dec 8 15:11 ..
lrwxrwxrwx 1 root root 112 Jun 28 14:30 hb-f6d699aa-f58f-449e-9a51-6e965562f178 -> /dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-hb--f6d699aa--f58f--449e--9a51--6e965562f178
lrwxrwxrwx 1 root root 71 Jun 28 12:22 MGT -> /dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-MGT
lrwxrwxrwx 1 root root 113 Nov 12 03:26 VHD-10e0a90f-6f47-41f4-910c-53a1514a0f2c -> /dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--10e0a90f--6f47--41f4--910c--53a1514a0f2c
lrwxrwxrwx 1 root root 113 Dec 1 14:04 VHD-19fd1780-dacf-4fa9-8615-cd3204df6b14 -> /dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--19fd1780--dacf--4fa9--8615--cd3204df6b14
lrwxrwxrwx 1 root root 113 Nov 12 03:26 VHD-b51b4cb7-f2c5-4ba3-98d3-4a23ff599ad9 -> /dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--b51b4cb7--f2c5--4ba3--98d3--4a23ff599ad9
lrwxrwxrwx 1 root root 113 Dec 1 14:01 VHD-bb171b0f-e806-4620-8b2c-0ca46203a431 -> /dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--bb171b0f--e806--4620--8b2c--0ca46203a431
lrwxrwxrwx 1 root root 113 Nov 12 03:26 VHD-e04aa8fa-93ce-4133-8fcb-35c9ce57ed08 -> /dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--e04aa8fa--93ce--4133--8fcb--35c9ce57ed08
[***@x3 ~]# vhd-util read -p -n /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
Failed to open /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba: -2
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba appears invalid; dumping headers

Then I decided to see to which chain it belongs to. I simply looked SMlog for VHD *1a240d45[VHD](20.000G//136.000M|n) after a srscan. Result of changed vdis:
(Uppon request I can provide the whole chain.)
*555b38bf[VHD](20.000G//2.695G|n)
5afd5849[VHD](20.000G//8.000M|n)
*02e8b56b[VHD](20.000G//12.000M|n)
*1a240d45[VHD](20.000G//136.000M|n) <- corrupted
*c99e711b[VHD](20.000G//136.000M|n)
34a03c9a[VHD](20.000G//8.000M|a)
5c12db0a[VHD](20.000G//8.000M|n)

I have noticed that above numbers are part of UUID so I did additional search on the whole database and these are the results:
*555b38bf[VHD](20.000G//2.695G|n) -> not foud
5afd5849[VHD](20.000G//8.000M|n) -> found in template_spool_ref. status ready, template ID 292
*02e8b56b[VHD](20.000G//12.000M|n) -> not found
*1a240d45[VHD](20.000G//136.000M|n) <- corrupted and not found
*c99e711b[VHD](20.000G//136.000M|n) -> not found
34a03c9a[VHD](20.000G//8.000M|a) -> found in shapshot_store_ref. state destroyed, volume id 661, from template id 292
5c12db0a[VHD](20.000G//8.000M|n) -> found in shapshot_store_ref. state destroyed

Here are my assumptions / questions. Please advise / comment.
Is last snapshot: 5c12db0a in chain linked to the first: 555b38bf or the chain ends with 34a03c9a?
Does anyone have a link to documentation explaining this output?
What do the starts ie. * mean?
To which VM based on the above output is the *1a240d45[VHD]related?
How to make the error go away? How to safely delete *1a240d45[VHD]?

All suggestions are welcome!

Thank you.
F.

Post by Dag Sonstebo
Hi France,
have a look at the following writeup I did a few weeks back - http://www.shapeblue.com/recovery-of-vms-to-new-cloudstack-instance/
It should give you an idea how to troubleshoot the VHD chain and how to tie it back to DB entries and potential VM owner.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

Post by France
Hi Guys,
after battling with cloudstack (4.3.2+) for the whole day with another issue (unable to create snapshots for one VM, which I also failed to fix) I ran into message on XenServer 6.0.2+Hotfixes SMlog (copy pasted at the bottom).
Even thou i kinda understand XenServers relations of VM -> VBD -> VDI -> SR -> PBD, i failed to find out to which VM does this corrupted .vhd snapshot belong.
Can you please help me find it? Afterwards, we can find the appropriate action in order to remove the error and make coalesce work.
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
uuid ( RO) : 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
name-label ( RW): base copy
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): generate_config; update; resize; destroy; clone; copy; snapshot
sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
sr-name-label ( RO): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
virtual-size ( RO): 21474836480
physical-utilisation ( RO): 12582912
location ( RO): 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
type ( RO): User
sharable ( RO): false
read-only ( RO): true
storage-lock ( RO): false
managed ( RO): false
parent ( RO): <not in database>
missing ( RO): false
sm-config (MRO): vhd-parent: 555b38bf-fe63-41b2-9609-e16f4e26b274; vdi_type: vhd; vhd-blocks: eJxjYBgFo2AUjIKRCQAFAAAB
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
uuid ( RO) : 3b6e386d-8736-6b6b-7006-f3f4df9bd586
name-label ( RW): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
name-description ( RW): 208 - censored
host ( RO): <shared>
type ( RO): lvmoiscsi
content-type ( RO): user
Which is on one of many HA iscsi targets mounted on the XenServer cluster.
uuid ( RO) : f471ef16-4670-2f5e-967e-7e98d4721fe7
host-uuid ( RO): f6d699aa-f58f-449e-9a51-6e965562f178
sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
device-config (MRO): targetIQN: censoredi:storage.censored; target: censored; SCSIid: 1target2lun1
currently-attached ( RO): true)
[8756] 2015-12-07 21:50:45.118758 ['/usr/sbin/vhd-util', 'check', '--debug', '-n', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
[8756] 2015-12-07 21:50:45.131975 FAILED: (rc 22) stdout: 'primary footer invalid: invalid cookie
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba appears invalid; dumping metadata
-------------------
Cookie : conectix
Features : (0x00000002) <RESV>
File format version : Major: 1, Minor: 0
Data offset : 512
Timestamp : Mon Dec 7 18:19:46 2015
Creator Application : 'tap'
Creator version : Major: 1, Minor: 3
Creator OS : Unknown!
Original disk size : 20480 MB (21474836480 Bytes)
Current disk size : 20480 MB (21474836480 Bytes)
Geometry : Cyl: 41610, Hds: 16, Sctrs: 63
: = 20479 MB (21474754560 Bytes)
Disk type : Differencing hard disk
Checksum : 0xffffef85|0xffffef85 (Good!)
UUID : 8f25299d-d7cc-44fe-a42b-95b4e9a31a47
Saved state : No
Hidden : 1
-------------------
Cookie : cxsparse
Data offset (unusd) : 18446744073709
Table offset : 1536
Header version : 0x00010000
Max BAT size : 1048576
Block size : 2097152 (2 MB)
Parent name : VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
Parent UUID : 8cece4c9-8c94-47b9-9f7a-cb6023da9b72
Parent timestamp : Mon Dec 7 18:19:46 2015
Checksum : 0xffffc6ed|0xffffc6ed (Good!)
--------------------
locator: : 0
code : PLAT_CODE_MACX
data_space : 512
data_length : 110
data_offset : 4327424
decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
locator: : 1
code : PLAT_CODE_W2KU
data_space : 512
data_length : 206
data_offset : 4327936
decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
locator: : 2
code : PLAT_CODE_W2RU
data_space : 512
data_length : 206
data_offset : 4328448
decoded name : ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
-------------------
Batmap offset : 4196352
Batmap size (secs) : 256
Batmap version : 0x00010002
Checksum : 0xffffff7f|0xffffff7f (Good!)
', stderr: ''
[8756] 2015-12-07 21:50:45.132293 ['/usr/sbin/vhd-util', 'query', '--debug', '-s', '-n', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
[8756] 2015-12-07 21:50:45.145222 SUCCESS
[8756] 2015-12-07 21:50:45.145502 lock: tried lock /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr, acquired: True (exists: True)
[8756] 2015-12-07 21:50:45.145635 lock: released /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr
[8756] 2015-12-07 21:50:45.145752 ['/usr/sbin/lvchange', '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba', '-p', 'r']
<8756> 2015-12-07 21:50:45.229864 *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
<8756> 2015-12-07 21:50:45.229919 ***********************
<8756> 2015-12-07 21:50:45.229969 * E X C E P T I O N *
<8756> 2015-12-07 21:50:45.230017 ***********************
<8756> 2015-12-07 21:50:45.230077 coalesce: EXCEPTION util.SMException, VHD *1a240d45[VHD](20.000G//136.000M|n) corrupted
<8756> 2015-12-07 21:50:45.230129 File "/opt/xensource/sm/cleanup.py", line 1397, in coalesce
self._coalesce(vdi)
File "/opt/xensource/sm/cleanup.py", line 1587, in _coalesce
vdi._doCoalesce()
File "/opt/xensource/sm/cleanup.py", line 1050, in _doCoalesce
self.parent.validate()
File "/opt/xensource/sm/cleanup.py", line 1043, in validate
VDI.validate(self, fast)
File "/opt/xensource/sm/cleanup.py", line 633, in validate
raise util.SMException("VHD %s corrupted" % self)
<8756> 2015-12-07 21:50:45.230181 *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
<8756> 2015-12-07 21:50:45.230234 Coalesce failed, skipping
Sorry for possible mistakes, but after working for 13 hours, my concentration has dropped and thank you for your help.
Regards,
F.

Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.