Mar 29 04:10:35 a4 kernel: [ 6542.673070] mce: [Hardware Error]: Machine check events logged Mar 29 04:10:35 a4 kernel: [ 6542.673079] Memory failure: 0x6a0430: recovery action for free buddy page: Delayed Mar 29 04:10:35 a4 kernel: [ 6542.673116] [Hardware Error]: Deferred error, no action required. Mar 29 04:10:35 a4 kernel: [ 6542.673119] [Hardware Error]: CPU:0 (19:21:0) MC18_STATUS[Over|-|MiscV|AddrV|-|-|SyndV|UECC|Deferred|-|Scrub]: 0xdc2031000000011b Mar 29 04:10:35 a4 kernel: [ 6542.673124] [Hardware Error]: Error Addr: 0x00000006a04302c0 Mar 29 04:10:35 a4 kernel: [ 6542.673125] [Hardware Error]: IPID: 0x0000009600150f00, Syndrome: 0x77778a000b800000 Mar 29 04:10:35 a4 kernel: [ 6542.673128] [Hardware Error]: Unified Memory Controller Ext. Error Code: 0, DRAM ECC error. Mar 29 04:10:35 a4 kernel: [ 6542.673140] EDAC MC0: 1 UE on mc#0csrow#0channel#1 (csrow:0 channel:1 page:0x1ab10c0 offset:0x9c0 grain:64) Mar 29 04:10:35 a4 kernel: [ 6542.673142] [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD18 of the errors were uncorrectable (UE), 1 was correctable (CE). We noticed no software crashes from the unrecoverable errors, so the processor may have noticed the errors during scrubbing (it walks through the memory to detect and correct any flipped bits before they accumulate) rather than on a software read access. All these messages refer to "mc#0csrow#0channel#1 (csrow:0 channel:1", which indicates that it is a specific DIMM that causes these problems rather than a problem with the memory controller (i.e., the CPU), or the mainboard in general. Also, when doing
grep . /sys/devices/system/edac/mc/mc*/rank*/dimm_?e_countwe saw that all errors were in rank1 (i.e., the same side of the same DIMM). The question was: which DIMM should we replace?
dmidecode -t memorygives us some information about the DIMMs, but unfortunately with no overlap with the information about the ranks. We eventually resorted to pulling a DIMM out and checking which ranks were still present when doing the grep above (fortunately only actually present ranks are shown), and repeating that until rank1 was no longer there. Then we replaced that DIMM with a working one, and we again have a machine that does not produce ECC errors frequently.
In case you also have an ASUS TUF Gaming B550M-Plus board, here are the mappings we found between ranks and DIMM slots.
dimm_location DIMM
rank csrow channel slot
0 0 0 B1
1 0 1 A1
2 1 0 B1
3 1 1 A1
4 2 0 B2
5 2 1 A2
6 3 0 B2
7 3 1 A2
Did ECC help in this case? Given that 18 of the 19 errors were
uncorrectable, one may doubt that. However, it gave the only
indication of a hardware problem we saw, and it allowed us to pinpoint
a failing component relatively quickly that would have been hard to
find out through occasional software failures; would we have started
to use memtest after a few unexplainable software crashes?
Just in case you are wondering about the symptoms: With a degraded bit in a DRAM device (i.e., DRAM chip, 9 per side on our DIMMs) we would expect correctable errors, and no spontaneous reboots. In case of some other degradation of a RAM device we would expect errors to be logged first in most cases. Our theory is that some common cause resulted in the sponaneous reboots and also caused degradation of a more significant part than one bit in a DRAM device (my wild guess would be that a row decoder was damaged). The fact that we don't know what this cause was worries me.