Another Milestone: Sectors Start to Fail

So, finally, after 1.96PB written to the disk, we have our first failures in the flash: 2 sectors have been reallocated from reserved space after they failed to erase properly. The disk is still working fine, and it can be read from and written to without problem, but clearly our workload is starting to take its toll:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   099   099   010    Pre-fail  2
  9 Power_On_Hours          099   099   000    Old_age   2487
 12 Power_Cycle_Count       099   099   000    Old_age   167
177 Wear_Leveling_Count     069   069   000    Pre-fail  1831
179 Used_Rsvd_Blk_Cnt_Tot   099   099   010    Pre-fail  2
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  099   099   010    Old_age   2
183 Runtime_Bad_Block       099   099   010    Pre-fail  2
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 064   060   000    Old_age   36
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   30
241 Total_LBAs_Written      098   098   000    Old_age   3830852167169

We have written over 13 times the warranted life of the disk (150TBW), and only now has the first problem emerged. But the disk is coping with this fine - it has space set aside to replace sectors as they fail, and so far the problems are only with erasing cells, so no data has been lost.

The test will continue, and we will monitor the failure rate for cells from now on.  Watch this space!