The End has Come

So, after over 7PB written the 850 Pro has finally given up the ghost. It's been an amazing run over the past 9 months, and the drive has vastly outlasted its puny 150TB warranty - by a factor of more than 46 times.

 
The Amazing Samsung 850 Pro 1TB SSD
 

Interestingly, the disk didn't fail while writing but during other testing. When attempting to enable drive security, an error code was returned and the SSD stopped responding to commands.  And despite numerous attempts to revive it, it is completely bricked.

In conclusion then, this is an amazing device - fast, consistent, long-lived - overall excellent!

7PB Written - Amazing!

Wow! 7PB written to a drive only assured for 150TB - that's more than 46 times the warranted amount, and the disk is still going strong. We're checking the data written to it every 0.1PB to ensure there is no corruption, and we check the SMART stats for errors too. Nothing. It's quite remarkable. The disk's expected wear levelling has been exhausted (stat now stuck at 1% remaining) and the reallocated sector count continues to climb as more and more sectors fail - but there's 31% remaining here still, and at the current rate this could equate to another couple of PB...

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   031   031   010    Pre-fail  4059
  9 Power_On_Hours          098   098   000    Old_age   5159
 12 Power_Cycle_Count       099   099   000    Old_age   183
177 Wear_Leveling_Count     001   001   000    Pre-fail  6519
179 Used_Rsvd_Blk_Cnt_Tot   031   031   010    Pre-fail  4059
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  031   031   010    Old_age   4059
183 Runtime_Bad_Block       031   031   010    Pre-fail  4059
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 060   052   000    Old_age   40
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   31
241 Total_LBAs_Written      095   095   000    Old_age   13672464730475

Write performance seems to vary gradually over time - recently it has typically been 518MB/s; a PB or two ago it was often 529MB/s. Once the test is complete, we will graph it to see if there are any trends worth noting. But for now, the disk will be left powered off for a week to check on short-term data retention, and then the wear test will resume.

6PB Written - and still going!

So - an amazing 6PB written so far, which is 40x the SSD's rated endurance of 150TBW - and it is still going strong. 3111 sectors have failed during erasure and have been reallocated, which is 53% of the pool - 47% remaining, so still a fair bit of life in the disk. We have also just completed a 9-day power-off to check for basic data retention, and everything was fine.

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   047   047   010    Pre-fail  3111
  9 Power_On_Hours          099   099   000    Old_age   4617
 12 Power_Cycle_Count       099   099   000    Old_age   181
177 Wear_Leveling_Count     007   007   000    Pre-fail  5580
179 Used_Rsvd_Blk_Cnt_Tot   047   047   010    Pre-fail  3111
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  047   047   010    Old_age   3111
183 Runtime_Bad_Block       047   047   010    Pre-fail  3111
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 057   052   000    Old_age   43
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   31
241 Total_LBAs_Written      095   095   000    Old_age   11718844550148

Wear-levelling count is getting towards its limit - only 7% life left - so it will be interesting to see what effect that has when it reaches 0, which can't be far off now.

Finally, in response to comments on an earlier post, I have added a md5 check to ensure that there is no unreported data corruption occurring. This will run every 100 writes to the disk, and verify that the data really does read back correctly.

5PB Written, 2476 Sectors Reallocated

So - 5PB written, no unrecoverable errors, and some interesting observations from the stats below. The rate of sector failures has fallen dramatically over the last PB written. It is difficult to be certain of the cause - it may simply be that the weaker flash cells have been largely eliminated now, and those remaining have plenty more life. It may also be related to the fact that, after the last pause at 4PBW, the disk was secure-erased and then Samsung Magician was used to over-provision it by 10% - though it's not clear why that should impact the SMART stats.  Whatever the cause, the SSD is holding up better than predicted:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   058   058   010    Pre-fail  2476
  9 Power_On_Hours          099   099   000    Old_age   4085
 12 Power_Cycle_Count       099   099   000    Old_age   177
177 Wear_Leveling_Count     023   023   000    Pre-fail  4646
179 Used_Rsvd_Blk_Cnt_Tot   058   058   010    Pre-fail  2476
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  058   058   010    Old_age   2476
183 Runtime_Bad_Block       058   058   010    Pre-fail  2476
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 057   052   000    Old_age   43
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   31
241 Total_LBAs_Written      096   096   000    Old_age   9766178591977

The disk will now be rested for a week, and then checked to ensure that the data it was holding can all be read back without error - a short retention test.  Then back to trying to wear it out!

4PB Written, 2250 Sectors Reallocated

Another milestone reached - 4PB written to the SSD, and it is still working fine! Sector fails have risen to 2254, but these have all occurred during erasure and so have not introduced any unrecoverable errors yet. Program Fail errors still stand at zero, as do Reported Uncorrectable. The SSD has reallocated 39% of its reserved space, with 61% still remaining.

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   061   061   010    Pre-fail  2254
  9 Power_On_Hours          099   099   000    Old_age   3558
 12 Power_Cycle_Count       099   099   000    Old_age   170
177 Wear_Leveling_Count     038   038   000    Pre-fail  3713
179 Used_Rsvd_Blk_Cnt_Tot   061   061   010    Pre-fail  2254
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  061   061   010    Old_age   2254
183 Runtime_Bad_Block       061   061   010    Pre-fail  2254
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 064   060   000    Old_age   36
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   30
241 Total_LBAs_Written      097   097   000    Old_age   7814058492462

Once again we will power off the server for a week and read back all the data on the SSD to check for retention. We will then secure erase the disk and increase the amount of reserved space (currently at the default) to 10% via the Samsung Magician software, and see what difference that makes to the SMART stats.

3PB Written, 800 Sectors Reallocated

So, another petabyte written and the disk is still working well - no unrecoverable errors so far and no drop in performance. Sector fails are creeping up steadily - just under 800 to date - but so far all these have occurred when the flash was being erased rather than written to. The SSD controller appears to be coping fine, allocating spare sectors from its reserved pool (14% used, 86% remaining):

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   086   086   010    Pre-fail  797
  9 Power_On_Hours          099   099   000    Old_age   3034
 12 Power_Cycle_Count       099   099   000    Old_age   169
177 Wear_Leveling_Count     053   053   000    Pre-fail  2788
179 Used_Rsvd_Blk_Cnt_Tot   086   086   010    Pre-fail  797
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  086   086   010    Old_age   797
183 Runtime_Bad_Block       086   086   010    Pre-fail  797
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 065   060   000    Old_age   35
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   30
241 Total_LBAs_Written      097   097   000    Old_age   5859439678108

The disk that was rated for 150TB of writes has now reached 20 times that and still has life in it. But there will be a cost - it is likely that the data retention period when powered off is much less than the warranted 12 months.  When powered on, the controller should periodically check that all cells are readable and swap out any looking a bit dodgy.

As previously, we will now conduct a mini retention test by powering the disk off for a week, and then checking that we can read back the huge file filling it correctly.

Short Retention Test

After 7 days powered off, we reconnected the disk and booted the test server. The aim was to check whether the flash, after 2PB written, could still retain data reliably. We had left a huge file completely filling the SSD before powering down the system, and read this back to check for errors. There were none.

So we have restarted the write testing, and will update as things progress.

Another Milestone: Sectors Start to Fail

So, finally, after 1.96PB written to the disk, we have our first failures in the flash: 2 sectors have been reallocated from reserved space after they failed to erase properly. The disk is still working fine, and it can be read from and written to without problem, but clearly our workload is starting to take its toll:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   099   099   010    Pre-fail  2
  9 Power_On_Hours          099   099   000    Old_age   2487
 12 Power_Cycle_Count       099   099   000    Old_age   167
177 Wear_Leveling_Count     069   069   000    Pre-fail  1831
179 Used_Rsvd_Blk_Cnt_Tot   099   099   010    Pre-fail  2
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  099   099   010    Old_age   2
183 Runtime_Bad_Block       099   099   010    Pre-fail  2
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 064   060   000    Old_age   36
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   30
241 Total_LBAs_Written      098   098   000    Old_age   3830852167169

We have written over 13 times the warranted life of the disk (150TBW), and only now has the first problem emerged. But the disk is coping with this fine - it has space set aside to replace sectors as they fail, and so far the problems are only with erasing cells, so no data has been lost.

The test will continue, and we will monitor the failure rate for cells from now on.  Watch this space!

Two Petabytes Written - Time for a Short Break

So, the 850 Pro continues to impress, and this is quite something: 2PBW and the flash is less than a third worn. According to SMART stats, we should still have another 4+PB (69%) to go:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   099   099   010    Pre-fail  7
  9 Power_On_Hours          099   099   000    Old_age   2507
 12 Power_Cycle_Count       099   099   000    Old_age   167
177 Wear_Leveling_Count     069   069   000    Pre-fail  1867
179 Used_Rsvd_Blk_Cnt_Tot   099   099   010    Pre-fail  7
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  099   099   010    Old_age   7
183 Runtime_Bad_Block       099   099   010    Pre-fail  7
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 064   060   000    Old_age   36
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   30
241 Total_LBAs_Written      098   098   000    Old_age   3906798501645

We're going to take the opportunity to test the flash's retention by powering off the system for a week, and then checking that we can read back a large data file filling the disk. If this is ok, we'll restart the testing and let it run for another 2 PB.

1.5PBW and Still Going Strong!

After we passed the 1PB mark, we moved the SSD to another server with a SATA 3 connection to speed things up. This has pretty much doubled the rate at which we can write to the disk:

953+0 records in
953+0 records out
1023275958272 bytes (1.0 TB) copied, 1925.34 s, 531 MB/s

And this is necessary - the SSD is showing no signs of giving up yet. Quite the opposite in fact - it has now reached 1.5PB written, and is still reporting 76% life left!

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   2222
 12 Power_Cycle_Count       099   099   000    Old_age   165
177 Wear_Leveling_Count     076   076   000    Pre-fail  1407
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 065   060   000    Old_age   35
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   30
241 Total_LBAs_Written      098   098   000    Old_age   2931030500253

We'll keep updating as things progress.

36-Hour Power-Off

Having just passed 1PB written, we decided to check that all was still well with the SSD by powering it off for 36 hours.  On restarting the server, we checked that the SSD was still visible and then tried reading the huge static file we had been using for the last stage of testing:

# dd if=/data/static of=/dev/null bs=1G iflag=direct
810+0 records in
810+0 records out
869730877440 bytes (870 GB) copied, 3033.31 s, 287 MB/s

No errors were reported, and performance was the same as earlier - limited by the SATA2 connection currently in use. SMART reports that the flash is only 18% worn - 82% still to go, so the tests are going to be running for a while yet!

Write Amplification at 85% Utilisation

We've already looked at the impact on write amplification of having 50% of the disk occupied. Now we'll look at the effect of increasing this to 85%, and see if this increases write amplification. First, let's review the expected life of the disk. The following 2 sets of SMART stats were taken when the wear levelling count dropped by 1%:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1777
 12 Power_Cycle_Count       099   099   000    Old_age   161
177 Wear_Leveling_Count     085   085   000    Pre-fail  849
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 066   060   000    Old_age   34
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   1769630340990
ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1843
 12 Power_Cycle_Count       099   099   000    Old_age   161
177 Wear_Leveling_Count     084   084   000    Pre-fail  910
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 065   060   000    Old_age   35
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   1894685865950

The volume of data written to cause this drop was:

(1894685865950 -1769630340990) * 512 = 64.03TB.

This is exactly in line with the last measurement, and supports the estimated life for the disk at around 6.4PBW.

Moving on to write amplification, we need to look at two more sets of SMART stats.

After run 2008:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1726
 12 Power_Cycle_Count       099   099   000    Old_age   161
177 Wear_Leveling_Count     086   086   000    Pre-fail  801
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 067   060   000    Old_age   33
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   1671265323927

After run 3568:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1970
 12 Power_Cycle_Count       099   099   000    Old_age   161
177 Wear_Leveling_Count     083   083   000    Pre-fail  1029
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 064   060   000    Old_age   36
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   2138526416722

Each test run writes 143GiB, so we can calculate the total data written during this period of the test from the number of test runs completed:

(3568 - 2008) * 143GiB / 1024 = 217.85TiB.

This correlates very closely with what the SSD has recorded as 'Total LBAs Written', but is actually slightly higher:

(2138526416722 -1671265323927) * 512 / (1024 * 1024 * 1024 * 1024) = 217.59TiB.

One possible explanation for this could be that, since the file is deleted immediately after it has been written, the last part of it is still in the SSD's cache and never actually makes it to flash storage.

Over the same period, the wear levelling count has gone up from 801 to 1029 - a total of 228. From the results in our previous post ’Estimating Reserved Space in the SSD’, this equates to:

228 * 0.9885TiB = 225.378TiB

And from this we can work out the write amplification factor:

225.378TiB / 217.59TiB = 1.036.

As before, this is a tiny level of write amplification, and indicates that the controller continues to do a fantastic job of minimising the impact of having a very large static file present. Wow!

Roaring Past The 1 Petabyte Mark

ONE PETABYTE WRITTEN!!! And still going strong - only 16% wear levelling used, 84% still to go:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1873
 12 Power_Cycle_Count       099   099   000    Old_age   161
177 Wear_Leveling_Count     084   084   000    Pre-fail  938
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 065   060   000    Old_age   35
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   1953192330681

This disk is storming on!  Watch this space for future updates.

Revisiting Write Amplification

Now let's take a look at the impact of having 50% of the disk occupied with a static file while the rest is written to repeatedly. What happens to write amplification - does the rate go up as the controller has to move data around to ensure the flash wears evenly? And by how much - do the figures indicate that there is a high degree of tolerance in different wear levels on individual blocks before background data movement takes place? We're starting to get into the nitty-gritty of the controller's behaviour and will see just how well it copes under reasonable stress. We'll start by looking at a couple of sets of SMART stats where the wear levelling count drops by 1%:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1371
 12 Power_Cycle_Count       099   099   000    Old_age   160
177 Wear_Leveling_Count     091   091   000    Pre-fail  485
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 063   060   000    Old_age   37
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   1030221277903
ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1436
 12 Power_Cycle_Count       099   099   000    Old_age   160
177 Wear_Leveling_Count     090   090   000    Pre-fail  546
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 065   060   000    Old_age   35
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   1155002071365

The volume of data written to cause this drop (taken from LBAs written, where each LBA is 512 bytes) is:

(1155002071365 - 1030221277903) * 512 = 63.89TB.

This is less than when previously measured, at 66.25TB, but still gives an estimated life for the disk of nearly 6.4PBW.

Now turning to measuring write amplification, we'll look at two more sets of SMART stats.

After run 511:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1347
 12 Power_Cycle_Count       099   099   000    Old_age   160
177 Wear_Leveling_Count     092   092   000    Pre-fail  464
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 063   060   000    Old_age   37
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   985300191097

After run 806:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1503
 12 Power_Cycle_Count       099   099   000    Old_age   160
177 Wear_Leveling_Count     089   089   000    Pre-fail  609
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 067   060   000    Old_age   33
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   1279782869530

Each test run writes 476GiB, so we can calculate the total data written during this period of the test from the number of test runs completed:

(806 - 511) * 476GiB / 1024 = 137.13TiB.

This correlates precisely with what the SSD has recorded as 'Total LBAs Written':

(1279782869530 - 985300191097) * 512 / (1024 * 1024 * 1024 * 1024) = 137.13TiB.

Over the same period, the wear levelling count has gone up from 464 to 609 - a total of 145. From the results in our previous post ’Estimating Reserved Space in the SSD’, this equates to:

145 * 0.9885TiB = 143.3325TiB

And from this we can work out the write amplification factor:

143.3325TiB / 137.13TiB = 1.045.

This is a tiny level of write amplification, and indicates that the controller is doing a fantastic job of minimising the impact of having a large static file present. The controller is clearly tolerating widely varying levels of wear across the SSD, only moving data when really necessary to ensure evenness. It will be interesting to compare this result with behaviour when the disk is 85% full - coming soon!

Stage 3b - Testing with SSD 85% Full

With 676TB written, and some interesting results regarding write amplification (coming in the next post), we have decided to modify this test to have a much larger static file. This time it will occupy 85% of the disk, and provide significantly more stress to the controller as it tries to ensure the flash wears evenly. A quick look at the SMART stats reveals that everything is still looking very healthy, so we will write another 150TB and then see how performance and longevity are affected:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1524
 12 Power_Cycle_Count       099   099   000    Old_age   160
177 Wear_Leveling_Count     089   089   000    Pre-fail  629
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 066   060   000    Old_age   34
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   1321709214696

Firmware Upgrade Issues

There has been a lot of talk in the press over the last few days about problems with upgrading the firmware in the 850 Pro series. Several people have reported apparently bricked SSDs after trying to install the latest version '2B6Q'. This had us concerned as we had just bought a number of new 850 Pro SSDs, and they all came with 2B6Q factory-installed. The question was whether it was the firmware itself that was at fault, or something about the upgrade process. We decided to contact Samsung for clarification, and had the following response which we thought was worth sharing:

"The firmware issues with this update were not caused by the firmware itself, but by the software solution. As you already have the newest firmware and it has been proven to function correctly and well, you can use the SSDs just as intended."

So it seems that the firmware itself is fine, and we will be upgrading our test SSD as soon as the process is fixed (the disk is currently running the previous version '1B6Q'). We'll let you know how things go.

Stage 3a - Testing with SSD 50% Full

So - 0.5PB written and time to move on to stage 3 of the test. For the next stage, we will half-fill the disk with a single large static file and then repeatedly fill the rest of the SSD with a second file. This will exercise the controller as it tries to balance wear across the whole of the flash, moving the static file around in the background, and should lead to more visible write amplification.

Test setup is very simple. The static file is created with:

# dd if=/dev/phony of=/data/static bs=1G count=476 oflag=direct
476+0 records in
476+0 records out
511101108224 bytes (511 GB) copied, 1863.89 s, 274 MB/s

And the test to be run repeatedly is:

dd if=/dev/phony of=/data/abc bs=1G count=476 oflag=direct

Once this has written approximately another 150TB, we will recalculate disk lifetime and write amplification and see how it compares with writing to an empty disk. It shouldn't take too long so check back soon.

Half a Petabyte and Still Going Strong!

Wow! Half a petabyte written and still 92% life left:

ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    Pre-fail  0
  9 Power_On_Hours          099   099   000    Old_age   1341
 12 Power_Cycle_Count       099   099   000    Old_age   160
177 Wear_Leveling_Count     092   092   000    Pre-fail  460
179 Used_Rsvd_Blk_Cnt_Tot   100   100   010    Pre-fail  0
181 Program_Fail_Cnt_Total  100   100   010    Old_age   0
182 Erase_Fail_Count_Total  100   100   010    Old_age   0
183 Runtime_Bad_Block       100   100   010    Pre-fail  0
187 Reported_Uncorrect      100   100   000    Old_age   0
190 Airflow_Temperature_Cel 063   060   000    Old_age   37
195 Hardware_ECC_Recovered  200   200   000    Old_age   0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   0
235 Unknown_Attribute       099   099   000    Old_age   28
241 Total_LBAs_Written      099   099   000    Old_age   977576355977

What more is there to say? Completely error-free so far, and still going strong. Watch this space!

Estimating Reserved Space in the SSD

Inside each SSD is a reserved section of storage that the controller can use for management and performance optimisation. It is also used for wear levelling, and accounts for the difference between the estimated and actual figures we saw in the previous post. We can estimate how large this space is from the measurements we taken so far, but first we need to know exactly how large the user area of the disk is. SMART helps with this by reporting 'User Capacity':

Device Model:     Samsung SSD 850 PRO 1TB
Firmware Version: EXM01B6Q
User Capacity:    1,024,209,543,168 bytes [1.02 TB]
Sector Size:      512 bytes logical/physical
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4c

Previously, we saw that writing 179.91TiB of data led to an increase in wear levelling count of 182. From this, we can calculate that each increment in the count is equivalent to:

179.91TiB / 182 = 0.9885TiB = 1086885367874 bytes.

Subtracting the user capacity, we get:

(1086885367874 - 1024209543168) / (1024 * 1024 * 1024) = 58.4GiB.

Working this out as a proportion of the overall disk capacity:

(58.4GiB / 1086885367874) * 100 = 5.77%.

That's a reasonable amount of space to set aside, and helps ensure that the disk runs efficiently, but according to Samsung does not include any room for over-provisioning - the user should allocate additional space for this.