民用SSD 关闭 写cache后的 糟糕性能表现
背景
昨天刚到手的一块SSD，建兴 ZETA 256GB的硬盘。
smartctl查看到的信息如下：
=== START OF INFORMATION SECTION ===
Device Model: LITEON LCH-256V2S
Serial Number: 002448118079
LU WWN Device Id: 5 002303 10035169e
Firmware Version: FC8020A
User Capacity: 256,060,514,304 bytes [256 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Dec 20 21:52:23 2015 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Write SCT (Get) XXX Error Recovery Control Command failed: scsi error aborted command
Wt Cache Reorder: N/A
在打开写缓存时，pg_test_fsync测试到的fsync写性能如下：
postgres@digoal-> pg_test_fsync
5 seconds per test
O_DIRECT supported on this platform for open_datasync and open_sync.
Compare file sync methods using one 8kB write:
(in wal_sync_method preference order, except fdatasync is Linux's default)
open_datasync 14640.971 ops/sec 68 usecs/op
fdatasync 13786.079 ops/sec 73 usecs/op
fsync 9384.938 ops/sec 107 usecs/op
fsync_writethrough n/a
open_sync 9606.181 ops/sec 104 usecs/op
Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync is Linux's default)
open_datasync 8157.733 ops/sec 123 usecs/op
fdatasync 10784.381 ops/sec 93 usecs/op
fsync 7117.690 ops/sec 140 usecs/op
fsync_writethrough n/a
open_sync 4913.559 ops/sec 204 usecs/op
Compare open_sync with different write sizes:
(This is designed to compare the cost of writing 16kB in different write
open_sync sizes.)
1 * 16kB open_sync write 7361.125 ops/sec 136 usecs/op
2 * 8kB open_sync writes 5274.986 ops/sec 190 usecs/op
4 * 4kB open_sync writes 3099.050 ops/sec 323 usecs/op
8 * 2kB open_sync writes 2048.264 ops/sec 488 usecs/op
16 * 1kB open_sync writes 1225.877 ops/sec 816 usecs/op
Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written on a different
descriptor.)
write, fsync, close 9403.945 ops/sec 106 usecs/op
write, close, fsync 9847.864 ops/sec 102 usecs/op
Non-sync'ed 8kB writes:
write 337716.595 ops/sec 3 usecs/op
看起来非常漂亮的性能指标。居然比我以前用的PCI-E接口的ocz还要好，有点不敢相信。
当我把硬盘的写缓存关闭后，再次测试，真实性能暴露。
[root@digoal parameters]# hdparm -W 0 /dev/sda
/dev/sda:
setting drive write-caching to 0 (off)
write-caching = 0 (off)
性能结果让人咂舌：
postgres@digoal-> pg_test_fsync
5 seconds per test
O_DIRECT supported on this platform for open_datasync and open_sync.
Compare file sync methods using one 8kB write:
(in wal_sync_method preference order, except fdatasync is Linux's default)
open_datasync 527.295 ops/sec 1896 usecs/op
fdatasync 519.180 ops/sec 1926 usecs/op
fsync 168.596 ops/sec 5931 usecs/op
fsync_writethrough n/a
open_sync 169.740 ops/sec 5891 usecs/op
Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync is Linux's default)
open_datasync 261.551 ops/sec 3823 usecs/op
fdatasync 517.381 ops/sec 1933 usecs/op
fsync 168.195 ops/sec 5945 usecs/op
fsync_writethrough n/a
open_sync 84.394 ops/sec 11849 usecs/op
Compare open_sync with different write sizes:
(This is designed to compare the cost of writing 16kB in different write
open_sync sizes.)
1 * 16kB open_sync write 168.287 ops/sec 5942 usecs/op
2 * 8kB open_sync writes 84.500 ops/sec 11834 usecs/op
4 * 4kB open_sync writes 42.232 ops/sec 23678 usecs/op
8 * 2kB open_sync writes 20.911 ops/sec 47823 usecs/op
16 * 1kB open_sync writes 10.288 ops/sec 97204 usecs/op
Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written on a different
descriptor.)
write, fsync, close 170.290 ops/sec 5872 usecs/op
write, close, fsync 171.121 ops/sec 5844 usecs/op
Non-sync'ed 8kB writes:
write 332315.202 ops/sec 3 usecs/op
用dd加direct,sync测试持续写入，在开启了写缓存的时候，性能还是很不错的，连续写入8GB数据。
但是在写入过程中，使用smartctl -x /dev/sda查看硬盘的信息时，又hang了。iowait飙升到100，问题可能还和NCQ有关，见上一篇文章。
如果这是一个普遍现象，企业用的话，还是要企业级的产品。