如何分析D状态进程

背景

在使用top查看进程状态时，我们有时候会看到D状态的进程。

       w: S  --  Process Status  
          The status of the task which can be one of:  
             ’D’ = uninterruptible sleep  
             ’R’ = running  
             ’S’ = sleeping  
             ’T’ = traced or stopped  
             ’Z’ = zombie  

D是一种不可中断的sleep，如果你发现大量的D状态的进程，这个时候这些进程实际上是没有在处理业务逻辑的。

例如使用PostgreSQL时，批量的往数据库导入数据，如果导入的数据量大到OS脏页回写的速度赶不上写入的速度时，并且用户刷dirty page的阈值到达，用户进程会需要主动刷脏页。

vm.dirty_background_ratio = 10  
vm.dirty_background_bytes = 0  
vm.dirty_ratio = 20  
vm.dirty_bytes = 0  
vm.dirty_writeback_centisecs = 50  
vm.dirty_expire_centisecs = 6000    

例如以上配置，OS脏页超过20%时，用户调write也需要主动的刷脏页，就会看到进程处于D状态，直到脏页水位下降到10%以下。

当然还有其他的原因会导致进程进入D状态，我们需要观察进程的stack，看看它处于什么状态。

例如处于R状态的COPY PostgreSQL进程，它的stack是什么样的？

cat /proc/17944/status ; echo -e "\n"; cat /proc/17944/stack  
Name:   postgres  
State:  R (running)  
Tgid:   17944  
Pid:    17944  
PPid:   57925  
TracerPid:      0  
Uid:    123293  123293  123293  123293  
Gid:    100     100     100     100  
Utrace: 0  
FDSize: 64  
Groups: 100 19001   
VmPeak: 272294920 kB  
VmSize:   119788 kB  
VmLck:         0 kB  
VmHWM:      3244 kB  
VmRSS:      2812 kB  
VmData:     2140 kB  
VmStk:       152 kB  
VmExe:      5852 kB  
VmLib:      2400 kB  
VmPTE:        64 kB  
VmSwap:        0 kB  
Threads:        1  
SigQ:   0/4131614  
SigPnd: 0000000000000000  
ShdPnd: 0000000000000000  
SigBlk: 0000000000000000  
SigIgn: 0000000001301800  
SigCgt: 0000000180006287  
CapInh: 0000000000000000  
CapPrm: 0000000000000000  
CapEff: 0000000000000000  
CapBnd: ffffffffffffffff  
Cpus_allowed:   ffffffff,ffffffff  
Cpus_allowed_list:      0-63  
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001  
Mems_allowed_list:      0  
voluntary_ctxt_switches:        55758  
nonvoluntary_ctxt_switches:     103995  
  
  
[<ffffffff8121b40a>] sys_semtimedop+0x81a/0x840  
[<ffffffffffffffff>] 0xffffffffffffffff  

例如处于D状态的COPY PostgreSQL进程，它的stack是什么样的？

可以看到它处于刷脏页速度受限的状态，与ext4内核有关。

cat /proc/17944/status ; echo -e "\n"; cat /proc/17944/stack  
Name:   postgres  
State:  D (disk sleep)  
Tgid:   17944  
Pid:    17944  
PPid:   57925  
TracerPid:      0  
Uid:    123293  123293  123293  123293  
Gid:    100     100     100     100  
Utrace: 0  
FDSize: 64  
Groups: 100 19001   
VmPeak: 272294920 kB  
VmSize:   119788 kB  
VmLck:         0 kB  
VmHWM:      3244 kB  
VmRSS:      2812 kB  
VmData:     2140 kB  
VmStk:       152 kB  
VmExe:      5852 kB  
VmLib:      2400 kB  
VmPTE:        64 kB  
VmSwap:        0 kB  
Threads:        1  
SigQ:   0/4131614  
SigPnd: 0000000000000000  
ShdPnd: 0000000000000000  
SigBlk: 0000000000000000  
SigIgn: 0000000001301800  
SigCgt: 0000000180006287  
CapInh: 0000000000000000  
CapPrm: 0000000000000000  
CapEff: 0000000000000000  
CapBnd: ffffffffffffffff  
Cpus_allowed:   ffffffff,ffffffff  
Cpus_allowed_list:      0-63  
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001  
Mems_allowed_list:      0  
voluntary_ctxt_switches:        55922  
nonvoluntary_ctxt_switches:     104189  
  
  
[<ffffffff81133cf0>] balance_dirty_pages_ratelimited_nr+0x2d0/0x9a0  
[<ffffffff8111f19a>] generic_file_buffered_write+0x1da/0x2e0  
[<ffffffff81120fe0>] __generic_file_aio_write+0x260/0x490  
[<ffffffff81121298>] generic_file_aio_write+0x88/0x100  
[<ffffffffa00b9463>] ext4_file_write+0x43/0xe0 [ext4]  
[<ffffffff8118863a>] do_sync_write+0xfa/0x140  
[<ffffffff81188938>] vfs_write+0xb8/0x1a0  
[<ffffffff81189231>] sys_write+0x51/0x90  
[<ffffffff8100c072>] system_call_fastpath+0x16/0x1b  
[<ffffffffffffffff>] 0xffffffffffffffff  

digoal’s 大量PostgreSQL文章入口