Jekyll2018-11-09T17:22:23+08:00https://billtian.github.io/digoal.blog/Digoal.Zhou’s BlogBill Tian's Blog. 记录技术成长及生活感误Bill Tianbilltian945@gmail.comPostgreSQL(PPAS 兼容Oracle) 从零开始入门手册 - 珍藏版2018-11-02T00:00:00+08:002018-11-02T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/11/02/02<h2 id="背景">背景</h2>
<p><a href="https://www.aliyun.com/product/rds/ppas">云数据库PPAS版</a>,是阿里云与EnterpriseDB公司(简称EDB)合作基于PostgreSQL高度兼容Oracle语法的数据库服务,为用户提供易于操作的迁移工具,兼容范围涵盖:PL/SQL、数据类型、高级函数、表分区等。</p>
<p>用户可以直接在阿里云<a href="https://www.aliyun.com/product/rds/ppas">购买PPAS</a>进行使用。</p>
<p>如果在购买PPAS前,想试用一下,可以去EDB网站下载,并部署相关的插件。</p>
<h1 id="一测试环境安装部署edb">一、测试环境安装部署EDB</h1>
<h2 id="服务器">服务器</h2>
<p>以阿里云ECS为例。</p>
<p>1、CPU</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
Stepping: 4
CPU MHz: 2500.008
BogoMIPS: 5000.01
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 33792K
NUMA node0 CPU(s): 0-63
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1
</code></pre></div></div>
<p>2、DISK</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 200G 0 disk
└─vda1 253:1 0 200G 0 part /
vdb 253:16 0 1.8T 0 disk
├─vgdata01-lv01 252:0 0 4T 0 lvm /data01
├─vgdata01-lv02 252:1 0 4T 0 lvm /data02
├─vgdata01-lv03 252:2 0 4T 0 lvm /data03
└─vgdata01-lv04 252:3 0 2T 0 lvm /data04
vdc 253:32 0 1.8T 0 disk
├─vgdata01-lv01 252:0 0 4T 0 lvm /data01
├─vgdata01-lv02 252:1 0 4T 0 lvm /data02
├─vgdata01-lv03 252:2 0 4T 0 lvm /data03
└─vgdata01-lv04 252:3 0 2T 0 lvm /data04
vdd 253:48 0 1.8T 0 disk
├─vgdata01-lv01 252:0 0 4T 0 lvm /data01
├─vgdata01-lv02 252:1 0 4T 0 lvm /data02
├─vgdata01-lv03 252:2 0 4T 0 lvm /data03
└─vgdata01-lv04 252:3 0 2T 0 lvm /data04
vde 253:64 0 1.8T 0 disk
├─vgdata01-lv01 252:0 0 4T 0 lvm /data01
├─vgdata01-lv02 252:1 0 4T 0 lvm /data02
├─vgdata01-lv03 252:2 0 4T 0 lvm /data03
└─vgdata01-lv04 252:3 0 2T 0 lvm /data04
vdf 253:80 0 1.8T 0 disk
├─vgdata01-lv01 252:0 0 4T 0 lvm /data01
├─vgdata01-lv02 252:1 0 4T 0 lvm /data02
├─vgdata01-lv03 252:2 0 4T 0 lvm /data03
└─vgdata01-lv04 252:3 0 2T 0 lvm /data04
vdg 253:96 0 1.8T 0 disk
├─vgdata01-lv01 252:0 0 4T 0 lvm /data01
├─vgdata01-lv02 252:1 0 4T 0 lvm /data02
├─vgdata01-lv03 252:2 0 4T 0 lvm /data03
└─vgdata01-lv04 252:3 0 2T 0 lvm /data04
vdh 253:112 0 1.8T 0 disk
├─vgdata01-lv01 252:0 0 4T 0 lvm /data01
├─vgdata01-lv02 252:1 0 4T 0 lvm /data02
├─vgdata01-lv03 252:2 0 4T 0 lvm /data03
└─vgdata01-lv04 252:3 0 2T 0 lvm /data04
vdi 253:128 0 1.8T 0 disk
├─vgdata01-lv01 252:0 0 4T 0 lvm /data01
├─vgdata01-lv02 252:1 0 4T 0 lvm /data02
├─vgdata01-lv03 252:2 0 4T 0 lvm /data03
└─vgdata01-lv04 252:3 0 2T 0 lvm /data04
</code></pre></div></div>
<p>3、MEM</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># free -m
total used free shared buff/cache available
Mem: 515815 318010 109199 8 88605 194736
Swap: 0 0 0
</code></pre></div></div>
<h2 id="操作系统">操作系统</h2>
<p>以CentOS 7.x x64为例。</p>
<p>1、存储</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dd if=/dev/zero of=/dev/vdb bs=1024 count=1024
dd if=/dev/zero of=/dev/vdc bs=1024 count=1024
dd if=/dev/zero of=/dev/vdd bs=1024 count=1024
dd if=/dev/zero of=/dev/vde bs=1024 count=1024
dd if=/dev/zero of=/dev/vdf bs=1024 count=1024
dd if=/dev/zero of=/dev/vdg bs=1024 count=1024
dd if=/dev/zero of=/dev/vdh bs=1024 count=1024
dd if=/dev/zero of=/dev/vdi bs=1024 count=1024
pvcreate /dev/vd[b-i]
vgcreate -A y -s 128M vgdata01 /dev/vd[b-i]
lvcreate -A y -i 8 -I 8 -L 4096GiB -n lv01 vgdata01
lvcreate -A y -i 8 -I 8 -L 4096GiB -n lv02 vgdata01
lvcreate -A y -i 8 -I 8 -L 4096GiB -n lv03 vgdata01
</code></pre></div></div>
<p>2、文件系统</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkfs.ext4 /dev/mapper/vgdata01-lv01 -m 0 -O extent,uninit_bg -E lazy_itable_init=1,stride=2,stripe_width=16 -b 4096 -T largefile -L lv01
mkfs.ext4 /dev/mapper/vgdata01-lv02 -m 0 -O extent,uninit_bg -E lazy_itable_init=1,stride=2,stripe_width=16 -b 4096 -T largefile -L lv02
mkfs.ext4 /dev/mapper/vgdata01-lv03 -m 0 -O extent,uninit_bg -E lazy_itable_init=1,stride=2,stripe_width=16 -b 4096 -T largefile -L lv03
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vi /etc/fstab
LABEL=lv01 /data01 ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0,data=writeback 0 0
LABEL=lv02 /data02 ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0,data=writeback 0 0
LABEL=lv03 /data03 ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0,data=writeback 0 0
mkdir /data01
mkdir /data02
mkdir /data03
mount -a
</code></pre></div></div>
<p>3、内核参数</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vi /etc/sysctl.conf
# add by digoal.zhou
fs.aio-max-nr = 1048576
fs.file-max = 76724600
# 可选:kernel.core_pattern = /data01/corefiles/core_%e_%u_%t_%s.%p
# /data01/corefiles 事先建好,权限777,如果是软链接,对应的目录修改为777
kernel.sem = 4096 2147483647 2147483646 512000
# 信号量, ipcs -l 或 -u 查看,每16个进程一组,每组信号量需要17个信号量。
kernel.shmall = 107374182
# 所有共享内存段相加大小限制 (建议内存的80%),单位为页。
kernel.shmmax = 274877906944
# 最大单个共享内存段大小 (建议为内存一半), >9.2的版本已大幅降低共享内存的使用,单位为字节。
kernel.shmmni = 819200
# 一共能生成多少共享内存段,每个PG数据库集群至少2个共享内存段
net.core.netdev_max_backlog = 10000
net.core.rmem_default = 262144
# The default setting of the socket receive buffer in bytes.
net.core.rmem_max = 4194304
# The maximum receive socket buffer size in bytes
net.core.wmem_default = 262144
# The default setting (in bytes) of the socket send buffer.
net.core.wmem_max = 4194304
# The maximum send socket buffer size in bytes.
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_keepalive_intvl = 20
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_mem = 8388608 12582912 16777216
net.ipv4.tcp_fin_timeout = 5
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syncookies = 1
# 开启SYN Cookies。当出现SYN等待队列溢出时,启用cookie来处理,可防范少量的SYN攻击
net.ipv4.tcp_timestamps = 1
# 减少time_wait
net.ipv4.tcp_tw_recycle = 0
# 如果=1则开启TCP连接中TIME-WAIT套接字的快速回收,但是NAT环境可能导致连接失败,建议服务端关闭它
net.ipv4.tcp_tw_reuse = 1
# 开启重用。允许将TIME-WAIT套接字重新用于新的TCP连接
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_wmem = 8192 65536 16777216
net.nf_conntrack_max = 1200000
net.netfilter.nf_conntrack_max = 1200000
vm.dirty_background_bytes = 409600000
# 系统脏页到达这个值,系统后台刷脏页调度进程 pdflush(或其他) 自动将(dirty_expire_centisecs/100)秒前的脏页刷到磁盘
# 默认为10%,大内存机器建议调整为直接指定多少字节
vm.dirty_expire_centisecs = 3000
# 比这个值老的脏页,将被刷到磁盘。3000表示30秒。
vm.dirty_ratio = 95
# 如果系统进程刷脏页太慢,使得系统脏页超过内存 95 % 时,则用户进程如果有写磁盘的操作(如fsync, fdatasync等调用),则需要主动把系统脏页刷出。
# 有效防止用户进程刷脏页,在单机多实例,并且使用CGROUP限制单实例IOPS的情况下非常有效。
vm.dirty_writeback_centisecs = 100
# pdflush(或其他)后台刷脏页进程的唤醒间隔, 100表示1秒。
vm.swappiness = 0
# 不使用交换分区
vm.mmap_min_addr = 65536
vm.overcommit_memory = 0
# 在分配内存时,允许少量over malloc, 如果设置为 1, 则认为总是有足够的内存,内存较少的测试环境可以使用 1 .
vm.overcommit_ratio = 90
# 当overcommit_memory = 2 时,用于参与计算允许指派的内存大小。
vm.swappiness = 0
# 关闭交换分区
vm.zone_reclaim_mode = 0
# 禁用 numa, 或者在vmlinux中禁止.
net.ipv4.ip_local_port_range = 40000 65535
# 本地自动分配的TCP, UDP端口号范围
fs.nr_open=20480000
# 单个进程允许打开的文件句柄上限
# 以下参数请注意
# vm.extra_free_kbytes = 4096000
# vm.min_free_kbytes = 2097152
# 如果是小内存机器,以上两个值不建议设置
# vm.nr_hugepages = 66536
# 建议shared buffer设置超过64GB时 使用大页,页大小 /proc/meminfo Hugepagesize
# vm.lowmem_reserve_ratio = 1 1 1
# 对于内存大于64G时,建议设置,否则建议默认值 256 256 32
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sysctl -p
</code></pre></div></div>
<p>4、资源限制</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vi /etc/security/limits.d/20-nproc.conf
# nofile超过1048576的话,一定要先将sysctl的fs.nr_open设置为更大的值,并生效后才能继续设置nofile.
* soft nofile 1024000
* hard nofile 1024000
* soft nproc unlimited
* hard nproc unlimited
* soft core unlimited
* hard core unlimited
* soft memlock unlimited
* hard memlock unlimited
</code></pre></div></div>
<p>5、关闭透明大页</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vi /etc/rc.local
touch /var/lock/subsys/local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
su - postgres -c "pg_ctl start"
</code></pre></div></div>
<p>6、防火墙</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>iptables 配置,略
</code></pre></div></div>
<p>7、selinux</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vi /etc/selinux/config
SELINUX=disabled
SELINUXTYPE=targeted
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># setenforce 0
setenforce: SELinux is disabled
</code></pre></div></div>
<p>8、依赖包</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yum -y install coreutils glib2 lrzsz dstat sysstat e4fsprogs xfsprogs ntp readline-devel zlib-devel openssl-devel pam-devel libxml2-devel libxslt-devel python-devel tcl-devel gcc gcc-c++ make smartmontools flex bison perl-devel perl-ExtUtils* openldap-devel jadetex openjade bzip2 iotop perf cmake3
</code></pre></div></div>
<p>建议重启一下机器。</p>
<h2 id="edb-数据库软件">EDB 数据库软件</h2>
<p>10.5 版本为例。</p>
<p>1、下载软件</p>
<p>https://www.enterprisedb.com/advanced-downloads</p>
<p>OPTION A - EDB POSTGRES™ ADVANCED SERVER</p>
<p>v10.5 , Linux x86-64 , Interactive Installer</p>
<p>注意:</p>
<p>Note: For the Enterprise subscription , choose EDB Postgres™ Advanced Server.</p>
<p>For the Standard subscription, choose PostgreSQL.</p>
<p>For the Developer subscription, choose either database.</p>
<p>The Developer subscription is for use in non-production environments only.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget https://get.enterprisedb.com/advstacks/edb-as10-server-10.5.12-1-linux-x64.run
</code></pre></div></div>
<p>2、安装软件</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>su - root
chmod 700 edb-as10-server-10.5.12-1-linux-x64.run
./edb-as10-server-10.5.12-1-linux-x64.run
Press [Enter] to continue:
Do you accept this license? [y/n]: y
同意许可。
指定EDB软件安装目录
Installation Directory [/opt/edb/as10]:
----------------------------------------------------------------------------
Select the components you want to install; clear the components you do not want
to install. Click Next when you are ready to continue.
选择要安装的组件
EDB Postgres Advanced Server [Y/n] :Y
pgAdmin 4 [Y/n] :n
StackBuilder Plus [Y/n] :n
Command Line Tools [Y/n] :Y
Is the selection above correct? [Y/n]: Y
</code></pre></div></div>
<p>2.1、数据、REDO目录</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># mkdir /data04/ppas10
# mkdir /data04/ppas10_wal
</code></pre></div></div>
<p>2.2、</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>----------------------------------------------------------------------------
Additional Directories
指定数据、REDO目录
Please select a directory under which to store your data.
Data Directory [/opt/edb/as10/data]: /data04/ppas10
Please select a directory under which to store your Write-Ahead Logs.
Write-Ahead Log (WAL) Directory [/opt/edb/as10/data/pg_wal]: /data04/ppas10_wal
----------------------------------------------------------------------------
Advanced Server Dialect
EDB Postgres Advanced Server can be configured in one of two "Dialects" - 1) Compatible with Oracle or 2) Compatible with Postgres.
If you select Compatible with Oracle, Advanced Server will be configured with appropriate data type conversions,
time and date formats, Oracle-styled operators, dictionary views and more.
This makes it easier to migrate or write new applications that are more compatible with the Oracle database.
If you select Compatible with Postgres, Advanced Server will be configured with standard PostgeSQL data types, time/date formats and operators.
指定兼容PostgreSQL还是兼容Oracle模式。
Advanced Server Dialect
[1] Compatible with Oracle
[2] Compatible with Postgres
Please choose an option [1] : 1
----------------------------------------------------------------------------
Please provide a password for the database superuser (enterprisedb). A locked
Unix user account (enterprisedb) will be created if not present.
设置数据库超级用户enterprisedb的密码
Password :
Retype Password :
----------------------------------------------------------------------------
Additional Configuration
Please select the port number the server should listen on.
设置数据库监听端口
Port [5444]: 4001
Select the locale to be used by the new database cluster.
Locale
[1] [Default locale]
....
[232] en_US
[233] en_US.iso88591
[234] en_US.iso885915
[235] en_US.utf8
...
[762] zh_CN
[763] zh_CN.gb2312
[764] zh_CN.utf8
[765] zh_HK.utf8
[766] zh_SG
[767] zh_SG.gb2312
[768] zh_SG.utf8
[769] zh_TW.euctw
[770] zh_TW.utf8
[771] zu_ZA
[772] zu_ZA.iso88591
[773] zu_ZA.utf8
设置location, 字符集
Please choose an option [1] : 235
Would you like to install sample tables and procedures?
是否安装样例模板
Install sample tables and procedures. [Y/n]: n
----------------------------------------------------------------------------
Dynatune Dynamic Tuning:
Server Utilization
Please select the type of server to determine the amount of system resources
that may be utilized:
配置数据库所在服务器的属性(独占,开发模式,混合模式)。
[1] Development (e.g. a developer's laptop)
[2] General Purpose (e.g. a web or application server)
[3] Dedicated (a server running only Advanced Server)
Please choose an option [2] : 3
----------------------------------------------------------------------------
Dynatune Dynamic Tuning:
Workload Profile
Please select the type of workload this server will be used for:
设置数据库的使用场景,OLTP或OLAP或混合模式
[1] Transaction Processing (OLTP systems)
[2] General Purpose (OLTP and reporting workloads)
[3] Reporting (Complex queries or OLAP workloads)
Please choose an option [1] : 2
----------------------------------------------------------------------------
Pre Installation Summary
The following settings will be used for the installation::
再次确认配置
Installation Directory: /opt/edb/as10
Server Installation Directory: /opt/edb/as10
Data Directory: /data04/ppas10
WAL Directory: /data04/ppas10_wal
Database Port: 4001
Database Superuser: enterprisedb
Operating System Account: enterprisedb
Database Service: edb-as-10
Command Line Tools Installation Directory: /opt/edb/as10
Press [Enter] to continue:
----------------------------------------------------------------------------
Setup is now ready to begin installing EDB Postgres Advanced Server on your
computer.
开始安装
Do you want to continue? [Y/n]: Y
----------------------------------------------------------------------------
Please wait while Setup installs EDB Postgres Advanced Server on your computer.
Installing EDB Postgres Advanced Server
0% ______________ 50% ______________ 100%
########################################
</code></pre></div></div>
<p>3、配置enterprisedb用户环境变量</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>修改软件目录OWNER
# chown -R enterprisedb:enterprisedb /opt/edb/as10
配置enterprisedb用户的环境变量
# vi /opt/edb/as10/.bash_profile
export PS1="$USER@`/bin/hostname -s`-> "
export PGPORT=4001
export PGDATA=/data04/ppas10
export LANG=en_US.utf8
export PGHOME=/opt/edb/as10
export LD_LIBRARY_PATH=$PGHOME/lib:/lib64:/usr/lib64:/usr/local/lib64:/lib:/usr/lib:/usr/local/lib:$LD_LIBRARY_PATH
export DATE=`date +"%Y%m%d%H%M"`
export PATH=$PGHOME/bin:$PATH:.
export MANPATH=$PGHOME/share/man:$MANPATH
export PGHOST=$PGDATA
export PGUSER=enterprisedb
export PGDATABASE=postgres
alias rm='rm -i'
alias ll='ls -lh'
alias cmake=cmake3
unalias vi
</code></pre></div></div>
<p>4、初始化数据库集群(安装软件时已初始化,以下略)</p>
<p>如果你需要在一台服务器上初始化多个数据库实例,可以参照执行:</p>
<p>数据目录与WAL日志目录(注意,这些是数据和redo日志目录,所以必须有别于其他数据库实例已经存在的目录。)</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># mkdir ppas10_8001
# mkdir /data04/ppas10_wal_8001
# chown -R enterprisedb:enterprisedb /data04/ppas10*
# chmod 700 /data04/ppas10_wal_8001
su - enterprisedb
</code></pre></div></div>
<p>如果你想兼容Oracle,使用如下手段初始化</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>initdb -D ppas10_8001 -E UTF8 --lc-collate=C --lc-ctype=en_US.utf8 -U enterprisedb -W -X /data04/ppas10_wal_8001 --redwood-like
</code></pre></div></div>
<p>如果你想兼容PostgreSQL,使用如下手段初始化</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>initdb -D ppas10_8001 -E UTF8 --lc-collate=C --lc-ctype=en_US.utf8 -U enterprisedb -W -X /data04/ppas10_wal_8001
</code></pre></div></div>
<p>当然,在初始化后,还可以通过修改参数来实现兼容Oracle或PostgreSQL</p>
<p><a href="../201805/20180522_04.md">《EDB PPAS (PostgreSQL plus advanced server) 10 参数模板 - 珍藏级》</a></p>
<p>5、安装Oracle OCI(可选)</p>
<p>安装Oracle OCI</p>
<p><a href="../201801/20180119_01.md">《PostgreSQL 商用版本EPAS(阿里云ppas) - 测试环境部署(EPAS 安装、配置、管理、Oracle DBLINK、外表)》</a></p>
<p>安装Oracle OCI。这样才可以在EDB数据库中建立ORACLE的DBLINK。</p>
<p>http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html</p>
<p>选择下载包含OCI的包( 需要输入Oracle的账号密码,可以免费注册。)</p>
<p><img src="../201801/20180119_01_pic_005.jpg" alt="pic" /></p>
<p><img src="../201801/20180119_01_pic_006.jpg" alt="pic" /></p>
<p>将安装包上传到服务器,解压,放到EDB的PGHOME目录(本文/opt/edb/as10),并建立软链。详情参考</p>
<p>https://www.enterprisedb.com/docs/en/10.0/Ora_Compat_Dev_Guide/Database_Compatibility_for_Oracle_Developers_Guide.1.124.html#</p>
<p>操作如下</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># cd /opt/edb/as10
[root@digoal ~]# mkdir oci
[root@digoal ~]# mv instantclient-basic-linux.x64-12.2.0.1.0.zip oci/
[root@digoal ~]# cd oci
[root@digoal oci]# ll
total 67356
-rw-r--r-- 1 root root 68965195 Jan 19 11:00 instantclient-basic-linux.x64-12.2.0.1.0.zip
[root@digoal oci]# unzip instantclient-basic-linux.x64-12.2.0.1.0.zip
Archive: instantclient-basic-linux.x64-12.2.0.1.0.zip
inflating: instantclient_12_2/adrci
inflating: instantclient_12_2/BASIC_README
inflating: instantclient_12_2/genezi
inflating: instantclient_12_2/libclntshcore.so.12.1
inflating: instantclient_12_2/libclntsh.so.12.1
inflating: instantclient_12_2/libipc1.so
inflating: instantclient_12_2/libmql1.so
inflating: instantclient_12_2/libnnz12.so
inflating: instantclient_12_2/libocci.so.12.1
inflating: instantclient_12_2/libociei.so
inflating: instantclient_12_2/libocijdbc12.so
inflating: instantclient_12_2/libons.so
inflating: instantclient_12_2/liboramysql12.so
inflating: instantclient_12_2/ojdbc8.jar
inflating: instantclient_12_2/uidrvci
inflating: instantclient_12_2/xstreams.jar
[root@digoal oci]# ll
total 67360
drwxr-xr-x 2 root root 4096 Jan 19 11:02 instantclient_12_2
-rw-r--r-- 1 root root 68965195 Jan 19 11:00 instantclient-basic-linux.x64-12.2.0.1.0.zip
[root@digoal oci]# cd instantclient_12_2/
[root@digoal instantclient_12_2]# ll
total 216696
-rwxrwxr-x 1 root root 44220 Jan 26 2017 adrci
-rw-rw-r-- 1 root root 363 Jan 26 2017 BASIC_README
-rwxrwxr-x 1 root root 57272 Jan 26 2017 genezi
-rwxrwxr-x 1 root root 8033199 Jan 26 2017 libclntshcore.so.12.1
-rwxrwxr-x 1 root root 71638263 Jan 26 2017 libclntsh.so.12.1
-r-xr-xr-x 1 root root 2981501 Jan 26 2017 libipc1.so
-r-xr-xr-x 1 root root 539065 Jan 26 2017 libmql1.so
-r-xr-xr-x 1 root root 6568149 Jan 26 2017 libnnz12.so
-rwxrwxr-x 1 root root 2218687 Jan 26 2017 libocci.so.12.1
-rwxrwxr-x 1 root root 124771800 Jan 26 2017 libociei.so
-r-xr-xr-x 1 root root 158543 Jan 26 2017 libocijdbc12.so
-r-xr-xr-x 1 root root 380996 Jan 26 2017 libons.so
-rwxrwxr-x 1 root root 116563 Jan 26 2017 liboramysql12.so
-r--r--r-- 1 root root 4036257 Jan 26 2017 ojdbc8.jar
-rwxrwxr-x 1 root root 240476 Jan 26 2017 uidrvci
-rw-rw-r-- 1 root root 74230 Jan 26 2017 xstreams.jar
[root@digoal instantclient_12_2]# cp libclntsh.so.12.1 /opt/edb/as10/lib/
[root@digoal instantclient_12_2]# cd /opt/edb/as10/lib
[root@digoal lib]# ln -s libclntsh.so.12.1 libclntsh.so
</code></pre></div></div>
<p>6、创建DBLINK,创建ORACLE外部表,略。参考上面的连接。</p>
<p>7、配置数据库参数</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>su - enterprisedb
cd $PGDATA
vi postgresql.conf
listen_addresses = '0.0.0.0'
port = 4001 # 多实例时,不同的数据库端口必须不同。
max_connections = 4000
superuser_reserved_connections = 13
unix_socket_directories = '.,/tmp'
unix_socket_permissions = 0700
row_security = on
tcp_keepalives_idle = 45
tcp_keepalives_interval = 10
tcp_keepalives_count = 10
shared_buffers = 32GB
huge_pages = try
max_prepared_transactions = 4000
work_mem = 8MB
maintenance_work_mem = 2GB
autovacuum_work_mem = 2GB
max_stack_depth = 4MB
dynamic_shared_memory_type = posix
shared_preload_libraries = '$libdir/dbms_pipe,$libdir/edb_gen,$libdir/dbms_aq'
vacuum_cost_delay = 0
bgwriter_delay = 10ms
bgwriter_lru_maxpages = 1000
bgwriter_lru_multiplier = 5.0
effective_io_concurrency = 0
max_worker_processes = 64
max_parallel_workers_per_gather = 8
max_parallel_workers = 24
wal_level = replica
synchronous_commit = off # 异步提交
full_page_writes = on # 如果文件系统支持COW,可以关闭。如果块设备支持8K(block_size决定)原子写,也可以关闭。开启有一定性能影响
wal_buffers = 64MB
wal_writer_delay = 10ms
wal_writer_flush_after = 1MB
checkpoint_timeout = 35min
max_wal_size = 64GB
min_wal_size = 16GB
checkpoint_completion_target = 0.1
archive_mode = on
archive_command = '/usr/bin/true'
max_wal_senders = 16
wal_keep_segments = 2048
wal_sender_timeout = 30s
max_replication_slots = 16
hot_standby = on
max_standby_archive_delay = 300s
max_standby_streaming_delay = 300s
wal_receiver_status_interval = 1s
wal_receiver_timeout = 30s
max_logical_replication_workers = 16
max_sync_workers_per_subscription = 4
random_page_cost = 1.1
effective_cache_size = 400GB
log_destination = 'csvlog'
logging_collector = on
log_truncate_on_rotation = on
log_min_duration_statement = 10s
log_checkpoints = on
log_connections = on
log_disconnections = on
log_error_verbosity = verbose
log_line_prefix = '%t '
log_lock_waits = on
log_statement = 'ddl'
log_temp_files = 0
log_timezone = 'PRC'
track_io_timing = on # 如果不需要跟踪IO的时间,可以关闭,开启有一定性能影响
track_functions = all
track_activity_query_size = 2048
autovacuum = on
log_autovacuum_min_duration = 0
autovacuum_max_workers = 6
autovacuum_freeze_max_age = 1200000000
autovacuum_multixact_freeze_max_age = 1400000000
autovacuum_vacuum_cost_delay = 0
statement_timeout = 300s
lock_timeout = 15s
idle_in_transaction_session_timeout = 60s
vacuum_freeze_table_age = 1150000000
vacuum_multixact_freeze_table_age = 1150000000
datestyle = 'redwood,show_time'
timezone = 'PRC'
lc_messages = 'en_US.utf8'
lc_monetary = 'en_US.utf8'
lc_numeric = 'en_US.utf8'
lc_time = 'en_US.utf8'
default_text_search_config = 'pg_catalog.english'
edb_redwood_date = on
edb_redwood_greatest_least = on
edb_redwood_strings = on
db_dialect = 'redwood'
edb_dynatune = 100
edb_dynatune_profile = mixed
timed_statistics = off
</code></pre></div></div>
<p>8、数据库防火墙配置</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>su - enterprisedb
cd $PGDATA
vi pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local replication all trust
host replication all 127.0.0.1/32 trust
host replication all ::1/128 trust
host all all 0.0.0.0/0 md5
</code></pre></div></div>
<p>9、重启数据库</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pg_ctl restart -m fast
</code></pre></div></div>
<h1 id="二ide">二、IDE</h1>
<h2 id="cli">cli</h2>
<p>https://www.postgresql.org/docs/current/static/app-psql.html</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enterprisedb@pg11-test-> psql
psql.bin (10.5.12)
Type "help" for help.
postgres=# \dt
Did not find any relations.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | ICU | Access privileges
-----------+--------------+----------+------------+------------+-----+-------------------------------
edb | enterprisedb | UTF8 | en_US.utf8 | en_US.utf8 | |
postgres | enterprisedb | UTF8 | en_US.utf8 | en_US.utf8 | |
template0 | enterprisedb | UTF8 | en_US.utf8 | en_US.utf8 | | =c/enterprisedb +
| | | | | | enterprisedb=CTc/enterprisedb
template1 | enterprisedb | UTF8 | en_US.utf8 | en_US.utf8 | | =c/enterprisedb +
| | | | | | enterprisedb=CTc/enterprisedb
(4 rows)
</code></pre></div></div>
<h2 id="gui">gui</h2>
<p>1、pem (Enterprisedb提供的GUI工具)</p>
<p>https://www.enterprisedb.com/software-downloads-postgres</p>
<p>2、pgadmin</p>
<p>https://www.pgadmin.org/</p>
<p><a href="../201810/20181018_03.md">《阿里云ppas 逻辑备份(导出)、还原 - 导出到本地、从本地导入》</a></p>
<h1 id="三压测">三、压测</h1>
<h2 id="tpc-b">TPC-B</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>su - enterprisedb
</code></pre></div></div>
<p>初始化1亿TPC-B数据</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pgbench -i -s 1000
</code></pre></div></div>
<p>只读测试</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pgbench -M prepared -n -r -P 1 -c 64 -j 64 -T 120 -S
transaction type: <builtin: select only>
scaling factor: 1000
query mode: prepared
number of clients: 64
number of threads: 64
duration: 120 s
number of transactions actually processed: 125854784
latency average = 0.061 ms
latency stddev = 0.062 ms
tps = 1048623.422874 (including connections establishing)
tps = 1048713.313561 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.001 \set aid random(1, 100000 * :scale)
0.060 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
</code></pre></div></div>
<p>读写混合测试</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pgbench -M prepared -n -r -P 1 -c 64 -j 64 -T 120
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1000
query mode: prepared
number of clients: 64
number of threads: 64
duration: 120 s
number of transactions actually processed: 10363880
latency average = 0.741 ms
latency stddev = 1.315 ms
tps = 86351.412706 (including connections establishing)
tps = 86359.549707 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.001 \set delta random(-5000, 5000)
0.050 BEGIN;
0.155 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.077 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.109 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.102 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.083 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
0.162 END;
</code></pre></div></div>
<h2 id="tpc-c">TPC-C</h2>
<p>参考</p>
<p><a href="../201809/20180913_01.md">《PostgreSQL 11 tpcc 测试(103万tpmC on ECS) - use sysbench-tpcc by Percona-Lab》</a></p>
<h1 id="四插件">四、插件</h1>
<p>https://www.enterprisedb.com/advanced-downloads</p>
<p>插件介绍、安装、使用</p>
<h2 id="图像">图像</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[root@pg11-test ~]# cd
[root@pg11-test ~]# . /opt/edb/as10/.bash_profile
-bash: unalias: vi: not found
root@pg11-test-> git clone https://github.com/postgrespro/imgsmlr
Cloning into 'imgsmlr'...
remote: Enumerating objects: 146, done.
remote: Total 146 (delta 0), reused 0 (delta 0), pack-reused 146
Receiving objects: 100% (146/146), 241.11 KiB | 149.00 KiB/s, done.
Resolving deltas: 100% (69/69), done.
root@pg11-test-> cd imgsmlr
root@pg11-test-> USE_PGXS=1 make clean
rm -f imgsmlr.so libimgsmlr.a libimgsmlr.pc
rm -f imgsmlr.o imgsmlr_idx.o
rm -rf data/*.hex
rm -rf results/ regression.diffs regression.out tmp_check/ tmp_check_iso/ log/ output_iso/
root@pg11-test-> USE_PGXS=1 make
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -DLINUX_OOM_ADJ=0 -O2 -DMAP_HUGETLB=0x40000 -fPIC -I. -I./ -I/opt/edb/as10/include/server -I/opt/edb/as10/include/internal -I/opt/local/Current/include -D_GNU_SOURCE -I/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/include/libxml2 -I/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/include -I/opt/local/Current/include/libxml2 -I/opt/local/Current/include -I/mnt/hgfs/edb-postgres.auto/server/source/libmm-edb.linux-x64/inst/include -c -o imgsmlr.o imgsmlr.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -DLINUX_OOM_ADJ=0 -O2 -DMAP_HUGETLB=0x40000 -fPIC -I. -I./ -I/opt/edb/as10/include/server -I/opt/edb/as10/include/internal -I/opt/local/Current/include -D_GNU_SOURCE -I/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/include/libxml2 -I/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/include -I/opt/local/Current/include/libxml2 -I/opt/local/Current/include -I/mnt/hgfs/edb-postgres.auto/server/source/libmm-edb.linux-x64/inst/include -c -o imgsmlr_idx.o imgsmlr_idx.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -DLINUX_OOM_ADJ=0 -O2 -DMAP_HUGETLB=0x40000 -fPIC -shared -o imgsmlr.so imgsmlr.o imgsmlr_idx.o -L/opt/edb/as10/lib -L/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/lib -L/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/lib -L/opt/local/Current/lib -L/mnt/hgfs/edb-postgres.auto/server/source/libmm-edb.linux-x64/inst/lib -Wl,--as-needed -Wl,-rpath,'/opt/edb/as10/lib',--enable-new-dtags -lgd
root@pg11-test-> USE_PGXS=1 make install
/bin/mkdir -p '/opt/edb/as10/lib'
/bin/mkdir -p '/opt/edb/as10/share/extension'
/bin/mkdir -p '/opt/edb/as10/share/extension'
/usr/bin/install -c -m 755 imgsmlr.so '/opt/edb/as10/lib/imgsmlr.so'
/usr/bin/install -c -m 644 .//imgsmlr.control '/opt/edb/as10/share/extension/'
/usr/bin/install -c -m 644 .//imgsmlr--1.0.sql '/opt/edb/as10/share/extension/'
root@pg11-test-> psql
psql.bin (10.5.12)
Type "help" for help.
postgres=# create extension imgsmlr;
CREATE EXTENSION
</code></pre></div></div>
<p>用法</p>
<p><a href="../201809/20180904_02.md">《PostgreSQL 11 相似图像搜索插件 imgsmlr 性能测试与优化 1 - 单机单表 (4亿图像)》</a></p>
<h2 id="中文分词">中文分词</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[root@pg11-test ~]# cd
[root@pg11-test ~]# . /opt/edb/as10/.bash_profile
-bash: unalias: vi: not found
git clone https://github.com/jaiminpan/pg_jieba
cd pg_jieba
git submodule update --init --recursive
mkdir build
cd build
cmake3 -DCMAKE_PREFIX_PATH=/opt/edb/as10 ..
make
make install
vi $PGDATA/postgresql.conf
shared_preload_libraries = '$libdir/pg_jieba.so,$libdir/dbms_pipe,$libdir/edb_gen,$libdir/dbms_aq'
</code></pre></div></div>
<p>重启数据库</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pg_ctl restart -m fast
postgres=# create extension pg_jieba;
CREATE EXTENSION
postgres=# select * from to_tsvector('jiebacfg', '小明硕士毕业于中国科学院计算所,后在日本京都大学深造');
to_tsvector
----------------------------------------------------------------------------------
'中国科学院':5 '小明':1 '日本京都大学':10 '毕业':3 '深造':11 '硕士':2 '计算所':6
(1 row)
postgres=# select * from to_tsvector('jiebacfg', '李小福是创新办主任也是云计算方面的专家');
to_tsvector
-------------------------------------------------------------------
'专家':11 '主任':5 '云计算':8 '创新':3 '办':4 '方面':9 '李小福':1
(1 row)
</code></pre></div></div>
<p>用法</p>
<p><a href="../201607/20160725_02.md">《如何加快PostgreSQL结巴分词pg_jieba加载速度》</a></p>
<h2 id="流计算">流计算</h2>
<p><a href="../201811/20181101_02.md">《PostgreSQL pipelinedb 流计算插件 - IoT应用 - 实时轨迹聚合》</a></p>
<h2 id="job">job</h2>
<p>https://www.enterprisedb.com/thank-you-4?anid=1256127</p>
<h2 id="内置插件">内置插件</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# create extension 这里按tab键补齐,口可以看到当前支持的插件。
adminpack chkpass dict_xsyn hstore insert_username ltree_plpython2u pg_buffercache pgstattuple plpython3u sslinfo tsm_system_time
amcheck citext earthdistance hstore_plperl intagg ltree_plpython3u pgcrypto pg_trgm pltcl sslutils unaccent
autoinc cube edb_cloneschema hstore_plperlu intarray ltree_plpythonu pg_freespacemap pg_visibility pltclu tablefunc "uuid-ossp"
bloom dblink edb_sharedplancache hstore_plpython2u isn moddatetime pg_prewarm pljava postgres_fdw tcn xml2
btree_gin dbms_scheduler file_fdw hstore_plpython3u lo pageinspect pgrowlocks plperl refint timetravel
btree_gist dict_int fuzzystrmatch hstore_plpythonu ltree parallel_clone pg_stat_statements plperlu seg tsm_system_rows
</code></pre></div></div>
<p>数据库内置插件,以及介绍</p>
<p>https://www.postgresql.org/docs/current/static/contrib.html</p>
<h1 id="五物理结构">五、物理结构</h1>
<p>PPT内有介绍物理结构</p>
<p><a href="../201801/20180121_01.md">《阿里云 PostgreSQL 产品生态;案例、开发实践、管理实践、数据库原理、学习资料、学习视频 - 珍藏级》</a></p>
<h1 id="六逻辑结构">六、逻辑结构</h1>
<p><a href="../201605/20160510_01.md">《PostgreSQL 逻辑结构 和 权限体系 介绍》</a></p>
<h1 id="七使用">七、使用</h1>
<p>1、连接数据库</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>psql -h HOST_OR_IP -p PORT -U DBUSER -d DBNAME
</code></pre></div></div>
<p>2、创建用户</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>create role rolename1 login encrypted password 'pwd';
</code></pre></div></div>
<p>3、创建数据库</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>create database newdb with template template0 encoding 'UTF8' lc_collate 'C' lc_ctype 'en_US.utf8';
</code></pre></div></div>
<p>4、克隆数据</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>create database clonedb1 with template postgres;
</code></pre></div></div>
<p>5、基本使用</p>
<p>http://www.postgresqltutorial.com/</p>
<p>6、高级用法</p>
<p><a href="../201802/20180226_05.md">《PostgreSQL SELECT 的高级用法(CTE, LATERAL, ORDINALITY, WINDOW, SKIP LOCKED, DISTINCT, GROUPING SETS, …) - 珍藏级》</a></p>
<p>7、应用场景</p>
<p><a href="../201706/20170601_02.md">《PostgreSQL、Greenplum 《如来神掌》 - 目录 - 珍藏级》</a></p>
<p>8、参考</p>
<p>https://www.postgresql.org/docs/current/static/index.html</p>
<h1 id="八继续阅读">八、继续阅读</h1>
<p>1、HA</p>
<p>patroni</p>
<p>2、只读节点</p>
<p>流复制</p>
<p><a href="../201809/20180917_01.md">《PostgreSQL 11 1000亿 tpcb、1000W tpcc 性能测试 - on 阿里云ECS + ESSD (含quorum based 0丢失多副本配置与性能测试)》</a></p>
<p><a href="../201803/20180326_01.md">《PostgreSQL 一主多从(多副本,强同步)简明手册 - 配置、压测、监控、切换、防脑裂、修复、0丢失 - 珍藏级》</a></p>
<p>3、容灾节点</p>
<p><a href="../201711/20171129_02.md">《PostgreSQL 10 on ECS 实施 流复制备库镜像+自动快照备份+自动备份验证+自动清理备份与归档 - 珍藏级》</a></p>
<p>4、持续增量备份</p>
<p><a href="../201711/20171129_02.md">《PostgreSQL 10 on ECS 实施 流复制备库镜像+自动快照备份+自动备份验证+自动清理备份与归档 - 珍藏级》</a></p>
<p>5、恢复到时间点</p>
<p><a href="../201512/20151220_06.md">《阿里云ApsaraDB RDS for PostgreSQL 最佳实践 - 6 任意时间点恢复》</a></p>
<p>6、逻辑备份</p>
<p>pg_dump</p>
<p>psql copy</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Command: COPY
Description: copy data between a file and a table
Syntax:
COPY table_name [ ( column_name [, ...] ) ]
FROM { 'filename' | PROGRAM 'command' | STDIN }
[ [ WITH ] ( option [, ...] ) ]
COPY { table_name [ ( column_name [, ...] ) ] | ( query ) }
TO { 'filename' | PROGRAM 'command' | STDOUT }
[ [ WITH ] ( option [, ...] ) ]
where option can be one of:
FORMAT format_name
OIDS [ boolean ]
FREEZE [ boolean ]
DELIMITER 'delimiter_character'
NULL 'null_string'
HEADER [ boolean ]
QUOTE 'quote_character'
ESCAPE 'escape_character'
FORCE_QUOTE { ( column_name [, ...] ) | * }
FORCE_NOT_NULL ( column_name [, ...] )
FORCE_NULL ( column_name [, ...] )
ENCODING 'encoding_name'
</code></pre></div></div>
<p>7、逻辑恢复</p>
<p>pg_restore</p>
<p>或</p>
<p>sql文档直接执行</p>
<p>或</p>
<p>copy</p>
<p>8、回收站</p>
<p><a href="../201504/20150429_01.md">《PostgreSQL Oracle 兼容性之 - 事件触发器实现类似Oracle的回收站功能》</a></p>
<p><a href="../201404/20140403_01.md">《PostgreSQL 回收站功能 - 基于HOOK的recycle bin pgtrashcan》</a></p>
<p>9、闪回</p>
<p><a href="../201710/20171010_01.md">《PostgreSQL flashback(闪回) 功能实现与介绍》</a></p>
<p><a href="../201408/20140828_01.md">《PostgreSQL 闪回 - flash back query emulate by trigger》</a></p>
<p>10、审计</p>
<p><a href="../201709/20170925_02.md">《PostgreSQL 事件触发器应用 - DDL审计记录 + 异步通知(notify)》</a></p>
<p><a href="../201505/20150515_01.md">《PostgreSQL 审计 - pg_audit module》</a></p>
<p><a href="../201806/20180616_01.md">《PostgreSQL 开启“审计日志、时间记录”带来的性能影响有多少?》</a></p>
<h1 id="九参考文档">九、参考文档</h1>
<p>https://www.enterprisedb.com/advanced-downloads</p>
<p><a href="https://www.aliyun.com/product/rds/ppas">阿里云ppas(兼容Oracle)</a></p>
<p><a href="https://promotion.aliyun.com/ntms/act/ppasadam.html">阿里云adam Oracle评估、迁移</a></p>
<p>https://www.postgresql.org/docs/current/static/app-psql.html</p>
<p>https://www.postgresql.org/docs/current/static/pgbench.html</p>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>Digoal.zhoudigoal@126.com背景 云数据库PPAS版,是阿里云与EnterpriseDB公司(简称EDB)合作基于PostgreSQL高度兼容Oracle语法的数据库服务,为用户提供易于操作的迁移工具,兼容范围涵盖:PL/SQL、数据类型、高级函数、表分区等。 用户可以直接在阿里云购买PPAS进行使用。 如果在购买PPAS前,想试用一下,可以去EDB网站下载,并部署相关的插件。 一、测试环境安装部署EDB 服务器 以阿里云ECS为例。 1、CPU # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 2 Core(s) per socket: 32 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz Stepping: 4 CPU MHz: 2500.008 BogoMIPS: 5000.01 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 33792K NUMA node0 CPU(s): 0-63 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 2、DISK # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 200G 0 disk └─vda1 253:1 0 200G 0 part / vdb 253:16 0 1.8T 0 disk ├─vgdata01-lv01 252:0 0 4T 0 lvm /data01 ├─vgdata01-lv02 252:1 0 4T 0 lvm /data02 ├─vgdata01-lv03 252:2 0 4T 0 lvm /data03 └─vgdata01-lv04 252:3 0 2T 0 lvm /data04 vdc 253:32 0 1.8T 0 disk ├─vgdata01-lv01 252:0 0 4T 0 lvm /data01 ├─vgdata01-lv02 252:1 0 4T 0 lvm /data02 ├─vgdata01-lv03 252:2 0 4T 0 lvm /data03 └─vgdata01-lv04 252:3 0 2T 0 lvm /data04 vdd 253:48 0 1.8T 0 disk ├─vgdata01-lv01 252:0 0 4T 0 lvm /data01 ├─vgdata01-lv02 252:1 0 4T 0 lvm /data02 ├─vgdata01-lv03 252:2 0 4T 0 lvm /data03 └─vgdata01-lv04 252:3 0 2T 0 lvm /data04 vde 253:64 0 1.8T 0 disk ├─vgdata01-lv01 252:0 0 4T 0 lvm /data01 ├─vgdata01-lv02 252:1 0 4T 0 lvm /data02 ├─vgdata01-lv03 252:2 0 4T 0 lvm /data03 └─vgdata01-lv04 252:3 0 2T 0 lvm /data04 vdf 253:80 0 1.8T 0 disk ├─vgdata01-lv01 252:0 0 4T 0 lvm /data01 ├─vgdata01-lv02 252:1 0 4T 0 lvm /data02 ├─vgdata01-lv03 252:2 0 4T 0 lvm /data03 └─vgdata01-lv04 252:3 0 2T 0 lvm /data04 vdg 253:96 0 1.8T 0 disk ├─vgdata01-lv01 252:0 0 4T 0 lvm /data01 ├─vgdata01-lv02 252:1 0 4T 0 lvm /data02 ├─vgdata01-lv03 252:2 0 4T 0 lvm /data03 └─vgdata01-lv04 252:3 0 2T 0 lvm /data04 vdh 253:112 0 1.8T 0 disk ├─vgdata01-lv01 252:0 0 4T 0 lvm /data01 ├─vgdata01-lv02 252:1 0 4T 0 lvm /data02 ├─vgdata01-lv03 252:2 0 4T 0 lvm /data03 └─vgdata01-lv04 252:3 0 2T 0 lvm /data04 vdi 253:128 0 1.8T 0 disk ├─vgdata01-lv01 252:0 0 4T 0 lvm /data01 ├─vgdata01-lv02 252:1 0 4T 0 lvm /data02 ├─vgdata01-lv03 252:2 0 4T 0 lvm /data03 └─vgdata01-lv04 252:3 0 2T 0 lvm /data04 3、MEM # free -m total used free shared buff/cache available Mem: 515815 318010 109199 8 88605 194736 Swap: 0 0 0 操作系统 以CentOS 7.x x64为例。 1、存储 dd if=/dev/zero of=/dev/vdb bs=1024 count=1024 dd if=/dev/zero of=/dev/vdc bs=1024 count=1024 dd if=/dev/zero of=/dev/vdd bs=1024 count=1024 dd if=/dev/zero of=/dev/vde bs=1024 count=1024 dd if=/dev/zero of=/dev/vdf bs=1024 count=1024 dd if=/dev/zero of=/dev/vdg bs=1024 count=1024 dd if=/dev/zero of=/dev/vdh bs=1024 count=1024 dd if=/dev/zero of=/dev/vdi bs=1024 count=1024 pvcreate /dev/vd[b-i] vgcreate -A y -s 128M vgdata01 /dev/vd[b-i] lvcreate -A y -i 8 -I 8 -L 4096GiB -n lv01 vgdata01 lvcreate -A y -i 8 -I 8 -L 4096GiB -n lv02 vgdata01 lvcreate -A y -i 8 -I 8 -L 4096GiB -n lv03 vgdata01 2、文件系统 mkfs.ext4 /dev/mapper/vgdata01-lv01 -m 0 -O extent,uninit_bg -E lazy_itable_init=1,stride=2,stripe_width=16 -b 4096 -T largefile -L lv01 mkfs.ext4 /dev/mapper/vgdata01-lv02 -m 0 -O extent,uninit_bg -E lazy_itable_init=1,stride=2,stripe_width=16 -b 4096 -T largefile -L lv02 mkfs.ext4 /dev/mapper/vgdata01-lv03 -m 0 -O extent,uninit_bg -E lazy_itable_init=1,stride=2,stripe_width=16 -b 4096 -T largefile -L lv03 vi /etc/fstab LABEL=lv01 /data01 ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0,data=writeback 0 0 LABEL=lv02 /data02 ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0,data=writeback 0 0 LABEL=lv03 /data03 ext4 defaults,noatime,nodiratime,nodelalloc,barrier=0,data=writeback 0 0 mkdir /data01 mkdir /data02 mkdir /data03 mount -a 3、内核参数 vi /etc/sysctl.conf # add by digoal.zhou fs.aio-max-nr = 1048576 fs.file-max = 76724600 # 可选:kernel.core_pattern = /data01/corefiles/core_%e_%u_%t_%s.%p # /data01/corefiles 事先建好,权限777,如果是软链接,对应的目录修改为777 kernel.sem = 4096 2147483647 2147483646 512000 # 信号量, ipcs -l 或 -u 查看,每16个进程一组,每组信号量需要17个信号量。 kernel.shmall = 107374182 # 所有共享内存段相加大小限制 (建议内存的80%),单位为页。 kernel.shmmax = 274877906944 # 最大单个共享内存段大小 (建议为内存一半), >9.2的版本已大幅降低共享内存的使用,单位为字节。 kernel.shmmni = 819200 # 一共能生成多少共享内存段,每个PG数据库集群至少2个共享内存段 net.core.netdev_max_backlog = 10000 net.core.rmem_default = 262144 # The default setting of the socket receive buffer in bytes. net.core.rmem_max = 4194304 # The maximum receive socket buffer size in bytes net.core.wmem_default = 262144 # The default setting (in bytes) of the socket send buffer. net.core.wmem_max = 4194304 # The maximum send socket buffer size in bytes. net.core.somaxconn = 4096 net.ipv4.tcp_max_syn_backlog = 4096 net.ipv4.tcp_keepalive_intvl = 20 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_time = 60 net.ipv4.tcp_mem = 8388608 12582912 16777216 net.ipv4.tcp_fin_timeout = 5 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syncookies = 1 # 开启SYN Cookies。当出现SYN等待队列溢出时,启用cookie来处理,可防范少量的SYN攻击 net.ipv4.tcp_timestamps = 1 # 减少time_wait net.ipv4.tcp_tw_recycle = 0 # 如果=1则开启TCP连接中TIME-WAIT套接字的快速回收,但是NAT环境可能导致连接失败,建议服务端关闭它 net.ipv4.tcp_tw_reuse = 1 # 开启重用。允许将TIME-WAIT套接字重新用于新的TCP连接 net.ipv4.tcp_max_tw_buckets = 262144 net.ipv4.tcp_rmem = 8192 87380 16777216 net.ipv4.tcp_wmem = 8192 65536 16777216 net.nf_conntrack_max = 1200000 net.netfilter.nf_conntrack_max = 1200000 vm.dirty_background_bytes = 409600000 # 系统脏页到达这个值,系统后台刷脏页调度进程 pdflush(或其他) 自动将(dirty_expire_centisecs/100)秒前的脏页刷到磁盘 # 默认为10%,大内存机器建议调整为直接指定多少字节 vm.dirty_expire_centisecs = 3000 # 比这个值老的脏页,将被刷到磁盘。3000表示30秒。 vm.dirty_ratio = 95 # 如果系统进程刷脏页太慢,使得系统脏页超过内存 95 % 时,则用户进程如果有写磁盘的操作(如fsync, fdatasync等调用),则需要主动把系统脏页刷出。 # 有效防止用户进程刷脏页,在单机多实例,并且使用CGROUP限制单实例IOPS的情况下非常有效。 vm.dirty_writeback_centisecs = 100 # pdflush(或其他)后台刷脏页进程的唤醒间隔, 100表示1秒。 vm.swappiness = 0 # 不使用交换分区 vm.mmap_min_addr = 65536 vm.overcommit_memory = 0 # 在分配内存时,允许少量over malloc, 如果设置为 1, 则认为总是有足够的内存,内存较少的测试环境可以使用 1 . vm.overcommit_ratio = 90 # 当overcommit_memory = 2 时,用于参与计算允许指派的内存大小。 vm.swappiness = 0 # 关闭交换分区 vm.zone_reclaim_mode = 0 # 禁用 numa, 或者在vmlinux中禁止. net.ipv4.ip_local_port_range = 40000 65535 # 本地自动分配的TCP, UDP端口号范围 fs.nr_open=20480000 # 单个进程允许打开的文件句柄上限 # 以下参数请注意 # vm.extra_free_kbytes = 4096000 # vm.min_free_kbytes = 2097152 # 如果是小内存机器,以上两个值不建议设置 # vm.nr_hugepages = 66536 # 建议shared buffer设置超过64GB时 使用大页,页大小 /proc/meminfo Hugepagesize # vm.lowmem_reserve_ratio = 1 1 1 # 对于内存大于64G时,建议设置,否则建议默认值 256 256 32 sysctl -p 4、资源限制 vi /etc/security/limits.d/20-nproc.conf # nofile超过1048576的话,一定要先将sysctl的fs.nr_open设置为更大的值,并生效后才能继续设置nofile. * soft nofile 1024000 * hard nofile 1024000 * soft nproc unlimited * hard nproc unlimited * soft core unlimited * hard core unlimited * soft memlock unlimited * hard memlock unlimited 5、关闭透明大页 vi /etc/rc.local touch /var/lock/subsys/local if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi su - postgres -c "pg_ctl start" 6、防火墙 iptables 配置,略 7、selinux vi /etc/selinux/config SELINUX=disabled SELINUXTYPE=targeted # setenforce 0 setenforce: SELinux is disabled 8、依赖包 yum -y install coreutils glib2 lrzsz dstat sysstat e4fsprogs xfsprogs ntp readline-devel zlib-devel openssl-devel pam-devel libxml2-devel libxslt-devel python-devel tcl-devel gcc gcc-c++ make smartmontools flex bison perl-devel perl-ExtUtils* openldap-devel jadetex openjade bzip2 iotop perf cmake3 建议重启一下机器。 EDB 数据库软件 10.5 版本为例。 1、下载软件 https://www.enterprisedb.com/advanced-downloads OPTION A - EDB POSTGRES™ ADVANCED SERVER v10.5 , Linux x86-64 , Interactive Installer 注意: Note: For the Enterprise subscription , choose EDB Postgres™ Advanced Server. For the Standard subscription, choose PostgreSQL. For the Developer subscription, choose either database. The Developer subscription is for use in non-production environments only. wget https://get.enterprisedb.com/advstacks/edb-as10-server-10.5.12-1-linux-x64.run 2、安装软件 su - root chmod 700 edb-as10-server-10.5.12-1-linux-x64.run ./edb-as10-server-10.5.12-1-linux-x64.run Press [Enter] to continue: Do you accept this license? [y/n]: y 同意许可。 指定EDB软件安装目录 Installation Directory [/opt/edb/as10]: ---------------------------------------------------------------------------- Select the components you want to install; clear the components you do not want to install. Click Next when you are ready to continue. 选择要安装的组件 EDB Postgres Advanced Server [Y/n] :Y pgAdmin 4 [Y/n] :n StackBuilder Plus [Y/n] :n Command Line Tools [Y/n] :Y Is the selection above correct? [Y/n]: Y 2.1、数据、REDO目录 # mkdir /data04/ppas10 # mkdir /data04/ppas10_wal 2.2、 ---------------------------------------------------------------------------- Additional Directories 指定数据、REDO目录 Please select a directory under which to store your data. Data Directory [/opt/edb/as10/data]: /data04/ppas10 Please select a directory under which to store your Write-Ahead Logs. Write-Ahead Log (WAL) Directory [/opt/edb/as10/data/pg_wal]: /data04/ppas10_wal ---------------------------------------------------------------------------- Advanced Server Dialect EDB Postgres Advanced Server can be configured in one of two "Dialects" - 1) Compatible with Oracle or 2) Compatible with Postgres. If you select Compatible with Oracle, Advanced Server will be configured with appropriate data type conversions, time and date formats, Oracle-styled operators, dictionary views and more. This makes it easier to migrate or write new applications that are more compatible with the Oracle database. If you select Compatible with Postgres, Advanced Server will be configured with standard PostgeSQL data types, time/date formats and operators. 指定兼容PostgreSQL还是兼容Oracle模式。 Advanced Server Dialect [1] Compatible with Oracle [2] Compatible with Postgres Please choose an option [1] : 1 ---------------------------------------------------------------------------- Please provide a password for the database superuser (enterprisedb). A locked Unix user account (enterprisedb) will be created if not present. 设置数据库超级用户enterprisedb的密码 Password : Retype Password : ---------------------------------------------------------------------------- Additional Configuration Please select the port number the server should listen on. 设置数据库监听端口 Port [5444]: 4001 Select the locale to be used by the new database cluster. Locale [1] [Default locale] .... [232] en_US [233] en_US.iso88591 [234] en_US.iso885915 [235] en_US.utf8 ... [762] zh_CN [763] zh_CN.gb2312 [764] zh_CN.utf8 [765] zh_HK.utf8 [766] zh_SG [767] zh_SG.gb2312 [768] zh_SG.utf8 [769] zh_TW.euctw [770] zh_TW.utf8 [771] zu_ZA [772] zu_ZA.iso88591 [773] zu_ZA.utf8 设置location, 字符集 Please choose an option [1] : 235 Would you like to install sample tables and procedures? 是否安装样例模板 Install sample tables and procedures. [Y/n]: n ---------------------------------------------------------------------------- Dynatune Dynamic Tuning: Server Utilization Please select the type of server to determine the amount of system resources that may be utilized: 配置数据库所在服务器的属性(独占,开发模式,混合模式)。 [1] Development (e.g. a developer's laptop) [2] General Purpose (e.g. a web or application server) [3] Dedicated (a server running only Advanced Server) Please choose an option [2] : 3 ---------------------------------------------------------------------------- Dynatune Dynamic Tuning: Workload Profile Please select the type of workload this server will be used for: 设置数据库的使用场景,OLTP或OLAP或混合模式 [1] Transaction Processing (OLTP systems) [2] General Purpose (OLTP and reporting workloads) [3] Reporting (Complex queries or OLAP workloads) Please choose an option [1] : 2 ---------------------------------------------------------------------------- Pre Installation Summary The following settings will be used for the installation:: 再次确认配置 Installation Directory: /opt/edb/as10 Server Installation Directory: /opt/edb/as10 Data Directory: /data04/ppas10 WAL Directory: /data04/ppas10_wal Database Port: 4001 Database Superuser: enterprisedb Operating System Account: enterprisedb Database Service: edb-as-10 Command Line Tools Installation Directory: /opt/edb/as10 Press [Enter] to continue: ---------------------------------------------------------------------------- Setup is now ready to begin installing EDB Postgres Advanced Server on your computer. 开始安装 Do you want to continue? [Y/n]: Y ---------------------------------------------------------------------------- Please wait while Setup installs EDB Postgres Advanced Server on your computer. Installing EDB Postgres Advanced Server 0% ______________ 50% ______________ 100% ######################################## 3、配置enterprisedb用户环境变量 修改软件目录OWNER # chown -R enterprisedb:enterprisedb /opt/edb/as10 配置enterprisedb用户的环境变量 # vi /opt/edb/as10/.bash_profile export PS1="$USER@`/bin/hostname -s`-> " export PGPORT=4001 export PGDATA=/data04/ppas10 export LANG=en_US.utf8 export PGHOME=/opt/edb/as10 export LD_LIBRARY_PATH=$PGHOME/lib:/lib64:/usr/lib64:/usr/local/lib64:/lib:/usr/lib:/usr/local/lib:$LD_LIBRARY_PATH export DATE=`date +"%Y%m%d%H%M"` export PATH=$PGHOME/bin:$PATH:. export MANPATH=$PGHOME/share/man:$MANPATH export PGHOST=$PGDATA export PGUSER=enterprisedb export PGDATABASE=postgres alias rm='rm -i' alias ll='ls -lh' alias cmake=cmake3 unalias vi 4、初始化数据库集群(安装软件时已初始化,以下略) 如果你需要在一台服务器上初始化多个数据库实例,可以参照执行: 数据目录与WAL日志目录(注意,这些是数据和redo日志目录,所以必须有别于其他数据库实例已经存在的目录。) # mkdir ppas10_8001 # mkdir /data04/ppas10_wal_8001 # chown -R enterprisedb:enterprisedb /data04/ppas10* # chmod 700 /data04/ppas10_wal_8001 su - enterprisedb 如果你想兼容Oracle,使用如下手段初始化 initdb -D ppas10_8001 -E UTF8 --lc-collate=C --lc-ctype=en_US.utf8 -U enterprisedb -W -X /data04/ppas10_wal_8001 --redwood-like 如果你想兼容PostgreSQL,使用如下手段初始化 initdb -D ppas10_8001 -E UTF8 --lc-collate=C --lc-ctype=en_US.utf8 -U enterprisedb -W -X /data04/ppas10_wal_8001 当然,在初始化后,还可以通过修改参数来实现兼容Oracle或PostgreSQL 《EDB PPAS (PostgreSQL plus advanced server) 10 参数模板 - 珍藏级》 5、安装Oracle OCI(可选) 安装Oracle OCI 《PostgreSQL 商用版本EPAS(阿里云ppas) - 测试环境部署(EPAS 安装、配置、管理、Oracle DBLINK、外表)》 安装Oracle OCI。这样才可以在EDB数据库中建立ORACLE的DBLINK。 http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html 选择下载包含OCI的包( 需要输入Oracle的账号密码,可以免费注册。) 将安装包上传到服务器,解压,放到EDB的PGHOME目录(本文/opt/edb/as10),并建立软链。详情参考 https://www.enterprisedb.com/docs/en/10.0/Ora_Compat_Dev_Guide/Database_Compatibility_for_Oracle_Developers_Guide.1.124.html# 操作如下 # cd /opt/edb/as10 [root@digoal ~]# mkdir oci [root@digoal ~]# mv instantclient-basic-linux.x64-12.2.0.1.0.zip oci/ [root@digoal ~]# cd oci [root@digoal oci]# ll total 67356 -rw-r--r-- 1 root root 68965195 Jan 19 11:00 instantclient-basic-linux.x64-12.2.0.1.0.zip [root@digoal oci]# unzip instantclient-basic-linux.x64-12.2.0.1.0.zip Archive: instantclient-basic-linux.x64-12.2.0.1.0.zip inflating: instantclient_12_2/adrci inflating: instantclient_12_2/BASIC_README inflating: instantclient_12_2/genezi inflating: instantclient_12_2/libclntshcore.so.12.1 inflating: instantclient_12_2/libclntsh.so.12.1 inflating: instantclient_12_2/libipc1.so inflating: instantclient_12_2/libmql1.so inflating: instantclient_12_2/libnnz12.so inflating: instantclient_12_2/libocci.so.12.1 inflating: instantclient_12_2/libociei.so inflating: instantclient_12_2/libocijdbc12.so inflating: instantclient_12_2/libons.so inflating: instantclient_12_2/liboramysql12.so inflating: instantclient_12_2/ojdbc8.jar inflating: instantclient_12_2/uidrvci inflating: instantclient_12_2/xstreams.jar [root@digoal oci]# ll total 67360 drwxr-xr-x 2 root root 4096 Jan 19 11:02 instantclient_12_2 -rw-r--r-- 1 root root 68965195 Jan 19 11:00 instantclient-basic-linux.x64-12.2.0.1.0.zip [root@digoal oci]# cd instantclient_12_2/ [root@digoal instantclient_12_2]# ll total 216696 -rwxrwxr-x 1 root root 44220 Jan 26 2017 adrci -rw-rw-r-- 1 root root 363 Jan 26 2017 BASIC_README -rwxrwxr-x 1 root root 57272 Jan 26 2017 genezi -rwxrwxr-x 1 root root 8033199 Jan 26 2017 libclntshcore.so.12.1 -rwxrwxr-x 1 root root 71638263 Jan 26 2017 libclntsh.so.12.1 -r-xr-xr-x 1 root root 2981501 Jan 26 2017 libipc1.so -r-xr-xr-x 1 root root 539065 Jan 26 2017 libmql1.so -r-xr-xr-x 1 root root 6568149 Jan 26 2017 libnnz12.so -rwxrwxr-x 1 root root 2218687 Jan 26 2017 libocci.so.12.1 -rwxrwxr-x 1 root root 124771800 Jan 26 2017 libociei.so -r-xr-xr-x 1 root root 158543 Jan 26 2017 libocijdbc12.so -r-xr-xr-x 1 root root 380996 Jan 26 2017 libons.so -rwxrwxr-x 1 root root 116563 Jan 26 2017 liboramysql12.so -r--r--r-- 1 root root 4036257 Jan 26 2017 ojdbc8.jar -rwxrwxr-x 1 root root 240476 Jan 26 2017 uidrvci -rw-rw-r-- 1 root root 74230 Jan 26 2017 xstreams.jar [root@digoal instantclient_12_2]# cp libclntsh.so.12.1 /opt/edb/as10/lib/ [root@digoal instantclient_12_2]# cd /opt/edb/as10/lib [root@digoal lib]# ln -s libclntsh.so.12.1 libclntsh.so 6、创建DBLINK,创建ORACLE外部表,略。参考上面的连接。 7、配置数据库参数 su - enterprisedb cd $PGDATA vi postgresql.conf listen_addresses = '0.0.0.0' port = 4001 # 多实例时,不同的数据库端口必须不同。 max_connections = 4000 superuser_reserved_connections = 13 unix_socket_directories = '.,/tmp' unix_socket_permissions = 0700 row_security = on tcp_keepalives_idle = 45 tcp_keepalives_interval = 10 tcp_keepalives_count = 10 shared_buffers = 32GB huge_pages = try max_prepared_transactions = 4000 work_mem = 8MB maintenance_work_mem = 2GB autovacuum_work_mem = 2GB max_stack_depth = 4MB dynamic_shared_memory_type = posix shared_preload_libraries = '$libdir/dbms_pipe,$libdir/edb_gen,$libdir/dbms_aq' vacuum_cost_delay = 0 bgwriter_delay = 10ms bgwriter_lru_maxpages = 1000 bgwriter_lru_multiplier = 5.0 effective_io_concurrency = 0 max_worker_processes = 64 max_parallel_workers_per_gather = 8 max_parallel_workers = 24 wal_level = replica synchronous_commit = off # 异步提交 full_page_writes = on # 如果文件系统支持COW,可以关闭。如果块设备支持8K(block_size决定)原子写,也可以关闭。开启有一定性能影响 wal_buffers = 64MB wal_writer_delay = 10ms wal_writer_flush_after = 1MB checkpoint_timeout = 35min max_wal_size = 64GB min_wal_size = 16GB checkpoint_completion_target = 0.1 archive_mode = on archive_command = '/usr/bin/true' max_wal_senders = 16 wal_keep_segments = 2048 wal_sender_timeout = 30s max_replication_slots = 16 hot_standby = on max_standby_archive_delay = 300s max_standby_streaming_delay = 300s wal_receiver_status_interval = 1s wal_receiver_timeout = 30s max_logical_replication_workers = 16 max_sync_workers_per_subscription = 4 random_page_cost = 1.1 effective_cache_size = 400GB log_destination = 'csvlog' logging_collector = on log_truncate_on_rotation = on log_min_duration_statement = 10s log_checkpoints = on log_connections = on log_disconnections = on log_error_verbosity = verbose log_line_prefix = '%t ' log_lock_waits = on log_statement = 'ddl' log_temp_files = 0 log_timezone = 'PRC' track_io_timing = on # 如果不需要跟踪IO的时间,可以关闭,开启有一定性能影响 track_functions = all track_activity_query_size = 2048 autovacuum = on log_autovacuum_min_duration = 0 autovacuum_max_workers = 6 autovacuum_freeze_max_age = 1200000000 autovacuum_multixact_freeze_max_age = 1400000000 autovacuum_vacuum_cost_delay = 0 statement_timeout = 300s lock_timeout = 15s idle_in_transaction_session_timeout = 60s vacuum_freeze_table_age = 1150000000 vacuum_multixact_freeze_table_age = 1150000000 datestyle = 'redwood,show_time' timezone = 'PRC' lc_messages = 'en_US.utf8' lc_monetary = 'en_US.utf8' lc_numeric = 'en_US.utf8' lc_time = 'en_US.utf8' default_text_search_config = 'pg_catalog.english' edb_redwood_date = on edb_redwood_greatest_least = on edb_redwood_strings = on db_dialect = 'redwood' edb_dynatune = 100 edb_dynatune_profile = mixed timed_statistics = off 8、数据库防火墙配置 su - enterprisedb cd $PGDATA vi pg_hba.conf # TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust # IPv6 local connections: host all all ::1/128 trust # Allow replication connections from localhost, by a user with the # replication privilege. local replication all trust host replication all 127.0.0.1/32 trust host replication all ::1/128 trust host all all 0.0.0.0/0 md5 9、重启数据库 pg_ctl restart -m fast 二、IDE cli https://www.postgresql.org/docs/current/static/app-psql.html enterprisedb@pg11-test-> psql psql.bin (10.5.12) Type "help" for help. postgres=# \dt Did not find any relations. postgres=# \l List of databases Name | Owner | Encoding | Collate | Ctype | ICU | Access privileges -----------+--------------+----------+------------+------------+-----+------------------------------- edb | enterprisedb | UTF8 | en_US.utf8 | en_US.utf8 | | postgres | enterprisedb | UTF8 | en_US.utf8 | en_US.utf8 | | template0 | enterprisedb | UTF8 | en_US.utf8 | en_US.utf8 | | =c/enterprisedb + | | | | | | enterprisedb=CTc/enterprisedb template1 | enterprisedb | UTF8 | en_US.utf8 | en_US.utf8 | | =c/enterprisedb + | | | | | | enterprisedb=CTc/enterprisedb (4 rows) gui 1、pem (Enterprisedb提供的GUI工具) https://www.enterprisedb.com/software-downloads-postgres 2、pgadmin https://www.pgadmin.org/ 《阿里云ppas 逻辑备份(导出)、还原 - 导出到本地、从本地导入》 三、压测 TPC-B su - enterprisedb 初始化1亿TPC-B数据 pgbench -i -s 1000 只读测试 pgbench -M prepared -n -r -P 1 -c 64 -j 64 -T 120 -S transaction type: <builtin: select only> scaling factor: 1000 query mode: prepared number of clients: 64 number of threads: 64 duration: 120 s number of transactions actually processed: 125854784 latency average = 0.061 ms latency stddev = 0.062 ms tps = 1048623.422874 (including connections establishing) tps = 1048713.313561 (excluding connections establishing) script statistics: - statement latencies in milliseconds: 0.001 \set aid random(1, 100000 * :scale) 0.060 SELECT abalance FROM pgbench_accounts WHERE aid = :aid; 读写混合测试 pgbench -M prepared -n -r -P 1 -c 64 -j 64 -T 120 transaction type: <builtin: TPC-B (sort of)> scaling factor: 1000 query mode: prepared number of clients: 64 number of threads: 64 duration: 120 s number of transactions actually processed: 10363880 latency average = 0.741 ms latency stddev = 1.315 ms tps = 86351.412706 (including connections establishing) tps = 86359.549707 (excluding connections establishing) script statistics: - statement latencies in milliseconds: 0.002 \set aid random(1, 100000 * :scale) 0.001 \set bid random(1, 1 * :scale) 0.001 \set tid random(1, 10 * :scale) 0.001 \set delta random(-5000, 5000) 0.050 BEGIN; 0.155 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid; 0.077 SELECT abalance FROM pgbench_accounts WHERE aid = :aid; 0.109 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid; 0.102 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid; 0.083 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP); 0.162 END; TPC-C 参考 《PostgreSQL 11 tpcc 测试(103万tpmC on ECS) - use sysbench-tpcc by Percona-Lab》 四、插件 https://www.enterprisedb.com/advanced-downloads 插件介绍、安装、使用 图像 [root@pg11-test ~]# cd [root@pg11-test ~]# . /opt/edb/as10/.bash_profile -bash: unalias: vi: not found root@pg11-test-> git clone https://github.com/postgrespro/imgsmlr Cloning into 'imgsmlr'... remote: Enumerating objects: 146, done. remote: Total 146 (delta 0), reused 0 (delta 0), pack-reused 146 Receiving objects: 100% (146/146), 241.11 KiB | 149.00 KiB/s, done. Resolving deltas: 100% (69/69), done. root@pg11-test-> cd imgsmlr root@pg11-test-> USE_PGXS=1 make clean rm -f imgsmlr.so libimgsmlr.a libimgsmlr.pc rm -f imgsmlr.o imgsmlr_idx.o rm -rf data/*.hex rm -rf results/ regression.diffs regression.out tmp_check/ tmp_check_iso/ log/ output_iso/ root@pg11-test-> USE_PGXS=1 make gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -DLINUX_OOM_ADJ=0 -O2 -DMAP_HUGETLB=0x40000 -fPIC -I. -I./ -I/opt/edb/as10/include/server -I/opt/edb/as10/include/internal -I/opt/local/Current/include -D_GNU_SOURCE -I/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/include/libxml2 -I/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/include -I/opt/local/Current/include/libxml2 -I/opt/local/Current/include -I/mnt/hgfs/edb-postgres.auto/server/source/libmm-edb.linux-x64/inst/include -c -o imgsmlr.o imgsmlr.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -DLINUX_OOM_ADJ=0 -O2 -DMAP_HUGETLB=0x40000 -fPIC -I. -I./ -I/opt/edb/as10/include/server -I/opt/edb/as10/include/internal -I/opt/local/Current/include -D_GNU_SOURCE -I/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/include/libxml2 -I/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/include -I/opt/local/Current/include/libxml2 -I/opt/local/Current/include -I/mnt/hgfs/edb-postgres.auto/server/source/libmm-edb.linux-x64/inst/include -c -o imgsmlr_idx.o imgsmlr_idx.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -DLINUX_OOM_ADJ=0 -O2 -DMAP_HUGETLB=0x40000 -fPIC -shared -o imgsmlr.so imgsmlr.o imgsmlr_idx.o -L/opt/edb/as10/lib -L/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/lib -L/opt/local/20160428/649c6f94-f2c0-4703-b065-99d58ae4acc6/lib -L/opt/local/Current/lib -L/mnt/hgfs/edb-postgres.auto/server/source/libmm-edb.linux-x64/inst/lib -Wl,--as-needed -Wl,-rpath,'/opt/edb/as10/lib',--enable-new-dtags -lgd root@pg11-test-> USE_PGXS=1 make install /bin/mkdir -p '/opt/edb/as10/lib' /bin/mkdir -p '/opt/edb/as10/share/extension' /bin/mkdir -p '/opt/edb/as10/share/extension' /usr/bin/install -c -m 755 imgsmlr.so '/opt/edb/as10/lib/imgsmlr.so' /usr/bin/install -c -m 644 .//imgsmlr.control '/opt/edb/as10/share/extension/' /usr/bin/install -c -m 644 .//imgsmlr--1.0.sql '/opt/edb/as10/share/extension/' root@pg11-test-> psql psql.bin (10.5.12) Type "help" for help. postgres=# create extension imgsmlr; CREATE EXTENSION 用法 《PostgreSQL 11 相似图像搜索插件 imgsmlr 性能测试与优化 1 - 单机单表 (4亿图像)》 中文分词 [root@pg11-test ~]# cd [root@pg11-test ~]# . /opt/edb/as10/.bash_profile -bash: unalias: vi: not found git clone https://github.com/jaiminpan/pg_jieba cd pg_jieba git submodule update --init --recursive mkdir build cd build cmake3 -DCMAKE_PREFIX_PATH=/opt/edb/as10 .. make make install vi $PGDATA/postgresql.conf shared_preload_libraries = '$libdir/pg_jieba.so,$libdir/dbms_pipe,$libdir/edb_gen,$libdir/dbms_aq' 重启数据库 pg_ctl restart -m fast postgres=# create extension pg_jieba; CREATE EXTENSION postgres=# select * from to_tsvector('jiebacfg', '小明硕士毕业于中国科学院计算所,后在日本京都大学深造'); to_tsvector ---------------------------------------------------------------------------------- '中国科学院':5 '小明':1 '日本京都大学':10 '毕业':3 '深造':11 '硕士':2 '计算所':6 (1 row) postgres=# select * from to_tsvector('jiebacfg', '李小福是创新办主任也是云计算方面的专家'); to_tsvector ------------------------------------------------------------------- '专家':11 '主任':5 '云计算':8 '创新':3 '办':4 '方面':9 '李小福':1 (1 row) 用法 《如何加快PostgreSQL结巴分词pg_jieba加载速度》 流计算 《PostgreSQL pipelinedb 流计算插件 - IoT应用 - 实时轨迹聚合》 job https://www.enterprisedb.com/thank-you-4?anid=1256127 内置插件 postgres=# create extension 这里按tab键补齐,口可以看到当前支持的插件。 adminpack chkpass dict_xsyn hstore insert_username ltree_plpython2u pg_buffercache pgstattuple plpython3u sslinfo tsm_system_time amcheck citext earthdistance hstore_plperl intagg ltree_plpython3u pgcrypto pg_trgm pltcl sslutils unaccent autoinc cube edb_cloneschema hstore_plperlu intarray ltree_plpythonu pg_freespacemap pg_visibility pltclu tablefunc "uuid-ossp" bloom dblink edb_sharedplancache hstore_plpython2u isn moddatetime pg_prewarm pljava postgres_fdw tcn xml2 btree_gin dbms_scheduler file_fdw hstore_plpython3u lo pageinspect pgrowlocks plperl refint timetravel btree_gist dict_int fuzzystrmatch hstore_plpythonu ltree parallel_clone pg_stat_statements plperlu seg tsm_system_rows 数据库内置插件,以及介绍 https://www.postgresql.org/docs/current/static/contrib.html 五、物理结构 PPT内有介绍物理结构 《阿里云 PostgreSQL 产品生态;案例、开发实践、管理实践、数据库原理、学习资料、学习视频 - 珍藏级》 六、逻辑结构 《PostgreSQL 逻辑结构 和 权限体系 介绍》 七、使用 1、连接数据库 psql -h HOST_OR_IP -p PORT -U DBUSER -d DBNAME 2、创建用户 create role rolename1 login encrypted password 'pwd'; 3、创建数据库 create database newdb with template template0 encoding 'UTF8' lc_collate 'C' lc_ctype 'en_US.utf8'; 4、克隆数据 create database clonedb1 with template postgres; 5、基本使用 http://www.postgresqltutorial.com/ 6、高级用法 《PostgreSQL SELECT 的高级用法(CTE, LATERAL, ORDINALITY, WINDOW, SKIP LOCKED, DISTINCT, GROUPING SETS, …) - 珍藏级》 7、应用场景 《PostgreSQL、Greenplum 《如来神掌》 - 目录 - 珍藏级》 8、参考 https://www.postgresql.org/docs/current/static/index.html 八、继续阅读 1、HA patroni 2、只读节点 流复制 《PostgreSQL 11 1000亿 tpcb、1000W tpcc 性能测试 - on 阿里云ECS + ESSD (含quorum based 0丢失多副本配置与性能测试)》 《PostgreSQL 一主多从(多副本,强同步)简明手册 - 配置、压测、监控、切换、防脑裂、修复、0丢失 - 珍藏级》 3、容灾节点 《PostgreSQL 10 on ECS 实施 流复制备库镜像+自动快照备份+自动备份验证+自动清理备份与归档 - 珍藏级》 4、持续增量备份 《PostgreSQL 10 on ECS 实施 流复制备库镜像+自动快照备份+自动备份验证+自动清理备份与归档 - 珍藏级》 5、恢复到时间点 《阿里云ApsaraDB RDS for PostgreSQL 最佳实践 - 6 任意时间点恢复》 6、逻辑备份 pg_dump psql copy Command: COPY Description: copy data between a file and a table Syntax: COPY table_name [ ( column_name [, ...] ) ] FROM { 'filename' | PROGRAM 'command' | STDIN } [ [ WITH ] ( option [, ...] ) ] COPY { table_name [ ( column_name [, ...] ) ] | ( query ) } TO { 'filename' | PROGRAM 'command' | STDOUT } [ [ WITH ] ( option [, ...] ) ] where option can be one of: FORMAT format_name OIDS [ boolean ] FREEZE [ boolean ] DELIMITER 'delimiter_character' NULL 'null_string' HEADER [ boolean ] QUOTE 'quote_character' ESCAPE 'escape_character' FORCE_QUOTE { ( column_name [, ...] ) | * } FORCE_NOT_NULL ( column_name [, ...] ) FORCE_NULL ( column_name [, ...] ) ENCODING 'encoding_name' 7、逻辑恢复 pg_restore 或 sql文档直接执行 或 copy 8、回收站 《PostgreSQL Oracle 兼容性之 - 事件触发器实现类似Oracle的回收站功能》 《PostgreSQL 回收站功能 - 基于HOOK的recycle bin pgtrashcan》 9、闪回 《PostgreSQL flashback(闪回) 功能实现与介绍》 《PostgreSQL 闪回 - flash back query emulate by trigger》 10、审计 《PostgreSQL 事件触发器应用 - DDL审计记录 + 异步通知(notify)》 《PostgreSQL 审计 - pg_audit module》 《PostgreSQL 开启“审计日志、时间记录”带来的性能影响有多少?》 九、参考文档 https://www.enterprisedb.com/advanced-downloads 阿里云ppas(兼容Oracle) 阿里云adam Oracle评估、迁移 https://www.postgresql.org/docs/current/static/app-psql.html https://www.postgresql.org/docs/current/static/pgbench.html digoal’s 大量PostgreSQL文章入口PostgreSQL pipelinedb 流计算插件 - IoT应用 - 实时轨迹聚合2018-11-01T00:00:00+08:002018-11-01T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/11/01/02<h2 id="背景">背景</h2>
<p>IoT场景,车联网场景,共享单车场景,人的行为位点等,终端实时上报的是孤立的位点,我们需要将其补齐成轨迹。</p>
<p>例如共享单车,下单,开锁,生成订单,骑行,关闭订单,关锁。这个过程有一个唯一的订单号,每次上报的位点会包含时间,订单号,位置。</p>
<p>根据订单号,将点聚合为轨迹。</p>
<p>使用pipelinedb插件,可以实时的实现聚合。</p>
<h2 id="例子">例子</h2>
<p>以ECS (centos 7.x x64), postgresql 10 为例</p>
<p>1、编译zeromq</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget https://github.com/zeromq/libzmq/releases/download/v4.2.5/zeromq-4.2.5.tar.gz
tar -zxvf zeromq-4.2.5.tar.gz
cd zeromq-4.2.5
./configure
make
make install
</code></pre></div></div>
<p>2、编译pipelinedb</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget https://github.com/pipelinedb/pipelinedb/archive/1.0.0rev4.tar.gz
tar -zxvf 1.0.0rev4.tar.gz
cd pipelinedb-1.0.0rev4/
vi Makefile
SHLIB_LINK += /usr/local/lib/libzmq.so -lstdc++
. /var/lib/pgsql/env.sh 1925
USE_PGXS=1 make
USE_PGXS=1 make install
</code></pre></div></div>
<p>3、配置postgresql.conf</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>max_worker_processes = 512
shared_preload_libraries = 'pipelinedb'
pipelinedb.num_combiners=16
pipelinedb.num_workers=8
pipelinedb.num_queues=4
pipelinedb.continuous_queries_enabled=true
</code></pre></div></div>
<p>重启</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pg_ctl restart -m fast
</code></pre></div></div>
<p>4、安装插件</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# create extension pipelinedb;
</code></pre></div></div>
<p>5、创建stream,实时写入轨迹点</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CREATE FOREIGN TABLE s1 ( order_id int8, ts timestamp, pos geometry )
SERVER pipelinedb;
</code></pre></div></div>
<p>6、创建Continue view,实时聚合</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CREATE VIEW cv1 WITH (action=materialize ) AS
select order_id, min(ts) min_ts, array_agg(ts||','||st_astext(pos)) as seg
from s1
group by order_id;
</code></pre></div></div>
<p>激活视图(默认已激活)</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>select pipelinedb.activate('public.cv1');
</code></pre></div></div>
<p>7、压测</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vi test.sql
\set order_id random(1,100000)
\set x random(70,90)
\set y random(120,125)
insert into s1 (order_id, ts, pos) values (:order_id, clock_timestamp(), st_makepoint(:x+10*random(), :y+10*random()));
pgbench -M prepared -n -r -P 1 -f ./test.sql -c 256 -j 256 -T 120
</code></pre></div></div>
<p>8、压测结果</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>transaction type: ./test.sql
scaling factor: 1
query mode: prepared
number of clients: 256
number of threads: 256
duration: 120 s
number of transactions actually processed: 17614607
latency average = 1.740 ms
latency stddev = 1.730 ms
tps = 146550.933776 (including connections establishing)
tps = 146906.482277 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set order_id random(1,10000000)
0.001 \set x random(70,90)
0.000 \set y random(120,125)
1.742 insert into s1 (order_id, ts, pos) values (:order_id, clock_timestamp(), st_makepoint(:x+10*random(), :y+10*random()));
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# \x
Expanded display is on.
-[ RECORD 17 ]---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
order_id | 8672585
min_ts | 2018-11-01 18:44:08.140027
seg | {"2018-11-01 18:44:08.140027,POINT(78.3615547642112 121.881739947945)","2018-11-01 18:44:11.739248,POINT(80.9645632216707 121.450987955555)"}
-[ RECORD 18 ]---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
order_id | 4011211
min_ts | 2018-11-01 18:44:08.166407
seg | {"2018-11-01 18:44:08.166407,POINT(87.126777020283 132.819293198176)","2018-11-01 18:44:11.524995,POINT(80.482944605872 126.906906872056)"}
-[ RECORD 19 ]---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
order_id | 2468486
min_ts | 2018-11-01 18:44:08.135136
seg | {"2018-11-01 18:44:08.135136,POINT(84.7732630362734 132.659516767599)","2018-11-01 18:44:20.603312,POINT(87.6352122295648 132.18647258915)","2018-11-01 18:44:19.447776,POINT(94.9817024609074 131.295661441982)"}
</code></pre></div></div>
<p>9、历史轨迹的保留</p>
<p>设置cv生命周期,自动清理老化数据</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# select pipelinedb.set_ttl('cv1', interval '1 hour' , 'min_ts');
-[ RECORD 1 ]-----
set_ttl | (3600,2)
</code></pre></div></div>
<p>创建目标持久化表</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>create table cv1_persist (like cv1);
</code></pre></div></div>
<p>创建时间字段索引(CV1)</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# create index idx_1 on cv1 (min_ts);
CREATE INDEX
</code></pre></div></div>
<p>ETL形式,将数据从cv抽取到目标持久化表</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# insert into cv1_persist select * from cv1 where min_ts <= '2018-01-01';
INSERT 0 0
</code></pre></div></div>
<h2 id="参考">参考</h2>
<p>https://github.com/pipelinedb/pipelinedb</p>
<p>http://zeromq.org/intro:get-the-software</p>
<p>https://www.pipelinedb.com/</p>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>Digoal.zhoudigoal@126.com背景 IoT场景,车联网场景,共享单车场景,人的行为位点等,终端实时上报的是孤立的位点,我们需要将其补齐成轨迹。 例如共享单车,下单,开锁,生成订单,骑行,关闭订单,关锁。这个过程有一个唯一的订单号,每次上报的位点会包含时间,订单号,位置。 根据订单号,将点聚合为轨迹。 使用pipelinedb插件,可以实时的实现聚合。 例子 以ECS (centos 7.x x64), postgresql 10 为例 1、编译zeromq wget https://github.com/zeromq/libzmq/releases/download/v4.2.5/zeromq-4.2.5.tar.gz tar -zxvf zeromq-4.2.5.tar.gz cd zeromq-4.2.5 ./configure make make install 2、编译pipelinedb wget https://github.com/pipelinedb/pipelinedb/archive/1.0.0rev4.tar.gz tar -zxvf 1.0.0rev4.tar.gz cd pipelinedb-1.0.0rev4/ vi Makefile SHLIB_LINK += /usr/local/lib/libzmq.so -lstdc++ . /var/lib/pgsql/env.sh 1925 USE_PGXS=1 make USE_PGXS=1 make install 3、配置postgresql.conf max_worker_processes = 512 shared_preload_libraries = 'pipelinedb' pipelinedb.num_combiners=16 pipelinedb.num_workers=8 pipelinedb.num_queues=4 pipelinedb.continuous_queries_enabled=true 重启 pg_ctl restart -m fast 4、安装插件 postgres=# create extension pipelinedb; 5、创建stream,实时写入轨迹点 CREATE FOREIGN TABLE s1 ( order_id int8, ts timestamp, pos geometry ) SERVER pipelinedb; 6、创建Continue view,实时聚合 CREATE VIEW cv1 WITH (action=materialize ) AS select order_id, min(ts) min_ts, array_agg(ts||','||st_astext(pos)) as seg from s1 group by order_id; 激活视图(默认已激活) select pipelinedb.activate('public.cv1'); 7、压测 vi test.sql \set order_id random(1,100000) \set x random(70,90) \set y random(120,125) insert into s1 (order_id, ts, pos) values (:order_id, clock_timestamp(), st_makepoint(:x+10*random(), :y+10*random())); pgbench -M prepared -n -r -P 1 -f ./test.sql -c 256 -j 256 -T 120 8、压测结果 transaction type: ./test.sql scaling factor: 1 query mode: prepared number of clients: 256 number of threads: 256 duration: 120 s number of transactions actually processed: 17614607 latency average = 1.740 ms latency stddev = 1.730 ms tps = 146550.933776 (including connections establishing) tps = 146906.482277 (excluding connections establishing) script statistics: - statement latencies in milliseconds: 0.002 \set order_id random(1,10000000) 0.001 \set x random(70,90) 0.000 \set y random(120,125) 1.742 insert into s1 (order_id, ts, pos) values (:order_id, clock_timestamp(), st_makepoint(:x+10*random(), :y+10*random())); postgres=# \x Expanded display is on. -[ RECORD 17 ]--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- order_id | 8672585 min_ts | 2018-11-01 18:44:08.140027 seg | {"2018-11-01 18:44:08.140027,POINT(78.3615547642112 121.881739947945)","2018-11-01 18:44:11.739248,POINT(80.9645632216707 121.450987955555)"} -[ RECORD 18 ]--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- order_id | 4011211 min_ts | 2018-11-01 18:44:08.166407 seg | {"2018-11-01 18:44:08.166407,POINT(87.126777020283 132.819293198176)","2018-11-01 18:44:11.524995,POINT(80.482944605872 126.906906872056)"} -[ RECORD 19 ]--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- order_id | 2468486 min_ts | 2018-11-01 18:44:08.135136 seg | {"2018-11-01 18:44:08.135136,POINT(84.7732630362734 132.659516767599)","2018-11-01 18:44:20.603312,POINT(87.6352122295648 132.18647258915)","2018-11-01 18:44:19.447776,POINT(94.9817024609074 131.295661441982)"} 9、历史轨迹的保留 设置cv生命周期,自动清理老化数据 postgres=# select pipelinedb.set_ttl('cv1', interval '1 hour' , 'min_ts'); -[ RECORD 1 ]----- set_ttl | (3600,2) 创建目标持久化表 create table cv1_persist (like cv1); 创建时间字段索引(CV1) postgres=# create index idx_1 on cv1 (min_ts); CREATE INDEX ETL形式,将数据从cv抽取到目标持久化表 postgres=# insert into cv1_persist select * from cv1 where min_ts <= '2018-01-01'; INSERT 0 0 参考 https://github.com/pipelinedb/pipelinedb http://zeromq.org/intro:get-the-software https://www.pipelinedb.com/ digoal’s 大量PostgreSQL文章入口PostgreSQL plpgsql 存储过程、函数 - 状态、异常变量打印、异常捕获… - GET [STACKED] DIAGNOSTICS2018-10-29T00:00:00+08:002018-10-29T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/10/29/03<h2 id="背景">背景</h2>
<p>使用GET STACKED DIAGNOSTICS捕获异常时的STACK内容。</p>
<p>使用GET DIAGNOSTICS捕获运行过程中的状态值。</p>
<h2 id="get-diagnostics捕获运行过程中的状态值">GET DIAGNOSTICS捕获运行过程中的状态值</h2>
<p>There are several ways to determine the effect of a command. The first method is to use the GET DIAGNOSTICS command, which has the form:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET [ CURRENT ] DIAGNOSTICS variable { = | := } item [ , ... ];
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET DIAGNOSTICS integer_var = ROW_COUNT;
</code></pre></div></div>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>ROW_COUNT</td>
<td>bigint</td>
<td>the number of rows processed by the most recent SQL command</td>
</tr>
<tr>
<td>RESULT_OID</td>
<td>oid</td>
<td>the OID of the last row inserted by the most recent SQL command (only useful after an INSERT command into a table having OIDs)</td>
</tr>
<tr>
<td>PG_CONTEXT</td>
<td>text</td>
<td>line(s) of text describing the current call stack (see Section 43.6.8)</td>
</tr>
</tbody>
</table>
<p>The GET DIAGNOSTICS command, previously described in Section 43.5.5, retrieves information about current execution state (whereas the GET STACKED DIAGNOSTICS command discussed above reports information about the execution state as of a previous error).</p>
<p>例子</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CREATE OR REPLACE FUNCTION outer_func() RETURNS integer AS $$
BEGIN
RETURN inner_func();
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION inner_func() RETURNS integer AS $$
DECLARE
stack text;
BEGIN
GET DIAGNOSTICS stack = PG_CONTEXT;
RAISE NOTICE E'--- Call Stack ---\n%', stack;
RETURN 1;
END;
$$ LANGUAGE plpgsql;
SELECT outer_func();
NOTICE: --- Call Stack ---
PL/pgSQL function inner_func() line 5 at GET DIAGNOSTICS
PL/pgSQL function outer_func() line 3 at RETURN
CONTEXT: PL/pgSQL function outer_func() line 3 at RETURN
outer_func
------------
1
(1 row)
</code></pre></div></div>
<h2 id="get-stacked-diagnostics捕获异常时的stack内容">GET STACKED DIAGNOSTICS捕获异常时的STACK内容</h2>
<p>GET STACKED DIAGNOSTICS … PG_EXCEPTION_CONTEXT returns the same sort of stack trace, but describing the location at which an error was detected, rather than the current location.</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>RETURNED_SQLSTATE</td>
<td>text</td>
<td>the SQLSTATE error code of the exception</td>
</tr>
<tr>
<td>COLUMN_NAME</td>
<td>text</td>
<td>the name of the column related to exception</td>
</tr>
<tr>
<td>CONSTRAINT_NAME</td>
<td>text</td>
<td>the name of the constraint related to exception</td>
</tr>
<tr>
<td>PG_DATATYPE_NAME</td>
<td>text</td>
<td>the name of the data type related to exception</td>
</tr>
<tr>
<td>MESSAGE_TEXT</td>
<td>text</td>
<td>the text of the exception’s primary message</td>
</tr>
<tr>
<td>TABLE_NAME</td>
<td>text</td>
<td>the name of the table related to exception</td>
</tr>
<tr>
<td>SCHEMA_NAME</td>
<td>text</td>
<td>the name of the schema related to exception</td>
</tr>
<tr>
<td>PG_EXCEPTION_DETAIL</td>
<td>text</td>
<td>the text of the exception’s detail message, if any</td>
</tr>
<tr>
<td>PG_EXCEPTION_HINT</td>
<td>text</td>
<td>the text of the exception’s hint message, if any</td>
</tr>
<tr>
<td>PG_EXCEPTION_CONTEXT</td>
<td>text</td>
<td>line(s) of text describing the call stack at the time of the exception (see Section 43.6.8)</td>
</tr>
</tbody>
</table>
<p>例子</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DECLARE
text_var1 text;
text_var2 text;
text_var3 text;
BEGIN
-- some processing which might cause an exception
...
EXCEPTION WHEN OTHERS THEN
GET STACKED DIAGNOSTICS text_var1 = MESSAGE_TEXT,
text_var2 = PG_EXCEPTION_DETAIL,
text_var3 = PG_EXCEPTION_HINT;
END;
</code></pre></div></div>
<h2 id="参考">参考</h2>
<p>https://www.postgresql.org/docs/11/static/plpgsql-control-structures.html#PLPGSQL-CALL-STACK</p>
<p>https://www.postgresql.org/docs/11/static/plpgsql-statements.html</p>
<p><a href="../201006/20100603_01.md">《Using “GET DIAGNOSTICS integer_var = ROW_COUNT;” capture rows effect by the last SQL》</a></p>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>Digoal.zhoudigoal@126.com背景 使用GET STACKED DIAGNOSTICS捕获异常时的STACK内容。 使用GET DIAGNOSTICS捕获运行过程中的状态值。 GET DIAGNOSTICS捕获运行过程中的状态值 There are several ways to determine the effect of a command. The first method is to use the GET DIAGNOSTICS command, which has the form: GET [ CURRENT ] DIAGNOSTICS variable { = | := } item [ , ... ]; GET DIAGNOSTICS integer_var = ROW_COUNT; Name Type Description ROW_COUNT bigint the number of rows processed by the most recent SQL command RESULT_OID oid the OID of the last row inserted by the most recent SQL command (only useful after an INSERT command into a table having OIDs) PG_CONTEXT text line(s) of text describing the current call stack (see Section 43.6.8) The GET DIAGNOSTICS command, previously described in Section 43.5.5, retrieves information about current execution state (whereas the GET STACKED DIAGNOSTICS command discussed above reports information about the execution state as of a previous error). 例子 CREATE OR REPLACE FUNCTION outer_func() RETURNS integer AS $$ BEGIN RETURN inner_func(); END; $$ LANGUAGE plpgsql; CREATE OR REPLACE FUNCTION inner_func() RETURNS integer AS $$ DECLARE stack text; BEGIN GET DIAGNOSTICS stack = PG_CONTEXT; RAISE NOTICE E'--- Call Stack ---\n%', stack; RETURN 1; END; $$ LANGUAGE plpgsql; SELECT outer_func(); NOTICE: --- Call Stack --- PL/pgSQL function inner_func() line 5 at GET DIAGNOSTICS PL/pgSQL function outer_func() line 3 at RETURN CONTEXT: PL/pgSQL function outer_func() line 3 at RETURN outer_func ------------ 1 (1 row) GET STACKED DIAGNOSTICS捕获异常时的STACK内容 GET STACKED DIAGNOSTICS … PG_EXCEPTION_CONTEXT returns the same sort of stack trace, but describing the location at which an error was detected, rather than the current location. Name Type Description RETURNED_SQLSTATE text the SQLSTATE error code of the exception COLUMN_NAME text the name of the column related to exception CONSTRAINT_NAME text the name of the constraint related to exception PG_DATATYPE_NAME text the name of the data type related to exception MESSAGE_TEXT text the text of the exception’s primary message TABLE_NAME text the name of the table related to exception SCHEMA_NAME text the name of the schema related to exception PG_EXCEPTION_DETAIL text the text of the exception’s detail message, if any PG_EXCEPTION_HINT text the text of the exception’s hint message, if any PG_EXCEPTION_CONTEXT text line(s) of text describing the call stack at the time of the exception (see Section 43.6.8) 例子 DECLARE text_var1 text; text_var2 text; text_var3 text; BEGIN -- some processing which might cause an exception ... EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS text_var1 = MESSAGE_TEXT, text_var2 = PG_EXCEPTION_DETAIL, text_var3 = PG_EXCEPTION_HINT; END; 参考 https://www.postgresql.org/docs/11/static/plpgsql-control-structures.html#PLPGSQL-CALL-STACK https://www.postgresql.org/docs/11/static/plpgsql-statements.html 《Using “GET DIAGNOSTICS integer_var = ROW_COUNT;” capture rows effect by the last SQL》 digoal’s 大量PostgreSQL文章入口PostgreSQL datediff 日期间隔(单位转换)兼容SQL用法2018-10-29T00:00:00+08:002018-10-29T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/10/29/02<h2 id="背景">背景</h2>
<p>使用datediff,对时间或日期相减,得到的间隔,转换为目标单位(日、月、季度、年、小时、秒。。。等)的数值。</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DATEDIFF ( datepart, {date|timestamp}, {date|timestamp} )
</code></pre></div></div>
<p>周</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>select datediff(week,'2009-01-01','2009-12-31') as numweeks;
numweeks
----------
52
(1 row)
</code></pre></div></div>
<p>季度</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>select datediff(qtr, '1998-07-01', current_date);
date_diff
-----------
40
(1 row)
</code></pre></div></div>
<p>PostgreSQL中时间和日期可以相互加减,得到同样的结果使用extract。</p>
<h2 id="postgresql-age-extract-epoch">PostgreSQL age, extract epoch</h2>
<p>使用age函数对时间进行计算,得到interval。</p>
<p>使用extract epoch对interval 转换得到秒。</p>
<p>根据需求计算,转换为其他单位:日、月、季度、年、小时、秒等。</p>
<h3 id="例子">例子</h3>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# SELECT age(TO_TIMESTAMP('2016-01-01', 'YYYY-MM-DD'),TO_TIMESTAMP('2015-03-01', 'YYYY-MM-DD'));
age
---------
10 mons
(1 row)
postgres=# SELECT EXTRACT(epoch FROM age(TO_TIMESTAMP('2016-01-01', 'YYYY-MM-DD'),TO_TIMESTAMP('2015-03-01', 'YYYY-MM-DD')));
date_part
-----------
25920000
(1 row)
</code></pre></div></div>
<p>重新计算即可</p>
<p>月</p>
<p>日</p>
<p>年</p>
<p>小时</p>
<p>周</p>
<p>。。。 。。。</p>
<p>例如</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>天
postgres=# SELECT EXTRACT(epoch FROM age(TO_TIMESTAMP('2016-01-01', 'YYYY-MM-DD'),TO_TIMESTAMP('2015-03-01', 'YYYY-MM-DD')))/86400;
?column?
----------
300
(1 row)
月
postgres=# SELECT EXTRACT(epoch FROM age(TO_TIMESTAMP('2016-01-01', 'YYYY-MM-DD'),TO_TIMESTAMP('2015-03-01', 'YYYY-MM-DD')))/86400/30;
?column?
----------
10
(1 row)
</code></pre></div></div>
<h3 id="extract支持的时间单位">extract支持的时间单位</h3>
<p>https://www.postgresql.org/docs/11/static/functions-datetime.html</p>
<p>src/backend/utils/adt/timestamp.c</p>
<h2 id="参考">参考</h2>
<p>https://docs.aws.amazon.com/zh_cn/redshift/latest/dg/r_DATEDIFF_function.html</p>
<p>https://www.postgresql.org/docs/11/static/functions-datetime.html</p>
<p>src/backend/utils/adt/timestamp.c</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/* interval_part()
* Extract specified field from interval.
*/
Datum
interval_part(PG_FUNCTION_ARGS)
{
text *units = PG_GETARG_TEXT_PP(0);
Interval *interval = PG_GETARG_INTERVAL_P(1);
float8 result;
int type,
val;
char *lowunits;
fsec_t fsec;
struct pg_tm tt,
*tm = &tt;
lowunits = downcase_truncate_identifier(VARDATA_ANY(units),
VARSIZE_ANY_EXHDR(units),
false);
type = DecodeUnits(0, lowunits, &val);
if (type == UNKNOWN_FIELD)
type = DecodeSpecial(0, lowunits, &val);
if (type == UNITS)
{
if (interval2tm(*interval, tm, &fsec) == 0)
{
switch (val)
{
case DTK_MICROSEC:
result = tm->tm_sec * 1000000.0 + fsec;
break;
case DTK_MILLISEC:
result = tm->tm_sec * 1000.0 + fsec / 1000.0;
break;
case DTK_SECOND:
result = tm->tm_sec + fsec / 1000000.0;
break;
case DTK_MINUTE:
result = tm->tm_min;
break;
case DTK_HOUR:
result = tm->tm_hour;
break;
case DTK_DAY:
result = tm->tm_mday;
break;
case DTK_MONTH:
result = tm->tm_mon;
break;
case DTK_QUARTER:
result = (tm->tm_mon / 3) + 1;
break;
case DTK_YEAR:
result = tm->tm_year;
break;
case DTK_DECADE:
/* caution: C division may have negative remainder */
result = tm->tm_year / 10;
break;
case DTK_CENTURY:
/* caution: C division may have negative remainder */
result = tm->tm_year / 100;
break;
case DTK_MILLENNIUM:
/* caution: C division may have negative remainder */
result = tm->tm_year / 1000;
break;
default:
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("interval units \"%s\" not supported",
lowunits)));
result = 0;
}
}
else
{
elog(ERROR, "could not convert interval to tm");
result = 0;
}
}
else if (type == RESERV && val == DTK_EPOCH)
{
result = interval->time / 1000000.0;
result += ((double) DAYS_PER_YEAR * SECS_PER_DAY) * (interval->month / MONTHS_PER_YEAR);
result += ((double) DAYS_PER_MONTH * SECS_PER_DAY) * (interval->month % MONTHS_PER_YEAR);
result += ((double) SECS_PER_DAY) * interval->day;
}
else
{
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("interval units \"%s\" not recognized",
lowunits)));
result = 0;
}
PG_RETURN_FLOAT8(result);
}
</code></pre></div></div>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>Digoal.zhoudigoal@126.com背景 使用datediff,对时间或日期相减,得到的间隔,转换为目标单位(日、月、季度、年、小时、秒。。。等)的数值。 DATEDIFF ( datepart, {date|timestamp}, {date|timestamp} ) 周 select datediff(week,'2009-01-01','2009-12-31') as numweeks; numweeks ---------- 52 (1 row) 季度 select datediff(qtr, '1998-07-01', current_date); date_diff ----------- 40 (1 row) PostgreSQL中时间和日期可以相互加减,得到同样的结果使用extract。 PostgreSQL age, extract epoch 使用age函数对时间进行计算,得到interval。 使用extract epoch对interval 转换得到秒。 根据需求计算,转换为其他单位:日、月、季度、年、小时、秒等。 例子 postgres=# SELECT age(TO_TIMESTAMP('2016-01-01', 'YYYY-MM-DD'),TO_TIMESTAMP('2015-03-01', 'YYYY-MM-DD')); age --------- 10 mons (1 row) postgres=# SELECT EXTRACT(epoch FROM age(TO_TIMESTAMP('2016-01-01', 'YYYY-MM-DD'),TO_TIMESTAMP('2015-03-01', 'YYYY-MM-DD'))); date_part ----------- 25920000 (1 row) 重新计算即可 月 日 年 小时 周 。。。 。。。 例如 天 postgres=# SELECT EXTRACT(epoch FROM age(TO_TIMESTAMP('2016-01-01', 'YYYY-MM-DD'),TO_TIMESTAMP('2015-03-01', 'YYYY-MM-DD')))/86400; ?column? ---------- 300 (1 row) 月 postgres=# SELECT EXTRACT(epoch FROM age(TO_TIMESTAMP('2016-01-01', 'YYYY-MM-DD'),TO_TIMESTAMP('2015-03-01', 'YYYY-MM-DD')))/86400/30; ?column? ---------- 10 (1 row) extract支持的时间单位 https://www.postgresql.org/docs/11/static/functions-datetime.html src/backend/utils/adt/timestamp.c 参考 https://docs.aws.amazon.com/zh_cn/redshift/latest/dg/r_DATEDIFF_function.html https://www.postgresql.org/docs/11/static/functions-datetime.html src/backend/utils/adt/timestamp.c /* interval_part() * Extract specified field from interval. */ Datum interval_part(PG_FUNCTION_ARGS) { text *units = PG_GETARG_TEXT_PP(0); Interval *interval = PG_GETARG_INTERVAL_P(1); float8 result; int type, val; char *lowunits; fsec_t fsec; struct pg_tm tt, *tm = &tt; lowunits = downcase_truncate_identifier(VARDATA_ANY(units), VARSIZE_ANY_EXHDR(units), false); type = DecodeUnits(0, lowunits, &val); if (type == UNKNOWN_FIELD) type = DecodeSpecial(0, lowunits, &val); if (type == UNITS) { if (interval2tm(*interval, tm, &fsec) == 0) { switch (val) { case DTK_MICROSEC: result = tm->tm_sec * 1000000.0 + fsec; break; case DTK_MILLISEC: result = tm->tm_sec * 1000.0 + fsec / 1000.0; break; case DTK_SECOND: result = tm->tm_sec + fsec / 1000000.0; break; case DTK_MINUTE: result = tm->tm_min; break; case DTK_HOUR: result = tm->tm_hour; break; case DTK_DAY: result = tm->tm_mday; break; case DTK_MONTH: result = tm->tm_mon; break; case DTK_QUARTER: result = (tm->tm_mon / 3) + 1; break; case DTK_YEAR: result = tm->tm_year; break; case DTK_DECADE: /* caution: C division may have negative remainder */ result = tm->tm_year / 10; break; case DTK_CENTURY: /* caution: C division may have negative remainder */ result = tm->tm_year / 100; break; case DTK_MILLENNIUM: /* caution: C division may have negative remainder */ result = tm->tm_year / 1000; break; default: ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("interval units \"%s\" not supported", lowunits))); result = 0; } } else { elog(ERROR, "could not convert interval to tm"); result = 0; } } else if (type == RESERV && val == DTK_EPOCH) { result = interval->time / 1000000.0; result += ((double) DAYS_PER_YEAR * SECS_PER_DAY) * (interval->month / MONTHS_PER_YEAR); result += ((double) DAYS_PER_MONTH * SECS_PER_DAY) * (interval->month % MONTHS_PER_YEAR); result += ((double) SECS_PER_DAY) * interval->day; } else { ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("interval units \"%s\" not recognized", lowunits))); result = 0; } PG_RETURN_FLOAT8(result); } digoal’s 大量PostgreSQL文章入口PostgreSQL listagg within group (order by) 聚合兼容用法 string_agg ( order by) - 行列变换,CSV构造…2018-10-29T00:00:00+08:002018-10-29T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/10/29/01<h2 id="背景">背景</h2>
<p>listagg — Rows to Delimited Strings</p>
<p>The listagg function transforms values from a group of rows into a list of values that are delimited by a configurable separator. Listagg is typically used to denormalize rows into a string of comma-separated values (CSV) or other comparable formats suitable for human reading.</p>
<p>Listagg does not apply any escaping: it is not generally possible to tell whether an occurrence of the separator in the result is an actual separator, or just part of a value. The safe use of listagg for electronic data interfaces is therefore limited to cases in which an unambiguous separator can be selected, e.g. when aggregating numbers, dates, or strings that are known to not contain the separator.</p>
<p>When implementing electronic data interfaces, arrays and document types (JSON, XML) are advantageous as they offer type safety, or at least proper escaping.</p>
<h2 id="postgresql-string_agg">PostgreSQL string_agg</h2>
<p>string_agg 代替listagg,实现同样功能。</p>
<p>建表</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# create table tbl1 (gid int, val text, ts timestamp default clock_timestamp());
CREATE TABLE
</code></pre></div></div>
<p>写入测试数据</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# insert into tbl1 values (1,'a'),(1,'b'),(1,null),(2,'test'),(2,'a""b"c'),(3,'fw');
INSERT 0 6
</code></pre></div></div>
<p>数据</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# select * from tbl1;
gid | val | ts
-----+--------+----------------------------
1 | a | 2018-10-29 21:00:24.593859
1 | b | 2018-10-29 21:00:24.593994
1 | | 2018-10-29 21:00:24.593997
2 | test | 2018-10-29 21:00:24.593998
2 | a""b"c | 2018-10-29 21:00:24.594
3 | fw | 2018-10-29 21:00:24.594001
(6 rows)
</code></pre></div></div>
<p>逆向聚合,双引号作为quote字符,转义文本内的双引号,空值使用NULL表示。</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# select gid, string_agg(coalesce('"'||replace(val,'"','\"')||'"','NULL'),',' order by ts desc) from tbl1 group by gid;
gid | string_agg
-----+--------------------
1 | NULL,"b","a"
2 | "a\"\"b\"c","test"
3 | "fw"
(3 rows)
</code></pre></div></div>
<p>正向聚合,双引号作为quote字符,转义文本内的双引号,空值使用NULL表示。</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# select gid, string_agg(coalesce('"'||replace(val,'"','\"')||'"','NULL'),',' order by ts) from tbl1 group by gid;
gid | string_agg
-----+--------------------
1 | "a","b",NULL
2 | "test","a\"\"b\"c"
3 | "fw"
(3 rows)
</code></pre></div></div>
<p>正向聚合,不使用QUOTE,直接去除NULL值</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# select gid, string_agg(val,',' order by ts) from tbl1 group by gid;
gid | string_agg
-----+-------------
1 | a,b
2 | test,a""b"c
3 | fw
(3 rows)
</code></pre></div></div>
<h2 id="参考">参考</h2>
<p><a href="../201705/20170504_04.md">《PostgreSQL 聚合表达式 FILTER , order , within group, over window 用法》</a></p>
<p><a href="../201504/20150407_01.md">《PostgreSQL aggregate function 3 : Aggregate Functions for Ordered-Set》</a></p>
<p>https://modern-sql.com/feature/listagg</p>
<p>https://www.postgresql.org/docs/11/static/functions-aggregate.html</p>
<p>https://wiki.postgresql.org/wiki/PostgreSQL_vs_SQL_Standard</p>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>Digoal.zhoudigoal@126.com背景 listagg — Rows to Delimited Strings The listagg function transforms values from a group of rows into a list of values that are delimited by a configurable separator. Listagg is typically used to denormalize rows into a string of comma-separated values (CSV) or other comparable formats suitable for human reading. Listagg does not apply any escaping: it is not generally possible to tell whether an occurrence of the separator in the result is an actual separator, or just part of a value. The safe use of listagg for electronic data interfaces is therefore limited to cases in which an unambiguous separator can be selected, e.g. when aggregating numbers, dates, or strings that are known to not contain the separator. When implementing electronic data interfaces, arrays and document types (JSON, XML) are advantageous as they offer type safety, or at least proper escaping. PostgreSQL string_agg string_agg 代替listagg,实现同样功能。 建表 postgres=# create table tbl1 (gid int, val text, ts timestamp default clock_timestamp()); CREATE TABLE 写入测试数据 postgres=# insert into tbl1 values (1,'a'),(1,'b'),(1,null),(2,'test'),(2,'a""b"c'),(3,'fw'); INSERT 0 6 数据 postgres=# select * from tbl1; gid | val | ts -----+--------+---------------------------- 1 | a | 2018-10-29 21:00:24.593859 1 | b | 2018-10-29 21:00:24.593994 1 | | 2018-10-29 21:00:24.593997 2 | test | 2018-10-29 21:00:24.593998 2 | a""b"c | 2018-10-29 21:00:24.594 3 | fw | 2018-10-29 21:00:24.594001 (6 rows) 逆向聚合,双引号作为quote字符,转义文本内的双引号,空值使用NULL表示。 postgres=# select gid, string_agg(coalesce('"'||replace(val,'"','\"')||'"','NULL'),',' order by ts desc) from tbl1 group by gid; gid | string_agg -----+-------------------- 1 | NULL,"b","a" 2 | "a\"\"b\"c","test" 3 | "fw" (3 rows) 正向聚合,双引号作为quote字符,转义文本内的双引号,空值使用NULL表示。 postgres=# select gid, string_agg(coalesce('"'||replace(val,'"','\"')||'"','NULL'),',' order by ts) from tbl1 group by gid; gid | string_agg -----+-------------------- 1 | "a","b",NULL 2 | "test","a\"\"b\"c" 3 | "fw" (3 rows) 正向聚合,不使用QUOTE,直接去除NULL值 postgres=# select gid, string_agg(val,',' order by ts) from tbl1 group by gid; gid | string_agg -----+------------- 1 | a,b 2 | test,a""b"c 3 | fw (3 rows) 参考 《PostgreSQL 聚合表达式 FILTER , order , within group, over window 用法》 《PostgreSQL aggregate function 3 : Aggregate Functions for Ordered-Set》 https://modern-sql.com/feature/listagg https://www.postgresql.org/docs/11/static/functions-aggregate.html https://wiki.postgresql.org/wiki/PostgreSQL_vs_SQL_Standard digoal’s 大量PostgreSQL文章入口PostgreSQL DNS 解析功能 - get ip, get host name2018-10-27T00:00:00+08:002018-10-27T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/10/27/01<h2 id="背景">背景</h2>
<p>在数据库中,根据主机名获得IP,根据IP解析主机名。</p>
<p>这个功能没有什么花哨,可以通过C函数或者PYTHON函数得到。</p>
<p>src/common/ip.c</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/*
* pg_getnameinfo_all - get name info for Unix, IPv4 and IPv6 sockets
*
* The API of this routine differs from the standard getnameinfo() definition
* in two ways: first, the addr parameter is declared as sockaddr_storage
* rather than struct sockaddr, and second, the node and service fields are
* guaranteed to be filled with something even on failure return.
*/
int
pg_getnameinfo_all(const struct sockaddr_storage *addr, int salen,
char *node, int nodelen,
char *service, int servicelen,
int flags)
{
int rc;
#ifdef HAVE_UNIX_SOCKETS
if (addr && addr->ss_family == AF_UNIX)
rc = getnameinfo_unix((const struct sockaddr_un *) addr, salen,
node, nodelen,
service, servicelen,
flags);
else
#endif
rc = getnameinfo((const struct sockaddr *) addr, salen,
node, nodelen,
service, servicelen,
flags);
if (rc != 0)
{
if (node)
strlcpy(node, "???", nodelen);
if (service)
strlcpy(service, "???", servicelen);
}
return rc;
}
</code></pre></div></div>
<h2 id="pgdnsres">pgdnsres</h2>
<p>有个python的插件,可以直接使用。</p>
<p>https://www.postgresql.org/ftp/projects/pgFoundry/pgdnsres/pgdnsres/1.1/</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/*
IPv4 DNS Resolution Functions for PostgreSQL
============================================
Author: Christian J. Dietrich <dietrich@internet-sicherheit.de>
Version: 1.1
License: BSD license
pg-dns-resolve contains PL/Python functions for DNS resolution at the SQL
prompt. Ever wanted to issue SELECT hostbyname('www.google.com') in
order to get its IP address(es) from the pgsql command line? Then
pg-dns-resolve is the right thing for you. See below for more examples.
INSTALLATION
============
* Make sure, you have ip4r installed. Get it from: http://pgfoundry.org/projects/ip4r/
* Make sure, you have PL/Pythonu installed and are allowed to add new functions
* PL/Pythonu must be built against Python >= 2.4
* psql [YOUR OPTIONS] < plpython_dns-functions.sql
EXAMPLES
========
NOTE: If you run any of the functions below on a massive
data set, your DNS resolver might misinterpret this as a
DoS attack and get somewhat angry. Thus, it is a good idea
to run your own local resolver.
For all of the functions there is a variant ending in "_n" which means
that on error, NULL is to be returned instead of an error string describing
the cause of the error. Some functions have a _s version which means they
return the result as a set, i.e. multiple rows.
Resolve the hostname for a given IP address:
db=# select dst, hostbyaddr(dst) from dns_per_ip limit 2;
dst | hostbyaddr
---------------+-----------------------------
192.168.1.1 | (1, 'Unbekannter Rechner')
193.232.128.6 | ns5.msk-ix.net
(2 rows)
Forward resolve www.google.de to (one of) its IP address:
db=# select hostbyname('www.google.de');
hostbyname
---------------
74.125.43.105
(1 row)
Note that on error, NULL is returned by hostbyname_n,
BUT hostbyname returns an error string instead. So if
you want to know why the resolution failed, use
hostbyname, otherwise use hostbyname_n.
db=# select hostbyname_n('deafd'), hostbyname('deafd');
hostbyname_n | hostbyname
---------------+----------------------------------------------------
| (-2, 'Der Name oder der Dienst ist nicht bekannt')
(1 row)
db=# select hostbyname_n('nonexistinghost') is NULL;
?column?
----------
true
(1 row)
If you need all IP addresses of a hostname, use addrsbyname.
DNS usually returns a different order of multiple IP addresses due
to round-robin. Note that, the list of IP addresses of
addrsbyname is sorted, thus two executions with the same
argument return the same list. This is very useful for comparisons.
db=# select addrsbyname('www.google.de');
addrsbyname
-------------------
74.125.43.103
74.125.43.104
74.125.43.105
74.125.43.106
74.125.43.147
74.125.43.99
(1 row)
If you want e.g. a comma-separated list instead of newline-separated list,
use your own separator string as the second argument to addrsbyname:
db=# select addrsbyname('www.google.de', ', ');
addrsbyname
-----------------------------------------------------------------------------------------
74.125.43.103, 74.125.43.104, 74.125.43.105, 74.125.43.106, 74.125.43.147, 74.125.43.99
(1 row)
hostsbyname works similar to addrsbyname. hostsbyname returns a list of
all hostnames associated with a given hostname, including aliases. As with
addrsbyname there are 2 variants, one using the default newline delimiter
to separate elements and one where you can specify the delimiter yourself.
The list of resulting hostnames is sorted.
db=# select hostsbyname('www.google.de', ', ');
hostsbyname
-------------------------------------------------
www.google.com, www.google.de, www.l.google.com
(1 row)
When working with sets, there are 4 interesting functions: addrsbyname_s and
addrsbyname_ns as well as hostsbyname_s and hostsbyname_ns.
Those return a set, i.e. multiple rows, instead of an aggregated
string and they are useful when working with statements such as
SELECT ...
FROM ...
WHERE xxx IN ( SELECT addrsbyname_ns('www.google.com') )
db=# SELECT addrsbyname_s('www.google.com');
addrsbyname_s
---------------
74.125.43.103
74.125.43.104
74.125.43.105
74.125.43.106
74.125.43.147
74.125.43.99
(6 rows)
Note the subtle difference: 6 rows instead of 1 row when comparing the output
of addrsbyname_s to that of addrsbyname.
db=# SELECT '74.125.43.103'::ip4 IN ( SELECT addrsbyname_s('www.google.com') );
?column?
----------
t
(1 row)
db=# SELECT hostsbyname_ns('www.google.com');
hostsbyname_ns
------------------
www.google.com
www.l.google.com
(2 rows)
Querying a non existing hostname will result in an empty set:
db=# SELECT hostsbyname_ns('nonexistinghost');
hostsbyname_ns
----------------
(0 rows)
A special case is forward-confirmed reverse DNS resolution (http://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS):
db=# SELECT fcrdns('192.203.230.10');
fcrdns
--------
f
(1 row)
db=# SELECT fcrdns('74.125.43.104');
fcrdns
--------
t
(1 row)
*/
/* ********** reverse resolution ********** */
-- returns the hostname for a given IP address
-- returns an error string for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION hostbyaddr (addr ip4)
RETURNS text
AS $$
import socket
if addr is None: return None
try:
hostname = socket.gethostbyaddr(addr)[0]
except Exception, e:
hostname = str(e)
return hostname
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostbyaddr (ip4) IS 'Returns the hostname for a given IP address. Returns an error string on resolution errors. Returns NULL on NULL input.';
-- returns the hostname for a given IP address
-- returns NULL for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION hostbyaddr_n (addr ip4)
RETURNS text
AS $$
import socket
if addr is None: return None
try:
hostname = socket.gethostbyaddr(addr)[0]
except Exception, e:
hostname = None
return hostname
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostbyaddr_n (ip4) IS 'Returns the hostname for a given IP address, NULL on error. Returns NULL on NULL input.';
-- returns true, if the given IP address passes forward-confirmed reverse DNS, false otherwise (also on error)
-- see http://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS
CREATE OR REPLACE FUNCTION fcrdns (addr ip4)
RETURNS boolean
AS $$
import socket
if addr is None: return None
try:
# reverse resolution of the given IP address returns all rhostnames
(hostname, aliaslist, ipaddrlist) = socket.gethostbyaddr(addr)
for rhostname in [hostname]+aliaslist:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(rhostname)
# if the given ip addr is at least once in ipaddrlist, we pass the test
if addr in ipaddrlist:
return True
# if the ip addr has not been in any of the ipaddrlists, we fail
return False
except Exception, e:
pass
return False
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION fcrdns (ip4) IS 'Returns true, if the given IP address passes forward-confirmed reverse DNS, false otherwise. see http://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS';
/* ********** forward resolution ********** */
-- returns an IP address for the given hostname
-- returns an error string for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION hostbyname (name text)
RETURNS text
AS $$
import socket
if name is None: return None
try:
addr = socket.gethostbyname(name)
except Exception, e:
addr = str(e)
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostbyname (text) IS 'Returns an IP address for the given hostname. Returns an error string on resolution errors. Returns NULL on NULL input.';
-- returns an IP address for the given hostname
-- returns NULL for non-resolvable hostnames
CREATE OR REPLACE FUNCTION hostbyname_n (name text)
RETURNS ip4
AS $$
import socket
if name is None: return None
try:
addr = socket.gethostbyname(name)
except Exception, e:
addr = None
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostbyname_n (text) IS 'Returns an IP address for the given hostname, NULL on error. Returns NULL on NULL input.';
-- returns all IP addresses for the given hostname
-- returns an error string for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION addrsbyname (name text)
RETURNS text
AS $$
import socket, string
if name is None: return None
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = string.join(sorted(ipaddrlist), '\n')
except Exception, e:
addr = str(e)
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION addrsbyname (text) IS 'Returns all IP addresses for the given hostname. Returns an error string on resolution errors. Returns NULL on NULL input.';
-- returns all IP addresses for the given hostname as a set (multiple rows)
-- returns an error string for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION addrsbyname_s (name text)
RETURNS SETOF ip4
AS $$
import socket, string
if name is None: return []
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = sorted(ipaddrlist)
except Exception, e:
addr = [str(e)]
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION addrsbyname_s (text) IS 'Returns all IP addresses for the given hostname as multiple rows. Returns an error string on resolution errors. Returns empty set on NULL input.';
-- returns all IP addresses for the given hostname as a string where the elements are separated by sep
-- returns an error string for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION addrsbyname (name text, sep text)
RETURNS text
AS $$
import socket, string
if name is None or sep is None: return None
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = string.join(sorted(ipaddrlist), sep)
except Exception, e:
addr = str(e)
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION addrsbyname (text, text) IS 'Returns all IP addresses for the given hostname with custom delimiter. Returns an error string on resolution errors. Returns NULL on NULL input.';
-- returns all IP addresses for the given hostname
-- returns NULL for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION addrsbyname_n (name text)
RETURNS text
AS $$
import socket, string
if name is None: return None
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = string.join(sorted(ipaddrlist), '\n')
except Exception, e:
addr = None
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION addrsbyname_n (text) IS 'Returns all IP addresses for the given hostname, NULL on error. Returns NULL on NULL input.';
-- returns all IP addresses for the given hostname as a set (multiple rows)
-- returns an empty set for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION addrsbyname_ns (name text)
RETURNS SETOF ip4
AS $$
import socket, string
if name is None: return []
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = sorted(ipaddrlist)
except Exception, e:
addr = ()
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION addrsbyname_ns (text) IS 'Returns all IP addresses for the given hostname as a set (multiple rows), empty set on error or NULL input.';
-- returns all IP addresses for the given hostname as a string where the elements are separated by sep
-- returns NULL for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION addrsbyname_n (name text, sep text)
RETURNS text
AS $$
import socket, string
if name is None or sep is None: return None
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = string.join(sorted(ipaddrlist), sep)
except Exception, e:
addr = None
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION addrsbyname_n (text, text) IS 'Returns all IP addresses for the given hostname with custom delimiter, NULL on error or NULL input.';
/* ********** hostname alias resolution ********** */
-- returns all hostnames for the given hostname
-- returns an error string for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION hostsbyname (name text)
RETURNS text
AS $$
import socket, string
if name is None: return None
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = string.join(sorted([hostname] + aliaslist), '\n')
except Exception, e:
addr = str(e)
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostsbyname (text) IS 'Returns all hostnames for the given hostname. Returns an error string on resolution errors. Returns NULL on NULL input.';
-- returns all hostnames for the given hostname as a string where the elements are separated by sep
-- returns an error string for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION hostsbyname (name text, sep text)
RETURNS text
AS $$
import socket, string
if name is None or sep is None: return None
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = string.join(sorted([hostname] + aliaslist), sep)
except Exception, e:
addr = str(e)
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostsbyname (text, text) IS 'Returns all hostnames for the given hostname with custom delimiter. Returns an error string on resolution errors. Returns NULL on NULL input.';
-- returns all hostnames for the given hostname as a set
-- returns an error string for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION hostsbyname_s (name text)
RETURNS SETOF text
AS $$
import socket, string
if name is None: return []
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = sorted([hostname] + aliaslist)
except Exception, e:
addr = [str(e)]
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostsbyname_s (text) IS 'Returns all hostnames for the given hostname as a set. Returns an error string on resolution errors. Returns empty set on NULL input.';
-- returns all hostnames for the given hostname
-- returns NULL for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION hostsbyname_n (name text)
RETURNS text
AS $$
import socket, string
if name is None: return None
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = string.join(sorted([hostname] + aliaslist), '\n')
except Exception, e:
addr = None
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostsbyname_n (text) IS 'Returns all hostnames for the given hostname, NULL on error. Returns NULL on NULL input.';
-- returns all hostnames for the given hostname as a string where the elements are separated by sep
-- returns NULL for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION hostsbyname_n (name text, sep text)
RETURNS text
AS $$
import socket, string
if name is None or sep is None: return None
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = string.join(sorted([hostname] + aliaslist), sep)
except Exception, e:
addr = None
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostsbyname_n (text, text) IS 'Returns all hostnames for the given hostname with custom delimiter, NULL on error. Returns NULL on NULL input.';
-- returns all hostnames for the given hostname as a set
-- returns an error string for non-resolvable hostnames or resolution errors
CREATE OR REPLACE FUNCTION hostsbyname_ns (name text)
RETURNS SETOF text
AS $$
import socket, string
if name is None: return []
try:
(hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name)
addr = sorted([hostname] + aliaslist)
except Exception, e:
addr = ()
return addr
$$ LANGUAGE plpythonu;
COMMENT ON FUNCTION hostsbyname_ns (text) IS 'Returns all hostnames for the given hostname as a set, on error or on NULL input returns empty set.';
/* end of file */
</code></pre></div></div>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>Digoal.zhoudigoal@126.com背景 在数据库中,根据主机名获得IP,根据IP解析主机名。 这个功能没有什么花哨,可以通过C函数或者PYTHON函数得到。 src/common/ip.c /* * pg_getnameinfo_all - get name info for Unix, IPv4 and IPv6 sockets * * The API of this routine differs from the standard getnameinfo() definition * in two ways: first, the addr parameter is declared as sockaddr_storage * rather than struct sockaddr, and second, the node and service fields are * guaranteed to be filled with something even on failure return. */ int pg_getnameinfo_all(const struct sockaddr_storage *addr, int salen, char *node, int nodelen, char *service, int servicelen, int flags) { int rc; #ifdef HAVE_UNIX_SOCKETS if (addr && addr->ss_family == AF_UNIX) rc = getnameinfo_unix((const struct sockaddr_un *) addr, salen, node, nodelen, service, servicelen, flags); else #endif rc = getnameinfo((const struct sockaddr *) addr, salen, node, nodelen, service, servicelen, flags); if (rc != 0) { if (node) strlcpy(node, "???", nodelen); if (service) strlcpy(service, "???", servicelen); } return rc; } pgdnsres 有个python的插件,可以直接使用。 https://www.postgresql.org/ftp/projects/pgFoundry/pgdnsres/pgdnsres/1.1/ /* IPv4 DNS Resolution Functions for PostgreSQL ============================================ Author: Christian J. Dietrich <dietrich@internet-sicherheit.de> Version: 1.1 License: BSD license pg-dns-resolve contains PL/Python functions for DNS resolution at the SQL prompt. Ever wanted to issue SELECT hostbyname('www.google.com') in order to get its IP address(es) from the pgsql command line? Then pg-dns-resolve is the right thing for you. See below for more examples. INSTALLATION ============ * Make sure, you have ip4r installed. Get it from: http://pgfoundry.org/projects/ip4r/ * Make sure, you have PL/Pythonu installed and are allowed to add new functions * PL/Pythonu must be built against Python >= 2.4 * psql [YOUR OPTIONS] < plpython_dns-functions.sql EXAMPLES ======== NOTE: If you run any of the functions below on a massive data set, your DNS resolver might misinterpret this as a DoS attack and get somewhat angry. Thus, it is a good idea to run your own local resolver. For all of the functions there is a variant ending in "_n" which means that on error, NULL is to be returned instead of an error string describing the cause of the error. Some functions have a _s version which means they return the result as a set, i.e. multiple rows. Resolve the hostname for a given IP address: db=# select dst, hostbyaddr(dst) from dns_per_ip limit 2; dst | hostbyaddr ---------------+----------------------------- 192.168.1.1 | (1, 'Unbekannter Rechner') 193.232.128.6 | ns5.msk-ix.net (2 rows) Forward resolve www.google.de to (one of) its IP address: db=# select hostbyname('www.google.de'); hostbyname --------------- 74.125.43.105 (1 row) Note that on error, NULL is returned by hostbyname_n, BUT hostbyname returns an error string instead. So if you want to know why the resolution failed, use hostbyname, otherwise use hostbyname_n. db=# select hostbyname_n('deafd'), hostbyname('deafd'); hostbyname_n | hostbyname ---------------+---------------------------------------------------- | (-2, 'Der Name oder der Dienst ist nicht bekannt') (1 row) db=# select hostbyname_n('nonexistinghost') is NULL; ?column? ---------- true (1 row) If you need all IP addresses of a hostname, use addrsbyname. DNS usually returns a different order of multiple IP addresses due to round-robin. Note that, the list of IP addresses of addrsbyname is sorted, thus two executions with the same argument return the same list. This is very useful for comparisons. db=# select addrsbyname('www.google.de'); addrsbyname ------------------- 74.125.43.103 74.125.43.104 74.125.43.105 74.125.43.106 74.125.43.147 74.125.43.99 (1 row) If you want e.g. a comma-separated list instead of newline-separated list, use your own separator string as the second argument to addrsbyname: db=# select addrsbyname('www.google.de', ', '); addrsbyname ----------------------------------------------------------------------------------------- 74.125.43.103, 74.125.43.104, 74.125.43.105, 74.125.43.106, 74.125.43.147, 74.125.43.99 (1 row) hostsbyname works similar to addrsbyname. hostsbyname returns a list of all hostnames associated with a given hostname, including aliases. As with addrsbyname there are 2 variants, one using the default newline delimiter to separate elements and one where you can specify the delimiter yourself. The list of resulting hostnames is sorted. db=# select hostsbyname('www.google.de', ', '); hostsbyname ------------------------------------------------- www.google.com, www.google.de, www.l.google.com (1 row) When working with sets, there are 4 interesting functions: addrsbyname_s and addrsbyname_ns as well as hostsbyname_s and hostsbyname_ns. Those return a set, i.e. multiple rows, instead of an aggregated string and they are useful when working with statements such as SELECT ... FROM ... WHERE xxx IN ( SELECT addrsbyname_ns('www.google.com') ) db=# SELECT addrsbyname_s('www.google.com'); addrsbyname_s --------------- 74.125.43.103 74.125.43.104 74.125.43.105 74.125.43.106 74.125.43.147 74.125.43.99 (6 rows) Note the subtle difference: 6 rows instead of 1 row when comparing the output of addrsbyname_s to that of addrsbyname. db=# SELECT '74.125.43.103'::ip4 IN ( SELECT addrsbyname_s('www.google.com') ); ?column? ---------- t (1 row) db=# SELECT hostsbyname_ns('www.google.com'); hostsbyname_ns ------------------ www.google.com www.l.google.com (2 rows) Querying a non existing hostname will result in an empty set: db=# SELECT hostsbyname_ns('nonexistinghost'); hostsbyname_ns ---------------- (0 rows) A special case is forward-confirmed reverse DNS resolution (http://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS): db=# SELECT fcrdns('192.203.230.10'); fcrdns -------- f (1 row) db=# SELECT fcrdns('74.125.43.104'); fcrdns -------- t (1 row) */ /* ********** reverse resolution ********** */ -- returns the hostname for a given IP address -- returns an error string for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION hostbyaddr (addr ip4) RETURNS text AS $$ import socket if addr is None: return None try: hostname = socket.gethostbyaddr(addr)[0] except Exception, e: hostname = str(e) return hostname $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostbyaddr (ip4) IS 'Returns the hostname for a given IP address. Returns an error string on resolution errors. Returns NULL on NULL input.'; -- returns the hostname for a given IP address -- returns NULL for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION hostbyaddr_n (addr ip4) RETURNS text AS $$ import socket if addr is None: return None try: hostname = socket.gethostbyaddr(addr)[0] except Exception, e: hostname = None return hostname $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostbyaddr_n (ip4) IS 'Returns the hostname for a given IP address, NULL on error. Returns NULL on NULL input.'; -- returns true, if the given IP address passes forward-confirmed reverse DNS, false otherwise (also on error) -- see http://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS CREATE OR REPLACE FUNCTION fcrdns (addr ip4) RETURNS boolean AS $$ import socket if addr is None: return None try: # reverse resolution of the given IP address returns all rhostnames (hostname, aliaslist, ipaddrlist) = socket.gethostbyaddr(addr) for rhostname in [hostname]+aliaslist: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(rhostname) # if the given ip addr is at least once in ipaddrlist, we pass the test if addr in ipaddrlist: return True # if the ip addr has not been in any of the ipaddrlists, we fail return False except Exception, e: pass return False $$ LANGUAGE plpythonu; COMMENT ON FUNCTION fcrdns (ip4) IS 'Returns true, if the given IP address passes forward-confirmed reverse DNS, false otherwise. see http://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS'; /* ********** forward resolution ********** */ -- returns an IP address for the given hostname -- returns an error string for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION hostbyname (name text) RETURNS text AS $$ import socket if name is None: return None try: addr = socket.gethostbyname(name) except Exception, e: addr = str(e) return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostbyname (text) IS 'Returns an IP address for the given hostname. Returns an error string on resolution errors. Returns NULL on NULL input.'; -- returns an IP address for the given hostname -- returns NULL for non-resolvable hostnames CREATE OR REPLACE FUNCTION hostbyname_n (name text) RETURNS ip4 AS $$ import socket if name is None: return None try: addr = socket.gethostbyname(name) except Exception, e: addr = None return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostbyname_n (text) IS 'Returns an IP address for the given hostname, NULL on error. Returns NULL on NULL input.'; -- returns all IP addresses for the given hostname -- returns an error string for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION addrsbyname (name text) RETURNS text AS $$ import socket, string if name is None: return None try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = string.join(sorted(ipaddrlist), '\n') except Exception, e: addr = str(e) return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION addrsbyname (text) IS 'Returns all IP addresses for the given hostname. Returns an error string on resolution errors. Returns NULL on NULL input.'; -- returns all IP addresses for the given hostname as a set (multiple rows) -- returns an error string for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION addrsbyname_s (name text) RETURNS SETOF ip4 AS $$ import socket, string if name is None: return [] try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = sorted(ipaddrlist) except Exception, e: addr = [str(e)] return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION addrsbyname_s (text) IS 'Returns all IP addresses for the given hostname as multiple rows. Returns an error string on resolution errors. Returns empty set on NULL input.'; -- returns all IP addresses for the given hostname as a string where the elements are separated by sep -- returns an error string for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION addrsbyname (name text, sep text) RETURNS text AS $$ import socket, string if name is None or sep is None: return None try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = string.join(sorted(ipaddrlist), sep) except Exception, e: addr = str(e) return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION addrsbyname (text, text) IS 'Returns all IP addresses for the given hostname with custom delimiter. Returns an error string on resolution errors. Returns NULL on NULL input.'; -- returns all IP addresses for the given hostname -- returns NULL for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION addrsbyname_n (name text) RETURNS text AS $$ import socket, string if name is None: return None try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = string.join(sorted(ipaddrlist), '\n') except Exception, e: addr = None return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION addrsbyname_n (text) IS 'Returns all IP addresses for the given hostname, NULL on error. Returns NULL on NULL input.'; -- returns all IP addresses for the given hostname as a set (multiple rows) -- returns an empty set for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION addrsbyname_ns (name text) RETURNS SETOF ip4 AS $$ import socket, string if name is None: return [] try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = sorted(ipaddrlist) except Exception, e: addr = () return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION addrsbyname_ns (text) IS 'Returns all IP addresses for the given hostname as a set (multiple rows), empty set on error or NULL input.'; -- returns all IP addresses for the given hostname as a string where the elements are separated by sep -- returns NULL for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION addrsbyname_n (name text, sep text) RETURNS text AS $$ import socket, string if name is None or sep is None: return None try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = string.join(sorted(ipaddrlist), sep) except Exception, e: addr = None return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION addrsbyname_n (text, text) IS 'Returns all IP addresses for the given hostname with custom delimiter, NULL on error or NULL input.'; /* ********** hostname alias resolution ********** */ -- returns all hostnames for the given hostname -- returns an error string for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION hostsbyname (name text) RETURNS text AS $$ import socket, string if name is None: return None try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = string.join(sorted([hostname] + aliaslist), '\n') except Exception, e: addr = str(e) return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostsbyname (text) IS 'Returns all hostnames for the given hostname. Returns an error string on resolution errors. Returns NULL on NULL input.'; -- returns all hostnames for the given hostname as a string where the elements are separated by sep -- returns an error string for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION hostsbyname (name text, sep text) RETURNS text AS $$ import socket, string if name is None or sep is None: return None try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = string.join(sorted([hostname] + aliaslist), sep) except Exception, e: addr = str(e) return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostsbyname (text, text) IS 'Returns all hostnames for the given hostname with custom delimiter. Returns an error string on resolution errors. Returns NULL on NULL input.'; -- returns all hostnames for the given hostname as a set -- returns an error string for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION hostsbyname_s (name text) RETURNS SETOF text AS $$ import socket, string if name is None: return [] try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = sorted([hostname] + aliaslist) except Exception, e: addr = [str(e)] return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostsbyname_s (text) IS 'Returns all hostnames for the given hostname as a set. Returns an error string on resolution errors. Returns empty set on NULL input.'; -- returns all hostnames for the given hostname -- returns NULL for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION hostsbyname_n (name text) RETURNS text AS $$ import socket, string if name is None: return None try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = string.join(sorted([hostname] + aliaslist), '\n') except Exception, e: addr = None return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostsbyname_n (text) IS 'Returns all hostnames for the given hostname, NULL on error. Returns NULL on NULL input.'; -- returns all hostnames for the given hostname as a string where the elements are separated by sep -- returns NULL for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION hostsbyname_n (name text, sep text) RETURNS text AS $$ import socket, string if name is None or sep is None: return None try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = string.join(sorted([hostname] + aliaslist), sep) except Exception, e: addr = None return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostsbyname_n (text, text) IS 'Returns all hostnames for the given hostname with custom delimiter, NULL on error. Returns NULL on NULL input.'; -- returns all hostnames for the given hostname as a set -- returns an error string for non-resolvable hostnames or resolution errors CREATE OR REPLACE FUNCTION hostsbyname_ns (name text) RETURNS SETOF text AS $$ import socket, string if name is None: return [] try: (hostname, aliaslist, ipaddrlist) = socket.gethostbyname_ex(name) addr = sorted([hostname] + aliaslist) except Exception, e: addr = () return addr $$ LANGUAGE plpythonu; COMMENT ON FUNCTION hostsbyname_ns (text) IS 'Returns all hostnames for the given hostname as a set, on error or on NULL input returns empty set.'; /* end of file */ digoal’s 大量PostgreSQL文章入口[转载] systemtap 跟踪分析 PostgreSQL2018-10-25T00:00:00+08:002018-10-25T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/10/25/01<h2 id="背景">背景</h2>
<h2 id="为什么要动态跟踪pg">为什么要动态跟踪pg</h2>
<p>1.想清楚数据库内部运行的细节</p>
<p>2.想获取更多地信息</p>
<h2 id="systemtap可以获取到哪些信息">systemtap可以获取到哪些信息</h2>
<ol>
<li>进入或退出某个函数触发事件,执行相应程序</li>
<li>打印函数堆栈</li>
<li>打印函数入参,和出参</li>
</ol>
<h2 id="安装过程">安装过程</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1.安装systemtap
yum install systemtap systemtap-runtime
2.手动安装依赖的内核调试信息包
kernel-debuginfo
kernel-debuginfo-common
kernel-devel
下载地址:
http://debuginfo.centos.org/7/x86_64/
3.支持pg需要的编译参数
--enable-dtrace(静态探测点)
简单例子
--触发事件打印当前事件
stap -v -e 'probe process("/home/postgres/pg10.5/bin/postgres").mark("query-start") {println(gettimeofday_s())}'
--打印该函数的参数
stap -L 'process("/home/postgres/pg10.5/bin/postgres").mark("query-start")'
--打印函数堆栈
stap -v -e 'probe process("/home/postgres/pg10.5/bin/postgres").mark("query-start") {print_usyms(ubacktrace())}'
--enable-debug
</code></pre></div></div>
<h2 id="详细语法细节">详细语法细节</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://spacewander.gitbooks.io/systemtapbeginnersguide_zh/content/1_Introduction.html
</code></pre></div></div>
<h2 id="怎么用systemtap跟踪pg呢">怎么用systemtap跟踪pg呢</h2>
<h3 id="insert表中有索引和没索引差距有多大">insert表中有索引和没索引差距有多大</h3>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--创建表
create table have_index(id int);
create index on have_index(id);
--插入语句
insert into have_index select generate_series(1,1000000);
</code></pre></div></div>
<p>systemtap脚本如下</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>stap -v -DMAXSKIPPED=200000 -e '
global var1
global var2
probe process("/home/postgres/pg10.5/bin/postgres").function("ExecInsert").call {
var1[pid(),0] = gettimeofday_us()
}
probe process("/home/postgres/pg10.5/bin/postgres").function("ExecInsert").return {
var1[pid(),1] = gettimeofday_us()
var2[1] <<< var1[pid(),1] - var1[pid(),0]
}
probe process("/home/postgres/pg10.5/bin/postgres").function("heap_insert").call {
var1[pid(),2] = gettimeofday_us()
}
probe process("/home/postgres/pg10.5/bin/postgres").function("heap_insert").return {
var1[pid(),3] = gettimeofday_us()
var2[2] <<< var1[pid(),3] - var1[pid(),2]
}
probe process("/home/postgres/pg10.5/bin/postgres").function("ExecInsertIndexTuples").call {
var1[pid(),4] = gettimeofday_us()
}
probe process("/home/postgres/pg10.5/bin/postgres").function("ExecInsertIndexTuples").return {
var1[pid(),5] = gettimeofday_us()
var2[3] <<< var1[pid(),5] - var1[pid(),4]
}
probe process("/home/postgres/pg10.5/bin/postgres").function("setLastTid").call {
var1[pid(),6] = gettimeofday_us()
}
probe process("/home/postgres/pg10.5/bin/postgres").function("setLastTid").return {
var1[pid(),7] = gettimeofday_us()
var2[4] <<< var1[pid(),7] - var1[pid(),6]
}
probe process("/home/postgres/pg10.5/bin/postgres").function("list_free").call {
var1[pid(),8] = gettimeofday_us()
}
probe process("/home/postgres/pg10.5/bin/postgres").function("list_free").return {
var1[pid(),9] = gettimeofday_us()
var2[5] <<< var1[pid(),9] - var1[pid(),8]
}
probe timer.s(5) {
if ( @count(var2[1]) > 0 ) {
printf("time : %d \n",gettimeofday_s())
printf("ExecInsert us min: %d, max: %d, avg: %d, sum: %d, count: %d\n", @min(var2[1]), @max(var2[1]), @avg(var2[1]), @sum(var2[1]), @count(var2[1]) )
printf("heap_insert us min: %d, max: %d, avg: %d, sum: %d, count: %d\n", @min(var2[2]), @max(var2[2]), @avg(var2[2]), @sum(var2[2]), @count(var2[2]) )
printf("setLastTid us min: %d, max: %d, avg: %d, sum: %d, count: %d\n", @min(var2[4]), @max(var2[4]), @avg(var2[4]), @sum(var2[4]), @count(var2[4]) )
printf("list_free us min: %d, max: %d, avg: %d, sum: %d, count: %d\n", @min(var2[5]), @max(var2[5]), @avg(var2[5]), @sum(var2[5]), @count(var2[5]) )
if ( @count(var2[3]) > 0 ){
printf("ExecInsertIndexTuples us min: %d, max: %d, avg: %d, sum: %d, count: %d\n", @min(var2[3]), @max(var2[3]), @avg(var2[3]), @sum(var2[3]), @count(var2[3]) )
}
printf(" \n")
}
}'
</code></pre></div></div>
<h3 id="打印函数堆栈">打印函数堆栈</h3>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>stap -v -DMAXSKIPPED=200000 -e '
probe process("/home/postgres/pg10.5/bin/postgres").function("heap_insert").call {
print_usyms(ubacktrace())
}
'
</code></pre></div></div>
<h3 id="insert-values和copy的差距有多大差别在哪些方面">insert values(),(),()和copy的差距有多大,差别在哪些方面</h3>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>create table test_copy (id int);
--insert插入
insert into test_copy values(1),(2),(3),(4),(5);
--copy导入
copy test_copy from '/home/postgres/test_copy.csv';
</code></pre></div></div>
<p>systemstap脚本</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>global var1
global var2
probe process("/home/postgres/pg10.5/bin/postgres").function("ExecInsert").call {
var1[pid(),0] = gettimeofday_us()
}
probe process("/home/postgres/pg10.5/bin/postgres").function("ExecInsert").return {
var1[pid(),1] = gettimeofday_us()
var2[1] <<< var1[pid(),1] - var1[pid(),0]
}
probe process("/home/postgres/pg10.5/bin/postgres").function("heap_insert").call {
var1[pid(),2] = gettimeofday_us()
}
probe process("/home/postgres/pg10.5/bin/postgres").function("heap_insert").return {
var1[pid(),3] = gettimeofday_us()
var2[2] <<< var1[pid(),3] - var1[pid(),2]
}
probe process("/home/postgres/pg10.5/bin/postgres").function("heap_multi_insert").call {
var1[pid(),4] = gettimeofday_us()
}
probe process("/home/postgres/pg10.5/bin/postgres").function("heap_multi_insert").return {
var1[pid(),5] = gettimeofday_us()
var2[3] <<< var1[pid(),5] - var1[pid(),4]
}
probe process("/home/postgres/pg10.5/bin/postgres").function("CopyFromInsertBatch").call {
var1[pid(),6] = gettimeofday_us()
}
probe process("/home/postgres/pg10.5/bin/postgres").function("CopyFromInsertBatch").return {
var1[pid(),7] = gettimeofday_us()
var2[4] <<< var1[pid(),7] - var1[pid(),6]
}
probe timer.s(5) {
if ( @count(var2[1]) > 0 ) {
printf("time : %d \n",gettimeofday_s())
printf("ExecInsert us min: %d, max: %d, avg: %d, sum: %d, count: %d\n", @min(var2[1]), @max(var2[1]), @avg(var2[1]), @sum(var2[1]), @count(var2[1]) )
printf("heap_insert us min: %d, max: %d, avg: %d, sum: %d, count: %d\n", @min(var2[2]), @max(var2[2]), @avg(var2[2]), @sum(var2[2]), @count(var2[2]) )
if (@count(var2[2]) >0 ){
printf("heap_multi_insert us min: %d, max: %d, avg: %d, sum: %d, count: %d\n", @min(var2[3]), @max(var2[3]), @avg(var2[3]), @sum(var2[3]), @count(var2[3]) )
printf("CopyFromInsertBatch us min: %d, max: %d, avg: %d, sum: %d, count: %d\n", @min(var2[4]), @max(var2[4]), @avg(var2[4]), @sum(var2[4]), @count(var2[4]) )
}
printf(" \n")
}
}
</code></pre></div></div>
<h3 id="gdb">gdb</h3>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://github.com/hellogcc/100-gdb-tips/blob/master/src/index.md
</code></pre></div></div>
<h3 id="pg内部日志打出">pg内部日志打出</h3>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elog(WARNING," ")
make && make install
pg_ctl restart
</code></pre></div></div>
<h2 id="参考">参考</h2>
<p>https://spacewander.gitbooks.io/systemtapbeginnersguide_zh/content/index.html</p>
<p>https://github.com/hellogcc/100-gdb-tips</p>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>王健背景 为什么要动态跟踪pgPostgreSQL 分区表、继承表 记录去重方法2018-10-22T00:00:00+08:002018-10-22T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/10/22/01<h2 id="背景">背景</h2>
<p>当使用数据库分区或继承功能,在PK层面上出现分区与分区,或分区与主表出现了重复的键值时,可以通过tableoid进行甄别,同时通过ONLY TABLE的操作方法进行删除。</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>select tableoid::regclass
delete|select|update|truncate only
</code></pre></div></div>
<h2 id="例子">例子</h2>
<p>创建测试表、继承分区,PK约束在独立的分区或主表上</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# create table p (id int primary key, info text, crt_time timestamp);
CREATE TABLE
postgres=# create table p0 (like p including all) inherits(p);
NOTICE: merging column "id" with inherited definition
NOTICE: merging column "info" with inherited definition
NOTICE: merging column "crt_time" with inherited definition
CREATE TABLE
postgres=# create table p1 (like p including all) inherits(p);
NOTICE: merging column "id" with inherited definition
NOTICE: merging column "info" with inherited definition
NOTICE: merging column "crt_time" with inherited definition
CREATE TABLE
postgres=# create table p2 (like p including all) inherits(p);
NOTICE: merging column "id" with inherited definition
NOTICE: merging column "info" with inherited definition
NOTICE: merging column "crt_time" with inherited definition
CREATE TABLE
postgres=# create table p3 (like p including all) inherits(p);
NOTICE: merging column "id" with inherited definition
NOTICE: merging column "info" with inherited definition
NOTICE: merging column "crt_time" with inherited definition
CREATE TABLE
</code></pre></div></div>
<p>往不同的分区写入PK重复的数据</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# insert into p values (1,'test',now());
INSERT 0 1
postgres=# insert into p0 values (1,'test',now());
INSERT 0 1
postgres=# insert into p1 values (1,'test',now());
INSERT 0 1
postgres=# insert into p2 values (1,'test',now());
INSERT 0 1
</code></pre></div></div>
<p>查询,你可能会不知道记录属于哪个表</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# select * from p;
id | info | crt_time
----+------+----------------------------
1 | test | 2018-10-22 09:26:55.456769
1 | test | 2018-10-22 09:26:58.441338
1 | test | 2018-10-22 09:27:01.149731
1 | test | 2018-10-22 09:27:03.389089
(4 rows)
</code></pre></div></div>
<p>通过tableoid进行甄别</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# select tableoid::regclass,* from p;
tableoid | id | info | crt_time
----------+----+------+----------------------------
p | 1 | test | 2018-10-22 09:26:55.456769
p0 | 1 | test | 2018-10-22 09:26:58.441338
p1 | 1 | test | 2018-10-22 09:27:01.149731
p2 | 1 | test | 2018-10-22 09:27:03.389089
(4 rows)
</code></pre></div></div>
<p>直接指定PK删除主表时,会将所有记录删除。</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# delete from p where id=1;
DELETE 4
postgres=# select tableoid::regclass,* from p;
tableoid | id | info | crt_time
----------+----+------+----------
(0 rows)
</code></pre></div></div>
<h2 id="deleteselectupdatetruncate-only-清除指定分区的数据">delete|select|update|truncate only 清除指定分区的数据</h2>
<p>https://www.postgresql.org/docs/11/static/sql-delete.html</p>
<p>通过only关键字,可以指定只操作当前表,不包括继承或子继承的表.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# insert into p values (1,'test',now());
INSERT 0 1
postgres=# insert into p0 values (1,'test',now());
INSERT 0 1
postgres=# insert into p1 values (1,'test',now());
INSERT 0 1
postgres=# insert into p2 values (1,'test',now());
INSERT 0 1
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# delete from only p where id=1;
DELETE 1
postgres=# select tableoid::regclass,* from p;
tableoid | id | info | crt_time
----------+----+------+----------------------------
p0 | 1 | test | 2018-10-22 09:27:47.510151
p1 | 1 | test | 2018-10-22 09:27:49.366293
p2 | 1 | test | 2018-10-22 09:27:51.255673
(3 rows)
postgres=# delete from only p2 where id=1;
DELETE 1
postgres=# select tableoid::regclass,* from p;
tableoid | id | info | crt_time
----------+----+------+----------------------------
p0 | 1 | test | 2018-10-22 09:27:47.510151
p1 | 1 | test | 2018-10-22 09:27:49.366293
(2 rows)
</code></pre></div></div>
<p>如果是单张表内的数据去重,请参考末尾连接。</p>
<h2 id="参考">参考</h2>
<p><a href="../201706/20170602_01.md">《PostgreSQL 数据去重方法大全》</a></p>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>Digoal.zhoudigoal@126.com背景 当使用数据库分区或继承功能,在PK层面上出现分区与分区,或分区与主表出现了重复的键值时,可以通过tableoid进行甄别,同时通过ONLY TABLE的操作方法进行删除。 select tableoid::regclass delete|select|update|truncate only 例子 创建测试表、继承分区,PK约束在独立的分区或主表上 postgres=# create table p (id int primary key, info text, crt_time timestamp); CREATE TABLE postgres=# create table p0 (like p including all) inherits(p); NOTICE: merging column "id" with inherited definition NOTICE: merging column "info" with inherited definition NOTICE: merging column "crt_time" with inherited definition CREATE TABLE postgres=# create table p1 (like p including all) inherits(p); NOTICE: merging column "id" with inherited definition NOTICE: merging column "info" with inherited definition NOTICE: merging column "crt_time" with inherited definition CREATE TABLE postgres=# create table p2 (like p including all) inherits(p); NOTICE: merging column "id" with inherited definition NOTICE: merging column "info" with inherited definition NOTICE: merging column "crt_time" with inherited definition CREATE TABLE postgres=# create table p3 (like p including all) inherits(p); NOTICE: merging column "id" with inherited definition NOTICE: merging column "info" with inherited definition NOTICE: merging column "crt_time" with inherited definition CREATE TABLE 往不同的分区写入PK重复的数据 postgres=# insert into p values (1,'test',now()); INSERT 0 1 postgres=# insert into p0 values (1,'test',now()); INSERT 0 1 postgres=# insert into p1 values (1,'test',now()); INSERT 0 1 postgres=# insert into p2 values (1,'test',now()); INSERT 0 1 查询,你可能会不知道记录属于哪个表 postgres=# select * from p; id | info | crt_time ----+------+---------------------------- 1 | test | 2018-10-22 09:26:55.456769 1 | test | 2018-10-22 09:26:58.441338 1 | test | 2018-10-22 09:27:01.149731 1 | test | 2018-10-22 09:27:03.389089 (4 rows) 通过tableoid进行甄别 postgres=# select tableoid::regclass,* from p; tableoid | id | info | crt_time ----------+----+------+---------------------------- p | 1 | test | 2018-10-22 09:26:55.456769 p0 | 1 | test | 2018-10-22 09:26:58.441338 p1 | 1 | test | 2018-10-22 09:27:01.149731 p2 | 1 | test | 2018-10-22 09:27:03.389089 (4 rows) 直接指定PK删除主表时,会将所有记录删除。 postgres=# delete from p where id=1; DELETE 4 postgres=# select tableoid::regclass,* from p; tableoid | id | info | crt_time ----------+----+------+---------- (0 rows) delete|select|update|truncate only 清除指定分区的数据 https://www.postgresql.org/docs/11/static/sql-delete.html 通过only关键字,可以指定只操作当前表,不包括继承或子继承的表. postgres=# insert into p values (1,'test',now()); INSERT 0 1 postgres=# insert into p0 values (1,'test',now()); INSERT 0 1 postgres=# insert into p1 values (1,'test',now()); INSERT 0 1 postgres=# insert into p2 values (1,'test',now()); INSERT 0 1 postgres=# delete from only p where id=1; DELETE 1 postgres=# select tableoid::regclass,* from p; tableoid | id | info | crt_time ----------+----+------+---------------------------- p0 | 1 | test | 2018-10-22 09:27:47.510151 p1 | 1 | test | 2018-10-22 09:27:49.366293 p2 | 1 | test | 2018-10-22 09:27:51.255673 (3 rows) postgres=# delete from only p2 where id=1; DELETE 1 postgres=# select tableoid::regclass,* from p; tableoid | id | info | crt_time ----------+----+------+---------------------------- p0 | 1 | test | 2018-10-22 09:27:47.510151 p1 | 1 | test | 2018-10-22 09:27:49.366293 (2 rows) 如果是单张表内的数据去重,请参考末尾连接。 参考 《PostgreSQL 数据去重方法大全》 digoal’s 大量PostgreSQL文章入口PostgreSQL 11 培训与认证设计 - 基础 , 管理 , 应用 , 内核 , 阿里云RDS PPAS(Oracle 兼容性) , 认证2018-10-21T00:00:00+08:002018-10-21T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/10/21/01<h2 id="背景">背景</h2>
<p>PostgreSQL 11 于2018-10-18发布。</p>
<p>https://www.postgresql.org/docs/11/static/release-11.html</p>
<p>基于PG 11与阿里云RDS PPAS(兼容Oracle),设计一套学习与认证课程。</p>
<h1 id="大纲">大纲</h1>
<h1 id="一基础部分">一、基础部分</h1>
<p>1、PG 11 新特性介绍</p>
<p>https://www.postgresql.org/docs/11/static/release-11.html</p>
<p>2、PG 11 环境部署</p>
<p>3、PG 11 配置文件讲解</p>
<p>4、GUI</p>
<p>5、CLI</p>
<p>6、商业、开源开发框架</p>
<p>7、SDK</p>
<p>8、PG 11 SQL语法</p>
<p>9、数据类型</p>
<p>10、内置函数</p>
<p>11、索引</p>
<p>12、函数开发</p>
<p>13、存储过程开发</p>
<p>14、强制并行</p>
<p>15、JIT</p>
<h1 id="二管理部分">二、管理部分</h1>
<p>1、物理架构</p>
<p>2、权限体系</p>
<p>3、日常维护</p>
<p>4、监控</p>
<p>5、诊断</p>
<p>6、优化</p>
<p>7、性能压测方法</p>
<p>8、安全加固</p>
<p>9、备份</p>
<p>10、恢复</p>
<p>11、容灾</p>
<p>12、HA</p>
<p>13、读写分离</p>
<p>14、sharding</p>
<p>15、单元化</p>
<p>16、小版本升级</p>
<p>17、跨版本升级</p>
<p>18、外部数据源</p>
<p>19、SQL审计</p>
<p>20、扩展接口</p>
<p>21、内置插件</p>
<p>22、外部插件</p>
<h1 id="三应用部分">三、应用部分</h1>
<p>1、全文检索</p>
<p>2、GIS</p>
<p>3、采样</p>
<p>4、加密</p>
<p>5、… 等</p>
<p>6、进阶 digoal’s github</p>
<p>https://github.com/digoal/blog/blob/master/README.md</p>
<h1 id="四oracle兼容部分">四、Oracle兼容部分</h1>
<p>1、PG orafce 兼容包</p>
<p>2、阿里云 RDS PPAS (oracle兼容性产品)介绍</p>
<p>3、PPAS 高级功能</p>
<p>4、PPAS ORACLE 兼容参数</p>
<p>5、PPAS PG 兼容参数</p>
<p>6、阿里云 ADAM Oracle迁移评估、迁移</p>
<p>7、阿里云 RDS PPAS API</p>
<h1 id="五内核部分">五、内核部分</h1>
<p>http://www.interdb.jp/pg/</p>
<h1 id="六认证部分">六、认证部分</h1>
<h2 id="管理员认证">管理员认证</h2>
<p>考察: 面</p>
<p>侧重管理</p>
<h2 id="专家认证">专家认证</h2>
<p>前置条件,通过管理员认证</p>
<p>考察: 深度</p>
<p>侧重应用、架构</p>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>Digoal.zhoudigoal@126.com背景 PostgreSQL 11 于2018-10-18发布。 https://www.postgresql.org/docs/11/static/release-11.html 基于PG 11与阿里云RDS PPAS(兼容Oracle),设计一套学习与认证课程。 大纲 一、基础部分 1、PG 11 新特性介绍 https://www.postgresql.org/docs/11/static/release-11.html 2、PG 11 环境部署 3、PG 11 配置文件讲解 4、GUI 5、CLI 6、商业、开源开发框架 7、SDK 8、PG 11 SQL语法 9、数据类型 10、内置函数 11、索引 12、函数开发 13、存储过程开发 14、强制并行 15、JIT 二、管理部分 1、物理架构 2、权限体系 3、日常维护 4、监控 5、诊断 6、优化 7、性能压测方法 8、安全加固 9、备份 10、恢复 11、容灾 12、HA 13、读写分离 14、sharding 15、单元化 16、小版本升级 17、跨版本升级 18、外部数据源 19、SQL审计 20、扩展接口 21、内置插件 22、外部插件 三、应用部分 1、全文检索 2、GIS 3、采样 4、加密 5、… 等 6、进阶 digoal’s github https://github.com/digoal/blog/blob/master/README.md 四、Oracle兼容部分 1、PG orafce 兼容包 2、阿里云 RDS PPAS (oracle兼容性产品)介绍 3、PPAS 高级功能 4、PPAS ORACLE 兼容参数 5、PPAS PG 兼容参数 6、阿里云 ADAM Oracle迁移评估、迁移 7、阿里云 RDS PPAS API 五、内核部分 http://www.interdb.jp/pg/ 六、认证部分 管理员认证 考察: 面 侧重管理 专家认证 前置条件,通过管理员认证 考察: 深度 侧重应用、架构 digoal’s 大量PostgreSQL文章入口Locking issue with concurrent DELETE / INSERT in PostgreSQL - 解法 advisory lock2018-10-18T00:00:00+08:002018-10-18T00:00:00+08:00https://billtian.github.io/digoal.blog/2018/10/18/04<h2 id="背景">背景</h2>
<p>两个会话,同时对同一个ID值的记录执行删除后插入,有一个会话会出现删除0,插入失败的现象。</p>
<h2 id="现象">现象</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CREATE TABLE test (id INT PRIMARY KEY);
INSERT INTO TEST VALUES (1);
INSERT INTO TEST VALUES (2);
</code></pre></div></div>
<p>执行如下几条SQL</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>begin;
1:DELETE FROM test WHERE id=1;
2:INSERT INTO test VALUES (1);
3:COMMIT;
</code></pre></div></div>
<p>现象</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>S1-1 runs (1 row deleted)
S2-1 runs (but is blocked since S1 has a write lock)
S1-2 runs (1 row inserted)
S1-3 runs, releasing the write lock
S2-1 runs, now that it can get the lock. But reports 0 rows deleted. HUH???
S2-2 runs, reports a unique key constraint violation
</code></pre></div></div>
<p>用户期望</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>S1-1 runs (1 row deleted)
S2-1 runs (but is blocked since S1 has a write lock)
S1-2 runs (1 row inserted)
S1-3 runs, releasing the write lock
S2-1 runs, now that it can get the lock. But reports 0 rows deleted. HUH??? 用户期望这里的S1新插入的记录可以被删除
S2-2 runs, reports a unique key constraint violation 不报错
</code></pre></div></div>
<h2 id="原因分析">原因分析</h2>
<p>行级锁。</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>S1-1 runs (1 row deleted)
S2-1 runs (but is blocked since S1 has a write lock) 锁冲突,因为这条记录(ROWID)已经被S1锁住了
S1-2 runs (1 row inserted)
S1-3 runs, releasing the write lock
S2-1 runs, now that it can get the lock. But reports 0 rows deleted. HUH??? S1提交,S2请求的这个ROWID已经被删除。所以DELETE 0
S2-2 runs, reports a unique key constraint violation 报错,因为S1已经插入了同一个ID
</code></pre></div></div>
<h2 id="deferable-约束无法解决这个问题">deferable 约束无法解决这个问题</h2>
<p>https://www.postgresql.org/docs/11/static/sql-createtable.html</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
</code></pre></div></div>
<p>https://www.postgresql.org/docs/11/static/sql-set-constraints.html</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SET CONSTRAINTS { ALL | name [, ...] } { DEFERRED | IMMEDIATE }
</code></pre></div></div>
<p>例子</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# create table t1(id int primary key deferrable) ;
CREATE TABLE
postgres=# insert into t1 values (1),(2);
INSERT 0 2
postgres=# begin;
BEGIN
postgres=# set constraints all deferred ;
SET CONSTRAINTS
postgres=# delete from t1 where id=1;
DELETE 1
postgres=# insert into t1 values (1);
INSERT 0 1
postgres=# end;
COMMIT
postgres=# begin;
BEGIN
postgres=# set constraints ALL deferred ;
SET CONSTRAINTS
postgres=# delete from t1 where id=1;
DELETE 0
postgres=# insert into t1 values (1); -- 不使用deferred,这里就直接报错
INSERT 0 1
使用deferred,现象不一样的地方,在事务结束时判断约束。
postgres=# end;
ERROR: duplicate key value violates unique constraint "t1_pkey"
DETAIL: Key (id)=(1) already exists.
</code></pre></div></div>
<h2 id="推荐解决方案">推荐解决方案</h2>
<p>adlock</p>
<p><a href="../201707/20170720_01.md">《advisory lock 实现高并发非堵塞式 业务锁》</a></p>
<p><a href="../201705/20170507_02.md">《PostgreSQL 使用advisory lock实现行级读写堵塞》</a></p>
<p><a href="../201610/20161020_02.md">《PostgreSQL 无缝自增ID的实现 - by advisory lock》</a></p>
<p><a href="../201610/20161018_01.md">《PostgreSQL 使用advisory lock或skip locked消除行锁冲突, 提高几十倍并发更新效率》</a></p>
<p><a href="../201509/20150914_01.md">《PostgreSQL 秒杀场景优化》</a></p>
<p>例子</p>
<p>S1,对ID=1进行操作</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# begin;
BEGIN
postgres=# select pg_try_advisory_xact_lock(1);
pg_try_advisory_xact_lock
----------------------
t
(1 row)
业务上判断,返回TRUE后,继续下面的请求
postgres=# delete from t1 where id=1;
DELETE 1
postgres=# insert into t1 values (1);
INSERT 0 1
postgres=# end;
COMMIT
事务结束自动释放pg_try_advisory_xact_lock
</code></pre></div></div>
<p>S2,对ID=1进行操作</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# begin;
BEGIN
postgres=# select pg_try_advisory_xact_lock(1);
pg_try_advisory_xact_lock
----------------------
f
(1 row)
返回FALSE,业务上不断重试,pg_try_advisory_xact_lock(1)
......
直到S1提交,返回TRUE后,继续下面的SQL请求
postgres=# delete from t1 where id=1;
DELETE 1
postgres=# insert into t1 values (1);
INSERT 0 1
postgres=# end;
COMMIT
</code></pre></div></div>
<h3 id="注意事项">注意事项</h3>
<p>adlock的id是库级冲突,例如lock(1),在同一个数据库再lock(1)就会冲突。</p>
<p>因此,如果在同一个库里面,期望对不同的表里面的数据使用同样的adlock手段,建议不同的表错开ID段,或者使用全局ID。</p>
<h2 id="小结">小结</h2>
<p>adlock为轻量级锁,在本文提到的业务场景中,逻辑上把两个事务中的SQL做成了串行执行。达到的效果满足业务上的需求。</p>
<h2 id="参考">参考</h2>
<p>https://dba.stackexchange.com/questions/27688/locking-issue-with-concurrent-delete-insert-in-postgresql</p>
<p>https://www.postgresql.org/docs/11/static/sql-createtable.html</p>
<p>https://www.postgresql.org/docs/11/static/sql-set-constraints.html</p>
<p><a rel="nofollow" href="http://info.flagcounter.com/h9V1"><img src="http://s03.flagcounter.com/count/h9V1/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_12/viewers_0/labels_0/pageviews_0/flags_0/" alt="Flag Counter" border="0" /></a></p>
<h2 id="digoals-大量postgresql文章入口"><a href="https://github.com/digoal/blog/blob/master/README.md" title="22709685feb7cab07d30f30387f0a9ae">digoal’s 大量PostgreSQL文章入口</a></h2>Digoal.zhoudigoal@126.com背景 两个会话,同时对同一个ID值的记录执行删除后插入,有一个会话会出现删除0,插入失败的现象。 现象 CREATE TABLE test (id INT PRIMARY KEY); INSERT INTO TEST VALUES (1); INSERT INTO TEST VALUES (2); 执行如下几条SQL begin; 1:DELETE FROM test WHERE id=1; 2:INSERT INTO test VALUES (1); 3:COMMIT; 现象 S1-1 runs (1 row deleted) S2-1 runs (but is blocked since S1 has a write lock) S1-2 runs (1 row inserted) S1-3 runs, releasing the write lock S2-1 runs, now that it can get the lock. But reports 0 rows deleted. HUH??? S2-2 runs, reports a unique key constraint violation 用户期望 S1-1 runs (1 row deleted) S2-1 runs (but is blocked since S1 has a write lock) S1-2 runs (1 row inserted) S1-3 runs, releasing the write lock S2-1 runs, now that it can get the lock. But reports 0 rows deleted. HUH??? 用户期望这里的S1新插入的记录可以被删除 S2-2 runs, reports a unique key constraint violation 不报错 原因分析 行级锁。 S1-1 runs (1 row deleted) S2-1 runs (but is blocked since S1 has a write lock) 锁冲突,因为这条记录(ROWID)已经被S1锁住了 S1-2 runs (1 row inserted) S1-3 runs, releasing the write lock S2-1 runs, now that it can get the lock. But reports 0 rows deleted. HUH??? S1提交,S2请求的这个ROWID已经被删除。所以DELETE 0 S2-2 runs, reports a unique key constraint violation 报错,因为S1已经插入了同一个ID deferable 约束无法解决这个问题 https://www.postgresql.org/docs/11/static/sql-createtable.html [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] https://www.postgresql.org/docs/11/static/sql-set-constraints.html SET CONSTRAINTS { ALL | name [, ...] } { DEFERRED | IMMEDIATE } 例子 postgres=# create table t1(id int primary key deferrable) ; CREATE TABLE postgres=# insert into t1 values (1),(2); INSERT 0 2 postgres=# begin; BEGIN postgres=# set constraints all deferred ; SET CONSTRAINTS postgres=# delete from t1 where id=1; DELETE 1 postgres=# insert into t1 values (1); INSERT 0 1 postgres=# end; COMMIT postgres=# begin; BEGIN postgres=# set constraints ALL deferred ; SET CONSTRAINTS postgres=# delete from t1 where id=1; DELETE 0 postgres=# insert into t1 values (1); -- 不使用deferred,这里就直接报错 INSERT 0 1 使用deferred,现象不一样的地方,在事务结束时判断约束。 postgres=# end; ERROR: duplicate key value violates unique constraint "t1_pkey" DETAIL: Key (id)=(1) already exists. 推荐解决方案 adlock 《advisory lock 实现高并发非堵塞式 业务锁》 《PostgreSQL 使用advisory lock实现行级读写堵塞》 《PostgreSQL 无缝自增ID的实现 - by advisory lock》 《PostgreSQL 使用advisory lock或skip locked消除行锁冲突, 提高几十倍并发更新效率》 《PostgreSQL 秒杀场景优化》 例子 S1,对ID=1进行操作 postgres=# begin; BEGIN postgres=# select pg_try_advisory_xact_lock(1); pg_try_advisory_xact_lock ---------------------- t (1 row) 业务上判断,返回TRUE后,继续下面的请求 postgres=# delete from t1 where id=1; DELETE 1 postgres=# insert into t1 values (1); INSERT 0 1 postgres=# end; COMMIT 事务结束自动释放pg_try_advisory_xact_lock S2,对ID=1进行操作 postgres=# begin; BEGIN postgres=# select pg_try_advisory_xact_lock(1); pg_try_advisory_xact_lock ---------------------- f (1 row) 返回FALSE,业务上不断重试,pg_try_advisory_xact_lock(1) ...... 直到S1提交,返回TRUE后,继续下面的SQL请求 postgres=# delete from t1 where id=1; DELETE 1 postgres=# insert into t1 values (1); INSERT 0 1 postgres=# end; COMMIT 注意事项 adlock的id是库级冲突,例如lock(1),在同一个数据库再lock(1)就会冲突。 因此,如果在同一个库里面,期望对不同的表里面的数据使用同样的adlock手段,建议不同的表错开ID段,或者使用全局ID。 小结 adlock为轻量级锁,在本文提到的业务场景中,逻辑上把两个事务中的SQL做成了串行执行。达到的效果满足业务上的需求。 参考 https://dba.stackexchange.com/questions/27688/locking-issue-with-concurrent-delete-insert-in-postgresql https://www.postgresql.org/docs/11/static/sql-createtable.html https://www.postgresql.org/docs/11/static/sql-set-constraints.html digoal’s 大量PostgreSQL文章入口