Digoal.zhou

Digoal.zhou

公益是一辈子的事, I'm digoal, Just do it.

单网卡绑多个IP时如何指定IP出口(VIP出口绑定) - use ip modify route table use vip as src trans in multi-IP bonded env

6 minute read

背景

在某些场景中, 当一个服务器的一块网卡上面配置了多个IP时, 例如虚拟IP, 可能想指定虚拟IP地址作为出口地址.

例如对这个虚拟IP有鉴权要求的场景. 如PostgreSQL的pg_hba.conf.

如在集中的流复制standby场景, 当集中的主机DOWN掉的话, 希望把虚拟IP切走, 同时生产机也只允许这个虚拟IP来访问的情况.

那么需要改写集中流复制的主机的路由表, 让其出口为虚拟IP.

默认情况下当有多个IP时, 路由是primary优先的.

如下, 默认的出口是 inet 172.16.3.111/24 brd 172.16.3.255 scope global eth0 而不是eth0:1

[root@db-172-16-3-111 network-scripts]# ip addr show  
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN   
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  
    inet 127.0.0.1/8 scope host lo  
    inet6 ::1/128 scope host   
       valid_lft forever preferred_lft forever  
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000  
    link/ether b8:ca:3a:6d:fe:b0 brd ff:ff:ff:ff:ff:ff  
    inet 172.16.3.111/24 brd 172.16.3.255 scope global eth0  
    inet 172.16.3.1/24 brd 172.16.1.255 scope global secondary eth0:1

为了达到指定出口IP的目的, 可以通过修改路由表来实现.

[root@db-172-16-3-111 ~]# which ip  
/sbin/ip

在没有创建eth0:1的情况下的默认路由如下 :

[root@db-172-16-3-111 ~]# ip route  
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111   
169.254.0.0/16 dev eth0  scope link  metric 1002   
default via 172.16.3.1 dev eth0

新增一个IP地址

[root@db-172-16-3-111 ~]# cd /etc/sysconfig/network-scripts/  
  
[root@db-172-16-3-111 network-scripts]# cp ifcfg-eth0 ifcfg-eth0:1  
[root@db-172-16-3-111 network-scripts]# vi ifcfg-eth0:1  
DEVICE="eth0:1"  
BOOTPROTO="static"  
IPADDR="172.16.3.105"  
NETMASK="255.255.255.0"  
HWADDR="B8:CA:3A:6D:FE:B0"  
IPV6INIT="no"  
MTU="1500"  
NM_CONTROLLED="no"  
ONBOOT="no"  
TYPE="Ethernet"  
UUID="37fb27ad-4293-43ba-a3b7-8e94fa563384"  
  
[root@db-172-16-3-111 network-scripts]# ifup eth0:1  
Determining if ip address 172.16.3.105 is already in use for device eth0...  
[root@db-172-16-3-111 network-scripts]# ifconfig  
eth0      Link encap:Ethernet  HWaddr B8:CA:3A:6D:FE:B0    
          inet addr:172.16.3.111  Bcast:172.16.3.255  Mask:255.255.255.0  
          inet6 addr: fe80::baca:3aff:fe6d:feb0/64 Scope:Link  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1  
          RX packets:536391 errors:0 dropped:0 overruns:0 frame:0  
          TX packets:88849 errors:0 dropped:0 overruns:0 carrier:0  
          collisions:0 txqueuelen:1000   
          RX bytes:718980563 (685.6 MiB)  TX bytes:4883966 (4.6 MiB)  
          Memory:dcb00000-dcc00000   
  
eth0:1    Link encap:Ethernet  HWaddr B8:CA:3A:6D:FE:B0    
          inet addr:172.16.3.105  Bcast:172.16.3.255  Mask:255.255.255.0  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1  
          Memory:dcb00000-dcc00000

然后添加路由, 目的是增加一条metric=1的路由, 使得去整个广播域和网关走新增的这个地址出去.

[root@db-172-16-3-111 network-scripts]# ip route add default via 172.16.3.1 dev eth0 src 172.16.3.105 metric 1  
[root@db-172-16-3-111 network-scripts]# ip route add 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1

然后删除原路由条目

[root@db-172-16-3-111 network-scripts]# ip route del default via 172.16.3.1 dev eth0  
[root@db-172-16-3-111 network-scripts]# ip route del 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111

注意, 最好增加2个条目, 如果虚拟IP切走的话, 还能正常走eth0出去的路由. 只是metric改大一点.

[root@db-172-16-3-111 network-scripts]# ip route add default via 172.16.3.1 dev eth0 src 172.16.3.111 metric 100  
[root@db-172-16-3-111 network-scripts]# ip route add 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 metric 100

添加完后的路由表

[root@db-172-16-3-111 network-scripts]# ip route  
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1   
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111  metric 100   
169.254.0.0/16 dev eth0  scope link  metric 1002   
default via 172.16.3.1 dev eth0  src 172.16.3.105  metric 1   
default via 172.16.3.1 dev eth0  metric 100

看看路由有没有生效, 已经使用了我们配置的metric=1的路由条目.

[root@db-172-16-3-111 network-scripts]# ip route get 192.168.1.1  
192.168.1.1 via 172.16.3.1 dev eth0  src 172.16.3.105   
    cache  mtu 1500 advmss 1460 hoplimit 64  
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.1  
172.16.3.1 dev eth0  src 172.16.3.105   
    cache  mtu 1500 advmss 1460 hoplimit 64

关闭这个虚拟IP, 走metric=100的路由条目

[root@db-172-16-3-111 network-scripts]# ifdown eth0:1  
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.1  
172.16.3.1 dev eth0  src 172.16.3.111   
    cache  mtu 1500 advmss 1460 hoplimit 64  
[root@db-172-16-3-111 network-scripts]# ip route get 192.168.1.1  
192.168.1.1 via 172.16.3.1 dev eth0  src 172.16.3.111   
    cache  mtu 1500 advmss 1460 hoplimit 64

关闭eth0:1后, 路由条目自动消失.

[root@db-172-16-3-111 network-scripts]# ip route  
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111  metric 100 	  
169.254.0.0/16 dev eth0  scope link  metric 1002   
default via 172.16.3.1 dev eth0  metric 100   
  
[root@db-172-16-3-111 network-scripts]# ifup eth0:1  
Determining if ip address 172.16.3.105 is already in use for device eth0...  
  
[root@db-172-16-3-111 network-scripts]# ip route  
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111  metric 100   
169.254.0.0/16 dev eth0  scope link  metric 1002   
default via 172.16.3.1 dev eth0   
default via 172.16.3.1 dev eth0  metric 100

如果虚拟IP切回来, 又要重新写路由, 比较土的方法是重启network服务, 然后执行前面一样的步骤添加路由.

[root@db-172-16-3-111 network-scripts]# service network restart  
Shutting down interface eth0:  [  OK  ]  
Shutting down loopback interface:  [  OK  ]  
Bringing up loopback interface:  [  OK  ]  
Bringing up interface eth0:  Determining if ip address 172.16.3.111 is already in use for device eth0...  
[  OK  ]  
  
[root@db-172-16-3-111 network-scripts]# ip route  
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111   
169.254.0.0/16 dev eth0  scope link  metric 1002   
default via 172.16.3.1 dev eth0   
  
[root@db-172-16-3-111 network-scripts]# ip route add default via 172.16.3.1 dev eth0 src 172.16.3.105 metric 1  
[root@db-172-16-3-111 network-scripts]# ip route add 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1  
  
[root@db-172-16-3-111 network-scripts]# ip route del default via 172.16.3.1 dev eth0  
[root@db-172-16-3-111 network-scripts]# ip route del 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111  
  
[root@db-172-16-3-111 network-scripts]# ip route add default via 172.16.3.1 dev eth0 src 172.16.3.111 metric 100  
[root@db-172-16-3-111 network-scripts]# ip route add 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 metric 100

比较方便的方法是写配置文件, 或者是使用iptables .

配置文件比较多, 参考

/usr/share/doc/initscripts-9.03.40/sysconfig.txt

这里用到ifup-post调用的ifup-routes

/etc/sysconfig/network-scripts/route-

  Contains lines that specify additional routes that should be added when the  
  associated interface is brought up.  
  
  The files are processed by the ifup-routes script and uses the /sbin/ipcalc  
  utility for all network masks and numbers. Routes are specified using the  
  syntax:  
  
    ADDRESSn=<network>  
    NETMASKn=<network/prefix mask>  
    GATEWAYn=<next-hop router/gateway IP address>  
  
  The "n" is expected to be consecutive positive integers starting from 0.  
  For example:  
  
    ADDRESS0=192.168.2.0  
    NETMASK0=255.255.255.0  
    GATEWAY0=192.168.1.1  
  
  adds a network route to the 192.168.2.0 network via the gateway at  
  192.168.1.1. Since you must already have a route to the network of the  
  gateway, there is no need to specify a device.  
  
  Note: The ifup-routes script also supports an older syntax designed to be  
  used directly as an argument to "/sbin/ip route add".  
  If no "ADDRESSn" lines are found the following will still  
  work:  
    
  192.168.2.0/24 dev ppp0  
    
  adds a network route to the 192.168.2.0 network through ppp0.

所以需要给绑定了多个IP的网络设备写一个自定义路由配置文件

/etc/sysconfig/network-scripts/route-eth0

内容如下 :

172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 metric 100  
default via 172.16.3.1 dev eth0 src 172.16.3.111 metric 100  
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1  
default via 172.16.3.1 dev eth0 src 172.16.3.105 metric 1

这个配置文件是在ifup-post后调用ip route add来添加路由.

但是仅此还有问题, 因为默认的话还是会添加2条路由, 网关和本网段的路由.

172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111   
default via 172.16.3.1 dev eth0   
[root@db-172-16-3-111 ~]# ip route  
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111   
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1   
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111  metric 100   
169.254.0.0/16 dev eth0  scope link  metric 1002   
default via 172.16.3.1 dev eth0   
default via 172.16.3.1 dev eth0  src 172.16.3.105  metric 1   
default via 172.16.3.1 dev eth0  src 172.16.3.111  metric 100

为了去除这两条路由, 需要调整一下配置文件.

最终的配置

把网关去掉

vi /etc/sysconfig/network  
and  
vi /etc/sysconfig/network-scripts/ifcfg-eth0  
vi /etc/sysconfig/network-scripts/ifcfg-eth0:1

把GATEWAY=172.16.3.1删除

同时调整

/etc/sysconfig/network-scripts/route-eth0

最终的内容如下, 优先匹配小的子网掩码 :

default via 172.16.3.1 dev eth0 src 172.16.3.111 metric 100  
172.16.3.0/25 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1  
172.16.3.128/25 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1  
default via 172.16.3.1 dev eth0 src 172.16.3.105 metric 1

重启network服务后, 路由表如下 :

[root@db-172-16-3-111 network-scripts]# ip route  
172.16.3.0/25 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1   
172.16.3.128/25 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1   
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111   
169.254.0.0/16 dev eth0  scope link  metric 1002   
default via 172.16.3.1 dev eth0  src 172.16.3.105  metric 1   
default via 172.16.3.1 dev eth0  src 172.16.3.111  metric 100

ifup , ifdown, service network等都会自动读取这个配置文件, 所以不需要在手工修改了.

另外注意, 不要直接使用ip命令对IP进行配置.

[root@db-172-16-3-111 network-scripts]# ip route get 192.168.1.1  
192.168.1.1 via 172.16.3.1 dev eth0  src 172.16.3.105   
    cache  mtu 1500 advmss 1460 hoplimit 64  
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.1  
172.16.3.1 dev eth0  src 172.16.3.105   
    cache  mtu 1500 advmss 1460 hoplimit 64  
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.0  
172.16.3.0 dev eth0  src 172.16.3.105   
    cache  mtu 1500 advmss 1460 hoplimit 64  
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.127  
172.16.3.127 dev eth0  src 172.16.3.105   
    cache  mtu 1500 advmss 1460 hoplimit 64  
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.128  
172.16.3.128 dev eth0  src 172.16.3.105   
    cache  mtu 1500 advmss 1460 hoplimit 64  
  
[root@db-172-16-3-111 network-scripts]# ifdown eth0:1  
[root@db-172-16-3-111 network-scripts]# ip route  
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111   
169.254.0.0/16 dev eth0  scope link  metric 1002   
default via 172.16.3.1 dev eth0  src 172.16.3.111  metric 100   
[root@db-172-16-3-111 network-scripts]# ifup eth0:1  
Determining if ip address 172.16.3.105 is already in use for device eth0...  
RTNETLINK answers: File exists  
[root@db-172-16-3-111 network-scripts]# ip route  
172.16.3.0/25 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1   
172.16.3.128/25 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1   
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111   
169.254.0.0/16 dev eth0  scope link  metric 1002   
default via 172.16.3.1 dev eth0  src 172.16.3.105  metric 1   
default via 172.16.3.1 dev eth0  src 172.16.3.111  metric 100

参考

1. /usr/share/doc/initscripts-9.03.40/sysconfig.txt

2. man ip

3. man ifup

Flag Counter

digoal’s 大量PostgreSQL文章入口

You May Also Enjoy

PostgreSQL(PPAS 兼容Oracle) 从零开始入门手册 - 珍藏版

17 minute read

背景 云数据库PPAS版，是阿里云与EnterpriseDB公司(简称EDB)合作基于PostgreSQL高度兼容Oracle语法的数据库服务，为用户提供易于操作的迁移工具，兼容范围涵盖：PL/SQL、数据类型、高级函数、表分区等。 用户可以直接在阿里云购买PPAS进行使用。 如果在购买PPAS前，想试用一下...

PostgreSQL pipelinedb 流计算插件 - IoT应用 - 实时轨迹聚合

1 minute read

背景 IoT场景，车联网场景，共享单车场景，人的行为位点等，终端实时上报的是孤立的位点，我们需要将其补齐成轨迹。 例如共享单车，下单，开锁，生成订单，骑行，关闭订单，关锁。这个过程有一个唯一的订单号，每次上报的位点会包含时间，订单号，位置。 根据订单号，将点聚合为轨迹。 使用pipelinedb插件，可以实...

PostgreSQL datediff 日期间隔（单位转换）兼容SQL用法

2 minute read

背景 使用datediff，对时间或日期相减，得到的间隔，转换为目标单位(日、月、季度、年、小时、秒。。。等)的数值。 DATEDIFF ( datepart, {date|timestamp}, {date|timestamp} ) 周 select datediff(week,'2009-01-0...