图数据库 CayLey + PostgreSQL

背景

CayLey是GO语言写的一个图数据库引擎，支持RESTful API，内置查询编辑器和可视化，支持MQL和JAVASCRIPT查询接口，后端存储支持文件格式，PostgreSQL，mongodb，LevelDB，Bolt。模块化设计，扩展后端存储非常容易。

本文将以PostgreSQL为例，演示一下CayLey的使用。

安装go：

yum install -y go  

执行以下命令，克隆cayley和依赖：

mkdir -p ~/cayley && cd ~/cayley  
export GOPATH=`pwd`  
export PATH=$PATH:~/cayley/bin  
mkdir -p bin pkg src/github.com/google  
cd src/github.com/google  
git clone https://github.com/google/cayley  
cd cayley  
go get github.com/tools/godep  
godep restore  
go build ./cmd/cayley  

样本数据：

$ ll data  
-rw-rw-r--. 1 postgres postgres 26M Jan 17 21:45 30kmoviedata.nq.gz  
-rw-rw-r--. 1 postgres postgres 463 Jan 17 21:45 testdata.nq  
$ gunzip 30kmoviedata.nq.gz  

cayley使用帮助：

$ ./cayley --help  
No command --help  
  
Usage:  
  cayley COMMAND [flags]  
  
Commands:  
  init      Create an empty database.  
  load      Bulk-load a quad file into the database.  
  http      Serve an HTTP endpoint on the given host and port.  
  dump      Bulk-dump the database into a quad file.  
  repl      Drop into a REPL of the given query language.  
  version   Version information.  
  
Flags:  
  -alsologtostderr=false: log to standard error as well as files  
  -assets="": Explicit path to the HTTP assets.  
  -config="": Path to an explicit configuration file.  
  -db="memstore": Database Backend.  
  -dbpath="/tmp/testdb": Path to the database.  
  -dump="dbdump.nq": Quad file to dump the database to (".gz" supported, "-" for stdout).  
  -dump_type="quad": Quad file format ("json", "quad", "gml", "graphml").  
  -format="cquad": Quad format to use for loading ("cquad" or "nquad").  
  -host="127.0.0.1": Host to listen on (defaults to all).  
  -ignoredup=false: Don't stop loading on duplicated key on add  
  -ignoremissing=false: Don't stop loading on missing key on delete  
  -init=false: Initialize the database before using it. Equivalent to running `cayley init` followed by the given command.  
  -load_size=10000: Size of quadsets to load  
  -log_backtrace_at=:0: when logging hits line file:N, emit a stack trace  
  -log_dir="": If non-empty, write log files in this directory  
  -logstashtype="": enable logstash logging and define the type  
  -logstashurl="172.17.42.1:5042": logstash url and port  
  -logtostderr=false: log to standard error instead of files  
  -port="64210": Port to listen on.  
  -prof="": Output profiling file.  
  -quads="": Quad file to load before going to REPL.  
  -query_lang="gremlin": Use this parser as the query language.  
  -read_only=false: Disable writing via HTTP.  
  -replication="single": Replication method.  
  -stderrthreshold=0: logs at or above this threshold go to stderr  
  -timeout=30s: Elapsed time until an individual query times out.  
  -v=0: log level for V logs  
  -vmodule=: comma-separated list of pattern=N settings for file-filtered logging  

假设已有一个PostgreSQL数据库。

IP : 192.168.150.132  
PORT : 1921  
DBNAME : postgres  
USER : digoal  
PWD : digoal_pwd  

初始化

./cayley init -db=sql -dbpath="postgres://digoal:digoal_pwd@192.168.150.132:1921/postgres?sslmode=disable"  

导入数据

./cayley load -quads="data/" -db=sql -dbpath="postgres://digoal:digoal_pwd@192.168.150.132:1921/postgres?sslmode=disable"  

50亿测试数据约2TB。

开启repl或http接口服务。

./cayley repl -db=sql -dbpath="postgres://digoal:digoal_pwd@192.168.150.132:1921/postgres?sslmode=disable" -host="0.0.0.0" -port="64210"  

或

./cayley http -db=sql -dbpath="postgres://digoal:digoal_pwd@192.168.150.132:1921/postgres?sslmode=disable" -host="0.0.0.0" -port="64210"  

使用http接口的图例：

Query Shape：

后端是PostgreSQL时，Cayley自动将MQL或JAVASCRIPT自动转换成SQL到数据库查询，并返回结果。

对于PostgreSQL作为后端的场景，优化的手段：

1. 使用GPU加速HASH JOIN和数据扫描。

2. 使用分区表，减少无用块扫描。

3. 其他通用的PG优化手段

如果数据量大到单库的计算资源和IO资源性能支撑不住，可以用Greenplum来实现分布式查询。

查询接口：

Javascript/Gremlin API documentation

图对象

根据节点ID，检索，返回路径

路径对象

路径相交，节点匹配等

查询路径对象

数值转换，等。

参考

1. https://github.com/google/cayley

digoal’s 大量PostgreSQL文章入口