There are cases when you want to access your Neo4j instance remotely and you live in an environment where direct access is not possible. This might be caused by technical or organizational restrictions.
One generic solution to this kind of problems is using a VPN. Another alternative to be discussed in this blog post is using a reverse proxy server. I want to show how you can proxy Neo4j using Nginx. Continue reading
Neo4j 3.2.0 was released last week at GraphConnect Europe. Among lots of cool new features, unfortunately it has one new “feature” making life of APOC users little bit harder, esp. if you run Neo4j from docker.
Since 3.0 you can enrich Cypher with your own stored procedures. Those are written in Java (or any other JVM language) and get deployed to the /plugins folder. In 3.1 user defined functions were added, followed by aggregate functions in 3.2.
All of them use annotations like @Context GraphDatabaseService db to retrieve a reference to the database itself. For getting deeper into Neo4j’s machinery room one could have @Context GraphDatabaseAPI api allowing you full access. This full access can be abused to break out of the permissions system added in 3.1.
Therefore in 3.2. all procedures and functions run in a sandboxed environment disallowing potentially harmful dependencies like GraphDatabaseAPI from being injected. Using a config option dbms.security.procedures.unrestricted=<procedure names> you can deactivate the sandboxing for a given list of procedures. This config option also allows wildcards like *.
Couple of procedure/functions in APOC depend on using internal components and therefore need to be added as unrestricted procedures. This can be achieved by using apoc.* as value for this config option.
In a regular (aka non-docker) deployment you would just adopt conf/neo4j.conf with that setting.
When starting a Neo4j docker container you can dynamically amend config settings using -e. Everything starting with -e NEO4J_<configKey>=<convifgValue> will be amended to the config file. Be aware that dots need be rewritten with underscore.
There is a gotcha: trying to use -e NEO4J_dbms_security_procedures_unrestricted=apoc.* does not work since the shell expands the wildcard * with all file in current directory. Even apoc.\* doesn’t help. I suspect docker internally tries another expansion. What I found finally working is using three backslashes: -e NEO4J_dbms_security_procedures_unrestricted=apoc.\\\*
My typical docker command for firing up a throw-away database to be used for demos and trainings typically looks like this:
|
1 2 3 4 |
docker run --rm -e NEO4J_AUTH=none \ -e NEO4J_dbms_security_procedures_unrestricted=apoc.\\\* \ -v $PWD/plugins:/plugins \ -p 7474:7474 -p 7687:7687 neo4j:3.2.0-enterprise |
Of course, the apoc jar file needs to reside in plugins folder of the directory where the command is fired from.
Recently Deutsche Bahn started wifi for everyone travelling with a ICE. So far I had trips where it just worked great, on other trips I could not even connect – not on mobile phone nor on my laptop.
We have great signal strength, wifi on my phone works like a charm. Connecting to the wifi works nicely on the laptop as well. But I cannot connect to the login page for accepting t&c.
I have a Thinkpad X1 Yoga laptop running Ubuntu 16.04. Among a gazillion of other packages docker is installed – mostly for dealing with lots of neo4j databases (of course ;-). The wifi (SSID: WIFIOnICE) itself is not authenticated but upon accessing the first webpage you get redirected to URL http://www.wifionice.de. Here I got a “cannot connect” error message in the browser. DNS lookups however worked fine – on couple of other WIFI issues DNS is a common culprit. Using dig www.wifionice.de I’ve learned that this hostname resolves to IP address 172.18.10.10. Next to check are the routing tables:
|
1 2 3 4 5 6 7 8 |
stefan@x1 ➜ sudo route -n Kernel-IP-Routentabelle Ziel Router Genmask Flags Metric Ref Use Iface 0.0.0.0 172.16.0.1 0.0.0.0 UG 600 0 0 wlp4s0 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 br-034d1e2af367 172.16.0.0 0.0.0.0 255.255.0.0 U 600 0 0 wlp4s0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-034d1e2af367 |
Interesting, packages to 172.18.0.0/16 are routed to a weird interface called br-034d1e2af367 and are therefore not set via the wifi device. This finding justifies a loud WTF! This bridge interface is established by docker. Since I’m just a docker user without deep understanding of its internals I cannot really explain its exact purpose. But I don’t have to 😉
It’s good enough to just disable the bridge network interface while doing the wifi authentication:
|
1 2 3 |
ihtsudo ifconfig br-034d1e2af367 down open http://www.wifionice.de in your webbrowser and press the "go online" button sudo ifconfig br-034d1e2af367 up |
After that operation I could use the internet on a ICE train without hassle.
Don’t have this one yet. I seems you can tweak the bridge’s IP number using --bip <CIDR> upon docker startup. But I couldn’t find out the details for now. Happy to read your helpful comments here.
This post is mainly intended as a self-reminder for future train trips. If it’s helpful to others as well I’m more than happy. As a reference I’ve posted this in German language to a question on a forum of Deutsch Bahn as well.
This autumn I’ve attended couple of conferences. It’s always interesting to learn about new tools popping up. One that took my attention is Apache Zeppelin. Zeppelin is a notebook style application focussing on data analytics. I’ll show in this article how Zeppelin can be used to access data residing in your favourite graph database Neo4j.
Neo4j Enterprise edition ships with a feature to take consistent backups while the database is up and running. Using config option dbms.backup.address you can expose the port to the outside.
However that port should not be exposed to public networks for two reasons. First the backup is transferred unencrypted and second no authentication is used. On online backups should only be used inside a trusted environment.
In couple of cases I wanted to grab a backup on a remote server and the only way to accessing it is via ssh. Nothing easier than this, see the following one-liner (be sure to scroll horizontally):
|
1 |
ssh <user>@<server> 'TARGET=`mktemp -d`; (neo4j-backup -from localhost -to $TARGET &> /dev/null); tar -zcf - -C $TARGET .; rm -rf $TARGET' > graph.db.tar.gz |
First we create a temporary directory to be used as target for the backup. On most systems this is created under /tmp, so be sure to have enough available space there.
It’s important to mute the call to neo4j-backup in order to not pollute the transferred archive with textual report output of neo4j-backup – therefore we redirect stdout and stderr to /dev/null.
Next step is transferring the target directory as gzipped tar archive over the wire and store it locally. As last step the temporary directory gets deleted.
This blog post quickly introduces one of my side projects called neo4j-csv-firehose.
The goal is to enrich Cypher’s great LOAD CSV command by adding support for non-csv datasources. For now there’s the capability to use jdbc datasources. In future other sources might be added – think of json, xml, …
It features three deployment modes:
LOAD CSV
jdbc: style URLs is added.
The project’s README should give you all you need to get started. Your feedback is highly appreciated.
This article summarizes the required and optional communication channels for a Neo4j enterprise cluster and provides some sample rules. For the firewall rules I’m using Ubuntu’s ufw package.
As a sample setup we assume the following servers. All of them have 2 network interfaces, eth0 is for communication with outside world, eth1 is for cluster communication.
| name | eth0 ip address | eth1 ip address |
|---|---|---|
| server1 | 172.16.0.1 | 192.168.1.1 |
| server2 | 172.16.0.2 | 192.168.1.2 |
| server3 | 172.16.0.3 | 192.168.1.3 |
By default Neo4j listens on port 7474 for http and on 7473 for https style requests to the rest api. To allow for remote access you need to set in neo4j-server.properties:
|
1 |
org.neo4j.server.webserver.address=172.16.0.1 // or 0.0.0.0 for all interfaces |
Inbound access needs to be granted using
|
1 |
ufw allow in on eth0 proto tcp from any to any port 7474 |
When using SSL the rule needs to hit port 7473 instead.
A Neo4j cluster uses two different communication channels: one for cluster management (joining/leaving the cluster, master elections, etc.) and one for transaction propagation. By default ports 5001 and 6001 are used. On all cluster members we need to allow inbound and outbound traffic for these to the other cluster members:
|
1 2 3 4 5 6 7 |
# cluster management: inbound and outbound ufw allow in on eth1 proto tcp from 192.168.1.0/24 to any port 5001 ufw allow out on eth1 proto tcp to 192.168.1.0/24 port 5001 # transaction propagation: inbound and outbound ufw allow in on eth1 proto tcp from 192.168.1.0/24 to any port 6001 ufw allow out on eth1 proto tcp to 192.168.1.0/24 port 6001 |
Neo4j enterprise supports online backup – the default port for this 6362. To enable remote backup you need to set in neo4j.properties:
|
1 |
online_backup_server=192.168.1.1:6362 // or 0.0.0.0:6362 for listening also on eth0 |
The corresponding ufw command is:
|
1 2 |
ufw allow out on eth1 proto tcp to 192.168.1.0/24 port 6362 ufw allow in on eth1 proto tcp from 192.168.1.0/24 to any port 6362 |
This is pretty tricky. Under the hoods neo4j remote shell uses Java RMI. When a new connection is established the client communicates with server:1337. During this control session a secondary port is negotiated – unfortunately a random port is used. So the client opens a second connection to the server with the negotiated port – therefore we have to open up basically all ports for the ip addresses acting as shell clients:
|
1 2 |
ufw allow in on eth1 proto tcp from <shell-client-ip> ufw allow out on eth1 proto tcp to <shell-client-ip> |
There might be a more sophisticated approach by implementing a custom RMISocketFactory and register it with the JVM as described in the JVM docs. I have not yet tried this, so if you have explored that path yourself I’d appreciate to hear your solution to this.
The goal of that blog post is to provide a reasonable haproxy configuration suitable for most neo4j clusters. It covers the Neo4j specific needs (authentication, stickyness, request routing) suitable for typically deployments.
When running a Neo4j cluster you typically want to front it with a load balancer. The load balancer itself is not part of Neo4j since a lot of customers already have hardware load balancers in their infrastructure. In case they have not or want some easy-to-go solution for that the recommended tool is haproxy.
In a Neo4j cluster its perfectly possible to run any operation (reads and writes) on any cluster member. However performance-wise it’s a good idea to send write requests directly to the master and send read-only requests to all cluster member or just to the slaves – depending how busy your master is.
These days most folks interact with Neo4j by sending Cypher commands through the transactional cypher end point. This endpoint uses always a HTTP POST regardless if the Cypher commands are intended to perform read or write operation. This causes some pain for load balancing since we need to distinguish reads from writes.
One approach for this is to use a additional custom HTTTP header on the client side. Assume any request that should perform writes gets a X-Write:1 header in the request. Then haproxy just needs to check for this header in and acl and associate this acl with the backend containing only the master instance.
Another idea is that the load balancer inspects the request’s payload (aka the Cypher command you’re sending). If that payload contains write clauses like CREATE, MERGE, SET, DELETE, REMOVE or FOREACH it is most likely a write operation to be directed to the master. Luckily haproxy has capbilites to inspect the payload and apply regex matches.
A third approach would be to encapsulate each and every of your use cases into an unmanaged extension and use Cypher, Traversal API or core API inside as an implementation detail. By using the semantics of REST appropriately you use GET for side-effect free reads, PUT, POST and DELETE for writes. These can be used in haproxy’s acl as well very easily.
Another issue we have to deal when setting up the load balancer with is authentication. Since Neo4j 2.2 the database is protected by default with username and password. Of course we can switch that feature off, but if security is a concern it’s wise to have it enabled. In combination with haproxy we need to be aware that haproxy needs to use authentication as well for its status checks to identify who’s available and who’s master.
I’ve spent some time to fiddle out a haproxy config addressing all the points mentioned above. Let’s go through the relevant parts below line by line – for the rest I delegate to the excellent documentation for haproxy. Please note that this config requires a recent version of haproxy:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
global daemon maxconn 256 stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m defaults mode http option httplog timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend http-in bind *:8090 acl write_method method POST DELETE PUT acl write_hdr hdr_val(X-Write) eq 1 acl write_payload payload(0,0) -m reg -i CREATE|MERGE|SET|DELETE|REMOVE acl tx_cypher_endpoint path_beg /db/data/transaction http-request set-var(txn.tx_cypher_endpoint) bool(true) if tx_cypher_endpoint use_backend neo4j-master if write_hdr use_backend neo4j-master if tx_cypher_endpoint write_payload use_backend neo4j-all if tx_cypher_endpoint use_backend neo4j-master if write_method default_backend neo4j-all backend neo4j-all option httpchk GET /db/manage/server/ha/available HTTP/1.0\r\nAuthorization:\ Basic\ bmVvNGo6MTIz #option httpchk GET /db/manage/server/ha/available acl tx_cypher_endpoint var(txn.tx_cypher_endpoint),bool stick-table type integer size 1k expire 70s # slightly higher with org.neo4j.server.transaction.timeout stick match path,word(4,/) if tx_cypher_endpoint stick store-response hdr(Location),word(6,/) if tx_cypher_endpoint server s1 127.0.0.1:7474 maxconn 32 check server s2 127.0.0.1:7475 maxconn 32 check server s3 127.0.0.1:7476 maxconn 32 check backend neo4j-master option httpchk GET /db/manage/server/ha/master HTTP/1.0\r\nAuthorization:\ Basic\ bmVvNGo6MTIz #option httpchk GET /db/manage/server/ha/master server s1 127.0.0.1:7474 maxconn 32 check server s2 127.0.0.1:7475 maxconn 32 check server s3 127.0.0.1:7476 maxconn 32 check listen admin bind *:8080 stats enable stats realm Haproxy\ Statistics stats auth admin:123 |
|
16 17 18 19 |
acl write_method method POST DELETE PUT acl write_hdr hdr_val(X-Write) eq 1 acl write_payload payload(0,0) -m reg -i CREATE|MERGE|SET|DELETE|REMOVE acl tx_cypher_endpoint path_beg /db/data/transaction |
The acl declare conditions to be used later one. We have acls to identify if the incoming request
X-Write:1|
20 |
http-request set-var(txn.tx_cypher_endpoint) bool(true) if tx_cypher_endpoint |
We do a nasty trick here: we store a internal variable holding a boolean depending on wether the request is for the tx endpoint. Later on we refer back to that variable. The rationale is that haproxy cannot access acls in other sections of the config file.
|
21 22 23 24 |
use_backend neo4j-master if write_hdr use_backend neo4j-master if tx_cypher_endpoint write_payload use_backend neo4j-all if tx_cypher_endpoint use_backend neo4j-master if write_method |
The acls defined above decide to which backend a request should be directed. We have one endpoint for the neo4j cluster’s master and another one for a pool of all available neo4j instances (master and slaves). A request is send to master if
X-Write:1 header is set, or|
28 29 |
option httpchk GET /db/manage/server/ha/available HTTP/1.0\r\nAuthorization:\ Basic\ bmVvNGo6MTIz #option httpchk GET /db/manage/server/ha/available |
|
39 40 |
option httpchk GET /db/manage/server/ha/master HTTP/1.0\r\nAuthorization:\ Basic\ bmVvNGo6MTIz #option httpchk GET /db/manage/server/ha/master |
A backend needs to check which of its member is available. Therefore Neo4j has a REST endpoint exposing the individual instance’s status. In case we use Neo4j authentication the requests to the status endpoint need to be authenticated. Thanks to stackoverflow, I’ve figured out that you can add a Authorization to these requests. As usual, the value for the auth header is a base64 encoded string of “<username>:<password>”. On a unix command line use:
|
1 |
echo "neo4j:123"| base64 |
In case you’ve disabled authentication in Neo4j, use the commented lines instead.
|
30 31 32 33 |
acl tx_cypher_endpoint var(txn.tx_cypher_endpoint),bool stick-table type integer size 1k expire 70s # slightly higher with org.neo4j.server.transaction.timeout stick match path,word(4,/) if tx_cypher_endpoint stick store-response hdr(Location),word(6,/) if tx_cypher_endpoint |
Using our previously stored variable, we get the value back and use it to declare an acl in this backend. To make sure any request for the same transaction number is sent to the same neo4j instance we create a sticky table to store the transaction ids. You might want to adopt the size of that table (here 1k) to your needs, as well the expire time should be aligned with neo4j’s org.neo4j.server.transaction.timeout setting. It defaults to 60s, so if we expire the sticky table entry after 70s we’re on the safe side. If a http response from the tx endpoint has a Location header we store its 6th word – that is the transaction number – in the sticky table. If we hit the tx endpoint we check if the request path’s 4th word (the transaction id) is found in the sticky table. If so, the request gets send to the same server again, otherwise we use the default policy (round-robin).
|
34 35 36 |
server s1 127.0.0.1:7474 maxconn 32 check server s2 127.0.0.1:7475 maxconn 32 check server s3 127.0.0.1:7476 maxconn 32 check |
These lines hold a list of all cluster instances. Be sure to align the maxconn value with the amount of max_server_threads. By default Neo4j uses 10 threads per CPU core. So 32 is a good value for a 4 core box.
|
45 46 47 48 49 |
listen admin bind *:8080 stats enable stats realm Haproxy\ Statistics stats auth admin:123 |
Haproxy provides a nice status dashbord. In most cases you want to protect that with a password. 
I hope this post could provide you some insights on how to use haproxy in front of a neo4j cluster efficiently. Of course I am aware that my experience and knowledge with haproxy is limited. So I appreciate any feedback to improve the config described here.
Based on couple of questions I should have more clearly mentioned that the sketched setup requires a haproxy 1.6 which is currently in development.
Haproxy 1.6 has been released GA on Oct 23rd – so no need to use a in-dev version any more.
There’s a docker image providing a preconfigured haproxy for neo4j available at https://github.com/UKHomeOffice/docker-neo4j-haproxy. Kudos to Giles.
At a onsite workshop with a new potential customer in wonderful Zurich I was challenged with a requirement I’ve never had in the last few years with Cypher.
Since I cannot disclose any details of the use case or the data used let’s create an artificial example with a similar structure:
The goal of the query is to find the most occupied table for each room. As an example, I’ve created a Neo4j console at http://console.neo4j.org/r/ufmpwi.
The challenging here is that we want to do a kind of local filtering: for a given room we need the most occupied table. Calling the reduce function to the rescue. The idea is to run the reduce with 3 state variables:
|
1 2 3 |
RETURN reduce(x=[0,0,0], i IN [1,2,2,5,2,1] | CASE WHEN i>x[2] THEN [x[1],x[1]+1,o] ELSE [x[0], x[1]+1,x[2]] END )[0] |
Reduce allows just one single state variable to be used but we can use a three element array instead (which is one variable 🙂 ). When the current element is larger than the maximum so far (aka the 3rd element of the state), we update the first element to the current position. The second element (current index) is always incremented. If the current element is not larger then the maximum so far, we just increment the current count (2nd element) and keep the other values:
The full query is:
|
1 2 3 4 5 6 7 8 9 10 |
MATCH (:Restaurant)-[:HAS_ROOM]->(room)-[:HAS_TABLE]->(table) OPTIONAL MATCH (guest)-[:SITS_AT]->(table) WITH room, table, count(guest) AS guestsAtTable WITH room, collect(table) AS tables, collect(guestsAtTable) AS occupation RETURN room, tables[reduce(x=[0,0,0], o IN occupation | CASE WHEN o>x[2] THEN [x[1], x[1]+1,o] ELSE [x[0], x[1]+1,x[2]] END )[0]] AS mostOccupied, tables, occupation |
The first line is pretty much obvious. Since there might be tables without guests OPTIONAL MATCH is required in line 2.
Cypher does not allow you to do direct aggregations of aggregations. Using multiple WITH helps here. In line 3 we first calculate counts of guests per table. Line 4 returns one line per room with two collections – one holding the tables, the other their occupation. Note that both collections do have the same order. Finally from line 5 on the reduce function is applied to find the most occupied table in each room.
In a previous blog post I’ve explained the differences of the different types of indexes being available in Neo4j. A common requirement for a lot of projects is the usage of fulltext indexes. With current versions of Neo4j (2.1.5 as of now) this can only be accomplished with the usage of manual indexes.
In this article I want to explain how you can use language specific analyzers for fulltext indexing and how to do regex searches for those.
When looking at the reference manual on fulltext indexing there is the notion of providing a custom analyzer class by specifying a config parameter analyzer upon index creation. It’s value is the full class name of the analyzer. There are two ways to create a manual index this, either using java api
|
1 2 3 4 5 6 7 |
GraphDatabaseService graphDb = .... IndexManager indexManager = graphDb.index() try (Transaction tx = graphDb.beginTx()) { Map<String,String> params = Collections.singletonMap("analyzer", "my.package.Analyzer") Index index = indexManager.forNodes("myfulltextindex", params); } |
or using REST API (using the wonderful httpie http command line client)
|
1 2 |
http -v -j localhost:7474/db/data/index/node \ name=myfulltextindex config:='{"analyzer":"my.package.Analyzer"}' |
Lucene provides an optional set of language specific analyzers. These analyzers have some knowledge on the language their operating on and use that for word stemming, see http://www.evelix.ch/unternehmen/Blog/evelix/2013/11/11/inner-workings-of-the-german-analyzer-in-lucene for details on the internals of the GermanAnalyzer. As an example the German word for houses “Häuser” is stemmed to its singular form “Haus”. Consequently a query for “Haus” retrieves all both, occurrences of “Haus” and “Häuser”.
The language specific analyzers are residing in an optional jar file called lucene-analyzers-3.6.2.jar that is not shipping by default with Neo4j. Therefore copy lucene-analyzers-3.6.2.jar into Neo4j’s plugins folder.
When trying e.g. to use Lucene’s GermanAnalyzer using
|
1 2 |
http -v -j localhost:7474/db/data/index/node name=fulltext_de \ config:='{"analyzer":"org.apache.lucene.analysis.de.GermanAnalyzer"}' |
you get back a HTTP status 500. The log files show up a strange exception java.lang.InstantiationException: org.apache.lucene.analysis.de.GermanAnalyzer. The reason for this exception is that Neo4j tries to instantiate the analyzer class using a noarg default constructor. Unfortunately Lucene’s language specific analyzers don’t have such a constructor, see javadocs. The solution for this is write a thin analyzer class with a default constructor. Internally that class uses the Lucene provided analyzer as a delegate.
In order to simplify the process of setting this up I’ve create a small project on github called neo4j-fti. It contains the mentioned wrappers in package org.neo4j.contrib.fti.analyzers for all languages having a lucene analyzer. It also provides a kernel extension to Neo4j to automatically create fulltext indexes by a config option. In neo4j.properties you need to set:
|
1 2 |
fullTextIndexes=fulltext_de:org.neo4j.contrib.fti.analyzers.German,\ fulltext_en:org.neo4j.contrib.fti.analyzers.English |
Additionally this project features an example how to use regular expression for search an index. Using Java API you need to pass a Lucene RegexQuery based on a Term holding your regular expression. The RegexQuery class isn’t part of lucene-core either, so be sure to have lucene-queries in your Neo4j’s plugins folder as well. This example is exposed in a unmanaged extension using the following code snippet:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
try (Transaction tx = graphDatabaseService.beginTx()) { IndexManager indexManager = graphDatabaseService.index(); if (!indexManager.existsForNodes(indexName)) { throw new IllegalArgumentException("index " + indexName + " does not exist"); } Index index = indexManager.forNodes(indexName); IndexHits hits = index.query(new RegexQuery(new Term(field, regex))); List result = new ArrayList<>(); for (Node node: hits) { result.add(node.getId()); } } |
Assuming a index named fulltext_de has been configured using the German analyzer (see above), use the following code using httpie again to create a node, add it to the fulltext index and perform a regular expression index query:
|
1 2 3 4 5 6 7 8 9 10 |
# create a node http -j localhost:7474/db/data/cypher query="create (n:Blog {description:'Auf der Straße stehen fünf Häuser'}) return id(n)" # put it to the index: http -j localhost:7474/db/data/index/node/fulltext_de \ uri="http://localhost:7474/db/data/node/xxxx" \ key="description" value="Auf der Straße stehen fünf Häuser" # query the index for words starting with "h" and ending with "s" http localhost:7474/regex/fulltext_de/description/h.*s |