Saturday, December 10, 2016

Distributed in-memory Cache - Apache Ignite


Apache Ignite is highly performant because it has a deep history of distributed deployment with sql, transactional and locking support.  It was recently open sourced to Apache.

Mixed metal and VM's
[19:05:22] Ignite node started OK (id=7352a302)
[19:05:22] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=12.0GB]
[19:05:32] Topology snapshot [ver=2, servers=2, clients=0, CPUs=8, heap=24.0GB]
[19:05:42] Topology snapshot [ver=3, servers=3, clients=0, CPUs=12, heap=36.0GB]
[19:05:50] Topology snapshot [ver=4, servers=4, clients=0, CPUs=16, heap=48.0GB]
[19:05:57] Topology snapshot [ver=5, servers=5, clients=0, CPUs=24, heap=60.0GB]
[19:06:07] Topology snapshot [ver=6, servers=6, clients=0, CPUs=30, heap=68.0GB]

[19:06:16] Topology snapshot [ver=7, servers=7, clients=0, CPUs=36, heap=76.0GB]

Bare Metal NUCs


VMware RHEL 7.3 Virtual Machines (bridged mode) across two physical hosts




GridGain Enterprise Edition 7.5 corresponds to Apache Ignite 1.5 in February 2016

Release 1.8.0 8 Dec 2016

Amazon Image
2015-06-25


  • Download Apache Ignite as ZIP archive from https://ignite.apache.org/
  • Unzip ZIP archive into the installation folder in your system
  • Set IGNITE_HOME environment variable to point to the installation folder and make sure there is no trailing / in the path (this step is optional)


Maven/Gradle dependencies

<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-core</artifactId>
    <version>${ignite.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-spring</artifactId>
    <version>${ignite.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-indexing</artifactId>
    <version>${ignite.version}</version>

</dependency>

Setup


Open ports on all machines
47100 = local
48100 = shared memory

   90  wget http://apache.mirror.iweb.ca//ignite/1.8.0/apache-ignite-fabric-1.8.0-bin.zip
   91  unzip apache-ignite-fabric-1.8.0-bin.zip 

export IGNITE_HOME=/root/ignite-180


 bin/ignite.sh examples/config/example-ignite.xml

Thursday, December 1, 2016

Running High Memory Java Applications on Amazon EC2 2048Gb 128 vCPU Memory Optimized x1.32xlarge instances

Amazon has a huge instance that we can use to run massive memory models.
Here I am creating hypercube (see CM-1 from the 90's).  On a normal 64G system like a Haswell 5820K board fully populated with DDR4 we top out at 22 dimensions.


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852    3801596 2010941256         64      13396     238792
-/+ buffers/cache:    3549408 2011193444
Swap:            0          0          0



[ec2-user@ip-172-31-57-105 ~]$ java -version
java version "1.7.0_121"
OpenJDK Runtime Environment (amzn-2.6.8.1.69.amzn1-x86_64 u121-b00)
OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)
[ec2-user@ip-172-31-57-105 ~]$ sudo yum install java-1.8.0
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                                                                                                                | 2.1 kB     00:00     
amzn-updates/latest                                                                                                                             | 2.3 kB     00:00     
Resolving Dependencies
--> Running transaction check
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed
--> Processing Dependency: java-1.8.0-openjdk-headless = 1:1.8.0.111-1.b15.25.amzn1 for package: 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64
--> Running transaction check
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed
amzn-main/latest/filelists_db                                                                                                                   | 5.1 MB     00:00     
amzn-updates/latest/filelists_db                                                                                                                | 1.2 MB     00:00     
--> Processing Dependency: lksctp-tools for package: 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64
--> Running transaction check
---> Package lksctp-tools.x86_64 0:1.0.10-7.7.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================================================
 Package                                          Arch                        Version                                          Repository                         Size
=======================================================================================================================================================================
Installing:
 java-1.8.0-openjdk                               x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                      227 k
Installing for dependencies:
 java-1.8.0-openjdk-headless                      x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                       39 M
 lksctp-tools                                     x86_64                      1.0.10-7.7.amzn1                                 amzn-main                          89 k

Transaction Summary
=======================================================================================================================================================================
Install  1 Package (+2 Dependent packages)

Total download size: 39 M
Installed size: 102 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                                   | 227 kB     00:00     
(2/3): java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                          |  39 MB     00:00     
(3/3): lksctp-tools-1.0.10-7.7.amzn1.x86_64.rpm                                                                                                 |  89 kB     00:00     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                   48 MB/s |  39 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 
  Installing : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       2/3 
  Installing : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                3/3 
  Verifying  : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 
  Verifying  : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                2/3 
  Verifying  : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       3/3 

Installed:
  java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1                                                                                                                 

Dependency Installed:
  java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1                                 lksctp-tools.x86_64 0:1.0.10-7.7.amzn1                                

Complete!

[ec2-user@ip-172-31-57-105 ~]$ sudo alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
   2           /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

Enter to keep the current selection[+], or type selection number: 2
[ec2-user@ip-172-31-57-105 ~]$ java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)


[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 1
22 dimensional hypercube: 1480643986605
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
47345 time: Total Ram: 8450998272 free: 2292978440

[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852   20609684 1994133168         68      14064     406412
-/+ buffers/cache:   20189208 1994553644
Swap:            0          0          0

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 2 -server 
22 dimensional hypercube: 1480644388217
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
43511 time: Total Ram: 8432648192 free: 2267825616

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 
23 dimensional hypercube: 1480644473000
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.
140562 time: Total Ram: 18184929280 free: 5885736464

top - 02:09:40 up  2:05,  2 users,  load average: 39.26, 21.54, 10.94
Tasks: 730 total,   1 running, 729 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 19851880k used, 1994890972k free,    14168k buffers
Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
  7999 ec2-user  20   0 39.9g  15g  14m S 8058.5  0.8  96:23.57 java  

Tasks: 729 total,   2 running, 727 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.8%us,  0.4%sy,  0.0%ni, 36.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 32436576k used, 1982306276k free,    14304k buffers
Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
  8226 ec2-user  20   0 39.8g  27g  15m S 8098.1  1.4 255:48.30 java  

[vagrant@nsp-latest workspace]$ free
              total        used        free      shared  buff/cache   available
Mem:        7495248      263396     5030584       10720     2201268     6923444
Swap:       1048572      171480      877092

[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852   33580832 1981162020         68      14372     406416
-/+ buffers/cache:   33160044 1981582808

Swap:            0          0          0


running with extra memory


[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 24 1 
24 dimensional hypercube: 1480649540504
_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.
101381 time: Total Ram: 526849671168 free: 471873878864



[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 
23 dimensional hypercube: 1480649699358
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.

50312 time: Total Ram: 526849671168 free: 496612985560



Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 52853856k used, 1961888996k free,    18704k buffers
Swap:        0k total,        0k used,        0k free,   436572k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 43697 ec2-user  20   0  538g  46g  14m S 99.9  2.4   1:26.89 java    


[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 25 1 
25 dimensional hypercube: 1480649786242
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.

289783 time: Total Ram: 526849671168 free: 397656555744


[ec2-user@ip-172-31-57-105 ~]$ nohup java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1 &
[1] 44231
[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44231  43947 91 03:55 pts/0    00:00:05 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

[ec2-user@ip-172-31-57-105 ~]$ tail -f nohup.out

2 dimensional hypercube: 1480650669281
_created 4 bidirectional links between 4 routers in the 2 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
3 dimensional hypercube: 1480650669283
_created 12 bidirectional links between 8 routers in the 3 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
4 dimensional hypercube: 1480650669283
_created 32 bidirectional links between 16 routers in the 4 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
5 dimensional hypercube: 1480650669283
_created 80 bidirectional links between 32 routers in the 5 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
6 dimensional hypercube: 1480650669284
_created 192 bidirectional links between 64 routers in the 6 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
7 dimensional hypercube: 1480650669284
_created 448 bidirectional links between 128 routers in the 7 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
8 dimensional hypercube: 1480650669286
_created 1024 bidirectional links between 256 routers in the 8 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
9 dimensional hypercube: 1480650669287
_created 2304 bidirectional links between 512 routers in the 9 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
10 dimensional hypercube: 1480650669288
_created 5120 bidirectional links between 1024 routers in the 10 dimensional HyperCube.
3 time: Total Ram: 1843972800512 free: 1826792931288
11 dimensional hypercube: 1480650669291
_created 11264 bidirectional links between 2048 routers in the 11 dimensional HyperCube.
5 time: Total Ram: 1843972800512 free: 1826792931288
12 dimensional hypercube: 1480650669296
_created 24576 bidirectional links between 4096 routers in the 12 dimensional HyperCube.
10 time: Total Ram: 1843972800512 free: 1826792931288
13 dimensional hypercube: 1480650669306
_created 53248 bidirectional links between 8192 routers in the 13 dimensional HyperCube.
21 time: Total Ram: 1843972800512 free: 1826792931288
14 dimensional hypercube: 1480650669327
_created 114688 bidirectional links between 16384 routers in the 14 dimensional HyperCube.
38 time: Total Ram: 1843972800512 free: 1826792931288
15 dimensional hypercube: 1480650669365
_created 245760 bidirectional links between 32768 routers in the 15 dimensional HyperCube.
82 time: Total Ram: 1843972800512 free: 1826792931288
16 dimensional hypercube: 1480650669447
_created 524288 bidirectional links between 65536 routers in the 16 dimensional HyperCube.
174 time: Total Ram: 1843972800512 free: 1826792931288
17 dimensional hypercube: 1480650669621
_created 1114112 bidirectional links between 131072 routers in the 17 dimensional HyperCube.
384 time: Total Ram: 1843972800512 free: 1826792931288
18 dimensional hypercube: 1480650670005
_created 2359296 bidirectional links between 262144 routers in the 18 dimensional HyperCube.
923 time: Total Ram: 1843972800512 free: 1826792931288
19 dimensional hypercube: 1480650670928
_created 4980736 bidirectional links between 524288 routers in the 19 dimensional HyperCube.
1970 time: Total Ram: 1843972800512 free: 1826792931288
20 dimensional hypercube: 1480650672898
_created 10485760 bidirectional links between 1048576 routers in the 20 dimensional HyperCube.
4677 time: Total Ram: 1843972800512 free: 1826792931288
21 dimensional hypercube: 1480650677576
_created 22020096 bidirectional links between 2097152 routers in the 21 dimensional HyperCube.
9739 time: Total Ram: 1843972800512 free: 1818202996656
22 dimensional hypercube: 1480650687314
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
21156 time: Total Ram: 1843972800512 free: 1809613062048
23 dimensional hypercube: 1480650708470
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.
44953 time: Total Ram: 1843972800512 free: 1783843258176


Tasks: 743 total,   1 running, 742 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 55841328k used, 1958901524k free,    19244k buffers
Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                

 44386 ec2-user  20   0 1865g  49g  14m S 100.2  2.6   1:19.67 java

24 dimensional hypercube: 1480651245506
_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.
101306 time: Total Ram: 1843972800512 free: 1732303649872
25 dimensional hypercube: 1480651346812
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.
264376 time: Total Ram: 1843972800512 free: 1603454628832



Tasks: 742 total,   1 running, 741 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 278517036k used, 1736225816k free,    19408k buffers
Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
 44386 ec2-user  20   0 1865g 261g  15m S 100.2 13.6   8:45.96 java  


26 dimensional hypercube: 1480651611188
_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.

656891 time: Total Ram: 1843972800512 free: 1647312996576


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852  661273556 1353469296         72      19672     436804

-/+ buffers/cache:  660817080 1353925772

 44386 ec2-user  20   0 1865g 625g  15m S 99.9 32.6  82:37.39 java   


Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1088382060k used, 926360792k free,    19984k buffers
Swap:        0k total,        0k used,        0k free,   436804k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 44386 ec2-user  20   0 1870g 1.0t  15m S 100.2 53.7 313:16.41 java   



25 dimensional hypercube: 1480651346812
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.
264376 time: Total Ram: 1843972800512 free: 1603454628832
26 dimensional hypercube: 1480651611188
_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.
656891 time: Total Ram: 1843972800512 free: 1647312996576
27 dimensional hypercube: 1480652268079
_created 1811939328 bidirectional links between 134217728 routers in the 27 dimensional HyperCube.
1433987 time: Total Ram: 1843972800512 free: 1253174190840
28 dimensional hypercube: 1480653702066
_created 3758096384 bidirectional links between 268435456 routers in the 28 dimensional HyperCube.
4437315 time: Total Ram: 1843972800512 free: 1001914613768
29 dimensional hypercube: 1480658139381
[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852 1992699228   22043624         64      28708     437000
-/+ buffers/cache: 1992233520   22509332
Swap:            0          0          0
[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44386      1 99 03:59 ?        18-01:29:52 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

Tasks: 739 total,   1 running, 738 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1992699740k used, 22043112k free,    28736k buffers
Swap:        0k total,        0k used,        0k free,   437000k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
 44386 ec2-user  20   0 1898g 1.8t  15m S 8059.8 98.5  26052:50 java  


Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1734049420k used, 280693432k free,    32136k buffers
Swap:        0k total,        0k used,        0k free,   437292k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1888g 1.6t  14m S 8045.2 85.7   5836:37 java  


Tasks: 744 total,   1 running, 743 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1971207064k used, 43535788k free,    33936k buffers
Swap:        0k total,        0k used,        0k free,   437308k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1886g 1.8t  14m S 100.0 97.4  13311:12 java   


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached

Mem:    2014742852 1971206692   43536160         64      33944     437308

29 dimensional hypercube: 1480681783260
_created 7784628224 bidirectional links between 536870912 routers in the 29 dimensional HyperCube.

16316219 time: Total Ram: 1843972800512 free: 147852676280

10 min 26d 30g
25 min 27d 75g
75 min 28d 185g
225 min 29d 460g
800 min 30d 1300g

Wednesday, November 30, 2016

Density Functional Theory

Carbon Based Lifeforms "Only illogics can find, hidden flaws in a straight logic line.  Only erratics can recognize, errors in patterns of a perfect size."

Get the Thule case for your Macbook Pro 15 https://www.thule.com/en-us/ca/computer-phone-cases/laptop-cases/thule-gauntlet-30-macbook-pro-sleeve-15-_-tl_85854234856

Tuesday, November 8, 2016

Neo4j 3.1 Graph Database for Distributed Development

see http://wiki.obrienlabs.cloud/display/DEV/Graph+Database+Service+with+Neo4J

Neo4j has released its 3.1.0 version of their graph database as of Dec 2016.  

Neo4j 3.1 Distributed Server Graph Database

This section details setting up Neo4j 3.1 Server on several metal and VM instances of RHEL 7.3
follow https://neo4j.com/docs/operations-manual/current/installation/linux/tarball/
Download: neo4j-enterprise-3.1.0-unix.tar.gz
copy it over to each box (I don't use a chef/ansible combo yet on these)

Source
https://github.com/obrienlabs/nbi-neo4j-embedded-aws-war

Add a node to the embedded graph db
(bolt enabled) http browser - user:neo4j pass:password



[root@obrien ~]# scp michael@192.168.15.7:/Users/michael/Dropbox/neo4j/neo4j-enterprise-3.1.0-unix.tar.gz .
[root@obrien neo4j-enterprise-3.1.0]# vi ~/.bashrc
export NEO4J_HOME=/root/neo4j-enterprise-3.1.0
I am blindly running as root inside my private subnet so I increase the file handle limits as

vi /etc/security/limits.conf
neo4j soft nofile 40000
neo4j hard nofile 40000
root soft nofile 40000
root hard nofile 40000

[root@obrien bin]# ./neo4j console
Starting Neo4j.
2016-12-18 23:25:26.073+0000 INFO  No SSL certificate found, generating a self-signed certificate..
2016-12-18 23:25:26.499+0000 INFO  Starting...
2016-12-18 23:25:27.227+0000 INFO  Bolt enabled on localhost:7687.
2016-12-18 23:25:27.241+0000 INFO  Initiating metrics...
2016-12-18 23:25:30.054+0000 INFO  Started.
2016-12-18 23:25:30.239+0000 INFO  Mounted REST API at: /db/manage
2016-12-18 23:25:31.331+0000 INFO  Remote interface available at http://localhost:7474/

Neo4j 3.1 Embedded Graph Database


This section details several ways to get an embedded version of Neo4j running in a spring container.

https://neo4j.com/docs/java-reference/current/

Results

Amazon Elastic Beanstalk deployment to t2.tiny 2.3.1

/biometric-nbi/FrontController?action=graph

SLF4J: Found binding in [jar:file:/var/lib/tomcat8/webapps/ROOT/WEB-INF/lib/neo4j-security-enterprise-3.1.0-M13-beta3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
01-Dec-2016 04:15:23.925 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.0.36
01-Dec-2016 04:15:23.928 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.4.23-31.54.amzn1.x86_64
01-Dec-2016 04:15:23.928 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.24.amzn1.x86_64/jre
01-Dec-2016 04:15:23.929 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_101-b13
01-Dec-2016 04:15:42.683 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 18422 ms
register: org.obrienlabs.nbi.graph.service.HaMonitor@b37b7a9
processAction: graph
<state>Node[0]+0:0</state>
processAction: graph
<state>Node[1]+1:1</state>

pom.xml

      <neo4j.version>3.1.0-M13-beta3</neo4j.version>
  </properties>
  <dependencies>
      <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-cypher</artifactId>
        <version>${neo4j.version}</version>
    </dependency> 
    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-enterprise</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-server</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-ha</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-management</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-server-enterprise</artifactId>
        <version>${neo4j.version}</version>

    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-bolt</artifactId>
        <version>${neo4j.version}</version>

    </dependency> 

spring.xml (embedded)


    <context:annotation-config />
    <context:spring-configured />
    <!-- rest annotations -->
    <mvc:annotation-driven />
    
      <context:component-scan base-package="org.obrienlabs.nbi.graph.service" />

    <!-- in cases where the DAO's are in a separate jar - list them -->
    <bean id="daoFacade" class="org.obrienlabs.nbi.graph.service.ApplicationService"/>
    <bean id="IGraphDatabaseService" class="org.obrienlabs.nbi.graph.service.GraphDatabaseServiceImpl"/>

  
    <util:map id="config">
        <entry key="enable_remote_shell" value="true"/>
        <entry key="ha.server_id" value="1"/>
        <entry key="ha.initial_hosts" value="127.0.0.1:5001"/>
    </util:map>
    
    <bean id="haMonitor" class="com.obrienlabs.graph.service.HaMonitor"/>
    <bean id="graphDbFactory" class="com.obrienlabs.graph.service.ExtendedHighlyAvailableGraphDatabaseFactory">
        <constructor-arg ref="haMonitor" />
    </bean>
    
    <bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newEmbeddedDatabaseBuilder">
        <constructor-arg value="neo4j"/>
    </bean>
    <bean id="graphDbBuilderFinal" factory-bean="graphDbBuilder" factory-method="setConfig">
        <constructor-arg ref="config"/>
    </bean>
    <bean id="graphDatabaseService" factory-bean="graphDbBuilderFinal" factory-method="newGraphDatabase" 
    destroy-method="shutdown" />

servlet

    @Autowired
    @Qualifier("daoFacade")

    private ApplicationServiceLocal service;

spring bean hierarchy


public interface ApplicationServiceLocal {
String getGraph();
}

@Service("daoFacade")
public class ApplicationService implements ApplicationServiceLocal {
@Inject

private IGraphDatabaseService graphDatabaseService;
}
public interface IGraphDatabaseService {
Node createNode();
Node getNodeById( long id);


}
@Named("graphDatabaseService")
public class GraphDatabaseServiceImpl implements IGraphDatabaseService {
@Inject 
GraphDatabaseService databaseService;

@Override
public Node createNode() {
Node node = null;
      try ( org.neo4j.graphdb.Transaction tx = databaseService.beginTx() )  { 
            node =  databaseService.createNode();
            node.setProperty( "message", " node1 " );
            tx.success();
        }
return node;

}


HA listener override

We are currently running the older paxos not the raft based causal cluster HA mode

2016-12-19 00:54:47.703+0000 INFO  [o.n.c.m.p.PaxosClusterMemberAvailability] Listening at:cluster://127.0.0.1:5001

package org.obrienlabs.nbi.graph.service;

import org.neo4j.kernel.ha.HighlyAvailableGraphDatabase;
import org.neo4j.kernel.ha.cluster.HighAvailabilityMemberChangeEvent;
import org.neo4j.kernel.ha.cluster.HighAvailabilityMemberListener;

public class HaMonitor implements HighAvailabilityMemberListener {

    protected HighlyAvailableGraphDatabase db;
    
    public void setDb(HighlyAvailableGraphDatabase aInDb) {
        db = aInDb;
    }
    
@Override
public void masterIsElected(HighAvailabilityMemberChangeEvent event) {
System.out.println("MasterIsElected");
}

@Override
public void masterIsAvailable(HighAvailabilityMemberChangeEvent event) {
System.out.println("masterIsAvailable");
}

@Override
public void slaveIsAvailable(HighAvailabilityMemberChangeEvent event) {
System.out.println("slaveIsAvailabl");
}

@Override
public void instanceStops(HighAvailabilityMemberChangeEvent event) {
System.out.println("instanceStops");
}

@Override
public void instanceDetached(HighAvailabilityMemberChangeEvent event) {
System.out.println("instanceDetached");
}

}

package org.obrienlabs.nbi.graph.service;

import java.io.File;
import java.util.Map;

import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.factory.GraphDatabaseBuilder;
import org.neo4j.graphdb.factory.GraphDatabaseFactoryState;

import org.neo4j.graphdb.factory.GraphDatabaseSettings;
import org.neo4j.graphdb.factory.HighlyAvailableGraphDatabaseFactory;
import org.neo4j.kernel.ha.HighlyAvailableGraphDatabase;
import org.neo4j.kernel.ha.cluster.HighAvailabilityMemberStateMachine;

public class ExtendedHighlyAvailableGraphDatabaseFactory extends HighlyAvailableGraphDatabaseFactory {

    HaMonitor haMonitor;
        GraphDatabaseSettings.BoltConnector bolt = GraphDatabaseSettings.boltConnector( "0" );
    public ExtendedHighlyAvailableGraphDatabaseFactory(HaMonitor aInHaMonitor)  {
        haMonitor = aInHaMonitor;
    }

    @Override
     protected GraphDatabaseBuilder.DatabaseCreator createDatabaseCreator(
                final File storeDir, final GraphDatabaseFactoryState state)  {
            return new GraphDatabaseBuilder.DatabaseCreator() {

                @Override
                public GraphDatabaseService newDatabase( final Map<String, String> config ) {
                    config.put( "unsupported.dbms.ephemeral", "false" );
                    // enable bolt to enable 7474 http browser
                    config.put( bolt.type.name(), "BOLT" );
                    config.put( bolt.enabled.name(), "true" );
                    config.put( bolt.address.name(), "localhost:7687" );
                    
                    HighlyAvailableGraphDatabase haDB = 
                    new HighlyAvailableGraphDatabase( storeDir, config, state.databaseDependencies());
                    haMonitor.setDb(haDB);
                    HighAvailabilityMemberStateMachine memberStateMachine = 
                    haDB.getDependencyResolver().resolveDependency(HighAvailabilityMemberStateMachine.class);
                    if ( memberStateMachine != null ) {
                        memberStateMachine.addHighAvailabilityMemberListener(haMonitor);
                        System.out.println("register: " +  haMonitor);
                    }
                    return haDB;
                }
            };
        }

}


Issue: constructor arg takes a file not a string anymore
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'graphDbBuilder' defined in ServletContext resource [/WEB-INF/spring.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [java.io.File]: Could not convert factory method argument value of type [java.lang.String] to required type [java.io.File]: Failed to convert value of type [java.lang.String] to required type [java.io.File]; nested exception is java.lang.IllegalArgumentException: The resource path [/../work/graph.db] has been normalized to [null] which is not valid

in
    <bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newEmbeddedDatabaseBuilder">
    <!-- bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newHighlyAvailableDatabaseBuilder"-->
        <constructor-arg value="neo4j"/>

    </bean>

This works

<constructor-arg value="neo4j"/>
This does not anymore - we need to pass in a ${} variable via spring

<constructor-arg value="../neo4j"/>

https://neo4j.com/docs/java-reference/current/#tutorial-traversal


Getting the Neo4j 7474 http browser working on an Embedded graph.db

The 7474 browser no longer directly works out of the box on Neo4j 3.x as it did in 2.3.  Partially because of the introduction of bolt.
Going through the following workarounds to get it working again

Open a Bolt connector to your embedded instance to get GUI administration and other benefits
section 2.14 of



2016-12-18 03:37:42.112+0000 INFO  [o.n.k.i.DiagnosticsManager] com.sun.management.jmxremote.port = 8086
2016-12-18 03:37:42.490+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt Server extension loaded.

2016-12-18 03:37:42.490+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt enabled on localhost:7687.



It turns out with a bit of help from the following post we can still bring up a wrapping NeoServer around our embedded mode Neo4j 3.1.0 server. This code however is not integrated yet with our HA listener overrides on the constructor tree we bootstrap in Spring - so is a POC. 

With this NeoServer wrapper code - we don’t need the secondary workaround where we bring up a 2nd Neo4j server mode instance, bring up its’ 7474 browser and modify the bolt address (default 7687) in the config to point to the exposed (7688) embedded mode graph.db - without bringing either Neo4j server down as long as they point to different graph.db instances - in effect sharing the 7474 browser as a bolt client.

Disclaimer: I believe a proper architecture involves a clustered set of server mode bolt enabled Neo4j servers.
This undocumented/internal API is used as a temporary workaround to get the neo4j browser up on an embedded Neo4j server in 3.1.0 as it used to work under the WrappingNeoServerBootstrapper in 2.3 so we can continue to use in-jvm Dijkstra algos until we are able to rework/benchmark the same or greater traversal speed via a bolt enabled cluster.

FrontController/spring service bean starts graph
```
2016-12-20 01:28:04.186+0000 INFO  Starting...
2016-12-20 01:28:04.533+0000 INFO  Bolt enabled on localhost:7687.
2016-12-20 01:28:04.534+0000 INFO  Initiating metrics...
2016-12-20 01:28:06.091+0000 INFO  Started.
2016-12-20 01:28:07.009+0000 INFO  Remote interface available at http://localhost:7575/
<state>community single [/ec2-user/graph2.db] : community single [/ec2-user/graph2.db/data/databases/graph.db]</state>
```

after opening browser

```
    // http://stackoverflow.com/questions/30074232/replacement-for-deprecated-wrappingneoserverbootstrapper
    // Warning: internal/unsupported API - NeoServer wrappers are usually only run in  server mode - not embedded
        ServerBootstrapper serverBootstrapper = new EnterpriseBootstrapper();//CommunityBootstrapper();
        NeoServer directNeoServer = serverBootstrapper.getServer();
        int i = serverBootstrapper.start(new File("/ec2-user"), // will resolve to /ec2-user/data/databases/graph.db 
        Optional.of(new File("/Users/michaeal/Documents/Neo4j/.neo4j.conf")), //  stub only
            Pair.of("dbms.connector.http.address","0.0.0.0:7575"),
            Pair.of("dbms.connector.http.enabled","true" ),
            Pair.of("dbms.connector.http.type", "HTTP" ),
            Pair.of("dbms.connector.http.tls_level", "DISABLED")
            // BOLT not advised if we run an embedded Neo4j DB inside a NeoServer wrapper
            // BOLT should be enabled if you run a standalone server's browser against this embedded bolt port
            // WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver. Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavailable, using the wrong connection URL or temporary network problems. If you have enabled encryption, ensure your browser is configured to trust the certificate Neo4j is configured to use. WebSocket `readyState` is: 3
             ,Pair.of("dbms.connector.bolt.address","0.0.0.0:7688"),
            Pair.of("dbms.connector.bolt.enabled","true" ),
            Pair.of("dbms.connector.bolt.type", "HTTP" ),
            Pair.of("dbms.connector.bolt.tls_level", "DISABLED")           
        );
        NeoServer neoServer = serverBootstrapper.getServer();
        GraphDatabaseService graph = neoServer.getDatabase().getGraph();  
```


Debugging we see that we get the proper Jetty server available usually only to server mode Neo4j not embedded mode.
```
AbstractNeoServer.<clinit>() line: 114
EnterpriseBootstrapper.createNeoServer(Config, GraphDatabaseDependencies, LogProvider) line: 45
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 90
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

CommunityNeoServer.<clinit>() line: 54
EnterpriseBootstrapper.createNeoServer(Config, GraphDatabaseDependencies, LogProvider) line: 45
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 90
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

Config.get(Setting<T>) line: 137
EnterpriseNeoServer(AbstractNeoServer).<init>(Config, Database$Factory, GraphDatabaseFacadeFactory$Dependencies, LogProvider) line: 159
EnterpriseNeoServer(CommunityNeoServer).<init>(Config, Database$Factory, GraphDatabaseFacadeFactory$Dependencies, LogProvider) line: 69
EnterpriseNeoServer.<init>(Config, GraphDatabaseFacadeFactory$Dependencies, LogProvider) line: 105
EnterpriseBootstrapper.createNeoServer(Config, GraphDatabaseDependencies, LogProvider) line: 45
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 90
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

Jetty9WebServer.<init>(LogProvider, Config) line: 135
EnterpriseNeoServer(CommunityNeoServer).createWebServer() line: 89
EnterpriseNeoServer.createWebServer() line: 131
EnterpriseNeoServer(AbstractNeoServer).init() line: 181
EnterpriseNeoServer(AbstractNeoServer).start() line: 196
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 91
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

GraphDatabaseFacadeFactory.initFacade(File, Map<String,String>, Dependencies, GraphDatabaseFacade) line: 141
EnterpriseGraphDatabase.<init>(File, Map<String,String>, Dependencies) line: 36
EnterpriseNeoServer.lambda$static$1(Config, GraphDatabaseFacadeFactory$Dependencies) line: 90
1274309106.newGraphDatabase(Config, GraphDatabaseFacadeFactory$Dependencies) line: not available
LifecycleManagingDatabase.start() line: 89
LifeSupport$LifecycleInstance.start() line: 433
LifeSupport.start() line: 107
EnterpriseNeoServer(AbstractNeoServer).start() line: 199
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 91
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

Jetty9WebServer.loadJAXRSResource(SessionManager, String, JaxRsServletHolderFactory) line: 536
Jetty9WebServer.loadJAXRSClasses(SessionManager, String) line: 522
Jetty9WebServer.loadAllMounts() line: 409
Jetty9WebServer.start() line: 171
EnterpriseNeoServer(AbstractNeoServer).startWebServer() line: 326
EnterpriseNeoServer(AbstractNeoServer).start() line: 218
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 91
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

serverBootstrapper EnterpriseBootstrapper  (id=299)
dependencies GraphDatabaseDependencies  (id=347)
log FormattedLog  (id=573)
server EnterpriseNeoServer  (id=357)
serverAddress "0.0.0.0:7575" (id=575)
shutdownHook ServerBootstrapper$1  (id=577)

neoServer EnterpriseNeoServer  (id=357)
….
webServer Jetty9WebServer  (id=406)

from NeoServer
graph EnterpriseGraphDatabase  (id=417)
contextFactory Neo4jTransactionalContextFactory  (id=1119)
defaultTransactionTimeout 0
indexManager Suppliers$1  (id=1121)
nodeActions StandardNodeActions  (id=1123)
relActions StandardRelationshipActions  (id=1128)
schema SchemaImpl  (id=1132)
spi ClassicCoreSPI  (id=425)

Without NeoServer
graphDb GraphDatabaseFacade  (id=150)
contextFactory Neo4jTransactionalContextFactory  (id=1182)
defaultTransactionTimeout 0
indexManager Suppliers$1  (id=1183)
nodeActions StandardNodeActions  (id=1184)
relActions StandardRelationshipActions  (id=1208)
schema SchemaImpl  (id=1219)
spi ClassicCoreSPI  (id=160)
```


```
2016-12-20 02:15:02.825+0000 INFO  Starting...
2016-12-20 02:15:03.304+0000 INFO  Bolt enabled on 0.0.0.0:7688.
2016-12-20 02:15:03.316+0000 INFO  Initiating metrics...
2016-12-20 02:15:06.232+0000 INFO  Started.
2016-12-20 02:15:06.412+0000 INFO  Mounted REST API at: /db/manage
2016-12-20 02:15:07.103+0000 INFO  Remote interface available at http://localhost:7575/
<state> : enterprise single [/ec2-user/data/databases/graph.db]</state>
```

settings and logs
```
cat /ec2-user/logs/debug.log
2016-12-20 02:15:03.225+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.enabled=true
2016-12-20 02:15:03.225+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.security.auth_enabled=true
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.type=HTTP
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.directories.neo4j_home=/ec2-user
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.directories.import=import
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.tls_level=DISABLED
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.type=HTTP
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.logs.http.enabled=false
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.edition=enterprise
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.address=0.0.0.0:7688
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.address=0.0.0.0:7575
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.enabled=true
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.tls_level=DISABLED
2016-12-20 02:15:03.304+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt Server extension loaded.
2016-12-20 02:15:03.304+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt enabled on 0.0.0.0:7688.
2016-12-20 02:15:04.687+0000 INFO  [o.n.k.i.DiagnosticsManager] Kernel version: 3.1.0,16a782b42d76ca37db72958eb2565cf6aa671a29
```

These errors are non-fatal related to JMX and can be fixed
```
2016-12-20 02:17:30.759+0000 ERROR [o.n.b.v.r.ErrorReporter] Client triggered an unexpected error [UnknownError]: Exception invoking method writeBufSize, reference 59659a03-461e-4532-9b98-7afe3a2e5704. Exception invoking method writeBufSize
javax.management.RuntimeOperationsException: Exception invoking method writeBufSize
at org.apache.tomcat.util.modeler.BaseModelMBean.getAttribute(BaseModelMBean.java:196)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at org.neo4j.kernel.builtinprocs.JmxQueryProcedure.toNeo4jValue(JmxQueryProcedure.java:131)
Caused by: java.lang.NullPointerException
at org.apache.tomcat.util.net.SocketProperties.getTxBufSize(SocketProperties.java:300)
```


7575 http and 7688 bolt ports
```
obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7575
tcp46      0      0  *.7575                 *.*                    LISTEN      131072 131072  49013      0
tcp4       0      0  127.0.0.1.7575         127.0.0.1.60685        TIME_WAIT   407296 146988  49013      0
tcp6       0      0  ::1.7575               ::1.60699              TIME_WAIT   407284 146808  49013      0
tcp6       0      0  ::1.7575               ::1.60700              TIME_WAIT   407284 146808  49013      0
obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7688
tcp6       0      0  ::1.7688               ::1.60704              ESTABLISHED 406582 146808  49013      0
tcp6       0      0  ::1.60704              ::1.7688               ESTABLISHED 398196 146808  48165      0
tcp6       0      0  ::1.7688               ::1.60702              ESTABLISHED 406570 146808  49013      0
tcp6       0      0  ::1.60702              ::1.7688               ESTABLISHED 398185 146808  48165      0
tcp6       0      0  ::1.7688               ::1.60701              ESTABLISHED 407255 146808  49013      0
tcp6       0      0  ::1.60701              ::1.7688               ESTABLISHED 407628 146808  48165      0
tcp46      0      0  *.7688                 *.*                    LISTEN      131072 131072  49013      0
obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 8080
tcp4       0      0  127.0.0.1.8080         127.0.0.1.60584        FIN_WAIT_2  408104 146988  49013      0
tcp4     994      0  127.0.0.1.60584        127.0.0.1.8080         CLOSE_WAIT  408128 146988  42992      0
tcp46      0      0  *.8080                 *.*                    LISTEN      131072 131072  49013      0
udp4       0      0  *.*                    *.*                                196724   9216  38080      0
```


This code however is not integrated yet with our HA listener overrides on the constructor tree we bootstrap in Spring - the following scrubbed/distilled mirror POC of our corporate production code will be adjusted in the next post.

```
public class ExtendedHighlyAvailableGraphDatabaseFactory extends HighlyAvailableGraphDatabaseFactory {
    private HaMonitor haMonitor;
    private GraphDatabaseSettings.BoltConnector bolt = GraphDatabaseSettings.boltConnector( "0" );
    @Override
     protected GraphDatabaseBuilder.DatabaseCreator createDatabaseCreator(
                final File storeDir, final GraphDatabaseFactoryState state)  {
            return new GraphDatabaseBuilder.DatabaseCreator() {
                @Override
                public GraphDatabaseService newDatabase( final Map<String, String> config ) {
                    HighlyAvailableGraphDatabase haDB = 
                    new HighlyAvailableGraphDatabase( storeDir, config, state.databaseDependencies());
                    haMonitor.setDb(haDB);
                    HighAvailabilityMemberStateMachine memberStateMachine = 
                    haDB.getDependencyResolver().resolveDependency(HighAvailabilityMemberStateMachine.class);
                    if ( memberStateMachine != null ) {
                        memberStateMachine.addHighAvailabilityMemberListener(haMonitor);
                    }
                    return haDB;
                } };}

```

Neo4j 3.1.0 HA (paxos) Embedded mode HighlyAvailableGraphDatabase wrapped with an EnterpriseBootstrapper NeoServer

see

In this code we get everything (except causal cluster using the raft protocol).

With a few modifications I was able to wrap a HighlyAvailableGraphDatabase with a EnterpriseBootstrapper.

There are some non-fatal exceptions around JMX reporting in debug.log likely related to my 8.0.28 tomcat version but the graph db running embedded in Tomcat is OK.

        2016-12-21 16:20:00.574+0000 INFO  Bolt enabled on 0.0.0.0:7688.
        2016-12-21 16:20:09.554+0000 INFO  Attempting to join cluster of [127.0.0.1:5001]
        2016-12-21 16:20:11.566+0000 INFO  Creating new cluster with name [neo4j.ha]...
        2016-12-21 16:20:11.607+0000 INFO  Instance 1 (this server)  entered the cluster
        2016-12-21 16:20:12.164+0000 INFO  I am 1, moving to master
        2016-12-21 16:20:12.293+0000 INFO  Instance 1 (this server)  was elected as coordinator
        2016-12-21 16:20:12.462+0000 INFO  I am 1, successfully moved to master
        2016-12-21 16:20:12.513+0000 INFO  Instance 1 (this server)  is available as master at ha://127.0.0.1:6001?serverId=1 with StoreId{creationTime=1482199697648, randomId=7800059877674392627, storeVersion=15531981201765894, upgradeTime=1482199697648, upgradeId=1}
        2016-12-21 16:20:14.495+0000 INFO  Database available for write transactions
        2016-12-21 16:20:31.917+0000 INFO  Mounted REST API at: /db/manage
        2016-12-21 16:20:53.264+0000 INFO  Remote interface available at http://localhost:7575/
        register: org.obrienlabs.nbi.graph.service.HaMonitor@1c0f80c9


        public class ExtendedHighlyAvailableGraphDatabaseFactory extends HighlyAvailableGraphDatabaseFactory {
            private HaMonitor haMonitor;    
            public GraphDatabaseService getEnterpriseGraphDatabase(final File _storeDir, final GraphDatabaseFactoryState _state,  
            final Map<String, String> _config) {
                String graphPath = "/ec2-user/graph2.db" ;
            // http://stackoverflow.com/questions/30074232/replacement-for-deprecated-wrappingneoserverbootstrapper
            // Warning: internal/unsupported API - NeoServer wrappers are usually only run in  server mode - not embedded
                EnterpriseBootstrapper serverBootstrapper = new EnterpriseBootstrapper();//CommunityBootstrapper();
                int i = serverBootstrapper.start(new File("/ec2-user"), // will resolve to /ec2-user/data/databases/graph.db 
                Optional.of(new File("/some-path/Neo4j/.neo4j.conf")), //  stub only
    Pair.of("dbms.connector.http.address","0.0.0.0:7575"),
                    Pair.of("dbms.connector.http.enabled","true" ),
                    Pair.of("dbms.connector.http.type", "HTTP" ),
                    Pair.of("dbms.connector.http.tls_level", "DISABLED"),
         Pair.of("dbms.connector.bolt.address","0.0.0.0:7688"),
                    Pair.of("dbms.connector.bolt.enabled","true" ),
                    Pair.of("dbms.connector.bolt.type", "HTTP" ),
                    Pair.of("dbms.connector.bolt.tls_level", "DISABLED") ,
                    Pair.of("ha.initial_hosts", "127.0.0.1:5001"),
                       Pair.of("ha.server_id", "1"),
                    Pair.of("dbms.mode", EnterpriseNeoServer.Mode.HA.toString()));
                NeoServer neoServer = serverBootstrapper.getServer();
                GraphDatabaseService graph = neoServer.getDatabase().getGraph(); 
                return graph;
            }
        
            @Override
            protected GraphDatabaseBuilder.DatabaseCreator createDatabaseCreator(
                        final File storeDir, final GraphDatabaseFactoryState state)  {
                    return new GraphDatabaseBuilder.DatabaseCreator() {
                        @Override
                        public GraphDatabaseService newDatabase( final Map<String, String> config ) {
                            // section 2.14 of http://neo4j.com/docs/java-reference/current/#tutorials-java-embedded       
                            /*HighlyAvailableGraphDatabase haDB = 
                            new HighlyAvailableGraphDatabase( storeDir, config, state.databaseDependencies());*/
                            HighlyAvailableGraphDatabase haDB =(HighlyAvailableGraphDatabase) getEnterpriseGraphDatabase( storeDir, state,  config);
                            haMonitor.setDb(haDB);
                            // Add the HA monitor listener
                            HighAvailabilityMemberStateMachine memberStateMachine = 
                            haDB.getDependencyResolver().resolveDependency(HighAvailabilityMemberStateMachine.class);
                            if ( memberStateMachine != null ) {
                                memberStateMachine.addHighAvailabilityMemberListener(haMonitor);
                            }
                            return haDB;
                        }};}

Transactions go through

    2016-12-21 20:51:07.478+0000 INFO  [o.n.k.i.s.c.CountsTracker] About to rotate counts store at transaction 9 to [/ec2-user/data/databases/graph.db/neostore.counts.db.b], from [/ec2-user/data/databases/graph.db/neostore.counts.db.a].
    2016-12-21 20:51:07.480+0000 INFO  [o.n.k.i.s.c.CountsTracker] Successfully rotated counts store at transaction 9 to [/ec2-user/data/databases/graph.db/neostore.counts.db.b], from [/ec2-user/data/databases/graph.db/neostore.counts.db.a].
    2016-12-21 20:51:07.483+0000 INFO  [o.n.k.i.t.l.c.CheckPointerImpl] Check Pointing triggered by scheduler for time threshold [9]:  Store flush completed

Settings from the mbean in jconsole are

    2016-12-21 20:55:46.503+0000 INFO  [o.n.k.i.DiagnosticsManager] --- REQUESTED diagnostics START ---
    2016-12-21 20:55:46.503+0000 INFO  [o.n.k.i.DiagnosticsManager] Graph Database: enterprise ha StoreId{creationTime=1482199697648, randomId=7800059877674392627, storeVersion=15531981201765894, upgradeTime=1482199697648, upgradeId=1}
    2016-12-21 20:55:46.503+0000 INFO  [o.n.k.i.DiagnosticsManager] Kernel version: 3.1.0,16a782b42d76ca37db72958eb2565cf6aa671a29
    2016-12-21 20:55:46.503+0000 INFO  [o.n.k.i.DiagnosticsManager] Neo4j Kernel properties:
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.enabled=true
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.mode=HA
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.block_size.labels=56
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.security.auth_enabled=true
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.type=HTTP
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.directories.neo4j_home=/ec2-user
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.directories.import=import
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.ephemeral=false
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.tls_level=DISABLED
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.type=HTTP
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] ha.server_id=1
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.block_size.strings=120
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.logs.http.enabled=false
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.block_size.array_properties=120
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] ha.initial_hosts=127.0.0.1:5001
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.allow_format_migration=false
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.edition=enterprise
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.address=0.0.0.0:7688
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.address=0.0.0.0:7575
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.enabled=true
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.tls_level=DISABLED

7575(jetty server) and 7688 (bolt) ports are open

    obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7575
    tcp46      0      0  *.7575                 *.*                    LISTEN      131072 131072  49013      0
    tcp4       0      0  127.0.0.1.7575         127.0.0.1.60685        TIME_WAIT   407296 146988  49013      0
    tcp6       0      0  ::1.7575               ::1.60699              TIME_WAIT   407284 146808  49013      0
    tcp6       0      0  ::1.7575               ::1.60700              TIME_WAIT   407284 146808  49013      0
    obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7688
    tcp6       0      0  ::1.7688               ::1.60704              ESTABLISHED 406582 146808  49013      0
    tcp6       0      0  ::1.60704              ::1.7688               ESTABLISHED 398196 146808  48165      0
    tcp6       0      0  ::1.7688               ::1.60702              ESTABLISHED 406570 146808  49013      0
    tcp6       0      0  ::1.60702              ::1.7688               ESTABLISHED 398185 146808  48165      0
    tcp6       0      0  ::1.7688               ::1.60701              ESTABLISHED 407255 146808  49013      0
    tcp6       0      0  ::1.60701              ::1.7688               ESTABLISHED 407628 146808  48165      0
    tcp46      0      0  *.7688                 *.*                    LISTEN      131072 131072  49013      0
    obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 8080
    tcp4       0      0  127.0.0.1.8080         127.0.0.1.60584        FIN_WAIT_2  408104 146988  49013      0
    tcp4     994      0  127.0.0.1.60584        127.0.0.1.8080         CLOSE_WAIT  408128 146988  42992      0
    tcp46      0      0  *.8080                 *.*                    LISTEN      131072 131072  49013      0
    udp4       0      0  *.*                    *.*                                196724   9216  38080      0

Disclaimer: I believe a proper architecture involves a clustered set of server mode bolt enabled Neo4j servers using causal clustering under the raft protocol.
This undocumented/internal API is used as a temporary workaround to get the neo4j browser up on an embedded Neo4j server in 3.1.0 as it used to work under the WrappingNeoServerBootstrapper in 2.3.  Use of the embedded server for faster traversals was the use case - performance must be reevaluated to see if a bolt based architecture is more optimal for the traversal API.
/michael

Neo4j 3.1 Technology

https://neo4j.com/blog/neo4j-3-1-beta-release/?ref=home
- causal clustering (with read replicas and synced read/write clusters)
- bolt can drive load balancing (with read/write splitting)
- 4 security roles with list and terminate control
- node and id reallocation from deleted records (uuid reuse?)
- max query duration

Test Code

Run the following to bring up a fully populated hypercube and test the memory limits of your server and browser
package org.obrienlabs.nbi.graph.test;

import static org.neo4j.driver.v1.Values.parameters;

import org.neo4j.driver.v1.AuthTokens;
import org.neo4j.driver.v1.Driver;
import org.neo4j.driver.v1.GraphDatabase;
import org.neo4j.driver.v1.Session;
import org.neo4j.driver.v1.StatementResult;
import org.neo4j.driver.v1.Transaction;

public class BoltUnitTest {

public static long linkStep = 0;
public static long linkCount = 0;
public static long pk = 0;
public static long pair = 0;
public static long linkMax = 0;
    private static final String url = "bolt://192.168.0.26:7687";
    private static final String username =  "neo4j";
    private static final String password = "mainstreet"; 
private void link(String label, long pair0, long pair1, long pk0, long pk1, Session session) {
         try (Transaction writeTransaction = session.beginTransaction()) {
        writeTransaction.success();
        writeTransaction.run( 
        "MATCH (a:Node"+ pair0 + "),(b:Node"+ pair1 + ") WHERE a.name = {p0} AND b.name = {p1} CREATE (a)-[r:" + label + "]->(b)",
                        parameters( "p0", pk0, "p1", pk1));
        StatementResult edge1  = writeTransaction.run( 
        "MATCH (a:Node"+ pair0 + "),(b:Node"+ pair1 + ") WHERE a.name = {p0} AND b.name = {p1} CREATE (a)-[r:" + label + "]->(b) RETURN r",
                        parameters( "p0", pk1, "p1", pk0));
        writeTransaction.success();
        linkStep--;
        linkCount++;
        if(linkStep < 0) {
        linkStep = 16;
        System.out.println("link: " + pk0 + "-" + pk1 + " : " + linkCount + " of " + linkMax);
        }
        }
}

private void create(long pk0, long pair, Session session) {
        try (Transaction writeTransaction = session.beginTransaction()) {
        StatementResult left  = writeTransaction.run( "CREATE (a:Node" + pair + " {name: {pk}}) return a",
                    parameters( "pk", pk));
        pk++;
        writeTransaction.success();
        }
}

private void createL(int power, long id, Session session) {
// base
if(power == 0) {
        create(id,  0, session);
        create(id + 1, 0,  session);
        link("L0", 0, 0, id ,id + 1, session);
        //pair++;
        return;
}
// recursive
int ext = 1 << power;
createL(power - 1, id, session);
createL(power - 1, id + ext, session);
for(int i=0; i<ext; i++) {
link("L" + power, 0, 0, id + i, id + ext + i, session);
}
}
// match(n:Node0) where n.name=0 return(n);
public void populateHypercube() throws Exception {
  long count = 0;
  int lastPower = 16;
  int power = 16;
  linkMax = (1 << power) * (power + 1);
        Driver driver = GraphDatabase.driver(url,  AuthTokens.basic(username, password)); 
        try (Session session = driver.session()) {
            try (Transaction writeTransaction = session.beginTransaction()) {
            for(int i=0;i<lastPower + 1;i++) {
            writeTransaction.run(" match (:Node0)-[r:L" + i + "]->(:Node0) delete r");
            }
        writeTransaction.run(" match(n) delete(n)");
        writeTransaction.success();
            }
        createL(power, count, session);
        }
        driver.close();
    }
 
// https://github.com/neo4j/neo4j-java-driver/blob/1.1/examples/src/main/java/org/neo4j/docs/driver/MinimalWorkingExample.java
public static void main(String[] args) {
BoltUnitTest test = new BoltUnitTest();
try {
test.populateHypercube();
} catch(Exception e) {
e.printStackTrace();
}
}

}

Troubleshooting

./neo4j start does not work
Fix: make sure your java 8 home is set like

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home

Links



https://neo4j.com/docs/java-reference/current/javadocs/deprecated-list.html
references deprecated wrapper



Total Pageviews

Followers