Thursday, December 1, 2016

Running High Memory Java Applications on Amazon EC2 2048Gb 128 vCPU Memory Optimized x1.32xlarge instances

Amazon has a huge instance that we can use to run massive memory models.
Here I am creating hypercube (see CM-1 from the 90's).  On a normal 64G system like a Haswell 5820K board fully populated with DDR4 we top out at 22 dimensions.


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852    3801596 2010941256         64      13396     238792
-/+ buffers/cache:    3549408 2011193444
Swap:            0          0          0



[ec2-user@ip-172-31-57-105 ~]$ java -version
java version "1.7.0_121"
OpenJDK Runtime Environment (amzn-2.6.8.1.69.amzn1-x86_64 u121-b00)
OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)
[ec2-user@ip-172-31-57-105 ~]$ sudo yum install java-1.8.0
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                                                                                                                | 2.1 kB     00:00     
amzn-updates/latest                                                                                                                             | 2.3 kB     00:00     
Resolving Dependencies
--> Running transaction check
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed
--> Processing Dependency: java-1.8.0-openjdk-headless = 1:1.8.0.111-1.b15.25.amzn1 for package: 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64
--> Running transaction check
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed
amzn-main/latest/filelists_db                                                                                                                   | 5.1 MB     00:00     
amzn-updates/latest/filelists_db                                                                                                                | 1.2 MB     00:00     
--> Processing Dependency: lksctp-tools for package: 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64
--> Running transaction check
---> Package lksctp-tools.x86_64 0:1.0.10-7.7.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================================================
 Package                                          Arch                        Version                                          Repository                         Size
=======================================================================================================================================================================
Installing:
 java-1.8.0-openjdk                               x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                      227 k
Installing for dependencies:
 java-1.8.0-openjdk-headless                      x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                       39 M
 lksctp-tools                                     x86_64                      1.0.10-7.7.amzn1                                 amzn-main                          89 k

Transaction Summary
=======================================================================================================================================================================
Install  1 Package (+2 Dependent packages)

Total download size: 39 M
Installed size: 102 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                                   | 227 kB     00:00     
(2/3): java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                          |  39 MB     00:00     
(3/3): lksctp-tools-1.0.10-7.7.amzn1.x86_64.rpm                                                                                                 |  89 kB     00:00     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                   48 MB/s |  39 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 
  Installing : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       2/3 
  Installing : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                3/3 
  Verifying  : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 
  Verifying  : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                2/3 
  Verifying  : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       3/3 

Installed:
  java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1                                                                                                                 

Dependency Installed:
  java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1                                 lksctp-tools.x86_64 0:1.0.10-7.7.amzn1                                

Complete!

[ec2-user@ip-172-31-57-105 ~]$ sudo alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
   2           /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

Enter to keep the current selection[+], or type selection number: 2
[ec2-user@ip-172-31-57-105 ~]$ java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)


[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 1
22 dimensional hypercube: 1480643986605
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
47345 time: Total Ram: 8450998272 free: 2292978440

[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852   20609684 1994133168         68      14064     406412
-/+ buffers/cache:   20189208 1994553644
Swap:            0          0          0

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 2 -server 
22 dimensional hypercube: 1480644388217
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
43511 time: Total Ram: 8432648192 free: 2267825616

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 
23 dimensional hypercube: 1480644473000
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.
140562 time: Total Ram: 18184929280 free: 5885736464

top - 02:09:40 up  2:05,  2 users,  load average: 39.26, 21.54, 10.94
Tasks: 730 total,   1 running, 729 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 19851880k used, 1994890972k free,    14168k buffers
Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
  7999 ec2-user  20   0 39.9g  15g  14m S 8058.5  0.8  96:23.57 java  

Tasks: 729 total,   2 running, 727 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.8%us,  0.4%sy,  0.0%ni, 36.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 32436576k used, 1982306276k free,    14304k buffers
Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
  8226 ec2-user  20   0 39.8g  27g  15m S 8098.1  1.4 255:48.30 java  

[vagrant@nsp-latest workspace]$ free
              total        used        free      shared  buff/cache   available
Mem:        7495248      263396     5030584       10720     2201268     6923444
Swap:       1048572      171480      877092

[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852   33580832 1981162020         68      14372     406416
-/+ buffers/cache:   33160044 1981582808

Swap:            0          0          0


running with extra memory


[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 24 1 
24 dimensional hypercube: 1480649540504
_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.
101381 time: Total Ram: 526849671168 free: 471873878864



[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 
23 dimensional hypercube: 1480649699358
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.

50312 time: Total Ram: 526849671168 free: 496612985560



Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 52853856k used, 1961888996k free,    18704k buffers
Swap:        0k total,        0k used,        0k free,   436572k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 43697 ec2-user  20   0  538g  46g  14m S 99.9  2.4   1:26.89 java    


[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 25 1 
25 dimensional hypercube: 1480649786242
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.

289783 time: Total Ram: 526849671168 free: 397656555744


[ec2-user@ip-172-31-57-105 ~]$ nohup java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1 &
[1] 44231
[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44231  43947 91 03:55 pts/0    00:00:05 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

[ec2-user@ip-172-31-57-105 ~]$ tail -f nohup.out

2 dimensional hypercube: 1480650669281
_created 4 bidirectional links between 4 routers in the 2 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
3 dimensional hypercube: 1480650669283
_created 12 bidirectional links between 8 routers in the 3 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
4 dimensional hypercube: 1480650669283
_created 32 bidirectional links between 16 routers in the 4 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
5 dimensional hypercube: 1480650669283
_created 80 bidirectional links between 32 routers in the 5 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
6 dimensional hypercube: 1480650669284
_created 192 bidirectional links between 64 routers in the 6 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
7 dimensional hypercube: 1480650669284
_created 448 bidirectional links between 128 routers in the 7 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
8 dimensional hypercube: 1480650669286
_created 1024 bidirectional links between 256 routers in the 8 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
9 dimensional hypercube: 1480650669287
_created 2304 bidirectional links between 512 routers in the 9 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
10 dimensional hypercube: 1480650669288
_created 5120 bidirectional links between 1024 routers in the 10 dimensional HyperCube.
3 time: Total Ram: 1843972800512 free: 1826792931288
11 dimensional hypercube: 1480650669291
_created 11264 bidirectional links between 2048 routers in the 11 dimensional HyperCube.
5 time: Total Ram: 1843972800512 free: 1826792931288
12 dimensional hypercube: 1480650669296
_created 24576 bidirectional links between 4096 routers in the 12 dimensional HyperCube.
10 time: Total Ram: 1843972800512 free: 1826792931288
13 dimensional hypercube: 1480650669306
_created 53248 bidirectional links between 8192 routers in the 13 dimensional HyperCube.
21 time: Total Ram: 1843972800512 free: 1826792931288
14 dimensional hypercube: 1480650669327
_created 114688 bidirectional links between 16384 routers in the 14 dimensional HyperCube.
38 time: Total Ram: 1843972800512 free: 1826792931288
15 dimensional hypercube: 1480650669365
_created 245760 bidirectional links between 32768 routers in the 15 dimensional HyperCube.
82 time: Total Ram: 1843972800512 free: 1826792931288
16 dimensional hypercube: 1480650669447
_created 524288 bidirectional links between 65536 routers in the 16 dimensional HyperCube.
174 time: Total Ram: 1843972800512 free: 1826792931288
17 dimensional hypercube: 1480650669621
_created 1114112 bidirectional links between 131072 routers in the 17 dimensional HyperCube.
384 time: Total Ram: 1843972800512 free: 1826792931288
18 dimensional hypercube: 1480650670005
_created 2359296 bidirectional links between 262144 routers in the 18 dimensional HyperCube.
923 time: Total Ram: 1843972800512 free: 1826792931288
19 dimensional hypercube: 1480650670928
_created 4980736 bidirectional links between 524288 routers in the 19 dimensional HyperCube.
1970 time: Total Ram: 1843972800512 free: 1826792931288
20 dimensional hypercube: 1480650672898
_created 10485760 bidirectional links between 1048576 routers in the 20 dimensional HyperCube.
4677 time: Total Ram: 1843972800512 free: 1826792931288
21 dimensional hypercube: 1480650677576
_created 22020096 bidirectional links between 2097152 routers in the 21 dimensional HyperCube.
9739 time: Total Ram: 1843972800512 free: 1818202996656
22 dimensional hypercube: 1480650687314
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
21156 time: Total Ram: 1843972800512 free: 1809613062048
23 dimensional hypercube: 1480650708470
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.
44953 time: Total Ram: 1843972800512 free: 1783843258176


Tasks: 743 total,   1 running, 742 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 55841328k used, 1958901524k free,    19244k buffers
Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                

 44386 ec2-user  20   0 1865g  49g  14m S 100.2  2.6   1:19.67 java

24 dimensional hypercube: 1480651245506
_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.
101306 time: Total Ram: 1843972800512 free: 1732303649872
25 dimensional hypercube: 1480651346812
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.
264376 time: Total Ram: 1843972800512 free: 1603454628832



Tasks: 742 total,   1 running, 741 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 278517036k used, 1736225816k free,    19408k buffers
Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
 44386 ec2-user  20   0 1865g 261g  15m S 100.2 13.6   8:45.96 java  


26 dimensional hypercube: 1480651611188
_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.

656891 time: Total Ram: 1843972800512 free: 1647312996576


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852  661273556 1353469296         72      19672     436804

-/+ buffers/cache:  660817080 1353925772

 44386 ec2-user  20   0 1865g 625g  15m S 99.9 32.6  82:37.39 java   


Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1088382060k used, 926360792k free,    19984k buffers
Swap:        0k total,        0k used,        0k free,   436804k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 44386 ec2-user  20   0 1870g 1.0t  15m S 100.2 53.7 313:16.41 java   



25 dimensional hypercube: 1480651346812
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.
264376 time: Total Ram: 1843972800512 free: 1603454628832
26 dimensional hypercube: 1480651611188
_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.
656891 time: Total Ram: 1843972800512 free: 1647312996576
27 dimensional hypercube: 1480652268079
_created 1811939328 bidirectional links between 134217728 routers in the 27 dimensional HyperCube.
1433987 time: Total Ram: 1843972800512 free: 1253174190840
28 dimensional hypercube: 1480653702066
_created 3758096384 bidirectional links between 268435456 routers in the 28 dimensional HyperCube.
4437315 time: Total Ram: 1843972800512 free: 1001914613768
29 dimensional hypercube: 1480658139381
[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852 1992699228   22043624         64      28708     437000
-/+ buffers/cache: 1992233520   22509332
Swap:            0          0          0
[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44386      1 99 03:59 ?        18-01:29:52 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

Tasks: 739 total,   1 running, 738 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1992699740k used, 22043112k free,    28736k buffers
Swap:        0k total,        0k used,        0k free,   437000k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
 44386 ec2-user  20   0 1898g 1.8t  15m S 8059.8 98.5  26052:50 java  


Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1734049420k used, 280693432k free,    32136k buffers
Swap:        0k total,        0k used,        0k free,   437292k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1888g 1.6t  14m S 8045.2 85.7   5836:37 java  


Tasks: 744 total,   1 running, 743 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1971207064k used, 43535788k free,    33936k buffers
Swap:        0k total,        0k used,        0k free,   437308k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1886g 1.8t  14m S 100.0 97.4  13311:12 java   


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached

Mem:    2014742852 1971206692   43536160         64      33944     437308

29 dimensional hypercube: 1480681783260
_created 7784628224 bidirectional links between 536870912 routers in the 29 dimensional HyperCube.

16316219 time: Total Ram: 1843972800512 free: 147852676280

10 min 26d 30g
25 min 27d 75g
75 min 28d 185g
225 min 29d 460g
800 min 30d 1300g

Wednesday, November 30, 2016

Density Functional Theory

Carbon Based Lifeforms "Only illogics can find, hidden flaws in a straight logic line.  Only erratics can recognize, errors in patterns of a perfect size."

Tuesday, November 8, 2016

Neo4j Graph Database


Neo4j is close to releasing the 3.1.0 version of their graph database.  This page details several ways to get an embedded version of neo4j running in a spring container

https://neo4j.com/docs/java-reference/current/

Results

Amazon Elastic Beanstalk deployment to t2.tiny 2.3.1

/biometric-nbi/FrontController?action=graph

SLF4J: Found binding in [jar:file:/var/lib/tomcat8/webapps/ROOT/WEB-INF/lib/neo4j-security-enterprise-3.1.0-M13-beta3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
01-Dec-2016 04:15:23.925 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.0.36
01-Dec-2016 04:15:23.928 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.4.23-31.54.amzn1.x86_64
01-Dec-2016 04:15:23.928 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.24.amzn1.x86_64/jre
01-Dec-2016 04:15:23.929 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_101-b13
01-Dec-2016 04:15:42.683 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 18422 ms
register: org.obrienlabs.nbi.graph.service.HaMonitor@b37b7a9
processAction: graph
<state>Node[0]+0:0</state>
processAction: graph
<state>Node[1]+1:1</state>

pom.xml

      <neo4j.version>3.1.0-M13-beta3</neo4j.version>
  </properties>
  <dependencies>
      <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-cypher</artifactId>
        <version>${neo4j.version}</version>
    </dependency> 
    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-enterprise</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-server</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-ha</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-management</artifactId>
        <version>${neo4j.version}</version>


    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-server-enterprise</artifactId>
        <version>${neo4j.version}</version>

    </dependency> 

spring.xml (embedded)


    <context:annotation-config />
    <context:spring-configured />
    <!-- rest annotations -->
    <mvc:annotation-driven />
    
      <context:component-scan base-package="org.obrienlabs.nbi.graph.service" />

    <!-- in cases where the DAO's are in a separate jar - list them -->
    <bean id="daoFacade" class="org.obrienlabs.nbi.graph.service.ApplicationService"/>
    <bean id="IGraphDatabaseService" class="org.obrienlabs.nbi.graph.service.GraphDatabaseServiceImpl"/>

  
    <util:map id="config">
        <entry key="enable_remote_shell" value="true"/>
        <!--entry key="org.neo4j.server.database.location" value="../work/graph.db"  /-->
        <entry key="ha.server_id" value="1"/>
        <entry key="ha.initial_hosts" value="127.0.0.1:5001"/>
    </util:map>
    
    <bean id="haMonitor" class="com.obrienlabs.graph.service.HaMonitor"/>
    <bean id="graphDbFactory" class="com.obrienlabs.graph.service.ExtendedHighlyAvailableGraphDatabaseFactory">
        <constructor-arg ref="haMonitor" />
    </bean>
    <!--bean id="graphDbFactory" class="org.neo4j.graphdb.factory.HighlyAvailableGraphDatabaseFactory"/-->
    
    <bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newEmbeddedDatabaseBuilder">
    <!--bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newHighlyAvailableDatabaseBuilder"-->
        <constructor-arg value="neo4j"/>
    </bean>
    <bean id="graphDbBuilderFinal" factory-bean="graphDbBuilder" factory-method="setConfig">
        <constructor-arg ref="config"/>
    </bean>
    <bean id="graphDatabaseService" factory-bean="graphDbBuilderFinal" factory-method="newGraphDatabase" 
    destroy-method="shutdown" />

servlet

    @Autowired
    @Qualifier("daoFacade")

    private ApplicationServiceLocal service;

spring bean hierarchy


public interface ApplicationServiceLocal {
String getGraph();
}

@Service("daoFacade")
public class ApplicationService implements ApplicationServiceLocal {
@Inject

private IGraphDatabaseService graphDatabaseService;
}
public interface IGraphDatabaseService {
Node createNode();
Node getNodeById( long id);


}
@Named("graphDatabaseService")
public class GraphDatabaseServiceImpl implements IGraphDatabaseService {
@Inject 
GraphDatabaseService databaseService;

@Override
public Node createNode() {
Node node = null;
      try ( org.neo4j.graphdb.Transaction tx = databaseService.beginTx() )  { 
            node =  databaseService.createNode();
            node.setProperty( "message", " node1 " );
            tx.success();
        }
return node;

}


HA listener override

package org.obrienlabs.nbi.graph.service;

import org.neo4j.kernel.ha.HighlyAvailableGraphDatabase;
import org.neo4j.kernel.ha.cluster.HighAvailabilityMemberChangeEvent;
import org.neo4j.kernel.ha.cluster.HighAvailabilityMemberListener;

public class HaMonitor implements HighAvailabilityMemberListener {

    protected HighlyAvailableGraphDatabase db;
    
    public void setDb(HighlyAvailableGraphDatabase aInDb) {
        db = aInDb;
    }
    
@Override
public void masterIsElected(HighAvailabilityMemberChangeEvent event) {
System.out.println("MasterIsElected");
}

@Override
public void masterIsAvailable(HighAvailabilityMemberChangeEvent event) {
System.out.println("masterIsAvailable");
}

@Override
public void slaveIsAvailable(HighAvailabilityMemberChangeEvent event) {
System.out.println("slaveIsAvailabl");
}

@Override
public void instanceStops(HighAvailabilityMemberChangeEvent event) {
System.out.println("instanceStops");
}

@Override
public void instanceDetached(HighAvailabilityMemberChangeEvent event) {
System.out.println("instanceDetached");
}

}

package org.obrienlabs.nbi.graph.service;

import java.io.File;
import java.util.Map;

import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.factory.GraphDatabaseBuilder;
import org.neo4j.graphdb.factory.GraphDatabaseFactoryState;
import org.neo4j.graphdb.factory.HighlyAvailableGraphDatabaseFactory;
import org.neo4j.kernel.ha.HighlyAvailableGraphDatabase;
import org.neo4j.kernel.ha.cluster.HighAvailabilityMemberStateMachine;

public class ExtendedHighlyAvailableGraphDatabaseFactory extends HighlyAvailableGraphDatabaseFactory {

    HaMonitor haMonitor;
    public ExtendedHighlyAvailableGraphDatabaseFactory(HaMonitor aInHaMonitor)  {
        haMonitor = aInHaMonitor;
    }

    @Override
     protected GraphDatabaseBuilder.DatabaseCreator createDatabaseCreator(
                final File storeDir, final GraphDatabaseFactoryState state)  {
            return new GraphDatabaseBuilder.DatabaseCreator() {

                @Override
                public GraphDatabaseService newDatabase( final Map<String, String> config ) {
                    config.put( "unsupported.dbms.ephemeral", "false" );
                    HighlyAvailableGraphDatabase haDB = 
                    new HighlyAvailableGraphDatabase( storeDir, config, state.databaseDependencies());
                    haMonitor.setDb(haDB);
                    HighAvailabilityMemberStateMachine memberStateMachine = 
                    haDB.getDependencyResolver().resolveDependency(HighAvailabilityMemberStateMachine.class);
                    if ( memberStateMachine != null ) {
                        memberStateMachine.addHighAvailabilityMemberListener(haMonitor);
                        System.out.println("register: " +  haMonitor);
                    }
                    return haDB;
                }
            };
        }

}


Issue: constructor arg takes a file not a string anymore
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'graphDbBuilder' defined in ServletContext resource [/WEB-INF/spring.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [java.io.File]: Could not convert factory method argument value of type [java.lang.String] to required type [java.io.File]: Failed to convert value of type [java.lang.String] to required type [java.io.File]; nested exception is java.lang.IllegalArgumentException: The resource path [/../work/graph.db] has been normalized to [null] which is not valid

in
    <bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newEmbeddedDatabaseBuilder">
    <!-- bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newHighlyAvailableDatabaseBuilder"-->
        <constructor-arg value="neo4j"/>

    </bean>

This works

<constructor-arg value="neo4j"/>
This does not anymore

<constructor-arg value="../neo4j"/>

https://neo4j.com/docs/java-reference/current/#tutorial-traversal

Neo4j 3.1 introduces
https://neo4j.com/blog/neo4j-3-1-beta-release/?ref=home
- causal clustering (with read replicas and synced read/write clusters)
- bolt can drive load balancing (with read/write splitting)
- 4 security roles with list and terminate control
- node and id reallocation from deleted records (uuid reuse?)
- max query duration

Troubleshooting
./neo4j start does not work
Fix: make sure your java 8 home is set like

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home

Links
https://neo4j.com/docs/java-reference/current/javadocs/deprecated-list.html
references deprecated wrapper



Total Pageviews

Followers