Forums  > Software  > Java C/C++ local inter-process communication  
     
Page 1 of 1
Display using:  

Gurz


Total Posts: 33
Joined: Aug 2007
 
Posted: 2016-06-29 11:46
I have to move large amounts of data from native C++ code into a JVM for further processing.

I do consider using JNI, but would love more flexibility regarding communication patterns and architecture that IPC would give me, at the cost of latency and throughput.

Speaking of that:
I estimate I'd need latency (roundtrip time) at lower millisecs and data size of 1 to 10MB.

I'm looking now at different transports and protocols like pipes, sockets, shared memory, AMQP, ..., and there are (are there?) several "solutions" that somehow look like they'd fit: ZeroMQ, Thrift, Avro,... - sorry for mixing apples and oranges.

Oh, and this is supposed to be an "enterprise" solution - used by others, low maintenance and in use for several years.

Any insights?

EspressoLover


Total Posts: 205
Joined: Jan 2015
 
Posted: 2016-06-29 12:25
How structured is the data? And how coupled do you need the two codebases? Are you just handing off a big chunk of plain-old-data between the two systems? If you're just sending a whole bunch of numeric and text fields in standard layout, I'd consider just using Unix pipes and a plaintext CSV representation. It's definitely not as efficient as binary, but both C++ and Java are pretty f'in fast at text formatting/parsing.

If it meets your performance requirements, then plaintext piping is way more maintainable and lower overhead then having a dedicated message broker, dealing with clients/servers, etc. If that's too slow, I'd suggest looking into Redis pub/sub. With ZeroMQ/RabbitMQ you're dealing with more headaches for features your probably don't need in this context, like persistent queues.

If your data's really structured, or you really need tight coupling between the C++ and Java code, then Avro. But if you're just going for repeated passes of the same object, then this is way more overhead then you need. Coupling two separate codebases in completely different languages makes maintainability challenging.

contango_and_cash


Total Posts: 57
Joined: Sep 2015
 
Posted: 2016-06-29 22:09
What about something like this

Gurz


Total Posts: 33
Joined: Aug 2007
 
Posted: 2016-07-01 14:50
Thanks, you two!

We would like to keep our codebases decoupled, and keep the big/fast payloads simply structured. Putting something more expressive on top later (e.g. serialization with exchanged schema) shouldn't be that hard.

If we had 1:1 consumer-producer, I'd look into building a shared memory setup with two "unidirectional channels" - but it looks like we will have n producers (C++) and 1 (for now) consumer and some need for two-way communication, and building that kind of coordination myself... yeah, smells a bit like fast (but simple routing) and local (over network is secondary) message brokering.

Besides 0MQ and RabbitMQ - any other approaches that come to mind? Kafka?

contango_and_cash


Total Posts: 57
Joined: Sep 2015
 
Posted: 2016-07-01 16:01
redis seems like a good option or you could create some sort of erlang mailbox listener in the middle.

aws has a lot of stuff that could solve this problem but you introduce latency.

i think you just need some performant warehouse in the middle.

Polter


Total Posts: 130
Joined: Jun 2008
 
Posted: 2016-07-02 20:57
How about Aeron?
https://github.com/real-logic/Aeron

Potential limitation for your use case:
"Note the the C++ implementation is client only, the Java Media Driver process must be running to support the C++ and Java clients."
https://github.com/real-logic/Aeron/wiki/Cpp-Programming-Guide

ast4


Total Posts: 390
Joined: Aug 2007
 
Posted: 2016-08-18 14:21
Posting for posterity..

I prefer to protobuf (google's offering) over thrift. Google has been pretty good about continued support.

For transport, depends.. You can get some serious juice out of lock free data structures if your problem is simple enough. Are you running multiproc? You could scale with rdma as well if that's available to you.

"Mathematicians are machines for turning coffee into theorems!"

nym


Total Posts: 32
Joined: Aug 2012
 
Posted: 2016-11-07 20:42
If you need latency of a few milliseconds you need to consider a dedicated hardware.
Standard JVM usually can not perform context switching in less then 50ms.

This does not take into account the fact that you have to serialize/de-serialize the information and 10MB are not exactly a few bits

Previous Thread :: Next Thread 
Page 1 of 1