Forums  > Software  > kafka vs zeromq  
     
Page 1 of 1
Display using:  

svisstack


Total Posts: 318
Joined: Feb 2014
 
Posted: 2018-11-15 18:45
What do you think guys about the kafka vs zeromq for the market data? I'm using the zeromq but considering kafka for some part of the project as pilot. Concerns: lack of control around the issues which could arise from using kafka eg. latency overhead; Any experiences?

www.coinapi.io && www.cryptotick.com

Maggette


Total Posts: 1093
Joined: Jun 2007
 
Posted: 2018-11-15 19:51
Kafka is more than a message queue that dies a lot of stuff for you. For many applications that is a good thing, for some it might be bad.

Kafka is JVM....that is considered evil by some.
Loss of ordering over partitions comes with the distributed nature of that beast.
You might need confluent stuff for a bearable experience with schema registries

But I guess you are aware of all that.

I like that it integrates well with other apache stuff and micro services via kafka streams api is a very awesome and lightweight way to create stateful stream applications without flink or storm

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

prikolno


Total Posts: 40
Joined: Jul 2018
 
Posted: 2018-11-16 04:31
Market data doesn't require many messaging semantics and I'm guessing you're not building complex distributed applications on top. If latency overhead was the only concern, I'd build my own (lightweight) data distribution protocol with pub/sub semantics over a reliable transport layer like Aeron.

Is the market data traversing NAT gateways? Do you need topic filtering?

svisstack


Total Posts: 318
Joined: Feb 2014
 
Posted: 2018-11-16 12:34
>> Market data doesn't require many messaging semantics and I'm guessing you're not building complex distributed applications on top.

Don't assume that, infrastructure is highly distributed.

>> If latency overhead was the only concern, I'd build my own (lightweight) data distribution protocol with pub/sub semantics over a reliable transport layer like Aeron.

I actually have exactly that on ZeroMQ at the moment, infrastructure grew to 100+ servers and there are cases when I need more than pub/sub including off-site geo-awareness + complex failover logic.

Deciding on abstracting the logic and expanding the ZeroMQ usage or piloting another thing. ZeroMQ works great, so the decision on Kafka must be strong to act.

>> Is the market data traversing NAT gateways? Do you need topic filtering?

Both yes. I actually have topic filtering on the ZeroMQ already.




www.coinapi.io && www.cryptotick.com

EspressoLover


Total Posts: 362
Joined: Jan 2015
 
Posted: 2018-11-17 03:45
Why not include RabbitMQ in the running? Seems like a potentially happy medium, where it would be easier than Kafka to adapt a pre-existing ZeroMQ system.

> off-site geo-awareness + complex failover logic.

Have you just considered keeping zeroMQ but adding a service discovery layer (etcd, consul, or zookeeper)? Depending on your specific needs, geo-location and high-availability can probably be handled inside service discovery in a way that's agnostic to the underlying message layer.


Good questions outrank easy answers. -Paul Samuelson

svisstack


Total Posts: 318
Joined: Feb 2014
 
Posted: 2018-11-18 13:12
@EspressoLover: you are correct, it's actually rabbitmq+X vs zmq+consul vs zmq+custom_solution.

I actually didn't consider using the third-party software because I think my solution can be correct and simple without introducing additional complexity, but I did not know about the consul project for some reason (looks like good fit) and must rethink that now; as it could solve unforeseen issues in the future (like this issue at the moment).

Thanks for the input. I will need to think about it now.

www.coinapi.io && www.cryptotick.com
Previous Thread :: Next Thread 
Page 1 of 1