Topics

[EXTERNAL] Re: 0MQ Alternatives - Tuesday's meeting

Dellabianca, Alberto
 

Drasko,

Thanks for your reply, I understand your points.

There are also MQTT broker implementations in Golang (e.g. https://github.com/VolantMQ/volantmq) although I never tested them.
Just curious: do you know on a Raspberry Pi how many messages/second can NATS process? (assuming 1 publisher and 1 subscriber for simplicity).

Thx

Alberto

 

 

From: Drasko DRASKOVIC [mailto:drasko@...]
Sent: Friday, March 16, 2018 11:38 PM
To: Dellabianca, Alberto <Alberto.Dellabianca@...>
Cc: James.White2@...; edgex-golang@...
Subject: [EXTERNAL] Re: [Edgex-golang] 0MQ Alternatives - Tuesday's meeting

 

On Sat, Mar 17, 2018 at 2:59 AM, Dellabianca, Alberto
<Alberto.Dellabianca@...> wrote:
>
> Great summary Jim.
>
> Any thoughts on having in the list also MQTT? It would be a broker option like NATS, it is lightweight
NATS is more lightweight then majority MQTT brokers out there

> and widely used in the community.

Not for the purpose of microservice inter-communication.

>
> Great implementations are Mosquitto Broker

Not scalable (can not cluster easily like NATS for HA/FT). Plus
written in C, so we'll have to recompile for ARM wih C compiler- bring
us back to the problem because of which we are changing ZMQ. Plus I am
not sure it works on Windows (probably not).

> and Paho Client (in different languages), not sure if the license is compatible with LF requirements though. We are already using Paho Client in EdgeX however.

For device interface, and this is OK.

>
> Also I understand that core-data is already somewhat capable to subscribe to MQTT topics?

Export data, only as a MQTT client for the cloud (not a broker).

MQTT has several other problems - like that for simplicity of the
protocol it does not keep origin (publisher) info, so it is impossible
to prevent loopbacks when you publish and subscribe on the same topic.
Finally, and probably most importantly - besides pure PUB/SUB:
https://nats.io/documentation/concepts/nats-pub-sub/ - NATS offers
other patterns, like REQ/RSP and Queueing that are very useful and can
be eventually achieved with MQTT only with additional work.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic

Drasko DRASKOVIC <drasko@...>
 

On Sat, Mar 17, 2018 at 6:11 AM, Dellabianca, Alberto
<Alberto.Dellabianca@...> wrote:
Drasko,

Thanks for your reply, I understand your points.

There are also MQTT broker implementations in Golang (e.g.
https://github.com/VolantMQ/volantmq) although I never tested them.
Just curious: do you know on a Raspberry Pi how many messages/second can
NATS process? (assuming 1 publisher and 1 subscriber for simplicity).
That depends on your RAM, but NATS is a monster:
https://seroter.wordpress.com/2016/05/16/modern-open-source-messaging-apache-kafka-rabbitmq-and-nats-in-action/
"NATS was originally built with Ruby and achieved a respectable 150k
messages per second. The team rewrote it in Go, and now you can do an
absurd 8-11 million messages per second. It’s tiny, just a 3MB Docker
image"

Mosquitto can achieve ~130k on a serious machine, if really pushed to
the edge. MQTT aedes can go ~200k:
https://www.nearform.com/blog/performance-reaching-ludicrous-speed/

I would not be surprised that NATS goes beyond 200k mps on RPim but I
never did the benchmarks - you must check with NATS community.

BR,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic

Dellabianca, Alberto
 

NATS seems to scale up very well indeed...
A good start would be if a EdgeX could sustain just 1% of what you mentioned about Mosquitto (130k —> 1.3k mps) on a RPi...
Thanks for sharing the stats.
Alberto

________________________________________
From: Drasko DRASKOVIC <drasko@...>
Sent: Saturday, March 17, 2018 12:24:58 AM
To: Dellabianca, Alberto
Cc: James.White2@...; edgex-golang@...
Subject: Re: [EXTERNAL] Re: [Edgex-golang] 0MQ Alternatives - Tuesday's meeting

On Sat, Mar 17, 2018 at 6:11 AM, Dellabianca, Alberto
<Alberto.Dellabianca@...> wrote:
Drasko,

Thanks for your reply, I understand your points.

There are also MQTT broker implementations in Golang (e.g.
https://github.com/VolantMQ/volantmq<https://github.com/VolantMQ/volantmq>) although I never tested them.
Just curious: do you know on a Raspberry Pi how many messages/second can
NATS process? (assuming 1 publisher and 1 subscriber for simplicity).
That depends on your RAM, but NATS is a monster:
https://seroter.wordpress.com/2016/05/16/modern-open-source-messaging-apache-kafka-rabbitmq-and-nats-in-action/<https://seroter.wordpress.com/2016/05/16/modern-open-source-messaging-apache-kafka-rabbitmq-and-nats-in-action/>
"NATS was originally built with Ruby and achieved a respectable 150k
messages per second. The team rewrote it in Go, and now you can do an
absurd 8-11 million messages per second. It’s tiny, just a 3MB Docker
image"

Mosquitto can achieve ~130k on a serious machine, if really pushed to
the edge. MQTT aedes can go ~200k:
https://www.nearform.com/blog/performance-reaching-ludicrous-speed/<https://www.nearform.com/blog/performance-reaching-ludicrous-speed/>

I would not be surprised that NATS goes beyond 200k mps on RPim but I
never did the benchmarks - you must check with NATS community.

BR,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com<http://www.mainflux.com> | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic<https://www.linkedin.com/in/draskodraskovic>
Twitter: @draskodraskovic

James.White2@...
 

Dell - Internal Use - Confidential

Alberto,
In the code base of the Java core data and export distro, there were replacement classes and commented-out configuration for use of MQTT in place of ZeroMQ. We initially used MQTT (Paho client - ActiveMQ broker) in Project Fuse. But we found that just too heavy for some of the smaller environments.

Having said that, as we go forward and pick a "reference implementation" alternate to ZeroMQ, I would also like to again allow for those with larger platforms and use cases that need it to replace whatever we choose (Nats, etc.) with their own implementation of message infrastructure (like MQTT or even one of the rejected alternatives if it is more appropriate to their needs). We need to isolate the messaging function from the rest of the micro service so that alternatives are more easily dropped in place. That's the goal.
Jim

-----Original Message-----
From: Dellabianca, Alberto [mailto:Alberto.Dellabianca@...]
Sent: Saturday, March 17, 2018 7:01 AM
To: Drasko DRASKOVIC <drasko@...>
Cc: White2, James <James_White2@...>; edgex-golang@...
Subject: Re: [EXTERNAL] Re: [Edgex-golang] 0MQ Alternatives - Tuesday's meeting

NATS seems to scale up very well indeed...
A good start would be if a EdgeX could sustain just 1% of what you mentioned about Mosquitto (130k -> 1.3k mps) on a RPi...
Thanks for sharing the stats.
Alberto

________________________________________
From: Drasko DRASKOVIC <drasko@...>
Sent: Saturday, March 17, 2018 12:24:58 AM
To: Dellabianca, Alberto
Cc: James.White2@...; edgex-golang@...
Subject: Re: [EXTERNAL] Re: [Edgex-golang] 0MQ Alternatives - Tuesday's meeting

On Sat, Mar 17, 2018 at 6:11 AM, Dellabianca, Alberto <Alberto.Dellabianca@...> wrote:
Drasko,

Thanks for your reply, I understand your points.

There are also MQTT broker implementations in Golang (e.g.
https://github.com/VolantMQ/volantmq<https://github.com/VolantMQ/volantmq>) although I never tested them.
Just curious: do you know on a Raspberry Pi how many messages/second
can NATS process? (assuming 1 publisher and 1 subscriber for simplicity).
That depends on your RAM, but NATS is a monster:
https://seroter.wordpress.com/2016/05/16/modern-open-source-messaging-apache-kafka-rabbitmq-and-nats-in-action/<https://seroter.wordpress.com/2016/05/16/modern-open-source-messaging-apache-kafka-rabbitmq-and-nats-in-action/>
"NATS was originally built with Ruby and achieved a respectable 150k messages per second. The team rewrote it in Go, and now you can do an absurd 8-11 million messages per second. It's tiny, just a 3MB Docker image"

Mosquitto can achieve ~130k on a serious machine, if really pushed to the edge. MQTT aedes can go ~200k:
https://www.nearform.com/blog/performance-reaching-ludicrous-speed/<https://www.nearform.com/blog/performance-reaching-ludicrous-speed/>

I would not be surprised that NATS goes beyond 200k mps on RPim but I never did the benchmarks - you must check with NATS community.

BR,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com<http://www.mainflux.com> | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic<https://www.linkedin.com/in/draskodraskovic>
Twitter: @draskodraskovic