Topics

[Edgex-devel] EdgeX Foundry exporter for Prometheus

Fede Claramonte
 

Tony,

from my point of view it does not make sense to use different version of different components of the monorepo at the same time. We can only assure that everything is working (with tests and blackbox testing) if all the packages are from the same monorepo version. Not having one repo but several only make this harder for us.

As an example with linux, will you choose to use different versions for each kernel module? network from 4.4, scsi from 4.8, graphics from 4.12. Normally no, but If you really need this you can do it, but it is not supported and you are on your own. I think the monorepo should have the same philosophy, this is why I think only a version for all the monorepo is the best and simplest option, but the versioning is a different discussion.

Regards,
Fede

On 20/02/18 19:12, espy wrote:
On 02/19/2018 09:52 AM, Drasko DRASKOVIC wrote:
Hi Jim,
it will be hard to accept PRs before we merge the monorepo. Let's try
all to make this to happen soon.
Drasko --

I never saw an answer about my versioning question re: the monorepo.  To me, it's a deal-breaker if external clients have to treat all Go library packages within the monorepo as a single version.

Let's discuss at the meeting later this afternoon.

Regards,
/tony




Kei,
Prometheus is a good chice in my opinion. I will analyze your code. I
was expecting rather that each service will separately export it's
metrics to Prometheus, and we have yet to define which of this metrics
should be exported.

Please keep edgex-golang mainling list in loop.

Dejan,
I will need your help for this Prometheus integration, I know you did
similar on Mainflux.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com   |  Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division |   Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic

On Mon, Feb 19, 2018 at 3:03 PM,  <James.White2@...> wrote:
Dell - Internal Use - Confidential

Thanks Kei.
You are correct, the edgex-go repo will be the home of our new EdgeX mono-repo for all our Go code.  However, it is not quite ready yet (you might notice it stands empty today).

I'd ask that you work with Janlo Isidorovic (copied) who is the chariman of our Applications WG to coordinate PR requests on the existing export-go repo and, when the time is ready, the work group move it to edgex-go.

Thanks for your contribution
Jim
-----Original Message-----
From: Kei Ohmura [mailto:ohmura.kei@...]
Sent: Monday, February 19, 2018 3:48 AM
To: White2, James <James_White2@...>
Cc: Keith Steele <keith@...>; steve@...; OHMURA Kei <ohmura.kei@...>
Subject: Re: EdgeX Foundry exporter for Prometheus

Hi,

I wrote edgex_fowarder as a just prototype which pushes metrics to the Prometheus.
https://github.com/ohmk/edgex_exporter/blob/master/forwarder/edgex_forwarder.go

I think this code should be implemented as the part of export-go.

If I understood correctly, all golang repositories will be merged into one repository.
https://github.com/edgexfoundry/edgex-go

So community will ready to accept contribution, I'll submit patches to the edgex-go repository.

Thanks,

2018-02-12 1:03 GMT+09:00  <James.White2@...>:
Kei,

Your English is far far better than my ability to try to communicate in your native language.   So thank you for your patience and communicating with me in my native tongue.

You are right, metadata does not collect those metrics.  So having another process do that makes some sense.  We look forward to seeing your work Kei.

Thanks
Jim
________________________________________
From: Kei Ohmura <ohmura.kei@...>
Sent: Sunday, February 11, 2018 7:01 AM
To: White2, James
Cc: Keith Steele; steve@...; OHMURA Kei
Subject: Re: EdgeX Foundry exporter for Prometheus

Hi Jim,


I answer the question 2).

2) We have often thought about a more generic north side interface to
meta data (and even command) that would allow applications (like
Prometheus) access to Metadata and/or command information.  so the
idea of an interface to these is a good idea.  What I am trying to
figure out is what would your edgex-exporter-for-Prometheus do that
is not being done by the APIs of Metadata today?  What extra servic
or capability is offered by this exporter?   Looks like you are
thinking metrics collection might be one thing?  And if it is needed,
is there a way for us to make this generic for things other than
Prometheus as well??
I don't understand well what you mean because of my English skill,
sorry about that. Anyway what I'd like to do is to collect metrics of
EdgeX Foundry. As far as I know current Metadata doesn't support API
for exporting metrics like the following:

* edgex_total_events
   * Total number of events which are inserted into CoreData.
* edgex_total_readings
   * Total number of readings which are inserted into CoreData.
* edgex_total_services
   * Total number of microservices connected to Consul.


Thanks,

2018-02-09 0:29 GMT+09:00  <James.White2@...>:
Hi Kei
thanks for this and I now have a much better understanding of what you want to do.  This is good stuff!!

couple of questions...
1) why would prometheus need to pull the data from core data when the export distro could push it to Prometheus? Making requests of core data directly, while possible, is potentially impactful of a service that must be high throughput.  If something (like Prometheus) on the north were to make too big of a request, it could impact the south side data collection.  So is there something you think export distro is not going to be able to do to push Prometheus the data?

2) We have often thought about a more generic north side interface to meta data (and even command) that would allow applications (like Prometheus) access to Metadata and/or command information.  so the idea of an interface to these is a good idea.  What I am trying to figure out is what would your edgex-exporter-for-Prometheus do that is not being done by the APIs of Metadata today?  What extra service or capability is offered by this exporter?   Looks like you are thinking metrics collection might be one thing?  And if it is needed, is there a way for us to make this generic for things other than Prometheus as well??

In general, I am very supportive of your effort and definitely would like to see a connector to Prometheus.  I have copied Keith Steele and Steve Osselton from IoTech because I believe they are looking at some sore of connection to Prometheus as well - so we should make sure everyone's input is in the eventual design and implementation if we can.

________________________________________
From: Kei Ohmura <ohmura.kei@...>
Sent: Thursday, February 8, 2018 1:38 AM
To: White2, James
Cc: OHMURA Kei
Subject: RFC: EdgeX Foundry exporter for Prometheus

Hi Jim,

I'm sending my thought to you.


# Goal
My goal is to export metrics of EdgeX Foundry microservices to
management system which is running on cloud and to monitor edge
servers including microservices which are running on it.


# Why prometheus
Prometheus is an open-source systems monitoring and alerting toolkit
with an active ecosystem and users.
Prometheus has many exporters:
https://github.com/prometheus/docs/blob/master/content/docs/instrumen
ting/exporters.md

So we can monitor not only EdgeX Foundry but also other microservices
running on edge servers just by implementing a exporter for EdgeX
Foundry.


# Architecture
Please refer to the attached file.
If you'd like to get hardware and OS metrics (such as cpu usage), you
only need to run
node_exporter(https://github.com/prometheus/node_exporter) on the
edge server.

I think that we need to implement similar functions into export
services (distrubution?) for users who don't use prometheus.


# Export metrics
I've already implemented the below:
(https://github.com/ohmk/edgex_exporter)

* edgex_total_devices
   * Total number devices connected to EdgeX Foundry.
* edgex_total_events
   * Total number of events which are inserted into CoreData.
* edgex_total_readings
   * Total number of readings which are inserted into CoreData.
* edgex_total_services
   * Total number of microservices connected to Consul.


# TODO
We need to discuss the following:
* What metrics each microservice should export
   * Also export services should have api for metrics?
     * ex. how many clients registered?
* Defining and implementing API of microservices to get metrics

I think that the above is very important regardless of Prometheus
support.


Thanks,
Kei
_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang
_______________________________________________
EdgeX-Devel mailing list
EdgeX-Devel@...
https://lists.edgexfoundry.org/mailman/listinfo/edgex-devel

Drasko DRASKOVIC <drasko@...>
 

Tony,
for the sake of simplicity - would it be acceptable that we start with
single version and then delve deeper when a particular use-case need
comes?

I can see potential need for this: serviceA changed, monorepo is
bumped to a higher version, but you do not want to update all the
services now. Keep in mind however that Docker image updates are
incremental, so the deltas of unchanged images are zero. I think that
Snappy works the same way.

Also - we will have 2 releases per year. And in reality, you will
probably run 10-ish services on a relatively powerful industrial GW -
why not change them all twice per year? Update will probably take just
a few minutes.

My take is that it is hard to estimate now the scenarios, but we can
start simple and learn how to walk before we run.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic


On Tue, Feb 20, 2018 at 8:38 PM, Fede Claramonte
<fclaramonte@...> wrote:
Tony,

from my point of view it does not make sense to use different version of
different components of the monorepo at the same time. We can only assure
that everything is working (with tests and blackbox testing) if all the
packages are from the same monorepo version. Not having one repo but several
only make this harder for us.

As an example with linux, will you choose to use different versions for each
kernel module? network from 4.4, scsi from 4.8, graphics from 4.12. Normally
no, but If you really need this you can do it, but it is not supported and
you are on your own. I think the monorepo should have the same philosophy,
this is why I think only a version for all the monorepo is the best and
simplest option, but the versioning is a different discussion.

Regards,
Fede


On 20/02/18 19:12, espy wrote:

On 02/19/2018 09:52 AM, Drasko DRASKOVIC wrote:

Hi Jim,
it will be hard to accept PRs before we merge the monorepo. Let's try
all to make this to happen soon.

Drasko --

I never saw an answer about my versioning question re: the monorepo. To
me, it's a deal-breaker if external clients have to treat all Go library
packages within the monorepo as a single version.

Let's discuss at the meeting later this afternoon.

Regards,
/tony




Kei,
Prometheus is a good chice in my opinion. I will analyze your code. I
was expecting rather that each service will separately export it's
metrics to Prometheus, and we have yet to define which of this metrics
should be exported.

Please keep edgex-golang mainling list in loop.

Dejan,
I will need your help for this Prometheus integration, I know you did
similar on Mainflux.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic

On Mon, Feb 19, 2018 at 3:03 PM, <James.White2@...> wrote:

Dell - Internal Use - Confidential

Thanks Kei.
You are correct, the edgex-go repo will be the home of our new EdgeX
mono-repo for all our Go code. However, it is not quite ready yet (you
might notice it stands empty today).

I'd ask that you work with Janlo Isidorovic (copied) who is the chariman
of our Applications WG to coordinate PR requests on the existing export-go
repo and, when the time is ready, the work group move it to edgex-go.

Thanks for your contribution
Jim
-----Original Message-----
From: Kei Ohmura [mailto:ohmura.kei@...]
Sent: Monday, February 19, 2018 3:48 AM
To: White2, James <James_White2@...>
Cc: Keith Steele <keith@...>; steve@...; OHMURA Kei
<ohmura.kei@...>
Subject: Re: EdgeX Foundry exporter for Prometheus

Hi,

I wrote edgex_fowarder as a just prototype which pushes metrics to the
Prometheus.

https://github.com/ohmk/edgex_exporter/blob/master/forwarder/edgex_forwarder.go

I think this code should be implemented as the part of export-go.

If I understood correctly, all golang repositories will be merged into
one repository.
https://github.com/edgexfoundry/edgex-go

So community will ready to accept contribution, I'll submit patches to
the edgex-go repository.

Thanks,

2018-02-12 1:03 GMT+09:00 <James.White2@...>:

Kei,

Your English is far far better than my ability to try to communicate in
your native language. So thank you for your patience and communicating
with me in my native tongue.

You are right, metadata does not collect those metrics. So having
another process do that makes some sense. We look forward to seeing your
work Kei.

Thanks
Jim
________________________________________
From: Kei Ohmura <ohmura.kei@...>
Sent: Sunday, February 11, 2018 7:01 AM
To: White2, James
Cc: Keith Steele; steve@...; OHMURA Kei
Subject: Re: EdgeX Foundry exporter for Prometheus

Hi Jim,


I answer the question 2).

2) We have often thought about a more generic north side interface to
meta data (and even command) that would allow applications (like
Prometheus) access to Metadata and/or command information. so the
idea of an interface to these is a good idea. What I am trying to
figure out is what would your edgex-exporter-for-Prometheus do that
is not being done by the APIs of Metadata today? What extra servic
or capability is offered by this exporter? Looks like you are
thinking metrics collection might be one thing? And if it is needed,
is there a way for us to make this generic for things other than
Prometheus as well??

I don't understand well what you mean because of my English skill,
sorry about that. Anyway what I'd like to do is to collect metrics of
EdgeX Foundry. As far as I know current Metadata doesn't support API
for exporting metrics like the following:

* edgex_total_events
* Total number of events which are inserted into CoreData.
* edgex_total_readings
* Total number of readings which are inserted into CoreData.
* edgex_total_services
* Total number of microservices connected to Consul.


Thanks,

2018-02-09 0:29 GMT+09:00 <James.White2@...>:

Hi Kei
thanks for this and I now have a much better understanding of what you
want to do. This is good stuff!!

couple of questions...
1) why would prometheus need to pull the data from core data when the
export distro could push it to Prometheus? Making requests of core data
directly, while possible, is potentially impactful of a service that must be
high throughput. If something (like Prometheus) on the north were to make
too big of a request, it could impact the south side data collection. So is
there something you think export distro is not going to be able to do to
push Prometheus the data?

2) We have often thought about a more generic north side interface to
meta data (and even command) that would allow applications (like Prometheus)
access to Metadata and/or command information. so the idea of an interface
to these is a good idea. What I am trying to figure out is what would your
edgex-exporter-for-Prometheus do that is not being done by the APIs of
Metadata today? What extra service or capability is offered by this
exporter? Looks like you are thinking metrics collection might be one
thing? And if it is needed, is there a way for us to make this generic for
things other than Prometheus as well??

In general, I am very supportive of your effort and definitely would
like to see a connector to Prometheus. I have copied Keith Steele and Steve
Osselton from IoTech because I believe they are looking at some sore of
connection to Prometheus as well - so we should make sure everyone's input
is in the eventual design and implementation if we can.

________________________________________
From: Kei Ohmura <ohmura.kei@...>
Sent: Thursday, February 8, 2018 1:38 AM
To: White2, James
Cc: OHMURA Kei
Subject: RFC: EdgeX Foundry exporter for Prometheus

Hi Jim,

I'm sending my thought to you.


# Goal
My goal is to export metrics of EdgeX Foundry microservices to
management system which is running on cloud and to monitor edge
servers including microservices which are running on it.


# Why prometheus
Prometheus is an open-source systems monitoring and alerting toolkit
with an active ecosystem and users.
Prometheus has many exporters:
https://github.com/prometheus/docs/blob/master/content/docs/instrumen
ting/exporters.md

So we can monitor not only EdgeX Foundry but also other microservices
running on edge servers just by implementing a exporter for EdgeX
Foundry.


# Architecture
Please refer to the attached file.
If you'd like to get hardware and OS metrics (such as cpu usage), you
only need to run
node_exporter(https://github.com/prometheus/node_exporter) on the
edge server.

I think that we need to implement similar functions into export
services (distrubution?) for users who don't use prometheus.


# Export metrics
I've already implemented the below:
(https://github.com/ohmk/edgex_exporter)

* edgex_total_devices
* Total number devices connected to EdgeX Foundry.
* edgex_total_events
* Total number of events which are inserted into CoreData.
* edgex_total_readings
* Total number of readings which are inserted into CoreData.
* edgex_total_services
* Total number of microservices connected to Consul.


# TODO
We need to discuss the following:
* What metrics each microservice should export
* Also export services should have api for metrics?
* ex. how many clients registered?
* Defining and implementing API of microservices to get metrics

I think that the above is very important regardless of Prometheus
support.


Thanks,
Kei

_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang
_______________________________________________
EdgeX-Devel mailing list
EdgeX-Devel@...
https://lists.edgexfoundry.org/mailman/listinfo/edgex-devel