Topics

EdgeX Foundry exporter for Prometheus

Drasko DRASKOVIC <drasko@...>
 

Hi Jim,
it will be hard to accept PRs before we merge the monorepo. Let's try
all to make this to happen soon.

Kei,
Prometheus is a good chice in my opinion. I will analyze your code. I
was expecting rather that each service will separately export it's
metrics to Prometheus, and we have yet to define which of this metrics
should be exported.

Please keep edgex-golang mainling list in loop.

Dejan,
I will need your help for this Prometheus integration, I know you did
similar on Mainflux.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

On Mon, Feb 19, 2018 at 3:03 PM, <James.White2@...> wrote:
Dell - Internal Use - Confidential

Thanks Kei.
You are correct, the edgex-go repo will be the home of our new EdgeX mono-repo for all our Go code. However, it is not quite ready yet (you might notice it stands empty today).

I'd ask that you work with Janlo Isidorovic (copied) who is the chariman of our Applications WG to coordinate PR requests on the existing export-go repo and, when the time is ready, the work group move it to edgex-go.

Thanks for your contribution
Jim
-----Original Message-----
From: Kei Ohmura [mailto:ohmura.kei@...]
Sent: Monday, February 19, 2018 3:48 AM
To: White2, James <James_White2@...>
Cc: Keith Steele <keith@...>; steve@...; OHMURA Kei <ohmura.kei@...>
Subject: Re: EdgeX Foundry exporter for Prometheus

Hi,

I wrote edgex_fowarder as a just prototype which pushes metrics to the Prometheus.
https://github.com/ohmk/edgex_exporter/blob/master/forwarder/edgex_forwarder.go

I think this code should be implemented as the part of export-go.

If I understood correctly, all golang repositories will be merged into one repository.
https://github.com/edgexfoundry/edgex-go

So community will ready to accept contribution, I'll submit patches to the edgex-go repository.

Thanks,

2018-02-12 1:03 GMT+09:00 <James.White2@...>:
Kei,

Your English is far far better than my ability to try to communicate in your native language. So thank you for your patience and communicating with me in my native tongue.

You are right, metadata does not collect those metrics. So having another process do that makes some sense. We look forward to seeing your work Kei.

Thanks
Jim
________________________________________
From: Kei Ohmura <ohmura.kei@...>
Sent: Sunday, February 11, 2018 7:01 AM
To: White2, James
Cc: Keith Steele; steve@...; OHMURA Kei
Subject: Re: EdgeX Foundry exporter for Prometheus

Hi Jim,


I answer the question 2).

2) We have often thought about a more generic north side interface to
meta data (and even command) that would allow applications (like
Prometheus) access to Metadata and/or command information. so the
idea of an interface to these is a good idea. What I am trying to
figure out is what would your edgex-exporter-for-Prometheus do that
is not being done by the APIs of Metadata today? What extra servic
or capability is offered by this exporter? Looks like you are
thinking metrics collection might be one thing? And if it is needed,
is there a way for us to make this generic for things other than
Prometheus as well??
I don't understand well what you mean because of my English skill,
sorry about that. Anyway what I'd like to do is to collect metrics of
EdgeX Foundry. As far as I know current Metadata doesn't support API
for exporting metrics like the following:

* edgex_total_events
* Total number of events which are inserted into CoreData.
* edgex_total_readings
* Total number of readings which are inserted into CoreData.
* edgex_total_services
* Total number of microservices connected to Consul.


Thanks,

2018-02-09 0:29 GMT+09:00 <James.White2@...>:
Hi Kei
thanks for this and I now have a much better understanding of what you want to do. This is good stuff!!

couple of questions...
1) why would prometheus need to pull the data from core data when the export distro could push it to Prometheus? Making requests of core data directly, while possible, is potentially impactful of a service that must be high throughput. If something (like Prometheus) on the north were to make too big of a request, it could impact the south side data collection. So is there something you think export distro is not going to be able to do to push Prometheus the data?

2) We have often thought about a more generic north side interface to meta data (and even command) that would allow applications (like Prometheus) access to Metadata and/or command information. so the idea of an interface to these is a good idea. What I am trying to figure out is what would your edgex-exporter-for-Prometheus do that is not being done by the APIs of Metadata today? What extra service or capability is offered by this exporter? Looks like you are thinking metrics collection might be one thing? And if it is needed, is there a way for us to make this generic for things other than Prometheus as well??

In general, I am very supportive of your effort and definitely would like to see a connector to Prometheus. I have copied Keith Steele and Steve Osselton from IoTech because I believe they are looking at some sore of connection to Prometheus as well - so we should make sure everyone's input is in the eventual design and implementation if we can.

________________________________________
From: Kei Ohmura <ohmura.kei@...>
Sent: Thursday, February 8, 2018 1:38 AM
To: White2, James
Cc: OHMURA Kei
Subject: RFC: EdgeX Foundry exporter for Prometheus

Hi Jim,

I'm sending my thought to you.


# Goal
My goal is to export metrics of EdgeX Foundry microservices to
management system which is running on cloud and to monitor edge
servers including microservices which are running on it.


# Why prometheus
Prometheus is an open-source systems monitoring and alerting toolkit
with an active ecosystem and users.
Prometheus has many exporters:
https://github.com/prometheus/docs/blob/master/content/docs/instrumen
ting/exporters.md

So we can monitor not only EdgeX Foundry but also other microservices
running on edge servers just by implementing a exporter for EdgeX
Foundry.


# Architecture
Please refer to the attached file.
If you'd like to get hardware and OS metrics (such as cpu usage), you
only need to run
node_exporter(https://github.com/prometheus/node_exporter) on the
edge server.

I think that we need to implement similar functions into export
services (distrubution?) for users who don't use prometheus.


# Export metrics
I've already implemented the below:
(https://github.com/ohmk/edgex_exporter)

* edgex_total_devices
* Total number devices connected to EdgeX Foundry.
* edgex_total_events
* Total number of events which are inserted into CoreData.
* edgex_total_readings
* Total number of readings which are inserted into CoreData.
* edgex_total_services
* Total number of microservices connected to Consul.


# TODO
We need to discuss the following:
* What metrics each microservice should export
* Also export services should have api for metrics?
* ex. how many clients registered?
* Defining and implementing API of microservices to get metrics

I think that the above is very important regardless of Prometheus
support.


Thanks,
Kei

espy
 

On 02/19/2018 09:52 AM, Drasko DRASKOVIC wrote:
Hi Jim,
it will be hard to accept PRs before we merge the monorepo. Let's try
all to make this to happen soon.
Drasko --

I never saw an answer about my versioning question re: the monorepo. To me, it's a deal-breaker if external clients have to treat all Go library packages within the monorepo as a single version.

Let's discuss at the meeting later this afternoon.

Regards,
/tony



Kei,
Prometheus is a good chice in my opinion. I will analyze your code. I
was expecting rather that each service will separately export it's
metrics to Prometheus, and we have yet to define which of this metrics
should be exported.
Please keep edgex-golang mainling list in loop.
Dejan,
I will need your help for this Prometheus integration, I know you did
similar on Mainflux.
Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor
www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France
LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovicOn Mon, Feb 19, 2018 at 3:03 PM, <James.White2@...> wrote:
Dell - Internal Use - Confidential

Thanks Kei.
You are correct, the edgex-go repo will be the home of our new EdgeX mono-repo for all our Go code. However, it is not quite ready yet (you might notice it stands empty today).

I'd ask that you work with Janlo Isidorovic (copied) who is the chariman of our Applications WG to coordinate PR requests on the existing export-go repo and, when the time is ready, the work group move it to edgex-go.

Thanks for your contribution
Jim
-----Original Message-----
From: Kei Ohmura [mailto:ohmura.kei@...]
Sent: Monday, February 19, 2018 3:48 AM
To: White2, James <James_White2@...>
Cc: Keith Steele <keith@...>; steve@...; OHMURA Kei <ohmura.kei@...>
Subject: Re: EdgeX Foundry exporter for Prometheus

Hi,

I wrote edgex_fowarder as a just prototype which pushes metrics to the Prometheus.
https://github.com/ohmk/edgex_exporter/blob/master/forwarder/edgex_forwarder.go

I think this code should be implemented as the part of export-go.

If I understood correctly, all golang repositories will be merged into one repository.
https://github.com/edgexfoundry/edgex-go

So community will ready to accept contribution, I'll submit patches to the edgex-go repository.

Thanks,

2018-02-12 1:03 GMT+09:00 <James.White2@...>:
Kei,

Your English is far far better than my ability to try to communicate in your native language. So thank you for your patience and communicating with me in my native tongue.

You are right, metadata does not collect those metrics. So having another process do that makes some sense. We look forward to seeing your work Kei.

Thanks
Jim
________________________________________
From: Kei Ohmura <ohmura.kei@...>
Sent: Sunday, February 11, 2018 7:01 AM
To: White2, James
Cc: Keith Steele; steve@...; OHMURA Kei
Subject: Re: EdgeX Foundry exporter for Prometheus

Hi Jim,


I answer the question 2).

2) We have often thought about a more generic north side interface to
meta data (and even command) that would allow applications (like
Prometheus) access to Metadata and/or command information. so the
idea of an interface to these is a good idea. What I am trying to
figure out is what would your edgex-exporter-for-Prometheus do that
is not being done by the APIs of Metadata today? What extra servic
or capability is offered by this exporter? Looks like you are
thinking metrics collection might be one thing? And if it is needed,
is there a way for us to make this generic for things other than
Prometheus as well??
I don't understand well what you mean because of my English skill,
sorry about that. Anyway what I'd like to do is to collect metrics of
EdgeX Foundry. As far as I know current Metadata doesn't support API
for exporting metrics like the following:

* edgex_total_events
* Total number of events which are inserted into CoreData.
* edgex_total_readings
* Total number of readings which are inserted into CoreData.
* edgex_total_services
* Total number of microservices connected to Consul.


Thanks,

2018-02-09 0:29 GMT+09:00 <James.White2@...>:
Hi Kei
thanks for this and I now have a much better understanding of what you want to do. This is good stuff!!

couple of questions...
1) why would prometheus need to pull the data from core data when the export distro could push it to Prometheus? Making requests of core data directly, while possible, is potentially impactful of a service that must be high throughput. If something (like Prometheus) on the north were to make too big of a request, it could impact the south side data collection. So is there something you think export distro is not going to be able to do to push Prometheus the data?

2) We have often thought about a more generic north side interface to meta data (and even command) that would allow applications (like Prometheus) access to Metadata and/or command information. so the idea of an interface to these is a good idea. What I am trying to figure out is what would your edgex-exporter-for-Prometheus do that is not being done by the APIs of Metadata today? What extra service or capability is offered by this exporter? Looks like you are thinking metrics collection might be one thing? And if it is needed, is there a way for us to make this generic for things other than Prometheus as well??

In general, I am very supportive of your effort and definitely would like to see a connector to Prometheus. I have copied Keith Steele and Steve Osselton from IoTech because I believe they are looking at some sore of connection to Prometheus as well - so we should make sure everyone's input is in the eventual design and implementation if we can.

________________________________________
From: Kei Ohmura <ohmura.kei@...>
Sent: Thursday, February 8, 2018 1:38 AM
To: White2, James
Cc: OHMURA Kei
Subject: RFC: EdgeX Foundry exporter for Prometheus

Hi Jim,

I'm sending my thought to you.


# Goal
My goal is to export metrics of EdgeX Foundry microservices to
management system which is running on cloud and to monitor edge
servers including microservices which are running on it.


# Why prometheus
Prometheus is an open-source systems monitoring and alerting toolkit
with an active ecosystem and users.
Prometheus has many exporters:
https://github.com/prometheus/docs/blob/master/content/docs/instrumen
ting/exporters.md

So we can monitor not only EdgeX Foundry but also other microservices
running on edge servers just by implementing a exporter for EdgeX
Foundry.


# Architecture
Please refer to the attached file.
If you'd like to get hardware and OS metrics (such as cpu usage), you
only need to run
node_exporter(https://github.com/prometheus/node_exporter) on the
edge server.

I think that we need to implement similar functions into export
services (distrubution?) for users who don't use prometheus.


# Export metrics
I've already implemented the below:
(https://github.com/ohmk/edgex_exporter)

* edgex_total_devices
* Total number devices connected to EdgeX Foundry.
* edgex_total_events
* Total number of events which are inserted into CoreData.
* edgex_total_readings
* Total number of readings which are inserted into CoreData.
* edgex_total_services
* Total number of microservices connected to Consul.


# TODO
We need to discuss the following:
* What metrics each microservice should export
* Also export services should have api for metrics?
* ex. how many clients registered?
* Defining and implementing API of microservices to get metrics

I think that the above is very important regardless of Prometheus
support.


Thanks,
Kei
_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang