Topics

Core-Data Refactor Available for Comments

Trevor.Conn@...
 

Dell Customer Communication

Hi folks – Thanks to those who attended the presentation this morning of the refactoring strategy I’ve applied to core-data-go over the last couple weeks. If you weren’t able to attend the call but are interested in reviewing the changes, please watch the recording of the meeting available here:

 

https://zoom.us/recording/play/NfVb5k1ji1EnHTE0Bc_2Lzfw96F9qb5KgXHlhUv8WUbTwo09nNIfbdJHAkCV4Rha

 

The refactoring presentation starts at the 25 minute mark.

 

Here is the URL to the repo in our holding area:

https://github.com/edgexfoundry-holding/core-data-refactor

 

To briefly summarize, here are the primary points of interest in the refactoring:

1.)    The following application services are now initialized through a factory which returns an interface:

a.       Routing

b.      Metadata clients (Device Client / Service Client)

c.       MQPublisher

2.)    Removal of global variables at the package level to access application services

3.)    Business logic moved out of controllers and into aggregates folder

a.       Note the use of channels in core/aggregates/events/event.go::AddNewEvent() to increase separation of concerns.

4.)    Routing and controller implementation moved into a specific package and out of core/data package

5.)    Unit test coverage focusing on the aggregate – the domain business logic – leaving out application services (like routing)

a.       See core/aggregates/events/providers.go for abstraction definitions

b.      See core/aggregates/events/event_test.go for mock initialization and “injection”

 

As I said on the call, there is more work that could be done here but this is a significant chunk that establishes the direction the Core Services team would like to move toward. We are looking to the community to identify any major gaps or challenges that we may have missed in taking this approach. Initial testing on this work via the blackbox testing is going well, but additional changes may come in as we continue to test and refine.

 

Comments are welcome. We may need to set up some review calls or, if you like, you can create issues/PRs against the repo.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Trevor.Conn@...
 

Just a reminder that this proposed change is available for review. I've received some minor comments from our contributor yanghua so far, but that's it.


Please let me know of any comments you may have (via GitHub issues or email) by this coming Thursday, April 26.


Trevor Conn
Senior Principal Software Engineer
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA


From: Conn, Trevor
Sent: Thursday, April 12, 2018 12:40 PM
To: EdgeX-GoLang@...
Cc: edgex-tsc-core@...
Subject: Core-Data Refactor Available for Comments
 

Dell Customer Communication

Hi folks – Thanks to those who attended the presentation this morning of the refactoring strategy I’ve applied to core-data-go over the last couple weeks. If you weren’t able to attend the call but are interested in reviewing the changes, please watch the recording of the meeting available here:

 

https://zoom.us/recording/play/NfVb5k1ji1EnHTE0Bc_2Lzfw96F9qb5KgXHlhUv8WUbTwo09nNIfbdJHAkCV4Rha

 

The refactoring presentation starts at the 25 minute mark.

 

Here is the URL to the repo in our holding area:

https://github.com/edgexfoundry-holding/core-data-refactor

 

To briefly summarize, here are the primary points of interest in the refactoring:

1.)    The following application services are now initialized through a factory which returns an interface:

a.       Routing

b.      Metadata clients (Device Client / Service Client)

c.       MQPublisher

2.)    Removal of global variables at the package level to access application services

3.)    Business logic moved out of controllers and into aggregates folder

a.       Note the use of channels in core/aggregates/events/event.go::AddNewEvent() to increase separation of concerns.

4.)    Routing and controller implementation moved into a specific package and out of core/data package

5.)    Unit test coverage focusing on the aggregate – the domain business logic – leaving out application services (like routing)

a.       See core/aggregates/events/providers.go for abstraction definitions

b.      See core/aggregates/events/event_test.go for mock initialization and “injection”

 

As I said on the call, there is more work that could be done here but this is a significant chunk that establishes the direction the Core Services team would like to move toward. We are looking to the community to identify any major gaps or challenges that we may have missed in taking this approach. Initial testing on this work via the blackbox testing is going well, but additional changes may come in as we continue to test and refine.

 

Comments are welcome. We may need to set up some review calls or, if you like, you can create issues/PRs against the repo.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Drasko DRASKOVIC <drasko@...>
 

Hi Trevor,
idea to move towards DDD looks great to me.

If you are moving in this direction, I think it would be interesting
to examine go-kit: https://gokit.io/. It divides domains and provides
onion architecture, while providing logging and instrumentation. It
tends to collect and incorporate best practices.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic

On Fri, Apr 20, 2018 at 11:18 PM, <Trevor.Conn@...> wrote:
Just a reminder that this proposed change is available for review. I've
received some minor comments from our contributor yanghua so far, but that's
it.


Please let me know of any comments you may have (via GitHub issues or email)
by this coming Thursday, April 26.


Trevor Conn
Senior Principal Software Engineer
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX USA
________________________________
From: Conn, Trevor
Sent: Thursday, April 12, 2018 12:40 PM
To: EdgeX-GoLang@...
Cc: edgex-tsc-core@...
Subject: Core-Data Refactor Available for Comments


Dell Customer Communication

Hi folks – Thanks to those who attended the presentation this morning of the
refactoring strategy I’ve applied to core-data-go over the last couple
weeks. If you weren’t able to attend the call but are interested in
reviewing the changes, please watch the recording of the meeting available
here:



https://zoom.us/recording/play/NfVb5k1ji1EnHTE0Bc_2Lzfw96F9qb5KgXHlhUv8WUbTwo09nNIfbdJHAkCV4Rha



The refactoring presentation starts at the 25 minute mark.



Here is the URL to the repo in our holding area:

https://github.com/edgexfoundry-holding/core-data-refactor



To briefly summarize, here are the primary points of interest in the
refactoring:

1.) The following application services are now initialized through a
factory which returns an interface:

a. Routing

b. Metadata clients (Device Client / Service Client)

c. MQPublisher

2.) Removal of global variables at the package level to access
application services

3.) Business logic moved out of controllers and into aggregates folder

a. Note the use of channels in
core/aggregates/events/event.go::AddNewEvent() to increase separation of
concerns.

4.) Routing and controller implementation moved into a specific package
and out of core/data package

5.) Unit test coverage focusing on the aggregate – the domain business
logic – leaving out application services (like routing)

a. See core/aggregates/events/providers.go for abstraction definitions

b. See core/aggregates/events/event_test.go for mock initialization and
“injection”



As I said on the call, there is more work that could be done here but this
is a significant chunk that establishes the direction the Core Services team
would like to move toward. We are looking to the community to identify any
major gaps or challenges that we may have missed in taking this approach.
Initial testing on this work via the blackbox testing is going well, but
additional changes may come in as we continue to test and refine.



Comments are welcome. We may need to set up some review calls or, if you
like, you can create issues/PRs against the repo.



Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA



James.White2@...
 

Dell - Internal Use - Confidential

Hi Drasko,
A question for my help and understanding... back in November, we all looked at GoKit and I recall a number of members (including you and Janko if I recall correctly) suggested GoKit came with some complexities that you all felt did not serve the community well (we even noted that in our architectural notes to discourage its use by our community). Trevor did a relook and we can investigate again going forward, but I am curious if you and others in the Go community feel it has made changes (or our project has made enough advancement) that GoKit may be warranted. What are your thoughts now in light of the current direction??

jim

-----Original Message-----
From: EdgeX-GoLang@... [mailto:EdgeX-GoLang@...] On Behalf Of Drasko DRASKOVIC
Sent: Saturday, April 21, 2018 8:30 AM
To: Conn, Trevor
Cc: EdgeX-GoLang@...; edgex-tsc-core@...
Subject: Re: [Edgex-golang] Core-Data Refactor Available for Comments

Hi Trevor,
idea to move towards DDD looks great to me.

If you are moving in this direction, I think it would be interesting to examine go-kit: https://gokit.io/. It divides domains and provides onion architecture, while providing logging and instrumentation. It tends to collect and incorporate best practices.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic


On Fri, Apr 20, 2018 at 11:18 PM, <Trevor.Conn@...> wrote:
Just a reminder that this proposed change is available for review.
I've received some minor comments from our contributor yanghua so far,
but that's it.


Please let me know of any comments you may have (via GitHub issues or
email) by this coming Thursday, April 26.


Trevor Conn
Senior Principal Software Engineer
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX USA
________________________________
From: Conn, Trevor
Sent: Thursday, April 12, 2018 12:40 PM
To: EdgeX-GoLang@...
Cc: edgex-tsc-core@...
Subject: Core-Data Refactor Available for Comments


Dell Customer Communication

Hi folks – Thanks to those who attended the presentation this morning
of the refactoring strategy I’ve applied to core-data-go over the last
couple weeks. If you weren’t able to attend the call but are
interested in reviewing the changes, please watch the recording of the
meeting available
here:



https://zoom.us/recording/play/NfVb5k1ji1EnHTE0Bc_2Lzfw96F9qb5KgXHlhUv
8WUbTwo09nNIfbdJHAkCV4Rha



The refactoring presentation starts at the 25 minute mark.



Here is the URL to the repo in our holding area:

https://github.com/edgexfoundry-holding/core-data-refactor



To briefly summarize, here are the primary points of interest in the
refactoring:

1.) The following application services are now initialized through a
factory which returns an interface:

a. Routing

b. Metadata clients (Device Client / Service Client)

c. MQPublisher

2.) Removal of global variables at the package level to access
application services

3.) Business logic moved out of controllers and into aggregates folder

a. Note the use of channels in
core/aggregates/events/event.go::AddNewEvent() to increase separation
of concerns.

4.) Routing and controller implementation moved into a specific package
and out of core/data package

5.) Unit test coverage focusing on the aggregate – the domain business
logic – leaving out application services (like routing)

a. See core/aggregates/events/providers.go for abstraction definitions

b. See core/aggregates/events/event_test.go for mock initialization and
“injection”



As I said on the call, there is more work that could be done here but
this is a significant chunk that establishes the direction the Core
Services team would like to move toward. We are looking to the
community to identify any major gaps or challenges that we may have missed in taking this approach.
Initial testing on this work via the blackbox testing is going well,
but additional changes may come in as we continue to test and refine.



Comments are welcome. We may need to set up some review calls or, if
you like, you can create issues/PRs against the repo.



Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA



Drasko DRASKOVIC <drasko@...>
 

Hi Jim,

On Sat, Apr 21, 2018 at 8:04 PM, <James.White2@...> wrote:
Dell - Internal Use - Confidential

Hi Drasko,
A question for my help and understanding... back in November, we all looked at GoKit and I recall a number of members (including you and Janko if I recall correctly) suggested GoKit came with some complexities that you all felt did not serve the community well (we even noted that in our architectural notes to discourage its use by our community). Trevor did a relook and we can investigate again going forward, but I am curious if you and others in the Go community feel it has made changes (or our project has made enough advancement) that GoKit may be warranted. What are your thoughts now in light of the current direction??
My concerns with go-kit at the time was that newcomers to Go
programming would be overwhelmed, because go-kit imposes a certain
framework that demands more discipline. However, go-kit actually tries
to collect and follow the best architectural practices, and for that
reason it must introduce certain conventions.

When I saw that Trevor headed in DDD direction (which I welcome), I
realized that it would be potentially beneficial to leverage go-kit,
because we started to make architecture more complex anyway (and
hopefully now we are all skilled Gophers ;)). This is maybe
unavoidable - as soon as we go beyond PoC and start building
enterprise product, then subjects of testing, logging and
instrumenting become very important.

I proposed re-evaluation of go-kit to avoid potentially reinventing
the wheel (i.e. inventing our file structures and similar -
practically re-inventing the framework). Mainflux project has been
using go-kit for quite some time (even before discussion of it's usage
in EdgeX), and for the same reasons that we face now in EdgeX - domain
division, logging, instrumenting, etc. And we are also following DDD
practice - for example with repositories (will be familiar to Trevor):
https://github.com/mainflux/mainflux/blob/master/manager/manager.go#L14
and aggregates, but also with logging:
https://github.com/mainflux/mainflux/blob/master/manager/api/logging.go
(https://gokit.io/faq/#logging-mdash-how-should-i-aggregate-my-logs)
and metrics: https://github.com/mainflux/mainflux/blob/master/manager/api/metrics.go
(https://gokit.io/faq/#observability-mdash-which-monitoring-systems-are-supported)
that are handled through onion architecture that framework imposes.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic

James.White2@...
 

Dell - Internal Use - Confidential

Thanks Drasko. Really good feedback. I'll address with the team on my return from Hannover Messe.
Jim

-----Original Message-----
From: Drasko DRASKOVIC [mailto:drasko@...]
Sent: Saturday, April 21, 2018 11:11 PM
To: White2, James
Cc: Conn, Trevor; EdgeX-GoLang@...; edgex-tsc-core@...
Subject: Re: [Edgex-golang] Core-Data Refactor Available for Comments

Hi Jim,

On Sat, Apr 21, 2018 at 8:04 PM, <James.White2@...> wrote:
Dell - Internal Use - Confidential

Hi Drasko,
A question for my help and understanding... back in November, we all looked at GoKit and I recall a number of members (including you and Janko if I recall correctly) suggested GoKit came with some complexities that you all felt did not serve the community well (we even noted that in our architectural notes to discourage its use by our community). Trevor did a relook and we can investigate again going forward, but I am curious if you and others in the Go community feel it has made changes (or our project has made enough advancement) that GoKit may be warranted. What are your thoughts now in light of the current direction??
My concerns with go-kit at the time was that newcomers to Go programming would be overwhelmed, because go-kit imposes a certain framework that demands more discipline. However, go-kit actually tries to collect and follow the best architectural practices, and for that reason it must introduce certain conventions.

When I saw that Trevor headed in DDD direction (which I welcome), I realized that it would be potentially beneficial to leverage go-kit, because we started to make architecture more complex anyway (and hopefully now we are all skilled Gophers ;)). This is maybe unavoidable - as soon as we go beyond PoC and start building enterprise product, then subjects of testing, logging and instrumenting become very important.

I proposed re-evaluation of go-kit to avoid potentially reinventing the wheel (i.e. inventing our file structures and similar - practically re-inventing the framework). Mainflux project has been using go-kit for quite some time (even before discussion of it's usage in EdgeX), and for the same reasons that we face now in EdgeX - domain division, logging, instrumenting, etc. And we are also following DDD practice - for example with repositories (will be familiar to Trevor):
https://github.com/mainflux/mainflux/blob/master/manager/manager.go#L14
and aggregates, but also with logging:
https://github.com/mainflux/mainflux/blob/master/manager/api/logging.go
(https://gokit.io/faq/#logging-mdash-how-should-i-aggregate-my-logs)
and metrics: https://github.com/mainflux/mainflux/blob/master/manager/api/metrics.go
(https://gokit.io/faq/#observability-mdash-which-monitoring-systems-are-supported)
that are handled through onion architecture that framework imposes.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic