Date   
Re: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages

Steve Osselton
 

Hi Trevor,

This all looks pretty sensible to me.

Cheers Steve.

On 30 June 2018 at 02:41, <Trevor.Conn@...> wrote:

Dell Customer Communication

Hi all – I wanted to send out an email as a heads-up to one of the decisions that came from our last F2F in Palo Alto. This was discussed during the Device Services working group and I wanted to give everyone visibility and an opportunity to comment before changes get made.

 

The Problem

Currently the EdgeX GoLang implementation has no separation between its internal domain model and the models used to pass data out of the system. They are one and the same. Likewise, there is no visibility as to which service client is used by an external application versus solely used internal to the EdgeX platform. This means we are at risk of making changes to either our data models or our service client code that break any applications which have imported those items. We have no way to insulate our consumers from changes we make to the application based on an internal need, nor do we have the ability to hide internal/private details of our data model since our data model is also part of our client contract.

 

The Recommendation

In order to facilitate the separation of internal/external domain models and logic, we realized that we would need to restructure the application packages. At a high level, the changes we decided to make are as follows:

 

/edgex-go/core

·         will be re-purposed to contain those models and libraries which are usable by external applications

·         will include the following:

o   clients package – service clients usable by external applications

o   models package – data models usable by external applications

§  NOTE: These will be identical to the current property bags found today in /edgex-go/core/domain/models

/edgex-go/internal

·         This package will be extended to include all code that we do not want external applications explicitly relying on as recommended by the golang-standards project layout

·         That means the following packages will be moved into here in some fashion

o   /edgex-go/core/command

o   /edgex-go/core/core-data

o   /edgex-go/core/core-metadata

o   /edgex-go/export

o   /edgex-go/support

§  Consul-client, domain, l ogging

§  Service client packages will most likely go to core above

·         Domain models included herein may start off as identical to those found today in /edgex-go/core/domain/models but have the flexibility to change and move around over time according to the internal needs of the platform.

                         

We are looking at making this change early in the Delhi development cycle, so please provide feedback in a timely fashion if you have it. Also please note that any external applications which are directly linking to the current package structure within the master branch of EdgeX will break when we do this. That is unavoidable. We are doing this now so that the pill isn’t too bitter to swallow later.

 

I look forward to your comments.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 




--
Technical Director
IOTech Systems Ltd.

Re: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages

Trevor.Conn@...
 

Dell Customer Communication

To continue the thread, I wanted to give visibility to Issue #368. I’ve submitted the first PR against that issue to address the items below and after doing some work, my thinking has shifted a touch. Please see below:

 

I'm about to submit the first PR for this issue in which the majority of the delineated code has been moved to its new location. I'm adding this comment to record my thoughts at this point on what the end goal is going to look like. There will be a couple more subsequent PRs I think.

  • edgex-go/core to be renamed to edgex-go/external
    All code importable by 3rd party applications wishing to integrate with EdgeX will go here
  • edgex-go/core/domain to be removed and models moved up to edgex-go/external/models.
    These models will serve as the data model for external client application consumption
  • edgex-go/core/domain/enums will be collapsed into the above models package
  • Instead of an edgex-go/internal/app directory containing all of the services, I decided to preserve the "core" / "export" / "support" packages as representative of the logical tiers of the platform. Therefore the service implementation code will go into a respective directory such as internal/core/metadata or internal/export/distro
  • We have to refactor database-specific types out of our domain model (such as the BSON ObjectIDs)
    This should instead be a string on the model and the Mongo provider itself will be responsible for validating any platform specific datatype requirements.

We had previously discussed keeping the edgex-go/core directory as the location of externally usable code but I’m thinking an edgex-go/external directory makes more sense, as described above. The term “core” has a specific meaning within EdgeX, referring to a specific tier within the platform.

 

The following directories will ultimately go away:

 

Edgex-go/core

Edgex-go/export

Edgex-go/support

 

Their respective clients would go into the edgex-go/external/clients package and their implementation logic would go into

 

edgex-go/internal/core

edgex-go/internal/export

edgex-go/internal/support

 

In particular I want to see if there is any objection to the external package and proposed grouping.

 

Work done so far is here PR #369

 

Thanks,

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

 

 

From: Steve Osselton [mailto:steve@...]
Sent: Monday, July 2, 2018 7:39 AM
To: Conn, Trevor
Cc: EdgeX-GoLang@...; edgex-tsc-core@...
Subject: Re: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages

 

Hi Trevor,

 

This all looks pretty sensible to me.

 

Cheers Steve.

 

On 30 June 2018 at 02:41, <Trevor.Conn@...> wrote:

Dell Customer Communication

Hi all – I wanted to send out an email as a heads-up to one of the decisions that came from our last F2F in Palo Alto. This was discussed during the Device Services working group and I wanted to give everyone visibility and an opportunity to comment before changes get made.

 

The Problem

Currently the EdgeX GoLang implementation has no separation between its internal domain model and the models used to pass data out of the system. They are one and the same. Likewise, there is no visibility as to which service client is used by an external application versus solely used internal to the EdgeX platform. This means we are at risk of making changes to either our data models or our service client code that break any applications which have imported those items. We have no way to insulate our consumers from changes we make to the application based on an internal need, nor do we have the ability to hide internal/private details of our data model since our data model is also part of our client contract.

 

The Recommendation

In order to facilitate the separation of internal/external domain models and logic, we realized that we would need to restructure the application packages. At a high level, the changes we decided to make are as follows:

 

/edgex-go/core

·         will be re-purposed to contain those models and libraries which are usable by external applications

·         will include the following:

o   clients package – service clients usable by external applications

o   models package – data models usable by external applications

§  NOTE: These will be identical to the current property bags found today in /edgex-go/core/domain/models

/edgex-go/internal

·         This package will be extended to include all code that we do not want external applications explicitly relying on as recommended by the golang-standards project layout

·         That means the following packages will be moved into here in some fashion

o   /edgex-go/core/command

o   /edgex-go/core/core-data

o   /edgex-go/core/core-metadata

o   /edgex-go/export

o   /edgex-go/support

§  Consul-client, domain, l ogging

§  Service client packages will most likely go to core above

·         Domain models included herein may start off as identical to those found today in /edgex-go/core/domain/models but have the flexibility to change and move around over time according to the internal needs of the platform.

                         

We are looking at making this change early in the Delhi development cycle, so please provide feedback in a timely fashion if you have it. Also please note that any external applications which are directly linking to the current package structure within the master branch of EdgeX will break when we do this. That is unavoidable. We are doing this now so that the pill isn’t too bitter to swallow later.

 

I look forward to your comments.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 



 

--

Technical Director

IOTech Systems Ltd.

Re: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages

Stella Yu
 

Hi Trevor,

 

I think it is a great idea to separate internal and external models. We went through a lot of pain by mixing the two and wished we had them separate a lot earlier.

 

As for package organizations, internal and external are clear to users, how about all the other directories then? Should everything outside of internal implies external? Our team is adopting this structure (https://github.com/golang-standards/project-layout). Instead of “external”, would things fit better inside “api” or “cmd”?

 

Cheers,

Stella

 

From: <EdgeX-TSC-Core@...> on behalf of "Trevor.Conn@..." <Trevor.Conn@...>
Date: Friday, June 29, 2018 at 6:42 PM
To: "EdgeX-GoLang@..." <EdgeX-GoLang@...>, "EdgeX-TSC-Core@..." <EdgeX-TSC-Core@...>
Subject: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages

 

Dell Customer Communication


Hi all – I wanted to send out an email as a heads-up to one of the decisions that came from our last F2F in Palo Alto. This was discussed during the Device Services working group and I wanted to give everyone visibility and an opportunity to comment before changes get made.

 

The Problem

Currently the EdgeX GoLang implementation has no separation between its internal domain model and the models used to pass data out of the system. They are one and the same. Likewise, there is no visibility as to which service client is used by an external application versus solely used internal to the EdgeX platform. This means we are at risk of making changes to either our data models or our service client code that break any applications which have imported those items. We have no way to insulate our consumers from changes we make to the application based on an internal need, nor do we have the ability to hide internal/private details of our data model since our data model is also part of our client contract.

 

The Recommendation

In order to facilitate the separation of internal/external domain models and logic, we realized that we would need to restructure the application packages. At a high level, the changes we decided to make are as follows:

 

/edgex-go/core

  • will be re-purposed to contain those models and libraries which are usable by external applications
  • will include the following:
    • clients package – service clients usable by external applications
    • models package – data models usable by external applications
      • NOTE: These will be identical to the current property bags found today in /edgex-go/core/domain/models

/edgex-go/internal

  • This package will be extended to include all code that we do not want external applications explicitly relying on as recommended by the golang-standards project layout
  • That means the following packages will be moved into here in some fashion
    • /edgex-go/core/command
    • /edgex-go/core/core-data
    • /edgex-go/core/core-metadata
    • /edgex-go/export
    • /edgex-go/support
      • Consul-client, domain, l ogging
      • Service client packages will most likely go to core above
  • Domain models included herein may start off as identical to those found today in /edgex-go/core/domain/models but have the flexibility to change and move around over time according to the internal needs of the platform.

                         

We are looking at making this change early in the Delhi development cycle, so please provide feedback in a timely fashion if you have it. Also please note that any external applications which are directly linking to the current package structure within the master branch of EdgeX will break when we do this. That is unavoidable. We are doing this now so that the pill isn’t too bitter to swallow later.

 

I look forward to your comments.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages

Trevor.Conn@...
 

Hi Stella -- Thanks for the feedback and your suggestions on the package separation for external into api and pkg. I have made those changes and have updated the relevant PR.


https://github.com/edgexfoundry/edgex-go/pull/369


Verify jobs are running now and then I just need someone from the community to approve and merge.


Trevor Conn
Senior Principal Software Engineer
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA


From: EdgeX-TSC-Core@... <EdgeX-TSC-Core@...> on behalf of Stella Yu <stella.yu@...>
Sent: Monday, July 9, 2018 1:38 PM
To: Conn, Trevor; EdgeX-GoLang@...; EdgeX-TSC-Core@...
Subject: Re: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages
 

Hi Trevor,

 

I think it is a great idea to separate internal and external models. We went through a lot of pain by mixing the two and wished we had them separate a lot earlier.

 

As for package organizations, internal and external are clear to users, how about all the other directories then? Should everything outside of internal implies external? Our team is adopting this structure (https://github.com/golang-standards/project-layout). Instead of “external”, would things fit better inside “api” or “cmd”?

 

Cheers,

Stella

 

From: <EdgeX-TSC-Core@...> on behalf of "Trevor.Conn@..." <Trevor.Conn@...>
Date: Friday, June 29, 2018 at 6:42 PM
To: "EdgeX-GoLang@..." <EdgeX-GoLang@...>, "EdgeX-TSC-Core@..." <EdgeX-TSC-Core@...>
Subject: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages

 

Dell Customer Communication


Hi all – I wanted to send out an email as a heads-up to one of the decisions that came from our last F2F in Palo Alto. This was discussed during the Device Services working group and I wanted to give everyone visibility and an opportunity to comment before changes get made.

 

The Problem

Currently the EdgeX GoLang implementation has no separation between its internal domain model and the models used to pass data out of the system. They are one and the same. Likewise, there is no visibility as to which service client is used by an external application versus solely used internal to the EdgeX platform. This means we are at risk of making changes to either our data models or our service client code that break any applications which have imported those items. We have no way to insulate our consumers from changes we make to the application based on an internal need, nor do we have the ability to hide internal/private details of our data model since our data model is also part of our client contract.

 

The Recommendation

In order to facilitate the separation of internal/external domain models and logic, we realized that we would need to restructure the application packages. At a high level, the changes we decided to make are as follows:

 

/edgex-go/core

  • will be re-purposed to contain those models and libraries which are usable by external applications
  • will include the following:
    • clients package – service clients usable by external applications
    • models package – data models usable by external applications
      • NOTE: These will be identical to the current property bags found today in /edgex-go/core/domain/models

/edgex-go/internal

  • This package will be extended to include all code that we do not want external applications explicitly relying on as recommended by the golang-standards project layout
  • That means the following packages will be moved into here in some fashion
    • /edgex-go/core/command
    • /edgex-go/core/core-data
    • /edgex-go/core/core-metadata
    • /edgex-go/export
    • /edgex-go/support
      • Consul-client, domain, l ogging
      • Service client packages will most likely go to core above
  • Domain models included herein may start off as identical to those found today in /edgex-go/core/domain/models but have the flexibility to change and move around over time according to the internal needs of the platform.

                         

We are looking at making this change early in the Delhi development cycle, so please provide feedback in a timely fashion if you have it. Also please note that any external applications which are directly linking to the current package structure within the master branch of EdgeX will break when we do this. That is unavoidable. We are doing this now so that the pill isn’t too bitter to swallow later.

 

I look forward to your comments.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages

espy
 

On 7/7/18 6:32 PM, Trevor.Conn@... wrote:

Dell Customer Communication

To continue the thread, I wanted to give visibility to Issue #368. I’ve submitted the first PR against that issue to address the items below and after doing some work, my thinking has shifted a touch. Please see below:

 

I'm about to submit the first PR for this issue in which the majority of the delineated code has been moved to its new location. I'm adding this comment to record my thoughts at this point on what the end goal is going to look like. There will be a couple more subsequent PRs I think.

  • edgex-go/core to be renamed to edgex-go/external
    All code importable by 3rd party applications wishing to integrate with EdgeX will go here
  • edgex-go/core/domain to be removed and models moved up to edgex-go/external/models.
    These models will serve as the data model for external client application consumption
  • edgex-go/core/domain/enums will be collapsed into the above models package
  • Instead of an edgex-go/internal/app directory containing all of the services, I decided to preserve the "core" / "export" / "support" packages as representative of the logical tiers of the platform. Therefore the service implementation code will go into a respective directory such as internal/core/metadata or internal/export/distro
  • We have to refactor database-specific types out of our domain model (such as the BSON ObjectIDs)
    This should instead be a string on the model and the Mongo provider itself will be responsible for validating any platform specific datatype requirements.
Would the Mongo provider (or other db providers) export a function that would allow other code to validate ids?

Also this still doesn't solve the problem that we're using actual database ids in our REST APIs.  It's probably outside the scope for Dehli, but I think we should give strong consideration to moving to a guid model for ids as Steve Osselton suggested at the last F2F.

There's also the dual-purpose nature of the Device member of some of our objects that should be addressed as well (the current solution is a band-aid).

Regards,
/tony


We had previously discussed keeping the edgex-go/core directory as the location of externally usable code but I’m thinking an edgex-go/external directory makes more sense, as described above. The term “core” has a specific meaning within EdgeX, referring to a specific tier within the platform.

 

The following directories will ultimately go away:

 

Edgex-go/core

Edgex-go/export

Edgex-go/support

 

Their respective clients would go into the edgex-go/external/clients package and their implementation logic would go into

 

edgex-go/internal/core

edgex-go/internal/export

edgex-go/internal/support

 

In particular I want to see if there is any objection to the external package and proposed grouping.

 

Work done so far is here PR #369

 

Thanks,

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

 

 

From: Steve Osselton [mailto:steve@...]
Sent: Monday, July 2, 2018 7:39 AM
To: Conn, Trevor
Cc: EdgeX-GoLang@...; edgex-tsc-core@...
Subject: Re: [Edgex-tsc-core] Delhi -- Cleaner Separation of Internal/External Packages

 

Hi Trevor,

 

This all looks pretty sensible to me.

 

Cheers Steve.

 

On 30 June 2018 at 02:41, <Trevor.Conn@...> wrote:

Dell Customer Communication

Hi all – I wanted to send out an email as a heads-up to one of the decisions that came from our last F2F in Palo Alto. This was discussed during the Device Services working group and I wanted to give everyone visibility and an opportunity to comment before changes get made.

 

The Problem

Currently the EdgeX GoLang implementation has no separation between its internal domain model and the models used to pass data out of the system. They are one and the same. Likewise, there is no visibility as to which service client is used by an external application versus solely used internal to the EdgeX platform. This means we are at risk of making changes to either our data models or our service client code that break any applications which have imported those items. We have no way to insulate our consumers from changes we make to the application based on an internal need, nor do we have the ability to hide internal/private details of our data model since our data model is also part of our client contract.

 

The Recommendation

In order to facilitate the separation of internal/external domain models and logic, we realized that we would need to restructure the application packages. At a high level, the changes we decided to make are as follows:

 

/edgex-go/core

·         will be re-purposed to contain those models and libraries which are usable by external applications

·         will include the following:

o   clients package – service clients usable by external applications

o   models package – data models usable by external applications

§  NOTE: These will be identical to the current property bags found today in /edgex-go/core/domain/models

/edgex-go/internal

·         This package will be extended to include all code that we do not want external applications explicitly relying on as recommended by the golang-standards project layout

·         That means the following packages will be moved into here in some fashion

o   /edgex-go/core/command

o   /edgex-go/core/core-data

o   /edgex-go/core/core-metadata

o   /edgex-go/export

o   /edgex-go/support

§  Consul-client, domain, l ogging

§  Service client packages will most likely go to core above

·         Domain models included herein may start off as identical to those found today in /edgex-go/core/domain/models but have the flexibility to change and move around over time according to the internal needs of the platform.

                         

We are looking at making this change early in the Delhi development cycle, so please provide feedback in a timely fashion if you have it. Also please note that any external applications which are directly linking to the current package structure within the master branch of EdgeX will break when we do this. That is unavoidable. We are doing this now so that the pill isn’t too bitter to swallow later.

 

I look forward to your comments.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 



 

--

Technical Director

IOTech Systems Ltd.


Docker-compose on the remote server

Drasko DRASKOVIC <drasko@...>
 

Hi all,
what's the main purpose of keeping the docker-compose.yaml on the
remote server (https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/compose-files/docker-compose-california-0.6.0.yml)?

It is obscuring the code/deployment, it is unclear form pure
observation of the code what is being run. Moreover, it is getting
harder to run and fine-tune the composition. If someone want to
propose changes to the docker-compose, how is this done, when file is
not in the VCS?

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Re: [Edgex-devel] Docker-compose on the remote server

Trevor.Conn@...
 

Dell Customer Communication

We had an extremely old and outdated docker-compose.yaml in the /edgex-go/docker directory and we fielded many questions from people who were new to EdgeX about why our compose file didn't work. It turned out they were all attempting to raise the containers from this old, outdated file.

From this we learned that the first thing many users will do in order to take EdgeX for an initial spin is cd into /edgex-go/docker and execute "docker-compose up". We put the mechanism in there to load the latest release docker-compose.yaml via curl so that the users would be able to easily load up the latest release. If they want to then tweak the file to use local images, that will be easy now that they have the file locally.

With regard to proposing changes to the compose file, that should be easy enough since the location is in the docker-launch.sh script. From there the user can see the appropriate repo against which to submit a PR.

Trevor

-----Original Message-----
From: EdgeX-Devel@... [mailto:EdgeX-Devel@...] On Behalf Of Drasko DRASKOVIC
Sent: Thursday, July 12, 2018 6:14 PM
To: edgex-devel@...; edgex-golang@...
Subject: [Edgex-devel] Docker-compose on the remote server

Hi all,
what's the main purpose of keeping the docker-compose.yaml on the remote server (https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/compose-files/docker-compose-california-0.6.0.yml)?

It is obscuring the code/deployment, it is unclear form pure observation of the code what is being run. Moreover, it is getting harder to run and fine-tune the composition. If someone want to propose changes to the docker-compose, how is this done, when file is not in the VCS?

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Re: [Edgex-devel] Docker-compose on the remote server

Drasko DRASKOVIC <drasko@...>
 

Hi Trevor,

On Fri, Jul 13, 2018 at 1:21 AM, <Trevor.Conn@...> wrote:
Dell Customer Communication

We had an extremely old and outdated docker-compose.yaml in the /edgex-go/docker directory and we fielded many questions from people who were new to EdgeX about why our compose file didn't work. It turned out they were all attempting to raise the containers from this old, outdated file.

From this we learned that the first thing many users will do in order to take EdgeX for an initial spin is cd into /edgex-go/docker and execute "docker-compose up". We put the mechanism in there to load the latest release docker-compose.yaml via curl so that the users would be able to easily load up the latest release. If they want to then tweak the file to use local images, that will be easy now that they have the file locally.

With regard to proposing changes to the compose file, that should be easy enough since the location is in the docker-launch.sh script. From there the user can see the appropriate repo against which to submit a PR.
OK, got it. You want that we maintain docker-compose.yaml over here:
https://github.com/edgexfoundry/developer-scripts/blob/master/compose-files/docker-compose.yml

We can maybe add this info README, it will be easier to spot than
opening the scripts, etc...

BR,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Re: [Edgex-devel] Docker-compose on the remote server

James.White2@...
 

Hi Drasko,

I am sorry, I am not following your question. Are you asking why we have a docker compose file tied to a particular version (and therefore multiple files)? Or are you asking how to make recommendations to change the docker compose files? The docker compose file is version controlled.
jim

-----Original Message-----
From: EdgeX-Devel@... [mailto:EdgeX-Devel@...] On Behalf Of Drasko DRASKOVIC
Sent: Thursday, July 12, 2018 6:14 PM
To: edgex-devel@...; edgex-golang@...
Subject: [Edgex-devel] Docker-compose on the remote server

Hi all,
what's the main purpose of keeping the docker-compose.yaml on the remote server (https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/compose-files/docker-compose-california-0.6.0.yml)?

It is obscuring the code/deployment, it is unclear form pure observation of the code what is being run. Moreover, it is getting harder to run and fine-tune the composition. If someone want to propose changes to the docker-compose, how is this done, when file is not in the VCS?

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Re: [Edgex-devel] Docker-compose on the remote server

James.White2@...
 

Thanks Drasko - saw Trevor and your exchange after I sent. Good call on addition to READMe. Will review/approve it goes through checks.
Jim

-----Original Message-----
From: Drasko DRASKOVIC [mailto:drasko@...]
Sent: Thursday, July 12, 2018 7:16 PM
To: White2, James
Cc: edgex-devel@...; edgex-golang@...
Subject: Re: [Edgex-devel] Docker-compose on the remote server

Hi Jim,
no problem, I saw that chosen location is in other repo.

I've sent PR that puts this info in the README to avoid confusion:
https://github.com/edgexfoundry/edgex-go/pull/391

BR,
Drasko


On Fri, Jul 13, 2018 at 1:35 AM, <James.White2@...> wrote:
Hi Drasko,

I am sorry, I am not following your question. Are you asking why we have a docker compose file tied to a particular version (and therefore multiple files)? Or are you asking how to make recommendations to change the docker compose files? The docker compose file is version controlled.
jim
-----Original Message-----
From: EdgeX-Devel@... [mailto:EdgeX-Devel@...] On Behalf Of Drasko DRASKOVIC
Sent: Thursday, July 12, 2018 6:14 PM
To: edgex-devel@...; edgex-golang@...
Subject: [Edgex-devel] Docker-compose on the remote server

Hi all,
what's the main purpose of keeping the docker-compose.yaml on the remote server (https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/compose-files/docker-compose-california-0.6.0.yml)?

It is obscuring the code/deployment, it is unclear form pure observation of the code what is being run. Moreover, it is getting harder to run and fine-tune the composition. If someone want to propose changes to the docker-compose, how is this done, when file is not in the VCS?

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic


--
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Re: [Edgex-devel] Docker-compose on the remote server

Drasko DRASKOVIC <drasko@...>
 

Hi Jim,
no problem, I saw that chosen location is in other repo.

I've sent PR that puts this info in the README to avoid confusion:
https://github.com/edgexfoundry/edgex-go/pull/391

BR,
Drasko

On Fri, Jul 13, 2018 at 1:35 AM, <James.White2@...> wrote:
Hi Drasko,

I am sorry, I am not following your question. Are you asking why we have a docker compose file tied to a particular version (and therefore multiple files)? Or are you asking how to make recommendations to change the docker compose files? The docker compose file is version controlled.
jim
-----Original Message-----
From: EdgeX-Devel@... [mailto:EdgeX-Devel@...] On Behalf Of Drasko DRASKOVIC
Sent: Thursday, July 12, 2018 6:14 PM
To: edgex-devel@...; edgex-golang@...
Subject: [Edgex-devel] Docker-compose on the remote server

Hi all,
what's the main purpose of keeping the docker-compose.yaml on the remote server (https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/compose-files/docker-compose-california-0.6.0.yml)?

It is obscuring the code/deployment, it is unclear form pure observation of the code what is being run. Moreover, it is getting harder to run and fine-tune the composition. If someone want to propose changes to the docker-compose, how is this done, when file is not in the VCS?

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic


--
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Re: [Edgex-devel] Docker-compose on the remote server

Drasko DRASKOVIC <drasko@...>
 

Argh, I accidentally removed Snap package info.

Let me correct this.

BR,
Drasko

On Fri, Jul 13, 2018 at 2:16 AM, Drasko DRASKOVIC <drasko@...> wrote:
Hi Jim,
no problem, I saw that chosen location is in other repo.

I've sent PR that puts this info in the README to avoid confusion:
https://github.com/edgexfoundry/edgex-go/pull/391

BR,
Drasko


On Fri, Jul 13, 2018 at 1:35 AM, <James.White2@...> wrote:
Hi Drasko,

I am sorry, I am not following your question. Are you asking why we have a docker compose file tied to a particular version (and therefore multiple files)? Or are you asking how to make recommendations to change the docker compose files? The docker compose file is version controlled.
jim
-----Original Message-----
From: EdgeX-Devel@... [mailto:EdgeX-Devel@...] On Behalf Of Drasko DRASKOVIC
Sent: Thursday, July 12, 2018 6:14 PM
To: edgex-devel@...; edgex-golang@...
Subject: [Edgex-devel] Docker-compose on the remote server

Hi all,
what's the main purpose of keeping the docker-compose.yaml on the remote server (https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/compose-files/docker-compose-california-0.6.0.yml)?

It is obscuring the code/deployment, it is unclear form pure observation of the code what is being run. Moreover, it is getting harder to run and fine-tune the composition. If someone want to propose changes to the docker-compose, how is this done, when file is not in the VCS?

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic




--
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter: @draskodraskovic
--
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Core WG Meeting -- CBOR Proposal Diagram

Trevor.Conn@...
 

Dell Customer Communication

Hi all – As discussed during last week’s Core working group call, we will take some time this week to review a high level proposal for the implementation of CBOR ingestion and export within EdgeX. I have prepared some slides which you can find here:

 

https://wiki.edgexfoundry.org/display/FA/Core+Working+Group

 

Scroll down to the entry for the July 19th meeting and you’ll see “CBOR Integration Proposal_v1.pdf”

 

This is to give any parties who are interested an opportunity to review the slides before we discuss on Thursday morning and to come prepared with any questions they may have.

 

Talk to you then. Thanks!

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Decoupling Models From Mongo

Trevor.Conn@...
 

Hi all -- As part of the effort to separate EdgeX's internal and external model representations, we've made good progress in restructuring the project packages to clearly delineate which code is publicly exportable and which code is usable only my EdgeX internals. Currently, all application logic -- both internal and external -- is pointed to the externally available models located in pkg/models. The long term goal is to change that so that EdgeX has an internal representation of its model which can be changed independently of the contract model.


One of the challenges I've been wrestling with is the coupling we have to Mongo. Our models currently specify BSON serialization metadata (something no client should ever care about) as well as BSON.ObjectID properties and our application logic manages the creation and assignment of BSON.ObjectIDs rather than the underlying database platform. In the future state, if we'd like for the database to be swappable, we have to eliminate explicit knowledge of how to manage database keys and push any metadata about translating a type to a database representation down into the database layer.


Fede has made a good start down a path that will facilitate cleaner separation with his types in the db/mongo package, but there's more work that needs to be done. I want to summarize some points that I have in mind based on a day and a half of wrestling with how to achieve a clean separation between our business logic and the requirements of the underlying storage platform. These should not be taken as decisions, but visibility into what I'm currently thinking and an open request for comment.


  • Push all key and referential integrity management down to the database
    • This means no object should expose the internal primary key/_id value to the application above the database layer (like db/mongo above)
    • This means we standardize on a user-specified UUID which identifies entries and is the primary means by which we perform lookups from the application layer
    • This means the given UUID field/column needs to be explictly indexed
      • Mongo example: db.event.createIndex({uuid: 1}, {name: "ix_event_uuid"})
    • This eliminates the need to force myriad, native database key datatypes into a generic representation (like string) for later verification and casting.
    • No application logic should ever be responsible for assigning these values
  • For models at the database layer (like db/mongo above), suggest we use "Id" for the property containing the actual database key value
  • For models at all other layers (which will NOT have the above property) suggest we use "Key" as a string containing the user-defined UUID
  • In cases where it is necessary for the database layer to obtain IDs of newly inserted records, we must require that the "driver" used by EdgeX is capable of providing this functionalty.
    • Example use case: see the AddEvent function
      • Our current mongo driver (gopkg.in/mgo.v2/bson) is incapable of returning the IDs of the newly inserted records for the readings, and so we must create those IDs in the application logic so that the DBRefs are aligned correctly when the event is stored.
      • If, however, we used the official Mongo go provider we would be able to obtain the IDs of newly inserted records due to support for Mongo's insertOne and insertMany capabilities.
    • NOTE: Our current mgo.v2 driver is no longer maintained, so we need to switch anyway!!!
Thanks for your attention in reading this. I'm not going to proceed any further down this path until I can get some feedback and clarity on the above points. As I said at the beginning, there are real benefits to the work done so far and once the export services have been re-organized to fit in the new package structure, we'll be in much better shape. But there's more work to be done and the primary intent of the original issue I created hasn't been addressed yet.

To ensure we follow through on making sure EdgeX is flexible enough to operate in a customer-defined environment, we have to decouple vendor-specific logic and types from the main application.


Trevor Conn
Senior Principal Software Engineer
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA

Re: Decoupling Models From Mongo

espy
 

On 7/20/18 12:22 PM, Trevor.Conn@... wrote:

Hi all -- As part of the effort to separate EdgeX's internal and external model representations, we've made good progress in restructuring the project packages to clearly delineate which code is publicly exportable and which code is usable only my EdgeX internals. Currently, all application logic -- both internal and external -- is pointed to the externally available models located in pkg/models. The long term goal is to change that so that EdgeX has an internal representation of its model which can be changed independently of the contract model.


One of the challenges I've been wrestling with is the coupling we have to Mongo. Our models currently specify BSON serialization metadata (something no client should ever care about) as well as BSON.ObjectID properties and our application logic manages the creation and assignment of BSON.ObjectIDs rather than the underlying database platform. In the future state, if we'd like for the database to be swappable, we have to eliminate explicit knowledge of how to manage database keys and push any metadata about translating a type to a database representation down into the database layer.


Fede has made a good start down a path that will facilitate cleaner separation with his types in the db/mongo package, but there's more work that needs to be done. I want to summarize some points that I have in mind based on a day and a half of wrestling with how to achieve a clean separation between our business logic and the requirements of the underlying storage platform. These should not be taken as decisions, but visibility into what I'm currently thinking and an open request for comment.


  • Push all key and referential integrity management down to the database
    • This means no object should expose the internal primary key/_id value to the application above the database layer (like db/mongo above)
    • This means we standardize on a user-specified UUID which identifies entries and is the primary means by which we perform lookups from the application layer
+1
    • This means the given UUID field/column needs to be explictly indexed
      • Mongo example: db.event.createIndex({uuid: 1}, {name: "ix_event_uuid"})
Are their penalties for for having more than field being indexed?

    • This eliminates the need to force myriad, native database key datatypes into a generic representation (like string) for later verification and casting.
    • No application logic should ever be responsible for assigning these values
This likely means we need constructors for all of our external objects which use a common newGUUID() function.

  • For models at the database layer (like db/mongo above), suggest we use "Id" for the property containing the actual database key value
  • For models at all other layers (which will NOT have the above property) suggest we use "Key" as a string containing the user-defined UUID
I don't love this, as key is usually used in the context of SQL-like databases.  I think the GUIID should be identified as "Id", and internal DB keys should use "key" or "dbId", or something along those lines.
  • In cases where it is necessary for the database layer to obtain IDs of newly inserted records, we must require that the "driver" used by EdgeX is capable of providing this functionalty.
    • Example use case: see the AddEvent function
      • Our current mongo driver (gopkg.in/mgo.v2/bson) is incapable of returning the IDs of the newly inserted records for the readings, and so we must create those IDs in the application logic so that the DBRefs are aligned correctly when the event is stored.
      • If, however, we used the official Mongo go provider we would be able to obtain the IDs of newly inserted records due to support for Mongo's insertOne and insertMany capabilities.
    • NOTE: Our current mgo.v2 driver is no longer maintained, so we need to switch anyway!!!
+1

While listening to the Mongo presentation yesterday I decided to take a look at the license of mgo.v2 (written by one of my co-workers), and noticed it's *unsupported*.

Thanks for your attention in reading this. I'm not going to proceed any further down this path until I can get some feedback and clarity on the above points. As I said at the beginning, there are real benefits to the work done so far and once the export services have been re-organized to fit in the new package structure, we'll be in much better shape. But there's more work to be done and the primary intent of the original issue I created hasn't been addressed yet.

To ensure we follow through on making sure EdgeX is flexible enough to operate in a customer-defined environment, we have to decouple vendor-specific logic and types from the main application.
Thanks for the write-up Trevor, some really good stuff here.

Regards,
/tony




Trevor Conn
Senior Principal Software Engineer
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA

How can I control the ouput of logs based on log level

sunjj@mingdutech.com
 

Hi,
    How can I control the ouput of logs based on log level 
Thanks,
jianjiao


孙建蛟 -- 研发部  IOT
************************************************************
浙江明度智控科技有限公司
公司地址:浙江省杭州市滨江区江虹南路316号京安创业园
工厂地址:江苏省昆山市汉浦路1937号欣昆产业园
电 话:0571-88196008   传 真:0571-86718570
邮 箱:sunjj@mingdutech.com

Re: How can I control the ouput of logs based on log level

James.White2@...
 

Jianjiano,

 

Courtesy of Akram Ahmad from my team, the following should help:

 

Assuming you are working with the new Go services.  In the Go services you can set the log level in each service call with this function

func (lc EdgeXLogger) log(logLevel string, msg string, labels []string) error {

}

 

To set the level more universally for that service, set the following config option:

/config/application/logging.level.root :  This specifies the level of logging mechanism, the value is one of TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF.

 

Similar options and capability are also in the older Java services if you need it.

 

See https://nexus.edgexfoundry.org/content/sites/docs/staging/master/docs/_build/html/Ch-Logging.html for more info on the logging service.

 

Hope this helps.

Jim

 

From: EdgeX-GoLang@... [mailto:EdgeX-GoLang@...] On Behalf Of sunjj@...
Sent: Monday, August 06, 2018 9:52 PM
To: EdgeX-GoLang
Subject: [Edgex-golang] How can I control the ouput of logs based on log level

 

Hi,

    How can I control the ouput of logs based on log level 

Thanks,

jianjiao

 


孙建蛟 -- 研发部  IOT

************************************************************

浙江明度智控科技有限公司

公司地址:浙江省杭州市滨江区江虹南路316号京安创业园

工厂地址:江苏省昆山市汉浦路1937号欣昆产业园

话:0571-88196008   真:0571-86718570

箱:sunjj@mingdutech.com

Test/Mock Tools in Go

Trevor.Conn@...
 

Dell Customer Communication

Hi all – I assigned myself an issue yesterday for taking a small piece of the EdgeX core services and refactoring toward greater unit test coverage. I believe that some of the techniques I first introduced to the codebase in the holding repo will be of benefit in this regard. However in order to accomplish that, I need help from mocking tools. At the time I created that holding repo project, I evaluated several tools in the Go ecosystem and the ones I settled on are:

 

Mockery – Used for creating mock types from interfaces, eliminates writing of boilerplate code.

Testify – Used for implementing test conditions and evaluations within mocked methods

 

I chose these because they were flexible and seemed to have the least overhead for integration. One of the other ones I looked at was Ginkgo but it seemed like overkill.

 

I am asking to see if anyone has strong feelings in this area about a particular package. If so, please call it out and I’ll take a look. Otherwise, does anyone have issues with using the above packages?

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

回复: Re: [Edgex-golang] How can I control the ouput of logs based on log level

sunjj@mingdutech.com
 

Hi Jim,
Thank you for your answer.  sorry I didn't describe the question clearly yesterday.

I am using the Go service of core data.  Is there a configuration in  cmd/core-data/res/configuration.toml  just as java's application.yml(logging.level.org.edgexfoundry=INFO)。

Thank you!
Jianjiao



孙建蛟 -- 研发部  IOT
************************************************************
浙江明度智控科技有限公司
公司地址:浙江省杭州市滨江区江虹南路316号京安创业园
工厂地址:江苏省昆山市汉浦路1937号欣昆产业园
电 话:0571-88196008   传 真:0571-86718570
邮 箱:sunjj@mingdutech.com

 
发件人: James.White2
发送时间: 2018-08-07 23:32
抄送: Akram.Ahmad@...
主题: Re: [Edgex-golang] How can I control the ouput of logs based on log level

Jianjiano,

 

Courtesy of Akram Ahmad from my team, the following should help:

 

Assuming you are working with the new Go services.  In the Go services you can set the log level in each service call with this function

func (lc EdgeXLogger) log(logLevel string, msg string, labels []string) error {

}

 

To set the level more universally for that service, set the following config option:

/config/application/logging.level.root :  This specifies the level of logging mechanism, the value is one of TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF.

 

Similar options and capability are also in the older Java services if you need it.

 

See https://nexus.edgexfoundry.org/content/sites/docs/staging/master/docs/_build/html/Ch-Logging.html for more info on the logging service.

 

Hope this helps.

Jim

 

From: EdgeX-GoLang@... [mailto:EdgeX-GoLang@...] On Behalf Of sunjj@...
Sent: Monday, August 06, 2018 9:52 PM
To: EdgeX-GoLang
Subject: [Edgex-golang] How can I control the ouput of logs based on log level

 

Hi,

    How can I control the ouput of logs based on log level 

Thanks,

jianjiao

 


孙建蛟 -- 研发部  IOT

************************************************************

浙江明度智控科技有限公司

公司地址:浙江省杭州市滨江区江虹南路316号京安创业园

工厂地址:江苏省昆山市汉浦路1937号欣昆产业园

话:0571-88196008   真:0571-86718570

箱:sunjj@mingdutech.com

Re: Test/Mock Tools in Go

Smith, Nick A <Nick.a.Smith@...>
 

FWIW internally we’ve had great success with Testify for mocking and asserts. The test code is generally clean and easy to read.

 

Nick

 



 
Nick Smith
Technology Strategist
Tel: +44 1223 703485
@thalesesecurity

Thales eSecurity
One Station Square
Cambridge CB1 2GA
United Kingdom



www.thalesesecurity.com

From: EdgeX-GoLang@... [mailto:EdgeX-GoLang@...] On Behalf Of Trevor.Conn@...
Sent: 07 August 2018 16:49
To: edgex-golang@...
Subject: [Edgex-golang] Test/Mock Tools in Go

 

Dell Customer Communication

Hi all – I assigned myself an issue yesterday for taking a small piece of the EdgeX core services and refactoring toward greater unit test coverage. I believe that some of the techniques I first introduced to the codebase in the holding repo will be of benefit in this regard. However in order to accomplish that, I need help from mocking tools. At the time I created that holding repo project, I evaluated several tools in the Go ecosystem and the ones I settled on are:

 

Mockery – Used for creating mock types from interfaces, eliminates writing of boilerplate code.

Testify – Used for implementing test conditions and evaluations within mocked methods

 

I chose these because they were flexible and seemed to have the least overhead for integration. One of the other ones I looked at was Ginkgo but it seemed like overkill.

 

I am asking to see if anyone has strong feelings in this area about a particular package. If so, please call it out and I’ll take a look. Otherwise, does anyone have issues with using the above packages?

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA