Topics

go repos concerns

Fede Claramonte
 

Hi all,

First of all I want to thank Dell for the initial go code for the core microservices. Good work.

I have been learning about the core Go code and I have some concerns about the layout of the code on different repositories.

We have several repos, each one with one different edgex module or lib. Each repo uses glide to vendor all dependencies, including the other edgex modules. So we have the same code copied over all the repositories in the vendor folder(e.g. core-domain-go, support-domain-go). My concerns:

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.

On the other hand we have the layout Drasko set up for export services. It contains all the needed code to compile several microservices and it is trivial to know when any change breaks some other module in the repo.

I would propose to use something similar to what Drasko did, i.e. having all/most of the edgex Go code in only one repo as follows:

edgex-go
├── cmd                 // The main file for each microservice in each folder
│   ├── core-data
│   ├── core-metadata
│   ├── export-client
│   ├── export-distro
│   └── support-logging
├── core                // core-domain-go
│   ├── clients         // core-clients-go
│   ├── data            // core-data-go
│   └── metadata        // core-metadata-go
├── device              // Device helplers
│   └── virtual         // device virtual
├── dockerfiles
├── export // The export repository
│ ├── client
│   └── distro
└── support             // support-domain-go
    └── logging

With this layout it will be easier to work with the code, all in only one repo. After changing something in edgex-go/core/XXX.go you just need to build and run the tests locally to know if the change breaks something. This is also a lot easier for CI, as there is no dependencies between different repos.

Opinions?

Fede

Drasko DRASKOVIC <drasko.draskovic@...>
 

Hi Fede,
this was the layout I was proposing from the start for Go rewrite -
this is what I was referring to when I said "monorepo". A single repo
would hold all the EdgeX code, separated in independent services that
live in different directories. We did the same thing with Mainflux -
which in the beginning had separate repos, and now all services are
live in one repo with identical structure:
https://github.com/Mainflux/mainflux. The result was significant
simplification of development and maintaining process.

Monorepo would simplify CI, testing and would keep overall project
code more consistent. I support the idea.

Best regards,
Drasko

On Mon, Dec 18, 2017 at 3:40 PM, Fede Claramonte
<fclaramonte@...> wrote:
Hi all,

First of all I want to thank Dell for the initial go code for the core
microservices. Good work.

I have been learning about the core Go code and I have some concerns about
the layout of the code on different repositories.

We have several repos, each one with one different edgex module or lib. Each
repo uses glide to vendor all dependencies, including the other edgex
modules. So we have the same code copied over all the repositories in the
vendor folder(e.g. core-domain-go, support-domain-go). My concerns:

1) Each time a repo changes we all will need to update the vendor folder
manually. I think it will be wearisome and error-prone.

2) Another problem is to know if a change in any module (e.g.
core-domain-go) breaks some other module. We will need to manually compile
all the dependent modules to make sure we will not break any other module.

3) Previous point will be also influence how CI is implemented, it will need
to launch several jobs after a change to test if it breaks some other
module.

On the other hand we have the layout Drasko set up for export services. It
contains all the needed code to compile several microservices and it is
trivial to know when any change breaks some other module in the repo.

I would propose to use something similar to what Drasko did, i.e. having
all/most of the edgex Go code in only one repo as follows:

edgex-go
├── cmd // The main file for each microservice in each
folder
│ ├── core-data
│ ├── core-metadata
│ ├── export-client
│ ├── export-distro
│ └── support-logging
├── core // core-domain-go
│ ├── clients // core-clients-go
│ ├── data // core-data-go
│ └── metadata // core-metadata-go
├── device // Device helplers
│ └── virtual // device virtual
├── dockerfiles
├── export // The export repository
│ ├── client
│ └── distro
└── support // support-domain-go
└── logging

With this layout it will be easier to work with the code, all in only one
repo. After changing something in edgex-go/core/XXX.go you just need to
build and run the tests locally to know if the change breaks something. This
is also a lot easier for CI, as there is no dependencies between different
repos.

Opinions?

Fede

_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang

Jeremy Phelps
 

Hi Fede,
These are actually all really good points, I appreciate you bringing them up.  I agree that we need to discuss this as there are some cases where it could cause us some troubles.
For example...if the dependencies are vendored is there a way to pin to a specific version?

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.
    Not only that but how do we manage different versions under vendoring...there may be a case where the vendored code will be a release branch and not the tip of master for example.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.
    This is not a unique problem in this case I think.  This is just an issue with mico-services in general.  We can lessen the impact by strict adherence to sematic versioning and some smoke tests before we kick off blackbox-testing.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.
    I think this is what blackbox-testing would cover no?

On Mon, Dec 18, 2017 at 8:40 AM, Fede Claramonte <fclaramonte@...> wrote:
Hi all,

First of all I want to thank Dell for the initial go code for the core microservices. Good work.

I have been learning about the core Go code and I have some concerns about the layout of the code on different repositories.

We have several repos, each one with one different edgex module or lib. Each repo uses glide to vendor all dependencies, including the other edgex modules. So we have the same code copied over all the repositories in the vendor folder(e.g. core-domain-go, support-domain-go). My concerns:

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.

On the other hand we have the layout Drasko set up for export services. It contains all the needed code to compile several microservices and it is trivial to know when any change breaks some other module in the repo.

I would propose to use something similar to what Drasko did, i.e. having all/most of the edgex Go code in only one repo as follows:

edgex-go
├── cmd                 // The main file for each microservice in each folder
│   ├── core-data
│   ├── core-metadata
│   ├── export-client
│   ├── export-distro
│   └── support-logging
├── core                // core-domain-go
│   ├── clients         // core-clients-go
│   ├── data            // core-data-go
│   └── metadata        // core-metadata-go
├── device              // Device helplers
│   └── virtual         // device virtual
├── dockerfiles
├── export // The export repository
│ ├── client
│   └── distro
└── support             // support-domain-go
    └── logging

With this layout it will be easier to work with the code, all in only one repo. After changing something in edgex-go/core/XXX.go you just need to build and run the tests locally to know if the change breaks something. This is also a lot easier for CI, as there is no dependencies between different repos.

Opinions?

Fede

_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...y.org
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang

James.White2@...
 

Fede, Drasko and Jeremy,

thanks all for the comments and review.  I will point you all to the fact that we did hash on this problem for some time in the working group meetings.  At the time, it was decided that each WG could feel free to implement as they see fit for this first implementation.  In the core WG we wanted libraries in different repo for the ability to share/modify them like we do in Java.  For others like in the Applications WG, you all wanted a single repo.


At that time, we all acknowledged that different approaches was going to create issues where duplication would occur in some cases (which we said was the responsibility of the maintainers to keep up with changes) and create complexity of many repo in other cases.


So while I follow and appreciate the discussion here, I fail to see any new evidence or arguments being introduced.  So are we all asking to open up this decision to new review?  And if we are reopening the discussion, are people wishing to change their original view point?


If we are just going to open this discussion to rehash the same issues and people are still holding to their same opinions, I am not sure of the value in doing so.  Instead, I'd like to have the teams make a little more progress in developing their respective Go services and then start to look at the issue again with more evidence of why we need to move in a particular direction.


Am I missing something?

Jim


From: edgex-golang-bounces@... <edgex-golang-bounces@...> on behalf of Jeremy Phelps <jphelps@...>
Sent: Monday, December 18, 2017 9:01 AM
To: Fede Claramonte
Cc: edgex-golang@...
Subject: Re: [Edgex-golang] go repos concerns
 
Hi Fede,
These are actually all really good points, I appreciate you bringing them up.  I agree that we need to discuss this as there are some cases where it could cause us some troubles.
For example...if the dependencies are vendored is there a way to pin to a specific version?

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.
    Not only that but how do we manage different versions under vendoring...there may be a case where the vendored code will be a release branch and not the tip of master for example.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.
    This is not a unique problem in this case I think.  This is just an issue with mico-services in general.  We can lessen the impact by strict adherence to sematic versioning and some smoke tests before we kick off blackbox-testing.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.
    I think this is what blackbox-testing would cover no?

On Mon, Dec 18, 2017 at 8:40 AM, Fede Claramonte <fclaramonte@...> wrote:
Hi all,

First of all I want to thank Dell for the initial go code for the core microservices. Good work.

I have been learning about the core Go code and I have some concerns about the layout of the code on different repositories.

We have several repos, each one with one different edgex module or lib. Each repo uses glide to vendor all dependencies, including the other edgex modules. So we have the same code copied over all the repositories in the vendor folder(e.g. core-domain-go, support-domain-go). My concerns:

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.

On the other hand we have the layout Drasko set up for export services. It contains all the needed code to compile several microservices and it is trivial to know when any change breaks some other module in the repo.

I would propose to use something similar to what Drasko did, i.e. having all/most of the edgex Go code in only one repo as follows:

edgex-go
├── cmd                 // The main file for each microservice in each folder
│   ├── core-data
│   ├── core-metadata
│   ├── export-client
│   ├── export-distro
│   └── support-logging
├── core                // core-domain-go
│   ├── clients         // core-clients-go
│   ├── data            // core-data-go
│   └── metadata        // core-metadata-go
├── device              // Device helplers
│   └── virtual         // device virtual
├── dockerfiles
├── export // The export repository
│ ├── client
│   └── distro
└── support             // support-domain-go
    └── logging

With this layout it will be easier to work with the code, all in only one repo. After changing something in edgex-go/core/XXX.go you just need to build and run the tests locally to know if the change breaks something. This is also a lot easier for CI, as there is no dependencies between different repos.

Opinions?

Fede

_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...y.org
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang

Jeremy Phelps
 

Hi Jim,
I remember the discussion and do not think we should re-hash.  My points were to  answer the 3 points that Fede brought up (sorry for any confusion).  I think we need to solve Point number 1; points 2 and 3 are general problems that you will find in any mico-service architecture I think.
Jeremy

On Mon, Dec 18, 2017 at 10:18 AM, <James.White2@...> wrote:

Fede, Drasko and Jeremy,

thanks all for the comments and review.  I will point you all to the fact that we did hash on this problem for some time in the working group meetings.  At the time, it was decided that each WG could feel free to implement as they see fit for this first implementation.  In the core WG we wanted libraries in different repo for the ability to share/modify them like we do in Java.  For others like in the Applications WG, you all wanted a single repo.


At that time, we all acknowledged that different approaches was going to create issues where duplication would occur in some cases (which we said was the responsibility of the maintainers to keep up with changes) and create complexity of many repo in other cases.


So while I follow and appreciate the discussion here, I fail to see any new evidence or arguments being introduced.  So are we all asking to open up this decision to new review?  And if we are reopening the discussion, are people wishing to change their original view point?


If we are just going to open this discussion to rehash the same issues and people are still holding to their same opinions, I am not sure of the value in doing so.  Instead, I'd like to have the teams make a little more progress in developing their respective Go services and then start to look at the issue again with more evidence of why we need to move in a particular direction.


Am I missing something?

Jim


From: edgex-golang-bounces@lists.edgexfoundry.org <edgex-golang-bounces@lists.edgexfoundry.org> on behalf of Jeremy Phelps <jphelps@...>
Sent: Monday, December 18, 2017 9:01 AM
To: Fede Claramonte
Cc: edgex-golang@lists.edgexfoundry.org
Subject: Re: [Edgex-golang] go repos concerns
 
Hi Fede,
These are actually all really good points, I appreciate you bringing them up.  I agree that we need to discuss this as there are some cases where it could cause us some troubles.
For example...if the dependencies are vendored is there a way to pin to a specific version?

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.
    Not only that but how do we manage different versions under vendoring...there may be a case where the vendored code will be a release branch and not the tip of master for example.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.
    This is not a unique problem in this case I think.  This is just an issue with mico-services in general.  We can lessen the impact by strict adherence to sematic versioning and some smoke tests before we kick off blackbox-testing.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.
    I think this is what blackbox-testing would cover no?

On Mon, Dec 18, 2017 at 8:40 AM, Fede Claramonte <fclaramonte@caviumnetworks.com> wrote:
Hi all,

First of all I want to thank Dell for the initial go code for the core microservices. Good work.

I have been learning about the core Go code and I have some concerns about the layout of the code on different repositories.

We have several repos, each one with one different edgex module or lib. Each repo uses glide to vendor all dependencies, including the other edgex modules. So we have the same code copied over all the repositories in the vendor folder(e.g. core-domain-go, support-domain-go). My concerns:

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.

On the other hand we have the layout Drasko set up for export services. It contains all the needed code to compile several microservices and it is trivial to know when any change breaks some other module in the repo.

I would propose to use something similar to what Drasko did, i.e. having all/most of the edgex Go code in only one repo as follows:

edgex-go
├── cmd                 // The main file for each microservice in each folder
│   ├── core-data
│   ├── core-metadata
│   ├── export-client
│   ├── export-distro
│   └── support-logging
├── core                // core-domain-go
│   ├── clients         // core-clients-go
│   ├── data            // core-data-go
│   └── metadata        // core-metadata-go
├── device              // Device helplers
│   └── virtual         // device virtual
├── dockerfiles
├── export // The export repository
│ ├── client
│   └── distro
└── support             // support-domain-go
    └── logging

With this layout it will be easier to work with the code, all in only one repo. After changing something in edgex-go/core/XXX.go you just need to build and run the tests locally to know if the change breaks something. This is also a lot easier for CI, as there is no dependencies between different repos.

Opinions?

Fede

_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...y.org
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang


Fede Claramonte
 

Jim,

I bringing it back to discussion because I didn't had previous experience with go and having several repos. Now I have and I think we are making it hard for the developers to work on the go codebase.


Making big changes in the repos layout is easy now that we are beginning. It will be harder latter once we have more commits in the repositories.


But it is a good idea to postpone the discussion until we get used to the actual layout and experience how friendly is the workflow.


Regards,

Fede


On 18/12/17 17:18, James.White2@... wrote:

Fede, Drasko and Jeremy,

thanks all for the comments and review.  I will point you all to the fact that we did hash on this problem for some time in the working group meetings.  At the time, it was decided that each WG could feel free to implement as they see fit for this first implementation.  In the core WG we wanted libraries in different repo for the ability to share/modify them like we do in Java.  For others like in the Applications WG, you all wanted a single repo.


At that time, we all acknowledged that different approaches was going to create issues where duplication would occur in some cases (which we said was the responsibility of the maintainers to keep up with changes) and create complexity of many repo in other cases.


So while I follow and appreciate the discussion here, I fail to see any new evidence or arguments being introduced.  So are we all asking to open up this decision to new review?  And if we are reopening the discussion, are people wishing to change their original view point?


If we are just going to open this discussion to rehash the same issues and people are still holding to their same opinions, I am not sure of the value in doing so.  Instead, I'd like to have the teams make a little more progress in developing their respective Go services and then start to look at the issue again with more evidence of why we need to move in a particular direction.


Am I missing something?

Jim


From: edgex-golang-bounces@... <edgex-golang-bounces@...> on behalf of Jeremy Phelps <jphelps@...>
Sent: Monday, December 18, 2017 9:01 AM
To: Fede Claramonte
Cc: edgex-golang@...
Subject: Re: [Edgex-golang] go repos concerns
 
Hi Fede,
These are actually all really good points, I appreciate you bringing them up.  I agree that we need to discuss this as there are some cases where it could cause us some troubles.
For example...if the dependencies are vendored is there a way to pin to a specific version?

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.
    Not only that but how do we manage different versions under vendoring...there may be a case where the vendored code will be a release branch and not the tip of master for example.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.
    This is not a unique problem in this case I think.  This is just an issue with mico-services in general.  We can lessen the impact by strict adherence to sematic versioning and some smoke tests before we kick off blackbox-testing.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.
    I think this is what blackbox-testing would cover no?

On Mon, Dec 18, 2017 at 8:40 AM, Fede Claramonte <fclaramonte@...> wrote:
Hi all,

First of all I want to thank Dell for the initial go code for the core microservices. Good work.

I have been learning about the core Go code and I have some concerns about the layout of the code on different repositories.

We have several repos, each one with one different edgex module or lib. Each repo uses glide to vendor all dependencies, including the other edgex modules. So we have the same code copied over all the repositories in the vendor folder(e.g. core-domain-go, support-domain-go). My concerns:

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.

On the other hand we have the layout Drasko set up for export services. It contains all the needed code to compile several microservices and it is trivial to know when any change breaks some other module in the repo.

I would propose to use something similar to what Drasko did, i.e. having all/most of the edgex Go code in only one repo as follows:

edgex-go
├── cmd                 // The main file for each microservice in each folder
│   ├── core-data
│   ├── core-metadata
│   ├── export-client
│   ├── export-distro
│   └── support-logging
├── core                // core-domain-go
│   ├── clients         // core-clients-go
│   ├── data            // core-data-go
│   └── metadata        // core-metadata-go
├── device              // Device helplers
│   └── virtual         // device virtual
├── dockerfiles
├── export // The export repository
│ ├── client
│   └── distro
└── support             // support-domain-go
    └── logging

With this layout it will be easier to work with the code, all in only one repo. After changing something in edgex-go/core/XXX.go you just need to build and run the tests locally to know if the change breaks something. This is also a lot easier for CI, as there is no dependencies between different repos.

Opinions?

Fede

_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...y.org
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang


Fede Claramonte
 

Jeremy,

My second point was about just breaking the compilation. Some module not compiling because of changes in another repo.

Fede


On 18/12/17 16:01, Jeremy Phelps wrote:
Hi Fede,
These are actually all really good points, I appreciate you bringing them up.  I agree that we need to discuss this as there are some cases where it could cause us some troubles.
For example...if the dependencies are vendored is there a way to pin to a specific version?

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.
    Not only that but how do we manage different versions under vendoring...there may be a case where the vendored code will be a release branch and not the tip of master for example.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.
    This is not a unique problem in this case I think.  This is just an issue with mico-services in general.  We can lessen the impact by strict adherence to sematic versioning and some smoke tests before we kick off blackbox-testing.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.
    I think this is what blackbox-testing would cover no?

On Mon, Dec 18, 2017 at 8:40 AM, Fede Claramonte <fclaramonte@...> wrote:
Hi all,

First of all I want to thank Dell for the initial go code for the core microservices. Good work.

I have been learning about the core Go code and I have some concerns about the layout of the code on different repositories.

We have several repos, each one with one different edgex module or lib. Each repo uses glide to vendor all dependencies, including the other edgex modules. So we have the same code copied over all the repositories in the vendor folder(e.g. core-domain-go, support-domain-go). My concerns:

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.

On the other hand we have the layout Drasko set up for export services. It contains all the needed code to compile several microservices and it is trivial to know when any change breaks some other module in the repo.

I would propose to use something similar to what Drasko did, i.e. having all/most of the edgex Go code in only one repo as follows:

edgex-go
├── cmd                 // The main file for each microservice in each folder
│   ├── core-data
│   ├── core-metadata
│   ├── export-client
│   ├── export-distro
│   └── support-logging
├── core                // core-domain-go
│   ├── clients         // core-clients-go
│   ├── data            // core-data-go
│   └── metadata        // core-metadata-go
├── device              // Device helplers
│   └── virtual         // device virtual
├── dockerfiles
├── export // The export repository
│ ├── client
│   └── distro
└── support             // support-domain-go
    └── logging

With this layout it will be easier to work with the code, all in only one repo. After changing something in edgex-go/core/XXX.go you just need to build and run the tests locally to know if the change breaks something. This is also a lot easier for CI, as there is no dependencies between different repos.

Opinions?

Fede

_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...y.org
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang


James.White2@...
 

Sounds good Fede.  I too am just starting to get familiar with Go Lang and the issues associated my self.  

Let me add a bit of time for us to discuss this issue at the Face - to - Face meeting.  This will also give us a chance for Jeremy to get some of the dev ops work done to produce the Go Lang artifacts and containers and provide additional feedback.

This will still allow us time to react/change prior to the California release in June.

Jim


From: Fede Claramonte <fclaramonte@...>
Sent: Tuesday, December 19, 2017 8:12 AM
To: White2, James; jphelps@...; drasko@...
Cc: edgex-golang@...
Subject: Re: [Edgex-golang] go repos concerns
 

Jim,

I bringing it back to discussion because I didn't had previous experience with go and having several repos. Now I have and I think we are making it hard for the developers to work on the go codebase.


Making big changes in the repos layout is easy now that we are beginning. It will be harder latter once we have more commits in the repositories.


But it is a good idea to postpone the discussion until we get used to the actual layout and experience how friendly is the workflow.


Regards,

Fede


On 18/12/17 17:18, James.White2@... wrote:

Fede, Drasko and Jeremy,

thanks all for the comments and review.  I will point you all to the fact that we did hash on this problem for some time in the working group meetings.  At the time, it was decided that each WG could feel free to implement as they see fit for this first implementation.  In the core WG we wanted libraries in different repo for the ability to share/modify them like we do in Java.  For others like in the Applications WG, you all wanted a single repo.


At that time, we all acknowledged that different approaches was going to create issues where duplication would occur in some cases (which we said was the responsibility of the maintainers to keep up with changes) and create complexity of many repo in other cases.


So while I follow and appreciate the discussion here, I fail to see any new evidence or arguments being introduced.  So are we all asking to open up this decision to new review?  And if we are reopening the discussion, are people wishing to change their original view point?


If we are just going to open this discussion to rehash the same issues and people are still holding to their same opinions, I am not sure of the value in doing so.  Instead, I'd like to have the teams make a little more progress in developing their respective Go services and then start to look at the issue again with more evidence of why we need to move in a particular direction.


Am I missing something?

Jim


From: edgex-golang-bounces@... <edgex-golang-bounces@...> on behalf of Jeremy Phelps <jphelps@...>
Sent: Monday, December 18, 2017 9:01 AM
To: Fede Claramonte
Cc: edgex-golang@...
Subject: Re: [Edgex-golang] go repos concerns
 
Hi Fede,
These are actually all really good points, I appreciate you bringing them up.  I agree that we need to discuss this as there are some cases where it could cause us some troubles.
For example...if the dependencies are vendored is there a way to pin to a specific version?

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.
    Not only that but how do we manage different versions under vendoring...there may be a case where the vendored code will be a release branch and not the tip of master for example.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.
    This is not a unique problem in this case I think.  This is just an issue with mico-services in general.  We can lessen the impact by strict adherence to sematic versioning and some smoke tests before we kick off blackbox-testing.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.
    I think this is what blackbox-testing would cover no?

On Mon, Dec 18, 2017 at 8:40 AM, Fede Claramonte <fclaramonte@...> wrote:
Hi all,

First of all I want to thank Dell for the initial go code for the core microservices. Good work.

I have been learning about the core Go code and I have some concerns about the layout of the code on different repositories.

We have several repos, each one with one different edgex module or lib. Each repo uses glide to vendor all dependencies, including the other edgex modules. So we have the same code copied over all the repositories in the vendor folder(e.g. core-domain-go, support-domain-go). My concerns:

1) Each time a repo changes we all will need to update the vendor folder manually. I think it will be wearisome and error-prone.

2) Another problem is to know if a change in any module (e.g. core-domain-go) breaks some other module. We will need to manually compile all the dependent modules to make sure we will not break any other module.

3) Previous point will be also influence how CI is implemented, it will need to launch several jobs after a change to test if it breaks some other module.

On the other hand we have the layout Drasko set up for export services. It contains all the needed code to compile several microservices and it is trivial to know when any change breaks some other module in the repo.

I would propose to use something similar to what Drasko did, i.e. having all/most of the edgex Go code in only one repo as follows:

edgex-go
├── cmd                 // The main file for each microservice in each folder
│   ├── core-data
│   ├── core-metadata
│   ├── export-client
│   ├── export-distro
│   └── support-logging
├── core                // core-domain-go
│   ├── clients         // core-clients-go
│   ├── data            // core-data-go
│   └── metadata        // core-metadata-go
├── device              // Device helplers
│   └── virtual         // device virtual
├── dockerfiles
├── export // The export repository
│ ├── client
│   └── distro
└── support             // support-domain-go
    └── logging

With this layout it will be easier to work with the code, all in only one repo. After changing something in edgex-go/core/XXX.go you just need to build and run the tests locally to know if the change breaks something. This is also a lot easier for CI, as there is no dependencies between different repos.

Opinions?

Fede

_______________________________________________
EdgeX-GoLang mailing list
EdgeX-GoLang@...y.org
https://lists.edgexfoundry.org/mailman/listinfo/edgex-golang


Drasko DRASKOVIC <drasko@...>
 

Hi Fede,

On Tue, Dec 19, 2017 at 4:12 PM, <James.White2@...> wrote:
Sounds good Fede. I too am just starting to get familiar with Go Lang and
the issues associated my self.

Let me add a bit of time for us to discuss this issue at the Face - to -
Face meeting. This will also give us a chance for Jeremy to get some of the
dev ops work done to produce the Go Lang artifacts and containers and
provide additional feedback.

This will still allow us time to react/change prior to the California
release in June.
If you have some time (and would like to learn), I would also suggest
setting-up a PoC GitHub repo (for example in your private GitHub
account) and doing integration of existing core repos. I can help you
with this. This way on F2F meting in Orlando we can present something
that works and then the advantages would be more tangeable, IMHO.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Fede Claramonte
 

Hi all,

we just wanted to share the work we have done with the monorepo:

https://github.com/feclare/edgex-go

It is automatically generated with this script:

https://gist.github.com/feclare/8dba191e8cf77864fe5eed38b380f13a

Regards,
Fede

On 19/12/17 17:18, Drasko DRASKOVIC wrote:
Hi Fede,

On Tue, Dec 19, 2017 at 4:12 PM, <James.White2@...> wrote:
Sounds good Fede. I too am just starting to get familiar with Go Lang and
the issues associated my self.

Let me add a bit of time for us to discuss this issue at the Face - to -
Face meeting. This will also give us a chance for Jeremy to get some of the
dev ops work done to produce the Go Lang artifacts and containers and
provide additional feedback.

This will still allow us time to react/change prior to the California
release in June.
If you have some time (and would like to learn), I would also suggest
setting-up a PoC GitHub repo (for example in your private GitHub
account) and doing integration of existing core repos. I can help you
with this. This way on F2F meting in Orlando we can present something
that works and then the advantages would be more tangeable, IMHO.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Drasko DRASKOVIC <drasko@...>
 

Thanks a lot Fede.

This looks very good and simple from my point of view. It's concise
and visually we can understand easily the whole system.

This approach will significantly simplify CI, testing, release
versioning and tagging, and many other things. It will also keep code
consistent, SW developers will have code from other micro-services to
look as examples and stay consistent. Technical battier will be
lowered and this will increase adoption rate.

I am definitely in flavour for this approach. As I mentioned earlier,
merging our per-service-repo into a single repo for Mainflux
(https://github.com/mainflux/mainflux) was one of the best decisions
we made for the project. It was much simpler to work on the project.

In the end, I would reccomend this video:
https://www.youtube.com/watch?v=lV8-1S28ycM, but you can also see that
Goole keeps tons of code in the single repo for the same reasons I
mentioned earlier: https://www.youtube.com/watch?v=W71BTkUbdqE

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

On Thu, Jan 18, 2018 at 3:00 PM, Fede Claramonte
<fclaramonte@...> wrote:
Hi all,

we just wanted to share the work we have done with the monorepo:

https://github.com/feclare/edgex-go

It is automatically generated with this script:

https://gist.github.com/feclare/8dba191e8cf77864fe5eed38b380f13a

Regards,
Fede

On 19/12/17 17:18, Drasko DRASKOVIC wrote:

Hi Fede,

On Tue, Dec 19, 2017 at 4:12 PM, <James.White2@...> wrote:

Sounds good Fede. I too am just starting to get familiar with Go Lang
and
the issues associated my self.

Let me add a bit of time for us to discuss this issue at the Face - to -
Face meeting. This will also give us a chance for Jeremy to get some of
the
dev ops work done to produce the Go Lang artifacts and containers and
provide additional feedback.

This will still allow us time to react/change prior to the California
release in June.
If you have some time (and would like to learn), I would also suggest
setting-up a PoC GitHub repo (for example in your private GitHub
account) and doing integration of existing core repos. I can help you
with this. This way on F2F meting in Orlando we can present something
that works and then the advantages would be more tangeable, IMHO.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic