Date   
Re: Welcome to EdgeX-GoLang@lists.edgexfoundry.org

qq
 

hello


15599633@...

 
Date: 2018-09-18 20:28
Subject: Welcome to EdgeX-GoLang@...

Hello,

Welcome to the EdgeX-GoLang@... group at EdgeX Foundry. Please take a moment to review this message.

To learn more about the EdgeX-GoLang@... group, please visit https://lists.edgexfoundry.org/g/EdgeX-GoLang

To start sending messages to members of this group, simply send email to EdgeX-GoLang@...

If you do not wish to belong to EdgeX-GoLang@..., you may unsubscribe by sending an email to EdgeX-GoLang+unsubscribe@...

To see and modify all of your groups, go to https://lists.edgexfoundry.org


Regards,

The EdgeX-GoLang@... Moderator

Support-Logging Mongo Configuration

Trevor.Conn@...
 

Dell - Internal Use - Confidential

I have a question for the community –

 

I’m working on the ConfigV2 changes for the support-logging service and in its configuration for Mongo, it has an anomalous property for MongoCollection. This is in addition to the database name signified by the MongoDB property. These are currently set as follows:

 

MongoDB = 'logging' #DBName

MongoCollection = 'logEntry' #Collection within the above DB

 

No other service specifies a collection in addition to the database name, instead choosing to just write all documents into the specified database. Indeed having a collection called “logEntry” inside of a “logging” database sounds redundant to me.

 

ConfigV2 has the following structure for specifying database configuration settings:

[Databases]
  [Databases.Primary]
  Host = 'localhost'
  Name = 'coredata'
  Password = ''
  Port = 27017
  Username = ''
  Timeout = 5000
  Type = 'mongodb'

 

I would like to standardize on this by removing the need for a MongoCollection from the support-logging configuration. Are there any objections?

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: Support-Logging Mongo Configuration

Joan Duran
 

In the other microservices (core-data, core-metada and core-client) the collection names used by MongoDB are defined in internal/pkg/db/db.go.

As you said, should be fine to remove this parameter and add a const (LogCollection?) in internal/pkg/db/db.go file.

Best regards,
Joan


El 26/09/18 a les 23:48, Trevor.Conn@... ha escrit:

Dell - Internal Use - Confidential

I have a question for the community –

 

I’m working on the ConfigV2 changes for the support-logging service and in its configuration for Mongo, it has an anomalous property for MongoCollection. This is in addition to the database name signified by the MongoDB property. These are currently set as follows:

 

MongoDB = 'logging' #DBName

MongoCollection = 'logEntry' #Collection within the above DB

 

No other service specifies a collection in addition to the database name, instead choosing to just write all documents into the specified database. Indeed having a collection called “logEntry” inside of a “logging” database sounds redundant to me.

 

ConfigV2 has the following structure for specifying database configuration settings:

[Databases]
  [Databases.Primary]
  Host = 'localhost'
  Name = 'coredata'
  Password = ''
  Port = 27017
  Username = ''
  Timeout = 5000
  Type = 'mongodb'

 

I would like to standardize on this by removing the need for a MongoCollection from the support-logging configuration. Are there any objections?

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 


Serverless

Drasko DRASKOVIC <drasko@...>
 

Interesting watch: https://www.youtube.com/watch?v=Va6Ve9oKNyY

We did not mention this one:
- https://serverless.com/event-gateway/
- https://github.com/serverless/event-gateway

BR,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor
EdgeX Foundry TSC member

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

Re: [Edgex-devel] Serverless

Tiejun Chen <tiejunc@...>
 

Drasko,

Have you worked on enabling serverless/faas on EdgeX? On my side I'm interested in this area.

Thanks
Tiejun

-----Original Message-----
From: EdgeX-Devel@... <EdgeX-Devel@...>
On Behalf Of Drasko DRASKOVIC
Sent: Wednesday, October 17, 2018 3:10 AM
To: edgex-devel@...; edgex-golang@...
Subject: [Edgex-devel] Serverless

Interesting watch:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.yout
ube.com%2Fwatch%3Fv%3DVa6Ve9oKNyY&amp;data=02%7C01%7Ctiejunc%40v
mware.com%7Ccd1de4381a674350fa1c08d6339af187%7Cb39138ca3cee4b4aa4
d6cd83d9dd62f0%7C1%7C0%7C636753137908225102&amp;sdata=iG%2BMu7
PKKEwKz%2BlZ8VOYAo2K9ncmt0j0Iyh6WrTdkc0%3D&amp;reserved=0

We did not mention this one:
-
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fserverless.
com%2Fevent-
gateway%2F&amp;data=02%7C01%7Ctiejunc%40vmware.com%7Ccd1de4381a6
74350fa1c08d6339af187%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%
7C636753137908235108&amp;sdata=rVm1EFKtwFOSYLyzneP%2B0hICZBZeZFH8
d3DlSFBXgyw%3D&amp;reserved=0
-
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co
m%2Fserverless%2Fevent-
gateway&amp;data=02%7C01%7Ctiejunc%40vmware.com%7Ccd1de4381a67435
0fa1c08d6339af187%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C63
6753137908235108&amp;sdata=JOl5VENJx6JP7Qxgbr9jUSK5GtWXBiAZJDYBT9y3
w7Q%3D&amp;reserved=0

BR,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor
EdgeX Foundry TSC member

https://na01.safelinks.protection.outlook.com/?url=www.mainflux.com&amp;da
ta=02%7C01%7Ctiejunc%40vmware.com%7Ccd1de4381a674350fa1c08d6339af1
87%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C636753137908235
108&amp;sdata=G15Aqhw2bEH1TCAgxIrbOdJxdY2fc3nJOTDzqn%2F0KPM%3D&a
mp;reserved=0 | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linke
din.com%2Fin%2Fdraskodraskovic&amp;data=02%7C01%7Ctiejunc%40vmware.c
om%7Ccd1de4381a674350fa1c08d6339af187%7Cb39138ca3cee4b4aa4d6cd83d
9dd62f0%7C1%7C0%7C636753137908235108&amp;sdata=TcEtM%2Bui7djf%2F
rY0kzHgQgkD2gardk0Zuuov73l4UDM%3D&amp;reserved=0
Twitter:@draskodraskovic

Re: [Edgex-devel] Serverless

Drasko DRASKOVIC <drasko@...>
 

Hi Tiejun,
Application WG is working on making new architecture for Export Services.

So far we analyzing 2 approaches - one with the message bus, and other
with Functions as a Service.

You can follow Application WG work here:
https://wiki.edgexfoundry.org/display/FA/Applications+Working+Group
and join the meetings.

Best regards,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor
EdgeX Foundry TSC member

www.mainflux.com | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn: https://www.linkedin.com/in/draskodraskovic
Twitter:@draskodraskovic

On Wed, Oct 17, 2018 at 2:44 AM Tiejun Chen <tiejunc@...> wrote:

Drasko,

Have you worked on enabling serverless/faas on EdgeX? On my side I'm interested in this area.

Thanks
Tiejun

-----Original Message-----
From: EdgeX-Devel@... <EdgeX-Devel@...>
On Behalf Of Drasko DRASKOVIC
Sent: Wednesday, October 17, 2018 3:10 AM
To: edgex-devel@...; edgex-golang@...
Subject: [Edgex-devel] Serverless

Interesting watch:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.yout
ube.com%2Fwatch%3Fv%3DVa6Ve9oKNyY&amp;data=02%7C01%7Ctiejunc%40v
mware.com%7Ccd1de4381a674350fa1c08d6339af187%7Cb39138ca3cee4b4aa4
d6cd83d9dd62f0%7C1%7C0%7C636753137908225102&amp;sdata=iG%2BMu7
PKKEwKz%2BlZ8VOYAo2K9ncmt0j0Iyh6WrTdkc0%3D&amp;reserved=0

We did not mention this one:
-
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fserverless.
com%2Fevent-
gateway%2F&amp;data=02%7C01%7Ctiejunc%40vmware.com%7Ccd1de4381a6
74350fa1c08d6339af187%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%
7C636753137908235108&amp;sdata=rVm1EFKtwFOSYLyzneP%2B0hICZBZeZFH8
d3DlSFBXgyw%3D&amp;reserved=0
-
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co
m%2Fserverless%2Fevent-
gateway&amp;data=02%7C01%7Ctiejunc%40vmware.com%7Ccd1de4381a67435
0fa1c08d6339af187%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C63
6753137908235108&amp;sdata=JOl5VENJx6JP7Qxgbr9jUSK5GtWXBiAZJDYBT9y3
w7Q%3D&amp;reserved=0

BR,
Drasko DRASKOVIC
Mainflux Author and Technical Advisor
EdgeX Foundry TSC member

https://na01.safelinks.protection.outlook.com/?url=www.mainflux.com&amp;da
ta=02%7C01%7Ctiejunc%40vmware.com%7Ccd1de4381a674350fa1c08d6339af1
87%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C636753137908235
108&amp;sdata=G15Aqhw2bEH1TCAgxIrbOdJxdY2fc3nJOTDzqn%2F0KPM%3D&a
mp;reserved=0 | Industrial IoT Cloud
-------------------------------------------------------------------
Engineering Division | Paris, France

LinkedIn:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linke
din.com%2Fin%2Fdraskodraskovic&amp;data=02%7C01%7Ctiejunc%40vmware.c
om%7Ccd1de4381a674350fa1c08d6339af187%7Cb39138ca3cee4b4aa4d6cd83d
9dd62f0%7C1%7C0%7C636753137908235108&amp;sdata=TcEtM%2Bui7djf%2F
rY0kzHgQgkD2gardk0Zuuov73l4UDM%3D&amp;reserved=0
Twitter:@draskodraskovic

Rob Pike on Go 2.0

Trevor.Conn@...
 

Dell Customer Communication

https://youtu.be/RIvL2ONhFBI

 

FYI -- An interesting presentation from Rob Pike on the current thinking (no commitments yet) around the major new features planned for Go 2.0

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Building containers behind a http proxy

David Urbina
 

Hello! I have been trying to build the containers using "make docker" but given the fact that I am in a Intranet behind a proxy it simply fails. The only solution I could find was to modify the Makefile by adding "--build-arg HTTP_PROXY=$(HTTP_PROXY)" to every "docker build". Is there a better way to do this?

Thanks.

Re: Building containers behind a http proxy

Trevor.Conn@...
 

Dell Customer Communication

Have you tried exporting those vars in the terminal session itself and then running “make docker”? That should cause anything in the terminal session that initiates an http connection to use the specified proxy settings and you won’t have to edit the file. That said, if the proxy blocks a website that glide tries to pull a dependency from, you won’t have a way around that without talking to your network admin.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

 

From: EdgeX-GoLang@... [mailto:EdgeX-GoLang@...] On Behalf Of edgex@...
Sent: Tuesday, November 27, 2018 4:59 PM
To: EdgeX-GoLang@...
Subject: [Edgex-golang] Building containers behind a http proxy

 

[EXTERNAL EMAIL]

Hello! I have been trying to build the containers using "make docker" but given the fact that I am in a Intranet behind a proxy it simply fails. The only solution I could find was to modify the Makefile by adding "--build-arg HTTP_PROXY=$(HTTP_PROXY)" to every "docker build". Is there a better way to do this?

Thanks.

Re: Building containers behind a http proxy

David Urbina
 

I do have the HTTP_PROXY exported in my terminal session.

Digging a little more on the error, I can see that "docker build" does get the proxy from nthe terminal but it do not pass it to any the internal commands in the Dockerfile.* scripts. For example, "RUN apk" fails because it does not get the proxy.

Docker documentation mentions that the "--build-arg" must be used to pass the proxy to the container been built. But this certainly requires to modify the EdgeX's Makefile.

Is there a better way? or should It open an issue?

Thanks.

FYI New GlobalSign Mongo Driver

Trevor.Conn@...
 

Dell - Internal Use - Confidential

Hi all – This is to let you know that we now have a new Mongo driver in the master branch of EdgeX thanks to Lenny Goodell @ Intel.

 

PR #915 was just merged

Information about the new driver can be found here

 

Members of our Go developer community please note that the next time you update/rebase your feature branches from master, you will need to delete (or rename) your glide.lock file and execute a “make prepare” to pull the new driver into your vendor directory. From there you can do a “make build” as normal.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Anyone Using VSCode?

Trevor.Conn@...
 

Dell Customer Communication

Is there anyone out there using VSCode as their primary IDE for work on EdgeX-Go that would corroborate the suggestion in this issue?

 

https://github.com/edgexfoundry/edgex-go/issues/981

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: [Edgex-tsc-devops] Preliminary Doc on Moving to Go Modules

Trevor.Conn@...
 

Hi all -- I'm going to piggy back on the thread below and loop in the Go email distro for more discussion about modules.

I've attached a V2 of the previous document that was sent. It includes two more bullet points w/r/t DevOps considerations, the final one being reference to a larger section about Go modules and how their adoption may affect the structure of our code repos as well as how we manage tags.


As I said below, I'm offering this proposal in the spirit of discussion. It represents the state of my thinking at the moment. There are some questions that I'd like to get feedback on from both the DevOps team and our Go experts. In addition to any feedback here, I imagine we'll be talking about this in the Working Group calls this week.


Trevor Conn
Senior Principal Software Engineer
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA


From: EdgeX-TSC-DevOps@... <EdgeX-TSC-DevOps@...> on behalf of Gregg, James R <james.r.gregg@...>
Sent: Monday, January 7, 2019 12:21 PM
To: Conn, Trevor; edgex-tsc-devops@...
Subject: Re: [Edgex-tsc-devops] Preliminary Doc on Moving to Go Modules
 

[EXTERNAL EMAIL]

Thanks Trevor.

I’ll add this to the agenda for this week.

 

~James R.Gregg

 

From: EdgeX-TSC-DevOps@... [mailto:EdgeX-TSC-DevOps@...] On Behalf Of Trevor.Conn@...
Sent: Monday, January 7, 2019 11:19 AM
To: edgex-tsc-devops@...
Subject: [Edgex-tsc-devops] Preliminary Doc on Moving to Go Modules

 

Hi all -- I've attached a document that contains everything I can think of w/r/t accommodating Go Modules in our CI/CD pipeline. Moving to modules for dependency mgmt is a goal for the Edinburgh deliverable.

 

The intent is not to consider the attached doc as a final draft or plan. I'd like to use it as a basis for discussion during this week's upcoming DevOps call. From this we can work toward a plan so that we can have this done well in advance of mid-April.

 

Thanks.

 

Trevor Conn
Senior Principal Software Engineer
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA

JSON Marshal vs Decode

Trevor.Conn@...
 

Hi all – I’m sending this email out as a follow up to today’s Core WG call. The recording of the call can be found on the wiki page and discussion pertinent to this topic starts at about the 30 minute mark.

 

https://wiki.edgexfoundry.org/display/FA/Core+Working+Group

 

The issue in a nutshell is that in large measure throughout the EdgeX Go services, we ingest and deserialize JSON requests in the following manner:

 

var e models.Event

dec := json.NewDecoder(r.Body)

err := dec.Decode(&e)

 

Or, alternatively stated

 

var pw models.ProvisionWatcher
json.NewDecoder(r.Body).Decode(&pw)

 

The mechanism used here to decode the request body from a stream containing JSON into a given type does NOT cause the custom unmarshaling logic present in many contracts to be called. This was proven and demo’ed on the call. In essence, this means we have an issue guaranteeing the integrity of requests received by all services in edgex-go that utilize the above pattern.

 

I presented a solution to this problem on the call. The depth to which we embrace the solution and the associated timeframe is TBD. If you wish to more closely review the code that I showed during the call, I have made that available via the following feature branches.

 

https://github.com/tsconn23/edgex-go/tree/marshal_bug

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug

 

If you are interested in this topic, I am happy to take any questions/comments via this email thread or Slack. Thanks.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: JSON Marshal vs Decode

Trevor.Conn@...
 

Whoops – It’s always good to commit one’s changes, not simply add them. They’re there now. Thanks for the catch, Lenny.

 

Trevor

 

From: Goodell, Leonard <leonard.goodell@...>
Sent: Thursday, March 7, 2019 2:01 PM
To: Conn, Trevor; edgex-tsc-core@...; EdgeX-GoLang@...
Subject: RE: JSON Marshal vs Decode

 

[EXTERNAL EMAIL]

Hi Trevor,

  Not seeing your changes. Did you push them to your branches. i.e. “This branch is even with edgexfoundry:master.” 😉

 

Thanks!

   Lenny

 

From: EdgeX-TSC-Core@... <EdgeX-TSC-Core@...> On Behalf Of Trevor.Conn@...
Sent: Thursday, March 07, 2019 11:43 AM
To: edgex-tsc-core@...; EdgeX-GoLang@...
Subject: [Edgex-tsc-core] JSON Marshal vs Decode

 

Hi all – I’m sending this email out as a follow up to today’s Core WG call. The recording of the call can be found on the wiki page and discussion pertinent to this topic starts at about the 30 minute mark.

 

https://wiki.edgexfoundry.org/display/FA/Core+Working+Group

 

The issue in a nutshell is that in large measure throughout the EdgeX Go services, we ingest and deserialize JSON requests in the following manner:

 

var e models.Event

dec := json.NewDecoder(r.Body)

err := dec.Decode(&e)

 

Or, alternatively stated

 

var pw models.ProvisionWatcher
json.NewDecoder(r.Body).Decode(&pw)

 

The mechanism used here to decode the request body from a stream containing JSON into a given type does NOT cause the custom unmarshaling logic present in many contracts to be called. This was proven and demo’ed on the call. In essence, this means we have an issue guaranteeing the integrity of requests received by all services in edgex-go that utilize the above pattern.

 

I presented a solution to this problem on the call. The depth to which we embrace the solution and the associated timeframe is TBD. If you wish to more closely review the code that I showed during the call, I have made that available via the following feature branches.

 

https://github.com/tsconn23/edgex-go/tree/marshal_bug

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug

 

If you are interested in this topic, I am happy to take any questions/comments via this email thread or Slack. Thanks.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: JSON Marshal vs Decode

Goodell, Leonard <leonard.goodell@...>
 

Hi Trevor,

  Not seeing your changes. Did you push them to your branches. i.e. “This branch is even with edgexfoundry:master.” 😉

 

Thanks!

   Lenny

 

From: EdgeX-TSC-Core@... <EdgeX-TSC-Core@...> On Behalf Of Trevor.Conn@...
Sent: Thursday, March 07, 2019 11:43 AM
To: edgex-tsc-core@...; EdgeX-GoLang@...
Subject: [Edgex-tsc-core] JSON Marshal vs Decode

 

Hi all – I’m sending this email out as a follow up to today’s Core WG call. The recording of the call can be found on the wiki page and discussion pertinent to this topic starts at about the 30 minute mark.

 

https://wiki.edgexfoundry.org/display/FA/Core+Working+Group

 

The issue in a nutshell is that in large measure throughout the EdgeX Go services, we ingest and deserialize JSON requests in the following manner:

 

var e models.Event

dec := json.NewDecoder(r.Body)

err := dec.Decode(&e)

 

Or, alternatively stated

 

var pw models.ProvisionWatcher
json.NewDecoder(r.Body).Decode(&pw)

 

The mechanism used here to decode the request body from a stream containing JSON into a given type does NOT cause the custom unmarshaling logic present in many contracts to be called. This was proven and demo’ed on the call. In essence, this means we have an issue guaranteeing the integrity of requests received by all services in edgex-go that utilize the above pattern.

 

I presented a solution to this problem on the call. The depth to which we embrace the solution and the associated timeframe is TBD. If you wish to more closely review the code that I showed during the call, I have made that available via the following feature branches.

 

https://github.com/tsconn23/edgex-go/tree/marshal_bug

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug

 

If you are interested in this topic, I am happy to take any questions/comments via this email thread or Slack. Thanks.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: JSON Marshal vs Decode

Ian Johnson
 

Hi Trevor, sorry for missing the call today so perhaps I'm missing some context here, but why doesn't the custom unmarshaller method get called for the type you are using? 

A problem I see with the solution you have in your branch is that you're unmarshalling the json and then you re-marshal it again just to get the custom error checking logic in your marshaller to run. This seems both inefficent and confusing.

I think that a better way to handle this is to re-define your custom Unmarshaller for a pointer to the type as that will cause it to run and be used. As you have it right now, it's I don't think it's being used because you call Decode you're using a pointer, and as such unmarshalling into a pointer, but the custom method you've defined is a method on a object directly and thus is ignored. However you will find if you try the trivial implementation for this Unmarshaller which is just to call json.Unmarshal on the data passed in to your custom unmarshaller method that you will enter an infinite loop. A good trick I've seen before is to use a type alias and an "anonymous struct" with struct embedding to run the generic unmarshaller on the data you're provided but in a way that it doesn't call the custom method you've defined. You can see an example of this here: https://gist.github.com/anonymouse64/bd0321890ce34ba39b92c536bc9080d0
I recommend this approach over the re-encoding method you have today.

Thanks,
Ian 


On Thu, Mar 7, 2019 at 12:43 PM <Trevor.Conn@...> wrote:

Hi all – I’m sending this email out as a follow up to today’s Core WG call. The recording of the call can be found on the wiki page and discussion pertinent to this topic starts at about the 30 minute mark.

 

https://wiki.edgexfoundry.org/display/FA/Core+Working+Group

 

The issue in a nutshell is that in large measure throughout the EdgeX Go services, we ingest and deserialize JSON requests in the following manner:

 

var e models.Event

dec := json.NewDecoder(r.Body)

err := dec.Decode(&e)

 

Or, alternatively stated

 

var pw models.ProvisionWatcher
json.NewDecoder(r.Body).Decode(&pw)

 

The mechanism used here to decode the request body from a stream containing JSON into a given type does NOT cause the custom unmarshaling logic present in many contracts to be called. This was proven and demo’ed on the call. In essence, this means we have an issue guaranteeing the integrity of requests received by all services in edgex-go that utilize the above pattern.

 

I presented a solution to this problem on the call. The depth to which we embrace the solution and the associated timeframe is TBD. If you wish to more closely review the code that I showed during the call, I have made that available via the following feature branches.

 

https://github.com/tsconn23/edgex-go/tree/marshal_bug

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug

 

If you are interested in this topic, I am happy to take any questions/comments via this email thread or Slack. Thanks.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: JSON Marshal vs Decode

Trevor.Conn@...
 

Hi Ian – See below in blue

 

Trevor

 

From: Ian Johnson <ian.johnson@...>
Sent: Thursday, March 7, 2019 5:04 PM
To: Conn, Trevor
Cc: edgex-tsc-core@...; EdgeX-GoLang@...
Subject: Re: [Edgex-golang] JSON Marshal vs Decode

 

[EXTERNAL EMAIL]

Hi Trevor, sorry for missing the call today so perhaps I'm missing some context here, but why doesn't the custom unmarshaller method get called for the type you are using? 

 

We covered this pretty extensively in the call. If you’ve watched that already and still have a question, we should talk 1x1.

 

A problem I see with the solution you have in your branch is that you're unmarshalling the json and then you re-marshal it again just to get the custom error checking logic in your marshaller to run. This seems both inefficent and confusing.

 

Please listen to the call. I did this, actually, because of some feedback you provided some time ago on one of the SMA PRs. I would like to find a better solution.

 

I think that a better way to handle this is to re-define your custom Unmarshaller for a pointer to the type as that will cause it to run and be used. As you have it right now, it's I don't think it's being used because you call Decode you're using a pointer, and as such unmarshalling into a pointer, but the custom method you've defined is a method on a object directly and thus is ignored. However you will find if you try the trivial implementation for this Unmarshaller which is just to call json.Unmarshal on the data passed in to your custom unmarshaller method that you will enter an infinite loop. A good trick I've seen before is to use a type alias and an "anonymous struct" with struct embedding to run the generic unmarshaller on the data you're provided but in a way that it doesn't call the custom method you've defined. You can see an example of this here: https://gist.github.com/anonymouse64/bd0321890ce34ba39b92c536bc9080d0

I recommend this approach over the re-encoding method you have today.

 

I will take a look

 

Thanks,

Ian 

On Thu, Mar 7, 2019 at 12:43 PM <Trevor.Conn@...> wrote:

Hi all – I’m sending this email out as a follow up to today’s Core WG call. The recording of the call can be found on the wiki page and discussion pertinent to this topic starts at about the 30 minute mark.

 

https://wiki.edgexfoundry.org/display/FA/Core+Working+Group

 

The issue in a nutshell is that in large measure throughout the EdgeX Go services, we ingest and deserialize JSON requests in the following manner:

 

var e models.Event

dec := json.NewDecoder(r.Body)

err := dec.Decode(&e)

 

Or, alternatively stated

 

var pw models.ProvisionWatcher
json.NewDecoder(r.Body).Decode(&pw)

 

The mechanism used here to decode the request body from a stream containing JSON into a given type does NOT cause the custom unmarshaling logic present in many contracts to be called. This was proven and demo’ed on the call. In essence, this means we have an issue guaranteeing the integrity of requests received by all services in edgex-go that utilize the above pattern.

 

I presented a solution to this problem on the call. The depth to which we embrace the solution and the associated timeframe is TBD. If you wish to more closely review the code that I showed during the call, I have made that available via the following feature branches.

 

https://github.com/tsconn23/edgex-go/tree/marshal_bug

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug

 

If you are interested in this topic, I am happy to take any questions/comments via this email thread or Slack. Thanks.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: JSON Marshal vs Decode

Trevor.Conn@...
 

We may be moving towards a better solution with Ian’s suggestion.

 

I have created a follow up branch here:

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug_ian

 

Only the contracts require changes in this case and, as Ian says below, the change is to add the Unmarshal function to the ProvisionWatcher where the receiver is a pointer. There’s still one problem though, suggestions are welcome.

 

This still doesn’t fix the bug. I can now step into the Unmarshal function, so we’re closer. But the related Unmarshal in the OperatingState is never called because it’s not supplied on the request – in either the ProvisionWatcher itself or the related DeviceService.

 

Here is the example request I am using to test. This comes from the core-metadata-importer collection from the blackbox tests.

 

POST http://localhost:48081/api/v1/provisionwatcher

{"_class":"org.edgexfoundry.domain.meta.ProvisionWatcher","name":"lonworks watcher test","identifiers":{"MAC":"00-05-1C-C1-99-99","HTTP":"10.0.0.4"},"origin":1471806386919,"profile":{"name":"variable speed motor profile"},"service":{"name":"home variable speed motor device service"}}

 

If you’re going to test this locally you will need to run the core-metadata-importer collection in your local environment first due to the Device Service relationship.

 

Trevor

 

P.S. Even with this, I still think having a more streamlined request makes sense. But if we can work this out, it could be a good middling step that we can implement for Edinburgh.

 

From: Ian Johnson <ian.johnson@...>
Sent: Thursday, March 7, 2019 5:04 PM
To: Conn, Trevor
Cc: edgex-tsc-core@...; EdgeX-GoLang@...
Subject: Re: [Edgex-golang] JSON Marshal vs Decode

 

[EXTERNAL EMAIL]

Hi Trevor, sorry for missing the call today so perhaps I'm missing some context here, but why doesn't the custom unmarshaller method get called for the type you are using? 

 

A problem I see with the solution you have in your branch is that you're unmarshalling the json and then you re-marshal it again just to get the custom error checking logic in your marshaller to run. This seems both inefficent and confusing.

 

I think that a better way to handle this is to re-define your custom Unmarshaller for a pointer to the type as that will cause it to run and be used. As you have it right now, it's I don't think it's being used because you call Decode you're using a pointer, and as such unmarshalling into a pointer, but the custom method you've defined is a method on a object directly and thus is ignored. However you will find if you try the trivial implementation for this Unmarshaller which is just to call json.Unmarshal on the data passed in to your custom unmarshaller method that you will enter an infinite loop. A good trick I've seen before is to use a type alias and an "anonymous struct" with struct embedding to run the generic unmarshaller on the data you're provided but in a way that it doesn't call the custom method you've defined. You can see an example of this here: https://gist.github.com/anonymouse64/bd0321890ce34ba39b92c536bc9080d0

I recommend this approach over the re-encoding method you have today.

 

Thanks,

Ian 

On Thu, Mar 7, 2019 at 12:43 PM <Trevor.Conn@...> wrote:

Hi all – I’m sending this email out as a follow up to today’s Core WG call. The recording of the call can be found on the wiki page and discussion pertinent to this topic starts at about the 30 minute mark.

 

https://wiki.edgexfoundry.org/display/FA/Core+Working+Group

 

The issue in a nutshell is that in large measure throughout the EdgeX Go services, we ingest and deserialize JSON requests in the following manner:

 

var e models.Event

dec := json.NewDecoder(r.Body)

err := dec.Decode(&e)

 

Or, alternatively stated

 

var pw models.ProvisionWatcher
json.NewDecoder(r.Body).Decode(&pw)

 

The mechanism used here to decode the request body from a stream containing JSON into a given type does NOT cause the custom unmarshaling logic present in many contracts to be called. This was proven and demo’ed on the call. In essence, this means we have an issue guaranteeing the integrity of requests received by all services in edgex-go that utilize the above pattern.

 

I presented a solution to this problem on the call. The depth to which we embrace the solution and the associated timeframe is TBD. If you wish to more closely review the code that I showed during the call, I have made that available via the following feature branches.

 

https://github.com/tsconn23/edgex-go/tree/marshal_bug

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug

 

If you are interested in this topic, I am happy to take any questions/comments via this email thread or Slack. Thanks.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Re: JSON Marshal vs Decode

Goodell, Leonard
 

So the issue is the custom validation can’t run if the incoming JSON does exist for that object, which makes sense. Seems to me it is then the responsibility of the parent object’s custom validation to handle this case, i.e. simple empty value check.

 

In this case have UnmarshalJSON() in models/provisionwatcher.go should check a.OperatingState for empty string or constant representing ‘empty’ before setting it.

 

Lenny

 

From: EdgeX-GoLang@... <EdgeX-GoLang@...> On Behalf Of Trevor.Conn@...
Sent: Thursday, March 07, 2019 4:58 PM
To: ian.johnson@...
Cc: edgex-tsc-core@...; EdgeX-GoLang@...
Subject: Re: [Edgex-golang] JSON Marshal vs Decode

 

We may be moving towards a better solution with Ian’s suggestion.

 

I have created a follow up branch here:

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug_ian

 

Only the contracts require changes in this case and, as Ian says below, the change is to add the Unmarshal function to the ProvisionWatcher where the receiver is a pointer. There’s still one problem though, suggestions are welcome.

 

This still doesn’t fix the bug. I can now step into the Unmarshal function, so we’re closer. But the related Unmarshal in the OperatingState is never called because it’s not supplied on the request – in either the ProvisionWatcher itself or the related DeviceService.

 

Here is the example request I am using to test. This comes from the core-metadata-importer collection from the blackbox tests.

 

POST http://localhost:48081/api/v1/provisionwatcher

{"_class":"org.edgexfoundry.domain.meta.ProvisionWatcher","name":"lonworks watcher test","identifiers":{"MAC":"00-05-1C-C1-99-99","HTTP":"10.0.0.4"},"origin":1471806386919,"profile":{"name":"variable speed motor profile"},"service":{"name":"home variable speed motor device service"}}

 

If you’re going to test this locally you will need to run the core-metadata-importer collection in your local environment first due to the Device Service relationship.

 

Trevor

 

P.S. Even with this, I still think having a more streamlined request makes sense. But if we can work this out, it could be a good middling step that we can implement for Edinburgh.

 

From: Ian Johnson <ian.johnson@...>
Sent: Thursday, March 7, 2019 5:04 PM
To: Conn, Trevor
Cc: edgex-tsc-core@...; EdgeX-GoLang@...
Subject: Re: [Edgex-golang] JSON Marshal vs Decode

 

[EXTERNAL EMAIL]

Hi Trevor, sorry for missing the call today so perhaps I'm missing some context here, but why doesn't the custom unmarshaller method get called for the type you are using? 

 

A problem I see with the solution you have in your branch is that you're unmarshalling the json and then you re-marshal it again just to get the custom error checking logic in your marshaller to run. This seems both inefficent and confusing.

 

I think that a better way to handle this is to re-define your custom Unmarshaller for a pointer to the type as that will cause it to run and be used. As you have it right now, it's I don't think it's being used because you call Decode you're using a pointer, and as such unmarshalling into a pointer, but the custom method you've defined is a method on a object directly and thus is ignored. However you will find if you try the trivial implementation for this Unmarshaller which is just to call json.Unmarshal on the data passed in to your custom unmarshaller method that you will enter an infinite loop. A good trick I've seen before is to use a type alias and an "anonymous struct" with struct embedding to run the generic unmarshaller on the data you're provided but in a way that it doesn't call the custom method you've defined. You can see an example of this here: https://gist.github.com/anonymouse64/bd0321890ce34ba39b92c536bc9080d0

I recommend this approach over the re-encoding method you have today.

 

Thanks,

Ian 

On Thu, Mar 7, 2019 at 12:43 PM <Trevor.Conn@...> wrote:

Hi all – I’m sending this email out as a follow up to today’s Core WG call. The recording of the call can be found on the wiki page and discussion pertinent to this topic starts at about the 30 minute mark.

 

https://wiki.edgexfoundry.org/display/FA/Core+Working+Group

 

The issue in a nutshell is that in large measure throughout the EdgeX Go services, we ingest and deserialize JSON requests in the following manner:

 

var e models.Event

dec := json.NewDecoder(r.Body)

err := dec.Decode(&e)

 

Or, alternatively stated

 

var pw models.ProvisionWatcher
json.NewDecoder(r.Body).Decode(&pw)

 

The mechanism used here to decode the request body from a stream containing JSON into a given type does NOT cause the custom unmarshaling logic present in many contracts to be called. This was proven and demo’ed on the call. In essence, this means we have an issue guaranteeing the integrity of requests received by all services in edgex-go that utilize the above pattern.

 

I presented a solution to this problem on the call. The depth to which we embrace the solution and the associated timeframe is TBD. If you wish to more closely review the code that I showed during the call, I have made that available via the following feature branches.

 

https://github.com/tsconn23/edgex-go/tree/marshal_bug

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug

 

If you are interested in this topic, I am happy to take any questions/comments via this email thread or Slack. Thanks.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA