Re: [Edgex-golang] JSON Marshal vs Decode

Trevor.Conn@...
 

Hi Ian – See below in blue

 

Trevor

 

From: Ian Johnson <ian.johnson@...>
Sent: Thursday, March 7, 2019 5:04 PM
To: Conn, Trevor
Cc: edgex-tsc-core@...; EdgeX-GoLang@...
Subject: Re: [Edgex-golang] JSON Marshal vs Decode

 

[EXTERNAL EMAIL]

Hi Trevor, sorry for missing the call today so perhaps I'm missing some context here, but why doesn't the custom unmarshaller method get called for the type you are using? 

 

We covered this pretty extensively in the call. If you’ve watched that already and still have a question, we should talk 1x1.

 

A problem I see with the solution you have in your branch is that you're unmarshalling the json and then you re-marshal it again just to get the custom error checking logic in your marshaller to run. This seems both inefficent and confusing.

 

Please listen to the call. I did this, actually, because of some feedback you provided some time ago on one of the SMA PRs. I would like to find a better solution.

 

I think that a better way to handle this is to re-define your custom Unmarshaller for a pointer to the type as that will cause it to run and be used. As you have it right now, it's I don't think it's being used because you call Decode you're using a pointer, and as such unmarshalling into a pointer, but the custom method you've defined is a method on a object directly and thus is ignored. However you will find if you try the trivial implementation for this Unmarshaller which is just to call json.Unmarshal on the data passed in to your custom unmarshaller method that you will enter an infinite loop. A good trick I've seen before is to use a type alias and an "anonymous struct" with struct embedding to run the generic unmarshaller on the data you're provided but in a way that it doesn't call the custom method you've defined. You can see an example of this here: https://gist.github.com/anonymouse64/bd0321890ce34ba39b92c536bc9080d0

I recommend this approach over the re-encoding method you have today.

 

I will take a look

 

Thanks,

Ian 

On Thu, Mar 7, 2019 at 12:43 PM <Trevor.Conn@...> wrote:

Hi all – I’m sending this email out as a follow up to today’s Core WG call. The recording of the call can be found on the wiki page and discussion pertinent to this topic starts at about the 30 minute mark.

 

https://wiki.edgexfoundry.org/display/FA/Core+Working+Group

 

The issue in a nutshell is that in large measure throughout the EdgeX Go services, we ingest and deserialize JSON requests in the following manner:

 

var e models.Event

dec := json.NewDecoder(r.Body)

err := dec.Decode(&e)

 

Or, alternatively stated

 

var pw models.ProvisionWatcher
json.NewDecoder(r.Body).Decode(&pw)

 

The mechanism used here to decode the request body from a stream containing JSON into a given type does NOT cause the custom unmarshaling logic present in many contracts to be called. This was proven and demo’ed on the call. In essence, this means we have an issue guaranteeing the integrity of requests received by all services in edgex-go that utilize the above pattern.

 

I presented a solution to this problem on the call. The depth to which we embrace the solution and the associated timeframe is TBD. If you wish to more closely review the code that I showed during the call, I have made that available via the following feature branches.

 

https://github.com/tsconn23/edgex-go/tree/marshal_bug

https://github.com/tsconn23/go-mod-core-contracts/tree/marshal_bug

 

If you are interested in this topic, I am happy to take any questions/comments via this email thread or Slack. Thanks.

 

Trevor Conn

Senior Principal Software Engineer

Dell Technologies | IoT DellTech

Trevor.Conn@...

Round Rock, TX USA

 

Join EdgeX-TSC-Core@lists.edgexfoundry.org to automatically receive all group messages.