Topics

Overview of Pre-Delhi Dynamic Device Discovery design

espy
 

As discussed during today's Device Service working group meeting, I volunteered to write a description of the original dynamic device discovery mechanism.

[Jim - feel free to comment and/or correct my description]

Regards,
/tony

---

Prior to the Delhi release, EdgeX Foundry and device services based on the Java SDK supported a feature called "dynamic device discovery". This feature was originally created in order to support wireless device protocols such as Bluetooth Low Energy, where a given device might not always be available. In such a scenario, the idea was to provide EdgeX with a set of filtering rules which could be used to dynamically add a "discovered" (i.e. available) device to the system if the rules produced a match.

This mechanism relied on Core Metadata object called a ProvisionWatcher which was used to define filtering rules.  A ProvisionWatcher has the following fields:
  • name
  • device-service - name of the owning service
  • device-profile - profile assigned to new matching devices
  • identifiers - map of attributes used to identify a device (eg. name, port, MAC, uuid, …)
  • operating-state - initial operating state for new devices
Note - identifiers can be fully-qualified or could use the wildcard ('*') for pattern matching. This allows a single ProvisionWatcher to be used to match multiple devices.

Device services (via code provided by the Java SDK) provided an endpoint called /discovery. Calling this endpoint resulted in the device service calling an internal Scan method in the device service's ProtocolDriver. The result of the Scan method was a list of available devices. For each device in this list, the ProtocolDriver provided a map of attributes that can be used to identify devices of a given protocol.  For instance for BLE devices this map might contain a MAC address and a service UUID.

The device SDK code would iterate through the scan results looking for new devices. If a new device was detected, then the SDK would attempt to match the device against any existing ProvisionWatcher objects associated with the device service.  When matches were found, the SDK would add the new device using the device profile and operating state specified in the ProvisionWatcher.

The final piece of the puzzle is Support Scheduler, which via Schedule and ScheduleEvent objects could be used to periodically call a device service's /discovery endpoint.

= Problems =

There are a few problems with this approach to dynamic device discovery.

1. Device deletion - if a device is deleted from the device service (and Core Metadata), on subsequent scans if the device was found to be available again, it would be re-added.

This problem is where the idea of device blacklists came from. In addition to deleting a device, you'd also add it to a blacklist object (in the ProvisionWatcher?). I'd argue that this is just adding to already complicated approach. Instead of black-listing the device, why not modify (or delete) the matching ProvisionWatcher before deleting the device?

2. Scan are synchronous - the /discovery endpoint didn't support asynchronous scan results.

3. ProvisionWatchers contain device profiles. This is the same problem we currently have with embedded device profiles in devices.  This makes ProvisionWatchers fairly heavy-weight objects, and adds complexity if/when a device profile is updated.

4. Scanning in a fixed period is less than ideal, as scanning can impact performance and also may impacts battery life for certain devices. Ideally the logic for determing when to scan should be the responsibilty of the device service, not pre-configured using Support Scheduler.

Example Java configuration for BLE ProvisionWatchers:

default.watcher.name=BLE-Watcher,BLE-Watcher-2,BLE-Watcher-3
default.watcher.service=device-bluetooth,device-bluetooth,device-bluetooth
default.watcher.profile=SensorTag,XDK,BLELight
default.watcher.name_identifiers=.*CC2650.*,XDK.*,.*beLight.*

Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?

James.White2@...
 

Thanks Tony.  I’ll have a look today or tomorrow, update what I can, and put it out somewhere accessible on the Wiki so that we can use it at next week’s DS WG call.

 

j

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of espy
Sent: Monday, June 3, 2019 5:53 PM
To: edgex-tsc-device-services@...
Subject: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design

 

[EXTERNAL EMAIL]

As discussed during today's Device Service working group meeting, I volunteered to write a description of the original dynamic device discovery mechanism.

[Jim - feel free to comment and/or correct my description]

Regards,
/tony

---

Prior to the Delhi release, EdgeX Foundry and device services based on the Java SDK supported a feature called "dynamic device discovery". This feature was originally created in order to support wireless device protocols such as Bluetooth Low Energy, where a given device might not always be available. In such a scenario, the idea was to provide EdgeX with a set of filtering rules which could be used to dynamically add a "discovered" (i.e. available) device to the system if the rules produced a match.

This mechanism relied on Core Metadata object called a ProvisionWatcher which was used to define filtering rules.  A ProvisionWatcher has the following fields:

  • name
  • device-service - name of the owning service
  • device-profile - profile assigned to new matching devices
  • identifiers - map of attributes used to identify a device (eg. name, port, MAC, uuid, …)
  • operating-state - initial operating state for new devices

Note - identifiers can be fully-qualified or could use the wildcard ('*') for pattern matching. This allows a single ProvisionWatcher to be used to match multiple devices.

Device services (via code provided by the Java SDK) provided an endpoint called /discovery. Calling this endpoint resulted in the device service calling an internal Scan method in the device service's ProtocolDriver. The result of the Scan method was a list of available devices. For each device in this list, the ProtocolDriver provided a map of attributes that can be used to identify devices of a given protocol.  For instance for BLE devices this map might contain a MAC address and a service UUID.

The device SDK code would iterate through the scan results looking for new devices. If a new device was detected, then the SDK would attempt to match the device against any existing ProvisionWatcher objects associated with the device service.  When matches were found, the SDK would add the new device using the device profile and operating state specified in the ProvisionWatcher.

The final piece of the puzzle is Support Scheduler, which via Schedule and ScheduleEvent objects could be used to periodically call a device service's /discovery endpoint.

= Problems =

There are a few problems with this approach to dynamic device discovery.

1. Device deletion - if a device is deleted from the device service (and Core Metadata), on subsequent scans if the device was found to be available again, it would be re-added.

This problem is where the idea of device blacklists came from. In addition to deleting a device, you'd also add it to a blacklist object (in the ProvisionWatcher?). I'd argue that this is just adding to already complicated approach. Instead of black-listing the device, why not modify (or delete) the matching ProvisionWatcher before deleting the device?

2. Scan are synchronous - the /discovery endpoint didn't support asynchronous scan results.

3. ProvisionWatchers contain device profiles. This is the same problem we currently have with embedded device profiles in devices.  This makes ProvisionWatchers fairly heavy-weight objects, and adds complexity if/when a device profile is updated.

4. Scanning in a fixed period is less than ideal, as scanning can impact performance and also may impacts battery life for certain devices. Ideally the logic for determing when to scan should be the responsibilty of the device service, not pre-configured using Support Scheduler.

Example Java configuration for BLE ProvisionWatchers:

default.watcher.name=BLE-Watcher,BLE-Watcher-2,BLE-Watcher-3
default.watcher.service=device-bluetooth,device-bluetooth,device-bluetooth
default.watcher.profile=SensorTag,XDK,BLELight
default.watcher.name_identifiers=.*CC2650.*,XDK.*,.*beLight.*

Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?

Iain Anderson
 



On 03/06/2019 23:53, espy wrote:


Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?


What if the device service were to always add any new device it found (either by some proactive mechanism of its own or as a result of the /discovery endpoint being hit), but mark it as (new attribute) "unconfigured" or similar?

You could then have a microservice that processed these new devices according to ProvisionWatcher rules, and enabled them if appropriate. It could register (could it?) with core-metadata for device update callbacks so that new devices were processed as soon as they were discovered. And if the ProvisionWatcher mechanism is not suitable in some cases, alternative services for processing new devices could be plugged in, or interactive methods could be used.

Regards,
Iain

James.White2@...
 

Iain,

 

While I could see the benefits of a separate service to handle these duties, aren’t some provisioning operations going to be unique to the device and protocol.  While handling typical metadata ops is straightforward, I could potentially see that some engagements to enable and wake a device could be pretty specific.

 

j

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of Iain Anderson
Sent: Friday, June 7, 2019 5:14 AM
To: EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design

 

[EXTERNAL EMAIL]

 

On 03/06/2019 23:53, espy wrote:



Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?


What if the device service were to always add any new device it found (either by some proactive mechanism of its own or as a result of the /discovery endpoint being hit), but mark it as (new attribute) "unconfigured" or similar?

You could then have a microservice that processed these new devices according to ProvisionWatcher rules, and enabled them if appropriate. It could register (could it?) with core-metadata for device update callbacks so that new devices were processed as soon as they were discovered. And if the ProvisionWatcher mechanism is not suitable in some cases, alternative services for processing new devices could be plugged in, or interactive methods could be used.

Regards,
Iain

Iain Anderson
 

Hi Jim, I'm only thinking about the metadata aspects here. The device service would still have to do any enable/wakeup things when a device moved into a "configured" state. This would tie in with what is being talked about in https://github.com/edgexfoundry/device-sdk-go/issues/276

Regards,
Iain

On 07/06/2019 13:12, James.White2@... wrote:

Iain,

 

While I could see the benefits of a separate service to handle these duties, aren’t some provisioning operations going to be unique to the device and protocol.  While handling typical metadata ops is straightforward, I could potentially see that some engagements to enable and wake a device could be pretty specific.

 

j

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of Iain Anderson
Sent: Friday, June 7, 2019 5:14 AM
To: EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design

 

[EXTERNAL EMAIL]

 

On 03/06/2019 23:53, espy wrote:



Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?


What if the device service were to always add any new device it found (either by some proactive mechanism of its own or as a result of the /discovery endpoint being hit), but mark it as (new attribute) "unconfigured" or similar?

You could then have a microservice that processed these new devices according to ProvisionWatcher rules, and enabled them if appropriate. It could register (could it?) with core-metadata for device update callbacks so that new devices were processed as soon as they were discovered. And if the ProvisionWatcher mechanism is not suitable in some cases, alternative services for processing new devices could be plugged in, or interactive methods could be used.

Regards,
Iain


E.Cotter@...
 

​Ian,


So would each "new" device have its unique id defined in this "discovery" endpoint?

I could see the same service potentially coming up and down and having it re-listed with provisionWatcher?


The idea of individual device services "discovering" other devices makes sense.  Distinguishing them as unique would be some what imperative unless we wanted a lot of traffic routing back and forth between all the device services. 


Eric Cotter


From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> on behalf of Iain Anderson <iain@...>
Sent: Friday, June 7, 2019 10:26 AM
To: White2, James; EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design
 

[EXTERNAL EMAIL]

Hi Jim, I'm only thinking about the metadata aspects here. The device service would still have to do any enable/wakeup things when a device moved into a "configured" state. This would tie in with what is being talked about in https://github.com/edgexfoundry/device-sdk-go/issues/276

Regards,
Iain

On 07/06/2019 13:12, James.White2@... wrote:

Iain,

 

While I could see the benefits of a separate service to handle these duties, aren’t some provisioning operations going to be unique to the device and protocol.  While handling typical metadata ops is straightforward, I could potentially see that some engagements to enable and wake a device could be pretty specific.

 

j

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of Iain Anderson
Sent: Friday, June 7, 2019 5:14 AM
To: EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design

 

[EXTERNAL EMAIL]

 

On 03/06/2019 23:53, espy wrote:



Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?


What if the device service were to always add any new device it found (either by some proactive mechanism of its own or as a result of the /discovery endpoint being hit), but mark it as (new attribute) "unconfigured" or similar?

You could then have a microservice that processed these new devices according to ProvisionWatcher rules, and enabled them if appropriate. It could register (could it?) with core-metadata for device update callbacks so that new devices were processed as soon as they were discovered. And if the ProvisionWatcher mechanism is not suitable in some cases, alternative services for processing new devices could be plugged in, or interactive methods could be used.

Regards,
Iain


Iain Anderson
 

I guess this raises a couple of questions -

1) Do we envisage multiple device services of the same type running in a topology that allows more than one of them to 'see' the same device? This gives us issues with the question of whether a device is new or not, as a DS only knows about its own devices. Running a ProvisionWatcher mechanism inside the DS can help with this if you can make the matching criteria work such that any potential device will only be matched by one DS's watcher.

2) How can we uniquely identify a device? I have assumed that if a discovered device has matching ProtocolProperties with a known device, the discovered device is in fact the one we already know. Can we be sure of that for all device types or do we need to delegate the "are these the same" test to the protocol driver?

- Iain


On 07/06/2019 16:44, E.Cotter@... wrote:

​Ian,


So would each "new" device have its unique id defined in this "discovery" endpoint?

I could see the same service potentially coming up and down and having it re-listed with provisionWatcher?


The idea of individual device services "discovering" other devices makes sense.  Distinguishing them as unique would be some what imperative unless we wanted a lot of traffic routing back and forth between all the device services. 


Eric Cotter


From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> on behalf of Iain Anderson <iain@...>
Sent: Friday, June 7, 2019 10:26 AM
To: White2, James; EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design
 

[EXTERNAL EMAIL]

Hi Jim, I'm only thinking about the metadata aspects here. The device service would still have to do any enable/wakeup things when a device moved into a "configured" state. This would tie in with what is being talked about in https://github.com/edgexfoundry/device-sdk-go/issues/276

Regards,
Iain

On 07/06/2019 13:12, James.White2@... wrote:

Iain,

 

While I could see the benefits of a separate service to handle these duties, aren’t some provisioning operations going to be unique to the device and protocol.  While handling typical metadata ops is straightforward, I could potentially see that some engagements to enable and wake a device could be pretty specific.

 

j

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of Iain Anderson
Sent: Friday, June 7, 2019 5:14 AM
To: EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design

 

[EXTERNAL EMAIL]

 

On 03/06/2019 23:53, espy wrote:



Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?


What if the device service were to always add any new device it found (either by some proactive mechanism of its own or as a result of the /discovery endpoint being hit), but mark it as (new attribute) "unconfigured" or similar?

You could then have a microservice that processed these new devices according to ProvisionWatcher rules, and enabled them if appropriate. It could register (could it?) with core-metadata for device update callbacks so that new devices were processed as soon as they were discovered. And if the ProvisionWatcher mechanism is not suitable in some cases, alternative services for processing new devices could be plugged in, or interactive methods could be used.

Regards,
Iain



E.Cotter@...
 

Iain,


1) I would think multiple device services could be very possible. I'm not sure about the same type but you could certainly 12 factor your device service and have multiple instances from a deployment perspective. But multiple instances of a like or similar device service is another topic.


"This gives us issues with the question of whether a device is new or not, as a DS only knows about its own devices. Running a ProvisionWatcher mechanism inside the DS can help with this if you can make the matching criteria work such that any potential device will only be matched by one DS's watcher.​"  ~ Iain


I think this is what Jim was alluding to.  Where each distinct device service "type" would find a way to discover it's own kin.  This could potentially eliminate a lot of extra traffic and somewhat logically partition the discovery process. 



2) My understanding is that if you had multiple device service types running you could run into similar issues. Camera(s), Temperature sensors, Access control devices, etc.  So would you not have all three of those device services presumably trying to notify ProvisionWatcher with each device services version of what it deemed a unique device?  This is given we have no distinction requirement for discovery in our devices discovered example protocols. 


Eric 


From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> on behalf of Iain Anderson <iain@...>
Sent: Friday, June 7, 2019 11:18 AM
To: EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design
 

[EXTERNAL EMAIL]

I guess this raises a couple of questions -

1) Do we envisage multiple device services of the same type running in a topology that allows more than one of them to 'see' the same device? This gives us issues with the question of whether a device is new or not, as a DS only knows about its own devices. Running a ProvisionWatcher mechanism inside the DS can help with this if you can make the matching criteria work such that any potential device will only be matched by one DS's watcher.

2) How can we uniquely identify a device? I have assumed that if a discovered device has matching ProtocolProperties with a known device, the discovered device is in fact the one we already know. Can we be sure of that for all device types or do we need to delegate the "are these the same" test to the protocol driver?

- Iain


On 07/06/2019 16:44, E.Cotter@... wrote:

​Ian,


So would each "new" device have its unique id defined in this "discovery" endpoint?

I could see the same service potentially coming up and down and having it re-listed with provisionWatcher?


The idea of individual device services "discovering" other devices makes sense.  Distinguishing them as unique would be some what imperative unless we wanted a lot of traffic routing back and forth between all the device services. 


Eric Cotter


From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> on behalf of Iain Anderson <iain@...>
Sent: Friday, June 7, 2019 10:26 AM
To: White2, James; EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design
 

[EXTERNAL EMAIL]

Hi Jim, I'm only thinking about the metadata aspects here. The device service would still have to do any enable/wakeup things when a device moved into a "configured" state. This would tie in with what is being talked about in https://github.com/edgexfoundry/device-sdk-go/issues/276

Regards,
Iain

On 07/06/2019 13:12, James.White2@... wrote:

Iain,

 

While I could see the benefits of a separate service to handle these duties, aren’t some provisioning operations going to be unique to the device and protocol.  While handling typical metadata ops is straightforward, I could potentially see that some engagements to enable and wake a device could be pretty specific.

 

j

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of Iain Anderson
Sent: Friday, June 7, 2019 5:14 AM
To: EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design

 

[EXTERNAL EMAIL]

 

On 03/06/2019 23:53, espy wrote:



Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?


What if the device service were to always add any new device it found (either by some proactive mechanism of its own or as a result of the /discovery endpoint being hit), but mark it as (new attribute) "unconfigured" or similar?

You could then have a microservice that processed these new devices according to ProvisionWatcher rules, and enabled them if appropriate. It could register (could it?) with core-metadata for device update callbacks so that new devices were processed as soon as they were discovered. And if the ProvisionWatcher mechanism is not suitable in some cases, alternative services for processing new devices could be plugged in, or interactive methods could be used.

Regards,
Iain



James.White2@...
 

Tony,

Thanks for the excellent work on this.  After digging up and rummaging through the code, I provided a few minor corrections to some of the flow and provided a sequence diagram so that if people want, they can find the code in the old Java SDK.  The substance of your content as it relates to the workflow was right on – just some of the classes mentioned were off.

 

I posted a doc with your write up, the sequence diagram and additional comments in the Wiki – so that people out of the mailing forum and in China can have access.  Find it here (PDF and Word format):  https://wiki.edgexfoundry.org/display/FA/Device+and+Device+SDK+Working+Group#DeviceandDeviceSDKWorkingGroup-Documents

 

I am not sure I follow your last point in your proposal and something we can discuss and clarify in the next WG.

 

Eventually we can/should fold the final results into your requirements document.  BTW - the requirements document is on Google docs which we should move so that those in China can have access at some point.

 

Jim

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of espy
Sent: Monday, June 3, 2019 5:53 PM
To: edgex-tsc-device-services@...
Subject: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design

 

[EXTERNAL EMAIL]

As discussed during today's Device Service working group meeting, I volunteered to write a description of the original dynamic device discovery mechanism.

[Jim - feel free to comment and/or correct my description]

Regards,
/tony

---

Prior to the Delhi release, EdgeX Foundry and device services based on the Java SDK supported a feature called "dynamic device discovery". This feature was originally created in order to support wireless device protocols such as Bluetooth Low Energy, where a given device might not always be available. In such a scenario, the idea was to provide EdgeX with a set of filtering rules which could be used to dynamically add a "discovered" (i.e. available) device to the system if the rules produced a match.

This mechanism relied on Core Metadata object called a ProvisionWatcher which was used to define filtering rules.  A ProvisionWatcher has the following fields:

  • name
  • device-service - name of the owning service
  • device-profile - profile assigned to new matching devices
  • identifiers - map of attributes used to identify a device (eg. name, port, MAC, uuid, …)
  • operating-state - initial operating state for new devices

Note - identifiers can be fully-qualified or could use the wildcard ('*') for pattern matching. This allows a single ProvisionWatcher to be used to match multiple devices.

Device services (via code provided by the Java SDK) provided an endpoint called /discovery. Calling this endpoint resulted in the device service calling an internal Scan method in the device service's ProtocolDriver. The result of the Scan method was a list of available devices. For each device in this list, the ProtocolDriver provided a map of attributes that can be used to identify devices of a given protocol.  For instance for BLE devices this map might contain a MAC address and a service UUID.

The device SDK code would iterate through the scan results looking for new devices. If a new device was detected, then the SDK would attempt to match the device against any existing ProvisionWatcher objects associated with the device service.  When matches were found, the SDK would add the new device using the device profile and operating state specified in the ProvisionWatcher.

The final piece of the puzzle is Support Scheduler, which via Schedule and ScheduleEvent objects could be used to periodically call a device service's /discovery endpoint.

= Problems =

There are a few problems with this approach to dynamic device discovery.

1. Device deletion - if a device is deleted from the device service (and Core Metadata), on subsequent scans if the device was found to be available again, it would be re-added.

This problem is where the idea of device blacklists came from. In addition to deleting a device, you'd also add it to a blacklist object (in the ProvisionWatcher?). I'd argue that this is just adding to already complicated approach. Instead of black-listing the device, why not modify (or delete) the matching ProvisionWatcher before deleting the device?

2. Scan are synchronous - the /discovery endpoint didn't support asynchronous scan results.

3. ProvisionWatchers contain device profiles. This is the same problem we currently have with embedded device profiles in devices.  This makes ProvisionWatchers fairly heavy-weight objects, and adds complexity if/when a device profile is updated.

4. Scanning in a fixed period is less than ideal, as scanning can impact performance and also may impacts battery life for certain devices. Ideally the logic for determing when to scan should be the responsibilty of the device service, not pre-configured using Support Scheduler.

Example Java configuration for BLE ProvisionWatchers:

default.watcher.name=BLE-Watcher,BLE-Watcher-2,BLE-Watcher-3
default.watcher.service=device-bluetooth,device-bluetooth,device-bluetooth
default.watcher.profile=SensorTag,XDK,BLELight
default.watcher.name_identifiers=.*CC2650.*,XDK.*,.*beLight.*

Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?

Malini Bhandaru
 

Thank you Tony and Jim!

Yet to figure out how to get a comment pin in the google doc! A few comments:

  1. I like that we maintain declaratively (file/db), able to view/maintain state across reboots, devices that we seek to “ignore”. Even if a device is already registered, it will show up during a scan, so a quick way to move on is nice, perhaps some canonical naming convention using the identifier attributes, skip if in { ignore or covered} list.
  2. VMware in some early PoC used the discovery feature of EdgeX. Their newest IoT monitoring and management solution explicitly registers.
  3. Both “discover” and “Register” approaches are good to support.

Perhaps the VMpare “register” approach might with some transform tool render itself as a provision watcher.

  1. Definitely like the discovery frequency to be protocol /device-service specific.  By default enable/disable based on energy consumption?
  2. That said, there is value to the end-point, to trigger discovery. For instance air-drop on our iphones scans when we click the air-drop button.  
  3. Config-file/CLI/UI edits of these settings makes sense.

Regards

Malini

 

From: <EdgeX-TSC-Device-Services@...> on behalf of "James.White2 via Lists.Edgexfoundry.Org" <James.White2=dell.com@...>
Reply-To: "White, James (EMC)" <James_White2@...>
Date: Friday, June 7, 2019 at 12:48 PM
To: "espy@..." <espy@...>, "EdgeX-TSC-Device-Services@..." <EdgeX-TSC-Device-Services@...>
Cc: "EdgeX-TSC-Device-Services@..." <EdgeX-TSC-Device-Services@...>
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design

 

Tony,

Thanks for the excellent work on this.  After digging up and rummaging through the code, I provided a few minor corrections to some of the flow and provided a sequence diagram so that if people want, they can find the code in the old Java SDK.  The substance of your content as it relates to the workflow was right on – just some of the classes mentioned were off.

 

I posted a doc with your write up, the sequence diagram and additional comments in the Wiki – so that people out of the mailing forum and in China can have access.  Find it here (PDF and Word format):  https://wiki.edgexfoundry.org/display/FA/Device+and+Device+SDK+Working+Group#DeviceandDeviceSDKWorkingGroup-Documents

 

I am not sure I follow your last point in your proposal and something we can discuss and clarify in the next WG.

 

Eventually we can/should fold the final results into your requirements document.  BTW - the requirements document is on Google docs which we should move so that those in China can have access at some point.

 

Jim

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of espy
Sent: Monday, June 3, 2019 5:53 PM
To: edgex-tsc-device-services@...
Subject: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design

 

[EXTERNAL EMAIL]

As discussed during today's Device Service working group meeting, I volunteered to write a description of the original dynamic device discovery mechanism.

[Jim - feel free to comment and/or correct my description]

Regards,
/tony

---

Prior to the Delhi release, EdgeX Foundry and device services based on the Java SDK supported a feature called "dynamic device discovery". This feature was originally created in order to support wireless device protocols such as Bluetooth Low Energy, where a given device might not always be available. In such a scenario, the idea was to provide EdgeX with a set of filtering rules which could be used to dynamically add a "discovered" (i.e. available) device to the system if the rules produced a match.

This mechanism relied on Core Metadata object called a ProvisionWatcher which was used to define filtering rules.  A ProvisionWatcher has the following fields:

  • name
  • device-service - name of the owning service
  • device-profile - profile assigned to new matching devices
  • identifiers - map of attributes used to identify a device (eg. name, port, MAC, uuid, …)
  • operating-state - initial operating state for new devices

Note - identifiers can be fully-qualified or could use the wildcard ('*') for pattern matching. This allows a single ProvisionWatcher to be used to match multiple devices.

Device services (via code provided by the Java SDK) provided an endpoint called /discovery. Calling this endpoint resulted in the device service calling an internal Scan method in the device service's ProtocolDriver. The result of the Scan method was a list of available devices. For each device in this list, the ProtocolDriver provided a map of attributes that can be used to identify devices of a given protocol.  For instance for BLE devices this map might contain a MAC address and a service UUID.

The device SDK code would iterate through the scan results looking for new devices. If a new device was detected, then the SDK would attempt to match the device against any existing ProvisionWatcher objects associated with the device service.  When matches were found, the SDK would add the new device using the device profile and operating state specified in the ProvisionWatcher.

The final piece of the puzzle is Support Scheduler, which via Schedule and ScheduleEvent objects could be used to periodically call a device service's /discovery endpoint.

= Problems =

There are a few problems with this approach to dynamic device discovery.

1. Device deletion - if a device is deleted from the device service (and Core Metadata), on subsequent scans if the device was found to be available again, it would be re-added.

This problem is where the idea of device blacklists came from. In addition to deleting a device, you'd also add it to a blacklist object (in the ProvisionWatcher?). I'd argue that this is just adding to already complicated approach. Instead of black-listing the device, why not modify (or delete) the matching ProvisionWatcher before deleting the device?

2. Scan are synchronous - the /discovery endpoint didn't support asynchronous scan results.

3. ProvisionWatchers contain device profiles. This is the same problem we currently have with embedded device profiles in devices.  This makes ProvisionWatchers fairly heavy-weight objects, and adds complexity if/when a device profile is updated.

4. Scanning in a fixed period is less than ideal, as scanning can impact performance and also may impacts battery life for certain devices. Ideally the logic for determing when to scan should be the responsibilty of the device service, not pre-configured using Support Scheduler.

Example Java configuration for BLE ProvisionWatchers:

default.watcher.name=BLE-Watcher,BLE-Watcher-2,BLE-Watcher-3
default.watcher.service=device-bluetooth,device-bluetooth,device-bluetooth
default.watcher.profile=SensorTag,XDK,BLELight
default.watcher.name_identifiers=.*CC2650.*,XDK.*,.*beLight.*

Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?

espy
 

On 6/3/19 6:53 PM, Tony Espy wrote:

As discussed during today's Device Service working group meeting, I volunteered to write a description of the original dynamic device discovery mechanism.

[Jim - feel free to comment and/or correct my description]

Regards,
/tony

---

Prior to the Delhi release, EdgeX Foundry and device services based on the Java SDK supported a feature called "dynamic device discovery". This feature was originally created in order to support wireless device protocols such as Bluetooth Low Energy, where a given device might not always be available. In such a scenario, the idea was to provide EdgeX with a set of filtering rules which could be used to dynamically add a "discovered" (i.e. available) device to the system if the rules produced a match.

This mechanism relied on Core Metadata object called a ProvisionWatcher which was used to define filtering rules.  A ProvisionWatcher has the following fields:
  • name
  • device-service - name of the owning service
  • device-profile - profile assigned to new matching devices
  • identifiers - map of attributes used to identify a device (eg. name, port, MAC, uuid, …)
  • operating-state - initial operating state for new devices
Note - identifiers can be fully-qualified or could use the wildcard ('*') for pattern matching. This allows a single ProvisionWatcher to be used to match multiple devices.

Device services (via code provided by the Java SDK) provided an endpoint called /discovery. Calling this endpoint resulted in the device service calling an internal Scan method in the device service's ProtocolDriver. The result of the Scan method was a list of available devices. For each device in this list, the ProtocolDriver provided a map of attributes that can be used to identify devices of a given protocol.  For instance for BLE devices this map might contain a MAC address and a service UUID.

The device SDK code would iterate through the scan results looking for new devices. If a new device was detected, then the SDK would attempt to match the device against any existing ProvisionWatcher objects associated with the device service.  When matches were found, the SDK would add the new device using the device profile and operating state specified in the ProvisionWatcher.

As long promised, here's my suggestion on how we handle blacklisting devices. As pointed out above, a ProvisionWatcher has a field called "identifiers" which is a map of strings which represent device attributes which can be used to identify a device. The key in this case identifies the attribute to match against (e.g. 'Name', 'MAC', 'uuid', ...). When a device service ProtocolDriver returns a list of devices in response to a Scan call, a corresponding map of device identifiers is returned. The SDK then looks for matches between the devices in the scan list and it's current ProvisionWatchers. The values in the ProvisionWatcher identifiers map support a basic glob-style [1] wildcard character "*".

So if a ProvisionWatcher defined identifiers as [["Name", "acme*"], "MAC", "38:Fc:E7*"], then any devices found in a subsequent scan results list with 'Name' that matches "acme*" and MAC address with the prefix "38:Fc:E7" would be added (if they don't already exist).

In order to exclude specific devices when a pattern has many potential matches, the idea has been tossed about to add a blacklist. Although you could argue that adjusting identifiers to exclude devices is one way to go, adding an explicit blacklist mechanism also has merit.

I think the simplest approach would be to add a new field called "blacklist" which is map of string arrays/lists which define values which if found in the attributes of a scanned device, prevent it from being added.  With the example above, if the device "acme13" was found to be compromised, all you'd need to do is add "acme13" to the "Names" list, and the original identifiers would still match against all other devices with the name prefix "acme". Likewise you could also specify a different identifier (if supported by the device service) in the blacklist (i.e. a specific MAC address).

To keep it's simple, let's just call the new field blacklist and make it a map of string slices:

blacklist    map[string][]

Two questions:

1) Should the blacklist support the "*" wildcard char?

2) Should we implement glob style pattern matching (eg. '?' or [...])?

Regards,
/tony

[1] https://en.wikipedia.org/wiki/Glob_(programming)


The final piece of the puzzle is Support Scheduler, which via Schedule and ScheduleEvent objects could be used to periodically call a device service's /discovery endpoint.

= Problems =

There are a few problems with this approach to dynamic device discovery.

1. Device deletion - if a device is deleted from the device service (and Core Metadata), on subsequent scans if the device was found to be available again, it would be re-added.

This problem is where the idea of device blacklists came from. In addition to deleting a device, you'd also add it to a blacklist object (in the ProvisionWatcher?). I'd argue that this is just adding to already complicated approach. Instead of black-listing the device, why not modify (or delete) the matching ProvisionWatcher before deleting the device?

2. Scan are synchronous - the /discovery endpoint didn't support asynchronous scan results.

3. ProvisionWatchers contain device profiles. This is the same problem we currently have with embedded device profiles in devices.  This makes ProvisionWatchers fairly heavy-weight objects, and adds complexity if/when a device profile is updated.

4. Scanning in a fixed period is less than ideal, as scanning can impact performance and also may impacts battery life for certain devices. Ideally the logic for determing when to scan should be the responsibilty of the device service, not pre-configured using Support Scheduler.

Example Java configuration for BLE ProvisionWatchers:

default.watcher.name=BLE-Watcher,BLE-Watcher-2,BLE-Watcher-3
default.watcher.service=device-bluetooth,device-bluetooth,device-bluetooth
default.watcher.profile=SensorTag,XDK,BLELight
default.watcher.name_identifiers=.*CC2650.*,XDK.*,.*beLight.*

Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?

Iain Anderson
 

I've updated https://github.com/edgexfoundry/go-mod-core-contracts/issues/111 in light of the suggestions here. Please review and comment.

Regards,
Iain


On 22/07/2019 18:54, espy wrote:

On 6/3/19 6:53 PM, Tony Espy wrote:

As discussed during today's Device Service working group meeting, I volunteered to write a description of the original dynamic device discovery mechanism.

[Jim - feel free to comment and/or correct my description]

Regards,
/tony

---

Prior to the Delhi release, EdgeX Foundry and device services based on the Java SDK supported a feature called "dynamic device discovery". This feature was originally created in order to support wireless device protocols such as Bluetooth Low Energy, where a given device might not always be available. In such a scenario, the idea was to provide EdgeX with a set of filtering rules which could be used to dynamically add a "discovered" (i.e. available) device to the system if the rules produced a match.

This mechanism relied on Core Metadata object called a ProvisionWatcher which was used to define filtering rules.  A ProvisionWatcher has the following fields:
  • name
  • device-service - name of the owning service
  • device-profile - profile assigned to new matching devices
  • identifiers - map of attributes used to identify a device (eg. name, port, MAC, uuid, …)
  • operating-state - initial operating state for new devices
Note - identifiers can be fully-qualified or could use the wildcard ('*') for pattern matching. This allows a single ProvisionWatcher to be used to match multiple devices.

Device services (via code provided by the Java SDK) provided an endpoint called /discovery. Calling this endpoint resulted in the device service calling an internal Scan method in the device service's ProtocolDriver. The result of the Scan method was a list of available devices. For each device in this list, the ProtocolDriver provided a map of attributes that can be used to identify devices of a given protocol.  For instance for BLE devices this map might contain a MAC address and a service UUID.

The device SDK code would iterate through the scan results looking for new devices. If a new device was detected, then the SDK would attempt to match the device against any existing ProvisionWatcher objects associated with the device service.  When matches were found, the SDK would add the new device using the device profile and operating state specified in the ProvisionWatcher.

As long promised, here's my suggestion on how we handle blacklisting devices. As pointed out above, a ProvisionWatcher has a field called "identifiers" which is a map of strings which represent device attributes which can be used to identify a device. The key in this case identifies the attribute to match against (e.g. 'Name', 'MAC', 'uuid', ...). When a device service ProtocolDriver returns a list of devices in response to a Scan call, a corresponding map of device identifiers is returned. The SDK then looks for matches between the devices in the scan list and it's current ProvisionWatchers. The values in the ProvisionWatcher identifiers map support a basic glob-style [1] wildcard character "*".

So if a ProvisionWatcher defined identifiers as [["Name", "acme*"], "MAC", "38:Fc:E7*"], then any devices found in a subsequent scan results list with 'Name' that matches "acme*" and MAC address with the prefix "38:Fc:E7" would be added (if they don't already exist).

In order to exclude specific devices when a pattern has many potential matches, the idea has been tossed about to add a blacklist. Although you could argue that adjusting identifiers to exclude devices is one way to go, adding an explicit blacklist mechanism also has merit.

I think the simplest approach would be to add a new field called "blacklist" which is map of string arrays/lists which define values which if found in the attributes of a scanned device, prevent it from being added.  With the example above, if the device "acme13" was found to be compromised, all you'd need to do is add "acme13" to the "Names" list, and the original identifiers would still match against all other devices with the name prefix "acme". Likewise you could also specify a different identifier (if supported by the device service) in the blacklist (i.e. a specific MAC address).

To keep it's simple, let's just call the new field blacklist and make it a map of string slices:

blacklist    map[string][]

Two questions:

1) Should the blacklist support the "*" wildcard char?

2) Should we implement glob style pattern matching (eg. '?' or [...])?

Regards,
/tony

[1] https://en.wikipedia.org/wiki/Glob_(programming)


The final piece of the puzzle is Support Scheduler, which via Schedule and ScheduleEvent objects could be used to periodically call a device service's /discovery endpoint.

= Problems =

There are a few problems with this approach to dynamic device discovery.

1. Device deletion - if a device is deleted from the device service (and Core Metadata), on subsequent scans if the device was found to be available again, it would be re-added.

This problem is where the idea of device blacklists came from. In addition to deleting a device, you'd also add it to a blacklist object (in the ProvisionWatcher?). I'd argue that this is just adding to already complicated approach. Instead of black-listing the device, why not modify (or delete) the matching ProvisionWatcher before deleting the device?

2. Scan are synchronous - the /discovery endpoint didn't support asynchronous scan results.

3. ProvisionWatchers contain device profiles. This is the same problem we currently have with embedded device profiles in devices.  This makes ProvisionWatchers fairly heavy-weight objects, and adds complexity if/when a device profile is updated.

4. Scanning in a fixed period is less than ideal, as scanning can impact performance and also may impacts battery life for certain devices. Ideally the logic for determing when to scan should be the responsibilty of the device service, not pre-configured using Support Scheduler.

Example Java configuration for BLE ProvisionWatchers:

default.watcher.name=BLE-Watcher,BLE-Watcher-2,BLE-Watcher-3
default.watcher.service=device-bluetooth,device-bluetooth,device-bluetooth
default.watcher.profile=SensorTag,XDK,BLELight
default.watcher.name_identifiers=.*CC2650.*,XDK.*,.*beLight.*

Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?


Trevor.Conn@...
 

Some thoughts after reading this and going back over the discussion thread


1.) Are we sure the Provision Watcher isn't something that should live solely in the device service? Metadata will still contain devices/profiles/services that are active and part of the landscape, but the DS is the arbiter of onboarding.

2.) Putting the ProvisionWatcher in the DS eliminates reliance on the Scheduler b/c you can manage the rediscovery of devices internally, the same way we do with AutoEvents.

3.) The pushback to point 1 above might be that giving Metadata the responsibility for ProvisionWatchers could prevent a moving device -- such as a wearable going from building to building -- from onboarding via any DS. This would require a split deployment of EdgeX where the DS is on a gateway and the core services are running in a central location. Alternatively, we could support the ability for DS to talk horizontally to each other in environments with multiple complete deployments of EdgeX. I would prefer a pub/sub (not ZeroMQ) for this instead of a REST call.

4.) Another objection could be -- if the DS controls the Provision Watcher and the DS goes down, how does it re-hydrate that information? There could either be a simple persistence layer to handle this or, in the case where DS can talk to each other, it could request the ProvisionWatcher criteria from another service instance.


Thinking out loud.


Trevor Conn
Technical Staff Engineer
Core Working Group Chair of EdgeX Foundry
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA


From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> on behalf of Iain Anderson <iain@...>
Sent: Tuesday, July 23, 2019 9:46 AM
To: EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-device-services] Overview of Pre-Delhi Dynamic Device Discovery design
 

[EXTERNAL EMAIL]

I've updated https://github.com/edgexfoundry/go-mod-core-contracts/issues/111 in light of the suggestions here. Please review and comment.

Regards,
Iain


On 22/07/2019 18:54, espy wrote:

On 6/3/19 6:53 PM, Tony Espy wrote:

As discussed during today's Device Service working group meeting, I volunteered to write a description of the original dynamic device discovery mechanism.

[Jim - feel free to comment and/or correct my description]

Regards,
/tony

---

Prior to the Delhi release, EdgeX Foundry and device services based on the Java SDK supported a feature called "dynamic device discovery". This feature was originally created in order to support wireless device protocols such as Bluetooth Low Energy, where a given device might not always be available. In such a scenario, the idea was to provide EdgeX with a set of filtering rules which could be used to dynamically add a "discovered" (i.e. available) device to the system if the rules produced a match.

This mechanism relied on Core Metadata object called a ProvisionWatcher which was used to define filtering rules.  A ProvisionWatcher has the following fields:
  • name
  • device-service - name of the owning service
  • device-profile - profile assigned to new matching devices
  • identifiers - map of attributes used to identify a device (eg. name, port, MAC, uuid, …)
  • operating-state - initial operating state for new devices
Note - identifiers can be fully-qualified or could use the wildcard ('*') for pattern matching. This allows a single ProvisionWatcher to be used to match multiple devices.

Device services (via code provided by the Java SDK) provided an endpoint called /discovery. Calling this endpoint resulted in the device service calling an internal Scan method in the device service's ProtocolDriver. The result of the Scan method was a list of available devices. For each device in this list, the ProtocolDriver provided a map of attributes that can be used to identify devices of a given protocol.  For instance for BLE devices this map might contain a MAC address and a service UUID.

The device SDK code would iterate through the scan results looking for new devices. If a new device was detected, then the SDK would attempt to match the device against any existing ProvisionWatcher objects associated with the device service.  When matches were found, the SDK would add the new device using the device profile and operating state specified in the ProvisionWatcher.

As long promised, here's my suggestion on how we handle blacklisting devices. As pointed out above, a ProvisionWatcher has a field called "identifiers" which is a map of strings which represent device attributes which can be used to identify a device. The key in this case identifies the attribute to match against (e.g. 'Name', 'MAC', 'uuid', ...). When a device service ProtocolDriver returns a list of devices in response to a Scan call, a corresponding map of device identifiers is returned. The SDK then looks for matches between the devices in the scan list and it's current ProvisionWatchers. The values in the ProvisionWatcher identifiers map support a basic glob-style [1] wildcard character "*".

So if a ProvisionWatcher defined identifiers as [["Name", "acme*"], "MAC", "38:Fc:E7*"], then any devices found in a subsequent scan results list with 'Name' that matches "acme*" and MAC address with the prefix "38:Fc:E7" would be added (if they don't already exist).

In order to exclude specific devices when a pattern has many potential matches, the idea has been tossed about to add a blacklist. Although you could argue that adjusting identifiers to exclude devices is one way to go, adding an explicit blacklist mechanism also has merit.

I think the simplest approach would be to add a new field called "blacklist" which is map of string arrays/lists which define values which if found in the attributes of a scanned device, prevent it from being added.  With the example above, if the device "acme13" was found to be compromised, all you'd need to do is add "acme13" to the "Names" list, and the original identifiers would still match against all other devices with the name prefix "acme". Likewise you could also specify a different identifier (if supported by the device service) in the blacklist (i.e. a specific MAC address).

To keep it's simple, let's just call the new field blacklist and make it a map of string slices:

blacklist    map[string][]

Two questions:

1) Should the blacklist support the "*" wildcard char?

2) Should we implement glob style pattern matching (eg. '?' or [...])?

Regards,
/tony

[1] https://en.wikipedia.org/wiki/Glob_(programming)


The final piece of the puzzle is Support Scheduler, which via Schedule and ScheduleEvent objects could be used to periodically call a device service's /discovery endpoint.

= Problems =

There are a few problems with this approach to dynamic device discovery.

1. Device deletion - if a device is deleted from the device service (and Core Metadata), on subsequent scans if the device was found to be available again, it would be re-added.

This problem is where the idea of device blacklists came from. In addition to deleting a device, you'd also add it to a blacklist object (in the ProvisionWatcher?). I'd argue that this is just adding to already complicated approach. Instead of black-listing the device, why not modify (or delete) the matching ProvisionWatcher before deleting the device?

2. Scan are synchronous - the /discovery endpoint didn't support asynchronous scan results.

3. ProvisionWatchers contain device profiles. This is the same problem we currently have with embedded device profiles in devices.  This makes ProvisionWatchers fairly heavy-weight objects, and adds complexity if/when a device profile is updated.

4. Scanning in a fixed period is less than ideal, as scanning can impact performance and also may impacts battery life for certain devices. Ideally the logic for determing when to scan should be the responsibilty of the device service, not pre-configured using Support Scheduler.

Example Java configuration for BLE ProvisionWatchers:

default.watcher.name=BLE-Watcher,BLE-Watcher-2,BLE-Watcher-3
default.watcher.service=device-bluetooth,device-bluetooth,device-bluetooth
default.watcher.profile=SensorTag,XDK,BLELight
default.watcher.name_identifiers=.*CC2650.*,XDK.*,.*beLight.*

Food for thought…

Instead of the ProvisionWatcher approach, should devices be explicitly added individually and a new attribute added which indicates "presence"?


Iain Anderson
 



On 23/07/2019 16:34, Trevor.Conn@... wrote:

Some thoughts after reading this and going back over the discussion thread


1.) Are we sure the Provision Watcher isn't something that should live solely in the device service? Metadata will still contain devices/profiles/services that are active and part of the landscape, but the DS is the arbiter of onboarding.

2.) Putting the ProvisionWatcher in the DS eliminates reliance on the Scheduler b/c you can manage the rediscovery of devices internally, the same way we do with AutoEvents.



The intent as I understand it is that core-metadata handles the persistence of PWs, provides the interface for managing your PW collection and handles consistency issues (a DeviceProfile may not be deleted if a PW specifies it). The DS is where the matching logic will live. So the DS will hit the provisionwatcher/servicename endpoint on core-metadata at startup, and listen for updates on its callback handler - in the same way as it does for Devices. Then when discovery occurs the DS will use its local cache of PWs to determine whether to create new devices.

As far as the Scheduler goes, the intention is to perform discovery in various circumstances:
1) When the DS /discovery endpoint is hit (by support-scheduler or otherwise)
2) On a configurable schedule internal to the DS
3) On programmatic request from the DS implementation

Regards,
Iain