Date   

Re: Writable Config Settings Audit

Ranganathan, Ramya
 

Great start Trevor!

These settings can go in as config files where the robot files reside ie (Test Scenarios/Use case).

This enables robot files to iterate through various config files that are under that use case .

For eg:

https://github.com/edgexfoundry-holding/edgex-taf/tree/master/TAF/testScenarios/UC_Color_Log_Demo

 

Thanks,

Ramya

 

 

From: EdgeX-TSC-QA-Test@... [mailto:EdgeX-TSC-QA-Test@...] On Behalf Of Trevor.Conn@...
Sent: Monday, September 9, 2019 12:18 PM
To: EdgeX-TSC-QA-Test@...
Subject: [Edgex-tsc-QA-Test] Writable Config Settings Audit

 

Hi all -- In last week's QA/Test working group meeting I offered to do an audit of all of the Writable settings in the Core/Support/Export services so that we could look at the possibility of toggling these settings and how they would affect the blackbox tests. Please see the attached spreadsheet. There is one tab defined for each service.

 

If a setting has "No" in the "Blackbox Relevant?" column, it may still be considered for the overall end-to-end integration testing. For example, we might toggle the LogLevel of a given service on the fly from "INFO" to "DEBUG" and then have a test verify whether or not we are receiving errors of the specified level in the log.

 

If additional information is required, let me know. Otherwise we can address follow up at this week's QA/Test call.

 

Trevor Conn
Technical Staff Engineer

Core Working Group Chair of EdgeX Foundry

Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA


Writable Config Settings Audit

Trevor.Conn@...
 

Hi all -- In last week's QA/Test working group meeting I offered to do an audit of all of the Writable settings in the Core/Support/Export services so that we could look at the possibility of toggling these settings and how they would affect the blackbox tests. Please see the attached spreadsheet. There is one tab defined for each service.


If a setting has "No" in the "Blackbox Relevant?" column, it may still be considered for the overall end-to-end integration testing. For example, we might toggle the LogLevel of a given service on the fly from "INFO" to "DEBUG" and then have a test verify whether or not we are receiving errors of the specified level in the log.


If additional information is required, let me know. Otherwise we can address follow up at this week's QA/Test call.


Trevor Conn
Technical Staff Engineer
Core Working Group Chair of EdgeX Foundry
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA


Upcoming Event: EdgeX QA/Test WG: Validation/Testing - Thu, 09/05/2019 4:00pm-5:00pm, Please RSVP #cal-reminder

EdgeX-TSC-QA-Test@lists.edgexfoundry.org Calendar <EdgeX-TSC-QA-Test@...>
 

Reminder: EdgeX QA/Test WG: Validation/Testing

When: Thursday, 5 September 2019, 4:00pm to 5:00pm, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/635232828

An RSVP is requested. Click here to RSVP

Organizer: EdgeX-TSC-QA-Test@...

Description: EdgeX QA/Test WG: Validation/Testing meeting. Meeting content posted to QA/Test Wiki.
Meeting Lead: Andy Foster, EdgeX QA/Test WG Chair, andy@...
-----Join Zoom Meeting
https://zoom.us/j/635232828

One tap mobile
+16699006833,,635232828# US (San Jose)
+16465588656,,635232828# US (New York)

Dial by your location
        +1 669 900 6833 US (San Jose)
        +1 646 558 8656 US (New York)
        +1 855 880 1246 US Toll-free
        +1 877 369 0926 US Toll-free
Meeting ID: 635 232 828
Find your local number: https://zoom.us/u/abzs1KyOkW


Re: Common Device Service Test Cases in Robot file

Ranganathan, Ramya
 

Cloud,

 

The code looks neat and readable. Good job here!

One additional input at this stage is that you can start putting these robot files under TAF->TestScenarios.

Once Python functions are identified for the keywords in robot files, these python code can be put in TAF->TestCaseApps folder or TAF->Utils folder depending on the function.

 

Hope it makes sense.

 

Thanks,

Ramya

 

 

From: EdgeX-TSC-QA-Test@... [mailto:EdgeX-TSC-QA-Test@...] On Behalf Of Dobkins, Jacob
Sent: Thursday, September 5, 2019 2:07 PM
To: Cloud Tsai <cloud@...>; edgex-tsc-qa-test@...; EdgeX-TSC-Certification@...; EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-QA-Test] Common Device Service Test Cases in Robot file

 

Hey Cloud,

I unfortunately don’t have a lot of experience with the EdgeX Device Services, so I can’t give any advice in regards to the execution-order/structure that you have made (which seems to be the primary focus of the code as-is). However, I do have a personal preference suggestion as far as formatting is concerned.

Robot has two syntax-styles for its white-space/delimiting of Keywords: Tab-delimited, or Pipe+Tab-delimited.

 

In your example it would look like:

*** Test Cases ***

Device_TC0001a - Device adminState is LOCKED

    Execute Command     echo “test”

 

In my preference it would look like:

*** Test Cases ***

| Device_TC0001a - Device adminState is LOCKED  |

|   | Execute Command   | echo "test"   |

 

Now the immediate downside is that the plugin for Intellisense/syntax-highlight does not recognize this format and thus looks uglier.
The upside? In many cases you will have string arguments that utilize white-space (possibly even tabs) which could trick Robot into thinking it’s two separate parameters and thus requires a lot of character-escaping to mitigate. By using the Pipe+Tab style you mostly avoid this and have a clearer idea of what each section of a line does in relation to the keyword.

 

That said my observation of the Robot community is that your syntax is preferred for .robot files, and my syntax is preferred for .python document-strings. Note, my examples are from SSHLibrary which is developed by the same people who developed Robotframework itself, so I would say their examples probably trump my preferences, but just food for thought.

 

Jacob Dobkins

Work Mobile:        +1 (480) 232-1289 

iNet Skype Phone: +1 (480) 552-7539

 

From: EdgeX-TSC-QA-Test@... <EdgeX-TSC-QA-Test@...> On Behalf Of Cloud Tsai
Sent: Wednesday, September 4, 2019 6:59 AM
To: edgex-tsc-qa-test@...; EdgeX-TSC-Certification@...; EdgeX-TSC-Device-Services@...
Subject: [Edgex-tsc-QA-Test] Common Device Service Test Cases in Robot file

 

Hi all,

 

We are writing the test cases according to the Device Service requirement document.

The attachment is the initial version.  It's just a beginning and not runable.

We are keeping refining them and make them could be executed in edgex-taf.

Please provide your feedback if you have time.

The suggested reading tool is Visual Studio Code with Robot Framework Intellisense extension, but you can still read in any text editor.

 

--

Best Regards,

Cloud Tsai


Re: Common Device Service Test Cases in Robot file

Dobkins, Jacob
 

Hey Cloud,

I unfortunately don’t have a lot of experience with the EdgeX Device Services, so I can’t give any advice in regards to the execution-order/structure that you have made (which seems to be the primary focus of the code as-is). However, I do have a personal preference suggestion as far as formatting is concerned.

Robot has two syntax-styles for its white-space/delimiting of Keywords: Tab-delimited, or Pipe+Tab-delimited.

 

In your example it would look like:

*** Test Cases ***

Device_TC0001a - Device adminState is LOCKED

    Execute Command     echo “test”

 

In my preference it would look like:

*** Test Cases ***

| Device_TC0001a - Device adminState is LOCKED  |

|   | Execute Command   | echo "test"   |

 

Now the immediate downside is that the plugin for Intellisense/syntax-highlight does not recognize this format and thus looks uglier.
The upside? In many cases you will have string arguments that utilize white-space (possibly even tabs) which could trick Robot into thinking it’s two separate parameters and thus requires a lot of character-escaping to mitigate. By using the Pipe+Tab style you mostly avoid this and have a clearer idea of what each section of a line does in relation to the keyword.

 

That said my observation of the Robot community is that your syntax is preferred for .robot files, and my syntax is preferred for .python document-strings. Note, my examples are from SSHLibrary which is developed by the same people who developed Robotframework itself, so I would say their examples probably trump my preferences, but just food for thought.

 

Jacob Dobkins

Work Mobile:        +1 (480) 232-1289 

iNet Skype Phone: +1 (480) 552-7539

 

From: EdgeX-TSC-QA-Test@... <EdgeX-TSC-QA-Test@...> On Behalf Of Cloud Tsai
Sent: Wednesday, September 4, 2019 6:59 AM
To: edgex-tsc-qa-test@...; EdgeX-TSC-Certification@...; EdgeX-TSC-Device-Services@...
Subject: [Edgex-tsc-QA-Test] Common Device Service Test Cases in Robot file

 

Hi all,

 

We are writing the test cases according to the Device Service requirement document.

The attachment is the initial version.  It's just a beginning and not runable.

We are keeping refining them and make them could be executed in edgex-taf.

Please provide your feedback if you have time.

The suggested reading tool is Visual Studio Code with Robot Framework Intellisense extension, but you can still read in any text editor.

 

--

Best Regards,

Cloud Tsai


Re: Common Device Service Test Cases in Robot file

Ranganathan, Ramya
 

Great start Cloud! We will review the attachments and get back!

Thanks,

Ramya

 

 

From: EdgeX-TSC-QA-Test@... [mailto:EdgeX-TSC-QA-Test@...] On Behalf Of Cloud Tsai
Sent: Wednesday, September 4, 2019 7:03 AM
To: Cloud Tsai <cloud@...>
Cc: edgex-tsc-qa-test@...; EdgeX-TSC-Certification@...; EdgeX-TSC-Device-Services@...
Subject: Re: [Edgex-tsc-QA-Test] Common Device Service Test Cases in Robot file

 

Please also see the test plan about these tests,

 

On Wed, 4 Sep 2019 at 21:59, Cloud Tsai via Lists.Edgexfoundry.Org <cloud=iotechsys.com@...> wrote:

Hi all,

 

We are writing the test cases according to the Device Service requirement document.

The attachment is the initial version.  It's just a beginning and not runable.

We are keeping refining them and make them could be executed in edgex-taf.

Please provide your feedback if you have time.

The suggested reading tool is Visual Studio Code with Robot Framework Intellisense extension, but you can still read in any text editor.

 

--

Best Regards,

Cloud Tsai


 

--

Best Regards,

Cloud Tsai


Re: Common Device Service Test Cases in Robot file

Cloud Tsai
 

Please also see the test plan about these tests,


On Wed, 4 Sep 2019 at 21:59, Cloud Tsai via Lists.Edgexfoundry.Org <cloud=iotechsys.com@...> wrote:
Hi all,

We are writing the test cases according to the Device Service requirement document.
The attachment is the initial version.  It's just a beginning and not runable.
We are keeping refining them and make them could be executed in edgex-taf.
Please provide your feedback if you have time.
The suggested reading tool is Visual Studio Code with Robot Framework Intellisense extension, but you can still read in any text editor.

--
Best Regards,
Cloud Tsai



--
Best Regards,
Cloud Tsai


Common Device Service Test Cases in Robot file

Cloud Tsai
 

Hi all,

We are writing the test cases according to the Device Service requirement document.
The attachment is the initial version.  It's just a beginning and not runable.
We are keeping refining them and make them could be executed in edgex-taf.
Please provide your feedback if you have time.
The suggested reading tool is Visual Studio Code with Robot Framework Intellisense extension, but you can still read in any text editor.

--
Best Regards,
Cloud Tsai


Upcoming Event: EdgeX QA/Test WG: Validation/Testing - Thu, 08/29/2019 4:00pm-5:00pm, Please RSVP #cal-reminder

EdgeX-TSC-QA-Test@lists.edgexfoundry.org Calendar <EdgeX-TSC-QA-Test@...>
 

Reminder: EdgeX QA/Test WG: Validation/Testing

When: Thursday, 29 August 2019, 4:00pm to 5:00pm, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/635232828

An RSVP is requested. Click here to RSVP

Organizer: EdgeX-TSC-QA-Test@...

Description: EdgeX QA/Test WG: Validation/Testing meeting. Meeting content posted to QA/Test Wiki.
Meeting Lead: Andy Foster, EdgeX QA/Test WG Chair, andy@...
-----Join Zoom Meeting
https://zoom.us/j/635232828

One tap mobile
+16699006833,,635232828# US (San Jose)
+16465588656,,635232828# US (New York)

Dial by your location
        +1 669 900 6833 US (San Jose)
        +1 646 558 8656 US (New York)
        +1 855 880 1246 US Toll-free
        +1 877 369 0926 US Toll-free
Meeting ID: 635 232 828
Find your local number: https://zoom.us/u/abzs1KyOkW


Core Services -- Minor Version Compatibility Testing

Trevor.Conn@...
 

Hi all -- Per the QA/Test Working Group last week I took an action item to start fleshing out a draft containing thoughts related to possible minor version backward compatibility testing. I have uploaded the document to the Core Working Group page on the wiki under "Supporting Documents" for the August 29 meeting.


Comments are welcome, either in reply to this thread or via the meeting on the 29th. Once those are all collated, the doc will be reviewed in the QA/Test group also on the 29th.


Trevor Conn
Technical Staff Engineer
Core Working Group Chair of EdgeX Foundry
Dell Technologies | IoT DellTech
Trevor_Conn@...
Round Rock, TX  USA


Upcoming Event: EdgeX QA/Test WG: Validation/Testing - Thu, 08/22/2019 4:00pm-5:00pm, Please RSVP #cal-reminder

EdgeX-TSC-QA-Test@lists.edgexfoundry.org Calendar <EdgeX-TSC-QA-Test@...>
 

Reminder: EdgeX QA/Test WG: Validation/Testing

When: Thursday, 22 August 2019, 4:00pm to 5:00pm, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/635232828

An RSVP is requested. Click here to RSVP

Organizer: EdgeX-TSC-QA-Test@...

Description: EdgeX QA/Test WG: Validation/Testing meeting. Meeting content posted to QA/Test Wiki.
Meeting Lead: Andy Foster, EdgeX QA/Test WG Chair, andy@...
-----Join Zoom Meeting
https://zoom.us/j/635232828

One tap mobile
+16699006833,,635232828# US (San Jose)
+16465588656,,635232828# US (New York)

Dial by your location
        +1 669 900 6833 US (San Jose)
        +1 646 558 8656 US (New York)
        +1 855 880 1246 US Toll-free
        +1 877 369 0926 US Toll-free
Meeting ID: 635 232 828
Find your local number: https://zoom.us/u/abzs1KyOkW


Upcoming Event: EdgeX QA/Test WG: Validation/Testing - Thu, 08/15/2019 4:00pm-5:00pm, Please RSVP #cal-reminder

EdgeX-TSC-QA-Test@lists.edgexfoundry.org Calendar <EdgeX-TSC-QA-Test@...>
 

Reminder: EdgeX QA/Test WG: Validation/Testing

When: Thursday, 15 August 2019, 4:00pm to 5:00pm, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/635232828

An RSVP is requested. Click here to RSVP

Organizer: EdgeX-TSC-QA-Test@...

Description: EdgeX QA/Test WG: Validation/Testing meeting. Meeting content posted to QA/Test Wiki.
Meeting Lead: Andy Foster, EdgeX QA/Test WG Chair, andy@...
-----Join Zoom Meeting
https://zoom.us/j/635232828

One tap mobile
+16699006833,,635232828# US (San Jose)
+16465588656,,635232828# US (New York)

Dial by your location
        +1 669 900 6833 US (San Jose)
        +1 646 558 8656 US (New York)
        +1 855 880 1246 US Toll-free
        +1 877 369 0926 US Toll-free
Meeting ID: 635 232 828
Find your local number: https://zoom.us/u/abzs1KyOkW


Re: EdgeX Foundry v1.0.1 Performance Summary

Cloud Tsai
 

Hi all,

Here is the updated version of the performance summary.
We include the resource usage table of Redis in this version.

On Wed, 14 Aug 2019 at 21:58, Cloud Tsai <cloud@...> wrote:
Hi all,

We collected the performance summary on Raspberry Pi and Dell Gateway 3000 as attached.  It includes the test with security services and Redis.
Thanks for the suggestion from Robin, I've added some remark on the end to end exported time table.  Hope it becomes clearer.
Here are some notes for the report:
  1. For the start up time, Mongo is shorter than Redis.  With security, there are more services, so the start up time is increased.
  2. For the end to end exported, Redis is quicker than Mongo.
  3. We haven't included the resource usage of Redis.
  4. Opened a bug for core-data with Redis  https://github.com/edgexfoundry/edgex-go/issues/1643
    That's why some of the end to end exported time is N/A.
  5. Kong image is unavailable for ARM64, so we don't collect the result with security on Raspberry Pi
We can discuss more on QA WG meeting.

--
Best Regards,
Cloud Tsai


--
Best Regards,
Cloud Tsai


EdgeX Foundry v1.0.1 Performance Summary

Cloud Tsai
 

Hi all,

We collected the performance summary on Raspberry Pi and Dell Gateway 3000 as attached.  It includes the test with security services and Redis.
Thanks for the suggestion from Robin, I've added some remark on the end to end exported time table.  Hope it becomes clearer.
Here are some notes for the report:
  1. For the start up time, Mongo is shorter than Redis.  With security, there are more services, so the start up time is increased.
  2. For the end to end exported, Redis is quicker than Mongo.
  3. We haven't included the resource usage of Redis.
  4. Opened a bug for core-data with Redis  https://github.com/edgexfoundry/edgex-go/issues/1643
    That's why some of the end to end exported time is N/A.
  5. Kong image is unavailable for ARM64, so we don't collect the result with security on Raspberry Pi
We can discuss more on QA WG meeting.

--
Best Regards,
Cloud Tsai


Upcoming Event: EdgeX QA/Test WG: Validation/Testing - Thu, 08/08/2019 4:00pm-5:00pm, Please RSVP #cal-reminder

EdgeX-TSC-QA-Test@lists.edgexfoundry.org Calendar <EdgeX-TSC-QA-Test@...>
 

Reminder: EdgeX QA/Test WG: Validation/Testing

When: Thursday, 8 August 2019, 4:00pm to 5:00pm, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/635232828

An RSVP is requested. Click here to RSVP

Organizer: EdgeX-TSC-QA-Test@...

Description: EdgeX QA/Test WG: Validation/Testing meeting. Meeting content posted to QA/Test Wiki.
Meeting Lead: Andy Foster, EdgeX QA/Test WG Chair, andy@...
-----Join Zoom Meeting
https://zoom.us/j/635232828

One tap mobile
+16699006833,,635232828# US (San Jose)
+16465588656,,635232828# US (New York)

Dial by your location
        +1 669 900 6833 US (San Jose)
        +1 646 558 8656 US (New York)
        +1 855 880 1246 US Toll-free
        +1 877 369 0926 US Toll-free
Meeting ID: 635 232 828
Find your local number: https://zoom.us/u/abzs1KyOkW


Re: [EdgeX-tsc-certification] [Edgex-tsc-QA-Test] [Edgex-tsc-device-services] Prototype- Blackbox test of Device-Virtual with Robot Framework

Rodney
 

While scanning through the log and report files found in Playground_RobotFramework I stumbled across this:

/*                                                                                                                                                                                                          
    Copyright 2008-2013                                                                                                                                                                                     
        Matthias Ehmann,                                                                                                                                                                                    
        Michael Gerhaeuser,                                                                                                                                                                                 
        Carsten Miller,                                                                                                                                                                                     
        Bianca Valentin,                                                                                                                                                                                    
        Alfred Wassermann,                                                                                                                                                                                  
        Peter Wilfahrt                                                                                                                                                                                      
    Dual licensed under the Apache License Version 2.0, or LGPL Version 3 licenses.                                                                                                                         
    You should have received a copy of the GNU Lesser General Public License                                                                                                                                
    along with JSXCompressor.  If not, see <http://www.gnu.org/licenses/>.                                                                                                                                  
    You should have received a copy of the Apache License along with JSXCompressor.                                                                                                                         
    If not, see <http://www.apache.org/licenses/>.                                                                                                                                                          
*/                                                                                                                                                                                                          

Were you aware of these?

~Rodney

On Aug 8, 2019, at 12:39, Cloud Tsai <cloud@...> wrote:

Thanks for the feedback, Lenny and Jim,

I thought we could put "how to run" aside for now, and focus on whether we could define the spec in Robot file.
I believe we still need to make some development in test script to achieve the complex scenario, not only resource.robot, but also some Python script.

On Fri, 9 Aug 2019 at 00:19, Goodell, Leonard <leonard.goodell@...> wrote:

Hi Cloud,

  Great start!! I think you should also point out that all developers running/creating tests (that’s all of us 😊 ) will be required to install the robot tool chain outlined in the prerequisites here: https://github.com/FelixTing/Playground_RobotFramework/blob/master/README.md

 

Thanks!

   Lenny

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of Cloud Tsai
Sent: Thursday, August 08, 2019 12:31 AM
To: edgex-tsc-qa-test@...; edgex-tsc-device-services@...; EdgeX-TSC-Certification@...
Subject: [Edgex-tsc-device-services] Prototype- Blackbox test of Device-Virtual with Robot Framework

 

Hi all,

 

We made a prototype of blackbox test of device-virtual using Robot.  Please help take a look whether it is appropriate.  I plan to present the attached slide in the next Certification working group meeting.  It hasn't been applied the taf structure, but it is runnable with the standard Robot framework.

https://github.com/FelixTing/Playground_RobotFramework  

 

The plan is to use robot file to define the spec (test cases) for all the services, and the test report is easy to understand by all kinds of users.

 

--

Best Regards,

Cloud Tsai





--
Best Regards,
Cloud Tsai


Re: [Edgex-tsc-device-services] Prototype- Blackbox test of Device-Virtual with Robot Framework

Cloud Tsai
 

Thanks for the feedback, Lenny and Jim,

I thought we could put "how to run" aside for now, and focus on whether we could define the spec in Robot file.
I believe we still need to make some development in test script to achieve the complex scenario, not only resource.robot, but also some Python script.

On Fri, 9 Aug 2019 at 00:19, Goodell, Leonard <leonard.goodell@...> wrote:

Hi Cloud,

  Great start!! I think you should also point out that all developers running/creating tests (that’s all of us 😊 ) will be required to install the robot tool chain outlined in the prerequisites here: https://github.com/FelixTing/Playground_RobotFramework/blob/master/README.md

 

Thanks!

   Lenny

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of Cloud Tsai
Sent: Thursday, August 08, 2019 12:31 AM
To: edgex-tsc-qa-test@...; edgex-tsc-device-services@...; EdgeX-TSC-Certification@...
Subject: [Edgex-tsc-device-services] Prototype- Blackbox test of Device-Virtual with Robot Framework

 

Hi all,

 

We made a prototype of blackbox test of device-virtual using Robot.  Please help take a look whether it is appropriate.  I plan to present the attached slide in the next Certification working group meeting.  It hasn't been applied the taf structure, but it is runnable with the standard Robot framework.

https://github.com/FelixTing/Playground_RobotFramework  

 

The plan is to use robot file to define the spec (test cases) for all the services, and the test report is easy to understand by all kinds of users.

 

--

Best Regards,

Cloud Tsai



--
Best Regards,
Cloud Tsai


Re: [Edgex-tsc-device-services] Prototype- Blackbox test of Device-Virtual with Robot Framework

Goodell, Leonard
 

Hi Cloud,

  Great start!! I think you should also point out that all developers running/creating tests (that’s all of us 😊 ) will be required to install the robot tool chain outlined in the prerequisites here: https://github.com/FelixTing/Playground_RobotFramework/blob/master/README.md

 

Thanks!

   Lenny

 

From: EdgeX-TSC-Device-Services@... <EdgeX-TSC-Device-Services@...> On Behalf Of Cloud Tsai
Sent: Thursday, August 08, 2019 12:31 AM
To: edgex-tsc-qa-test@...; edgex-tsc-device-services@...; EdgeX-TSC-Certification@...
Subject: [Edgex-tsc-device-services] Prototype- Blackbox test of Device-Virtual with Robot Framework

 

Hi all,

 

We made a prototype of blackbox test of device-virtual using Robot.  Please help take a look whether it is appropriate.  I plan to present the attached slide in the next Certification working group meeting.  It hasn't been applied the taf structure, but it is runnable with the standard Robot framework.

https://github.com/FelixTing/Playground_RobotFramework  

 

The plan is to use robot file to define the spec (test cases) for all the services, and the test report is easy to understand by all kinds of users.

 

--

Best Regards,

Cloud Tsai


Re: Prototype- Blackbox test of Device-Virtual with Robot Framework

Cloud Tsai
 

Hi Jim,  Sorry for the unclear description.  Here is the further explanation.

  • “It hasn't been applied the taf structure” – what does this mean?
    TAF stands for Test Automation Framework contributed by Intel team.  It is based on Robot and defines a good file and folder structure, util libraries, and naming convention.  Intel will help present in the QA WG meeting.
    https://github.com/edgexfoundry-holding/edgex-taf  
    https://github.com/edgexfoundry-holding/edgex-taf-common  
  • “The plan is to use robot file to define the spec (test cases) for all the services” – so we would get rid of the current blackbox tests long run??  Is this work that each work group needs to do or be involved in?  Would the goal be for all services by Geneva or which release is the target?
    We won't get rid of the current blackbox tests soon.
    It should be a gradually process, and follow the pace of certification work.  We need a place clearly define the spec.  Thus, it would start from Device Service.  Yes, we will need the help from each working group to define or review the spec in Robot files.
  • Any idea of how long it takes to run Robot tests verses postman tests?  Just a general idea of whether it takes more or less time to run tests?
    The duration of running the test does not depend on the test framework, but the complexity of test cases.  The tests of postman are all simple API test, so it would be quicker.  The tests in system integration level would take longer time, but in API level should be similar.
  • I assume the device test suite (which you show against the device virtual on slide 4) would be set up for each device service??  How or would the Robot tests be setup to test each new device service?  Do we need a separate test suite for each device service or would this be the same suite for all device services?  How would that work/look?
    This needs further discussion, and applying TAF might help achieve that.
  • While you say that people (users) writing the tests don’t really have to know how to program to write test cases (you summary slide), it would seem that they will need a developer to write the resource.robot as that is pretty detailed to the API and all the parameters, etc. and I think it would require the developer to write this – correct?
    That's the goal, but we are still figuring out how to make it easier.

On Thu, 8 Aug 2019 at 22:29, <James.White2@...> wrote:

Cloud and team – good work to start this.  Couple of questions/comments…

  • “It hasn't been applied the taf structure” – what does this mean?
  • “The plan is to use robot file to define the spec (test cases) for all the services” – so we would get rid of the current blackbox tests long run??  Is this work that each work group needs to do or be involved in?  Would the goal be for all services by Geneva or which release is the target?
  • Any idea of how long it takes to run Robot tests verses postman tests?  Just a general idea of whether it takes more or less time to run tests?
  • I assume the device test suite (which you show against the device virtual on slide 4) would be set up for each device service??  How or would the Robot tests be setup to test each new device service?  Do we need a separate test suite for each device service or would this be the same suite for all device services?  How would that work/look?
  • While you say that people (users) writing the tests don’t really have to know how to program to write test cases (you summary slide), it would seem that they will need a developer to write the resource.robot as that is pretty detailed to the API and all the parameters, etc. and I think it would require the developer to write this – correct?

 

jim

 

From: EdgeX-TSC-QA-Test@... <EdgeX-TSC-QA-Test@...> On Behalf Of Cloud Tsai
Sent: Thursday, August 8, 2019 2:31 AM
To: edgex-tsc-qa-test@...; edgex-tsc-device-services@...; EdgeX-TSC-Certification@...
Subject: [Edgex-tsc-QA-Test] Prototype- Blackbox test of Device-Virtual with Robot Framework

 

[EXTERNAL EMAIL]

Hi all,

 

We made a prototype of blackbox test of device-virtual using Robot.  Please help take a look whether it is appropriate.  I plan to present the attached slide in the next Certification working group meeting.  It hasn't been applied the taf structure, but it is runnable with the standard Robot framework.

https://github.com/FelixTing/Playground_RobotFramework  

 

The plan is to use robot file to define the spec (test cases) for all the services, and the test report is easy to understand by all kinds of users.

 

--

Best Regards,

Cloud Tsai



--
Best Regards,
Cloud Tsai


Re: Prototype- Blackbox test of Device-Virtual with Robot Framework

James.White2@...
 

Cloud and team – good work to start this.  Couple of questions/comments…

  • “It hasn't been applied the taf structure” – what does this mean?
  • “The plan is to use robot file to define the spec (test cases) for all the services” – so we would get rid of the current blackbox tests long run??  Is this work that each work group needs to do or be involved in?  Would the goal be for all services by Geneva or which release is the target?
  • Any idea of how long it takes to run Robot tests verses postman tests?  Just a general idea of whether it takes more or less time to run tests?
  • I assume the device test suite (which you show against the device virtual on slide 4) would be set up for each device service??  How or would the Robot tests be setup to test each new device service?  Do we need a separate test suite for each device service or would this be the same suite for all device services?  How would that work/look?
  • While you say that people (users) writing the tests don’t really have to know how to program to write test cases (you summary slide), it would seem that they will need a developer to write the resource.robot as that is pretty detailed to the API and all the parameters, etc. and I think it would require the developer to write this – correct?

 

jim

 

From: EdgeX-TSC-QA-Test@... <EdgeX-TSC-QA-Test@...> On Behalf Of Cloud Tsai
Sent: Thursday, August 8, 2019 2:31 AM
To: edgex-tsc-qa-test@...; edgex-tsc-device-services@...; EdgeX-TSC-Certification@...
Subject: [Edgex-tsc-QA-Test] Prototype- Blackbox test of Device-Virtual with Robot Framework

 

[EXTERNAL EMAIL]

Hi all,

 

We made a prototype of blackbox test of device-virtual using Robot.  Please help take a look whether it is appropriate.  I plan to present the attached slide in the next Certification working group meeting.  It hasn't been applied the taf structure, but it is runnable with the standard Robot framework.

https://github.com/FelixTing/Playground_RobotFramework  

 

The plan is to use robot file to define the spec (test cases) for all the services, and the test report is easy to understand by all kinds of users.

 

--

Best Regards,

Cloud Tsai