/
QA/Test Roadmap

QA/Test Roadmap

Owned by Andrew Foster
Last updated: Nov 03, 2019

'Barcelona': October 2017

'California': ~June 2018

'Delhi': ~October 2018

'Edinburgh': ~ April 2019

'Fuji': ~ October 2019

'Geneva': ~ April 2020


'Hanoi': ~ October 2020

Edinburgh Release (Committed)

  • With automated blackbox tests in place, the Edinburgh release will include better visualization/dashboard of test results (including look up and display of historical test results).  Candidate tools include Telegraf + Grafana + InfluxDB, DataDog, Prometheus. https://github.com/edgexfoundry/blackbox-testing/issues/105
  • The Edinburgh release will include the capture of resource metrics to monitor performance.  Metrics monitored will include memory and CPU consumption. https://github.com/edgexfoundry/edgex-go/issues/115
  • As a stretch goal, the Edinburgh release will include performance/load testing using a tool like Bender, Jmeter, Load Impact or other tool selected by the work group.
  • Security tests will be automated as mentioned above in the Security Group tasks. https://github.com/edgexfoundry/security-api-gateway/issues/44 & https://github.com/edgexfoundry/security-secret-store/issues/51
  • For the Edinburgh release, code contributors will now be required to supply additional or updated blackbox tests to cover changes made to the code base with any PR.  Working group leads and project committers are head accountable for this change in procedures.
  • As a stretch goal, the project will look to produce Swagger documentation to better support testing and API documentation in future releases.  RAML documentation will remain the officially API standard for Edinburgh, but additional Swagger documentation will be provided and weighed for use in future releases.

Backlog (April 2019)

Priorities shown in the table below are provisional and subject to review.

Item#DescriptionPriorityTarget Release
1Improved unit tests across services (improvements should be driven by specific WG dev teams)1Fuji
2Improve blackbox test structure including reorganization of the tests and better test case documentation (source ramya.ranganathan@intel.com)1Fuji
3New test framework (e.g. Robot or Cucumber) to support additional types of functional/blackbox and system integration tests e.g. Device Service or system level latency tests1Fuji
4Dockerize blackbox testing infrastructure for deployment simplification/flexibility2Geneva
5System integration tests – currently we don’t have any end to end tests e.g. Device Service read data -> Core Data -> Rules Engine or Export Service2Geneva
6Blackbox tests for new Application Services microservices1Fuji
7Blackbox test for Device Services  - Virtual, Modbus, MQTT, BACNet, OPC UA. Develop common set of tests that can be used against all Device Services.1Fuji
8Configuration testing – currently all blackbox tests run with single static configuration, we should test additional EdgeX configurations2Geneva
9Automated performance testing - API Load testing (measure response time) and metrics (CPU, memory) collection for all EdgeX microservices (this work was started during the Edinburgh iteration)1Fuji
10Automated performance testing - EdgeX microservice startup times (STRETCH GOAL FOR EDINBURGH) 1Fuji
11Automated performance testing - Automated system level latency and throughout testing (e.g. device read to export or device read to analytics to device actuation)1Fuji
12Automated performance testing - Baseline performance of service binaries no container3Hanoi
13Blackbox and performance test runs against other container technologies supported (e.g. snaps)3Hanoi
14Automated performance testing - The ability to create summary reports/dashboards of key EdgeX performance indicators with alerts if thresholds have been exceeded2Geneva
15Test coverage analysis e.g. using tools such as Codecov.io1Fuji
16Static code analysis e.g. using tools such as SonarQube or Coverity to identify badly written code, memory leaks and security vulnerabilities  2Geneva
17Tracing during testing e.g. based on Open Tracing standard such as Zipkin, Jaeger3Hanoi
18

Replacement of RAML API documentation with Swagger

2Geneva
19Blackbox tests moved into edgex-go repository.  Will allow backbox test to be more easily added/updated by developers as part of a PR resulting from an API change. 3Hanoi
20Run blackbox tests for an individual EdgeX microservice when a PR issued.3Hanoi


Fuji Release (Committed)

=

  • Improve blackbox test structure including reorganization of the tests and better test case documentation 
  • New test framework (e.g. Robot or Cucumber) to support additional types of functional/blackbox and system integration tests e.g. Device Service or system level latency tests
  • Blackbox tests for new Application Services microservices. Moved to Application Services WG
  • Blackbox test for Device Services  - Virtual, Modbus, MQTT, BACNet, OPC UA. Initially develop common set of tests that can be used against all Device Services. Moved to Device Services WG
  • Automated performance testing
    • API Load testing (measure response time) and metrics (CPU, memory) collection for all EdgeX microservices (this work was started during the Edinburgh iteration
  • Test coverage analysis e.g. using tools such as Codecov.io. Moved to Dev Ops WG
  • Replacement of RAML API documentation with Swagger


Geneva Planning

Backlog (November 2019)

Priorities shown in the table below are provisional and subject to review.

Item#DescriptionPriorityTarget Release
1Improved unit tests across services (improvements should be driven by specific WG dev teams)1Fuji
2Improve blackbox test structure including reorganization of the tests and better test case documentation (source ramya.ranganathan@intel.com)1Fuji
3New test framework (e.g. Robot or Cucumber) to support additional types of functional/blackbox and system integration tests e.g. Device Service or system level latency tests1Fuji
4Dockerize blackbox testing infrastructure for deployment simplification/flexibility2Geneva
5System integration tests – currently we don’t have any end to end tests e.g. Device Service read data -> Core Data -> Rules Engine or Export Service1Geneva
6Blackbox tests for new Application Services microservices1Geneva
7Blackbox test for Device Services  - Virtual, Modbus, MQTT, BACNet, OPC UA. Develop common set of tests that can be used against all Device Services.1Geneva
8Configuration testing – currently all blackbox tests run with single static configuration, we should test additional EdgeX configurations2Geneva
9Automated performance testing - API Load testing (measure response time) and metrics (CPU, memory) collection for all EdgeX microservices (this work was started during the Edinburgh iteration)2Geneva
10Automated performance testing - EdgeX microservice startup times (STRETCH GOAL FOR EDINBURGH) 1Fuji
11Automated performance testing - Automated system level latency and throughout testing (e.g. device read to export or device read to analytics to device actuation)1Fuji
12Automated performance testing - Baseline performance of service binaries no container3Hanoi
13Blackbox and performance test runs against other container technologies supported (e.g. snaps)3Hanoi
14Automated performance testing - The ability to create summary reports/dashboards of key EdgeX performance indicators with alerts if thresholds have been exceeded2Geneva
15Test coverage analysis e.g. using tools such as Codecov.io1Fuji
16Static code analysis e.g. using tools such as SonarQube or Coverity to identify badly written code, memory leaks and security vulnerabilities  2Geneva
17Tracing during testing e.g. based on Open Tracing standard such as Zipkin, Jaeger3Hanoi
18

Replacement of RAML API documentation with Swagger

1Geneva
19Blackbox tests moved into edgex-go repository.  Will allow backbox test to be more easily added/updated by developers as part of a PR resulting from an API change. 3Hanoi
20Run blackbox tests for an individual EdgeX microservice when a PR issued.3Hanoi
 21Edinburgh/Fuji backward compatibility testing 2 Geneva






Related content