Table of Contents
Introduction
What is EdgeX certification?
...
Appendix 1. Device Service Certification Notes
Introduction
This document describes a certification program for EdgeX Foundry. This document was developed by the Certification Working Group to
...
Note: The term “certification authority” is used in this document to describe the organization that will be conducting the certification testing/evaluation. It is yet to be determined if this will be done within the EdgeX community or by a third party.
What is EdgeX certification?
EdgeX certification represents a process where software components that interact with EdgeX are evaluated and confirmed to meet a set of technical requirements and standards that will help insure compatibility.
...
For the purposes of this document, we are focused on core services and those services that interact with core services (e.g., device services and application services).
Why is Certification needed?
EdgeX Foundry is based on loosely-coupled micro-services bound by common APIs. Entire subsections can be replaced, combined, etc., with proprietary, differentiated “EdgeX-compliant” offerings. The community and users want assurance that EdgeX will continue to function as designed even when components from other sources are added or replaced.
Certification is critical for multi-vendor ecosystems. By ensuring the services can be easily and reliably integrated to drive an outcome, users have reduced risk and greater confidence in adopting EdgeX and third party components.
Benefits to Stakeholders
There are three primary stakeholders for EdgeX certification. These include the EdgeX developer community, third party partners that use and build products around EdgeX, and the end users that are employing EdgeX to implement projects.
...
- Confirmed interoperability as of a given EdgeX release
- Faster implementation
- Reduced integration risks
- Higher reliability
Certification Levels
The EdgeX certification process will consist of two levels: self-assessment and formal certification.
Self-Assessment
The self-assessment process enables an organization to conduct their own evaluation using EdgeX provided tools to confirm meeting basic certification requirements. This assessment will be a prerequisite for submitting a component to EdgeX for formal certification. Self-assessment alone may be useful for organizations to confirm their code even if they have no interest in pursuing formal certification.
...
Self-assessment is not formally recognized by the EdgeX Foundry. I.e., it does not convey the use of “EdgeX certified” in marketing materials or listing in the EdgeX certified products list.
Formal Certification
Formal EdgeX certification testing will be conducted by a certification authority designated by the EdgeX TSC. Components that successfully complete the certification process will be announced on the EdgeX web site and those organizations will be able to promote their component using the certification logo.
...
Submitted code will not be made public or stored beyond the period needed for testing. All test results will be retained.
Technical Requirements
This section describes the technical requirements for self-assessment and certification.
Assumptions
This section states the assumptions and beliefs that were used during development of the technical requirements.
- All certification submissions and work will be conducted in the English language.
- The certification evaluation process does not require submitters to provide source code.
- Services submitted for certification are expected to be packaged in Docker/Docker Compose.
- EdgeX is agnostic with regard to platform hardware and OS, but that platform independence is not tested. Submitters are expected to state exceptions to platform independence.
- Certification is not dependent on the programming language or technology used in the service.
- EdgeX is orchestration agnostic. Services should be designed that way and should not make any assumption about orchestration/ QoS
- EdgeX has versions. The testing tools have versions. Certification will be to an EdgeX release. We assume the testing tools are appropriate for the EdgeX release.
Submission Requirements
Prior to submitting a service for certification, submitters should …
...
- Guidance on service throughput, performance or scale limitations
- Source code, makefiles, or other development artifacts
- Additional runtime artifacts (e.g., containers that run in alternate platform or environments)
- Alternate configurations
Core Service
Submitters, when providing a core service for certification, must also provide…
- Information on any operations, configuration, additional functionality, etc. that differ from the reference implementation (example – providing configuration defaults and options that differ from the reference implementation service).
- No documentation is required if the EdgeX reference implementation documentation applies to the service under review.
Device Service
Submitters, when providing a device service for certification, must also provide…
- The device (hardware, drivers, etc.) or device simulator required to test the device service.
- Indications if the device service was built with (and therefore compliant with) an EdgeX SDK. If not, the submitter must provide an indication of which features of the SDK are provided and not provided by the device service under consideration.
- Documentation explaining how devices are registered and provisioned with an example.
- Example applicable device profiles
- Guidance/expectations on the amounts and rates of data collection typically seen by the device service with regard to its typical devices.
Database Service
Submitters, when submitting a database using service for certification (Core Data, Metadata, Notifications, Logging, Export Client), must also provide…
- Database initialization code and how is it executed
- Persistence store details (if different from the reference implementation) including installation, cleanup, startup/shutdown procedures, etc.
- Guidance/expectation on data throughput, and scale
Client Application Service
Submitters, when submitting a user interface or any client user of EdgeX APIs must also provide…
- Information on the ability to handle single or multiple EdgeX instances
- Information on the ability to operate with or without the API Gateway running
Testing Requirements
The certification authority will …
...
- Explore and assess any code, configuration, or other artifacts for programming best practices, standardization, language idioms, etc.
- Explore and assess any code, configuration, or other artifacts for quality and depth of comments.
- Explore and assess any code, configuration, or other artifacts for compliance with copyright and license agreement indications.
Needed Tools
Execution of the self-assessment and certification process will require some tools and infrastructure to be assembled by various EdgeX working groups.
Product and Development
The following product and infrastructure items that will be needed:
Deliverable | Lead |
Test plan for each type of service | QA |
Black box tests for each type of service based on the test plan | QA + Dev |
Parameterize black box tests with the git location and access credentials so the certification authority can use/automate existing tests and point them to location that may not be under our control (submitter provided repository) | Dev |
Performance/longevity tests for 24 hour exercise of service | Dev |
Test of the certification tests.
| QA |
Process and instructions for executing tests
| QA + Certification |
Independent infrastructure that can be used by the certification testing team
| DevOps |
Repository only accessible to certification testing team to store raw test results and results report (for use in the event of later question) | DevOps |
Location for public to obtain self-assessment tests and documentation | DevOps |
Clean up process following completion of certification process
| DevOps + Certification |
Other notes:
- Consider using existing device services to confirm process and that they can be certified (test our own dog food)
- Assumption that some submitters will want to provide their code from a private repository. Need to investigate how to script pulling the code (private certs?) and running tests.
Process
The following items that will be needed to execute the certification process:
Deliverable | Responsible |
Application form to gather details about the requestor and code being submitted | Certification |
Legal agreement to confirm rules for code handling by certification authority and rights and responsibilities for use of Certification by requestor | Certification + LF |
Code submittal process
| Certification |
Record and store testing results/observations | Certification + DevOps |
Report of failed test with information on failure and process for remediation | Certification |
Create formal certification test results document | Certification |
Issue formal letter of certification or failure to submitter | Certification |
Periodic reports of certification learnings (anonymized) to the community | Certification |
Other notes:
- Testing process is private between submitter and EdgeX Foundry until it reaches conclusion.
Marketing and Promotion
In order to achieve the full benefits of the certification program it must be seen as having value and momentum. The following items will be helpful in that effort:
Deliverable | Responsible |
Web page(s) about certification
| Marketing |
Coordinate with submitter on when they want to make public announcement | Marketing |
Joint press release template that can be used to announce new certified component | Marketing |
Periodic follow up with users of certified components to confirm value and gather quotes that could be used for additional promotion | Marketing |
External Certification Authority
...
- Duties and requirements
- Selection criteria and evaluation process
- Cost and budget (handling payments)
- Service quality expectations
Certification Process
When an organization is ready to submit their service or product for certification, they will follow a standard process. The beginning of this process is an application.
...
- The Certification WG and TSC will be notified
- Once approved, the submitter will be notified
- Announcement of the certification will be coordinated with the submitter
- Press announcement
- Appearance on EdgeX website
Commercial Issues
This section covers various commercial issues that will be involved in the operation of the certification program.
Eligibility
Formal certification is available to any organization or developer that is willing and able to submit the required information and successfully pass through the testing. There is no requirement for a submitter to be a member of EdgeX Foundry or Linux Foundation. This is to encourage maximum participation in the certification program.
Fees
Certification is a process that will take some effort by both the submitter and the certification authority. Organizations that will most benefit from certification will be those that intend to make their components available in the market. These organizations should be expected to help cover the cost of conducting the certification program.
...
Q: Should there be a reduced fee for developers or organizations that are members of EdgeX Foundry?
Version Control
The certification process will be executed against the current release of EdgeX. The certification results and certificate will specify the version of EdgeX that was used for testing. There is no expiration of certification provided: 1) the certified component has not been modified, and 2) the component will be used with that release of EdgeX.
EdgeX Changes
EdgeX will continue to evolve with regular releases. It will be up to users to evaluate if the changes to associated APIs are significant enough to invalidate the certification for the EdgeX release they plan to use. It will be up to the developer of a component to decide if they want to certify their component on a new EdgeX release.
Component Changes
It is expected that certified components will change as developers fix bugs and add new functionality. We want to encourage a process of continuous improvement. However, it is easy to imagine scenarios where the changes could make a component no longer meet the certification goals and requirements.
...
A process for recertifying component changes will be needed. If the changes are for bug fixes, it could be that self-assessment and request for review is sufficient to recertify the component. For enhancements, it may be that a full certification process will be required. This will be determined as we get a better understanding of the certification process as well as the volume and extent of changes to certified components.
Revoking Certification
There may be situations where EdgeX Foundry determines it is necessary to revoke certification of a component. This could be due to
...
We do not expect this to happen often, but it is necessary to protect the integrity of the certification program.
Roadmap
As can be seen from the list of support tools required to support certification presented above, there is a lot of work required to get to the point of issuing meaningful certification of EdgeX components. We want to balance market value, time to market, and effort.
Setting Priorities
One of the key decisions is around which type of service should be the starting point. We have ranked them as follows:
Service Type | Value to Market | Level of Effort | Availability of Tests |
Core | Low | Low | High |
Device | High | High | Low |
Application | Medium | Medium | Low |
Level of effort = availability of tools to execute tests
...
It is our belief that Device Services offer the greatest value to the success of EdgeX. We are recommending that the EdgeX community dedicate itself to producing the necessary tools and infrastructure to accomplish certification of Device Services within two releases.
Phases
Given the complexity and effort to create a meaningful certification program, we are suggesting a phased approach in the crawl, walk, run model. Release targets for the phases will be defined in collaboration with the TSC and other WG.
Crawl
The goal of the Crawl phase is to be able to evaluate and declare an EdgeX Device Service component as being certified. The focus is on making sure the process and tests exist and are sufficient, even if highly manual for execution.
...
- A minimum of 5 EdgeX device services are certified
Walk
With a working process, the priority of the Walk phase will be to automate the process and make setting up tests and recording results more automated. A second goal will be to make a self-assessment test package available. We should gain sufficient knowledge during this phase to make a decision on the requirements for the certification authority role.
...
- Ability to execute certification tests of device services and gather results in a repeatable manner by someone who may not be an EdgeX expert or developer
- First release of self-assessment test package for Device Service
- Identify the requirements and selection process for a third party to act as certification authority
Run
With a working and automated process, the Run phase will focus on expanding tests to cover the remaining service types in addition to continuous improvement of the process.
...
- Certification of all EdgeX service types (Device, Application, Core)
- Establishment of a third party certification authority
Open Issues
This section contains questions that are still open and under discussion by the Certification WG.
...
Q: Any types of organizations that should have reduced or no fees?
Glossary
Validation: The assurance that a product, service, or system meets the needs of the customer and other identified stakeholders. Is the project designed to meet the desired objectives/success criteria?
...
- MUST is equivalent to REQUIRED and SHALL indicating that the definition is an absolute requirement.
- MUST NOT is equivalent to SHALL NOT and indicates that it is an absolute prohibition of the specs.
- SHOULD is equivalent to RECOMMENDED means that there are valid reasons to ignore a particular requirement, but the implications need to be weighed.
- SHOULD NOT and NOT RECOMMENDED means that a particular behavior may be acceptable or useful, but again, the implications need to be weighed.
- MAY means OPTIONAL and that the requirement is truly optional. Interoperability with different systems that may or may not implement an optional requirement must be done.
Appendix 1. Device Service Certification Notes
This appendix is provided to give some insight into the level of detail that may be required for conducting certification of each service type.
...