Interoperability Testing with the OpenADR Cloud VEN Tester
QualityLogic is in the midst of releasing an update to both the A and B profile OpenADR test harness. As with any release of an OpenADR implementation, a variety of conformance and interoperability testing is required.
After our developers completed code modifications, we ran a full conformance test. Each VEN test case in the test harness has a matching VTN test case, so playing the matching pairs of tests together is an effective way to confirm that all the test case payloads validate against the schema, follow the conformance rules, and have correct payload exchange patterns.
Once all the bugs were fixed from the conformance self-testing, we did an analysis of the code changes and came up with a subset list of test cases that were modified and needed to be tested with real VEN and VTN implementations. We initially focused our interop testing on the VEN test cases as they require less manual intervention. We used the EPRI VEN for testing the B profile push test cases, and the Universal Devices ISY 994 VEN for testing the A push and Pull test cases, as well as the B push test cases.
Our next step was to test the VTN test cases using the QualityLogic Cloud Test System for OpenADR VENs, a VTN in the cloud based on the open source release of the Electric Power Research Institute (EPRI) VTN. We were aware of a number of known bugs in the open source release, but felt that these could be worked around and that the Cloud VEN Tester could be effectively used to validate the test harnesses interoperability with a real VTN.
We ran 26 updated test cases against the Cloud VEN Tester, found five issues with test cases that were not found using our conformance self-test, and had just one show stopper where a test case could not be run due to a known bug in the VTN. The following narrative shares some of the lessons learned from this testing experience.
Helpful Hints for Using the OpenADR Cloud VEN Tester
First a few helpful hints for using the Cloud VEN Tester:
- Firewall Access: Testing push test cases required opening up firewall access to the VEN test cases. In my test environment I had to port forward from my cable modem to my router, then port forward from my router to my PC, then open up ports in my PC firewall to get everything working from a push perspective from the VTN in the cloud.
- Transport Address: When using the Cloud VEN Tester with an A push implementation, make sure that the transport address used to communicate with the VEN uses only the root portion of the address (Domain/IP and port). Do not put a trailing slash on the root address. If you put in the full address, the VTN just tacks a second copy of “/OpenADR2/Simple/EiEvent,” and things don’t work so well!
- Change Activation Delay: I noticed that after making many changes to the VTN through the Admin console it took up to 15 seconds for them to become active. This lead to numerous instances of changing a VTN setting, running the test case and having it still fail, trying to sort out why, and then magically having it pass the next time I ran the test case after the setting change had taken effect. This is most likely an artifact of separate processes running in the VTN Application code.
- Target VEN: While an event is not required to have any targeting information, with the EPRI VTN, the only way the VTN knows which VEN to dispatch the event to is if you set the VEN as a target for the event when creating it on the Cloud VEN Tester. From the event tabs “Target, VEN,” you select the VEN name from drop down list, which in turn causes the venID to be included in the eiTarget object. If you fail to set a VEN target, the event will not be dispatched to the VEN.
- Setting Units: When creating events with signals other than simple, you will need to set Units. The settings for signalName and signalType are pretty obvious; however you will need to drill down into the signal itself after clicking “Create Event” to set the units drop down menu. I made the mistake more than once of not setting the units when sending out an event payload, resulting in the VEN complaining.
- Reusing Events: One trick that was very helpful was to reuse a previously completed or cancelled event by editing the dtStart time, then republishing it.
- Update Event Duration: The VTN does not automatically sync the overall event time with sum of the interval durations. If you modify an event interval to extend duration, make sure to also update the overall event duration.
- Destroy Unneeded Events: Cancelled and completed events seemed to get sent along with new events when publishing them from the VTN in the cloud. While this probably wouldn’t cause problems with a real VEN, it caused some test cases to fail. We worked around this by destroying unneeded events using the VTN admin interface before running each event-related test case.
Known Issues in the OpenADR Cloud Ven Tester
We did run into a number of bugs in the Cloud VEN Tester implementation. Some were previously known while others were discovered as a result of changes to the test harness. Here is what we found:
- venName: At one point we attempted to register with the VTN and used the wrong venName. As many of you know, the EPRI VTN uses the VEN name as a unique identifier for the VEN. The VTN returned an error code in the oadrRegesteredParty payload to indicate a wrong ID (the venName), but also included just the string “PTS ” in the oadrRequestedOadrPollFreq, causing a schema validation error that added some confusion to sorting out what was going on.
- Device Class Target: Specifying a device class target in a signal causes a schema validation error. We had to skip running one test case to as there was no clear work around.
- Multiple Events: One of the new clarifications in the upcoming B profile specification update is that, if an oadrDistributeEvent payload has multiple events, the createdDateTime in each of the events must match and should reflect when the payload is created. Event createdDateTime in the VTN in the Cloud appears to reflect when the event was created, so multiple events in a payload will have different dates-time values. In the test harness, this triggered a conformance rule violation.
Find the Hidden Bugs
All-in-all, the Cloud VEN Tester was very helpful in identifying bugs we would not have otherwise found in the test harness update. For instance, some test cases were hard-coding the event modificationNumber as zero in an oadrCreateOpt payload. I made a change to an event, causing the modification number to increment, and then when I ran the test case it failed because the Cloud VEN Tester recognized that the modification number sent by the VEN did not reflect the most recently incremented value.
As we begin implementing the Cloud VEN Tester Roadmap over the coming months, we are confident that many of the minor annoyances and bugs noted above will be cleaned up, leading to a much more effective interoperability testing tool for VEN implementers.