Using SoapUI to Develop an Automated Functional Conformance Test for a REST API

As part of a customer contract, over the past 6 weeks QualityLogic’s web services testing team utilized SoapUI to develop an automated functional conformance test for a REST API that provides account management services to a variety of related websites. The 100 resource paths in the API all used http POST requests with an XML fragment contained in the body of the http message conveying the desired actions and/or response. One particular challenge with this project was that we were sharing the test environment with other groups and could make no assumptions about the availability static values in the back-end database.

Our SoapUI Review

• It is an amazingly high quality open source test tool. The use of groovy scripts, xPath, and dynamic property expansion created a very robust development environment. Although not being Java programmers, it was initially a bit confusing to sort out where Groovy stopped and Java began. A good book, Groovy in Action, solved that problem.
• We found test case development a bit more tedious than expected with a test cases consisting of a dozen steps and within each step perhaps a dozen assertions. It was just a lot of detail, with lots of mouse clicking to drill into things. We wrote some helper routines to consolidate a lot of the assertion checks which helped to some extent.
• We wound up buying the Pro version of SoapUI as we needed some technical support and this turned out to be a great decision. The support was excellent, we were more productive using the Pro version, and the code remained compatible with the open source version of SoapUI as long as we stayed away from certain Pro features.
• The SoapUI documentation was quite well written, but initially it was a challenge to sort out which things applied specifically to REST based testing.

REST API Project Details

1. We created a SoapUI project, a rest service within the project, defined each of the 100 resource paths within the service, and created an empty test suite.
2. We decided to use test suite properties as a set of global variables. Some properties were used to store dynamic data that was stored and retrieved by various test steps, while other properties contained static content that represented various possible values for customer records. Referencing these properties in a variety of ways throughout the test suite was simple. A few examples…
   • Simple Property Expansion:
     ${#TestSuite#password}
   • Dynamic Property Expansion where multiple comma separated values are stored in one property:
     ${=context.testCase.testSuite.properties["accountNumber"].value.split(",")[1]}
3. We made extensive use of random numbers for generating values that had to be unique. Yes, there was a remote possibility that we might get a test case failure as a result of a random number collision, but from a practical perspective, this was a non-issue.
4. We structured our test suite to run sequentially, with the following initialization routines:
   • Generate unique “random” values and store in properties
   • Use REST calls to populate the data base with sufficient framework data to register a customer. Example, define the various valid address types such as home, business, other, etc.
   • Use REST calls to register a set of customers using characteristics pulled from test suite properties. These customers will not be modified during test execution and will be used exclusively for “get” and “search” customer related testing.
   • Define a series of test cases that register customers for subsequent record updating. These can be called from other test cases as needed. Note that we encountered problems exporting and importing test cases that used the Goto Test step (they reference a GUID not the name of the test case), so we chose to call the test cases from a groovy script…
     def testCase = testRunner.testCase.testSuite.testCases["Another TestCase"]def properties = new com.eviware.soapui.support.types.StringToObjectMap()
def async = false
testCase.run(  properties, async )
5. Once the basic plumbing was in place we needed to grind out hundreds of test cases that when completed represented thousands of REST API calls and many thousands of test assertions.
6. Perhaps a note on assertions is warranted. There are a multitude of ways to validate that you get the expected values in a response using SoapUI. We utilized the following methods…
   • Contains assertion – On the surface this is a simple text comparison, but we also used regular expressions to accept multiple values such as (?s).*ALREADY.*|.*SUCCESS.* as well as property expansions.
   • Script Assertion – This was our most common assertion tool, which allowed to use the getXmlHolder object provided by SoapUI to easily query and assert conditions within a groovy script, such as..
     def groovyUtils = new com.eviware.soapui.support.GroovyUtils( context )def holder = groovyUtils.getXmlHolder(messageExchange.responseContent)
def xPath = "//statusLabel[../SystemLabel='" + \
context.testCase.testSuite.properties["SystemLabel"].value.split(",")[1] \
+ "' and ../statusLabel='SUCCESS']"
assert holder[xPath] == "SUCCESS"
   • xPath Match – This was nice tool in that I could use an xPath statement to grab a parent node, then compare expected results using both wild cards and property expansions child elements. The only gotcha that I found was that if I wasn’t very careful with my xPath statement and multiple parent blocks were returned, xPath Match would compare the first instance, which wasn’t always what I wanted. But with simpler scenarios, this was definitively a very quick and dirty way to check a bunch of response values.

Creating Helper Routines to Simplify the Assertion Testing Process

As most of our assertion testing involved comparing large sets of known values stored in test suite properties to API responses, we wrote some rather elaborate helper routines that simplified the assertion process. We did run into one obstacle in that we couldn’t find a clean way to “include” these routines in the assertions. The Pro version has a common library capability, but we didn’t want to lock ourselves into that. The open source version does allow a Jar file to be included in the soapUI\bin\ext folder, which would have addressed the issue, but we wanted the test suite to run on a generic install of SoapUI.

So, the utility routines were pasted into assertions throughout the suite (yuck). Just as the test suite was being finalized we discovered that there was a bug in the helper routines (not zero stuffing MD5 hash values) and were faced with editing 147 instances of the helper routines in the test suite (nightmare!). Fortunately, we were able to pull the entire SoapUI project into XMLSpy and globally edit the routines. This notion of editing the project from outside SoapUI might open up some interesting productivity opportunities.

We are now moving on to using SoapUI to load test this implementation and I will share our experiences with this effort in a week or so.

Contact Us