API Contract Testing, Tools and Impressions. Part One: REST

on

 

This post is the first on of a series of four about the making of API contracts based on standard API style. It is obvious that if you've collected all the requirements you are going to write down these mandates into a standard format contract. The standard formats are what we call API Specification (the most popular example is OpenAPI (OAS) but we can consider others like Protocol Buffer, GraphQL and AsyncAPI, for instance). The purpose of the API contract is exactly what a usual contract is, the list of specifications that describe the behavior of a specific API.

It is widely accepted in the IT community APIs are the best way to connect systems in a flexible and easy way. The different design of API standards rely on the features of protocols, versions of protocols and certain characteristics belonging to the architectural design (based on Remote Procedure Calls, Resources, queries, etc). It is critically important to know about the features of the different versions of protocols (e.g. HTTP 1.1, 2, 3, WS, etc) and the possibilities that they allow to the construction of modern software systems.

The current post has a very simple target: To test some approaches and tools to Contract-Driven RESTful API, how to test this contract in different phases and once it's done,  to get some conclusions.

 

Contract Driven API Design?

  1. As an API descriptor file, following a standard specification (OpenAPI Specification, Protobuf, etc).
  2. As a description  of behavior based on a set of Test Scenarios (e.g. using Gerkhin syntax).
  1. By following the BDD methodology and having a set of test scenarios written in Gerkhin syntax, implement the tests against the actual implementation of the API. As a real use case, you can imagine a testing app that acts as the client of your API and running some tests against it, or a JMeter test sending a set of calls against your API and recording the results.
  2. On the other side, we can consider the BDD test scenarios are not needed as the technical API specification description is enough, it is the actual API Contract.We are going to test some tools and different options in this regard.
 So, we'll start with the REST API Contract based on OAS.

  1. to test some tools and see what happens
  2. to find out what  scenario they fit better and their strong and weak points regarding OAS-based API Contract Testing.

Other not specialized and well known tools can be used for this purpose. JMeter and Gatling can be effective when testing APIs in a Continuous Delivery pipeline.

Anyway, I want to have a very practical hands-on approach in this series of posts. So, the idea is to actually use the tools and include here the outcome of the usage of the tools. This can make reading a bit more difficult and boring but on the other side you can see directly how the result looks like.

 

Step 1: Compose the API Contract, OAS file and Test Scenarios

The test scenarios will describe the behavior of our API.

Include a test scenario per endpoint/method for the service we are going to test according to the contract that has been provided.

Ideally we should write the API contract terms in Gerkhin syntax. Once this is done, perhaps is needed to make some adaptations to the specific tool. We'll see this right now.


Step 2: Get an API Server

The second step is to get an implemented working REST API according to the contract. I used  Node and Express for this. It's simple!

var app = require('express')();

app.get('/test', function(req, res) {
  res.json({id: "1", name: "John"});
})

app.post('/test', function(req, res) {
  res.json({message: 'Added!'});
})

app.put('/test', function(req, res) {
  res.json({message: 'Updated!'});
})

app.delete('/test', function(req, res) {
  res.json({message: 'Deleted!'});
})

app.listen(3000, () => {
 console.log("Server running on port 3000");
});

 

Now, we'll start with the tests....


import com.intuit.karate.junit4.Karate;
import cucumber.api.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Karate.class)
@CucumberOptions(features = "classpath:karate",
        monochrome = true,
        strict = true
)
public class APITest {
}
Feature: Testing a REST API as test scenarios with Karate

Scenario: Testing valid GET endpoint
  Given url 'http://localhost:3000/test'
  When method GET
  Then status 200
  And match $ == {id:"#notnull",name:"#notnull"}

Scenario: Testing an invalid GET endpoint - 404
  Given url 'http://localhost:3000/tes'
  When method GET
  Then status 404

Scenario: Testing valid POST endpoint
  Given url 'http://localhost:3000/test'
  And request { id: "1" , name: "John"}
  When method POST
  Then status 200
  And match $ == {message:"Added!"}

Scenario: Testing valid PUT endpoint
  Given url 'http://localhost:3000/test'
  And request { id: '1234' , name: 'John Smith'}
  When method PUT
  Then status 200
  And match $ == {message:"Updated!"}

Scenario: Testing valid DELETE endpoint
  Given url 'http://localhost:3000/test'
  When method DELETE
  Then status 200
  And match $ == {message:"Deleted!"}

  • Protocol Introspection:  No
  • Documentation: OK 
  • Easiness of implementation: OK
  • Effectiveness: OK
  • BDD integration: OK (via Gerkhin) 
  • API Descriptor Integration: No
  • Tool specific features: Acceptable (Gerkhin/Karate syntax)
  • Integration in regular build time tests: OK
  • Integration as separate/runtime tests: OK  
@Functional
Feature: Testing a REST API as test scenarios with REST Assured

  Scenario: Testing valid GET endpoint
    Given I access to the base URL
    When using the method GET
    Then I get a response status code 200
    And the field ID is not empty and the field NAME is not empty


import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(
        features = "src/test/resources/restassured",
        plugin = {"pretty"},
        glue = {"steps"},
        monochrome = true,
        strict = true
)
public class TestRunner {
} 


  • Protocol Introspection:  OK
  • Documentation: OK
  • Easiness of implementation: Acceptable
  • Effectiveness: OK
  • BDD integration: OK
  • Tool specific features: OK (Gerkhin syntax is generic)
  • API descriptor Integration: No
  • Integration in regular build time tests: OK
  • Integration as separate/runtime tests: OK (managed as a external tool)
  • Protocol Introspection:  No
  • Documentation: OK 
  • Easiness of implementation: OK (no implementation at all!)
  • Effectiveness: OK
  • BDD integration: No
  • Tool specific features: OK (the descriptor files have to follow the standards)
  • API descriptor Integration: OK
  • Integration in regular build time tests: No
  • Integration as separate/runtime tests: OK (managed as an external tool)
  1. Karate is probably the best option if you're taking a BDD approach that describes the behaviour of the API. It as an excellent integration and it is fast.
  2. However, Dredd is great as it is directly integrated with the API specification. As it runs with Node it is trivial to integrate Dredd with the Delivery pipeline.
  3. For more customized scenarios or in the case you have to use predefined features files to your API (and they do not match with Karate) REST-assured is a good option
  4. If you need complete information about the HTTP protocol aspects request/response, headers, payloads,  use REST-assured.

As you can see they are all excellent tools that can be used in different scenarios.However, it depends on what you consider the API contract actually is. A really useful and practical case is, the OAS file is actually the API contract. Following the approach, Dredd is an excellent option.

For my understanding, the API Contract is composed by different parts:

  • The Standard technical specification (OAS, Proto file, etc)
  • The Enterprise API description (Basically all the non-functional features that bring your APIs to be ready for the world at scale. We'll see a post about it!)
  • The semantics about the use cases
  • The parameters for discovery and consumption

See ya in the next episode!




Comments

Post a Comment