Automatic Requirements Management for Continuous Delivery. Part 2

 

 

..in the last post we talked about how to start with the capture and management of requirements as graphical diagrams using a specification called BPMN2. By using a regular tool like Eclipse IDE we could easily design a couple of simple use cases and reflect the UI flow on new BPMN2 diagrams.

Ready for a new chapter?

What now?

Well, we've got a plan as you could figured it out!

In our plan we have 4 areas although only two of them are really our focus now. Please note the plan has a specific flow or order in the actions.

Overview of the capture of requirements system.

We're interested for now in only a part of the flow, from steps 1 to 5:

1. The business analysts manage the information from UX sessions and feedback to compose the business logic in a graphical way using BPMN2 diagrams.
2. With every change the BPMN2 diagrams are packaged and uploaded to a repository as a dependency, keeping versions and traceability.
3. The BPMN Automation component reads this package and processes the diagram files assigning IDs that will be used as references by tests.
4. It creates and/or updates the items in JIRA. It uses a NoSQL database to persist other useful information and preserve the history of changes in requirements.

 

In the post with the first part  we described how the BA (Business Analyst) made diagrams to reflect the lessons learnt from the analysis. Easy. The BPMN project used Maven and Java to create a JAR file containing all the diagram files. There are some points to highlight. It's important:

1. Each diagram file reflects a specific unitary feature with a set of multiple scenarios with functional and/or not functional tests.
2. The diagrams are arranged in a folder structure. The first level will be JIRA projects while the second level will be the components in that projects. The set of features will be translated to JIRA issues in each component and project.

Reminder: We'll use a folder structure related to JIRA items. "FUSION" will be the JIRA project name and "TELLER" the name of the JIRA component in that project. The features into each BPMN2 file have a name that will be the name of the JIRA issue grouped in that component.

The JAR file is uploaded to an artifact repository (Nexus in our case) with a specific version. These artifacts will be used as dependencies by the BPMN Automation component to read the BPMN2 diagrams inside.

We talked about this in the last post (part 1). This is the area where the BPMN2 diagrams are composed and uploaded to the repository every time they change. This iterative evolution of the diagrams actually implements the agile iteration of business requirements with no manual action needed.

The Automation Area

The next step is to build an automatic way to process iterative changes in the business requirements. We have to:

• Get the last version of the files with the BPMN2 diagrams.
• Process the information in the diagrams and create new JIRA items if needed or update the already existing ones.
• Assign IDs to new test scenarios in order to match the tests that will meet these tests.
• Persist the processing information to get a full traceability and history of actions.
• Notify the involved actors (development engineers, managers, etc).

The Automation are involves several components you can see in below diagram. The Delivery Platform manages the Continuous Delivery pipelines and it is the star of the part 3.

The automation stuff happens here. The BPMN Automation processing is the grey box at bottom left, the white box is the JIRA server and the green one is our Delivery Platform containing the Continuous Delivery pipelines.

We'll have a look onto the start of this part, the BPMN Automation component...

Description

The BPMN Automation component is an ETL application that uses Spark for parallel processing and handling a huge number of files and data. It is written in Scala (with SBT) using Spark, Akka and the Atlassian Java REST client. The parsing of the BPMN2 diagram files is done with the Gherkin library that translates the test scenarios into each feature to the standard object model.

The Atlassian REST client saves a lot of problems and possible issues with regular HTTP clients. It is asynchronous and it is probably the best and fastest option to integrate JIRA into your code. Here you have several common examples for common usages.

Step 1: Get the diagrams

The diagrams are retrieved as a dependency. The cardinality is, one project per file. The format of files is the infamous JAR. We keep a version range for compatibility in order to handle the features in a a specific project/file (SBT or Maven will download the latest version for that range). The dependency is declared in this way:

libraryDependencies +=  "com.fts.cp" % "BPMN_RFX" % "1.+

Voila!, once the dependency is downloaded we can examine the diagram files inside.

We wanted to encapsulate the reading (directly from the dependencies) of the diagram files in a simple function like:

scenario("gets diagram BPMN files properly from directories in the BPMN dependency as XML objects") {
  val files = Files.getListOfDiagramFilesFromDirectory("FUSION/Teller")
  assert(files.length > 0)
}

We have to meet this test. As you'll be noticed the dependency with the diagrams should be open and the files should be read. Well, this is not trivial. The goal was to get an XML object that we could use and manipulate. The natural way in Scala is to convert automatically to a scala.xml.Elem object. Please have a look on the code below and if someone has a better and elegant solution I'll be delighted to change the code!

https://gist.github.com/jesusjavierdediego/b2a44ed9ce89c3349fc804bc714b1d55

This function is used to read and convert to objects all the files for each component in the project (remember that a JIRA component is just a group of issues/features). So, we iterate all the components for this project and we get the object representation from the BPMN2 files.

https://gist.github.com/jesusjavierdediego/27a38f0b358386dfc388ecc9f3348328

Step 2: Process the information within and use a history

Once we can handle the information into the BPMN2 files we process each file. First we have to know if it is a new feature, it updates an existing one with new information or it is an existing feature but without changes.
This is the look of a feature into the database. We use CosmosDB to directly store JSON representations of features:

The structure of a feature with test scenarios in the database.

There are two ways of processing data from the database. On one hand we want an acceptable latency to operate with a large set of data as for instance all the features for a given project. On the other hand, we have to be able to operate with single records handled individually. We use Spark SQL for the first case and Morphia for the second one. This strategy provides us with  Big Data features as well as the required flexibility for CRUD operations.

In this snippet the function getRequirementFeaturesFromDBByProject retrieves a large set of data using Spark while the function getRequirementFeatureFromDBByID retrieves a single record:

https://gist.github.com/jesusjavierdediego/f26845d244269ba854808e1b2d72d61e

Step 3: Update the features in JIRA

We mentioned the BPMN Automation component would manage the JIRA items to provide traceability and visibility of the status of quality and progress of the features in a given project.

Reminder: The results of the after-deployment tests (functional, stress, load, security) will be mapped to specific tests scenarios in JIRA that contain the narration of the requirement in the arcane language of  Gherkin the wizard.

The cardinality of items is important. On one side we have defined a BPMN2 diagram for a specific feature (including functional and not functional requirements as well as a context based on the description of a JIRA component in the project).

A given feature will be designed in a BPMN2 diagram that will result in a JIRA issue

The JIRA component concept is important as it is useful to define specific areas in the project. The component concept is used in the BPMN2 project structure as we saw above and it will be used as well in the JIRA automation process.

Before going forward we have to remember the structure in JIRA that will be followed during the automation process:

We'll use a test JIRA server just to see what are the results. Please note we have created a project called with the same name as the main folder in the diagrams structure (TESTPRO).

Our little JIRA server for test purposes

We have used the Atlassian REST client for Java. We have found problems with certain transitive dependencies (sal-api). The final entry in the build.sbt was (after some collision):

libraryDependencies += "com.atlassian.jira" % "jira-rest-java-client-core" % "3.0.0"
dependencyOverrides += "com.google.guava" % "guava" % "19.0

Just a caveat. Keep an eye on the version of Guava. A not compatible version (above 19.0 in this case) can raise a ClassNotFoundException

And we use the asynchronous REST client. Below you can find the factory to get a single instance of the client as well as the function to create a JIRA issue.

https://gist.github.com/jesusjavierdediego/c10d401bdd8970ffdbeb6a05c5e9c560

Then, you can create your new JIRA issues with the test scenarios into the description written in Gherkin syntax as they were written in the BPMN2 diagram. Please note the tag on each test scenario with unique test ID. This tag will be used later to match the test results.

Resulting JIRA Issue from the BPMN2 diagram

Well, we can do it better. A very interesting option is to use sub-tasks in the JIRA issue, specifically one sub-task as a bug per each test scenario in the BPMN2 diagram. In this case we build a sub-task for each test scenario. We'll assign the sub-task key to the after-deployment test. The result of each test is handled to update the status of the sub-task according to the results of the test. With this approach a simple look on the list of sub-tasks in a JIRA issue will provide the information about the quality status for that feature.

The created subtasks in the JIRA Issue from the scenarios defined in the BPMN2 diagram.

Step 4: Notify

Once all the JIRA issues (with a specific feature) and sub--tasks (with a test scenario each) have been created or updated all the members of the squads are notified with the changes. Assigned engineers will review the new description in the test scenarios modifying the existing tests or creating new ones. The notifications channels are email and Slack with Teams in a next phase.

Testing

The way of testing the BPMN Automation component involves conventional unit, integration and functional tests.

We use a test dependency with a simple test diagram to be used during the Test phase.

libraryDependencies += "com.fts.cp" % "BPMN_TEST" % "1.+" % Test

To perform the functional tests we'll need a running JIRA test server and a running CosmosDB database with the collection.

It is important to remark that the items in the database and JIRA test server have to be deleted after the tests to allow repeatability of results.

 

What's Next?

Well, the next part is exciting. It's about linking the results of the after-deployment tests managed by the Delivery Platform to the JIRA issues and sub-tasks to update the status according to the result (Fail/Pass). These are the steps 9 and 10 in the first diagram in the post. So, the Continuous Delivery pipelines will tell to JIRA what happened with the tests every time they run, keeping the JIRA status of issues and sub-tasks up to date according to the SDLC. We'll see all this stuff in the next and last part of this series.

Coming soon!