In Part One of this series we aimed to help developers by highlighting potential challenges that new graduates and folks new to the industry often face and some differences between the university atmosphere and working in an industrial software engineering context.
In Part Two we introduced the concept of domain-driven development and outlined a process of breaking problem statements down into constituent parts to form a domain model and accrue the benefits this approach offers.
Now in Part Three, we'll tackle production issues by recreating a scenario in a test environment, get to the root cause, and explain how component testing can help. We'll explain how we use technologies like H2 (a relational database manager), Mongo (an open source database), Spring (a Java UI Framework). Finally, we'll give some place you can go to learn more.
"I have a difficult production issue or a bug – what do I do?"
As a software engineer, one of the tasks that you are very likely to take on is production support. This typically involves responding to queries that end users may have about your application. In addition, you are also very likely to encounter a production issue/bug that you need to deal with. Sometimes the issue you're dealing with will be a known issue and you may have an existing workaround to address the fundamental issue. On other occasions, you will be required to establish the root cause as quickly as possible.
Establishing the root cause can be achieved using a variety of ways, such as:
- Inspecting the logs to spot the exception(s) / edge cases in the processing flow
- Querying a database to check what is being persisted
- Recreating the scenario in a test environment
Recreating the scenario in a test environment will be the focus for the rest of this post.
Scenario Recreation in a Test Environment
Scenario recreation is the process of taking the same inputs from one environment and replaying the inputs and/or invoking the same steps in another environment to recreate the same scenario. The good thing about this approach is because the recreation is done in a test environment, you can debug to a level that you can't necessarily do in a production environment. Scenario recreation can be used to not just solve production issues, but you can also use it to help get through User Acceptance Tests (UATs). This approach I actually used very recently to solve a UAT issue that we were facing.
Detail Behind Application and UAT Issue
In one of our platforms, we have a component called DataSourceManager. It is an orchestration component that manages the execution of our "Session State Model", that is modeled as finite state machine. The states are used in our application to inform our end users how far in the transaction lifecycle the processing of a session has gone, so it is important that this is done correctly. For instance, the "failed" state is used to inform our end users that there is a processing issue and we couldn't complete the transaction successfully.
During our UAT, we noticed that the state for sessions that failed were not persisting correctly. Instead of them having a final status of "failed", they were stuck on the post process state. In addition, the end time wasn't being populated.
For context, DataSourceManager runs as a Spring web service. For the UAT issue concerned, there were four parts/concepts that were key to understanding the root cause:
- DataSourceSessionCaller – Executes the state model for DataSourceSessions
- DataSourceSessionCallerExecutor – Manages the caller by executing retry logic for certain states based on configuration
- DataSourceSessionDataStore – Executes queries and transactions against our Mongo cluster
- DataSourceSession – The data entity that gets processed
What made the problem even more confusing was in the logs we spotted logging for the session that indicated that it had gone from post process into a failed state. I needed a way to drill into what our issue was. To solve this challenge, I wrote a spring component test to recreate the problem and address our UAT issue.
Component testing will help you learn to to think like a programmer because it helps you recognize and write encapsulated code that is much easier to test.
Spring Component Testing Usage
Because our application runs as a Spring web service, we were able to use this to our advantage. In Spring, you have the capability to run Spring component tests. Spring component tests are unit tests, but they give you the ability to run various pieces of your application locally in memory or stub out (mock) certain pieces of your application if necessary – as opposed to testing a class/component in isolation. In my use case, I used the the component test to run the DataSourceManager locally and instantiate a local H2 instance of Mongo.
In effect, I was able to run a local version of DataSourceManager as if I was running in a real production/test environment. Running locally also meant I could step through the code in my integrated developer environment (IDE) and spot where the issue is. Below were the key parts for my component test:
WebTestClient – A spring class that allows you to invoke rest requests. The external interface for DataSourceManager is REST; via the WebClientTest, I was able to invoke REST requests at my instance of DataSourceManager running locally.
Autowired WebApplication context – This is a spring annotation that enables usage of dependency injection for initializing components. This feature also enables you to choose when you wish to instantiate real or mocked versions of your classes.
Local Mongo Cluster Initialization – DataSourceManager uses Mongo as its database. Instead of connecting to the real infrastructure, this feature allowed me to leverage configuration to run a local in memory Mongo instance to enable my component test to run in a unit test capacity.
I created a test that sent in a DataSourceSession scenario that should result in the failed state scenario; the failed state is used to indicate when there are vendor feed or processing errors, so it is important that this state is rendered correctly. Using the component test, I eventually was able to get to the root cause of my particular problem. The issue was a result of a change we had made to one of the State classes. We had changed our Inprogress State class to write status updates to Mongo immediately as opposed to initially reading them off an in memory queue to improve our resilience. As a result of this change, prior to writing updates, we always query Mongo to get the latest version of our session to enable us to correctly perform optimistic locks; this is where our problem was.
The querying for the latest version of our DataSourceSession meant that the DataSourceSessionCallerExecutor had an outdated reference to the underlying DataSourceSession and as a result was missing information to enable it to correctly persist the final failed status by ensuring that the status field for the session in Mongo has a value of failed. This has since been corrected.
Due to the Spring component test running the components in memory, this enables the test to be run as part of Software Development Lifecycle (SDLC) build. So in future, if there is a change that is made that breaks this functionality, we have an in memory end-to-end test that will allow us to spot the issue before the change makes its way to a UAT and/or production environment.
Key Advantages of Component Testing Usage
Even though our application has very good unit tests, the scope of unit tests tends to be very class and/or method level specific. Component testing helps to address this gap as they are compact enough to run as part of a regular build but enable enough of the inner workings of an application to run together to test them in a manner that can be executed in an SDLC build. This extra safeguarding is something that I found useful and I expect has prevented a lot of other potential production and UAT issues from arising in the future.
Summary
If you wish to leverage component testing in your application, always bear in mind the following points to ensure you get the maximum benefit from it:
- Separation of concerns – Make sure key components are isolated to maximize test coverage and make it easier to spot issues.
- SDLC – Make sure your test is able to run as part of your SDLC build; having a component test should not result in a massive increase in build time for instance.
- Infrastructure isolation – If your application leverages certain infrastructure (i.e. a database), look to inject in memory equivalents into your component test so that you can concentrate on testing the behavior without having to spin up a dedicated environment to run your component test.
Part 4
Our next blog post in this series will be focused on why you need to take design into consideration when working in an agile environment. Stay tuned.
Additional material/further reading on testing
See https://www.gs.com/disclaimer/global_email for important risk disclosures, conflicts of interest, and other terms and conditions relating to this blog and your reliance on information contained in it.