August 5, 2022

How to Think Like a Programmer Part Three

Denis Akinmolasire

In Part One of this series we aimed to help developers by highlighting potential challenges that new graduates and folks new to the industry often face and some differences between the university atmosphere and working in an industrial software engineering context.

In Part Two we introduced the concept of domain-driven development and outlined a process of breaking problem statements down into constituent parts to form a domain model and accrue the benefits this approach offers.

Now in Part Three, we'll tackle production issues by recreating a scenario in a test environment, get to the root cause, and explain how component testing can help. We'll explain how we use technologies like H2 (a relational database manager), Mongo (an open source database), Spring (a Java UI Framework). Finally, we'll give some place you can go to learn more.

"I have a difficult production issue or a bug  what do I do?"

As a software engineer, one of the tasks that you are very likely to take on is production support. This typically involves responding to queries that end users may have about your application. In addition, you are also very likely to encounter a production issue/bug that you need to deal with. Sometimes the issue you're dealing with will be a known issue and you may have an existing workaround to address the fundamental issue. On other occasions, you will be required to establish the root cause as quickly as possible. 

Establishing the root cause can be achieved using a variety of ways, such as:

  • Inspecting the logs to spot the exception(s) / edge cases in the processing flow
  • Querying a database to check what is being persisted
  • Recreating the scenario in a test environment 

Recreating the scenario in a test environment will be the focus for the rest of this post.

Scenario Recreation in a Test Environment

Scenario recreation is the process of taking the same inputs from one environment and replaying the inputs and/or invoking the same steps in another environment to recreate the same scenario. The good thing about this approach is because the recreation is done in a test environment, you can debug to a level that you can't necessarily do in a production environment. Scenario recreation can be used to not just solve production issues, but you can also use it to help get through User Acceptance Tests (UATs). This approach I actually used very recently to solve a UAT issue that we were facing. 

Detail Behind Application and UAT Issue

In one of our platforms, we have a component called DataSourceManager. It is an orchestration component that manages the execution of our "Session State Model", that is modeled as finite state machine. The states are used in our application to inform our end users how far in the transaction lifecycle the processing of a session has gone, so it is important that this is done correctly. For instance, the "failed" state is used to inform our end users that there is a processing issue and we couldn't complete the transaction successfully.

During our UAT, we noticed that the state for sessions that failed were not persisting correctly. Instead of them having a final status of "failed", they were stuck on the post process state. In addition, the end time wasn't being populated.

For context, DataSourceManager runs as a Spring web service. For the UAT issue concerned, there were four parts/concepts that were key to understanding the root cause:

  • DataSourceSessionCaller Executes the state model for DataSourceSessions
  • DataSourceSessionCallerExecutor Manages the caller by executing retry logic for certain states based on configuration
  • DataSourceSessionDataStore Executes queries and transactions against our Mongo cluster
  • DataSourceSession The data entity that gets processed

What made the problem even more confusing was in the logs we spotted logging for the session that indicated that it had gone from post process into a failed state. I needed a way to drill into what our issue was. To solve this challenge, I wrote a spring component test to recreate the problem and address our UAT issue.

Component testing will help you learn to to think like a programmer because it helps you recognize and write encapsulated code that is much easier to test.

Diagram of the Data Source Session Manager with a caller and Executer feeding into the data store, all interacting with MongoDB
Diagram of the Data Source Session Manager with a caller and Executer feeding into the data store, all interacting with MongoDB

Spring Component Testing Usage

Because our application runs as a Spring web service, we were able to use this to our advantage. In Spring, you have the capability to run Spring component tests. Spring component tests are unit tests, but they give you the ability to run various pieces of your application locally in memory or stub out (mock) certain pieces of your application if necessary – as opposed to testing a class/component in isolation. In my use case, I used the the component test to run the DataSourceManager locally and instantiate a local H2 instance of Mongo.

In effect, I was able to run a local version of DataSourceManager as if I was running in a real production/test environment. Running locally also meant I could step through the code in my integrated developer environment (IDE) and spot where the issue is. Below were the key parts for my component test:

    WebTestClient – A spring class that allows you to invoke rest requests. The external interface for DataSourceManager is REST; via the WebClientTest, I was able to invoke REST requests at my instance of DataSourceManager running locally.

    Autowired WebApplication context – This is a spring annotation that enables usage of dependency injection for initializing components. This feature also enables you to choose when you wish to instantiate real or mocked versions of your classes.

    Local Mongo Cluster Initialization – DataSourceManager uses Mongo as its database. Instead of connecting to the real infrastructure, this feature allowed me to leverage configuration to run a local in memory Mongo instance to enable my component test to run in a unit test capacity.

Diagram of the interaction between the DataSourceManager and the DataSourceSessionManager component test as explained in the preceding text.
Diagram of the interaction between the DataSourceManager and the DataSourceSessionManager component test as explained in the preceding text.

I created a test that sent in a DataSourceSession scenario that should result in the failed state scenario; the failed state is used to indicate when there are vendor feed or processing errors, so it is important that this state is rendered correctly. Using the component test, I eventually was able to get to the root cause of my particular problem. The issue was a result of a change we had made to one of the State classes. We had changed our Inprogress State class to write status updates to Mongo immediately as opposed to initially reading them off an in memory queue to improve our resilience. As a result of this change, prior to writing updates, we always query Mongo to get the latest version of our session to enable us to correctly perform optimistic locks; this is where our problem was.

The querying for the latest version of our DataSourceSession meant that the DataSourceSessionCallerExecutor had an outdated reference to the underlying DataSourceSession and as a result was missing information to enable it to correctly persist the final failed status by ensuring that the status field for the session in Mongo has a value of failed. This has since been corrected.

Additional material/further reading on testing What is software testing?  Types of software testing  Unit Testing - What, how and why?  Test environments  Test Driven Development Kata
Additional material/further reading on testing What is software testing?  Types of software testing  Unit Testing - What, how and why?  Test environments  Test Driven Development Kata

Due to the Spring component test running the components in memory, this enables the test to be run as part of Software Development Lifecycle (SDLC) build. So in future, if there is a change that is made that breaks this functionality, we have  an in memory end-to-end test that will allow us to spot the issue before the change makes its way to a UAT and/or production environment.

Key Advantages of Component Testing Usage

Even though our application has very good unit tests, the scope of unit tests tends to be very class and/or method level specific. Component testing helps to address this gap as they are compact enough to run as part of a regular build but enable enough of the inner workings of an application to run together to test them in a manner that can be executed in an SDLC build. This extra safeguarding is something that I found useful and I expect has prevented a lot of other potential production and UAT issues from arising in the future.


If you wish to leverage component testing in your application, always bear in mind the following points to ensure you get the maximum benefit from it:

  • Separation of concerns – Make sure key components are isolated to maximize test coverage and make it easier to spot issues.
  • SDLC – Make sure your test is able to run as part of your SDLC build; having a component test should not result in a massive increase in build time for instance.
  • Infrastructure isolation – If your application leverages certain infrastructure (i.e. a database), look to inject in memory equivalents into your component test so that you can concentrate on testing the behavior without having to spin up a dedicated environment to run your component test.

Part 4

Our next blog post in this series will be focused on why you need to take design into consideration when working in an agile environment. Stay tuned.

Additional material/further reading on testing

See for important risk disclosures, conflicts of interest, and other terms and conditions relating to this blog and your reliance on information contained in it.

Certain solutions and Institutional Services described herein are provided via our Marquee platform. The Marquee platform is for institutional and professional clients only. This site is for informational purposes only and does not constitute an offer to provide the Marquee platform services described, nor an offer to sell, or the solicitation of an offer to buy, any security. Some of the services and products described herein may not be available in certain jurisdictions or to certain types of clients. Please contact your Goldman Sachs sales representative with any questions. Any data or market information presented on the site is solely for illustrative purposes. There is no representation that any transaction can or could have been effected on such terms or at such prices. Please see for additional information.
Transaction Banking services are offered by Goldman Sachs Bank USA (“GS Bank”). GS Bank is a New York State chartered bank, a member of the Federal Reserve System and a Member FDIC.
GS DAP™ is owned and operated by Goldman Sachs. This site is for informational purposes only and does not constitute an offer to provide, or the solicitation of an offer to provide access to or use of GS DAP™. Any subsequent commitment by Goldman Sachs to provide access to and / or use of GS DAP™ would be subject to various conditions, including, amongst others, (i) satisfactory determination and legal review of the structure of any potential product or activity, (ii) receipt of all internal and external approvals (including potentially regulatory approvals); (iii) execution of any relevant documentation in a form satisfactory to Goldman Sachs; and (iv) completion of any relevant system / technology / platform build or adaptation required or desired to support the structure of any potential product or activity.
Mosaic is a service mark of Goldman Sachs & Co. LLC. This service is made available in the United States by Goldman Sachs & Co. LLC and outside of the United States by Goldman Sachs International, or its local affiliates in accordance with applicable law and regulations. Goldman Sachs International and Goldman Sachs & Co. LLC are the distributors of the Goldman Sachs Funds. Depending upon the jurisdiction in which you are located, transactions in non-Goldman Sachs money market funds are affected by either Goldman Sachs & Co. LLC, a member of FINRA, SIPC and NYSE, or Goldman Sachs International. For additional information contact your Goldman Sachs representative. Goldman Sachs & Co. LLC, Goldman Sachs International, Goldman Sachs Liquidity Solutions, Goldman Sachs Asset Management, L.P., and the Goldman Sachs funds available through Goldman Sachs Liquidity Solutions and other affiliated entities, are under the common control of the Goldman Sachs Group, Inc.
© 2024 Goldman Sachs. All rights reserved.