Introduction
At Goldman Sachs, payments processing sits at the heart of our Transaction Banking (TxB) business. This post will focus on one of TxB’s services that enables instant payments for our corporate clients. First, we will give a high-level overview and present technical challenges associated with instant payments. We will then provide an explanation of the technical architecture, focusing on how TxB utilizes various products and services offered by Amazon Web Services (AWS) to achieve a highly scalable architecture with millisecond latency per transaction and 24x7 availability.
What is an Instant Payment?
An instant payment (also called a real-time payment) is any payment that settles between both the sender and the recipient within a few seconds and enables the near instantaneous transfer of funds between two bank accounts. An example of an instant payment is when one company pays another company immediately upon delivery of a product. The receiving company may release the product only after confirmation that the payment has been received.
An advantage of an instant payment scheme over a batch-based payments scheme (e.g., ACH payments) is that some batch-based payment schemes may take days to settle funds. Instant payments require greater throughput, lower latency, and higher availability of computing systems vs. these other payment schemes and are typically available 24x7x365 and can run across multiple platforms (e.g., mobile devices or traditional computing devices such as computers or laptops).
According to a study from the Federal Reserve, 80% of businesses in the United States are using a method of faster payments. As a result, there is significant demand for advanced technical solutions such as reliable payment processing infrastructure and scalable cloud-based platforms capable of handling high payments volumes. Instant payments create opportunities for innovative services and applications, pushing businesses to adopt advanced technologies to meet corporate demands for fast, secure, and seamless payment transactions.
High-Level Instant Payments Flow
There are five main actors involved in a typical instant payment processing scheme:
- The Sender of the Payment
- The Sender’s Bank
- The Payment Exchange Network
- The Recipient’s Bank
- The Receiver of the Payment
The high-level diagram below illustrates how an instant payment flows between sender and receiver.
In the illustration above, both the sender and the receiver happen to be corporate entities; however, nothing in the above flow precludes either actor from being individual consumers, government entities or other banks.
Most notably, the sending and receiving banks are where TxB plays the pivotal role of facilitating any instant payment transaction. TxB may act as both a sender and receiver of instant payments. This role is critical for interfacing with the Payment Exchange Network. TxB must ensure latency is minimized, and high throughput and high availability are achieved when payments are settled between these actors.
Technical and Functional Challenges Associated with Instant Payments
One of the biggest challenges in instant payments processing is that funds are required to move (also called settlement) between sender and receiver through a Payment Exchange Network within seconds. Once the payment is settled, the payment is considered final and irrevocable.
As stated above, the three primary goals of any instant payment technology solution are as follows:
- High Throughput: Instant payment processing typically occurs at peak times. Examples of peak processing times include:
- Certain days during the month, year, or quarter: for example, corporates paying their contractors or employees.
- Certain times during the day: for example, corporates managing their liquidity needs by settling funds among their various bank accounts held at other banks intraday.
- Low Latency: due to the strict settlement requirements of instant payments, any individual payment must be settled in (typically) under 15 seconds. This includes debiting the funds from the sender's account and crediting the funds into the receiver's account. The settlement of funds includes the Sender Bank and the Recipient Bank performing due diligence on the payments to ensure fraud or other such activities are not taking place.
- High Availability: payments processing should be available 24x7x365 (including holidays and weekends)
Typical Instant Payments Use Case
Below is a hypothetical scenario between two fictitious corporations and two fictitious banks where each corporation maintains their respective bank account.
Imagine WidgetTechCo has an Account with SummitPeakBank. WidgetTechCo manufactures widgets for InnovateTechCo. InnovateTechCo has an account at VistaTrustBank. It is the end of the month and InnovateTechCo would like to pay an invoice that is due to WidgetTechCo. This invoice is $2,000. The diagram below illustrates how InnovateTechCo will pay WidgetTechCo the invoice using an instant payments scheme.
VistaTrustBank will send this payment on behalf of InnovateTechCo to the Payment Exchange Network. The Network will identify that WidgetTechCo, who keeps an account at SummitPeakBank, needs to receive a $2,000 payment; hence, the Network instructs SummitPeakBank to deposit $2,000 into WidgetTechCo's Account.
The moment the payment instruction is published to SummitPeakBank by the Network, the Network starts a timer for SummitPeakBank. Within a few seconds (e.g., 10 seconds), SummitPeakBank must perform the following functions (at a minimum):
- Consume the payment instruction from the Network.
- Perform certain security validations to ensure the payment has indeed originated from the Network and not from some other source (i.e., the Network should be the only trusted source of the payment instruction). An example security validation may include a digital signature verification on the payment details.
- Perform payment-level validations. These validations include both technical and functional validations, including that the payment message adheres to industry formatting standards (e.g., ISO20022), and the payment message includes the correct currency, or the payment does not exceed a certain amount
- Execute certain payment processing rules that may adhere to SummitPeakBank's compliance and other internal payments processing due diligence standards.
- Perform any fraud or sanctions checks
If SummitPeakBank cannot respond within the given time limit to the Network, the Network will cancel the payment instruction - this effectively "unwinds" the entire payment. The Network will then instruct VistaTrustBank that the $2,000 payment to WidgetTechCo has failed to be completed and therefore the cycle must repeat itself (if needed). There are two scenarios forward after that: 1) VistaTrustBank may retry the payment instruction (it will be a new payment instruction); or 2) VistaTrustBank and SummitPeakBank communicate offline (via email or phone, for example) to investigate why the payment transaction has failed.
What makes this more interesting is that the Network may be sending payments at an extremely high transaction rate (e.g., 300 payments per second or more) during peak times and at any hour of the day, highlighting the need for high throughput, low latency and high availability.
Technical Architecture
The diagram above illustrates TxB's cloud first architecture for processing instant payments. In the following example, SummitPeakBank wants to send TxB a payment transaction; TxB in turn will then ensure that the payment is deposited into one of TxB's client's account.
- The Payment Exchange Network publishes a payment instruction for TxB
- TxB's Payment Gateway will consume this payment transaction from the Network; the Gateway will then perform the required security validations on the payment (for example, did this payment transaction originate from a trusted source, namely the Network itself is the only trusted source)
- After successfully validating the payment transaction from a security perspective, the Gateway will execute functional validations on the payment - for example, account, currency and amount validations, among other functional validations.
- Once all validations have been successfully executed, a positive acknowledgement (known as an "ACK") is published by the Gateway to the Network
- At each step of the above process, the Gateway may write or read payment-related data to/from an Amazon DynamoDB. DynamoDB helps us deal with the challenges associated with both throughput and latency during peak payment processing times at certain days of the day or month.
The three technologies chosen by TxB to help us tackle the high throughput, low latency, and high availability challenges discussed previously are as follows:
- Amazon ECS (Elastic Container Service) running on AWS Fargate as our compute layer for the Payment Gateway
- Amazon MSK (Managed Streaming for Apache Kafka) for inter-process communication between the Payment Gateway and other TxB Services
- Amazon DynamoDB as our operational datastore
AWS Fargate and Amazon ECS
To achieve high throughput and high availability for our instant payments architecture, the compute layer needs to scale based on the architecture’s resource demands with no developer intervention. AWS Fargate supports automatic scaling ("auto scaling") that fits our instant payments use case. This feature has the following advantages for our architecture:
- Reduces maintenance required by our DevOps staff. As the application’s resource demands increase or decrease, we do not need manual intervention to add or remove ECS tasks.
- ECS auto scaling allows us to scale based on the application-level (task) metrics (e.g., number of payments per second). We also required more fine-grained control on the auto scaling demands to control costs. Auto scaling on the instance level is too coarse and therefore the auto scaling may add more instances that would be over-kill for our application’s demand.
- If DevOps intervention is needed for any reason (e.g., due to urgent security patching of the runtime environment or an urgent bug fix) a simple redeployment of the ECS cluster would take only a few seconds which is vital to maintain overall throughput and availability for instant payment processing.
Amazon DynamoDB
DynamoDB is a fully managed service offering NoSQL capabilities and supports regional replication natively. For our instant payments architecture, DynamoDB was an ideal fit for the following reasons:
- Simple Schema: For any instant payment sent or received from the system, there is only one data point we needed to model to ensure we can uniquely identify the payment. This data point is the Payment ID. The rest of the payment’s data points (such as sender, receiver, amount, currency, and so on) are stored as JSON strings. Therefore, our payments data are extremely simple and does not call for a highly normalized schema. And finally, since the JSON strings may have slightly different internal data elements between one payment and another, we required the flexibility to store this data in a schema-less fashion. Moreover, since the payments data are based on the ISO20022 standard, evolution of this standard will not affect our schema where the schema may need to evolve in a traditional RDBMS.
- Payment Lifecycle Management: Each instant payment’s lifecycle is captured and milestoned. One of our functional requirements is to ensure we can track a payment’s lifecycle as it is progressing in the flow discussed previously. Therefore, this introduces not only a volume-related challenge, but also a “chattiness” challenge between the Payment Gateway and DynamoDB. Although any payment’s complete lifecycle is only a few seconds (e.g., under 10 seconds), over a dozen lifecycle events are generated for each payment over the course of these few seconds. These events must be saved (for regulatory, compliance and other internal reporting purposes). By using the native provisioned capacity feature of DynamoDB, we overcome our throughput challenges.
- DynamoDB Supports CDC (Change Data Capture) Streaming: Whenever a data item (i.e., a payment lifecycle event) is inserted (or updated) in DynamoDB, this item will be streamed to other downstream consumers (such as data lakes and other historical data stores). We store the bulk of a payment’s lifecycle event as a JSON string. On insert into DynamoDB, the JSON payload is streamed to any downstream system and these consumers may parse the payload for any further processing. By using this native CDC feature, we don’t need to build or deploy additional code to publish these payload data to other consumers, thus further simplifying the architecture.
- Support for Global Tables: As shown in the architecture diagram above, to achieve high availability, our instant payments stack is deployed on us-east and us-west. DynamoDB natively supports cross-region table replication using its Global Tables feature.
Amazon MSK (Kafka)
When an instant payment is being processed by TxB, some of the primary functions that need to be performed on the payment include security validations, payment-level functional checks including fraud checks, and payment enrichments. These functional checks will be handled by other TxB Services as shown in the diagram.
As discussed in the DynamoDB section, these call-outs to other TxB Services will generate lifecycle events per call. When calls by the Payment Gateway to other TxB Services are being made via Kafka, the request/response latency must be fast (single-digit milliseconds or lower) to ensure high throughput is achieved, especially during bursts of activity. We can achieve these low latency requirements by using Kafka.
When TxB exchanges payments with the Payment Exchange Network, we need to ensure that we are processing these payments exactly once to avoid duplicate payments processing. To achieve exactly once processing on producers, we take advantage of Kafka’s native idempotency feature. Contrasted with other messaging technologies (such as traditional queueing technologies), these messaging schemes do not support producer idempotency natively like Kafka without added engineering effort.
With respect to performance (message latency), in our lab environment we found that Kafka outperforms traditional queuing technologies by about 175% for our instant payments use case. In other words, given the same number of consumers and producers in a multi-threaded environment and the same payment messages load, the time taken (latency) for a message to be consumed by a queue-based consumer was far greater than that of a Kafka consumer.
One final consideration on why Kafka was chosen for our instant processing use case: Kafka’s ecosystem natively supports a Schema Registry. Due to the strict time requirements of processing instant payments and the importance of ensuring that Kafka producers and consumers adhere to the ISO20022 specifications, having native support in Kafka for schema validation is greatly beneficial. Since payments processing is a critical function for TxB, we must ensure that if there are schema (contract) violations between consumers and producers, we fail-fast. In this context, fail-fast means that Kafka producers or consumers reject the payment message before further processing of the payment. For our use case, failing fast on contract violations between consumer and producers is preferable over processing potentially erroneous payments. In addition, since the Schema Registry feature is natively supported by the Kafka ecosystem, no added latency is introduced to payments processing where otherwise we would have had to contend with latency if we had to validate the message contract using other means (such as consumers and producers building their own custom validations). In instant payments processing, validating the contract between producers and consumers using an external system (such as another TxB-built custom service) may introduce more latency. Therefore, the native enforcement of the data contract was a critical deciding factor for us.
Conclusion
Our instant payments architecture using Amazon MSK (Kafka), Amazon DynamoDB and AWS Fargate/Amazon ECS, presents a solution capable of addressing the challenges inherent in high-throughput, low-latency and 24x7 availability requirements. By using these services, our instant payment architecture can scale to larger payment volumes without compromising on throughout, latency or availability.
By using these managed services and their respective scaling automation, we have achieved cost savings in terms of both infrastructure and engineering effort.
See https://www.gs.com/disclaimer/global_email for important risk disclosures, conflicts of interest, and other terms and conditions relating to this blog and your reliance on information contained in it.
Solutions
Curated Data Security MasterData AnalyticsPlotTool ProPortfolio AnalyticsGS QuantTransaction BankingGS DAP®Liquidity Investing¹ Real-time data can be impacted by planned system maintenance, connectivity or availability issues stemming from related third-party service providers, or other intermittent or unplanned technology issues.
Transaction Banking services are offered by Goldman Sachs Bank USA ("GS Bank") and its affiliates. GS Bank is a New York State chartered bank, a member of the Federal Reserve System and a Member FDIC. For additional information, please see Bank Regulatory Information.
² Source: Goldman Sachs Asset Management, as of March 31, 2025.
Mosaic is a service mark of Goldman Sachs & Co. LLC. This service is made available in the United States by Goldman Sachs & Co. LLC and outside of the United States by Goldman Sachs International, or its local affiliates in accordance with applicable law and regulations. Goldman Sachs International and Goldman Sachs & Co. LLC are the distributors of the Goldman Sachs Funds. Depending upon the jurisdiction in which you are located, transactions in non-Goldman Sachs money market funds are affected by either Goldman Sachs & Co. LLC, a member of FINRA, SIPC and NYSE, or Goldman Sachs International. For additional information contact your Goldman Sachs representative. Goldman Sachs & Co. LLC, Goldman Sachs International, Goldman Sachs Liquidity Solutions, Goldman Sachs Asset Management, L.P., and the Goldman Sachs funds available through Goldman Sachs Liquidity Solutions and other affiliated entities, are under the common control of the Goldman Sachs Group, Inc.
Goldman Sachs & Co. LLC is a registered U.S. broker-dealer and futures commission merchant, and is subject to regulatory capital requirements including those imposed by the SEC, the U.S. Commodity Futures Trading Commission (CFTC), the Chicago Mercantile Exchange, the Financial Industry Regulatory Authority, Inc. and the National Futures Association.
FOR INSTITUTIONAL USE ONLY - NOT FOR USE AND/OR DISTRIBUTION TO RETAIL AND THE GENERAL PUBLIC.
This material is for informational purposes only. It is not an offer or solicitation to buy or sell any securities.
THIS MATERIAL DOES NOT CONSTITUTE AN OFFER OR SOLICITATION IN ANY JURISDICTION WHERE OR TO ANY PERSON TO WHOM IT WOULD BE UNAUTHORIZED OR UNLAWFUL TO DO SO. Prospective investors should inform themselves as to any applicable legal requirements and taxation and exchange control regulations in the countries of their citizenship, residence or domicile which might be relevant. This material is provided for informational purposes only and should not be construed as investment advice or an offer or solicitation to buy or sell securities. This material is not intended to be used as a general guide to investing, or as a source of any specific investment recommendations, and makes no implied or express recommendations concerning the manner in which any client's account should or would be handled, as appropriate investment strategies depend upon the client's investment objectives.
United Kingdom: In the United Kingdom, this material is a financial promotion and has been approved by Goldman Sachs Asset Management International, which is authorized and regulated in the United Kingdom by the Financial Conduct Authority.
European Economic Area (EEA): This marketing communication is disseminated by Goldman Sachs Asset Management B.V., including through its branches ("GSAM BV"). GSAM BV is authorised and regulated by the Dutch Authority for the Financial Markets (Autoriteit Financiële Markten, Vijzelgracht 50, 1017 HS Amsterdam, The Netherlands) as an alternative investment fund manager ("AIFM") as well as a manager of undertakings for collective investment in transferable securities ("UCITS"). Under its licence as an AIFM, the Manager is authorized to provide the investment services of (i) reception and transmission of orders in financial instruments; (ii) portfolio management; and (iii) investment advice. Under its licence as a manager of UCITS, the Manager is authorized to provide the investment services of (i) portfolio management; and (ii) investment advice.
Information about investor rights and collective redress mechanisms are available on www.gsam.com/responsible-investing (section Policies & Governance). Capital is at risk. Any claims arising out of or in connection with the terms and conditions of this disclaimer are governed by Dutch law.
To the extent it relates to custody activities, this financial promotion is disseminated by Goldman Sachs Bank Europe SE ("GSBE"), including through its authorised branches. GSBE is a credit institution incorporated in Germany and, within the Single Supervisory Mechanism established between those Member States of the European Union whose official currency is the Euro, subject to direct prudential supervision by the European Central Bank (Sonnemannstrasse 20, 60314 Frankfurt am Main, Germany) and in other respects supervised by German Federal Financial Supervisory Authority (Bundesanstalt für Finanzdienstleistungsaufsicht, BaFin) (Graurheindorfer Straße 108, 53117 Bonn, Germany; website: www.bafin.de) and Deutsche Bundesbank (Hauptverwaltung Frankfurt, Taunusanlage 5, 60329 Frankfurt am Main, Germany).
Switzerland: For Qualified Investor use only - Not for distribution to general public. This is marketing material. This document is provided to you by Goldman Sachs Bank AG, Zürich. Any future contractual relationships will be entered into with affiliates of Goldman Sachs Bank AG, which are domiciled outside of Switzerland. We would like to remind you that foreign (Non-Swiss) legal and regulatory systems may not provide the same level of protection in relation to client confidentiality and data protection as offered to you by Swiss law.
Asia excluding Japan: Please note that neither Goldman Sachs Asset Management (Hong Kong) Limited ("GSAMHK") or Goldman Sachs Asset Management (Singapore) Pte. Ltd. (Company Number: 201329851H ) ("GSAMS") nor any other entities involved in the Goldman Sachs Asset Management business that provide this material and information maintain any licenses, authorizations or registrations in Asia (other than Japan), except that it conducts businesses (subject to applicable local regulations) in and from the following jurisdictions: Hong Kong, Singapore, India and China. This material has been issued for use in or from Hong Kong by Goldman Sachs Asset Management (Hong Kong) Limited and in or from Singapore by Goldman Sachs Asset Management (Singapore) Pte. Ltd. (Company Number: 201329851H).
Australia: This material is distributed by Goldman Sachs Asset Management Australia Pty Ltd ABN 41 006 099 681, AFSL 228948 (‘GSAMA’) and is intended for viewing only by wholesale clients for the purposes of section 761G of the Corporations Act 2001 (Cth). This document may not be distributed to retail clients in Australia (as that term is defined in the Corporations Act 2001 (Cth)) or to the general public. This document may not be reproduced or distributed to any person without the prior consent of GSAMA. To the extent that this document contains any statement which may be considered to be financial product advice in Australia under the Corporations Act 2001 (Cth), that advice is intended to be given to the intended recipient of this document only, being a wholesale client for the purposes of the Corporations Act 2001 (Cth). Any advice provided in this document is provided by either of the following entities. They are exempt from the requirement to hold an Australian financial services licence under the Corporations Act of Australia and therefore do not hold any Australian Financial Services Licences, and are regulated under their respective laws applicable to their jurisdictions, which differ from Australian laws. Any financial services given to any person by these entities by distributing this document in Australia are provided to such persons pursuant to the respective ASIC Class Orders and ASIC Instrument mentioned below.
- Goldman Sachs Asset Management, LP (GSAMLP), Goldman Sachs & Co. LLC (GSCo), pursuant ASIC Class Order 03/1100; regulated by the US Securities and Exchange Commission under US laws.
- Goldman Sachs Asset Management International (GSAMI), Goldman Sachs International (GSI), pursuant to ASIC Class Order 03/1099; regulated by the Financial Conduct Authority; GSI is also authorized by the Prudential Regulation Authority, and both entities are under UK laws.
- Goldman Sachs Asset Management (Singapore) Pte. Ltd. (GSAMS), pursuant to ASIC Class Order 03/1102; regulated by the Monetary Authority of Singapore under Singaporean laws
- Goldman Sachs Asset Management (Hong Kong) Limited (GSAMHK), pursuant to ASIC Class Order 03/1103 and Goldman Sachs (Asia) LLC (GSALLC), pursuant to ASIC Instrument 04/0250; regulated by the Securities and Futures Commission of Hong Kong under Hong Kong laws
No offer to acquire any interest in a fund or a financial product is being made to you in this document. If the interests or financial products do become available in the future, the offer may be arranged by GSAMA in accordance with section 911A(2)(b) of the Corporations Act. GSAMA holds Australian Financial Services Licence No. 228948. Any offer will only be made in circumstances where disclosure is not required under Part 6D.2 of the Corporations Act or a product disclosure statement is not required to be given under Part 7.9 of the Corporations Act (as relevant).
FOR DISTRIBUTION ONLY TO FINANCIAL INSTITUTIONS, FINANCIAL SERVICES LICENSEES AND THEIR ADVISERS. NOT FOR VIEWING BY RETAIL CLIENTS OR MEMBERS OF THE GENERAL PUBLIC
Canada: This presentation has been communicated in Canada by GSAM LP, which is registered as a portfolio manager under securities legislation in all provinces of Canada and as a commodity trading manager under the commodity futures legislation of Ontario and as a derivatives adviser under the derivatives legislation of Quebec. GSAM LP is not registered to provide investment advisory or portfolio management services in respect of exchange-traded futures or options contracts in Manitoba and is not offering to provide such investment advisory or portfolio management services in Manitoba by delivery of this material.
Japan: This material has been issued or approved in Japan for the use of professional investors defined in Article 2 paragraph (31) of the Financial Instruments and Exchange Law ("FIEL"). Also, any description regarding investment strategies on or funds as collective investment scheme under Article 2 paragraph (2) item 5 or item 6 of FIEL has been approved only for Qualified Institutional Investors defined in Article 10 of Cabinet Office Ordinance of Definitions under Article 2 of FIEL.
Interest Rate Benchmark Transition Risks: This transaction may require payments or calculations to be made by reference to a benchmark rate ("Benchmark"), which will likely soon stop being published and be replaced by an alternative rate, or will be subject to substantial reform. These changes could have unpredictable and material consequences to the value, price, cost and/or performance of this transaction in the future and create material economic mismatches if you are using this transaction for hedging or similar purposes. Goldman Sachs may also have rights to exercise discretion to determine a replacement rate for the Benchmark for this transaction, including any price or other adjustments to account for differences between the replacement rate and the Benchmark, and the replacement rate and any adjustments we select may be inconsistent with, or contrary to, your interests or positions. Other material risks related to Benchmark reform can be found at https://www.gs.com/interest-rate-benchmark-transition-notice. Goldman Sachs cannot provide any assurances as to the materialization, consequences, or likely costs or expenses associated with any of the changes or risks arising from Benchmark reform, though they may be material. You are encouraged to seek independent legal, financial, tax, accounting, regulatory, or other appropriate advice on how changes to the Benchmark could impact this transaction.
Confidentiality: No part of this material may, without GSAM's prior written consent, be (i) copied, photocopied or duplicated in any form, by any means, or (ii) distributed to any person that is not an employee, officer, director, or authorized agent of the recipient.
GSAM Services Private Limited (formerly Goldman Sachs Asset Management (India) Private Limited) acts as the Investment Advisor, providing non-binding non-discretionary investment advice to dedicated offshore mandates, involving Indian and overseas securities, managed by GSAM entities based outside India. Members of the India team do not participate in the investment decision making process.