At Goldman Sachs, the infrastructure resources in our cloud can be managed using Infrastructure-as-Code (IaC). The engineering teams at Goldman Sachs request these resources, via code, on-demand and expect the provisioning to be completed in a matter of minutes. The various infrastructure resources include compute, network and storage along with other services. This post will explore how network connectivity requirements are fulfilled by PINACL.
PINACL is a GS Accelerate sponsored product that seeks to drive end-to-end automation of network security connectivity policies that control network traffic in an enterprise. PINACL allows connectivity policy to be authored and owned by application developers rather than network operators. It leverages network topology, built for the enterprise by PINACL, to discover the impact on the existing security posture and delivers the user's intent.
Enterprise Requirements
- Scale: Application developers and network operators submit multiple connectivity requests which results in a large number of connectivity flows.
- Consistency: Data should be strictly consistent across the system because inconsistencies could lead to unauthorized access and security incidents.
- Latency: Users would like to realize their infrastructure changes in a timely manner. Since a single connectivity request could impact multiple security enforcement points, the fulfillment system should have high throughput to keep the end-to-end latency low.
Solutions
- Highly scalable distributed services that asynchronously receive and deliver proper provisioning.
- PINACL uses CockroachDB as a datastore. CockroachDB is a distributed database with a SQL interface that promises consistency and partition tolerance.
PINACL Architecture
PINACL's architectural and design decisions are influenced by these challenges and solutions listed above. The product's capabilities can be broadly divided into two parts: upstream and downstream systems.
Upstream System
The upstream system accepts requests through REST API endpoints for realizing any change in the network security posture. It processes the events in an asynchronous manner and adds tasks to the policy fulfillment queue.
Description of the upstream system: A flow chart diagram detailing the flow from IAC application developers, network operators , event streams and topology changes funneling into the api server, the processor queue, pipeline processor fulfillment queue. * Each USER (use case) will have a dedicated / independent UPSTREAM SYSTEM processing the requests / changes.
High availability, low coupling, and high cohesion are the driving forces for the design of the upstream system.
Each use case like policy management by network engineers or software developers has been divided into individual services with single responsibility. Each component has its own dedicated upstream service independent of others making it easier to maintain and extend the system to support new use cases in the future. All the upstream systems communicate with the downstream system through the same fulfilment queue.
The upstream system is further divided into two parts—API server and Pipeline processor—to ensure high availability. The multiple instances or replicas of API servers can continue accepting and queuing the orders, even when the pipeline processor is down due to issues with the processing of any request. All the requests will eventually be processed by the pipeline processor.
API Server: Authorized users can interact with PINACL using the RESTful APIs exposed by the server component to CREATE / READ / UPDATE / DELETE the security policies managed by them.
Pipeline Processor: Processes user connectivity intent by first detecting the impacted rules and then with the help of network topology determines the security devices which need to be reprovisioned. These security devices are then added to the fulfillment queue which will be processed by the downstream system.
For example, consider the following network topology where there is a security device (SD1) blocking the network traffic between Host A and Host B.
A flow diagram showing the before and after of the connectivity between Host A and Host B with the security device interrupting the flow. In the after portion of the diagram is unblocked i- with the policy flowing through from PINACL allowing the security device being programmed.
Downstream / Fulfillment System
The downstream system processes the tasks in the policy fulfillment queue by preparing and pushing the latest policies to the security devices.
A diagram showing multiple upstream systems flowing into a priority based fulfilment queue into orchestraqtors controlled by requesting agents, then down into the vendor provided security devices.
The rate at which the upstream systems add tasks to the fulfillment queue can be high, driven by the volume of changes it receives, making horizontal scalability the driving factor in designing the downstream system in order to maintain low end-to-end latency. To ensure scalability, the fulfillment system adopts the Orchestrator - Agent pattern with task stealing approach, where the agents request (pull) the tasks from the orchestrator when it's free. We adapted a pull-based approach over a push-based one as the rate of processing and completion of each task varies widely based on the policy size and vendor. The agent nodes are elastic. During heavy load, new agent nodes can be dynamically added to the group to share the load / tasks and increase the throughput. Each task is a unit of work which involves pushing the security policy to a device.
Orchestrator acts as a mediator to group the same security devices currently pending in the queue to create a single task, when an agent requests for one. It also ensures the same security device is not allocated to different agents at the same time to avoid policy provisioning conflicts.
Agent composes the policy by optimizing / compressing the flows that will be pushed to a security device from many millions to a few thousand. The agent then translates and pushes the policy to the security device using the driver of the respective vendor.
Queue Control: Priority can be assigned to the task which influences the order of fulfillment which becomes effective in controlling the queue especially during heavy loads.
Data Management in PINACL
At a high level, the PINACL product needs to keep track of a few key datasets:
- Connectivity policy contains a collection of rules. Rules reference endpoints (or endpoint groups) and app protocol (or app protocol groups). Endpoint groups are defined as a collection of other endpoint groups (nested) or leaf endpoints. The rules and endpoint group relationships change often. They are entities with strong relationships between them and have their own lifetime and ownership.
- Network Topology information, which changes less frequently.
- Upstream system computes the impact caused by policy changes and determines which security devices need to be programmed. This requires joins between the entities and Recursive Common Table Expression, i.e. CTE to resolve nested dependencies.
Relational Database Choice
Both the upstream and downstream processing requires significant data joins to fulfill the requests. With most document DBs, server-side join is either non-existent or works with only non-sharded collections (e.g. MongoDB's lookup operator) and limited to equi-joins. This requires applications to perform client side joins which are relatively expensive due to network latency, amount of data transferred over the network, and the number of calls to the DB.
While a database which doesn't enforce any schema is easier to develop, there is value in enforcing a schema and referential integrity at the database layer. Datasets tend to outlive the application, so ensuring the data integrity is paramount and having the database prevent data quality issues due to bugs in the application is valuable.
With de-normalized documents in either JSON/BSON format, there is a significant overhead to the dataset size compared to a relational database. It is partly to do with denormalization introducing repeated values and partly to do with JSON/BSON requiring repeated string keys.
Support for ACID transactions simplifies application development without having to detect and workaround various failure scenarios.
Selecting CockroachDB
CockroachDB is an open source database providing support for relational data model, range sharded data partitioning, replication, and high availability. CockroachDB uses MVCC to provide ACID and more specifically serializable isolation level. It supports PostgreSQL wire protocol allowing existing PostgreSQL clients (eg: JDBC driver) to work seamlessly with CockroachDB.
While there are lots of features and benefits with using CockroachDB, here are a few that were important when making a selection:
- Supports transactions with serializable isolation level which simplifies application development.
- Leverages its MVCC to support point-in-time queries to reduce contention with other writes in the system.
- Range partitioning of data allows the data storage layer to scale by addition of more nodes to the cluster.
- Provides high availability out of the box, as long as the quorum is available i.e. N/2+1 nodes are up, with replication of every range N times across the cluster.
- Supports online backup by leveraging point-in-time query capabilities.
- Supports rolling upgrades.
- Supports IPv4 and IPv6 data types.
What's Next?
Today, PINACL supports multiple security device vendors and with the growing need of multi vendor solutions, we are planning to add support for more vendors. With the increasing adoption of cloud infrastructure along with on-premise, PINACL seeks to extend the support for the hybrid environment. The network topology consists of different kinds of devices which can enforce security like Router ACLs. PINACL is envisioned to become the single solution to manage all kinds of security enforcement points.
PINACL is actively hiring software engineers in Bengaluru, see roles here.
About GS Accelerate
Since launching in 2018, Goldman Sachs employees have submitted more than 1,800 ideas, many of which, including PINACL, have been funded, resulting in seven new products available for Goldman Sachs clients and people. GS Accelerate businesses are hiring across teams and functions in offices across the globe including New York, Dallas, and Birmingham. To view open roles, visit here.
PINACL technology is covered by pending and issued patents.
See https://www.gs.com/disclaimer/global_email for important risk disclosures, conflicts of interest, and other terms and conditions relating to this blog and your reliance on information contained in it.
Solutions
Curated Data Security MasterData AnalyticsPlotTool ProPortfolio AnalyticsGS QuantTransaction BankingGS DAP®Liquidity Investing¹ Real-time data can be impacted by planned system maintenance, connectivity or availability issues stemming from related third-party service providers, or other intermittent or unplanned technology issues.
Transaction Banking services are offered by Goldman Sachs Bank USA ("GS Bank") and its affiliates. GS Bank is a New York State chartered bank, a member of the Federal Reserve System and a Member FDIC. For additional information, please see Bank Regulatory Information.
² Source: Goldman Sachs Asset Management, as of March 31, 2025.
Mosaic is a service mark of Goldman Sachs & Co. LLC. This service is made available in the United States by Goldman Sachs & Co. LLC and outside of the United States by Goldman Sachs International, or its local affiliates in accordance with applicable law and regulations. Goldman Sachs International and Goldman Sachs & Co. LLC are the distributors of the Goldman Sachs Funds. Depending upon the jurisdiction in which you are located, transactions in non-Goldman Sachs money market funds are affected by either Goldman Sachs & Co. LLC, a member of FINRA, SIPC and NYSE, or Goldman Sachs International. For additional information contact your Goldman Sachs representative. Goldman Sachs & Co. LLC, Goldman Sachs International, Goldman Sachs Liquidity Solutions, Goldman Sachs Asset Management, L.P., and the Goldman Sachs funds available through Goldman Sachs Liquidity Solutions and other affiliated entities, are under the common control of the Goldman Sachs Group, Inc.
Goldman Sachs & Co. LLC is a registered U.S. broker-dealer and futures commission merchant, and is subject to regulatory capital requirements including those imposed by the SEC, the U.S. Commodity Futures Trading Commission (CFTC), the Chicago Mercantile Exchange, the Financial Industry Regulatory Authority, Inc. and the National Futures Association.
FOR INSTITUTIONAL USE ONLY - NOT FOR USE AND/OR DISTRIBUTION TO RETAIL AND THE GENERAL PUBLIC.
This material is for informational purposes only. It is not an offer or solicitation to buy or sell any securities.
THIS MATERIAL DOES NOT CONSTITUTE AN OFFER OR SOLICITATION IN ANY JURISDICTION WHERE OR TO ANY PERSON TO WHOM IT WOULD BE UNAUTHORIZED OR UNLAWFUL TO DO SO. Prospective investors should inform themselves as to any applicable legal requirements and taxation and exchange control regulations in the countries of their citizenship, residence or domicile which might be relevant. This material is provided for informational purposes only and should not be construed as investment advice or an offer or solicitation to buy or sell securities. This material is not intended to be used as a general guide to investing, or as a source of any specific investment recommendations, and makes no implied or express recommendations concerning the manner in which any client's account should or would be handled, as appropriate investment strategies depend upon the client's investment objectives.
United Kingdom: In the United Kingdom, this material is a financial promotion and has been approved by Goldman Sachs Asset Management International, which is authorized and regulated in the United Kingdom by the Financial Conduct Authority.
European Economic Area (EEA): This marketing communication is disseminated by Goldman Sachs Asset Management B.V., including through its branches ("GSAM BV"). GSAM BV is authorised and regulated by the Dutch Authority for the Financial Markets (Autoriteit Financiële Markten, Vijzelgracht 50, 1017 HS Amsterdam, The Netherlands) as an alternative investment fund manager ("AIFM") as well as a manager of undertakings for collective investment in transferable securities ("UCITS"). Under its licence as an AIFM, the Manager is authorized to provide the investment services of (i) reception and transmission of orders in financial instruments; (ii) portfolio management; and (iii) investment advice. Under its licence as a manager of UCITS, the Manager is authorized to provide the investment services of (i) portfolio management; and (ii) investment advice.
Information about investor rights and collective redress mechanisms are available on www.gsam.com/responsible-investing (section Policies & Governance). Capital is at risk. Any claims arising out of or in connection with the terms and conditions of this disclaimer are governed by Dutch law.
To the extent it relates to custody activities, this financial promotion is disseminated by Goldman Sachs Bank Europe SE ("GSBE"), including through its authorised branches. GSBE is a credit institution incorporated in Germany and, within the Single Supervisory Mechanism established between those Member States of the European Union whose official currency is the Euro, subject to direct prudential supervision by the European Central Bank (Sonnemannstrasse 20, 60314 Frankfurt am Main, Germany) and in other respects supervised by German Federal Financial Supervisory Authority (Bundesanstalt für Finanzdienstleistungsaufsicht, BaFin) (Graurheindorfer Straße 108, 53117 Bonn, Germany; website: www.bafin.de) and Deutsche Bundesbank (Hauptverwaltung Frankfurt, Taunusanlage 5, 60329 Frankfurt am Main, Germany).
Switzerland: For Qualified Investor use only - Not for distribution to general public. This is marketing material. This document is provided to you by Goldman Sachs Bank AG, Zürich. Any future contractual relationships will be entered into with affiliates of Goldman Sachs Bank AG, which are domiciled outside of Switzerland. We would like to remind you that foreign (Non-Swiss) legal and regulatory systems may not provide the same level of protection in relation to client confidentiality and data protection as offered to you by Swiss law.
Asia excluding Japan: Please note that neither Goldman Sachs Asset Management (Hong Kong) Limited ("GSAMHK") or Goldman Sachs Asset Management (Singapore) Pte. Ltd. (Company Number: 201329851H ) ("GSAMS") nor any other entities involved in the Goldman Sachs Asset Management business that provide this material and information maintain any licenses, authorizations or registrations in Asia (other than Japan), except that it conducts businesses (subject to applicable local regulations) in and from the following jurisdictions: Hong Kong, Singapore, India and China. This material has been issued for use in or from Hong Kong by Goldman Sachs Asset Management (Hong Kong) Limited and in or from Singapore by Goldman Sachs Asset Management (Singapore) Pte. Ltd. (Company Number: 201329851H).
Australia: This material is distributed by Goldman Sachs Asset Management Australia Pty Ltd ABN 41 006 099 681, AFSL 228948 (‘GSAMA’) and is intended for viewing only by wholesale clients for the purposes of section 761G of the Corporations Act 2001 (Cth). This document may not be distributed to retail clients in Australia (as that term is defined in the Corporations Act 2001 (Cth)) or to the general public. This document may not be reproduced or distributed to any person without the prior consent of GSAMA. To the extent that this document contains any statement which may be considered to be financial product advice in Australia under the Corporations Act 2001 (Cth), that advice is intended to be given to the intended recipient of this document only, being a wholesale client for the purposes of the Corporations Act 2001 (Cth). Any advice provided in this document is provided by either of the following entities. They are exempt from the requirement to hold an Australian financial services licence under the Corporations Act of Australia and therefore do not hold any Australian Financial Services Licences, and are regulated under their respective laws applicable to their jurisdictions, which differ from Australian laws. Any financial services given to any person by these entities by distributing this document in Australia are provided to such persons pursuant to the respective ASIC Class Orders and ASIC Instrument mentioned below.
- Goldman Sachs Asset Management, LP (GSAMLP), Goldman Sachs & Co. LLC (GSCo), pursuant ASIC Class Order 03/1100; regulated by the US Securities and Exchange Commission under US laws.
- Goldman Sachs Asset Management International (GSAMI), Goldman Sachs International (GSI), pursuant to ASIC Class Order 03/1099; regulated by the Financial Conduct Authority; GSI is also authorized by the Prudential Regulation Authority, and both entities are under UK laws.
- Goldman Sachs Asset Management (Singapore) Pte. Ltd. (GSAMS), pursuant to ASIC Class Order 03/1102; regulated by the Monetary Authority of Singapore under Singaporean laws
- Goldman Sachs Asset Management (Hong Kong) Limited (GSAMHK), pursuant to ASIC Class Order 03/1103 and Goldman Sachs (Asia) LLC (GSALLC), pursuant to ASIC Instrument 04/0250; regulated by the Securities and Futures Commission of Hong Kong under Hong Kong laws
No offer to acquire any interest in a fund or a financial product is being made to you in this document. If the interests or financial products do become available in the future, the offer may be arranged by GSAMA in accordance with section 911A(2)(b) of the Corporations Act. GSAMA holds Australian Financial Services Licence No. 228948. Any offer will only be made in circumstances where disclosure is not required under Part 6D.2 of the Corporations Act or a product disclosure statement is not required to be given under Part 7.9 of the Corporations Act (as relevant).
FOR DISTRIBUTION ONLY TO FINANCIAL INSTITUTIONS, FINANCIAL SERVICES LICENSEES AND THEIR ADVISERS. NOT FOR VIEWING BY RETAIL CLIENTS OR MEMBERS OF THE GENERAL PUBLIC
Canada: This presentation has been communicated in Canada by GSAM LP, which is registered as a portfolio manager under securities legislation in all provinces of Canada and as a commodity trading manager under the commodity futures legislation of Ontario and as a derivatives adviser under the derivatives legislation of Quebec. GSAM LP is not registered to provide investment advisory or portfolio management services in respect of exchange-traded futures or options contracts in Manitoba and is not offering to provide such investment advisory or portfolio management services in Manitoba by delivery of this material.
Japan: This material has been issued or approved in Japan for the use of professional investors defined in Article 2 paragraph (31) of the Financial Instruments and Exchange Law ("FIEL"). Also, any description regarding investment strategies on or funds as collective investment scheme under Article 2 paragraph (2) item 5 or item 6 of FIEL has been approved only for Qualified Institutional Investors defined in Article 10 of Cabinet Office Ordinance of Definitions under Article 2 of FIEL.
Interest Rate Benchmark Transition Risks: This transaction may require payments or calculations to be made by reference to a benchmark rate ("Benchmark"), which will likely soon stop being published and be replaced by an alternative rate, or will be subject to substantial reform. These changes could have unpredictable and material consequences to the value, price, cost and/or performance of this transaction in the future and create material economic mismatches if you are using this transaction for hedging or similar purposes. Goldman Sachs may also have rights to exercise discretion to determine a replacement rate for the Benchmark for this transaction, including any price or other adjustments to account for differences between the replacement rate and the Benchmark, and the replacement rate and any adjustments we select may be inconsistent with, or contrary to, your interests or positions. Other material risks related to Benchmark reform can be found at https://www.gs.com/interest-rate-benchmark-transition-notice. Goldman Sachs cannot provide any assurances as to the materialization, consequences, or likely costs or expenses associated with any of the changes or risks arising from Benchmark reform, though they may be material. You are encouraged to seek independent legal, financial, tax, accounting, regulatory, or other appropriate advice on how changes to the Benchmark could impact this transaction.
Confidentiality: No part of this material may, without GSAM's prior written consent, be (i) copied, photocopied or duplicated in any form, by any means, or (ii) distributed to any person that is not an employee, officer, director, or authorized agent of the recipient.
GSAM Services Private Limited (formerly Goldman Sachs Asset Management (India) Private Limited) acts as the Investment Advisor, providing non-binding non-discretionary investment advice to dedicated offshore mandates, involving Indian and overseas securities, managed by GSAM entities based outside India. Members of the India team do not participate in the investment decision making process.