Legend + Snowflake Native Apps = Fast, Easy, Secure Access to Data

How engineers at Goldman Sachs and Snowflake partnered to unlock native Snowflake performance on top of Legend APIs

Goldman Sachs's data strategy is tied directly to Legend and the firm's Financial Cloud for Data. Our latest offering gives our clients access to curated GS data and provides a client-owned AWS runtime to power the behind-the-scenes data movement. We are also users of Snowflake for relational data warehousing and data engineering. 

The idea to combine our capabilities was born from a simple conversation that happened in April 2022... Abhishek relates how the idea came about

Running our vendor data engineering team I have personally seen the time and energy required to set up teams with access to new datasets. It is a frustrating process - for both business users and engineers; 

At a recent Snowflake event, I learned about Native Apps, a marketplace of new capabilities they built on top of secure data sharing. Following the event, my manager stopped by my desk to learn more about these features and brainstormed how Snowflake Native Apps could potentially solve a critical and long-standing issue for our users

This conversation quickly led to an “ah ha!” moment: if we can combine Legend with the Snowflake Native App capabilities, we can unlock the full performance of Snowflake while still providing logical separation of client, vendor, and Goldman Sachs data. 

This new idea presented a great opportunity to work with internal researchers to understand the impact we could achieve. And the impact really was phenomenal - our research partners could setup new datasets in days instead of months. They could do this in a fully self-serve model while adhering to data governance standards. And our engineering team didn't have to support our business teams every single time they wanted new datasets or insights. A win-win for everyone!

But our ambitions didn't end there. Once we unlocked the foundational set of capabilities from this new solution, we wanted to push the boundaries of this innovation. The next step was for us to identify a real-world use case that could scale this not just internally at GS, but to external clients of the firm. We decided to leverage data from the newly launched Goldman Sachs Financial Cloud for Data to make the experience of sharing insights with our clients even more seamless. Adding more functionality within our existing application suite is a powerful value proposition for our clients.  We are excited to go-to-market with this service once Snowflake Native Apps are publicly available (expected end of this year).

Our problem statement:

A researcher at an institutional investor client who wants to access new datasets from various external sources has to work with a data engineer to complete a multi-step time consuming process. They need to spend a significant amount of time figuring out how their internal data stitches together with third party data across multiple ecosystems. It’s also hard on the engineering side to understand the intent of Quants/Researchers as they run into complex extract, transform and load (ETL) problems with multi hop operations yielding lack of ownership and performance issues. After this is all in production, the researcher still does not get lineage.

Our mission:

Allow business users to pull together all the data in need using an intuitive, self-service experience that is highly performant, secure and scalable.