What makes the modern stack so shaky

“In the modern data stack, at the heart of it, there’s the data warehouse—that part is good. But around it are all these point solutions that you have to stitch together. They don't talk to one another. You have to write custom scripts, integrate solutions, and that's not productive for anybody, right? And if you look at the on-prem world, the old enterprise world, there was a simple turnkey solution that solved your whole problem. That is the right direction to go.

“The modern data stack is not going to last very long.”

That’s the contrarian take of Raj Bains, CEO of SignalFire portfolio company Prophecy.io, which transforms raw data into reliable, analytics-ready data using visual pipelines. Not unlike the “Stacks” of trailers in the future slums imagined by Ready Player One, the disorganized, makeshift data stack is precarious.

Raj sat down with SignalFire co-founder, CTO, and long-time Google engineer Ilya Kirnos to hammer out what he sees as the right way to build the modern data stack. Point solutions are not it! They also discussed the challenges inherent in getting the best solutions funded—young investors working with young companies are missing historical context and repeating old mistakes.

Here’s a breakdown of the problem with today’s enterprise infrastructure, and how founders working on the data stack can dodge the pitfalls of the past.

‎How did we get here?

SignalFire CTO Ilya Kirnos: There's been a lot of discussion about a new modern data stack—that's kind of become a term of art almost, or a brand in the wake of successful companies like Snowflake. I would love to hear your thoughts.

Prophecy CEO Raj Bains: So now you look at it and say, what have companies like Snowflake done? It's providing the same functionality [data warehouses], done better with cloud architecture. Enterprise folks from the last generation who really are steeped in database technology had come in and built a product [on prem] that was nearly impossible to find.

Now on top of [these cloud-based solutions], what you have is a set of tools. And the question is, what about them? Are they modern? No. They’re all simpler and relearning the lessons that were learned 30 years ago. These people have not looked at the history and they're mostly selling to startups, where the users have not used sophisticated products. 

I don't think that many VCs know the history. I talk to a lot of them. A lot of them are really young folk who've come from management consulting or those backgrounds. They don't know what the old enterprise stack looks like. Our customers [at Prophecy] are very deep in these technologies, and they've been working with these technologies for 20 years.

‎[Your] very different approach is resonating with customers.

A complete solution has really resonated with customers: “I want you to solve my complete problem.” It does not resonate with investors because they want quick traction. So they fund a lot of point solutions, and now you have ten companies with tiny point solutions trying to scale up, running to a little bit of growth, and stalling.

If you go with a large solution that solves the complete problem, it doesn’t match the investor metrics. It’s super hard to get funded because you have a lot of money going into R&D, into supporting the first large customers because the product is important to the large customers—all of which should be good signs, but this makes it very hard to get funded.

You started with some of the biggest companies as your very first customers—even when you were a seed stage company, which is very unusual in my experience. Was that also seen as a strange way to build a company by investors?

Yes, I thought a few years ago it was really nice to go to and get absolutely top enterprise companies as customers, but by the time we came back for Series A, people were like, “Hey, the fashion has changed.” And now the fashion is you go bottoms up, go to mid-market, and then you're going to be able to transition to enterprise. And I'm wondering, has any company done that? They're like, “No, no, no, it's a new playbook. It's not played itself out yet, but that's the way to build a company….You don't know what you're doing. That's not how you build a company.”

And with that, it was almost impossible to get funding. So the fashion changes, and it leads to some very, very interesting conversations.

‎Serving the enterprise customer

Well, obviously, you've made it this far. You've been able to raise capital [more than $30M] for your low-code tool that makes raw data analytics-ready, despite some of these headwinds in the investor sentiment. And I feel like the markets sort of came your way in that now there is a lot more interest in selling into the enterprise. A lot of these companies are trying to move up market and sell into some of these large companies. That's challenging because as you pointed out, they're architected differently—there are different levels of complexity and requirements for those large customers.

I think we'll get stronger every year, right? Because we have substance and storytelling. And enterprises are now like, I don't want 20 tools to play with. I want one thing that solves my whole problem.

And if you have a product they're going to use every day, it's a centrally important thing to them. We’re building data pipelines—the entire business will come to a standstill if we do not do it right.

For something like that, they are going to need customer success, because the product isn’t going to be completely finished. There’s a need for hands-on training: there are going to be 150 engineers who are going to use the product every day, six hours a day. Of course, they're going to say, “Oh, how do I do this?”

[The challenge with the investor market is] they look at the spreadsheet and say, “Oh, I'm looking for efficient growth; you know, you’re spending on customer success.” But I have over a half-a-million contract with a Fortune 5 company. Of course they're going to expect this, and it's going to grow to a few million dollars. This focus on efficient growth precludes companies that are actually really important. You look at Salesforce, you look at Workday, they had PS [professional services] in the beginning. It's an important product, it's central to the business. How are you not going to have PS? So when you look at the general consensus of the market, it's just wrong.

Yeah, well it sounds like the toughest part has been the funding environment and the sentiment in the investor community, but you're finding great resonance with customers, and at the end of the day that's what's going to win.

Yes, we have single-mindedly focused on what our customers really want and given that to them, and that is working really, really well for us.

How do you see code versus low-code evolving?

You don't have to choose between code and visual. You can do visual drag-and-drop and that turns into dbt code—or SQL with dbt code to be more precise. Or you can open any of those projects and edit them visually. For any solution to become dominant, it has to enable the maximum number of data users in an enterprise. And we are squarely focused on that, and providing a complete solution.

What's next for Prophecy? How are enterprise data infrastructure companies thinking about AI?

Generative AI is going to shake up the industry quite a bit, there are a couple of large changes occurring that we’re smack in the middle of. 

One, we are seeing tremendous results from building applications that combine enterprise data with off-the-shelf large language models. Our belief is that for the vast majority of enterprise use cases, the era of hiring machine learning engineers who train proprietary models, or specialize LLMs is over. All you need is a private knowledge warehouse that provides private context along with questions and the LLMs give very relevant answers. In a few months, as tools become more available, you’ll see data engineers or application engineers turn around applications like support bots on private data in a week. Off-the-shelf models being so good will make it a data engineering problem rather than a machine learning problem.

We see generative AI–based copilots making low-code the default way to build applications and data pipelines in enterprises. We already see a 30–40% productivity boost from low-code adoption in our customer base. That, coupled with another 30–40% from copilots will make starting with code financially untenable for the majority of enterprise use cases. The future of enterprise development is definitely copilot-powered low-code.

Tying it back to Prophecy in the larger data tools ecosystem: our ability to build new visual components with Python to integrate with the AI ecosystem and our handling of unstructured data position us well for LLM-based applications. The fact that we have a complete product where metadata can power copilots positions us excellently well for productivity. Overall, we think we got the architecture right, and as the AI wave comes we’ll see most of the rest of the modern data stack fall apart.

Exciting times ahead!

Takeaways for building a better data stack

• Enterprise customers don't want the tangle of point solutions, they want a complete solution with customer success baked in
• Point solutions for the mid-market may be quicker to gain traction and funding, but they struggle to move up market to enterprise and turn into big companies
• Off-the-shelf LLMs combined with enterprise data sets, plus low-code tools and generative AI-based coding co-pilots will become the default way to develop data pipelines and enterprise applications

Building a startup in the data infrastructure space? We’d love to hear from you. Reach out to one of our investment team members.