Tomorrow’s machine learning startup opportunities

Fads fade. We believe machine learning (ML) is a classic piece of building software. Classics have an undeniable air of timelessness. They are hard to ignore and impossible to miss. Classics are forever. We believe ML is forever for software. In fact, at SignalFire, ML is woven into the software we build. We develop our own ML models in-house for our Beacon AI data platform, which we use for sourcing investments and helping portfolio companies with recruiting. That’s given us a deep appreciation for how ML tooling can improve a business, so we’re eager to invest in and support founders building the future of ML infrastructure.

Hot takes on machine learning and LLMs

1. For the time being, large companies will own model training for large language models (LLMs) and foundation models (FMs). Training requires a combination of proprietary/web-scale datasets and costly infrastructure, making them too big a challenge to be adequately addressed by smaller companies.
2. Companies can open-source models and methodology without fear of giving away their secret sauce since data is often proprietary and infrastructure is tough to build. The secret sauce is not primarily the recipe, but the rare ingredients and huge kitchen required. Facebook’s pytorch may be open-source, but you can’t build Facebook models on your own.
3. Foundation models unlock bigger TAMs for enterprises via additional services. Incumbents can charge higher annual contract value for premium AI features, and there is a high interest in companies like Fixie helping enterprises build with LLMs.
4. Foundation models are industry agnostic, making specialization a growth opportunity. To scale in the commercial medium, one has to fine-tune on industry-specific data, as well as have more control and visibility—for example, the ability to filter for brand safety and content moderation standards.
5. MLOps will be a critical component in the industry for the foreseeable future. For several years to come MLOps will be important for tasks where LLMs don’t perform reliably. And once large models become more efficient, MLOps will be able to fine-tune the models on a task, get data, manage artifacts, monitor, update a model to address failures, and more.

ML is a core component of the software we build and our portfolio

SignalFire truly understands ML because we’ve spent 10 years building our own ML models for many uses across venture: sourcing companies, completing due diligence, winning competitive deals, and supporting our portfolio with recruiting data and customer lead lists. We do not build a “robot general partner,” but rather utilize data to empower human decision-makers. For example, we use natural language processing (NLP) models to classify companies and surface them to the appropriate investors, classify potential recruiting leads for our portfolio companies to match their hiring needs, and use graph ML algorithms on the open-source community to find compelling technical projects, many of which are ML projects themselves!

MLOps or MLOops?

Machine learning infrastructure went from Matlab produced by Mathworks in 1984 to an explosion of tooling to operationalize models, referred to in the industry as MLOps. Up to 2021, $3.8B was invested in MLOps. The category has existed for a decade, and while there are a few companies valued at $1B+—such as Scale.ai, Weights & Biases, and DataRobot—there is no IPO yet. Which begs the question: MLOps or MLOops? Here’s how we think the timeline will play out.

ML infrastructure timeline

We believe that MLOps aren’t disappearing anytime soon. LLMs have raised the awareness of AI in our collective mindshare. While they can be useful, LLMs are not (yet) suitable for all tasks, such as time series forecasting, or where end users can’t afford to be wrong in even one percent of cases, such as self-driving cars. While the latest GPT-4 LLM has expanded to accept audio and image inputs—and other foundation models tackle image (DALL-E 2) and audio (AudioLM) generation—the high cost of inference and model fine-tuning are a barrier for high-volume, low-margin business applications.

For several years to come, MLOps will be needed for tasks where LLMs don’t perform or are cost prohibitive. And even once large models become more efficient, there will still be a need for MLOps to fine-tune the models on your task, get data, manage artifacts, monitor data, update the model when you see failure cases, and update the simpler models that manage spam filtering and toxicity detection, among other things. 

In the category of MLOps, we invested in annotation (Explosion), testing (Kolena), and compute (SaturnCloud). While there is no common tooling yet, there is a common workflow to build models. In the image below, you can see an incomplete sketch of companies operating in each of the categories, illustrating in one snapshot the common workflow for building ML models. It captures notable companies for each step, the different teams involved, and some of their publicly announced total funding. 

This graphic shows how operationalizing an ML model requires a wide variety of different tools and teams with deep expertise. Enterprise companies with the requisite data may still find it hard to staff all these teams with top AI talent—but they have the funds to purchase MLOps.

MLOps startup opportunities

We want to meet founders to solve common pain points for commercial customers who want to add AI skills. If you are building tooling that helps companies use ML to benefit the bottom line, please email [email protected]. 

Three of the areas we are especially interested to meet more MLOps startups are:

1. Visibility across the stack: Customers tell us their ML stacks are heterogeneous and hard to review and audit end-to-end. A head of AI at a large retail chain told us, “We have to work with many companies to get to 75% of what Google has internally as their ML platform.” He would pay top dollar for tooling that can provide the enterprise level of standardization and visibility end-to-end, vs. opinionated ML tooling for a point solution.   
2. Closing the skills gap: As the diagram makes clear, a multitude of different builders have to interact throughout the ML lifecycle. We are interested in startups that bridge the gap in skills between these different organizations as well as tooling that helps the engineers who build models to be more productive. 
3. Data engineering: We have more data than ever, and we are interested in tooling that makes the collection, transformation, and usage of data more efficient.

Native LLMOps 

Large language models are bringing faster time-to-value to enterprises. SignalFire recently invested in Fixie.ai in this space, a cloud-hosted Platform-as-a-Service that enables anyone to build and integrate smart agents that leverage the power of LLMs to solve problems based on natural language. We’re looking to meet more founders building native tooling to use LLMs for commercial use cases, particularly around these three areas:

1. Inference: Inference (applying an ML model to new data) is expensive, and optimizing it would be beneficial to many companies. Businesses need help with inference and also need help understanding the cost. Algorithmia pioneered MLOps; we would love to speak with teams building the next Algorithmia and startups helping build industrial-strength inference.
2. Business-specific context: Foundation models are industry agnostic; we are interested in tooling to verticalize a model, providing the vertical-specific context to the model, such as vector databases. 
3. Commercial control: To scale in the commercial medium, one has to have more control and visibility—for example, a tool to filter for brand safety. ChatGPT is cool but cannot be adopted and used in production if it uses profanity or risks saying something against brand standards that could be screenshotted and shared with everyone.

A VC engineered to help you scale

At SignalFire, we like to say “think of us as an extension of your team that scales with you.” Beyond our in-house Beacon AI, we built our full-time Portfolio Experience team with world-class operators across a variety of functions including the former Chief People Officer at Netflix for developing an engineer hiring strategy, the Chief Marketing Officer at Stripe to optimize your sales process, and the former Editor-At-Large at TechCrunch to help you convert the value you deliver into a persuasive story. Our approach was built around providing value to founders, leading to our net promoter score of 85 among founders, with 85% saying we are the most valuable investor on their cap table.

We are investors, and we don’t have a magic crystal ball; we are open to being proven wrong and we keep an open mind as we’re meeting founders. If you are building in the LLM space, come talk with us—we will share our full research, and we’d like to learn more about what you’re building and earn the option to be at the table when you’re raising. We’re also co-hosting plenty of AI events at our San Francisco office if you have ideas. Cold emails welcomed at [email protected]

Why Signalfire funded CodaMetrix to fix medical billing

Healthcare is under threat of an enormous labor shortage, caused in large part by an aging workforce. Across the industry—physicians, nurses, administrators—many workers are expected to retire in the next decade, with not enough young people entering the profession to replace them. According to the American Journal of Nursing, 4 million nurses are expected to retire by 2030. Add to that an increasing rate of burnout due to COVID. 

One particularly interesting, relatively unknown healthcare profession that’s also been affected is the medical coder. It’s a niche role—there are approximately 140,000 people in the U.S. and an additional 150,000 globally, in the trade. A coder’s job is to take doctors’ notes and come up with a combination of one or more Current Procedural Terminology (CPT) codes, and one or more ICD-10 codes (International Classification of Diseases, 10th Revision) that are used to express the type of care that was delivered to meet payers’ reimbursement guidelines and requirements. There are literally billions of combinations.

Depending on the type of care provided, this could quickly become a very complex task. For example, if you go to your primary care doctor for a routine appointment, it’s usually a highly structured visit where several standard checks are performed, maybe a blood test is ordered, and you’re good to go. But let’s think about something like an inpatient visit: you’re staying in a hospital for a week because you just had surgery; you have multiple, varied interactions with your surgeon, the anesthesiologist, other specialists, and multiple nurses; all over the course of those seven days. The work to record this activity accurately is enormous. The “doctors’ notes” from your weeklong stay can become the length of a term paper. Coders then have to extract the key information to generate dozens and dozens of billing codes, or worse yet in some cases the physicians have to do that themselves, which takes them away from patient care.

Across the U.S., health systems spend a collective total of $9 billion annually on medical coding. And despite the large spend, the current state of coding is challenging—it’s highly manual, and so errors happen— in fact the recent Becker’s Hospital CFO Report shows 42% of denials are due to coding inaccuracies.  Currently, miscoding can lead to costly back-and-forth communications between Providers and Payers, and undercoding can lead to Providers being underpaid for necessary services they delivered to patients.

Medical coding should be largely automated—enter “Applied AI”

Given SignalFire’s strength in data and analytics, AI, and specifically machine learning (ML) and natural language processing (NLP), we were particularly excited about how automation can play an outsized role in the medical coding space for four key reasons:

1. Coders are quickly aging out of the workforce. The average medical coder is approximately 50 years old, and fewer young people have been entering the field. Providers must find a way to scale their coding capacity, while making their existing employees more productive.
2. Coders are humans and humans make mistakes. We’ve all heard hospitals complain about Payers denying some portion of their claims. On the other hand, Payers complain that hospitals try to squeeze more reimbursement from them by billing for more services than were performed—a concept called “upcoding.” In reality, both sides are right, but very rarely do claims get denied because of fraud (intentional upcoding). Instead, the problem is usually human error. In the weeklong inpatient visit example above, it wouldn’t be unusual for a coder to make mistakes and input incorrect codes. Also, the list of codes changes and new codes are released annually or quarterly. Coders need training to keep up with the latest and they have to keep up with their CEU (Continuing Education Units) to maintain their certifications.
3. ML and NLP technology advancements have now made coding automation feasible with high accuracy. Natural language processing technology was not mature enough until recently to tackle challenges for automated coding. Many of these NLP systems required hand-annotated training data, which had poorer quality results. It wasn’t until 2013 that there was a big breakthrough in deep-learning–based methods (starting with a language model called word2vec), which created a step function improvement in accuracy across all NLP tasks. For example, Google Translate was first publicly released in 2006 but was able to improve accuracy by 60% when they switched to a deep learning model in 2016. Furthermore, NLP applied to the medical field saw exponential improvement in 2020 as researchers released deep learning language models focused specifically on medical notes (called BioBERT).
4. More compliant adoption of Electronic Health Record systems (“EHRs”) and highly scalable and affordable cloud computing. US healthcare Providers and Payers alike have made significant investments in electronification of both clinical and billing data. Now, this large corpus of data can be used by fairly reliable, secure, and affordable cloud and AI infrastructure to both clean the data to develop the ground truth data needed to train Machine Learning (ML) models that not only take advantage of more advanced NLP, but also add the pattern recognition and statistical analysis dimension to produce even better results.

Why now?

Between 2014 and 2018 many startups began working to apply the more mature NLP systems to medical coding. Fast forward to 2023 and we now finally see a small cohort of autonomous medical coding companies that have achieved product-market fit but are still in the early stages of crossing the chasm to achieve adoption widely. It’s not a surprise that companies have taken several years to get to this point. It requires an enormous amount of time to obtain a large training data set, train and backtest the models until they work, convince the typical hyper-conservative-unwilling-to-try-any-new-technology healthcare organization to use their solution, and finally integrate with all of their existing workflow solutions—all before any revenue is generated. But we believe the market is finally ready for these solutions, because ML and NLP technology advancements have enabled these solutions to actually deliver the results they claim.

Why CodaMetrix?

When we built a market map of all the medical coding automation companies that have product-market fit (defined as having at least a handful of customers and a minimum of several million dollars of revenue) there were only a few that made the list. At SignalFire, we believe the companies that will win over time are those with products built on a true machine learning approach and with teams that have experience working inside US-based health systems. CodaMetrix was the only company with both. 

ML is the right platform approach to this technology problem. We think it’s best positioned to extend across specialities over time. The alternative to using an ML model is a simpler rules-based NLP system—the concept that if a doctor wrote X, Y and Z keywords and phrases, then output would be specific billing (CPT and ICD) codes. However, the pure rules-based NLP systems do not scale well because changes in documentation style and every subsequent speciality product developed will require a new set of rules to be programmed. When a system relies on a large set of rules, that’s when it becomes “brittle” and will require lots of caring and feeding to function properly. Currently, companies in this space begin with one or two specialities (for instance, CodaMetrix started with radiology and pathology, and has now expanded to offer surgery, inpatient coding, etc.). But as the autonomous coding platforms need to handle a larger number of specialties and subspecialties, the ML approach, vs. NLP-centric, promises to have the chops to handle the complexities.    

However, ML platforms are useless without an abundance of accurate training data. Because CodaMetrix was incubated and launched inside Mass General Brigham (MGB)—one of the largest and most innovative health systems in the country—it has had unparalleled access to large volumes of training data on any specialty. The company’s cutting edge deep learning models can quickly “learn” the terminology and coding guidelines (or “patterns”) of a new speciality, enabling a rapid product development cycle. This allows CodaMetrix to get its new, tested products into the hands of customers faster than the competition.

One important, yet subtle, feature of its platform is “code transparency.” Because the company started from day one as a product built for a health system (MGB), the tech team also creatively designed the architecture to solve for a key challenge in the industry—showing the reasoning behind its AI decisions. Most machine learning models are black boxes, but CodaMetrix has a built-in audit trail feature in their product so one can see what information the system specifically used to reach its conclusion. This is incredibly rare, yet valuable, in machine learning solutions and a must-have in healthcare, where Providers are frequently audited by Payers to determine if they are coding appropriately.

CodaMetrix’s ability to build products tailored for health systems is driven by an experienced team who deeply understand the ecosystem. The executive team, which has decades of experience working inside and with health systems, is led by CEO Hamid Tabatabaie who was previously an Entrepreneur-in-Residence at MGB, and formerly the CEO and Founder of LifeIMAGE. 

The early results are impressive. Today, CodaMetrix is a multi-specialty platform that classifies codes across radiology, pathology, surgery, gastroenterology, and inpatient professional coding with customers across 10 health systems and major academic universities, representing 111 hospitals in 40 states, including Mass General Brigham, University of Colorado Medicine, Yale Medicine, and Henry Ford Health Systems. Customers using CodaMetrix have seen significant increase in cost savings and cash acceleration—from 70% reduction in manual labor, 59% reduction in denials, and improved cash collection by as many as 47 days.

Why CodaMetrix and MGB chose SignalFire

SignalFire is honored to be leading CodaMetrix’s $55M Series A alongside Frist Cressey Ventures, Martin Ventures, Yale Medicine, CU Healthcare Innovation Fund, and Mass General Brigham physician organizations. SignalFire’s proprietary AI data platform Beacon—which tracks 495 million employees and 80 million companies—stood out to CodaMetrix’s CEO Hamid Tabatabaie as a demonstration of our depth in AI and data as one of the key reason why he chose us to lead the Series A, amongst many others.

“SignalFire’s mantra of adding value beyond the check was evident from the onset and further pronounced with every step along the process. The impeccable combination of Yuanling Yuan’’s in-depth industry research, Tom Peterson’s continuous involvement, SignalFire’s in-house AI and data expertise, Chris Scoggins’ infectious drive for scale, Chris Farmer’s refreshing healthcare thesis, and the brilliant team of experts in talent, operations, and go-to-market strategists are clear reasons for us to have selected them as the lead investor.’ – Hamid Tabatabaie, CodaMetrix CEO

We are also thrilled to have the support of Mass General Brigham in selecting SignalFire as a long-term partner for CodaMetrix.

“Mass General Brigham is delighted to have SignalFire lead the Series A and attract a strong syndicate. SignalFire brings to CodaMetrix deep AI/ML and go-to-market commercialization expertise that will ensure CodaMetrix continues to build on those capabilities developed at Mass General Brigham and lead the market in autonomous medical coding to help drive down healthcare administrative costs and reduce physician burnout. In the short time that SignalFire has been involved with CodaMetrix, they have already demonstrated how their focus on data and GTM expertise will add value.” — Gaye Bok, Partner of AI and Digital Innovation Fund, Mass General Brigham

The team at CodaMetrix was excited about SignalFire’s unique XIR program, which pairs deeply experienced tech industry leaders from the firm’s Advisor Network with high-potential portfolio companies to help accelerate their growth. Tom Peterson, SignalFire’s XIR who will be very actively involved at the CodaMetrix board level and advising on day-to-day operations, boasts 20+ years of deep healthcare experience as the co-founder and former COO of Evolent Health (NYSE:EVH) and former executive director of the Advisory Board Company (acquired by United Health and Vista Equity Partners for $2.6B). 

“Given the strength of the leadership team and the connection to MGB, I was really impressed with the depth of CodaMetrix’s understanding of health systems and how that was combined with best-in-class technology to develop industry leading coding automation products. CodaMetrix is the only coding automation company out there that is built for the health system ecosystem by people who deeply understand it.” — Tom Peterson, SignalFire XIR and co-founder and former COO of Evolent Health (NYSE:EVH)

We are incredibly excited to see how the medical coding automation space evolves over the next decade. This solution is coming just in time, as hiring managers in the industry are likely to face a steep challenge replacing a soon-to-be retiring workforce. With investments like CodaMetrix, we see a future where healthcare Providers can save on labor costs, ensure proper billing for services rendered and achieve faster pay reimbursement thanks to a much lower rate of claim denials. All of these will help our overburdened health systems to remain financially solvent as they try to keep up with the rapidly increasing needs of U.S. patients.

 

---

 

*SignalFire may engage Affiliate Advisors, Retained Advisors, and other consultants as listed above to provide their expertise on a formal or ad hoc basis. They are not employed by SignalFire and do not provide investment advisory services to clients on behalf of SignalFire. For more information on their specific roles, please contact us. Portfolio Company Endorsements: Certain portfolio company founders listed above have not received any compensation for this feedback and did not invest in a SignalFire fund. Please refer to our website for additional disclosures.