The great LLM debate: product or infrastructure?

• Will LLMs mature into a pure-play product or evolve as infrastructure? The difference is profound for the development of the technology.
• Apple recently planted a flag in the infrastructure camp during their WWDC presentation, leveraging LLMs as tools to deliver a better user experience.
• At ModuleQ, we are in the Apple camp, viewing personalization and context as crucial ingredients for AI’s ability to drive the next generation of enterprise work fruitfully.
• Context is a crucial element in productively applying AI to enterprise workflows.
• Our UnpromptedAI solution uses various technological tools to deliver contextually rich information, a current limitation of LLMs.

There is still an open debate of the future direction of LLMs. Will they mature into a product or will they move mature as infrastructure? This fork in the road has profound implications for the future of how we interact with technology.

The product direction is akin to ChatGPT. The user navigates to the native home of the LLM and processes their workload through its prompt. Nearly 1 in 4 adults have tried ChatGPT, as of 1Q24, according to a Pew Research survey. The survey also points to a significant amount of “churn” across those users, as many do not come back. This suggests LLMs as a product need to refine their product-market fit.

The infrastructure direction is more like coding CoPilot, where the user is coding in their native terminal and the LLM provides support to their existing workflow. According to GitHub’s CEO Thomas Domke, there were over 1 million paying users of GitHub CoPilot as of 1Q24. The Daily Active Usage is presumably very high, given it works natively in one’s coding application. While the scale is different from casual LLM queries (admittedly a smaller subset of internet users), the stickiness or the offering is glaring.

The difference between the two functions sounds subtle, but it has meaningful implications for the development of AI. For a prompt to absorb workflows and tasks, it needs to broaden its abilities while also changing the user’s habit. Think about a Python coder switching to their web browser, prompting a code query, copying the output, pasting it in their notebook and testing the output, versus the native autocomplete of Github CoPilot. That additional friction means the service must be overwhelmingly better to change the worker’s habit.

This doesn’t mean it is impossible, especially when dealing with a new technology. LLMs might well be overwhelmingly better at many things, driving users to them to do all manner of tasks. Over 20 years ago, Google effectively achieved this, shaping the way we access the internet. As its search got better, users got more engrained in using it to navigate (even learning how to powerfully prompt it), to the point where the internet started to be shaped by Google. As of today, LLMs have key limitations that make this a challenge. However, the major architecture players (OpenAI, Anthropic, Mistral, and others) suggest that they are working on these deficiencies and are developing a meta-product that will be able to subsume these native workflows.

Apple’s Announcement: User-Centric Focus

For a while, across the major tech companies, Apple has been relatively quiet on the Generative AI front. That was until their introduction of Apple Intelligence at the June 2024 WWDC. This moniker is Apple’s umbrella solution for integrating AI technology across its suite of hardware products and solutions. As Benedict Evans, a prominent technology analyst and former partner at technology venture capitalist firm Andreessen Horowitz, wonderfully explained, Apple is firmly in the LLM as an infrastructure camp for their billion-plus users. In Evans’ words, they are “treating this as a technology to enable new classes of features and capabilities, where there is design and product management shaping what the technology does and what the user sees, not as an oracle that you ask for things.”

This takeaway is somewhat different from the headline the news media picked up, which was billed as a partnership between Apple and OpenAI, bringing ChatGPT to iPhones. But as Evans carefully infers from Apple’s preliminary details, the phone maker will likely be utilizing a variety of LLMs for different use cases, while also carving out specific queries that go directly to ChatGPT when the user’s persona or profile are not important for satisfying the request. Or as Evans puts it, “Apple is doing something slightly different - it’s proposing a single context model for everything you do on your phone, and powering features from that, rather than adding disconnected LLM-powered features at disconnected points across the company.”

This is the key distinction that the news media highlights when talking about the partnership. If you are drafting an email to your friend, Apple will use a different LLM approach toward enhancing the drafting process, versus a general request about an ingredient list, which will be pushed to ChatGPT (the premier tool for such requests). The difference is all about personalized workflows and tailoring different AI tools for those workflows.

Why We Think the Apple Approach is The Right Approach for Enterprise AI

Since ModuleQ’s founding, our goal has been to enrich the knowledge worker with personalized AI tools, allowing them to be better versions of themselves. Our calling card is delivering the right information to the right person at the right time. Those three variables (information, individual, and time) are driven through personalization. They align with Apple’s perspective on applying AI tools in a way that enhances the personalized human experience of interacting with technology.

In our view, context is a crucial element in applying AI to workflows. It is the key to delivering the right output that improves productivity and efficacy. We develop context by understanding the user, not forcing them to prompt the relevant information. As a result, our AI doesn’t disrupt existing workflows. We believe that the application of LLMs in the future of knowledge work must be tailored to their strengths, while also being aware of their weaknesses. In this worldview, LLMs are a notch in the quiver, not the entire kit and kaboodle.

Our Unprompted AI solution starts with the user’s persona: their role, seniority, and team. Our technology then integrates into the worker’s personal context (with things like their calendar, email, and CRM), enriching their knowledge work by delivering relevant information directly into the communications hub. This delivers increased productivity and output while avoiding task switching. The very term Unprompted represents our firm belief that users shouldn’t prompt everything they need to get their jobs done. Instead, the AI should understand their circumstance and deliver to them the right information. From this perspective, the AI starts from the user’s workflow and works backward, rather than shoehorning all workflows into an omniscient prompt.

The way LLMs fit into this equation is by processing different types of information within the worker’s existing flow. For example, an LLM is great at summarizing key points from a lengthy research report that a busy investment banker may not have time to read. It can be tasked with that when the banker needs a summary of such a report. It is less capable of surfacing subject-matter-specific facts with dead accuracy, as is so often required in financial services. For these workflows, different tools should be leveraged. And so, our approach has been to source the right tool for the right job in delivering the relevant information to the right person.

An Optimistic Future of Enterprise Work

Returning to Evans’ great piece, the question the entire sector is grappling with is the ceiling of LLMs as general-purpose technologies. If it is unlimited, will they simply be able to absorb all user-specific workflows through general omniscience (as Evans calls it: ‘software is dead’), or will they hit a plateau, thus needing to be calibrated differently for specific needs and use cases?

If the latter, then the Apple approach makes sense. While it is too soon to tell for sure, we are firmly in this camp. This is in part driven by our expertise and perspective on the AI landscape, but it is also due to our firm belief that AI can be a tool for human flourishing. It can do this by unlocking human potential, making humans happier and more productive workers. Just as we hope Apple’s effort will deliver this to the consumer, ModuleQ’s UnpromptedAI is leading the way across the enterprise.