When the GPT Store launched a few months ago, it seemed like we were witnessing the birth of a new software paradigm. Instead of traditional apps built with code, we’d have “prompts” — carefully crafted instructions that could transform a general-purpose AI into a specialized tool. The promise was enticing: anyone with a knack for prompt engineering could become the next App Store millionaire, without needing to learn complex programming languages or manage server infrastructure.

This idea wasn’t entirely new. We’ve seen similar shifts before. But the scale and potential of AI-driven apps seemed to dwarf previous revolutions. Here was a way to create sophisticated, intelligent applications simply by describing what you wanted in natural language.

But as often happens in technology, what appears to be a breakthrough can fizzle rapidly. I don’t really use the GPT store and no one else does either. Despite all the hype, it’s just not that talked about any more, and it’s not where big innovations in AI are getting launched (instead, these are rolling out on Arxiv or on platforms like Replicate).

And now, to put an exclamation mark on things, a new research paper has demonstrated a technique called “output2prompt” that can reverse-engineer these supposedly secret GPT prompts just by looking at the GPT’s output. Now, someone could reconstruct your app’s source code just by using it a few times.

There’s no more GPT store moat. But this also has a few other bigger implications beyond OpenAI’s shop:

1. Business Model Vulnerability: It challenges the entire business model of selling prompts. If your secret sauce can be easily replicated, what exactly are customers paying for? This is reminiscent of early concerns about JavaScript-based web apps, where the code was visible to anyone who knew how to open the browser’s developer tools.
2. AI vs Traditional Software: I think this highlights how fundamentally different AI is from traditional software. In a normal app, the logic is hidden away in compiled code, running on a server you control. But with language models, the logic is, in a sense, visible in every interaction. This transparency is both a strength and a weakness.
3. Intellectual Property Concerns: It raises complex questions about intellectual property in the age of AI. Can you really claim ownership of a prompt if it can be trivially extracted? This could have far-reaching implications for patents and trade secrets in the AI space.
4. Security Implications: If system prompts can be extracted, it potentially opens up new avenues for attackers to understand and exploit AI systems. This could be particularly bad for AI applications in sensitive domains like finance or healthcare.

In this post, I want to explore some of the downstream impacts of all of these. But first, I want to take a second and explore the output2prompt paper, which shows how you can extract the prompts from the outputs of the GPTs in the GPT store.

How output2prompt works

The core idea behind output2prompt is clever in its simplicity. By analyzing patterns in the AI’s responses, another AI can infer the instructions that produced those responses.

The researchers behind output2prompt employed a transformer encoder-decoder model with a sparse encoder. This architectural choice allows for efficient processing of multiple LLM outputs simultaneously, reducing both memory requirements and computational complexity. The model is trained on concatenated sets of LLM outputs to predict the corresponding original prompts.

What’s particularly impressive (or concerning, depending on your perspective) is how well this technique works. The researchers found they could reconstruct prompts with high accuracy, often needing fewer samples than you might expect. In their experiments, output2prompt outperformed previous extraction methods, requiring fewer samples and training epochs to achieve accurate results.

Perhaps even more striking is the method’s transferability. It works across different language models and types of prompts, with only minimal loss in performance when applied to new contexts. This suggests that the patterns it’s identifying are fundamental to how these models process and respond to prompts, rather than being quirks of a particular implementation.

Of course, there are limitations. Very complex prompts or those with specific examples are harder to extract precisely. And the quality of the extraction depends on having a diverse set of outputs to analyze. But these feel like speedbumps rather than roadblocks. Given the pace of progress in our field, it’s likely these limitations will be overcome sooner rather than later.

Why does this matter?

So what does this mean for the future of AI applications? If every LLM can have its system prompt extracted, how will that change the industry? I think we’re going to see several shifts:

1. Dynamic Prompting: We might move away from static, fixed prompts towards more dynamic approaches. Perhaps prompts will be generated on the fly, or combined with other techniques to make them harder to extract.
2. Emphasis on User Experience: There will be a greater focus on the overall user experience, rather than just the raw capabilities of the underlying model. The value will be in how seamlessly the AI integrates into a workflow, not just in what it can do.
3. Data as the New Moat: High-quality, proprietary datasets may become even more valuable. If prompts can be easily copied, unique training or RAG data could be the main differentiator.
4. Hybrid Approaches: We might see more systems that combine prompt-based AI with traditional coded components, creating a more robust and harder-to-replicate architecture.
5. New Security Measures: Just as we developed obfuscation techniques for JavaScript, we’ll likely see new methods emerge to protect valuable prompts from extraction.

This situation reminds me of the early days of the web, when people thought the value was in the HTML itself. But it quickly became clear that the real value was in the data, the user experience, and the network effects. Similarly, I suspect we’ll find that the value in AI apps isn’t in the prompt itself, but in the overall design of the system, the quality of the training data, and the ecosystem built around it.

There’s also an interesting parallel to open source software here. In both cases, the core technology is essentially visible to anyone who looks closely enough. The challenge is in creating something valuable on top of that open foundation. Companies have shown that it’s possible to build billion-dollar businesses on open source; perhaps we’ll see similar models emerge in the world of AI.

For founders working in this space, the lesson is pretty clear and maybe even not all that novel: don’t rely on the obscurity of your prompts for your competitive advantage. You’ll never be able to make bank by monetizing a just a single prompt.

Instead, focus on creating a holistic experience that can’t be easily replicated, even if someone knows exactly what prompts you’re using. This might involve:

1. Building a strong brand and user community
2. Continuously improving and updating your prompts based on user feedback
3. Combining AI capabilities with unique datasets or proprietary algorithms
4. Creating seamless integrations with other tools and workflows
5. Providing excellent customer support and education

It’s worth noting that this research also opens up some new opportunities. Tools for analyzing and improving prompts could become valuable in their own right. We might see the emergence of “prompt optimization” as a specialized field, similar to SEO for web content. TBD.

The ethical implications of this technique are also worth considering for a second. On one hand, it could lead to more transparency in AI systems, allowing users to better understand how these tools work. On the other hand, it could be used to copy and exploit others’ work, or to reverse-engineer systems for malicious purposes. As with many advances in AI, it will be crucial to develop ethical guidelines and perhaps (maybe) legal frameworks to govern its use (although I’m personally pretty skeptical of this approach).

Looking further ahead, this might accelerate the shift towards more advanced AI architectures. If simple prompt-based systems prove too vulnerable to extraction, we might see faster adoption of more sophisticated approaches like chained models, multi-agent systems, or AI that can dynamically modify its own prompts.

The upshot

In the end, this research is a reminder of a fundamental truth in technology: nothing stays secret for long. The real challenge isn’t in creating something clever, but in creating something that remains valuable even when all its secrets are known. It’s also a testament to the rapid progress in AI — the fact that we can now use AI to reverse-engineer other AI systems is remarkable in itself.

I’m sure about one thing: the most successful AI applications won’t be those with the cleverest prompts, but those that provide the most value to users in a way that can’t be easily replicated.

Easier said than done.

If you liked this post, check out AImodels.fyi to stay up to date with all the latest AI research.