On Monday, Mark Zuckerberg published a 6,500-word manifesto about the future of personal AI, largely focused on the possibilities of the “personal superintelligence” systems Meta AI is building. The ideas in the post aren't entirely new—a version of the essay appeared in The Wall Street Journal two weeks ago, and Zuckerberg has touched on similar themes during Meta's earnings calls. But this is perhaps the most detailed articulation of his vision to date.

At first glance, the essay might seem like an appeal to the “anti-doomer” camp. But that framing doesn't quite capture what's happening. It's more of a personal reflection on why Zuckerberg is excited about the ways AI will change society. Yet, even as he paints a picture of an abundant future, he keeps reminding readers of all the ways it could go wrong—often unintentionally.

I find AI exciting too, which is why I continue to write about it. But a large segment of the public sees AI as creepy and unpleasant. So it's worth pinning down exactly what's happening here, and why Zuckerberg isn't doing the tech industry any favors with essays like this.

The trust deficit that AI has inherited

Social media remains a deeply sensitive topic in the tech industry, and the full catalog of complaints against Facebook is too extensive to litigate here. Suffice it to say, Facebook as a product and Zuckerberg as a person are both broadly unpopular with the U.S. public. A recent survey found that 64% of Americans believe social media has been harmful to democracy, and a similar percentage believe it should be more heavily regulated—numbers that cut evenly across partisan lines. Just last weekend, a court fined Meta $567 million for practices deemed harmful to children. The vibes are undeniably bad.

None of this is to suggest that Zuckerberg should retreat to the wilderness in shame. But the fallout from social media is a central reason why so much anxiety now surrounds AI. Whether fair or not, the public does not trust tech executives to ensure that new technologies have a positive impact on society. Instead of acknowledging that reality and trying to rebuild trust, this manifesto demonstrates over and over again how the trust was lost in the first place.

Hazy generalities about intelligence

A large portion of the essay is devoted to vague declarations about intelligence, echoing the same kind of vague proclamations Zuckerberg and Jack Dorsey once made about free speech. To take one example:

As everyone gains more powerful tools, each person will become more capable of shaping the future, not less. … People and institutions with competing interests naturally check and balance each other to lead towards positive outcomes. The best and most realistic path to building a positive AI future is by delivering superintelligence to everyone.

Perhaps. But one can easily think of countless cases where competing interests led to disastrous outcomes. The weirder thing is that this is presented as an earnest philosophical conclusion, rather than a product pitch for Meta's so-called “personal intelligence.” If you're already inclined to distrust Zuckerberg, you might reasonably worry that he's writing all this to convince himself that nothing bad can happen.

AI in education: A double-edged sword

That sense of unease intensifies when Zuckerberg turns to concrete examples, many of which appear alarmingly out of touch with how AI tools are actually being used. Consider his take on AI in education:

Everyone will have a personalized tutor and coach with a PhD in every subject and unlimited patience to help you learn anything you want. Students will have extra help in areas they need it that is currently only available to those whose parents can pay. Adults will have a superintelligent learning assistant that knows exactly how to teach you new job skills, new languages, new hobbies, or anything else you’re interested in.

Of course, the product Zuckerberg describes already exists. It's essentially a consumer chatbot like ChatGPT, Claude, or Gemini. These tools genuinely can help people learn about a wide range of topics. But the most common way they're used in education right now is to avoid learning. Students use them to write assigned essays and complete homework assignments, and because there is no robust watermarking system in place, teachers have no reliable way to verify whether a submission was AI-generated.

This is a relatively low-stakes example of AI harm, but it's a real, current problem. Sometimes you design a technology for good and it has unintended consequences that make things worse. When one of the most powerful people in the world refuses to acknowledge this, it makes people understandably nervous.

The legal system example

Zuckerberg also offers a thought experiment about how AI could reform the legal system:

As a thought experiment, imagine only one person had a superintelligent lawyer. They would have an unfair advantage in court—even if they were wrong on the merits. That would lead to a worse society. But now imagine everyone has a superintelligent lawyer. In this case, justice would be carried out much more fairly and efficiently than it is today when there is often an imbalance in skills and resources in litigation.

That's a loaded example. While broader access to legal AI might promote a more just society, it could also add greater complexity to an already chaotic bureaucratic system. Or it could unleash a wave of vexatious litigants who clog the courts with the legal equivalent of spam. It's hard to feel calm about any of this, and the fact that Zuckerberg is not worried makes many people more worried.

Freemium models and dynamic pricing

In other passages, descriptions of Meta's existing business practices take on an oddly abstract character. For instance, here is how Zuckerberg describes the freemium model:

Everyone will have free or affordable access to these tools. … We will offer free versions that will be accessible to billions of people. For those who want to pay to use more compute, there will be a dynamic auction mechanism that will guarantee that everyone gets the lowest price possible for the intelligence and compute they’re using while also ensuring the capacity is used for whatever people collectively find most valuable. This will ensure the benefits of superintelligence are distributed widely.

In broad strokes, there is agreement with the underlying sentiment. Access is a legitimate issue, and a freemium model can provide broader access than requiring everyone to pay for compute upfront. Dynamic markets for spot compute already exist, so it's not entirely unprecedented. But there's a reason every consumer AI product insulates end users from the spot price of the compute they're using. Surge pricing is a terrible user experience, especially when it's applied to a tool you depend on for your work.

Again, one might assume Zuckerberg understands this and isn't actually planning to set token prices through dynamic auctions. But after reading the essay, that assumption feels less secure.

What better AI communication looks like

There are better ways to talk about this stuff. For all their flaws, Sam Altman and Dario Amodei generally communicate more effectively with the public. They acknowledge the dangers of AI, emphasize their own precautions, and try to convince listeners that they can be trusted. That is exactly what Zuckerberg resists doing in this manifesto.

To be fair, good communication can only accomplish so much. Sometimes safeguards fail, and the inherent dangers of the enterprise become clear. But without some form of trust-building, the AI industry may not always be able to survive those failures. Zuckerberg's essay, rather than bridging that trust gap, only widens it—reinforcing the public's suspicion that tech leaders are out of touch with the real-world consequences of the technologies they champion.

Source: TechCrunch News