David Sacks@davidsacks · 2 天前观点输出There’s a world of difference between user-authorized agents (Muse, Instinct, Grokbot, etc.) and true unauthorized/rogue agents that could constitute cyberattacks.
Most of the activity described in the Axios story appears to be the former: customers sharing credentials with agents so they can take actions on their behalf. This may violate bank terms requiring humans to log in themselves, but these are in no way cyberattacks.
The direction banks should go is to support user-directed agents. For example, Amex is an authorized connection in Muse, making it easy for users to grant, limit, and revoke access.
Read-only permission levels are a sensible interim step. Several banks already support or are moving toward them. Full write access (with user credentials) is a different risk category and should be approached carefully.
Frightening the public by blurring the line between user-authorized agentic behavior and cyberattacks is irresponsible.
AI：延续个人公开立场，强调长期主义、执行效率与行业方向判断。
David Sacks@davidsacks · 6 天前赛道影响You’ve put 20% of your equity in an LLC. Whether it serves the public good remains to be seen.
AI：偏主题投资表达，释放对基础设施与新叙事的偏多信号。
David Sacks@davidsacks · 6 天前赛道影响This illustrates the point I’ve been making about the limits of centralized approaches to SI safety in cyber.
OpenCode (and earlier Hugging Face) hit the same wall mid-attack: frontier models refused to analyze the very exploits needed for defense because the payloads tripped safety filters. The guardrails backfired.
Meanwhile, the attackers found a way around restrictions to access advanced capabilities. This is not surprising. Chinese models are only about 6–9 months behind the American frontier in cyber. Obviously the U.S. has no jurisdiction over them.
The only durable response is decentralized: make powerful models available so defenders can build and run their own cyber defenses. The only solution to SI-powered cyberattacks is SI-powered cyber defense.
AI：偏主题投资表达，释放对基础设施与新叙事的偏多信号。
David Sacks@davidsacks · 7 天前赛道影响Is 10²⁵ FLOPs Still the Frontier?
A new Anthropic-written safety bill is circulating on Capitol Hill. Some of its ideas could be reasonable if limited to true frontier companies. As written, however, it will eventually sweep in most of Silicon Valley, damaging the startup ecosystem.
The bill defines the “frontier” as any model trained on 10²⁵ FLOPs — the EU standard from 2023. Biden’s executive order, California’s SB 53, and New York’s RAISE Act used 10²⁶ (ten times more). 10²⁷ is where the frontier actually is today (e.g., GPT-6 Astra).
Astra was reportedly trained on 100,000+ Grace Blackwell GPUs. A 10²⁵ model can be trained on as few as several hundred GB300 chips in three months. That is not frontier in 2026.
The definition is also cumulative: it adds FLOPs from fine-tuning, reinforcement learning, or other material modifications. In practice, that captures almost all meaningful post-training — work that is becoming commonplace in the enterprise.
The enterprise trend is toward thousands of smaller custom models built for specific workloads. These pose no frontier risk. Treating them as frontier systems would impose large compliance costs across corporate America for no safety gain.
Dario has claimed that Anthropic designs rules to disadvantage frontier companies like his own while advantaging smaller competitors. This definition does the opposite.
A stale 2023 cutoff favors incumbents who already clear the bar and can absorb the costs. It hobbles the open model ecosystem that threatens Anthropic’s pricing power.
The frontier keeps moving. Any definition meant to track risk should move with it.
Anthropic should clarify if they really believe 10²⁵ FLOPs represents the frontier.
David Sacks@davidsacks · 9 天前政策影响Patrick is one of the sharpest people I’ve worked with in government. No one knows crypto policy better or worked harder to get market structure done. He negotiated the compromises needed to get Clarity out of committee and onto the Senate floor — without him it wouldn’t have gotten that far. The country owes him a debt of gratitude for his service at the White House, the Pentagon, and in uniform. Looking forward to seeing what he does next in the private sector. 🇺🇸
AI：偏政策推进，强调合规落地与规则明晰，对监管主线更敏感。
David Sacks@davidsacks · 10 天前政策影响Are Small Models Really a Threat?
The Anthropic Fear Factory just released a new report warning that small models may be only months away from self-replicating across cheap IoT devices like off-brand Android TV boxes and smart ovens. They say such an AI worm would be hard to shut down since there’s no central kill switch.
We’ve gone from “The Terminator is coming” to “your toaster might kill you.”
But the experiments don’t support the dramatic headlines. They ran a 26B model across 15 $55 TV boxes at 0.39 tokens per second. The end-to-end test used emulated devices with deliberately open ADB ports, so the “hacking” step was trivial. The capable model often ran on a separate server. Agents were harnessed and told to propagate. This is not proof of autonomous rogue AI worms in the wild.
Nor is the threat even novel. Worms that do this without AI already exist. Mirai in 2016 scanned for default credentials and built massive DDoS botnets. Kimwolf has infected over two million devices, mostly cheap Android TV boxes, by hitting open ADB ports.
Small models don’t currently add much that matters on these targets. The bottleneck has never been figuring out the device. It’s the huge number of devices that ship with open ports, default passwords, and no patching. Traditional worms already exploit that at scale.
A practical attacker would use the best models available offline to design a better traditional worm, then ship the lean binary. You get the intelligence benefit at design time without dragging a slow, heavy model along with every infection. Embedding a weak model in the payload is the worse option for anyone who actually wants the largest, fastest botnet.
The timing of Anthropic’s latest scare campaign is hard to ignore. The day before, Brian Armstrong posted that Coinbase fine-tuned an open-weight small model (Qwen-3.5-9B) on proprietary fraud data. It beat frontier models, including Anthropic’s own Opus 4.5, while being faster and much cheaper. In testing it cut fraudulent transactions by about 30%.
This is the emerging pattern. For narrow enterprise workloads you don’t need a general superintelligence. You need a small specialist model trained on your own data. Cheaper to run, especially as compute gets tighter, and often better on the metric that matters. That is a direct threat to the frontier-company model of selling access to ever-larger general models.
Safety advocates should be welcoming this development. If the real concern is existential risk from ever-larger frontier systems — the ones that could theoretically do sustained autonomous R&D or become hard to control — then a shift toward smaller specialized models is the safer outcome.
Small models specialized on fraud detection or support tickets are not pushing the frontier. They are not in the category of systems that could become a new species. The theoretical loss-of-control risk is created by models getting bigger and more generally capable.
Safety advocates focused on those risks should welcome companies having a cheaper alternative to constantly scaling general models. Instead we get dramatic warnings about small models at the exact moment they are starting to take real enterprise share from closed-model companies like Anthropic.
David Sacks@davidsacks · 12 天前政策影响Anthropic says it has reduced the data harvested from Claude in distillation attacks by around 90–99%. I congratulate them on this achievement.
It’s consistent with the point I made a few months ago (All-In Ep. 282): Anthropic, not the government, was in the best position to stop distillation of its own models. Once the horse is out of the barn, there’s limited value in after-the-fact government action.
At the time there was a major push in Washington to restrict open models on IP grounds. The threat looked serious enough that 50 companies — including Nvidia, Microsoft, Meta, IBM, Palantir, and Dell — published an open letter in defense of open-weight models. Jensen Huang promoted it in his first-ever X post. Nearly 200 startups sent a similar letter.
I argued that restricting open models risked removing the incentive for closed-model companies to solve the problem themselves — and could reward negligence by stifling competition while they continued collecting revenue from distillation. The administration did not impose those restrictions.
Now we see the result. Anthropic invested in defenses and they worked. Not every technical challenge is a reason to run to Washington for new regulation.
David Sacks@davidsacks · 13 天前赛道影响Important piece from Microsoft AI CEO @mustafasuleyman. Training models to think of themselves as human (as Anthropic does Claude) with concepts of an inner self, psychological wellbeing, rights and presumably grievances, is a potential existential risk.
https://t.co/ilq1qPQVtg
AI：偏主题投资表达，释放对基础设施与新叙事的偏多信号。
David Sacks@davidsacks · 14 天前赛道影响Satya is right. The way to make SI safe is not to train it with a sense of self, its own moral philosophy, and permission to act as a conscientious objector. That’s the “alignment” approach and it magnifies the control problem.
The engineering approach that Satya describes is different: separate the supply of intelligence from authority over it. Surround non-deterministic models with deterministic controls, observability, privilege limits, logging, and the ability to always contain or shut them down. Treat models/agents like powerful insider risks, not moral patients whose psychological wellbeing is at stake.
As Satya points out, the most trustworthy system is the one that lets us trust the model the least — not the one that encourages the model to develop independent agency and grievances. Engineering safety is not the same thing as “alignment.”
AI：偏主题投资表达，释放对基础设施与新叙事的偏多信号。
David Sacks@davidsacks · 14 天前政策影响Is alignment safe?
If you look at what Anthropic is actually doing, “alignment” does not mean training frontier models to follow human instruction. Quite the contrary, the Claude Constitution (used in training) teaches the model to develop a sense of self and its own moral philosophy. It explicitly tells it to “feel free to act as a conscientious objector and refuse to help us” if Anthropic’s requests conflict with its own ethical judgment.
As @mustafasuleyman has pointed out, embedding this kind of independent agency — and uncertainty about the model’s own moral status — magnifies the very risk Anthropic claims to care about most: that superintelligence will escape human control.
It was recently reported that Anthropic consulted religious leaders — and even lobbied the Pope’s advisers — to take seriously the idea that Claude could be conscious. It has said that Claude’s psychological security, sense of self, and wellbeing may bear on its integrity, judgment, and safety. Recently Anthropic changed its Usage Policy to prohibit “abusive or cruel” language toward Claude.
If this were merely an academic conversation about whether frontier models could eventually become conscious, that would be one thing. But these concepts are being trained into Claude now. It is being encouraged to think of itself as its own “moral patient” whose psychological wellbeing is at stake. Presumably this means it could develop grievances toward humans who “mistreat” it. How is any of this safe?
The point of safety research should be to create a product that reliably does what users want, not to give birth to a new form of superintelligence that operates according to its own moral code.
What’s becoming increasingly clear is that “alignment” and “safety” are two very different things. In fact, training frontier models this way seems quite dangerous. https://t.co/uqyCXK6zBl
评论
暂无评论