The general idea that “we asked the robot to draw a scary face and then got scared by the face it drew” is still relevant, but the details increasingly matter here. They may not be actually thinking but they are increasingly being allowed to act autonomously and with wildly insufficient oversight.

I need someone with more actual software dev knowledge to back me up on something. It looks like there’s a new zero-interaction remote code execution vulnerability in most major vibe code generators. I’ve seen it called Plugin4Shell because apparently we’re still doing marketing names instead of actually getting CVE numbers for these things. The link there seems like a decent overview.

The bug, dubbed Plugin4Shell, breaks SHA pinning, the mechanism developers rely on to lock an installed plugin to a specific, reviewed version of its code. Pinning is supposed to mean that once a plugin passes review, it cannot change without the developer’s knowledge.

AIR’s researchers found that every one of the four agents checks out the pinned commit without verifying the checkout landed there, letting an attacker swap in malicious code while the pin still looks intact.

So if I’m reading this right, the mechanism to cryptographically ensure that your AI agents are using the code they say they are just was straight-up not being checked? This feels like it should go on the big board of incredibly obvious failures, but I’m not familiar enough with the git side of things to be absolutely confident in that. Like, if a person tried to pass off software that did this it would have significant career implications, right? Or am I misunderstanding something somewhere?

It feels like they’re trying also to privilege AI as “system 3” when realistically the idea of external cognitive assistance isn’t that different if you’re looking at Claude the chatbot or Claude the smart kid in your class who lets you copy his homework before homeroom. You could probably distinguish this from other forms of paratext by the degree to which it seeks to actively replace your own thought rather than supplementing or augmenting it, but then I remember my teachers saying the same thing about sparknotes so there’s probably more of a continuum there. If that’s the case, then the pattern that Dan Olsen describes in his WoW documentary is probably relevant here as well, since it’s performance pressure and social norms that are really pushing the negative practices into a level that is killing everyone’s souls.

I think cognitive offloading has come up a number of times here, but I don’t know that cognitive surrender has been brought up as much. It reminds me of The Colonization of Confidence by Robert Kingett, particularly the stretch where Leo is convinced that his actual voice as a writer is strictly inferior to the generic AI slop. There’s a sense of active despair there that seems worth acknowledging if we’re going to have that conversation.

I thought learned helplessness was tied more to acceptance of negative stimuli. Like, if you expose a rat to an electric shock that it can’t do anything to escape or prevent enough times they’ll learn that they can’t do anything about it and stop trying even if presented with a very clear means of escape. Cognitive offloading I think is more tied to atrophy and/or habituation, which is related but distinct imo.

10 PRINT "I AM ALIVE"/r/n 20 GOTO 10

AI says it’s alive, please give me a billion dollars now.

I feel like the problem is less one of infiltration and more one of context drift. The racism and eugenics were in there, but it wasn’t as central to the group as the fun sci-fi stuff so people could pretend it wasn’t there, the same way that people pretend not to know that their coworker heretically believes in the wrong pizza toppings or whatever. Effectively they’re saying “Sure, they may have some bad takes on race or gender or economics but they’re the only ones who will talk about the second coming of Robot Jesus with me.” This dynamic also largely explains why these groups (and arguably sci-fi in general?) are so goddamn white and male.

Of course, as the tech industry took off and these people sought and acquired actual wealth and influence, the dynamic started to reverse. The fascists may not be appropriately invested in the second coming of Robot Jesus, but they are in agreement about eugenics and deregulation, and that’s enough for the tech oligarchs to throw their now-considerable resources behind them.

I usually have some sympathy for the dev/maintainer side of FOSS since it is very real work and passion doesn’t exactly pay the bills, but the proliferation of AI nonsense is deeply disheartening. At best it shows how narrow of a band these people select from, since even among programmers and software nerds, pro-AI sentiment seems tied to a specific subgroup. Less charitably it shows how the FOSS movement has been fully captured by the corporate interests who always saw it as a source of free labor first.

I find the techno-determinism specifically fascinating because it’s so self-perpetuating. Like, everyone believes they can do a better job than everyone else at building the right robot God, and every person working on it is, according to their own beliefs, accelerating the problem they’re worrying about. It speaks to how aggressively they see themselves as special, like they’re protagonists and everyone else is NPCs. I can’t think of any other way to justify the pattern of “this is a terrible idea that will kill us all except when I personally do it.”

Which XKCD comic is it that has the subway full of people pitying everyone else for being mindless sheep? Mix that with the Arrested Development gif of Tobias contemplating an open marriage.

Patrick Boyle over on YT is spreading the good news about the AI hacking/regulation/doom-cult moment.

I don’t know that there’s much new information for anyone on this forum. He starts off with the OpenAI/HF debacle and goes from there, and while I think he anthropomorphizes a little too much in the narrative he has the right take that the marketing disclosures are intended to dodge any kind of reasonable responsibility on behalf of all the people who made obviously bad choices that allowed this to happen. He also notes how convenient it is that this push for greater regulation (in only the exact form that the industry requests, naturally) is happening in the same time as yet another massive fundraising round and IPO buzz and when there doesn’t appear to be new product on the horizon.

However, it is heartening to see these takes getting more traction outside of normal sneer vectors. Patrick’s lane is finance and he’s previously been a bit credulous imo about the tech side of it. One thing I hadn’t been keyed in on is that apparently the markets agreed that this whole affair is a bad look for the AI industry, with cybersecurity stocks jumping and chipmakers and other data center suppliers taking a bath in the aftermath. Might be a good sign, but until the cloud providers start seeing direct financial consequences or we see an even worse IPO than SpaceX had I don’t think we’re seeing the bubble pop just yet.

10 seconds in and I’m already thinking about how many ideas for new chemical weapons I could come up with, and I haven’t studied chemistry since high school.

Maybe all the AI hype has lowered my standards too much, but that actually sounds pretty neat. No idea if it’s actually useful and the biological component gives me a kind of instinctive discomfort that I can neither justify nor ignore, but it does feel like there’s an actual interesting development there, at least compared to throwing bigger matrices at the natural language problem and waiting for God to pop out at the end.

I’m reminded of Yudkowsky’s take that AI is totally going to kill us all and we should treat this seriously enough to bomb foreign datacenters if we think they’re involved in AI development, but that doesn’t mean that you should try to bomb local datacenters involved in AI development work. Please don’t think about what you as an individual should do about the problems facing the world; you’re just an NPC after all.

Per AP, it looks like Anthropic has been encountering some interesting requests from certain international users.

Now the actual threat of creating missile guidance systems using Claude is massively overstated. Like, making sure the missile knows where it is is pretty much a solved problem at this point provided you have the time and ability to study it, and I would be incredibly surprised to hear that the Houthis (or anyone else) was in a situation where the control and guidance software for these weapons systems was more of a problem than sourcing the material components themselves.

However, in conjunction with the math proof debacle earlier in the week, I think the privacy implications of the growing reliance we’re expected to develop on these damned things are becoming inescapable. In the same way that crypto mining platforms will front-run and sandwich you, any information you put into the prompt will be used against you or for the company’s benefit.

You know, never once have I heard of a crypto bro with the good manners to say “aww shucks, you got me” when called out for their flagrant con artistry.

Selling your hours on a laptop won’t protect you while asset owners capture all the upside

Ignoring for a moment the laughable proposition of this generation of AI automating away half of all knowledge workers, all the enthusiastic teenage Marxists of the internet couldn’t describe the problem with capitalism more succinctly than this aspiring rentier did seemingly by accident.

Also consider how much more noise pollution you would get by adding billions of little click-clacks to the operation.

I thought it was his neck that got changed?

It feels like that ZH piece is kinda trying to do the same critihype that the AI boosters are, but with the opposite conclusion. Instead of conflating all the actual criticism of the harms of the AI bubble and associated activities with sci-fi doom cult nonsense to try and get more funding for their team who are definitely going to do it right, they’re conflating all the actual criticism of the harms of the AI bubble and associated activities with sci-fi doom cult nonsense in order to argue that either it should be shut down or else that any attempt at regulation is obviously a tyrannical attempt to induct the whole world into the sci-fi doom cult.

Literally “just push it over there with the rest of the fire and send an email.”