• 5 Posts
  • 287 Comments
Joined 3 years ago
Cake day: July 19th, 2023
  • nfultz@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 8th November 2026English
    9·
    19 hours ago

    todays adexchanger had a lolworthy note that the binance hack began with a search ad.

    If you hadn’t heard, Binance was totally pwned in a $6 billion crypto heist last week, suspected to be a North Korean operation. Well, Binance has now published what amounts to a preliminary autopsy report of the incident.

    The heist began when a Binance Wallet DevOps team member accidentally downloaded a malicious copy of Homebrew – a popular open-source tool – from a mirror site promoted in a search ad. The malware was spotted, but not before the hackers had obtained the real identities of Wallet DevOps team members.

    Using those identities, the North Korean hacking unit spent months targeting Binance’s wallet team with covert schemes. One involved a prolonged fake job interview that lured a Binance DevOps engineer into downloading an “assignment” onto their work device.

    After taking control of the Binance DevOps engineer’s laptop, the hacker used an unnamed internal AI tool to log in to the devices of other engineers. Access to multiple credentials allowed the heist to go through.

    Binance guy fell for “top link is ads”. COME ON.

    Never understood why my employer would drop $30m on Crowdstrike but not add uBlock Origin to their company chrome profile for free.

  • nfultz@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 1st November 2026English
    13·
    7 days ago

    https://www.todayintabs.com/p/eternal-black-october

    The problem with A.I. defense is the same asymmetry attackers have always had against defenders. After a hotel bombing in Brighton, England in October, 1984 that failed to kill Margaret Thatcher, the Irish Republican Army put out a statement that said “Today, we were unlucky. But remember we have only to be lucky once. You will have to be lucky always.“ As it turned out, Margaret Thatcher was lucky always, but there’s no reason to believe the rest of us will be. If an A.I. hack attempt works one time out of a thousand, that’s a fantastic success rate. If your A.I. sysop “secures” your server by deleting the production database and all the backups one time out of a thousand… well. Who’s gonna be first to sign up for those odds? Roll the dice! Apparently they always come up four.

  • nfultz@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 1st November 2026English
    13·
    11 days ago

    https://www.patreon.com/MaxRead/posts/ai-x-risk-and-ai-173181621

    worth the full read, but here’s the parts I liked

    But “x-risk” is not the only kind of “existential” risk about which AI enthusiasts and their converts tend to worry. This recent Ezra Klein column, written in the wake of OpenAI’s release of hundreds of mathematical proofs, is a nice articulation of something I’ve come to think of as “n-risk,” for “ennui risk” (with the mutual understanding that it’s not pronunciation-accurate so don’t bother making a nitpicky little comment about it): fear of the loss of comprehension, control, and meaning in the face of impossibly sophisticated godlike machines poses.

    We see the threat of “n-risk” in some of the emotional reaction of mathematicians faced with the results of OpenAI models bulldozing through open problems: a sense of having been surpassed and left behind, having the meaning from which one derives one’s worth totally emptied. In some sense, on a conceptual level and as a discursive object, x-risk is preferable to n-risk: it’s striking, dramatic, and offers us humans an ongoing chance at great feats of heroism and genius. N-risk has no such narrative reassurances. Maybe it’s better to burn out (as your body is turned into a paperclip) than to fade away.

    Here you might ask: Is the cause of melancholy and despair on the part of some mathematicians simply the knowledge that a computer system has surpassed their ability to understand or produce proofs? Or is it the realization that said computer system is privately owned and confidentially developed in service of accumulation rather than the pursuit of shared human knowledge? Or is it the contempt with which the people who work for the computer system companies and the hanger-on fans of these computer systems seem to regard the mathematicians? Or is it the sudden recognition that the pursuit of mathematical truths has already been so totally devalued by an anti-intellectual social system that you can no longer feel secure that the tradition to which you have devoted your life will continue as a space of intellectual inquiry, even given the development of these new tools? And so on.

    I mean, realistically, it’s all of the above.

  • nfultz@awful.systemstoTechTakes@awful.systems•On the feasibility of less illegal "AI coding tools" [effortpost]English
    1·
    19 days ago

    Hmm, I guess they could have written a lot of code lines, but its not like the code they write would be equivalent to million LOC of a coherent whole.

    I’d have guessed it was better for a training set to have lots of different mostly-known-good solutions to common problems and good visibility of the solution space, rather than one linux-sized program, even with the same bug rate between the two. I’d imagine leetcode is monetizing their user generated code this way too. IDK.

    I do actually have a decent GPU i bought 4 years ago in my PC, so maybe I’ll do a few simple experiments again… my preferred approach is to pick a simple but nonlinear problem e.g. binary multiplication of two large numbers, where you e.g. skew the training sample distribution or omit some number pairs and see if the resulting model can learn correct multiplication anyway.

    For that particular experiment, I’d also point you towards some of the 90s literature on “genetic programming”, because you could reuse most of your scaffolding for it. No GPU req’d, ought to learn it fairly quickly.

  • nfultz@awful.systemstoTechTakes@awful.systems•On the feasibility of less illegal "AI coding tools" [effortpost]English
    2·
    20 days ago

    Mercor has ~$1.5B revenue so napkin math would put at 10-20 million contractor hours per year. I’m not very fast but I can type out about a hundred lines an hour fairly regularly, especially boring code like unit tests, data cleaning or OO design patterns. LoC is a bad metric but still. For comparison, Linux and Chrome are ~40M LoC each, AAA games are 3-10M LoC.

    And to your point, they can do data augmentation by rotating nodes in the ASTs, changing variable names, and other tricks to make it even fatter.

    For comparison, the Software Heritage Archive of all open source is ~2 Petabytes / ~30B files. But not sure how much of it is AI slop at this point, or for dead languages / platforms.

  • nfultz@awful.systemstoTechTakes@awful.systems•On the feasibility of less illegal "AI coding tools" [effortpost]English
    7·
    21 days ago

    So, I was thinking about how modern LLMs got better at coding in the last 1 year, despite not having much good new training data.

    Half the job posts I see on linkedin are from snorkel, handshake and mercor etc to train llms for $50-90/hr. So I think there’s been quite a bit of new training data created as those companies scaled up.

    So everyone is paying openai to pay mercor to pay contractors to write code and then regurgitate it, rather than just paying for what they want directly.

    idk. I’m just skeptical that there was actually no human in the loop and that the gains were from unit tests / synthetic TDD.