• YTG123@feddit.ch
    link
    fedilink
    English
    arrow-up
    79
    arrow-down
    7
    ·
    9 months ago

    We need laws mandating respect of robots.txt. This is what happens when you don’t codify stuff

    • Echo Dot@feddit.uk
      link
      fedilink
      English
      arrow-up
      38
      arrow-down
      1
      ·
      9 months ago

      It’s a bad solution to a problem anyway. If we are going to legally mandate a solution I want to take the opportunity to come up with an actually better fix than the hacky solution that is robots.txt

    • patatahooligan@lemmy.world
      link
      fedilink
      English
      arrow-up
      24
      ·
      9 months ago

      AI companies will probably get a free pass to ignore robots.txt even if it were enforced by law. That’s what they’re trying to do with copyright and it looks likely that they’ll get away with it.

    • nutsack@lemmy.world
      link
      fedilink
      English
      arrow-up
      20
      arrow-down
      3
      ·
      edit-2
      9 months ago

      you can’t really make laws in the united states it’s too hard

      • SPRUNT@lemmy.world
        cake
        link
        fedilink
        English
        arrow-up
        24
        arrow-down
        3
        ·
        9 months ago

        The battle cry of conservatives everywhere: It’s too hard!

        Except if it involves oppressing minorities and women. Then it’s a moral imperative worth all the time and money you can shovel at it regardless of whether the desired outcome is realistic or not.

        • Jojo@lemm.ee
          link
          fedilink
          English
          arrow-up
          6
          ·
          9 months ago

          Seriously, could the party of “small government” get out of my business, please?

            • Jojo@lemm.ee
              link
              fedilink
              English
              arrow-up
              1
              ·
              9 months ago

              I just wish the push and pull of politics didn’t have to be played as a zero sum game. I wish someone could take the initiative and just…

              I think both parties in America sing pretty loud about “law and order.” I haven’t heard that cry particularly loudly from either side over the other. I don’t think I’ve heard anyone who claims to be a Democrat saying the end goal is “small government” but I have heard it from Republican voices.

              Honestly, I would really prefer if we were in a system that enabled more parties, so we didn’t have “parties” that did such contradictory things as the current ones…

              • JasonDJ@lemmy.zip
                link
                fedilink
                English
                arrow-up
                1
                ·
                edit-2
                9 months ago

                The GOP has historically been the party of law and order. Hence why they implied that blue lives matter more than black lives.

                thatsthejoke.png

                Just like how one party impeached a president of the other for obstruction and abuse of power, and the other impeached a president for checks notes lying about a blowjob.

    • AA5B@lemmy.world
      link
      fedilink
      English
      arrow-up
      21
      arrow-down
      9
      ·
      9 months ago

      Turning that into a law is ridiculous - you really can’t consider that more than advisory unless you enforce it with technical means. For example, maybe put it behind a login or captcha if you want only humans to see it

        • AA5B@lemmy.world
          link
          fedilink
          English
          arrow-up
          8
          arrow-down
          5
          ·
          edit-2
          9 months ago

          Yes, and there’s also no law against calling an unlisted phone number

          Also we already had this battle with robots.txt. In the beginning, search engines wouldn’t honor it either because they wanted the competitive advantage of more info, and websites trusted it too much and tried to wall off too much info that way.

          There were complaints, bad pr, lawsuits, call for a law

          It’s no longer the Wild West:

          • search engines are mature and generally honor robots.txt
          • websites use rate limiting to conserve resources and user logins to fence off data there’s a reason to fence off
          • truce: neither side is as greedy
          • there is no such law nor is that reasonable
    • ArmokGoB@lemmy.dbzer0.com
      link
      fedilink
      English
      arrow-up
      11
      ·
      9 months ago

      Sounds like the type of thing that would either be unenforceable or profitable to violate compared to the fines.

    • wabafee@lemmy.world
      link
      fedilink
      English
      arrow-up
      5
      arrow-down
      3
      ·
      edit-2
      9 months ago

      I hope not, laws tend to get outdated real fast. Who knows robots.txt might not even be used in the future and it just there adding space because of law reasons.

      • Tyfud@lemmy.world
        link
        fedilink
        English
        arrow-up
        12
        ·
        9 months ago

        You can describe the law in a similar way to a specification, and you can make it as broad as needed. Something like the file name shouldn’t ever come up as an issue.

        • GhostMatter@lemmy.ca
          link
          fedilink
          English
          arrow-up
          2
          ·
          9 months ago

          The law can be broad with allowances to define specifics by decree, executive order or the equivalent.

      • BreakDecks@lemmy.ml
        link
        fedilink
        English
        arrow-up
        10
        ·
        9 months ago

        robots.txt is a 30 year old standard. If we can write common sense laws around things like email and VoIP, we can do it for web standards too.

      • kingthrillgore@lemmy.ml
        link
        fedilink
        English
        arrow-up
        4
        ·
        9 months ago

        robots.txt has been an unofficial standard for 30 years and its augmented with sitemap.xml to help index uncrawlable pages, and Schema.org to expose contents for Semantic Web. I’m not stating it shouldn’t not be a law, but to suggest changing norms as a reason is a pretty weak counterargument, man.

      • Echo Dot@feddit.uk
        link
        fedilink
        English
        arrow-up
        5
        arrow-down
        2
        ·
        9 months ago

        We don’t need new laws we just need enforcement of existing laws. It is already illegal to copy copyrighted content, it’s just that the AI companies do it anyway and no one does anything about it.

        Enforcing respect for robots.txt doesn’t matter because the AI companies are already breaking the law.

        • BreakDecks@lemmy.ml
          link
          fedilink
          English
          arrow-up
          2
          ·
          9 months ago

          I think the issue is that existing laws don’t clearly draw a line that AI can cross. New laws may very well be necessary if you want any chance at enforcement.

          And without a law that defines documents like robots.txt as binding, enforcing respect for it isn’t “unnecessary”, it is impossible.

          I see no logic in complaining about lack of enforcement while actively opposing the ability to meaningfully enforce.

          • Echo Dot@feddit.uk
            link
            fedilink
            English
            arrow-up
            3
            ·
            9 months ago

            Copyright law in general needs changing though that’s the real problem. I don’t see the advantage of legally mandating that a hacky workaround solution becomes a legally mandated requirement.

            Especially because there are many many legitimate reasons to ignore robots.txt including it being misconfigured or it just been set up for search engines when your bot isn’t a search engine crawler.

    • XTornado@lemmy.ml
      link
      fedilink
      English
      arrow-up
      3
      arrow-down
      4
      ·
      9 months ago

      All my scrapping scripts go to shit…please no, I need automation to live…