• marcos@lemmy.world
    link
    fedilink
    arrow-up
    13
    arrow-down
    41
    ·
    4 months ago

    Nothing in my post (or in robots.txt) has any relation to distributing your content.

    • BlackPenguins@lemmy.world
      link
      fedilink
      arrow-up
      24
      arrow-down
      1
      ·
      edit-2
      4 months ago

      What else would they scrape your data for? Sure some could be for personal use but most of the time it will be to redistribute in a new medium. Like a recipe app importing recipes.

      • lud@lemm.ee
        link
        fedilink
        arrow-up
        7
        arrow-down
        2
        ·
        4 months ago

        Indexing is what “scrapers” mostly do.

        That’s how search engines work. If you don’t allow any scraping don’t be surprised if you get no visitors.

        • Album@lemmy.ca
          link
          fedilink
          arrow-up
          13
          ·
          4 months ago

          Search engine scrapers index. But that’s a subset of scrapers.

          There are data scrapers and content scrapers, and these are becoming more prolific as AI takes off and ppl need to feed it data.

          This post is specifically about AI scrapers.