• Album@lemmy.ca
    link
    fedilink
    arrow-up
    1
    ·
    4 months ago

    User agent catching is rather effective. You can serve different responses based on UA.

    So generally people will use a robots.txt to catch the bots that play nice and then use useragents to manage abusers.