86thumbs
  • Communities
  • Create Post
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
fattyfoods@feddit.nl to Open Source@lemmy.ml · 2 days ago

The Open-Source Software Saving the Internet From AI Bot Scrapers

www.404media.co

external-link
message-square
107
fedilink
  • cross-posted to:
  • technology@beehaw.org
578
external-link

The Open-Source Software Saving the Internet From AI Bot Scrapers

www.404media.co

fattyfoods@feddit.nl to Open Source@lemmy.ml · 2 days ago
message-square
107
fedilink
  • cross-posted to:
  • technology@beehaw.org
Anubis, which block AI scrapers from scraping websites to death, has been downloaded almost 200,000 times.
  • bdonvr@thelemmy.club
    link
    fedilink
    arrow-up
    16
    ·
    2 days ago

    Thanks. Anything special configuring it?

    • Captain Beyond@linkage.ds8.zone
      link
      fedilink
      arrow-up
      19
      ·
      edit-2
      2 days ago

      I keep my server config in a public git repo, but I don’t think you have to do anything really special to make it work with lemmy. Since I use Traefik I followed the guide for setting up Anubis with Traefik.

      I don’t expect to run into issues as Anubis specifically looks for user-agent strings that appear like human users (i.e. they contain the word “Mozilla” as most graphical web browsers do) any request clearly coming from a bot that identifies itself is left alone, and lemmy identifies itself as “Lemmy/{version} +{hostname}” in requests.

Open Source@lemmy.ml

opensource@lemmy.ml

remote_follow_modal_title

Create a post
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !opensource@lemmy.ml

All about open source! Feel free to ask questions, and share news, and interesting stuff!

Useful Links

  • Open Source Initiative
  • Free Software Foundation
  • Electronic Frontier Foundation
  • Software Freedom Conservancy
  • It’s FOSS
  • Android FOSS Apps Megathread

Rules

  • Posts must be relevant to the open source ideology
  • No NSFW content
  • No hate speech, bigotry, etc

Related Communities

  • !libre_culture@lemmy.ml
  • !libre_software@lemmy.ml
  • !libre_hardware@lemmy.ml
  • !linux@lemmy.ml
  • !technology@lemmy.ml

Community icon from opensource.org, but we are not affiliated with them.

community_visibility: public
globe

public_blurb

  • 344 users / day
  • 1.7K users / week
  • 3.37K users / month
  • 10.5K users / 6 months
  • number_of_local_subscribers
  • 38.7K subscribers
  • 2.3K Posts
  • 36.5K Comments
  • Modlog
  • mods:
  • Evan@lemmy.ml
  • kevincox@lemmy.ml
  • CrypticCoffee@lemmy.ml
    cake
  • Lettuce eat lettuce@lemmy.ml
  • UI: unknown version
  • BE: 0.19.5
  • Modlog
  • Instances
  • Docs
  • Code
  • join-lemmy.org