why are you blocking chatgpt from accessing the wiki?

mrmetaverse · 2026-01-10 16:11:05

It doesn't make sense to me. Many of us use chat and other LM powered tools to fix our configs. You are blocking access why?

`I attempted to pull “today’s tickets” directly from ArchWiki, but the site is currently protected by Anubis and is blocking automated access in this environment (I get an “Access Denied” page).`

Last edited by mrmetaverse (2026-01-10 16:12:00)

gromit · 2026-01-10 16:20:36

I'll be blunt about it: Generally we're doing this because the underlying infrastructure is crumbling under the load that the AI crawlers are putting on it. Read https://about.readthedocs.com/blog/2024 … ers-abuse/ to learn a bit more about the general phenomenon.

The Wiki itself should be freely available though, just the expensive endpoints like last changed pages, diffs etc. are guarded with Anubis.

Scimmia · 2026-01-10 16:21:22

Because these AI scrapers put far, far too much load on the server. They're getting blocked everywhere.

system72 · 2026-01-10 16:35:02

i got over 150k requests from openai on christmas targeting one of my instances, these scrapers are disrespectful and dont follow robots.txt, so solutions like anubis are needed

seth · 2026-01-10 17:03:50

Many of us use chat and other LM powered tools to fix our configs.

So you're asking for an LLM to be able to read the wiki so it can tell you what's in the wiki because you're too lazy to read the wiki or even press ctrl+f on it.

Stop wasting your time and everyones hardware with that.

From what I've seen here and gut-stimating, more than 2/3 of the LLM suggestions are flat out false.
The rest is between "missing the point but works by sheer dumb luck", "ridiculously clumsy" (this can actually pair with all other scenarios) and occasionally they hit the nail.

But if your strategy revolves around occasionally hitting the nail, you better invest in a screwdriver or some band-aid.

mrmetaverse · 2026-01-10 19:15:52

Thanks all, that actually makes sense, and I appreciate the answers. I was thinking it was some grand violation of freedom of information and open source, and that it was merely bias against use of AI tools. I have been managing my configs with AI tools for several years now, and I have a process. Statements lke "more than 2/3 of the LLM suggestions are flat out false." are false. Sure it doesn't get it right every time, but I am not "lazy" I use the tools like tools, and I get excellent results most of the time.

Not lazy! I will use ctrl+f and paste it into chat myself. Thanks!

mrmetaverse · 2026-01-10 19:17:00

Also TIL about anubis. One of my main sites get bombarded with bots and fake submissions all day. Going to look into this myself.

seth · 2026-01-10 20:51:11

"more than 2/3 of the LLM suggestions are flat out false." are false.

seth wrote:

what I've seen here and gut-stimating

You can assume a bias in that data because people will often show up because the AI got them into trouble.

Sure it doesn't get it right every time

If you can determine that, why are you asking the AI itfp?
If you cannot determine this, how do you know? (How do you know "it made it work" isn't just one of the "sheer dumb luck" cases?)

One of my main sites get bombarded with bots and fake submissions all day.

I guess many people are asking ChatGPT why the internet became so slow and fragile ever since ChatGPT is around… it's a mystery.

system72 · 2026-01-10 21:48:21

its sad that we require js solutions to a problem that really shouldnt exist

seth · 2026-01-10 21:55:34

Obligatory https://www.youtube.com/watch?v=5RyYrs5tu60

Also https://www.heise.de/en/background/Obit … 66991.html

LinuxLover471 · Today 06:39:43

If your question is answered, then edit the original post and add [SOLVED] to the start of the subject.

Arch Linux

#1 2026-01-10 16:11:05

why are you blocking chatgpt from accessing the wiki?

#2 2026-01-10 16:20:36

Re: why are you blocking chatgpt from accessing the wiki?

#3 2026-01-10 16:21:22

Re: why are you blocking chatgpt from accessing the wiki?

#4 2026-01-10 16:35:02

Re: why are you blocking chatgpt from accessing the wiki?

#5 2026-01-10 17:03:50

Re: why are you blocking chatgpt from accessing the wiki?

#6 2026-01-10 19:15:52

Re: why are you blocking chatgpt from accessing the wiki?

#7 2026-01-10 19:17:00

Re: why are you blocking chatgpt from accessing the wiki?

#8 2026-01-10 20:51:11

Re: why are you blocking chatgpt from accessing the wiki?

#9 2026-01-10 21:48:21

Re: why are you blocking chatgpt from accessing the wiki?

#10 2026-01-10 21:55:34

Re: why are you blocking chatgpt from accessing the wiki?

#11 Today 06:39:43

Re: why are you blocking chatgpt from accessing the wiki?

Board footer