Have a question about how the TVTropes wiki works? No one knows this community better than the people in it, so ask away! Ask the Tropers is the page you come to when you have a question burning in your brain and the support pages didn't help.
It's not for everything, though. For a list of all the resources for your questions, click here. You can also go to this Directory thread
for ongoing cleanup projects.
Any scraping algorithm or program would be blocked by our software, even if it's for the purpose you stated.
"It's Occam's Shuriken! If the answer is elusive, never rule out ninjas!"I'm confused. How would this algorithm tell apart a (sufficiently advanced) scraper from a browser or web spider?
This wouldn't be using a generic User-Agent header. This would be a very specialized header that I'm fairly certain will be unique to this particular script. If necessary, I can also slow it down as much as I like, so speed isn't an issue.
I just checked with a test script, and it would appear that it does, in fact, appear to be blocking based on the User-Agent header. I would like to stress that setting it really isn't an issue for me.
Back to the main point of this thread. I would like clearance to use the script I described before I actually start using it, because last I checked it was required for any sort of HTML scraper on this site. Though I can't seem to find where that policy was described.
You need permission from the admins to run any scraper. Moderators cannot grant it. Send a message to "The Staff" using our contact form
. We cannot offer any promises as to whether (or when) you will get a response.

Would it be acceptable to make a script that scrapes the site for the purposes of making a wick check easier?
Specifically, this script would download the "Related" page for a single article, choose an appropriate number of wicks, and then download the page source for all such wicks. For all requests, the script would identify itself via the User-Agent header.