I recently [edit: August 2006] discovered exactly how the Wayback Machine deals with changes to robots.txt.

First, some background. I have a weblog I’ve been running since 2002, switching from B2 to WordPress and changing the permalink structure twice (with appropriate HTTP redirects each time) as nicer structures became available. Unfortunately, some spiders kept hitting the old URLs over and over again, despite the fact that they forwarded with a 301 permanent redirect to the new locations. So, foolishly, I added the old links to robots.txt to get the spiders to stop.

Flash forward to earlier this week. I’ve made a post on Slashdot, which reminds me of a review I did of Might and Magic IX nearly four years ago. I head to my blog, pull up the post… and to my horror, discover that it’s missing half a sentence at the beginning of a paragraph and I don’t remember the sense of what I originally wrote!

My backups are too recent (ironic, that), so I hit the Wayback Machine. They only have the post going back to 2004, which is still missing the chunk of text. Then I remember that the link structure was different, so I try hitting the oldest archived copies of the main page, and I’m able to pull up the summary with a link to the original location. I click on it… and I see:

Excluded by robots.txt (or words to that effect).

Now this is a page that was not blocked at the time that ia_archiver spidered it, but that was later blocked. The Wayback machine retroactively blocked access to the page based on the robots.txt content. I searched through the documentation and couldn’t determine whether the data had actually been removed or just blocked, so I decided to alter my site’s robots.txt file, fire off a request for clarification, and see what happened.

As it turns out, several days later, they unblocked the file, and I was able to restore the missing text.

In summary, the Wayback Machine will block end-users from accessing anything that is in your current robots.txt file. If you remove the restriction from your robots.txt, it will re-enable access, but only if it had archived the page in the first place.

(Originally posted as a Slashdot comment.)

DC’s Archive line is their line of hardcover reprints on nice, glossy paper, usually following a character or team starting at the beginning of the series. DC has two sets of Flash archives right now:

• The Flash Archives: 4 volumes following Barry Allen from his first few appearances in Showcase through the start of his solo title, covering 1956–1962.
• The Golden Age Flash Archives: 2 volumes following Jay Garrick through the first 2 years of Flash Comics and All-Flash, covering 1940–1941.

The survey also asks about other reprint formats, including the paperback Chronicles series, the Omnibus series (hardcover, but lower-quality paper), and more thematic reprint sets (one suggestion is Flash: The Death of Iris Allen

So if, like me, you’re still hoping for that next volume of Golden Age Flash Archives—or any other classic DC book that hasn’t been reprinted in decades, if ever—stop on over and fill out the survey.

(via Comic Bloc Forums)

Are there more Legion, Flash or Justice League Archives coming? [VP of Sales Bob] Wayne said that when you get up to the issues that can be affordably bought by collectors the demand for the Archive Editions goes down.

Okay, this might apply to the Silver-Age material. The four Flash Archives books so far are up to Flash #132 (1962). When I was tracking down back-issues in the #133–140 range (the likely contents of a hypothetical book 5) about 6 or 7 years ago, I seem to remember finding reasonably good copies in the $5-15 range. (Better copies, of course, run into triple digits.)

But there’s still 8 years of Golden-Age material to cover, from 1942–1949: more than 75% of Jay Garrick’s solo run. And those books are much harder to find, with battered readers’ copies often selling for $40–150.

Moreover, those 8 years include the first appearances of every major Golden-Age Flash villain. The current 2 volumes of The Golden Age Flash Archives only make it through Flash Comics #24 and All-Flash #2. The Shade, the first real supervillain to grace the series’ pages, didn’t appear until Flash Comics #33. The Fiddler, the Thinker, the Turtle, the Rag Doll—even the original Rose & Thorn and Star Sapphire all made their appearances during those years, and haven’t shown up in the archives yet.

In fact, the only Golden-Age villain to get the archive treatment for his original appearances is the Rival, who showed up in Flash Comics #104 (the final issue of the series), and whose story was (oddly) included in the silver-age Flash Archives vol. 1.

Even when you factor in the golden-age stories that got reprinted in later Flash books, the “Greatest Stories” collections, and the occasional anthology series like DC Super-Stars, 75% of the stories have never been reprinted* since the 1940s.

I’ve been looking for Golden-Age issues for almost 2 years now, and by focusing on All-Flash (which typically included 3–4 short stories or 1 long one), I’ve managed to hit 50%. It’s gotten harder, largely because I’m less willing to pay the big bucks for random issues than for issues with specific characters.

Now, there is of course a question of whether there’s a demand for the material at all…but when a beat-up reader’s copy of a book goes for more than the archive would cost on Amazon, I’d say there’s no risk that the availability of back issues will interfere with the market for the archive.

(Expanded from a comment I made in the subsequent forum thread.)

*104 stories in Flash Comics, 1 in the Flash Comics Miniature edition, 29 in Comic Cavalcade, 71 stories in 32 issues of All-Flash, 3 completed but unpublished until years later = 208 total. Of those, 32 have appeared in the GA Flash and Comic Cavalcade Archives, 1 in the Silver Age Flash Archives, and another 16 in various other books. Just 49 out of 208. Only 23.5% It’s a little harder to go by issues instead of stories, since some issues of All-Flash have only partially been reprinted, but it seems to be roughly 41 out of 169 — which works out to 24.3%.

Edit Oct. 10: Confirmed! Today’s DC Comics Direct Channel Special lists the archive and its contents among the books scheduled for January-February.