Electronic bytes are not carved in marble

[Newsmaster]

There was an interesting panel discussion today on Copyright and Internet Archiving on NPR Science Friday. It included Brewster, Michael Keller, and Michael Hart.

I missed most of the show so I'll need to wait for the podcast, but I did find one comment of Brewster's potentially disturbing. In describing how easy it is for copyright owners to have content removed from the Wayback Machine, he said that not only will the Internet Archive honor requests for removal and that the crawler will obey robots.txt files, but IA will also RETROACTIVELY honor robots.txt commands.

As it applies to government, does this mean that a security wonk or web monkey can change a domain's robots.txt file at whim and have all of its archived content removed? If so, content could be jeopardized by the winds of politcal change or just simple misuse of the file. I have a site currently closed by robots.txt because its under construction and another closed because the content is unofficial yet rising above the official content in search results. Yet, I wouldn't want to have that same content removed or made inaccessible in the archives.

Does this same policy apply to Archive-It subscribers? Could someone come along, after the fact, and by tweaking the robots.txt or by making a request, have years of pre-existing content removed from public access?

In his presentation, Brewster warned of this threat, by citing the need for multiple web archives, geopolitically dispersed, under independent decision making authorities.

One would hope that there is a web archives somewhere, immune to local copyright laws, that crawls the public web and preserves and provides access to all "then public" content.