The point is that providing access to archives of web pages that were once public--especially if robots.txt is even retroactively honored--and CDL, while perhaps not adhering to the letter of copyright law, are sufficiently close to the spirit that most reasonable people see those actions as legit. (There's probably at least a case that you're just providing an equivalent proxy for physical access. IANAL) Especially by an entity which is reasonably viewed as an archive/library.