The four-factor test does not require that all factors pass. It's possible for a use to completely fail some of the factors but still qualify as fair use on the basis of the other factors.
No argument that CDL failed "amount and substantiality of"; they copied entire books. (This is, of course, exactly what they do with everything else, and that's not been a problem before.)
IA's "nature of" argument could have been reasonable for the subset of works that constituted nonfiction/educational material (there is a long history of the copying of those such work for educational use); that wouldn't have sufficed for other works, but it was a reasonable point for the subset of works it applied to.
For "purpose and character", the use was not hugely transformative, but it was turned into a different and more accessible form.
As for "effect of the use", I would argue that CDL was not in practice a substitute for most uses of a book or ebook. A book that you borrow, and have to return, can sometimes substitute for owning your own copy, but not always; in practice, the users of libraries and the users of bookstores overlap but I would venture that the majority of people who borrow a book from the library would not typically have bought the same book if the library didn't exist.
Suppose one built an automated apparatus that remotely opened a physical book and held individual pages up to a webcam for transmission to you on a video call. That's technically making a copy in the course of its operation, but you're still effectively reading the original physical book, with some assistance. (The Supreme Court ruling against Aereo would sadly probably be cited to shoot down such a model. That was a sad ruling as well; the opposite ruling would have enabled an incredible variety of uses and possibilities.) The 1:1 CDL mechanism seems effectively equivalent to that.