Copying has a specific definition under the law. Physical lending of books does not come close to meeting it. Applying any of the fair use tests to physical book lending thus does not make any sense. It's like applying the Central Hudson Test to my cooking of a grilled cheese sandwich: it's lawful, the government has minimal interest in regulating it, none of that matters because my sandwich isn't expressive.

If your whole point is that we could tear down all of copyright law and replace it with a system that allowed IA CDL, then, sure. We could do a lot of things. I'm not really here for that argument (because there just isn't enough to nerd out about in it, not because I have any problem with the exercise).

But as for the law as has existed in the United States for the last 50-odd years, I'm reminded of the words of a young Baltimore entrepreneur, who infamously said "you want it to be one way --- but it's the other way".