If your whole defense hinges on "borrowing books has to have an inherent delay of X hours/days/weeks before they can be given out again", that's a very weak point in today's day and age. It's like saying "sending mails is bad because it is nearly instantaneous, and you don't have to wait for the postman to deliver your letter".
On second thought, I don't think the demographic of "people buying (your) books" and "people borrowing (your) books from a library" have that much overlap.
People who borrow books from a library are usually people who either don't want to (or can't) buy all the books they like to read. In that case they are unlikely to buy the book anyway even if they can't find it in the library... OTOH people who love your books or your writings, or people who saw a review and think "I'd like to read that book" will buy it anyway and not read a scanned version of it on their small phone screen.
I mean I understand why authors would love it if libraries didn't exist and everyone had to buy the book to read it, I would probably be in the same boat if I were an author. But the calculation "1 borrowed book = 1 lost sale" is flawed the same way that software companies' "1 warez download = 1 lost sale" is flawed