That's true, but I still don't see it as the same as copying, even though I can't tell you exactly why.
One thing that is very different for books compared to other media would be the number of times a book is read. There are only a few books that I have read more than once. (Other people may say more than me, but it would still be a low number in proportion to all the books they must have read.) If I give you a book I have read and that I won't read again, then it doesn't deprive me of anything. That would not be the same with a record, CD or even a DVD, which I would listen to or watch multiple times. If I loaned you those, I would certainly expect them to be returned.
So, yes, I do agree that books are different in that respect which explains the charity shop sales and the shelves for loan in railway stations and youth hostels. However, physical books are not copied in the way that a digital copy can be, or a recording can be. That can be done on an industrial scale with an original item leading to millions of copies. That is the difference, I think.
PS. I think
@SilentRoamer just explained it better than me.