I am an academic and do the same reading of books for hours each day without phone distractions. The problem is precisely the less formal environments where, instead of a serious book, I'm going to be reading news or other light content. Hard to go for the ereader then instead of the phone.
What about adding some lighter content to the e-reader, even some fiction, to match the vibe you’re looking for in those less formal environments? If your e-reader is only associated with serious book, I can see where that would be a lot for a casual lunch that you want to feel like a break.