Whichever way you go, would you consider redoing it so that more of the widget space is consumed when possible. I would much prefer that this widget:
Looked like this instead:
Well, something like that. The point is to have as little blank space and as big covers as will fit. All portrait covers should be as tall as possible that they still reach the side edges of their cell and all landscape books should be as wide as possible and then just drawn as tall as they are. Tall skinny books would be the exception and would have extra space.
If you do that, the books will all be the same width, and the borders will be so small that they won't stand out.
In this case, only very tall but skinny covers will have extra padding, but that's a more obscure use-case. I just scanned though a chunk of my library and didn't see any.