5000+ books in Calibre Library- EACH duplicated in directory- not in Calibre UI

[chmohta]

Hi
I have a home network set-up wherein I have a windows PC serving as a media/ document server, which I connect through with my laptop etc. On the media server, I host my Calibre Library, which has in total 5060 books (some with multiple formats; making the actual file count to around 6000; collected over a 9 year period, some of which are rare books). I do not run a document server, but rather connect to the calibre library remotely (always ONLY a single instance running) on my laptop; and add books manually through the calibre interface.

The issue I am facing is this- that for EVERY single book in my library, even after removing duplicates from the Calibre UI; I have got a duplicate directory with all the 3 files- cover.jpg; .opf and the ebook itself.
I can't fathom how to correct this; and more importantly, ensure that the problem doesn't recur in future.
Any help towards this would be a great help, people.

many thanks in advance
regards
C

[chmohta]

have attached the screens for better reference to the problem.

[jhowell]

You have your calibre library on a networked drive. That can cause the sort of problem you are seeing.

See the FAQ: I am getting errors with my calibre library on a networked drive/NAS?

[chmohta]

yup, I know it.. unfortunately the constraint that I have got is this (am unsure if it is really a constraint) - my desktop media server is located in a basement multimedia room with all other stuff like Sky tv, xbox, av receiver etc.) and I rarely actually use it to upload/add books into Calibre.. My search and addition of books happen on the laptop; and so does my reading (on phone or Kindle; where I download it through the laptop-based instance of Calibre) . . I am not sure if I could do that on the content server set-up (not an expert).. I did, however, I know that as long as I am not running two calibre instances off the same library, it wouldn't cause issues.

The thing is, all seemed/worked well for last 8+ years- on the investigation of the modified dates of these duplicated, I can see the problem began happening only from Jan 2020.

Net net- is there a straightforward way for me to resolve this situation? Would the Move/Create New Library help here? And does that actually move the files on a different place on the HDD? or just "tags" the files with a new calibre metadata file?

regards
Adarsh

[Adoby]

It seems you have "orphan" books. This typically happen if you used some other tool than calibre to manage your books, and made a mistake. For instance copy a old version/copy/backup of the library OVER the current library. Having the library on a network share may or may not cause problems. In the calibre documentation you are clearly told not to! So don't be surprised when you experience strange problems if you disobey. (I have previously, for some years, successfully stored my library on a NAS, using NFS and Linux laptop client.)

The fix is simple: Create a new empty library with the same structure as the old. Then move the books from the old library to the new library, using calibre.

When you are done, what is left inside the folder of the old library is the orphan books. They may be duplicates only. But there may be some that are not duplicates. You can add all the books in the folder of the old library to the new library and skip all duplicates.

To avoid having this happen in the future you might consider storing the library on the laptop. And a backup copy on the network share. Keep the backup copy updated using some file sync utility. I like rsync from the command line, creating multi-generation snapshots, but there are many free Windows sync utilities you might use. Possibly also with multi-generation snapshot ability.

You can even use a copy of a backup copy as a read-only calibre library on some other device.

I have my calibre libraries stored on my laptop. And several backup rsync snapshot copies on the second SSD on my laptop. And several backup rsync snapshot copies on a NAS.

[theducks]

Calibre's Copy to Library: Delete after Copy ( AKA a Move)
'Moves files.'
You can only Move 'Known' files (those the GUI can access), the rest will be orphaned (GUI says none, but the Library folder has files).

When I first built up my Calibre Library waaay back in 2010, My PC was in my home office, my Paper Library in a storeroom.

Remote Desktop Connection (Windows # Pro on the host) and the free RDC client on a low end laptop (the heavy lifting is done on the host). Barcode reader, Cover scanner (if the metadata fetch did not find the correct one), on the work surface was all I needed (I did have a hardwired network, but a solid WiFi will work.) You can also use VNC Just be aware there may be security issues onsite (LAN), but not Malware.

As others have already pointed to the FAQ. Keep doing it your way and you will keep seeing this Grief, popping up form time to time.
The Q is not IF, but WHEN.

[chmohta]

thanks!! would try it. Needless question possibly- but do I need to necessarily do it on my mediaserver PC? or can I do that through the laptop instance of Calibre?

would need to learn/ read more about rsync etc soon I suppose. My only gripe in terms of copying the entire Calibre Library on the laptop - its the main reason why I had got this "server" in the first place- to avoid having to clog my laptop with stuff which I dont necessarily need too frequently - my library is roughly 180 GB right now (with duplicates- so may be around 90-100 GB in reality).

nut moan aside- thanks again for the quick help guys! really appreciate it. would let you know how the move of library goes. fingers crossed, it would work like a charm!

[theducks]

RDC runs on the host (where Calibre and the Library files are)

The laptop is really just a remote Screen and Keyboard (some versions also allow file copy between the remote and host. That was what I did when I scanned the cover (image scanner was attached to the Laptop), Barcode was a Keyboard wedge, so it was remoted to the host

[chmohta]

Originally Posted by Adoby

It seems you have "orphan" books. This typically happen if you used some other tool than calibre to manage your books, and made a mistake. For instance copy a old version/copy/backup of the library OVER the current library. Having the library on a network share may or may not cause problems. In the calibre documentation you are clearly told not to! So don't be surprised when you experience strange problems if you disobey. (I have previously, for some years, successfully stored my library on a NAS, using NFS and Linux laptop client.)

The fix is simple: Create a new empty library with the same structure as the old. Then move the books from the old library to the new library, using calibre.

When you are done, what is left inside the folder of the old library is the orphan books. They may be duplicates only. But there may be some that are not duplicates. You can add all the books in the folder of the old library to the new library and skip all duplicates.

To avoid having this happen in the future you might consider storing the library on the laptop. And a backup copy on the network share. Keep the backup copy updated using some file sync utility. I like rsync from the command line, creating multi-generation snapshots, but there are many free Windows sync utilities you might use. Possibly also with multi-generation snapshot ability.

You can even use a copy of a backup copy as a read-only calibre library on some other device.

I have my calibre libraries stored on my laptop. And several backup rsync snapshot copies on the second SSD on my laptop. And several backup rsync snapshot copies on a NAS.

So - the Move library trick didn't work but then I managed to solve it!!
essentially I tried the following two things:-
1. After opening the existing library in Calibre; went to Switch/Create Library>Move Current Library to a new location, with "Copy structure from current library ticked" -- this MOVED the duplicates as well; and left "mostly" EMPTY, the author named folders in the previous location. Mostly because, in a very few instances, the books folders (and the ebooks themselves) were also left unmoved. These were NOT ALL folders though.

2. next, I did the simplest thing - In library maintenance, I ran "check library" option and voila- it neatly provides the list of all the files/directories which are not in my db and an option to mark/ delete them. It also provided a list of incorrectly labelled books (i.e. filename was inconsistent with naming convention; and an option to delete empty folders!. Deleted one book first after checking there were indeed 2 instances in directory structure and it correctly deleted the orphan. Then ran the whole shabang; and my calibre library was correctly compressed from 195GB to 82 GB with all 5102 books still there with their unique instances!! And it took just 5 minutes for it to do this.