Missing Books & Library Maintenance fails to disclose problem

Aussiecubs · 11-27-2019, 02:21 AM

I have a library with 4,000 books in it. I recently discovered that 117 of those books do not actually exist. If I edit the metadata of one of these books, the box that displays the format for the file is empty. If I attempt to open the book to read it, I get "no available formats" error message. If I go to the physical directory of where the book should be, the directory is there but it is empty.

When I discovered the first missing book, I ran library maintenance to check if the information in the database matches the files. The results did not disclose a single "missing books format" error.

Following tips elsewhere on this forum, I ran a search for "formats:false" in the search bar and the search result displayed 117 books that satisfied this search criteria. So Calibre can confirm the error this way, but not via maintenance.

Is this a bug that anyone else has encountered?

theducks · 11-27-2019, 09:12 AM

That is not an 'error' (at least to Calibre

)
What you see is a very legitimate usage called an empty book.
There are 419 in my Library. These document books that exist in Dead Tree (Format)

in the storeroom.

Another common usage: Wishlist of things to come

Library Maintenance verified the Library matched what was found.
(The Library is really The DATABASE of what you have added / changed )
Since there were no integrity errors, that said one of two things: 1)You use Calibre properly (not by going under the skirts in the OS) to remove the format. 2) You had created an empty book entry.

The only unknown: Intent. Were either of those what you really wanted to do?

We can't answer that one.

Aussiecubs · 11-27-2019, 04:33 PM

Thank you Theducks for taking the time to reply and for your insight in to this problem. It is much appreciated.

Now you have explained to me the existence of this feature, I am left with a bit of a mystery of how I finished up with 117 empty book entries.

I have never created an empty book entry as such. The only way I have ever added a new book to Calibre is by firstly, physically obtaining the book in an .epub or .mobi format and then secondly, using the Add Books function to import and add the book to Calibre. This is usually done in a batch of say 5 or 10 or 20 books at a time. So I know for sure all of the books existed in an .epub or .mobi format when each entry was first created.

Having been created in this manner, it seems improbable that I have some how subsequently physically deleted the books from the folders where they resided. I simply have never gone into the files directory with a file browser and stared poking around to give me the opportunity to intentionally or accidentally start deleting various files.

I would also have had to have done this 117 times in 117 different physical locations, which again seems improbable.

Further, I use GoodSync to do a regular one way backup of my Calibre folder to another computer and to Dropbox.

Even if I had somehow or other accidentally physically deleted the .epub or .mobi files at the source or my computer system somehow or other did it without my active involvement, when doing my backups GoodSync would have warned me that the files no longer exist when it gives me the option of removing files from the target directory that no longer exist in the source directory. This has never happened. I only ever run GoodSync manually, so I review what it is about to do to ensure I don't have any autosyncing disasters.

So I am left with quite a mystery about how I arrived at the point of now having 117 empty book entries in my library. Any suggestions?

theducks · 11-27-2019, 05:44 PM

Any OS induced (with or without USER help

) errors would show as an error.

Do you download NEWS? There is a feature that removes older entries with the tag NEWS ((I think the option of Fetch News is 'Delete after n days" ), but that might have snagged your books if they had that tag.

The only other thing would be incorrect usage of the Delete Formats (for selected)
ALWAYS review the STATUS LINE at the bottom. it has a vale for Selected. It should match your expectations

Aussiecubs · 11-27-2019, 06:02 PM

Thanks Theducks for your further thoughts.

I do not download NEWS.

Also I cannot recall ever deleting a single book, or a particular format of a single book from Calibre since I have been using it.

I am a hoarder, I keep collecting and adding books, thinking I will get around to reading them someday. So accidental deletion through some sort of error in selection has not happened.

The mystery remains.

BetterRed · 11-27-2019, 06:44 PM

@Aussiecubs - When you say the book folders are empty, do you mean A) there are no ebook format files (EPUB. MOBI etc), or B) there no files at all, not even a metadata.opf file? If the latter then that would be very strange - every 'book' in the database should have a metadata.opf in the corresponding book folder.

What is the state of the 117 books on your backups. If they are likewise 'empty' that would explain why you didn't get a query from GS regarding what to do about deletions (fellow GS user here). Which would suggest that these books have always been 'empty', or the format files were deleted inadvertently soon after adding via one of the Remove format options -- i.e. before you ran the next GS backup

If you have suppressed the "Are you sure you want to do this" message you wouldn't know you you did it, especially if you've given any of the Remove formats options a keyboard shortcut - e.g. I have Alt+Del assigned to Remove specific format from selected books but I don't suppress any of the calibre "are you sure" messages.

BR

DoctorOhh · 11-27-2019, 10:07 PM

Quote:

Originally Posted by Aussiecubs

Further, I use GoodSync to do a regular one way backup of my Calibre folder to another computer and to Dropbox.

Has there ever been a time before Goodsync that you used the calibre library in Dropbox directly?

Many years ago this happened to me because I did not pause sync during book conversions or while adding books. Then to compound things when Library maintenance was first introduced I didn't look close enough when it said there were extra titles and simply deleted the extra titles. The issue was because the sync interfered, the metadata.db file (which is the heart of the library) wasn't updated as expected. So deleting extra titles was removing books that should have been re-added to the library.

These missing books were my fault, but I didn't realize it at the time. This hasn't happened to me in years, but every now and then I do have to clean up mistakes and put the the books back. My work flow is better now, not perfect, but I never lose books anymore.

Go into Dropbox and show deleted files, maybe some of the books still exist in Dropbox.

Aussiecubs · 11-28-2019, 02:09 AM

@BetterRed
Some folders are completely empty, some folders have a cover.jpg file and some folders have both a cover.jpg and a metadata.opf file. The only thing in common with them all is the fact that there is no .epbu or .mobi file.

As best as I can work out, the GoodSync backup directories seem to exactly mirror the Calibre directories on the computer.

I am absolutely certain I have not suppressed any "are you sure you want to do this" messages and I have not installed any keyboard shortcuts.

@DoctorOhh
This could be the reason. I did at one time have it set up with the Calibre directory inside a Dropbox folder. I only discovered the 117 missing format files recently as part of the process of moving the files to a new (non-Dropbox) folder and adopting a different backup strategy using GoodSync.

I checked the Dropbox deleted files folder but none of the files were there. I think Dropbox only stores my deleted files for 30 days so likely all this happened before then.

As it is only 117 and a fair amount of them are among my lower rated books they are no great loss. The important few that I am really keen on I think I can get my hands on them again.

@ All contributors
Thank you for your advice. A salutary lesson for me about the fact that you cannot be too careful about checking what you have and making sure that your have a fail proof backup plan.

Aussiecubs · 11-28-2019, 02:14 AM

PS thanks to all of this, I now have a better understanding of how Calibre actually works. I have used it for a fair time but not ever really poked my nose under the bonnet (under the hood for my USA friends) to see how the thing actually works.

Adoby · 11-30-2019, 09:48 PM

Sometimes when you add a bunch of books you may by mistake use the wrong import options.

If the books you import are saved from calibre they may consist of a folder for each book, a cover file, one or more actual ebook files and an opf-file with metadata about the book.

Such books should be imported as one book per folder. Then only one book entry is created using the metadata in the opf-file, for all the different book formats.

If it is imported as many books in the same folder then a separate entry is created for each actual ebook file AND one EMPTY book based on the opf-file. So that is one possible reason why you have empty books.

I suspect that you ran an import with the wrong add/import options at one time. You get warnings about duplicate books, but you may have ignored that... Don't do that.

One reason why bad import options are bad is that you may lose the metadata in the opf-file. And you will have to download it again, when all along that data was available but discarded by you. Just imagine how much time you might save for reading if you preserve the metadata in the opf-file.

Aussiecubs · 11-30-2019, 10:07 PM

@Adoby Thank you for your comments. I have never imported books that were previously saved or exported from Calibre. I have only added new .epub & .mobi files from a folder or folders via the Add Books option. However I will now certainly make sure I pay detailed attention to any duplicate books warning. Cheers from Downunder.

11-27-2019, 02:21 AM	#1
Aussiecubs Junior Member Posts: 9 Karma: 10 Join Date: Mar 2016 Location: Melbourne, Australia Device: Sony ereader	Missing Books & Library Maintenance fails to disclose problem I have a library with 4,000 books in it. I recently discovered that 117 of those books do not actually exist. If I edit the metadata of one of these books, the box that displays the format for the file is empty. If I attempt to open the book to read it, I get "no available formats" error message. If I go to the physical directory of where the book should be, the directory is there but it is empty. When I discovered the first missing book, I ran library maintenance to check if the information in the database matches the files. The results did not disclose a single "missing books format" error. Following tips elsewhere on this forum, I ran a search for "formats:false" in the search bar and the search result displayed 117 books that satisfied this search criteria. So Calibre can confirm the error this way, but not via maintenance. Is this a bug that anyone else has encountered?

11-27-2019, 04:33 PM	#3
Aussiecubs Junior Member Posts: 9 Karma: 10 Join Date: Mar 2016 Location: Melbourne, Australia Device: Sony ereader	Thank you Theducks for taking the time to reply and for your insight in to this problem. It is much appreciated. Now you have explained to me the existence of this feature, I am left with a bit of a mystery of how I finished up with 117 empty book entries. I have never created an empty book entry as such. The only way I have ever added a new book to Calibre is by firstly, physically obtaining the book in an .epub or .mobi format and then secondly, using the Add Books function to import and add the book to Calibre. This is usually done in a batch of say 5 or 10 or 20 books at a time. So I know for sure all of the books existed in an .epub or .mobi format when each entry was first created. Having been created in this manner, it seems improbable that I have some how subsequently physically deleted the books from the folders where they resided. I simply have never gone into the files directory with a file browser and stared poking around to give me the opportunity to intentionally or accidentally start deleting various files. I would also have had to have done this 117 times in 117 different physical locations, which again seems improbable. Further, I use GoodSync to do a regular one way backup of my Calibre folder to another computer and to Dropbox. Even if I had somehow or other accidentally physically deleted the .epub or .mobi files at the source or my computer system somehow or other did it without my active involvement, when doing my backups GoodSync would have warned me that the files no longer exist when it gives me the option of removing files from the target directory that no longer exist in the source directory. This has never happened. I only ever run GoodSync manually, so I review what it is about to do to ensure I don't have any autosyncing disasters. So I am left with quite a mystery about how I arrived at the point of now having 117 empty book entries in my library. Any suggestions? Last edited by Aussiecubs; 11-27-2019 at 04:38 PM.

11-30-2019, 09:48 PM	#10
Adoby Handy Elephant Posts: 1,737 Karma: 26785684 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Samsung Galaxy Tab S8 Ultra	Sometimes when you add a bunch of books you may by mistake use the wrong import options. If the books you import are saved from calibre they may consist of a folder for each book, a cover file, one or more actual ebook files and an opf-file with metadata about the book. Such books should be imported as one book per folder. Then only one book entry is created using the metadata in the opf-file, for all the different book formats. If it is imported as many books in the same folder then a separate entry is created for each actual ebook file AND one EMPTY book based on the opf-file. So that is one possible reason why you have empty books. I suspect that you ran an import with the wrong add/import options at one time. You get warnings about duplicate books, but you may have ignored that... Don't do that. One reason why bad import options are bad is that you may lose the metadata in the opf-file. And you will have to download it again, when all along that data was available but discarded by you. Just imagine how much time you might save for reading if you preserve the metadata in the opf-file. Last edited by Adoby; 11-30-2019 at 09:52 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Library Maintenance Problem	mitch13	Library Management	7	01-14-2019 07:41 PM
Problem with template functions and library maintenance	Terisa de morgan	Library Management	12	03-11-2017 09:43 PM
Missing Library/Library Maintenance button	babyshelley	Library Management	2	06-20-2015 01:05 PM
Library Maintenance - Missing Books	iwaddo77	Library Management	26	05-03-2013 10:49 AM
Library Maintenance option missing?	xterminal	Library Management	3	08-27-2012 03:48 PM

11-27-2019, 09:12 AM	#2
theducks Well trained by Cats Posts: 31,001 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	That is not an 'error' (at least to Calibre ) What you see is a very legitimate usage called an empty book. There are 419 in my Library. These document books that exist in Dead Tree (Format) in the storeroom. Another common usage: Wishlist of things to come Library Maintenance verified the Library matched what was found. (The Library is really The DATABASE of what you have added / changed ) Since there were no integrity errors, that said one of two things: 1)You use Calibre properly (not by going under the skirts in the OS) to remove the format. 2) You had created an empty book entry. The only unknown: Intent. Were either of those what you really wanted to do? We can't answer that one.

11-27-2019, 05:44 PM	#4
theducks Well trained by Cats Posts: 31,001 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Any OS induced (with or without USER help ) errors would show as an error. Do you download NEWS? There is a feature that removes older entries with the tag NEWS ((I think the option of Fetch News is 'Delete after n days" ), but that might have snagged your books if they had that tag. The only other thing would be incorrect usage of the Delete Formats (for selected) ALWAYS review the STATUS LINE at the bottom. it has a vale for Selected. It should match your expectations

11-27-2019, 06:02 PM	#5
Aussiecubs Junior Member Posts: 9 Karma: 10 Join Date: Mar 2016 Location: Melbourne, Australia Device: Sony ereader	Thanks Theducks for your further thoughts. I do not download NEWS. Also I cannot recall ever deleting a single book, or a particular format of a single book from Calibre since I have been using it. I am a hoarder, I keep collecting and adding books, thinking I will get around to reading them someday. So accidental deletion through some sort of error in selection has not happened. The mystery remains.

11-27-2019, 06:44 PM	#6
BetterRed null operator (he/him) Posts: 21,693 Karma: 29711016 Join Date: Mar 2012 Location: Sydney Australia Device: none	@Aussiecubs - When you say the book folders are empty, do you mean A) there are no ebook format files (EPUB. MOBI etc), or B) there no files at all, not even a metadata.opf file? If the latter then that would be very strange - every 'book' in the database should have a metadata.opf in the corresponding book folder. What is the state of the 117 books on your backups. If they are likewise 'empty' that would explain why you didn't get a query from GS regarding what to do about deletions (fellow GS user here). Which would suggest that these books have always been 'empty', or the format files were deleted inadvertently soon after adding via one of the Remove format options -- i.e. before you ran the next GS backup If you have suppressed the "Are you sure you want to do this" message you wouldn't know you you did it, especially if you've given any of the Remove formats options a keyboard shortcut - e.g. I have Alt+Del assigned to Remove specific format from selected books but I don't suppress any of the calibre "are you sure" messages. BR

11-28-2019, 02:09 AM	#8
Aussiecubs Junior Member Posts: 9 Karma: 10 Join Date: Mar 2016 Location: Melbourne, Australia Device: Sony ereader	@BetterRed Some folders are completely empty, some folders have a cover.jpg file and some folders have both a cover.jpg and a metadata.opf file. The only thing in common with them all is the fact that there is no .epbu or .mobi file. As best as I can work out, the GoodSync backup directories seem to exactly mirror the Calibre directories on the computer. I am absolutely certain I have not suppressed any "are you sure you want to do this" messages and I have not installed any keyboard shortcuts. @DoctorOhh This could be the reason. I did at one time have it set up with the Calibre directory inside a Dropbox folder. I only discovered the 117 missing format files recently as part of the process of moving the files to a new (non-Dropbox) folder and adopting a different backup strategy using GoodSync. I checked the Dropbox deleted files folder but none of the files were there. I think Dropbox only stores my deleted files for 30 days so likely all this happened before then. As it is only 117 and a fair amount of them are among my lower rated books they are no great loss. The important few that I am really keen on I think I can get my hands on them again. @ All contributors Thank you for your advice. A salutary lesson for me about the fact that you cannot be too careful about checking what you have and making sure that your have a fail proof backup plan.

11-28-2019, 02:14 AM	#9
Aussiecubs Junior Member Posts: 9 Karma: 10 Join Date: Mar 2016 Location: Melbourne, Australia Device: Sony ereader	PS thanks to all of this, I now have a better understanding of how Calibre actually works. I have used it for a fair time but not ever really poked my nose under the bonnet (under the hood for my USA friends) to see how the thing actually works.

11-30-2019, 10:07 PM	#11
Aussiecubs Junior Member Posts: 9 Karma: 10 Join Date: Mar 2016 Location: Melbourne, Australia Device: Sony ereader	@Adoby Thank you for your comments. I have never imported books that were previously saved or exported from Calibre. I have only added new .epub & .mobi files from a folder or folders via the Add Books option. However I will now certainly make sure I pay detailed attention to any duplicate books warning. Cheers from Downunder.

Advert

Advert