Deleting folders without OPF vs recreating OPF files

[wladdy]

After a large import, many of the new books did not appear in my library.

The 'Check library' tool, shows many 'Extra titles'.
When I look at the Calibre library folder structure in the Finder (I am on Mac OSX), I book folders with proper book files but no opf metadata file.

I assume that the missing books, the Extra titles and the folders without opf all correspond to the same thing.

Running 'Restore library' did not help.
Running 'Add books' again on some original files did bring back their missing books.
To get rid of the 'Extra books', I tried 'Moving library to new location', hoping that the folders without opf would be left behind. Unfortunately, they were copied as well.

I have two questions:

(1) How can I clear a Calibre library folder structure from all folders without opf files, which seem to be parasitic folders since the books they contain do do not appear in the GUI?

(2) If such folders cannot be easily removed from within Calibre (ie: without writing my own shell script), is there a way to create or to recreate the opf files?

Your answers and recommendations will be greatly appreciated, as they will condition what I do to restore my library. Thanks in advance! W.

PS - the following threads discuss similar issues
https://www.mobileread.com/forums/sho...d.php?t=255818
https://www.mobileread.com/forums/sho...d.php?t=256497

[BetterRed]

@wladdy - have a look at Library Maintenance->Library backup status.

The opf files are created by a low priority background thread, after adding a lot of books I think it may take a while to create the opf files. If you want to hurry things along, quit calibre, and run the calibredb --backup_metadata command.

BR

[wladdy]

Thank you, BetterRed.

'calibredb backup_metadata' does nothing.
'calibredb backup_metadata --all' returns an error message (attached below).
In both cases, I get the prompt back way too soon for any significant work to have been accomplished on the Calibre database.

Any other ideas would be most welcome.


Python function terminated unexpectedly: 'NoneType' object has no attribute 'title'
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 209, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/library/cli.py", line 1554, in main
File "site-packages/calibre/library/cli.py", line 1192, in command_backup_metadata
File "site-packages/calibre/db/legacy.py", line 355, in dump_metadata
File "site-packages/calibre/db/cache.py", line 57, in call_func_with_lock
File "site-packages/calibre/db/cache.py", line 1139, in dump_metadata
File "site-packages/calibre/library/cli.py", line 1175, in __call__
AttributeError: 'NoneType' object has no attribute 'title'

[BetterRed]

@wladdy - sorry I can't offer any insight on the error message, other than the obvious. I've only used the command to 'hurry things along' when I've changed something like Publisher that cascaded through '000s of books.

One suggestion (if not already tried) - try using the Copy to Library feature (you may need to add it to a Toolbar) to Move (Copy with delete) all the books in the Book list to a fresh library. That should leave the library with a set of "left over" author/book folders that need "sorting out".

First I would remove the old library from calibre (Library->Remove Library), it wont delete anything there, just lose sight of it. Then I would manually deal with the "left over" books - how I did that would depend on how many there were, what files existed and the mood I was in.

BR

[wladdy]

I ended up writing the following shell script to identify the folders with no OPF metadata file.

#!/bin/bash
L=""
r="/Volumes/Ext/CalibreLibrary/"
d=$(find "$r" -type d -mindepth 2 -maxdepth 2)
while read p
do
o=$(find "$p" -name *.opf)
if [ -z "$o" ]
then
L="$L"$'\n'"$p"
fi
done <<< "$d"
echo "$L" > "/Users/wladdy/Desktop/opfless"

Once I was convinced that I had identified the proper folders, I ran another script to delete them:

#!/bin/bash
L="/Users/wladdy/Desktop/opfless"
while read p
do
rm -r "$p"
done < $L

I include the scripts because I am a very novice Bash scripter, and I would be grateful for any comments to make my code less clumsy (I am sure that there must be a more direct identification approach, using just a 'find' command and no loops...)
However, these work well enough as that are. O course, any Mac user interested is welcome to them (just be sure to adapt the pathnames to your own system).

Thanks again to BetterRed, who made me realize that the generation of the opf metadate files continues in the background way after the import has finished. Actually, I think that what cause my problems was quitting Calibre and disconnecting the external drive containing my library before the job had ended. Maybe a future version of Calibre could include a warning to neither quit nor eject while the library is being worked on in the background.

Anyway, problem solved! Onward to fresh ones...

[eschwartz]

For starters, code indenting and userfriendly variable names would be nice.

You might want to investigate use of the "-exec" flag which tells find to execute commands for each found object.

[kovidgoyal]

That error comes from a bug in calibredb (https://github.com/kovidgoyal/calibr...f37952eef31548)

[BetterRed]

Originally Posted by wladdy

Thanks again to BetterRed, who made me realize that the generation of the opf metadate files continues in the background way after the import has finished. Actually, I think that what cause my problems was quitting Calibre and disconnecting the external drive containing my library before the job had ended. Maybe a future version of Calibre could include a warning to neither quit nor eject while the library is being worked on in the background.

Anyway, problem solved! Onward to fresh ones...

@wladdy - AFAIK the background task will end even if there are opf writes pending, next time calibre is run the task is restarted, I assume its driven via a dirty flag in the metadata.db database.

But if you're pulling a usb then you should probably make sure there are no calibre processes hanging around before you do so. My Mac fu ain't up to telling you how best to do that

BTW the message from Kovid indicates he has fixed the bug you reported in post #3 - it'll be in the next release

BR

[chaley]

Originally Posted by BetterRed

@wladdy - AFAIK the background task will end even if there are opf writes pending, next time calibre is run the task is restarted, I assume its driven via a dirty flag in the metadata.db database.

Confirmed: this is how it works.