Calibre & Moved Books

[nynaevelan]

Hi:

Calibre experts is there a way to recover the books if they have been moved to a directory outside of where Calibre thinks they are? I need to rebuild my library, I have the metadata file and the books but they are not where Calibre thinks they should be?

[MmavisO]

I'm no expert, and if I'm wrong or incomplete, I'm sure someone will be along shortly with a better answer.

On the toolbar, in the upper left corner is an add books button. There is a dropdown arrow with it and if you click the arrow, it gives you some options like whether you have your books in one directory or several. You select your option, and then drill to where your books are really located, and add them.

[nynaevelan]

Yes but this will add the books as duplicates, I am looking for a way for it to find the books that are already in the library. This would not be a problem if Calibre did not have the annoying habit of adding the id to the folder name, a behavior I wish I could stop. But unfortunately it looks like I am going to have to manually go through all 657 of my books individually.

[DoctorOhh]

The obvious first choice to correct this problem is to restore from backup.

Originally Posted by nynaevelan

Yes but this will add the books as duplicates, I am looking for a way for it to find the books that are already in the library.

Before re-adding the books make sure the If books with similar Titles and Authors found, merge the new files automatically feature is checked under Preferences - Adding Books (See attached). This will cause many of the books to be automatically merged with the existing metadata book entries.

On second thought this might not work if calibre still believes that the format already exists. Running a database integrity check first might correct calibre's assumption that certain formats still exist in its library and allow you to re-add the books as suggested.

After you re-add the 675 books whichever books are newly created will have to be manually merged with the existing metadata (See attached).

Originally Posted by nynaevelan

This would not be a problem if Calibre did not have the annoying habit of adding the id to the folder name, a behavior I wish I could stop.

Interesting and here I thought it wouldn't be a problem if you hadn't moved the books in the first place.

Originally Posted by nynaevelan

But unfortunately it looks like I am going to have to manually go through all 657 of my books individually.

Re-adding as suggested above may minimize the pain.

Good Luck.

[Worldwalker]

Originally Posted by nynaevelan

This would not be a problem if Calibre did not have the annoying habit of adding the id to the folder name, a behavior I wish I could stop.

You should not be tampering with calibre's private files. If you need the specific file for some reason, just have calibre export it to wherever it needs to be.

[nynaevelan]

Thank you I will give your suggestion a try. As to the assumption that I did something to the files is just that AN ASSUMPTION. For the record my hard drive failed and I had a backup of the files from their original location not the location that calibre was using them in, so I have to rebuild from the copies that I have. While I appreciate the assistance, please do not assume I have done something that I should not have.

I still believe it would be nice if Calibre had some kind of restore feature or a possibility to look in another location for the files and rebuild from that, especially since it gives you the option to export the metadata info with the book files. But lesson learned, once I have fixed them all, I will just setup my backup program to automatically keep backups of the files to prevent future problems.

[chaley]

Originally Posted by nynaevelan

I still believe it would be nice if Calibre had some kind of restore feature or a possibility to look in another location for the files and rebuild from that, especially since it gives you the option to export the metadata info with the book files. But lesson learned, once I have fixed them all, I will just setup my backup program to automatically keep backups of the files to prevent future problems.

Calibre does have a restore feature, since version 0.7.21. It is able to reconstruct the metadata.db file from the books in the library, using the OPFs that it automatically creates when the metadata has changed. See the command line command calibredb restore_database. The command assumes that it is operating in a calibre library folder hierarchy.

That said, it sounds like you have a copy of your library. Why not tell calibre to open that copy, using the switch library dialog? Have I missed something? Edit: re-reading the posts tells me that you don't have a backup per se, but instead the original sources. Clearly opening these as a library won't work.

[nynaevelan]

Originally Posted by chaley

Calibre does have a restore feature, since version 0.7.21. It is able to reconstruct the metadata.db file from the books in the library, using the OPFs that it automatically creates when the metadata has changed. See the command line command calibredb restore_database. The command assumes that it is operating in a calibre library folder hierarchy.

That said, it sounds like you have a copy of your library. Why not tell calibre to open that copy, using the switch library dialog? Have I missed something? Edit: re-reading the posts tells me that you don't have a backup per se, but instead the original sources. Clearly opening these as a library won't work.

I tried that but it did not work. I have copies of the metadata.db and opf files along with copies of the opf files for all the books and the original book sources, the only things I am missing are the directories with all the books in the place that Calibre is looking for them. I suppose I could start over and lose the date added info, something I would like to avoid. Tell me is it possible to remove all the books, re-add them as new additions and then import the metadata from the opf files, will this restore the original date added info along with retaining the saved searches and user categories??

What I am trying to do is retain my library with the customizations that I have created over the last several months. I am not an expert with Calibre so I do not know how to accomplish this, so I do appreciate any assistance you can offer as well as any advice you have on how to prevent this from happening in the future, when using backup programs what are all the files and locations I should be backing up so that I can restore everything should this happen again?

[nynaevelan]

Well I tried to re-add the books and although I had the option checked to merge the files automatically it did not link the new file types into the database. So, I deleted all the books and then re-added them. One good thing, it imported all the books and I was able to retain my original dates, I assume this is because this data is saved in the opf files. One bad thing, I lost all my covers, but this is not a major loss, I will fix them as I go along.

Thanks to all for your assistance.

Now off to setup my backup program to backup the directories so I do not need to go through this again. I did see some posts about dropbox, but I am a little hesitant about putting my books on the web so I need to research that before thinking about using something like that.

[Worldwalker]

Originally Posted by nynaevelan

As to the assumption that I did something to the files is just that AN ASSUMPTION.

True, but when you complain about how calibre names its private files, and state that you would prefer that it follow your preferences in naming them, it's a reasonable assumption that you're interacting with those files. Which, as I said, you shouldn't.

Covers are saved separately, I believe as .jpg files. If you can track them down, there's your errant covers. If your dead HD ate them, of course, you're back to retrieving/editing them.

As for Dropbox, I've been using it and so far it's been entirely private. I don't think I'd store credit card numbers there, but I have no worries about using it for books. If you want to try it, here's my referral link:
http://www.dropbox.com/referrals/NTgwODQwNDk5 (that gets us both some extra space).

[nynaevelan]

Originally Posted by Worldwalker

True, but when you complain about how calibre names its private files, and state that you would prefer that it follow your preferences in naming them, it's a reasonable assumption that you're interacting with those files. Which, as I said, you shouldn't.

Not necessarily, but that is a debate for another day. My only complaint in this is if the id was not included a new library could be created that would move the files to where they need to be for Calibre. I thought I was saving all the necessary files to restore the library but I did not realize I needed backups of those files also since I never go to that directory (see I normally do not mess with the files ). But now that I know the opf will retain the info I need, all is good because those are saved on two different drives.

Originally Posted by Worldwalker

Covers are saved separately, I believe as .jpg files. If you can track them down, there's your errant covers. If your dead HD ate them, of course, you're back to retrieving/editing them.

Yes I have a copy of most of them, the only ones I do not have are books I've added in the last week or so. I am in the process of finding the files that the batch download process was not able to find covers for and importing them into the db. I'll catch them all eventually.

Originally Posted by Worldwalker

As for Dropbox, I've been using it and so far it's been entirely private. I don't think I'd store credit card numbers there, but I have no worries about using it for books. If you want to try it, here's my referral link:
http://www.dropbox.com/referrals/NTgwODQwNDk5 (that gets us both some extra space).

Thanks for the info, I will look into this to see if it is something I would want to try. I am sure I will be back with questions regarding this if I decide to use it.

[Starson17]

Originally Posted by nynaevelan

My only complaint in this is if the id was not included a new library could be created that would move the files to where they need to be for Calibre.

Even if Calibre did not use the id in the folder name, you'd still be unable to easily recreate the library. Calibre modifies/shortens titles, it uses only one of the authors when there are multiple authors, etc. The id is essential to avoid duplicate folder names, so it's not likely to be removed.

[Starson17]

Originally Posted by nynaevelan

Well I tried to re-add the books and although I had the option checked to merge the files automatically it did not link the new file types into the database. So, I deleted all the books and then re-added them.

The automatic merge relies upon metadata for the new book to match the author and title to an existing record. The author has to match exactly, and the title needs to be close. Calibre can get the metadata 1) from inside the file, 2) from the filename, or 3) from an opf file. You need to select the best source for the metadata if you want the automatic merge option to work well. If it fails, you can always use the manual merge option. Just merge the newer books into the older records.

[nynaevelan]

Originally Posted by Starson17

The automatic merge relies upon metadata for the new book to match the author and title to an existing record. The author has to match exactly, and the title needs to be close. Calibre can get the metadata 1) from inside the file, 2) from the filename, or 3) from an opf file. You need to select the best source for the metadata if you want the automatic merge option to work well. If it fails, you can always use the manual merge option. Just merge the newer books into the older records.

I am not sure I understand you but there should be no problems with the import because the files were exported from Calibre on Saturday so they all should have matched perfectly. And after the import it said it merged them all, but none of them were linked into the new files although when I looked in the directory all the folders were created. Also, I am unsure what the correct settings should be, so maybe I did not have the settings setup correctly.

[Starson17]

Originally Posted by nynaevelan

I am not sure I understand you but there should be no problems with the import because the files were exported from Calibre on Saturday so they all should have matched perfectly.

You still need to have the setting correct for the source of the metadata. If you tell it to get it from filename, the regex needs to be correct.

And after the import it said it merged them all,

The message just tells you that it found matching records. It doesn't tell you if the matching record already had a matching format (which would have caused that file to be skipped - not merged)

It's possible that your starting database believed that the formats/files were there, when they were missing. Checking database integrity, and/or deleting formats would have fixed this. The automatic merge function will not add a format that already exists in the database.