Horror- Just lost 95% of my tags - help!

westofpecos · 04-26-2014, 04:31 AM

I had been gently testing calibre for the past few months with a small number of titles, to figure out if it would be able to manage the thousands of ebooks I am using as sources for a book I am writing. Last week I concluded enthusiastically that it would be perfect for the task.

So, over the past two weeks, I imported about 1500 ebooks into calibre, and spent tens of hours tagging them appropriately to what I use them for. To make sure that I would not have backup issues, I mounted the calibre library in a dropbox folder. I have about 20G of data, and Dropbox is all sync'ed.

This evening, I completed all the tagging I needed to do (a lot of hard work, it is 3:00am as I write). I checked the database as I always do every few hours, had calibre delete a single folder that did not have books showing in the db (it was empty), closed and reopened calibre, had a last look at the book list, and found that 95% of my tags are gone.

I am horrified, as I spend probably 50 hours tagging all these books. What is weird is that some books I tagged a few minutes ago are still tagged. But the very large majority of the books have lost all tagging. Those that still have tags seem to have all the tags they had.

Beyond the db check that I just did, these are the two significant actions I did recently:
- I upgraded a few hours ago to 1.33 from 1.32
- After completing the individual tagging, I applied a tag to 90% or so of the books in the library. After doing so, all the books looked appropriately tagged (so the tagging operation did not destroy anything I had). That is when I checked the db. After that, I closed calibre, then reopened it. That's when I found practically all the tags gone.

Any thoughts? I can't believe I just wasted 50 hours of tedious work.

And - no, I did not do any backups, I counted on dropbox to protect me from problems.

Thanks for your help!

chaley · 04-26-2014, 05:12 AM

Check dropbox to see if you have a conflicted copy of metadata.db in your library. If so then try renaming it back to metadata.db to see if it contains your tags.

In general it is not a good idea to have dropbox running while you are using calibre to make changes. Dropbox and calibre can fight over files being changed and usually dropbox wins, causing calibre to lose changes.

DoctorOhh · 04-26-2014, 06:43 AM

Quote:

Originally Posted by westofpecos

And - no, I did not do any backups, I counted on dropbox to protect me from problems.

As chaley stated Dropbox may have caused your issue, but it should bail you out as well. If you don't have a conflicted copy then just restore (via dropbox) previous versions of the metadata.db one at a time until you have one that shows your tags.

kacir · 04-26-2014, 07:18 AM

Quote:

Originally Posted by westofpecos

And - no, I did not do any backups, I counted on dropbox to protect me from problems.

Dropbox is like having a mirror. That doesn't protect you from doing something stupid with your database. it only protects you from a hardware failure.
You have to make a separate backup before any significant change in medatada, such as deleting large number of unwanted things or large search and replace.
You should also make a backup (not mirror and not live dropbox copy) every time you finish making significant change that would take many hours to recreate.

cybmole · 04-26-2014, 07:36 AM

I put a backup of the calibre library into dropbox. I tehn use a sync program - freefile sync to mirror my calibre library to my dropbox library whenever I am done with changing stuff in calibre.

I only run sync wien calibre is not active.

that way I have a real backup, both locally in dropbox folder, and in cloud.

Divingduck · 04-26-2014, 01:44 PM

In addition, if your book directories in Dropbox are ok, you can try to rebuild the Calibre database file metadata.db with Restore database. You should do this on a local drive.

westofpecos · 04-26-2014, 11:23 PM

Thanks for all the very helpful input.

I had no conflicting copies. I restored an older version of the db (through Dropbox) that showed a lot more content (per its size), and, weirdly, it lost all of its tags again

.

I ended up restoring a version that was an hour or so older. Once the db was restored, I checked it with Caliber, then re-imported by hand every folder that showed in the folder list but not in the book list, It was quite painful. Then I reentered the missing tags by hand - altogether an afternoon of work, but a lot less than I had invested before, thanks to the forum feedback, which made me realize that all the metadata is in the single db file.

I can't help wondering what caused the tag corruption. Looking at what tags got corrupted, I cannot figure out any way that I could have somehow screwed them up through the graphical interface, which is what I was using. I am worried that this might happen again...

Clearly I am going to be pretty anal about hand backups from here on. I could not find a simple way to set up calibre for automated backups of the some kind - did I miss something in the manual? It's very easy to do it by hand, though.

meeera · 04-26-2014, 11:57 PM

Are you not automatically backing up your whole hard drive regularly anyhow? You should be.

DoctorOhh · 04-27-2014, 12:47 AM

Quote:

Originally Posted by westofpecos

I can't help wondering what caused the tag corruption. Looking at what tags got corrupted, I cannot figure out any way that I could have somehow screwed them up through the graphical interface, which is what I was using. I am worried that this might happen again...

If Dropbox is the only other program accessing your library then simply pause syncing of dropbox while you are editing metadata or converting books and you should be fine.

johnoz123 · 04-27-2014, 02:35 AM

Quote:

Originally Posted by DoctorOhh

If Dropbox is the only other program accessing your library then simply pause syncing of dropbox while you are editing metadata or converting books and you should be fine.

Good advice.

Also, Dropbox can take longer than you might think to upload edited Calibre files. So you also need to be 100% sure any syncing is finished before you start editing again. I did some serious damage to my library by NOT following this.

cybmole · 04-27-2014, 06:50 AM

Quote:

Originally Posted by johnoz123

Good advice.

Also, Dropbox can take longer than you might think to upload edited Calibre files. So you also need to be 100% sure any syncing is finished before you start editing again. I did some serious damage to my library by NOT following this.

yes, I see Drobox at a slow crawl but uploading & downloading onto my other PC - with fast PC, fast broadband - I get it's server load issue at Dropbox

I've seen a download of ~100 changes to calibre library backup , going onto my other PC, from Dropbox, take 10 minutes to complete.

04-26-2014, 04:31 AM	#1
westofpecos Member Posts: 14 Karma: 10 Join Date: Apr 2014 Location: Wisconsin Device: Kindle 3, Kindle Touch, Kindle PW2, Nexus 7, Nexus 5	Horror- Just lost 95% of my tags - help! I had been gently testing calibre for the past few months with a small number of titles, to figure out if it would be able to manage the thousands of ebooks I am using as sources for a book I am writing. Last week I concluded enthusiastically that it would be perfect for the task. So, over the past two weeks, I imported about 1500 ebooks into calibre, and spent tens of hours tagging them appropriately to what I use them for. To make sure that I would not have backup issues, I mounted the calibre library in a dropbox folder. I have about 20G of data, and Dropbox is all sync'ed. This evening, I completed all the tagging I needed to do (a lot of hard work, it is 3:00am as I write). I checked the database as I always do every few hours, had calibre delete a single folder that did not have books showing in the db (it was empty), closed and reopened calibre, had a last look at the book list, and found that 95% of my tags are gone. I am horrified, as I spend probably 50 hours tagging all these books. What is weird is that some books I tagged a few minutes ago are still tagged. But the very large majority of the books have lost all tagging. Those that still have tags seem to have all the tags they had. Beyond the db check that I just did, these are the two significant actions I did recently: - I upgraded a few hours ago to 1.33 from 1.32 - After completing the individual tagging, I applied a tag to 90% or so of the books in the library. After doing so, all the books looked appropriately tagged (so the tagging operation did not destroy anything I had). That is when I checked the db. After that, I closed calibre, then reopened it. That's when I found practically all the tags gone. Any thoughts? I can't believe I just wasted 50 hours of tedious work. And - no, I did not do any backups, I counted on dropbox to protect me from problems. Thanks for your help!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Horror Freeman, Mary E. Wilkins: The Lost Ghost and Other Horror Stories. V1. 4 Dec 22012	crutledge	ePub Books	0	12-04-2012 01:15 PM
Horror Freeman, Mary E. Wilkins: The Lost Ghost and Other Horror Stories. V1. 4 Dec 22012	crutledge	BBeB/LRF Books	0	12-04-2012 01:13 PM
Lost Tags	ccowie	Calibre	2	04-06-2010 11:22 PM
PRS-505 All tags lost - Backup?	Bitman	Sony Reader	3	03-24-2010 09:31 PM
PRS-700 lost its tags	metageek	Calibre	4	08-13-2009 09:14 AM

04-26-2014, 05:12 AM	#2
chaley Grand Sorcerer Posts: 12,447 Karma: 8012886 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	Check dropbox to see if you have a conflicted copy of metadata.db in your library. If so then try renaming it back to metadata.db to see if it contains your tags. In general it is not a good idea to have dropbox running while you are using calibre to make changes. Dropbox and calibre can fight over files being changed and usually dropbox wins, causing calibre to lose changes.

04-26-2014, 07:36 AM	#5
cybmole Wizard Posts: 3,720 Karma: 1759970 Join Date: Sep 2010 Device: none	I put a backup of the calibre library into dropbox. I tehn use a sync program - freefile sync to mirror my calibre library to my dropbox library whenever I am done with changing stuff in calibre. I only run sync wien calibre is not active. that way I have a real backup, both locally in dropbox folder, and in cloud.

04-26-2014, 01:44 PM	#6
Divingduck Wizard Posts: 1,166 Karma: 1410083 Join Date: Nov 2010 Location: Germany Device: Sony PRS-650	In addition, if your book directories in Dropbox are ok, you can try to rebuild the Calibre database file metadata.db with Restore database. You should do this on a local drive.

04-26-2014, 11:23 PM	#7
westofpecos Member Posts: 14 Karma: 10 Join Date: Apr 2014 Location: Wisconsin Device: Kindle 3, Kindle Touch, Kindle PW2, Nexus 7, Nexus 5	Thanks for all the very helpful input. I had no conflicting copies. I restored an older version of the db (through Dropbox) that showed a lot more content (per its size), and, weirdly, it lost all of its tags again . I ended up restoring a version that was an hour or so older. Once the db was restored, I checked it with Caliber, then re-imported by hand every folder that showed in the folder list but not in the book list, It was quite painful. Then I reentered the missing tags by hand - altogether an afternoon of work, but a lot less than I had invested before, thanks to the forum feedback, which made me realize that all the metadata is in the single db file. I can't help wondering what caused the tag corruption. Looking at what tags got corrupted, I cannot figure out any way that I could have somehow screwed them up through the graphical interface, which is what I was using. I am worried that this might happen again... Clearly I am going to be pretty anal about hand backups from here on. I could not find a simple way to set up calibre for automated backups of the some kind - did I miss something in the manual? It's very easy to do it by hand, though.

04-26-2014, 11:57 PM	#8
meeera Grand Sorcerer Posts: 5,814 Karma: 68407974 Join Date: Dec 2011 Location: Australia Device: Kobo Libra 2, iPadMini4, iPad4, MBP; support other Kobo/Kindles	Are you not automatically backing up your whole hard drive regularly anyhow? You should be.

Advert

Advert