Wiped my Calibre Libraries

[Pizza_Cant_Read]

Feeling like I deserve a Darwin Award here. I deleted my libraries and am trying to restore. First, I created a new library, next I plugged in my phone and connected as a wireless device. After the sync it showed all the titles on my device. I highlighted and right-clicked 'add to library' and from there it hangs at "Downloading books from device."

Just wondering if there are any known issues with this. So glad I had everything backed up on my phone :-D

[Pizza_Cant_Read]

Here is a sample error message as I didn't save the others. Its most frustrating because it takes so freaking long to time out. Earlier, I was able to "Save To Disk" but that doesn't give me covers or other metadata which is why I am trying the "Add to Library" option (to reconstruct the library..):

Device responded with incorrect information

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 87, in run
File "site-packages/calibre/gui2/device.py", line 521, in _prepare_addable_books
File "site-packages/calibre/devices/smart_device_app/driver.py", line 50, in _synchronizer
File "site-packages/calibre/devices/smart_device_app/driver.py", line 1568, in prepare_addable_books
File "site-packages/calibre/devices/smart_device_app/driver.py", line 50, in _synchronizer
File "site-packages/calibre/devices/smart_device_app/driver.py", line 1550, in get_file
File "site-packages/calibre/devices/smart_device_app/driver.py", line 50, in _synchronizer
File "site-packages/calibre/devices/smart_device_app/driver.py", line 633, in _call_client
File "site-packages/calibre/devices/smart_device_app/driver.py", line 675, in _receive_from_client
ControlError: Device responded with incorrect information


Fourth time letting it time out with same error:

calibre, version 3.11.1
ERROR: Error: Error communicating with device

Device responded with incorrect information

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 87, in run
File "site-packages/calibre/gui2/device.py", line 521, in _prepare_addable_books
File "site-packages/calibre/devices/smart_device_app/driver.py", line 50, in _synchronizer
File "site-packages/calibre/devices/smart_device_app/driver.py", line 1568, in prepare_addable_books
File "site-packages/calibre/devices/smart_device_app/driver.py", line 50, in _synchronizer
File "site-packages/calibre/devices/smart_device_app/driver.py", line 1550, in get_file
File "site-packages/calibre/devices/smart_device_app/driver.py", line 50, in _synchronizer
File "site-packages/calibre/devices/smart_device_app/driver.py", line 633, in _call_client
File "site-packages/calibre/devices/smart_device_app/driver.py", line 675, in _receive_from_client
ControlError: Device responded with incorrect information

[chaley]

The add-to-library was never intended to do mass uploads. You are using iOS CC so I can't go further than that.

Adding a book to the library via CC doesn't give you covers or CC metadata because the device interface doesn't support that. However, the metadata should be in the book itself because unless you explicitly turned it off, calibre updates the metadata in the book when it sends to the device. Try saving a book to disk then adding it to calibre, ensuring that the calibre option Preferences / Adding books / "Read metadata from contents rather than file name" is checked. You will get the best results from epub and azw3.

It would be better if you could copy the book files from your device to the computer. Is that possible with iOS? If so, then you can do the copy then add in bulk.

[Pizza_Cant_Read]

Well, you know, there is no documentation concerning that. If I made an application I would be very explicit about its limitations. Everyone knows that if uploading hundreds of documents to an (older generation) Kindle that it won't index well. If a developer says it wasn't 'intended' for mass uploads then why doesn't it says so in the documentation? I appreciate your response but it should definitely be clarified within the documentation somewhere if there are 'intended' limitations for the software.

I can download the book files but that doesn't assist me in repopulating the "series" data. Many series, after porting from a directory structure required me to manually research online to fill in the "series" column.

In my last post, I mentioned I was able to save books to a manually via "Save to Disk" but the whole reason I use calibre and CC is to utilize the covers alongside the other metadata. Without that I could just upload it to the cloud and call it a lost cause.

If the Android version is more capable then I have no problem switching. I use an iPhone because my wife loves Apple way too much.

It is funny because this reminds me of a python programming project in college when some friends and I were writing a classroom polling program; the intention was that the professor could ask a question (like something from the homework) and the class would enter their answer (ie A, B, C, D, E) and from the results our professor would know what he needed to review. But there was some problem with the way results were received by the server.

How many documents can I upload back to my computer at a time? I have already began to do the work from hand but how do you suggest I proceed?

Thanks,

-pcr

[chaley]

Originally Posted by Pizza_Cant_Read

Well, you know, there is no documentation concerning that. If I made an application I would be very explicit about its limitations. Everyone knows that if uploading hundreds of documents to an (older generation) Kindle that it won't index well. If a developer says it wasn't 'intended' for mass uploads then why doesn't it says so in the documentation? I appreciate your response but it should definitely be clarified within the documentation somewhere if there are 'intended' limitations for the software.

My apologies.

It is worth noting that CC is not a calibre backup mechanism.

I can download the book files but that doesn't assist me in repopulating the 'series' data. Many series, after porting from a directory structure required me to manually research online to fill in the 'series' column.

Then you must be using mobi or pdf, or you haven't changed the setting I mentioned in my last post.

In my last post, I mentioned I was able to save books to a manually via "Save to Disk" but the whole reason I use calibre and CC is to utilize the covers alongside the other metadata. Without that I could just upload it to the cloud and call it a lost cause.

CC uses cover thumbnails, not the actual cover.

If the Android version is more capable then I have no problem switching. I use an iPhone because my wife loves Apple way too much.

I don't know if it is. Testing uploading massive number of books isn't a priority for me.

How many documents can I upload back to my computer at a time? I have already began to do the work from hand but how do you suggest I proceed?

I don't know how many books can be uploaded at once before something goes wrong.

I assume that there isn't another way to get to the files stored by CC? In Android that is easy to do.

[Pizza_Cant_Read]

Can you tell me it is a known bug and it will be fixed in a future release? I first converted all the files to azw3 but when they uploaded apparently it only sent mobi files rather than everything. Until now I figured I could figure it out later because I now calibre has advanced features I am not yet familiar with :/

[chaley]

Originally Posted by Pizza_Cant_Read

Can you tell me it is a known bug and it will be fixed in a future release? I first converted all the files to azw3 but when they uploaded apparently it only sent mobi files rather than everything. Until now I figured I could figure it out later because I now calibre has advanced features I am not yet familiar with :/

There isn't a bug. By default calibre sends exactly one format for a book to a device, the first format on a list. For most devices that list is set in calibre itself. For CC that list is found in Settings / Formats and folders (probably just Formats in iOS CC). If mobi is above azw3 in the list then calibre will send mobi. It is possible to override that selection in calibre when sending a book, but that is a one-by-one operation.

Note that if you do send two formats of the same book to CC then they will appear as two books, not one book with two formats.

[Pizza_Cant_Read]

Thank you for letting me know that if I send two formats of the same book to CC they will appear as two books.It is useful information because as CC uses the same metadata as calibre I would expect it to behave smiliarly.

The only reason I mentioned it is because you said I would get the best results from epub and azw3 formats. Most of my books were available in epub and azw3. Yet they were transferred it mobi.

I am not sure how one can say this isn't a bug involved. When I use calibre to send documents to my kindle it sends all of them. When I use calibre to save documents from the library on my Kindle to my computer it does exactly that. But if I activate the wireless service, and calibre acknowledges my phone as a device just as it would a Kindle I don't get the same results.

[chaley]

Originally Posted by Pizza_Cant_Read

Thank you for letting me know that if I send two formats of the same book to CC they will appear as two books.It is useful information because as CC uses the same metadata as calibre I would expect it to behave smiliarly.

This is the same behavior one sees with any device. Reason: calibre identifies books on a device by its filename. The filename includes the extension. Two different formats for the same book will have different extensions therefore different filenames, so they are two different books.

The only reason I mentioned it is because you said I would get the best results from epub and azw3 formats. Most of my books were available in epub and azw3. Yet they were transferred it mobi.

Epub and azw3 can contain the calibre metadata in the book. Mobi cannot. If mobi was transferred then it is because mobi has a higher priority.

I am not sure how one can say this isn't a bug involved. When I use calibre to send documents to my kindle it sends all of them. When I use calibre to save documents from the library on my Kindle to my computer it does exactly that. But if I activate the wireless service, and calibre acknowledges my phone as a device just as it would a Kindle I don't get the same results.

If you use calibre's send-to-device to send something to a device, any device, it sends exactly one format for each book, the highest in the list of acceptable formats. From calibre's point of view there is no difference between a kindle, the wireless device, or some other ereader. As far as calibre is concerned they are the same kind of thing -- a device.

If you don't get the same results then you aren't doing the same thing.

[Pizza_Cant_Read]

Originally Posted by Pizza_Cant_Read

Can you tell me it is a known bug and it will be fixed in a future release? I first converted all the files to azw3 but when they uploaded apparently it only sent mobi files rather than everything. Until now I figured I could figure it out later because I now calibre has advanced features I am not yet familiar with :/

I fear that amid beers and banging my head on the keyboard something here was lost.

Right next to "Library" in the top toolbar is "Device". After the initial sync it shows all of the documents I have in Calibre Companion. Just as with any other device I expect to be able to highlight those documents, right click and choose "Save to Library". Instead it hangs at 0% for <30 minutes before displaying the error I provided in an earlier post. Meanwhile Activity Monitor shows the process eating 100% of the CPU's resources. And that is after my phone is disconnected. I can't even kill it. Something weird is going on somewhere.

Any tips on how to get the meta data (specifically the series information) from CC and back in calibre? Because of limited time, it will take me weeks to find sources, rename them within calibre, repopulate the series column plus any conversion. Any advice you can offer on how I can do this more easily would be very much appreciated!

<3 Thanks <3

-pcr

[chaley]

Originally Posted by Pizza_Cant_Read

I fear that amid beers and banging my head on the keyboard something here was lost.

Right next to "Library" in the top toolbar is "Device". After the initial sync it shows all of the documents I have in Calibre Companion. Just as with any other device I expect to be able to highlight those documents, right click and choose "Save to Library". Instead it hangs at 0% for <30 minutes before displaying the error I provided in an earlier post. Meanwhile Activity Monitor shows the process eating 100% of the CPU's resources. And that is after my phone is disconnected. I can't even kill it. Something weird is going on somewhere.

It seems like the problem is on the calibre side. Calibre does use lots of resources when adding multiple books to a library, and it can take time. Calibre also does a lot of upfront processing before starting to add any books. That is why a standard calibre "hint" is to add books a few at a time. If calibre takes too long then CC will time out.

Questions:

1) Have you tried adding the books to your library 1 or 2 at a time? This could avoid calibre spending a lot of time upfront, thus avoiding the timeouts.

2) Have you tried using calibre's device view "Save to disk" (not the one in the library view) to copy the book files from your iOS device onto your calibre computer? If this works (and it might because it avoids the add-to-library step) then once you have the books you can add them directly to calibre. I just tried it with Android CC and 20 books. It worked fine, taking 1/4 second per book.

Any tips on how to get the meta data (specifically the series information) from CC and back in calibre? Because of limited time, it will take me weeks to find sources, rename them within calibre, repopulate the series column plus any conversion. Any advice you can offer on how I can do this more easily would be very much appreciated!

<3 Thanks <3

-pcr

There is no easy way that I know of to get an easy-to-use electronic copy of CC's metadata.

One possibility that on Android would be a rather large pain: in CC make a backup that contains the database. Copy that backup to the calibre computer. Open it, extract the database, then for each book extract the JSON-formatted metadata. I don't know if these steps are even possible on iOS CC.

Personally, I wouldn't bother with all the faffing around. Given that you have all the metadata on the iOS device and given that you can easily see it, I would do it by hand, assuming I can upload the formats as discussed above. First I would add the book files to calibre using the "use metadata in book file" option. Sort the list in calibre by title. Do the same sort in CC. Then for each book in calibre, find that book in CC (it should be easy if they are sorted the same), open CC book details, then enter the series metadata into calibre. Android CC supports swiping to see the book details of the "next" book. If iOS CC supports that (and I think it does), swipe to the next book. A down-arrow in calibre should take you to the same book. This process takes some seconds per book, which for 1,000 books translates to less than 2 hours.

BTW: do you happen to have a copy of calibre's metadata.db left over from the original disaster, even an older one? If so then you can recover more easily. What you would do is:

1. Create an empty calibre library. Close calibre.
2. Copy the recovered metadata.db to the empty calibre library.
3. Restart calibre. You should see all your books. Of course, the actual books aren't there. Calibre doesn't care.
4. Assuming you can recover the books using the device view "save to disk", then for each book file recovered, click on the same book in calibre then drag the book file onto the book details panel in calibre. That adds the book file to calibre's library using the existing book/metadata.

[HarryT]

I know it doesn't help you to say this, but I'm aghast that you did not have (multiple!) backups of your Calibre library. My Calibre library is one of my most valued possessions, and I have backups stored in multiple physical locations. When you do manage to re-build your library, for goodness sake back it up!

[Pizza_Cant_Read]

Thanks for your help guys. I thought I did have a backup on my phone but guess it doesn't work like that :-D But next time...next time I will back it up. calibre is enabling me to horde and organize on a whole new level. I'm going the way of you HarryT, no collections. I can find what I need using CC and then open it on my Kindle.

[DoctorOhh]

Originally Posted by Pizza_Cant_Read

Thanks for your help guys. I thought I did have a backup on my phone but guess it doesn't work like that :-D But next time...next time I will back it up. calibre is enabling me to horde and organize on a whole new level. I'm going the way of you HarryT, no collections. I can find what I need using CC and then open it on my Kindle.

Have you googled how to transfer files from your iphone to your PC? Seems there should be a way to do this without CC. Once on the PC you can add them to calibre.

[Pizza_Cant_Read]

Thanks again for all the help. I have the books themselves but they have spent their lives organized in directories. I am ashamed to say I have even more than was ever loaded in calibre so really my only problem is the metadata. I am confident that my sci-fi/fantasy novels and hopefully the literary classics will eventually be reorganized in calibre (spent about 5 hours the last couple days fixing titles/series information for Star Wars). But it may be the case that much of it will sit in my library with only title and author info xD

I haven't yet played with moving smaller batches of books using 'save to library' buuuuttt if it isn't going to give me metadata then as Chaley said, I might as well just get to work. Organizing isn't as fun as reading but I do enjoy the zen space I find when I get really into it, so there is that much going for it <3