Help copying from sony reader

[CraftingGrandma]

Here's what has happened. My calibre folder got deleted and I re-installed it and then proceeded to transfer all my books back from my sony to re-make it.

A good chunk of my books transferred over fine. The rest do not have the check mark that they are on the device. Their author and file name and covers tags and series did not copy with them and my calibre does not think they are the same file. Basically I think (and I may be using the wrong word here so please bear with me, but I tried to look around for the names of stuff to help you people help me) the meta data did not copy.

This old lady did some looking around in attempts to solve this problem myself and maybe learn something (and save some embarrassment) and thought I found the answer with the match book to library by right clicking on the books. It would only do one at a time, I figured slow was my punishment for deleting my folder and would just spend the time doing it until I noticed it changed the author on my sony to the messed up author from my calibre library.

So now that I am here asking for help I figured I would ask two things of you.

First, how do I make it copy the meta data (I hope that is the right word) covers, series, tags for a book in my calibre match the ones on my sony as opposed to my sony being matched to the calibre?

Second, is there a way to make it copy all of that stuff right from the start when I transfer books from my sony to my calibre so I do not have to match the books one by one?

[eschwartz]

When calibre first sends the books, all the metadata should be properly embedded in the book -- however, if you are using plugboards and such to apply metadata rules to the exported books, the embedded metadata might not match the library metadata. For example, I have a plugboard that automatically changes all titles to read "title (series, Book #)" -- which does not match the "title". So calibre also tracks the books on the device using a cache -- this will be a file on the device called "metadata.calibre"

Match books will tell calibre which device books map to which library books, and allow the metadata cache file to track them. This means calibre can better track which books are on the device already, and for certain devices, to manage metadata/collections automatically.


Getting those books back into calibre should be as simple as copying the books back into calibre, but depending on the ebook format, not all metadata may have been embedded in the book. I don't know how much metadata LRF for example supports, though EPUB should be fine.

Is the metadata for the books now in calibre consistent? That is to say, the messed up metadata -- is there a pattern to the madness? If so, we can probably fix that but I don't know any way to fix it if the books themselves don't have good metadata.

For the future, I would definitely advise backing up your library. One easy way would be to put it in Dropbox. (See the relevant information in my signature.)

[CraftingGrandma]

I apologize, but I did not follow much of that.

As far as I know the information is not messed up on the reader. As it shows up when I look at the connected device in calibre and can see the cover, the author, and tags.

I plug in my sony. Start calibre and then let it load the sony into calibre. I click on the devices button and transferred over the books. When it was done. there were about 400 that transfered over but do not have the check mark for on device and have all the missing and messed up information I talked about. When I click on device and find the same book missing it's cover tags and different author name in calibres main library I see everything there and the way it should be when looking at the book in the on device button. I can see the cover in the preview panel and it's tags.

Plus all the books on my sony were originally from calibre. So everything that was there came from calibre in the first place.

As for patterns. I notice that all the changed files tat are of a .txt nature have their titles changed to the entire first line of the book. and the author is changed to the entire second line.

.pdf has the title fine, but the author names are correct except they have a 3 digit number behind the correct name.

All of the files no matter what (at least the ones that did not transfer properly without that check mark for on device) are missing their covers, tags, and series. They are all missing/blank.

[JSWolf]

Plug the 650 into the USB. Once the computer sees the 650, using Windows Explorer, search for *.epub. Select all of the ePub and then drag/srop to your Calibre library. Do the same thing for PDF (search for *.pdf). That's all you need to do do get your eBooks from your 650 to your Calibre library.

[eschwartz]

Originally Posted by CraftingGrandma

I apologize, but I did not follow much of that.

As far as I know the information is not messed up on the reader. As it shows up when I look at the connected device in calibre and can see the cover, the author, and tags.

I plug in my sony. Start calibre and then let it load the sony into calibre. I click on the devices button and transferred over the books. When it was done. there were about 400 that transfered over but do not have the check mark for on device and have all the missing and messed up information I talked about. When I click on device and find the same book missing it's cover tags and different author name in calibres main library I see everything there and the way it should be when looking at the book in the on device button. I can see the cover in the preview panel and it's tags.

Plus all the books on my sony were originally from calibre. So everything that was there came from calibre in the first place.

As for patterns. I notice that all the changed files tat are of a .txt nature have their titles changed to the entire first line of the book. and the author is changed to the entire second line.

.pdf has the title fine, but the author names are correct except they have a 3 digit number behind the correct name.

All of the files no matter what (at least the ones that did not transfer properly without that check mark for on device) are missing their covers, tags, and series. They are all missing/blank.

TXT as a format does not support any metadata at all. If the books all have metadata in the filename, e.g. they are called "title - authors.txt" then you can delete those books, go to Preferences ==> Import/Export ==> Adding books, and uncheck "Read metadata from file contents rather than filename".
Then reimport all the TXT books, and they will be imported with that metadata at least. If the filenames are something different, tell me what they look like and I can give you a "regular expression pattern to use when trying to guess ebook metadata from filenames".

PDF is also a bit awkward in handling metadata. For those, I recommend just going to the Tag Browser (right side) and clicking Formats ==> PDF.
That will select all PDF books.
Then click CTRL+A (select all) and click the Edit Metadata toolbar. Select the last tab (Search and Replace) and enter the following settings:

Search Mode: Regular Expression
Search Field: Authors
Destination Field: Authors
Search for:

([^,]*)\d{3}

Replace with:

\1

That should remove all the three digit numbers at the end in bulk.

Finally, I would use the download metadata feature to download covers and series info. You can get tags too (personally, I have never found a useful tag )
Go to Preferences ==> Sharing ==> Metadata download, setup the right sources you trust (Goodreads is important, they should have series info) and then select all books and press CTRL+D (there should be a toolbar icon to start a Metadata download, but I always use the shortcut key ).

calibre will take a while and then popup with the results for you to approve one-by-one (or you can choose to accept all).


I don't know of any way to get that info from the Sony database.

[eschwartz]

Originally Posted by JSWolf

Plug the 650 into the USB. Once the computer sees the 650, using Windows Explorer, search for *.epub. Select all of the ePub and then drag/srop to your Calibre library. Do the same thing for PDF (search for *.pdf). That's all you need to do do get your eBooks from your 650 to your Calibre library.

There is no difference between adding from the device view and adding via Explorer.

[eschwartz]

Again, to keep this from happening in the future, back up your books. I suggest Dropbox, it is easy to use and I can shamelessly give you my referral link to sign up if you don't have an account already. (We both get a 500MB bonus if you use it.)

Then simply put your library in your Dropbox folder and it will be safe from accidental deletion (you get thirty days to undelete stuff from the Dropbox website).

[CraftingGrandma]

Thank you for your help so far eschwartz.

I'm still unclear on a few things.

You say that the txt files have no meta data stored with them, then why and where does it get all the tags and cover I see when I look at the same book in the connected device part of calibre? Is there no way to transfer over the information showed there?

And the same goes for the pdf file too. Your suggestion of downloading the tags was a good one, but the problem I run into is I have created many of my own tags that are not ones normally associated with the books I would like kept. Which is part of why I am so concerned with getting the tags to transfer over.

For example, one of the txt files. When I click on it in the calibre library it has no tags and no visible cover. Yet when I click on the same book in the device I can see the cover picture I originally set to the book when I first transferred it to my sony. I also see all of this in the preview panel under the cover


Series: Grandkids Poems
Tags: Poetry, Not Professional Poetry
Path: Click to open
D:\North, Judy\first poems, The - judy.txt
Collections: (Series)Grandkids Poems, Poetry, Not Professional Poetry



And what I see for the same book after I add the book to calibre and look at it from the library section is it with no cover displayed and this in the preview panel


Formats: TXT
Path: Click to open





I have been going at my problem under the assumption that if calibre can see the tags on the device and the cover that there is a way for it to be copied over I just do not know how to make it do it, but is it the case that it is actually a limitation of the calibre program that it can not transfer all that it can see and read from the books?

[eschwartz]

It is seeing that information from the Sony database. But I don't know of any way to copy that information off your Sony.

Yes, calibre can read it, it is possible... but I don't think anyone actually wrote any sort of functionality to do so.

Adding books works on the actual files, not the Sony database. The need to import metadata from that database was I guess never considered.


Hmm, perhaps it would be nice if it were possible to have calibre import books from Device view and use the metadata from that view -- it would make an intriguing feature request.

[chaley]

Originally Posted by eschwartz

It is seeing that information from the Sony database. But I don't know of any way to copy that information off your Sony.

Actually, the metadata is coming from the .metadata.calibre file on the device. The sony database doesn't contain tags.

Yes, calibre can read it, it is possible... but I don't think anyone actually wrote any sort of functionality to do so.

You are right, it is possible to import metadata from the .metadata.calibre file and therefore the associated device record, and you are right, no one has written that code.

One could convert the .metadata.calibre file into into individual .opf files for each known book on the device. These opf files could be combined with the book files (one per directory), permitting import of the book with the calibre metadata at the time the book was stored using existing calibre add books methods. Sounds like an opportunity for someone to develop a "related tool".

[eschwartz]

Ah, OK. I know the metadata.calibre file contains all kinds of stuff, I also believe calibre plugs directly into the Soy database -- I was unsure what was what (and somehow got the impression that that info was being seen in the device UI?)...

Same basic idea though.

[chaley]

Originally Posted by eschwartz

Ah, OK. I know the metadata.calibre file contains all kinds of stuff, I also believe calibre plugs directly into the Soy database -- I was unsure what was what (and somehow got the impression that that info was being seen in the device UI?)...

The most important thing that the Sony plugins pull from the sony database are the collections, which have no equivalent in calibre. IIRC the rest of the metadata comes from the file, but as it has been many years since I wrote much of that code I am not certain.

The file contains the complete calibre metadata in JSON format for each known book. I have written code to go from OPF to JSON, but have never had a need to go the other way.

[CraftingGrandma]

You two are talking way over this old lady's head.

But just to make sure I understand the few bits I think I do. There is no way to have all of the information in the sony .metadata.calibre (I think that is the file you two were saying contains all of the information I see in the preview panel form my sony when I click on a book) file to the claibre main library along with the book when you send them from it?

But there may be a way in the future if someone makes the program work that way?

And lastly, then my only option to get all of the stuff I see from my sony to the main library is to edit it all by hand one at a time?

Am I correct in all that I have gathered so far?

[eschwartz]

Correct.

[CraftingGrandma]

Thank you for all your help and taking the time to explain it to me.

Hopefully someday someone can make it happen in case this happens to anyone else. For now, I have a long time of rebuilding ahead of me.