Differences between library and reader: Force calibre to write metadata into ebook

[durtreg]

Hi,

For the third time now the combination of calibre/sony prs 650 have messed up my whole collection of books. I lost all bookmars and all my notes.

Over time the data in calibre starts to differ more and more from the data on the device. Double books show up even though there are no doubles on the device (I searched). Titles/authors start to differ.

In the end I cannot find the books on the device with the name I used to find them. This leaves me no choice as to delete all files on the device and write them again onto the device from the calibre library.

I the process I loose all my bookmarks/notes!

This is really annoying the hell out of me.

I don't know who exactly is the culprit (may be even the one in front of the monitor).

However I have some questions, and be grateful if someone could answer them:
1) is there a good way to cope with differences between calibre and the ebook reader?

i.E. is there a option which does "overwrite any ebook on the device with the data in the calibre library - preserving my notes/bookmarks"?

This would essentially replicate what is IMHO standard on virtually any sync tool I know:
A and B should be synced, so i.e. Firefox Sync allows me to define "A overwrites B", as well as "B overwrites A".


2) Keep metadata and opf/calibre data in sync
The FAQ says, that calibre can write metadata to pdf. Unfortunately it doesn't write the metadata into the pdf.

Neither in Linux nor in Windows am I able to force my carefully edited Titles/Authors from calibre into the pdf.
=> is there an option which does: "write all metadata of the selected books as they are in calibre into the pdf/epub"?

If these options are available in calibre and I simply haven't found them, then sorry. I spent a lot of time searching the calibre website as well as google to have an answer. However either I asked the wrong question or nobody had the same problem.
This seems strange as metadata-sync seems so basic that this should be working fine.

I really don't want to have to use Adobe Digital Editions as it's not Free Software and doesn't work with Linux.

Thanks to anyone helping me to use calibre.

[Manichean]

I don't own a Sony and thus don't know how Calibre interacts with them, but I'd advise you to additionally post what setting you selected for your metadata management (found in preferences->send to device). That'll help others find out what the problem is.

Originally Posted by durtreg

2) Keep metadata and opf/calibre data in sync
The FAQ says, that calibre can write metadata to pdf. Unfortunately it doesn't write the metadata into the pdf.

Neither in Linux nor in Windows am I able to force my carefully edited Titles/Authors from calibre into the pdf.
=> is there an option which does: "write all metadata of the selected books as they are in calibre into the pdf/epub"?

Calibre writes the metadata to the file

• when the book is being saved to disk
• when the book is being sent to a device
• when the file is converted
• when the book is downloaded via the content server

That said, not all formats support all metadata fields. Also, PDF seems to be somewhat difficult regarding metadata. If possible, try another format supported by your reader.

[durtreg]

Hi,
That was quick.

Settings on Windows (at work) are:
Calibre > Preferences > Sending books to devices:
- Metadata management: automatic management
- Save template: standard ({author_sort}/{title} - {authors})

Calibre > Preferences - Saving books to disk
all checkboxes set, except Replace space with underscores, change paths to lowercase

Reading your post, the option most suitable for me seems "when the book is being saved to disk".
- When does this happen?
- What I get by right-clicking on a book, selecting "save to disk" I get a dialog which allows me to export a book, but not to save it to the library?
- can I force all books to be rewritten?

About metadata: I would prefer to use the original (which mostly is pdf). Pdf does have a title, author information in the file.

Thank you

[DoctorOhh]

Welcome to Mobileread.

Originally Posted by durtreg

Over time the data in calibre starts to differ more and more from the data on the device. Double books show up even though there are no doubles on the device (I searched). Titles/authors start to differ.

How do you get books from calibre to your device?

Originally Posted by durtreg

2) Keep metadata and opf/calibre data in sync
The FAQ says, that calibre can write metadata to pdf. Unfortunately it doesn't write the metadata into the pdf. Neither in Linux nor in Windows am I able to force my carefully edited Titles/Authors from calibre into the pdf.

First, I believe some types of PDF files don't seem to accept changes using calibre's metadata writer. However most PDF files can be updated. Do you have the pdf metadata writer plugin enabled?

Originally Posted by durtreg

=> is there an option which does: "write all metadata of the selected books as they are in calibre into the pdf/epub"?

Update: I see you already answered this question.

Yes. Do you only use calibre to manage your books on your device? Do you have metadata management set to Automatic management (see attached). Also if you set this to automatic examine the Sony Device interface plugin to see what you can customize (see attached).

Originally Posted by durtreg

I really don't want to have to use Adobe Digital Editions as it's not Free Software and doesn't work with Linux.

The Linux is a sticking point but it is free to use in Windows. But 100s of thousands of Sony users use calibre without your problems so there is hope that we will be able to help you out.

Hopefully someone with further insight or the right questions can clear things up.

[Manichean]

Originally Posted by durtreg

Reading your post, the option most suitable for me seems "when the book is being saved to disk".
- When does this happen?
- What I get by right-clicking on a book, selecting "save to disk" I get a dialog which allows me to export a book, but not to save it to the library?
- can I force all books to be rewritten?

Ah, allow me to clarify. I am assuming, as is the intention of Calibre, that you use the program to transfer books to your device (as in you plug in the reader via USB and transfer the books using the "send to device" function). If you do that, your metadata should, at the very least, reflect the state at the time you transferred the book (automatic metadata management may vary that, but as I said before, I have no experience and thus will not elaborate). The copy inside the Calibre library is considered to be outside the users' direct reach for the purpose of using Calibre. Books should only be accessed using the content server, the send to device function or the save to disk function. Save to disk is considered to be a method to export the book to a hard disk that is no device. You cannot, to my knowledge, force all the library books to have their metadata updated unless you were to export your entire library using the save to disk function, in which case the exported files would be outside Calibre's reach, so you'd have to update their metadata (through overwriting during a re-export or something other) manually every time you edit the metadata inside Calibre.

dwanthny has a good questions: You do actually use Calibre to send your books to device, do you? And do you have the metadata writer plugin for PDF enabled (see preferences->plugins)? If so, considering that your metadata management is set to automatic, it is my understanding that Calibre would automatically keep the metadata on your device current every time you plug it in. (Again, I've never used the function... but that's what the help text says )

[jackie_w]

Sony readers are not quite the same as other readers. All the metadata for your books (authors, title etc) is stored in a database file on the reader (FYI it's called media.xml). It doesn't much care what filename you gave your books in the Send-to-device template but it does store the filename in the database.

If you want to make sure that this database always reflects what you currently have in Calibre (without any need to resend to device) then the way forward is a Metadata Plugboard for the Sony. Every time you connect, Calibre updates the Sony database with the latest metadata no matter which book format (epub, pdf, rtf, txt). I don't use Notes and bookmarks much, but as far as I can tell they don't get deleted if you're using a Plugboard.

If you would like some help setting one up I'm volunteering to help. Just let me know.

P.S. You can also maintain Sony Collections with Calibre. This is entirely separate from the Plugboard.

P.P.S You can also maintain Sony thumbnails with Calibre. This is entirely separate from the Plugboard and Collections.

[durtreg]

Originally Posted by dwanthny

Welcome to Mobileread.

How do you get books from calibre to your device?

First, I believe some types of PDF files don't seem to accept changes using calibre's metadata writer. However most PDF files can be updated. Do you have the pdf metadata writer plugin enabled?

The plugin is enabled: Metadata writer plugins > Set pdf metadata (0.7.37) by Kovid Goyal

What I do to test writing to pdf is:
1) change the title of say book X in calibre, then I quit calibre, go to its library Folder on the computer, find book X, right click and have a look at it's properties "title" in Windows explorer as well as my pdf Reader. The title stays empty.

2) Do the same, but instead of quitting calibre I just send book X to the device again (it was already there)
Now I have two books on the device with different names (this might be the reason for my disappearing bookmarks/notes). The new one has indeed the changed pdf title (as opposed to the local file in the calibre library on the computer)

The pdf version varies across my library. I tested this on 1.4 and 1.5 version of unencrypted pdf with no security enabled.

My expectation is that the pdf title property is changed on the local pdf

About how I send books to my device:
Right-Click > Send to device > Send to storage card B

Here indeed the pdf properties are NOW changed (this is the first time I do it with the windows version of calibre at work). This is good news. I will look into the matter as why this did not happen the first time when I did it with linux but only when I get home.

I truly hope that finally the sony reader will not again start to differ from my calibre library.

However some questions remain:
1) why isn't the local library changed (the pdf property on the PC). The name of the pdf is changed. I would expect the properties also to change or at least have the option to write all properties into the files

2) Is there a way to perserve notes/bookmarks across a complete delete/resend of my library onto the device? I am not convinced that the books on the device and the library will not again start to differ. Maybe another Sony user can help me out?

Thank you

[Manichean]

Originally Posted by durtreg

1) change the title of say book X in calibre, then I quit calibre, go to its library Folder on the computer, find book X, right click and have a look at it's properties "title" in Windows explorer as well as my pdf Reader. The title stays empty.

Read the above posts explaining when Calibre updates the metadata inside the files themselves. This is expected behaviour. Actually, you're not even supposed to care about the files inside the Calibre library, instead interface with it using either the GUI or the CLI tools.

2) Do the same, but instead of quitting calibre I just send book X to the device again (it was already there)
Now I have two books on the device with different names (this might be the reason for my disappearing bookmarks/notes). The new one has indeed the changed pdf title (as opposed to the local file in the calibre library on the computer)

If you changed the title, this is expected behaviour as well: Calibre computes the path according to the template you specified in the send to device- options. If the path is dependent on the title and you change the title, the old file will not get overwritten, instead a new one will additionally be written. That file, as you have seen, will have the updated metadata.

However some questions remain:
1) why isn't the local library changed (the pdf property on the PC). The name of the pdf is changed. I would expect the properties also to change or at least have the option to write all properties into the files

Because of the Calibre design philosophy. The library is seen as a database without user access.

[durtreg]

Firstly, thank you all. This board is indeed very helpful.

Sony readers are not quite the same as other readers. All the metadata for your books (authors, title etc) is stored in a database file on the reader (FYI it's called media.xml). It doesn't much care what filename you gave your books in the Send-to-device template but it does store the filename in the database.

Very interesting. That could mean that the differences between the reader and calibre over time have their origin in some Sony-buggy-way of messing with this XML and calibre not overwriting it.

Therefore I would be very interested in the following suggestion:

If you want to make sure that this database always reflects what you currently have in Calibre (without any need to resend to device) then the way forward is a Metadata Plugboard for the Sony. Every time you connect, Calibre updates the Sony database with the latest metadata no matter which book format (epub, pdf, rtf, txt). I don't use Notes and bookmarks much, but as far as I can tell they don't get deleted if you're using a Plugboard.

If you would like some help setting one up I'm volunteering to help. Just let me know.

Notes and bookmarks are very important to me. Learning spanish and marking the words I don't know and such.

[durtreg]

Originally Posted by Manichean

Read the above posts explaining when Calibre updates the metadata inside the files themselves. This is expected behaviour. Actually, you're not even supposed to care about the files inside the Calibre library, instead interface with it using either the GUI or the CLI tools.


If you changed the title, this is expected behaviour as well: Calibre computes the path according to the template you specified in the send to device- options. If the path is dependent on the title and you change the title, the old file will not get overwritten, instead a new one will additionally be written. That file, as you have seen, will have the updated metadata.


Because of the Calibre design philosophy. The library is seen as a database without user access.

I understand.

However, the first time the books were sent to the device (on linux-calibre), the properties were not on the device (I will look into this when I get home to my linux machine). They should have been handled the same way as here at my work computer - the right way. I will check if this is the result of some difference in settings between the two calibre installations or indeed a bug in the Linux implementation of the pdf writer.

I am now convinced that the behaviour I see is a result of the above mentioned Sony-XML-Metadata getting out of sync with calibre. It would explain other things like disappearing books on the device.

[Manichean]

Originally Posted by durtreg

However, the first time the books were sent to the device (on linux-calibre), the properties were not on the device (I will look into this when I get home to my linux machine). They should have been handled the same way as here at my work computer - the right way. I will check if this is the result of some difference in settings between the two calibre installations or indeed a bug in the Linux implementation of the pdf writer.

You're right, sending the books to device via Calibre ought to have updated the metadata even on Linux. I can't help you with that, though, seeing as I don't run Linux. Checking the config may be a good idea, though.

I am now convinced that the behaviour I see is a result of the above mentioned Sony-XML-Metadata getting out of sync with calibre. It would explain other things like disappearing books on the device.

Several things come to mind here: If you have automatic metadata management turned on, Calibre will change the collections if you edit certain metadata fields. This might make books disappear from one collection, thus seemingly getting lost. Another thing is that, if you have two independent Calibre libraries set up on your home computer and your work computer (by independent I mean not synced by, say, Dropbox or something similar) then the internal book IDs may differ in between the libraries. I don't know how Calibre determines which book is which, but I have a feeling that the ID might be involved. Thus, the two installations would probably get confused about which book is which.

[jackie_w]

These are some instructions to get a basic Metadata Plugboard specifically for a Sony.

Goto Preferences - Metadata Plugboard

In 'Add new plugboard',

in the 'Format' box, select device_db from the menu.
In the 'Device' box select PRS505 from the menu (I know you have a PRS650 but PRS505 is used for all Sonys from PRS505 to-date).

Based on the fact that you have used {author_sort} rather than {authors} in your Save Template I'm assuming you'll want to do the same in the Plugboard so ...

Set up 3 Source template/Destination field entries:

1. Source tmplt = {author_sort}
   Dest field = {authors}
   
   
2. Source tmplt = {title}
   Dest field = {title}
   
   
3. Source tmplt = {title}
   Dest field = {title_sort}

Click 'Save plugboard' button - Apply - Close.

Now each time you connect the Sony to calibre, Calibre will update the Authors field in the Sony database to Calibre's {author_sort} field for each book on the device. It also does the same when you send a new book to the Sony.

You now have a basic Sony Plugboard. Set this up first to make sure there are no problems and report back.

It may not be very exciting so far, but next we can tweak it to suit your tastes for display and sorting of Titles.

[Edit: I've added another couple of basic entries in the Plugboard for the Title and TitleSorter fields in the Sony database. If they are missing I'm not 100% sure whether Calibre would automatically update the fields each time you connect. I suspect only developer chaley knows the answer to that.]

[DoctorOhh]

Something else to consider if you are sharing one database/library between these two operating systems. Windows and Linux operating systems deal with the files differently. The default setting for Windows is case-insensitive and Linux is case-sensitive. This can only cause database confusion because The Exorcist is different from The exorcist. If you added it in one OS then changed the name (case) in the other OS you could end up with two database entries?

I don't know exactly the full ramification, but only bad things can come of it. Things that other people don't experience. Things that will drive you nuts because everyone else will say X and you are experiencing Y and don't understand why your system is acting different.

[jackie_w]

Excellent point, dwanthny.

[durtreg]

Originally Posted by dwanthny

Something else to consider if you are sharing one database/library between these two operating systems. Windows and Linux operating systems deal with the files differently. The default setting for Windows is case-insensitive and Linux is case-sensitive. This can only cause database confusion because The Exorcist is different from The exorcist. If you added it in one OS then changed the name (case) in the other OS you could end up with two database entries?

I don't know exactly the full ramification, but only bad things can come of it. Things that other people don't experience. Things that will drive you nuts because everyone else will say X and you are experiencing Y and don't understand why your system is acting different.

Good point. However being of the careful sort after having been burned the first time I took care to have only *one* PC (the linux pc) from/to which I synced my reader.

This means that this is the first time that I synced at work with Windows since the last meltdown.

I just copied the Linux-Calibre-Library onto this Windows PC-Calibre Library and deleted again all books. Did a complete "send to device" and now the pdf properties are on the device.

The disappearing books .. well I guess if a book completely disappears from the xml Sony uses then it might not show up in the list, or (as the pdf properties were previously not written) the name of the book might have changed to something entirely unrelated to the book and I could not find it.

Strange was, that using the notes I was able to switch back to the book on the reader - so it was still there, somewhere.

Now I believe (thanks to you all) that the solution is at hand.

If the reader decides to unmotivatedly change some Titles/Authors it can only do so by using the pdf filename or the pdf metadata. These are now in sync with the calibre library. So I will be able to find them again.

The plugboard should get rid of the annoying differences over time between reader/calibre. I just overwrite it every time.

The bookmarks/notes will not be touched by either calibre or the sony xml. So I hope to not loose them.

If I again have to do a delete/resend from calibre then I will take care to not delete the xmls on the device. With some luck the bookmarks will work afterwards.

I will write back when at my linux pc, when I am able to test the sync and the plugboard.

Thank you all