lost metadata on pdfs

[slushie]

first, i have to say what a great suite of tools you've created with calibre! it is a wonderful testament to the power of open source software not to mention your endless patience. thanks!

i just got my prs505 a few days ago so i am still learning the ropes, so please bear with me if i misunderstand something...

in calibre i have a few books loaded and i've configured the automatic download of metadata via the gui. in the 'library' section this all looks perfect! however, when i send my library to the reader i am losing all my configured metadata... the reader sees whatever is A. inside the pdf's title/author slots (in acroread File>Properties dialog), or B. just the filename as title, with all spaces and non-alphanumeric chars turning into underscores and no author.

this problem only seems to happen with PDF's, and only when i transfer from my library to the hardware... if i click 'Card' and double-click an item in that list i can fix the metadata manually.

i'm running linux, just updated to the latest version of calibre (0.4.89). let me know if you need any more information, i'm happy to help you help me any way i can!

thank you!

[kovidgoyal]

There was some change in the latest firmware for the 505 that causes it to always rescan books and update the metadata in the cache (the file media.xml) for books in the main memory. This doesn't matter for LRF since calibre sets the metadata correctly in LRF. It doesn't do that for PDF because there isn't a python library that can do that and writing one is non-trivial

[slushie]

thanks for your quick response.

i see how updating the pdf would be the ideal solution, though nontrivial of course... still, would it be possible for calibre to set the metadata manually after the transfer -- i should say, emulating the action of me double-clicking each field and editing it by hand -- because that seems to update the information perfectly well for files on the card. i have been doing this myself but i have many pdfs and the reader loads pdf's *much* faster than lrf's, so i'd rather not convert them all.

also, i am not a python developer but am a programmer by trade. i would gladly submit a patch if you could give me a hint to where to find what i'm looking for in the source

thanks!

[kovidgoyal]

calibre does set the metadata manually after an upload, it's just that ever since the latest firmware update, the reader overwrites the metadata set by calibre with metadata read from the file. This only happens for books in the main memory.

If you care enough, you can try to figure out why it happens. metadata is stored on the device in the file media.xml in the main memory and cache.xml on the storage card.

Transfer a book using calibre. Dont disconnect the reader. See media.xml. Now disconnect. Let it rescan the collection. Reconnect and compare the new media.xml to the old one.

If you want to submit a patch for editing PDF metadata, you should submit one to the pypdf project (it's what calibre uses for PDF metadata handling)

[pilotbob]

Originally Posted by kovidgoyal

If you care enough, you can try to figure out why it happens. metadata is stored on the device in the file media.xml in the main memory and cache.xml on the storage card.

Can you flag the file as read-only somehow?

BOb

[kovidgoyal]

Originally Posted by pilotbob

Can you flag the file as read-only somehow?

BOb

No, the reader needs to be able to edit the file for storing bookmarks, history pagination information etc

[slushie]

Originally Posted by kovidgoyal

This only happens for books in the main memory.

i'm definitely seeing this issue on the SD card as well...

Originally Posted by kovidgoyal

If you care enough, you can try to figure out why it happens. metadata is stored on the device in the file media.xml in the main memory and cache.xml on the storage card.

i'm going to look into this today and see if i can begin to get a grasp of it.

thanks for the info!

[=X=]

Originally Posted by slushie

also, i am not a python developer but am a programmer by trade. i would gladly submit a patch if you could give me a hint to where to find what i'm looking for in the source

thanks!

There is a Perl library called PDF::API2 that allows you to create and edit (including metatdata) PDF from a perl script.

You might want to try it out.

=X=

[pilotbob]

Originally Posted by =X=

There is a Perl library called PDF::API2 that allows you to create and edit (including metatdata) PDF from a perl script.

You might want to try it out.

=X=

Heh... just what calibre needs, another dependeny.

The page for it doesn't say anything about its license. Can I assume it is open source since it is hosted on sourceforge?

BOb

[kovidgoyal]

Originally Posted by pilotbob

Heh... just what calibre needs, another dependeny.

The page for it doesn't say anything about its license. Can I assume it is open source since it is hosted on sourceforge?

BOb

calibre is *not* going to embed a perl/java/ruby interpreter.

[slushie]

im a perl developer and the first thing i did was check CPAN to see if someone had a solution... and yes, PDF::API2 would seem to be a very complete pdf solution... but its rather huge, as a matter of fact. i think rather than try to put all of that alongside a big python app it would be better either to port the relevant parts of the library to python, or try to install sony's software in windows someplace and do some reversing to fix this issue... both seem to be interesting, but neither are not gonna be finished tonight ;]

edit: oh btw, that perl library is gpl2

[pilotbob]

Originally Posted by kovidgoyal

calibre is *not* going to embed a perl/java/ruby interpreter.

heh... I didn't think you would. Of course, Mac and Linux already have perl on them.

BOb

[=X=]

Originally Posted by pilotbob

.... Can I assume it is open source since it is hosted on sourceforge?

Yes it is GNU CopyLeft. I downloaded it from CPAN

Originally Posted by kovidgoyal

calibre is *not* going to embed a perl/java/ruby interpreter.

Well the post was really more intended for the developer who, had an itch to resolve his PDF problem.

But this does raise the opportunity to ask the question about calibre. As calibre continues to grow in popularity more and more are going to want to contribute to calibre. It would be nice if there was a plugin architecture to the tool where individuals could develop their component independently to the tool and have it seamlessly integrate to calibre.

Just a wish list I suppose

=X=

[kovidgoyal]

Originally Posted by =X=

Yes it is GNU CopyLeft. I downloaded it from CPAN



Well the post was really more intended for the developer who, had an itch to resolve his PDF problem.

But this does raise the opportunity to ask the question about calibre. As calibre continues to grow in popularity more and more are going to want to contribute to calibre. It would be nice if there was a plugin architecture to the tool where individuals could develop their component independently to the tool and have it seamlessly integrate to calibre.

Just a wish list I suppose

=X=

calibre does have a plugin architecture. For a new input format you need to write a plugin to convert format -> html for a new output format you need to write opf -> format. Once you do that, integrating it into calibre is trivial.

[=X=]

Originally Posted by kovidgoyal

calibre does have a plugin architecture. For a new input format you need to write a plugin to convert format -> html for a new output format you need to write opf -> format. Once you do that, integrating it into calibre is trivial.

Well not really, calibre has a modular design which makes it easy to add features. However this requires the 3rd party tool to get released with calibre. A plugin architecture allows indep developers to enhance the feature of a product without requiring a release or and update to include that plugin. A devloper can just install the plugin and integrate with the product.

Good examples is WordRMR or OOo FBTool that plugs into OpenOffice.

=X=