Source URL / Automated Download

[Demosthenex]

I've begun to use Calibre to manage my collection of IBM Redbooks. They are publicly available, and frequently change.

I currently have a shell script that remembers the source URL for a particular PDF, and I refresh my local copy at long intervals to keep up to date. It is a manual process, I just don't have to navigate the website and manage downloads.

Is there a way to have Calibre fetch a PDF from a source URL, and remember it in metadata to facilitate refreshing the copy later? Manually initiated is fine, like a "check for newer version" button for an ebook.

Thanks!

[kovidgoyal]

Out of the box, no. You can write your own plugin for it. Or if that is too much work, use the calibredb command line tool in your script.

[Demosthenex]

Thanks for the fast reply!

I wouldn't mind trying to automate it myself, I believe the key would be whether or not I can store an ad-hoc field with the ebook documenting the source URL.

Then I could write a plugin or script to test the url for new content.

Is Calibre's database flexible enough to add custom fields, or does this already exist?

[kovidgoyal]

Preferences->Add your own columns

[davidfor]

You can also add a url as an id. An example I have is "uri:http|//www.gutenberg.org/ebooks/62" ("A Princess of Mars" by Edgar Rice Burroughs). When I select the book, the ids show "www.gutenberg.org" which is click-able and goes directly to the url. Using that, makes it easy to get to the appropriate page and check the current status.

I actually looked at something like this for another site. The solution I was going to use was to create a new id type in the plugin. I got to the point where I could click it on the id to open the page for downloading the file and do a partial search from the metadata editor. I haven't finished this as I hit an issue with needing to login to the site for all the details. And I haven't had time to go back to it.

Also, I like the idea of what you are doing. Way back in my mainframe programming days, IBM Redbooks were frequently our saviours. Making them easy to get and update would have been great.

[Demosthenex]

I thought the idea of a "source" field was straightforward, it would be a URL directly to the PDF (no authentication). I'd then have a refresh which would check the date of the remote PDF vs the local, and update unless the size dropped significantly (ie: error).

The issue I just ran into is that I'd like to import my existing redbooks into my Calibre database, but got stuck because it appears I can't add metadata when I add a book.

calibredb add lets me specify a filename, but I can't feed it my source field at the same time. My initial efforts to search for the PDF via calibredb list by filename failed as well.

If calibredb add returned the UUID in its output, I could then call set_metadata...

Any other ideas?

[kovidgoyal]

After doing an add, simply do a list and the highest id will be the book you just added

[Demosthenex]

That's a good idea, and it does hold true, I can query the highest index number easily from the command line.

Where I just fell flat is trying to add metadata. Set metadata requires an OPF formatted file, and replaces all metadata on the title in question. Looking at another book I added a source field to and grabbed the OPF, the syntax is huge for the custom field.

At this point to do this in a batch script I would have to query the OPF of the newest title, and then have a script edit this XML to add a giant source block and then recommit it.

Did I miss something where I can just set one field on a ebook from the command line (ie: "calibredb book# set_field source 'http....' ")?

Thanks.

[kovidgoyal]

You get the OPF with show_metadata, edit it with some XML library and then use set_metadata. There's no single field way of setting metadata from the command line.

[toonhenry]

Definitely post the final solution when you have it. I would use this feature constantly, many open source projects automatically generate their documentation in multiple formats (html, pdf, epub...) and this would be an easy way to keep the most up to date version on my desktop/reader.

[Demosthenex]

Originally Posted by kovidgoyal

There's no single field way of setting metadata from the command line.

I just revisited this issue after leaving it alone for a long duration. I noticed that calibredb's add command allows me to specify title, isdn, author, and several other metadata items from the command line.

Could you consider it a feature request to add a parameter that accepts a key/value pair, and may only work for valid metadata keys (including custom ones)?

For example:

$ calibredb add -t "Book title" myfile.pdf

would be equivalent to, assuming -m is for metadata:

$ calibredb add -m "title=Book title" myfile.pdf

Keys would likely be restricted to a single lowercase word that corresponds to the "lookup name" or column key. Multiple instances of -m should be supported for multiple keys.

I could see this as being useful for any number of potential applications, not just for storing a URL.

Thanks for your consideration.