[GUI Plugin] FanFictionDownLoader

[JimmXinu]

Originally Posted by FaceDeer

I just thought of a potential everybody-wins solution: add an "ffdl_last_modified" optional date metadata that users can configure FFDL to populate. That way you can have it update only when FFDL actually modifies the story, in a way that you can configure however you think is optimal for fanfic downloading (for example, you might want to have it only change the date if the *text* of the story changes, ignoring any other metadata updates).

FFDL already has three dates (for most sites) in it's metadata:

• Published - when the story was first created (from the site)
• Updated - when the story was last updated (from the site, usually only when new chapters)
• Created - essentially 'now', when FFDL is being run (labeled Packaged on the title page)

That's what I consider optimal.

[FaceDeer]

Oh, "Created" is changed whenever the epub is modified? I'd thought that field meant "when this story was first added to Calibre" and didn't change after that. That's a piece of information that I didn't need to know for anything and so I ignored it. If it updates whenever the epub file is changed then I'll switch over to using that, it's exactly what I need.

[JimmXinu]

Originally Posted by FaceDeer

Oh, "Created" is changed whenever the epub is modified? I'd thought that field meant "when this story was first added to Calibre" and didn't change after that. That's a piece of information that I didn't need to know for anything and so I ignored it. If it updates whenever the epub file is changed then I'll switch over to using that, it's exactly what I need.

Correct. I have no idea now why I labeled it 'Packaged' in place and 'Created' in the other.

It will also be 'now' if you run 'update metadata only'.

[JimmXinu]

Originally Posted by JimmXinu

...
So I'll have to change how FFDL does author links. Hopefully I can get a new version out later today, but no promises.

Or not. The methods available at the moment to look up the existing author link are rather onerous (unless I missed one--a distinct possibility).

I've posted additional comments and a couple patches to the bug report. I hope to convince Kovid to include one patch or the other.

[JimmXinu]

Originally Posted by JimmXinu

... (unless I missed one--a distinct possibility).

I've posted additional comments and a couple patches to the bug report. I hope to convince Kovid to include one patch or the other.

Now a certainty. Kovid pointed out the correct new_api calls for me to use.

So attached is a version that will only update author links (and therefore last_modified on all those authors' books) when the link actually changes.

Kovid also says "Note that the next release will also not mark books as dirty if you set the same value as before for the author link." But we don't necessarily want to wait that long.

UPDATE Aug 28, 2013 - Remove obsolete beta versions

[Sefiriot]

Oooh thank you for the quick work! *makes grabby hands* Will having books marked as dirty do anything nasty to the database though?

[JimmXinu]

Originally Posted by Sefiriot

Oooh thank you for the quick work! *makes grabby hands* Will having books marked as dirty do anything nasty to the database though?

No. 'Dirty' in programming terms means something that has been changed and needs to be saved. He just means that the books won't be saved (which is what updates last_modified) if the value is 'set', but isn't actually changed.

[Sefiriot]

Originally Posted by JimmXinu

No. 'Dirty' in programming terms means something that has been changed and needs to be saved. He just means that the books won't be saved (which is what updates last_modified) if the value is 'set', but isn't actually changed.

Ah, good to know. 'Dirty' can mean different things in various contexts, so I wasn't completely sure what was meant.

[JimmXinu]

Version 1.7.38 - 24 Aug 2013

• Work around for last_modified vs author link update issue in calibre 1.0.0.
• Better error reporting of metadata update problems.

[FaceDeer]

That better error reporting thing might just have raised its first flags. Testing out my new system using the "Created"/"Packaged" date, I went and grabbed 1200 fanfics off of fanfiction.net that were a mixture of ones I had already and new ones. I ran into 12 instances of the error:

calibre, version 1.0.0
ERROR: Error Updating Metadata: <p>An error has occurred while FFDL was updating calibre's metadata for <a href='http://www.fanfiction.net/s/9587231/1/'>The Legend of the Titans</a>.</p>The ebook has been updated, but the metadata has not.

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 1117, in update_books_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 1535, in update_metadata
KeyError: None

However, inspecting several of the fanfics with errors afterward seemed to show that the metadata in my database had indeed been updated to match what was on the site. So maybe these are false positives?

I tried redownloading the twelve and the errors all popped up again, so this is repeatable for me at least. The fanfics in question are:

http://www.fanfiction.net/s/8220675/1/
http://www.fanfiction.net/s/9125974/1/
http://www.fanfiction.net/s/9497792/1/
http://www.fanfiction.net/s/9499806/1/
http://www.fanfiction.net/s/9515957/1/
http://www.fanfiction.net/s/9545052/1/
http://www.fanfiction.net/s/9572375/1/
http://www.fanfiction.net/s/9587231/1/
http://www.fanfiction.net/s/9617841/1/
http://www.fanfiction.net/s/9622698/1/
http://www.fanfiction.net/s/9627992/1/
http://www.fanfiction.net/s/9636580/1/

Edit: Unfortunately the new error reporting hasn't turned up anything regarding my trouble with FFDL on my laptop, it still just quietly hangs at the "updating calibre with fanfiction stories" step. I figure I'll wait until weekend to try out the next Calibre update, to see if anything gets fixed in that that affects this, and then try the tear-it-all-down-and-rebuild approach.

[JimmXinu]

Originally Posted by FaceDeer

That better error reporting thing might just have raised its first flags.

I tried redownloading the twelve and the errors all popped up again, so this is repeatable for me at least. The fanfics in question are:

All of those stories have an author with '&' or 'and' in their author name. Calibre is automatically splitting that into two different authors. So when FFDL tries to look up the author to set their link, it fails.

It looks like it's always done that and before the attempt to set the author link just failed silently.

Attached is a temporary version that gets it past error, but doesn't fix the fundamental problem.

I'll need to research that a bit more first--I'm not sure how to get calibre to accept '&' or 'and' in a single author's name.

UPDATE Aug 28, 2013 - Remove obsolete beta versions

[FaceDeer]

http://www.fanfiction.net/s/9622698/1/ has an author name with "with" in it instead of "and" or "&", Calibre might be splitting on that too.

What a clever program.

[theducks]

Originally Posted by FaceDeer

http://www.fanfiction.net/s/9622698/1/ has an author name with "with" in it instead of "and" or "&", Calibre might be splitting on that too.

What a clever program.

There is a tweak for this:

ID: authors_split_regex

By default, calibre splits a string containing multiple author names on
ampersands and the words "and" and "with". You can customize the splitting
by changing the regular expression below. Strings are split on whatever the
specified regular expression matches, in addition to ampersands.
Default: r'(?i),?\s+(and|with)\s+'

[JimmXinu]

Originally Posted by theducks

There is a tweak for this:

ID: authors_split_regex

By default, calibre splits a string containing multiple author names on
ampersands and the words "and" and "with". You can customize the splitting
by changing the regular expression below. Strings are split on whatever the
specified regular expression matches, in addition to ampersands.
Default: r'(?i),?\s+(and|with)\s+'

Yep, I saw that.

But that doesn't allow authors with '&' (that part is hard coded) and tweaks affect all libraries and books.

(FYI, calibre changes '&' in an author name to ';'. There's no working around that.)

[JimmXinu]

Here is a version that I think addresses the author splitting issue.

UPDATE Aug 31 2013 -- Remove obsolete beta versions