[GUI Plugin] FanFictionDownLoader

[theducks]

Originally Posted by JimmXinu

Considering last_modified isn't normally visible in the GUI (at least I couldn't find it except for from CLI), does it really matter? And one could argue that it is accurate.

.

The column name is Modified. The lookup name is last_modified

[FaceDeer]

I've been exploring the failure of my laptop to successfully download fanfics (while my desktop is succeeding just fine). The laptop's Windows 7, the desktop is Windows 8.

I fired up my laptop's calibre in debug mode and took a shot at downloading a fanfic. It hung, and the end of the log read:

Successfully downloaded:
Bezithan http://www.fimfiction.net/story/124989/

Unsuccessful:

add/update Bezithan http://www.fimfiction.net/story/124989/

Everything before that looked normal. The log just stopped when Calibre hung.

After I killed Calibre and went back to the library I'd tried downloading it into, I found that there was indeed an entry for that story complete with all its metadata, but that it didn't have any actual story associated with it - it was an empty book record with no formats. I went and dug up the epub FFDL created out of the temp directory, and it looked fine. So it would seem that FFDL successfully created the epub, successfully created the book record in Calibre, but then failed to bring the epub over and insert it. My first instinct when seeing something like that on Windows is to try running the program as administrator to see if that helps, but unfortunately it didn't in this case. Didn't really expect it.

I tried installing the 32 bit version of Calibre alongside the 64 bit one and it had the same symptoms. I'm a little reluctant to wipe Calibre out and reinstall entirely from scratch, since everything else is working okay and that would mean a lot of reinstallation and restoration work afterward. Any suggestions on how else I can investigate before I go to more drastic steps like that?

[JimmXinu]

Originally Posted by FaceDeer

Heh. I actually do make use of the last_modified date, myself.
...
It never occurred to me that updating a fanfic would result in last_modified being changed on other documents that hadn't been flagged for updating.

So you can testify that it didn't work that way in calibre 0.9? That it is a change that's happened in calibre 1.0?

Originally Posted by FaceDeer

If you want to be more rigorous anyway, perhaps make it so that FFDL doesn't modify other fanfics if the update wouldn't actually change the contents of the fanfic's author link field to a new value?

FFDL isn't modifying the other books. calibre's DB keeps (or did keep) the author link in a separate table from the books. I can make that update optional, or only for new authors, or only when changed.

But if the calibre behavior has changed, we should open a bug with calibre and get it changed back.

Attached is a version that doesn't ever set or change author link if you care to use it in the mean time.

UPDATE Aug 27 2013 -- Remove obsolete beta versions

[Sefiriot]

Originally Posted by JimmXinu

So you can testify that it didn't work that way in calibre 0.9? That it is a change that's happened in calibre 1.0?

FFDL isn't modifying the other books. calibre's DB keeps (or did keep) the author link in a separate table from the books. I can make that update optional, or only for new authors, or only when changed.

But if the calibre behavior has changed, we should open a bug with calibre and get it changed back.

Attached is a version that doesn't ever set or change author link if you care to use it in the mean time.

Yep, pretty sure it didn't work like that in 0.9-- I notice it because I usually use calibre's modified value to sort my library when I want to see what I just added and want to read. Can't really do that right now when every update also messes with things that haven't been.

I'm not sure if this new behaviour is intended to be a bug or a feature but seeing as Calibre Companion, which I'm certain many of us are using, can sort by Date (calibre)-- which is actually the last_modified date-- having this kind of thing happening will likely negatively impact that feature. I just want to read the latest update from author X, and would appreciate not having to remember what did update and wade through the umpteen other stories I have by X that haven't actually updated. >.>

Thanks for the new version: I'll test it out and see if all is as expected.

[FaceDeer]

Originally Posted by JimmXinu

So you can testify that it didn't work that way in calibre 0.9? That it is a change that's happened in calibre 1.0?

Afraid I can't testify one way or the other. I never noticed it updating the last_modified date in 0.9, but the way I was using it made it a very likely thing to overlook if it did (very few of the fanfics in my "incompletes" library are by the same author and even if last_modified did get updated on the wrong books it's not something that had very obvious effects). I only brought it up because you were asking if anyone actually used that field, and it turns out I do. A little bit.

[El_Pirato]

Originally Posted by JimmXinu

Exactly. The ominous legal warnings are sufficient to make us leery.

I'm looking for ways to bypass if that's possible.
I want to know how Raimond Eisele(Fanfiction Downloader.net) is able to bypass it.

[JimmXinu]

I've installed an 0.9 version again and verified for certain that the last_modified behavior has changed and opened a bug report about it.

However, even if Kovid fixes it, it will still be Friday before a new version of calibre with the fix would be released.

So if you depend on last_modified being correct, you may want to use the temp version of FFDL posted above.

[JimmXinu]

Originally Posted by FaceDeer

Any suggestions on how else I can investigate before I go to more drastic steps like that?

Are you using the same library on both desktop and laptop or different ones?

Does adding books not from FFDL work?

Have you tried a new, empty library? You can create one without touching your old one.

(Sorry, thought I hit post on this last night.)

[JimmXinu]

Originally Posted by theducks

The column name is Modified. The lookup name is last_modified

You're right, of course. I just missed it and was having a moment of frustration.

[Sefiriot]

Just gave your latest version a test run-- everything's back to previous behaviour and it looks like the last modified values aren't being touched anymore where they shouldn't be, so thanks.

[JimmXinu]

Originally Posted by Sefiriot

Just gave your latest version a test run-- everything's back to previous behaviour and it looks like the last modified values aren't being touched anymore where they shouldn't be, so thanks.

That's good, but keep in mind that version is a temporary measure only. If you add a story for a new author you won't get an author link in calibre for them.

[Sefiriot]

Hmm. I see from Kovid's reply at the bug tracker that the 0.9.x implementation was the buggy implementation--! Welp. Time to figure out a better way to sort things and stop relying so much on last modified then, since it seems that calibre does require regeneration of all OPFs when updating. I guess you'll recommend reverting to the current release for FFDL yes?

[JimmXinu]

Originally Posted by JimmXinu

I've installed an 0.9 version again and verified for certain that the last_modified behavior has changed and opened a bug report about it.

Remember when I said it was arguably the correct thing to do? That's exactly what Kovid's just ruled.

Changing author metadata modifies the metadata of all books associated with that author. That is the correct behavior. author metadata is stored in the opf files which means that changing it must update last_modified and thereby regeenrate the opf files for all books by that author.

The implementation in 0.9.x was the buggy one.

** Changed in: calibre
Status: New => Invalid

So I'll have to change how FFDL does author links. Hopefully I can get a new version out later today, but no promises.

[Sefiriot]

No hurry, take your time. This is arguably a very minor issue after all, in the grand scheme of things.

[FaceDeer]

I just thought of a potential everybody-wins solution: add an "ffdl_last_modified" optional date metadata that users can configure FFDL to populate (along the lines of the FFDL error column). That way you can have it update only when FFDL actually modifies the story, in a way that you can configure however you think is optimal for fanfic downloading (for example, you might want to have it only change the date if the *text* of the story changes, ignoring any other metadata updates).