[GUI Plugin] FanFictionDownLoader

[JimmXinu]

While the PI does have to take modal control again to do the db update, it doesn't have to stop and prompt the user for permission, as I understand it.

FFDL (optionally) lets the user skip that prompt and go straight to a Progress dialog to update the db when everything was successful.

[kiwidude]

Ummm... what if the user has an edit dialog open at the time that the plugin is ready to update the database? That assumption sounds flawed to me - at least for books that already exist in the database. How can the plugin "assume" modal control when the user may have opened their own modal dialog?

[JimmXinu]

Originally Posted by kiwidude

Ummm... what if the user has an edit dialog open at the time that the plugin is ready to update the database? That assumption sounds flawed to me - at least for books that already exist in the database. How can the plugin "assume" modal control when the user may have opened their own modal dialog?

Empirically, if the user has the metadata dialog for a book FFDL is downloading open when FFDL finishes (updating the book's metadata), and the user then 'OKs' the metadata dialog, the book will be updated with what the user just changed, possibly overwriting columns FFDL has just populated.

If the user has the metadata dialog for a book Count Pages is counting open when Count Pages finishes, the 'update library' dialog is opened 'underneath' the metadata dialog and can't be accessed until the metadata dialog is closed. Then the user can tell the PI to proceed, possibly overwriting columns that they just set manually.

I don't think either approach is fundamentally incorrect, just different design choices.

However, if you tell me that you know that there can be corruption (not just contradiction of assumptions) if the db changes while the edit metadata dialogs are open (or other PIs run?), I'll bow to your greater experience and stop letting people skip that dialog in FFDL, too.

(I think that may be what you meant by "concurrency and stale data to deal with", but you also could have just meant overwriting data.)

[kiwidude]

Personally I think your approach is playing with fire, and taking away the prompt has missed the point of why the dialog is modal. Making it modal with a prompt ensures that (a) the user finishes whatever operations they currently have in progress before the update occurs, and (b) ensures the user cannot start any other activity which might update the database while the update occurs. Take away the prompt and you prevent neither of these. The user could be doing "anything" with the database at the time FFDL prepares to update - whether it is individual edits or bulk replace operations. Your database integrity is completely shot if there happens to be a conflict between whatever activity the user was doing and the background operation. And don't forget there is a data caching layer for the library view as well.

If putting in the prompts is Kovid's approach to deal with this issue (as he does in metadata downloads) then my plugins do the same. It eliminates any risk of data issues.

[cryzed]

Something unrelated to the current discussion: Jimm, do you think it's possible for you to put the table of contents at the beginning of the generated file? I ask because the Kindle displays it at the very end of the story and usually you wouldn't find it there (at least I don't think so) -- don't you usually expect it to be located directly after the cover page? Additionally one needs to jump over the table of contents to set the pages-read count to the maximum, only then it displays as fully read in the Kindle.

[iatheia]

Originally Posted by cryzed

Something unrelated to the current discussion: Jimm, do you think it's possible for you to put the table of contents at the beginning of the generated file? I ask because the Kindle displays it at the very end of the story and usually you wouldn't find it there (at least I don't think so) -- don't you usually expect it to be located directly after the cover page? Additionally one needs to jump over the table of contents to set the pages-read count to the maximum, only then it displays as fully read in the Kindle.

This isn't really ffdl issue, more of a Calibre default (at least when handling mobi, not too sure about epub). You can change it in settings, though, personally I have yet to find a satisfactory solution, as in that case the metadata page isn't the first thing displayed.

[cryzed]

You are right, well that sucks. It should have another option: "Place table of contents after the cover page".

[JimmXinu]

Originally Posted by cryzed

Jimm, do you think it's possible for you to put the table of contents at the beginning of the generated file?

It's not really a calibre issue either--it's a mobi issue. The mobi format doesn't define TOC in metadata like epub, it only defines a pointer to where the TOC is in the content.

By default, mobis generated by FFDL will have the TOC right after the title page and the mobi metadata will point to it.

Converted mobis put it at the end because a) I think it's more 'standard' that way and b) they have no other idea where to put it.

Presumably you're converting mobis from epubs. You can add an *extra* TOC right after the title page by setting include_tocpage: true in the [epub] section of personal.ini. Then you'll end up with both, but the official TOC will still be at the end. Whether that's an improvement or not is a personal choice.

[cryzed]

Back when I developed Lemon I actually generated the table of contents myself (line 31+), but I'm assuming that's not something you would want to do? It would be specific only to the MOBI writer then. I can fully understand if you don't want to do that.

Edit: I think you are actually doing that already with the "include_tocpage" option, I just need a way to somehow disable the "official table of contents" generation now...

[JimmXinu]

Originally Posted by kiwidude

... And don't forget there is a data caching layer for the library view as well.

If putting in the prompts is Kovid's approach to deal with this issue (as he does in metadata downloads) then my plugins do the same. It eliminates any risk of data issues.

Okay. That, and looking at the special queue signalling code in self.gui.proceed_question, convinces me. I'll change FFDL to follow suit.

Many thanks for sharing your expertise, kiwidude.

[JimmXinu]

Originally Posted by cryzed

Back when I developed Lemon I actually generated the table of contents myself (line 31+), but I'm assuming that's not something you would want to do? It would be specific only to the MOBI writer then. I can fully understand if you don't want to do that.

Originally Posted by JimmXinu

By default, mobis generated by FFDL will have the TOC right after the title page and the mobi metadata will point to it.

Actually, I remembered that slightly incorrectly. The mobi.py output code is what includes the TOC. And it does mark that TOC in the metadata (lines 92 & 111).

[kiwidude]

@Jimm - no probs, just wanted to explain why Count Pages (and Extract ISBN) work that way. I can 100% understand and sympathise with why you wanted to try to avoid the prompt - we would all love that. However as you may know from elsewhere database locking approaches and strategies are a headache every concurrent application faces. I can totally understand why Kovid took the simplest approach possible, which unfortunately means an interactive prompt. And in the case of your plugin for a user doing count pages, it will now mean two of them.

The alternatives just become increasingly more complex everywhere. Every time you do an update to the database you have to be prepared that it could "fail" because (a) the user may have merged/deleted the book or otherwise changed it so it no longer represents what it "was", (b) the user may have changed the same data you are now wanting to update - so you have to decide which one "wins" either by overwriting or by rejecting the change (which has huge repercussions). And then of course (c) you have all the GUI complications, books potentially being added/removed/updated in the background means what you see on screen does not reflect "current reality" which is important when you try to edit or perform operations. Ugh.

Which is why I shall be interested to see what approach Kovid has in mind as part of his database back-end rewrite. If he is going to allow multiple instances of calibre to connect to the same database... fun, fun, fun...

[JimmXinu]

@kiwidude - I understand. I work primarily on servers, not GUI, so I was implicitly assuming transaction protection at least and not thinking about how a GUI app that evolved over time wouldn't have been designed with concurrent access in mind.

[JimmXinu]

Here's a first version with changes to make FFDL properly respect calibre's method of guaranteeing db consistency.

I'll try to make the 'view log' prettier, but I'm not sure what html features I have available there.

I'd appreciate it some people would try this out and see how it works for them.

Thanks.

[cryzed]

@JimmXinu, shame on me, I meant the EPUB writer of course, not the MOBI one. I mean you suggested to initially create the stories as EPUB files and then later on convert them with Calibre, so that's what I'm doing.