[GUI Plugin] FanFicFare

[Ser4nb2LUY6e]

Could you please update the wuxiaworld.co adapter to ignore chapters that have:

style="color:Gray;"

They are always going to be at the end of the list, e.g.:

                     <dd> <a style="" href="2057006.html">1071 We’re Going to the Union Government? Aren't We Walking Right Into the Trap?</a></dd>
                    
                    <dd> <a style="" href="2069605.html">1072 The Neo Armstrong Cyclone Jet Armstrong Cannon</a></dd>
                    
                    <dd> <a style="color:Gray;" href="2071965.html">1073 Oh My God, That Smell!</a></dd>
                    
                    
                </dl>

And they are chapters that are already listed in the chapter list, but for which the contents only says:

In preparation, Keguan Please wait a moment, after the updates, you need to refresh the page to get the latest updates!

So it makes no sense to download them.

TIA!

[JimmXinu]

New Test Version Posted

2019-07-05
- adapter_wuxiaworldco: Don't include grayed out 'In preparation' chapters
- Move BS version debug so it doesn't appear in CLI without -d.

Originally Posted by Ser4nb2LUY6e

Could you please update the wuxiaworld.co adapter to ignore chapters that have:

[Ser4nb2LUY6e]

Thanks a lot for the quick fix.

Is there an easy way to remove the last x already downloaded chapters from the epubs so that the next update will try to download them again, or do I need use "overwrite always" (which will take it's sweet time given that most books that now have "In preparation" chapters [I got unlucky and was running my update right at the time when they were updating their website] are 1000+ chapters long)?

[JimmXinu]

Originally Posted by Ser4nb2LUY6e

Thanks a lot for the quick fix.

Is there an easy way to remove the last x already downloaded chapters from the epubs ...

I don't know of a super easy way. I use Calibre's 'Edit book' to remove chapters when needed. You can selected a range of chapter files in the file browser pane and delete them all at once.

One possibility: if you still have the pre-update epubs in your trash can, you could drag them out to desktop or a folder, then drop them back into Calibre, which should overwrite the newer epub with the older one.

[Ser4nb2LUY6e]

I had to manually edit the books, but I seem to have it all sorted out now.

Another question: When using "Update existing FanFiction Books" with a whole list of books, would it be possible to have each book scheduled as a separate job automatically?

I mean, I can do that by hand already by selecting one book, trigger the function, while the job is still running select the next book, trigger the function, and so on.

And, generally, in case that a job only is for a single book, maybe have the progress reported for the job based on number of chapters to download instead of being stuck and then jump to 100% when done?

[JimmXinu]

Originally Posted by Ser4nb2LUY6e

Another question: When using "Update existing FanFiction Books" with a whole list of books, would it be possible to have each book scheduled as a separate job automatically?

I mean, I can do that by hand already by selecting one book, trigger the function, while the job is still running select the next book, trigger the function, and so on.

As you've noticed, by running all in one BG process, it limits the number of simultaneous downloads running.

But that's a feature, not a bug. Several supported sites will start blocking you if you generate too much traffic. This way, you can launch several without having to manage that yourself.

Because trust, getting blocked for a few days is worse than slower downloading.

Originally Posted by Ser4nb2LUY6e

And, generally, in case that a job only is for a single book, maybe have the progress reported for the job based on number of chapters to download instead of being stuck and then jump to 100% when done?

I've looked into that a few times. Getting meaningful progress reporting from BG process to the foreground would require significant re-architecting.

[Ser4nb2LUY6e]

The main issue I have with performance is that most of the books I update are in the 2-5 million words range already. Looking at the job log, there is a 30 second to 3 minute pause between the last entry of one book (after all new chapters have been downloaded) and the start of the next book.

Because of that, checking for updates for 42 books, 8 of which had a couple of new chapters, took over 20 minutes.

[JimmXinu]

That doesn't strike me as particularly poor time performance, depending on the site. You are talking about stories with millions of words and thousands of chapters.

Do you have FFF configured to call word counts? Or smarten punctuation? It also wouldn't surprise me if a lot of that delay comes from post-processing.

[Ser4nb2LUY6e]

I've all post processing options turned off (nothing checked, calculate word count set to No), other wise the update would take closer to an hour than 20 minutes.

I do realize that the stories I'm following are really large, but the low performance doesn't come from the site access.

If I don't use the "background metadata" option, then fetching metadata for 45 huge books takes about 2 minutes, which isn't too bad.

And if there are multiple new chapters, I can see from the job log that the chapters are downloading at a rate of about 1 chapter per second, so that's pretty good too.

It's basically the loading and saving of the epub that takes significant time as far as I can tell (no surprise there, they are huge).

Maybe schedule the individual books as individual jobs, but have a global list with domain name + lock (whatever that is in python) and acquire the lock while loading a page from that domain.

This would serialize the online access (hopefully preventing a possible ban as you mentioned) while still allowing local processing to take place in parallel.

Anyway, just an idea. FFF as is has saved me countless hours of manual work and I really appreciate the work you've put in.

[Sema]

Was anything changed with handling downloading from AO3? I like to download completed series as an anthology to calibre, which used to be particularly easy with AO3's series features and FFF's "downloading anthology into single EPUB" feature. I last did this a couple months ago with no problem. When I tried the same thing today, I got an error message that the tool din't find valid urls on that site.
Downloading a single story works fine, but since this is a 14 part series of short stories, I would have loved to have them in a single ebook.

[JimmXinu]

multi-reply

Originally Posted by Ser4nb2LUY6e

Maybe schedule the individual books as individual jobs, but have a global list with domain name + lock (whatever that is in python) and acquire the lock while loading a page from that domain.

Because Calibre runs on Windows, Linux and MacOS, it's difficult to implement IPC between threads in subprocesses.

I will think about this some more.

Originally Posted by Sema

Was anything changed with handling downloading from AO3? I like to download completed series as an anthology to calibre, which used to be particularly easy with AO3's series features and FFF's "downloading anthology into single EPUB" feature. I last did this a couple months ago with no problem. When I tried the same thing today, I got an error message that the tool din't find valid urls on that site.
Downloading a single story works fine, but since this is a 14 part series of short stories, I would have loved to have them in a single ebook.

Nothing's changed in that regard. However, I suspect that you aren't using the correct FFF option to make an anthology.

Use 'Make Anthology Epub from Web Page' or 'Get Story URLs from Web Page' with the 'For Anthology Epub' button.

If you are already doing that, please post the series URL that isn't working for you.

[Sema]

That's exactly the option I was using, and I tried both. I just had to translate the wording back from German. I was trying to download this series:
https://archiveofourown.org/series/915132
I know it's restricted to members, but firstly I am one, have been for years, and secondly, this isn't a problem with the individual download.

[JimmXinu]

That series page is user/pass protected. You'll need to save your AO3 user/pass in personal.ini to use it:

[archiveofourown.org]
username:XXXX
password:XXXX

It also looks like AO3 has added a "I have read & understood the new Terms of Service and Privacy Policy" page that pops up and sets a cookie session cookie. So far it isn't effecting FFF, but it might in future.

[Sema]

That helped, thanks for the very quick reply!

[JimmXinu]

Originally Posted by JimmXinu

Because Calibre runs on Windows, Linux and MacOS, it's difficult to implement IPC between threads in subprocesses.

I will think about this some more.

I have looked into this a bit more. Calibre has a mechanism for limiting the number of BG jobs actively running at a time, which could reasonably be assumed to be an acceptable limit. But there are issues with that, namely, that as each job finishes it calls the 'Proceed to Update Library' dialog for each one. So you have click 'Yes' for each one.

FFF could gather all those updates and do 'Proceed' once, but it's kinda complicated. The sort of async that is hard to get right the first time and difficult to debug.

Plus FFF's background job is already coded to run CPU#(from Calibre settings) threads simultaneously. I've checked and I can see different the threads running concurrently.

All in all, I don't think BG job per download is worth pursuing any further.