[GUI Plugin] FanFicFare

[JimmXinu]

If you are updating an existing epub from FanFicFare with 74 chapters, when chapter 75 comes out, FFF should only need to fetch chapter 1 (for metadata) and chapter 75 -- assuming you don't have check_next_chapter or meta_from_last_chapter set true and that it's been long enough that chapter 1 shows 75 chapters.

I would suspect that it's the chapter 1 lookup that's failing and you're assuming it needs all chapters.

If FFF really wants to download all chapters, that suggests that the epub has been changed or something else is going on.

[ownedbycats]

Originally Posted by JimmXinu

Here are a few ideas:

• SmartEject missing/dup searches?
• Keep a second copy of each in another column that is New Only and compare?
• Use Audit Log plugin?
• Use FFF's logpage?
• Remove changeable metadata from save path?

SmartEject catches such things well enough for me.

Actually, I think that last one might be doable. I already have #kobopath for part of my save template, so modifying that to also include the original author and title might work.

[JimmXinu]

Originally Posted by ownedbycats

Actually, I think that last one might be doable. I already have #kobopath for part of my save template, so modifying that to also include the original author and title might work.

Actually, I meant removing title/author altogether and using {id} instead.

[Ser4nb2LUY6e]

Any chance you could add a "Background Metadata" option to "Download from URLs" like it exists in "Update existing FanFiction Books"?

Also to pick up a previous discussion, "Download from URLs" is definitely not multi-threaded.

I wanted to download a bunch of stories (19) from novelfull.com, which is quite slow as I found that it doesn't work reliably without using flaresolverr.

I downloaded them all together in a single use of "Download from URLs" because I didn't want to sit around for 5 minutes each time waiting for "Downloading metadata for stories" before I can add the next one. (That's where a "Background Metadata" option would be great.)

That blocked calibre completely for almost an hour with "Downloading metadata for stories", followed by a job running for 49h which was clearly downloading one story, one chapter at a time linearly, despite calibre being configured for 18 background threads (I've got a 24core/48thread CPU).

A "Background Metadata" option to "Download from URLs" would at least have allowed me to add these manually as 19 different jobs without spending an hour waiting to add a new one every 5 minutes.

Then having "Download from URLs" either using multi-threading internally, or having a "create one job per book" option would have reduced that from 49h to maybe 3h.

(As I've said before, I would find a "create one job per book" option much more useful, as it means that I get access to the books as they are finished downloading, and don't have to wait for all of them to finish downloading.)

[JimmXinu]

Originally Posted by Ser4nb2LUY6e

Any chance you could add a "Background Metadata" option to "Download from URLs" like it exists in "Update existing FanFiction Books"?

No, there are too many features that depend on having the metadata, such as checking title/author dups. It's been discussed before.

Originally Posted by Ser4nb2LUY6e

Also to pick up a previous discussion, "Download from URLs" is definitely not multi-threaded.

Download is multi-threaded for different sites, but not a single site.

Some sites, if they see you downloading in parallel, will block you entirely. FFF is designed to avoid that happening. And no, I'm not interested in making that a per-site option.

novelfull.com in particular takes a long time to collect metadata because it only lists 50 chapters per page, so a story with 1500 chapters (not rare on that site) will need 30 requests just to get the chapter list for metadata.

If you like, you can experiment with making the slow_down_sleep_time setting lower than the default value of 2sec under [novelfull.com].

[Ser4nb2LUY6e]

Ah well, I'll just have to start them by hand as individual jobs then.

The "slow down sleep time" probably doesn't come into effect as I'm using flaresolver for novelfull.com (had trouble without) and request take between 3.5 and 4 seconds in that.

I can download from novelfull.com without issue with 12 jobs started separately be hand though.

Originally Posted by JimmXinu

No, there are too many features that depend on having the metadata, such as checking title/author dups. It's been discussed before.



Download is multi-threaded for different sites, but not a single site.

Some sites, if they see you downloading in parallel, will block you entirely. FFF is designed to avoid that happening. And no, I'm not interested in making that a per-site option.

novelfull.com in particular takes a long time to collect metadata because it only lists 50 chapters per page, so a story with 1500 chapters (not rare on that site) will need 30 requests just to get the chapter list for metadata.

If you like, you can experiment with making the slow_down_sleep_time setting lower than the default value of 2sec under [novelfull.com].

[Ser4nb2LUY6e]

Actually, it's all good, I just managed to implement it for myself:

dialogs.py

    
    # invoke the
    def ok_clicked(self):
        self.dialog_closing(None) # save persistent size.
        self.hide()
        for s in self.get_urlstext().splitlines(): # line added 
            self.go_signal.emit( self.get_fff_options(),
                                 s, #was self.get_urlstext()
                                 self.merge,
                                 self.extrapayload )

I acknowledge this isn't the right solution for everyone in every circumstance, but for me, that's exactly what I want.

Originally Posted by Ser4nb2LUY6e

Ah well, I'll just have to start them by hand as individual jobs then.

The "slow down sleep time" probably doesn't come into effect as I'm using flaresolver for novelfull.com (had trouble without) and request take between 3.5 and 4 seconds in that.

I can download from novelfull.com without issue with 12 jobs started separately be hand though.

[sios00]

Anyone knows if it is possible to have FanficFare still generate a cover using Generate Cover without replacing all other images inside a fic?

For example, in this fic https://archiveofourown.org/works/14...pters/96450513 at the bottom of chapter 60, the author has placed an image. However, downloading the fic with Generate Cover in use results in that image being replaced with the cover image instead. This has also happened to other fics with images in them.

Thank you!

[JimmXinu]

Originally Posted by sios00

Anyone knows if it is possible to have FanficFare still generate a cover using Generate Cover without replacing all other images inside a fic?

For example, in this fic https://archiveofourown.org/works/14...pters/96450513 at the bottom of chapter 60, the author has placed an image. However, downloading the fic with Generate Cover in use results in that image being replaced with the cover image instead. This has also happened to other fics with images in them.

I am seeing something weird going on there, but I'm not sure what yet.

Can you provide any additional examples? Especially one showing "replacing all other images"? Because the one above only appears to replace one of two places that the image appears.

UPDATE: Never mind--I think I figured it out.

[JimmXinu]

New Test Version Posted

2022-05-15
- Refactor img code in story.py to fix a problem when cover image also in story.

This should fix sios00's issue.

[riothamus]

Originally Posted by JimmXinu

If you are updating an existing epub from FanFicFare with 74 chapters, when chapter 75 comes out, FFF should only need to fetch chapter 1 (for metadata) and chapter 75 -- assuming you don't have check_next_chapter or meta_from_last_chapter set true and that it's been long enough that chapter 1 shows 75 chapters.

I would suspect that it's the chapter 1 lookup that's failing and you're assuming it needs all chapters.

If FFF really wants to download all chapters, that suggests that the epub has been changed or something else is going on.

Ok, so to avoid downloading all chapters with FFDL, I would download the first chapter and new chapter(s) (with FFDL) and then FFF should update normally still?

[JimmXinu]

Originally Posted by riothamus

Ok, so to avoid downloading all chapters with FFDL, I would download the first chapter and new chapter(s) (with FFDL) and then FFF should update normally still?

Yes. I find it easier to use a browser for updates.

[ownedbycats]

Originally Posted by JimmXinu

I am seeing something weird going on there, but I'm not sure what yet.

Can you provide any additional examples? Especially one showing "replacing all other images"? Because the one above only appears to replace one of two places that the image appears.

UPDATE: Never mind--I think I figured it out.

This is very weird. I just came across the same glitch myself in an AO3 fic and came here to report it, only to find it's been fixed.

I found a glitch in the new test version. This happens when trying to save personal.ini, even though I don't have a fastnovels section:

[JimmXinu]

That can happen if I remove a site but forget to remove the config section from defaults.ini.

But I didn't in this case--I downloaded the posted zip again to test. And [fastnovels.net] is the new name, it was [fastnovel.net] before. Which suggests you some how have old code with the new defaults.ini?

I'd download install the zip/install/restart calibre again.

[ownedbycats]

Interesting, looking at fastnovels.net in the defaults.ini was all red. Re-installing the test version fixed it though. I wonder how that happened. Maybe the first install got botched up somehow.