[GUI Plugin] FanFicFare

[applegaa]

Originally Posted by JimmXinu

Yes, I understand your request. But IMO, this is a very minor benefit.

I agree it's a minor benefit, and I appreciate your explanation. Thank you for responding.

[criscros]

For the past week or so, everytime I download a new story or update an existing one, after clicking ok to finish, FFF loops back to the window "update existing list".
If I select yes, it goes through the process, tells me there is no new update.
If I click yes to finish, it loops again and again until I click no.
I'm using calibre 8.16.2 and FFF 4.54.
The only thing in my personal.ini is to exclude author's notes.
All the stories I've been updating are on Royal Road.

Any ideas on how to fix it?

[JimmXinu]

Originally Posted by criscros

For the past week or so, everytime I download a new story or update an existing one, after clicking ok to finish, FFF loops back to the window "update existing list".
If I select yes, it goes through the process, tells me there is no new update.
If I click yes to finish, it loops again and again until I click no.
I'm using calibre 8.16.2 and FFF 4.54.
The only thing in my personal.ini is to exclude author's notes.
All the stories I've been updating are on Royal Road.

Any ideas on how to fix it?

That's a bit baffling.

I have three ideas:

1. There's a bug for RR that's corrected in the latest test version. I don't see this behavior myself, but your settings may invoke it. Try the latest test version, link below in my sig.
   
2. I see you have Action Chains installed--are you calling FFF from AC, or have AC configured to do anything on FFF completion?
   
3. How are you calling FFF for update? Is there a possibility for key bounce or something invoking it more than once? (A stretch, I admit, since it requires clicking 'yes'.)

[criscros]

Originally Posted by JimmXinu

That's a bit baffling.

I have three ideas:

1. There's a bug for RR that's corrected in the latest test version. I don't see this behavior myself, but your settings may invoke it. Try the latest test version, link below in my sig.
   
2. I see you have Action Chains installed--are you calling FFF from AC, or have AC configured to do anything on FFF completion?
   
3. How are you calling FFF for update? Is there a possibility for key bounce or something invoking it more than once? (A stretch, I admit, since it requires clicking 'yes'.)

Just installing the new test didn't make a difference.
So, I uninstalled FFF, totally cleared calibre_cache, restarted installed the new test and it quit looping.
Thanks, so happy it's working without looping again!

Is there a way to automate the process with Action Chains without having to click approval? I set up Action Changes a while ago, but only added update from the plugin. It was just as easy to use the FFF menu shortcut, so I never really used it. It would be nice to eliminate some of the extra clicks.

[JimmXinu]

Originally Posted by criscros

Just installing the new test didn't make a difference.
So, I uninstalled FFF, totally cleared calibre_cache, restarted installed the new test and it quit looping.
Thanks, so happy it's working without looping again!

Hopefully it stays fixed.

Originally Posted by criscros

Is there a way to automate the process with Action Chains without having to click approval?
...

Basically, no.

Kovid firmly recommends using the 'Proceed with updating your library' dialog for plugins changing things in the library and FFF honors that.

[estherflails]

Recently I've been having some trouble with Ao3 HTTP Errors, mostly "HTTP Error in FFF '403 Client Error: Forbidden for url". Sometimes I even get an error when using the "Get Story Urls from Web Page" function (I feel like it was a different error I saw before but this is the one I managed to get just now):

calibre, version 8.4.0
ERROR: Unhandled exception: <b>HTTPErrorFFF</b>:HTTP Error in FFF '525 Server Error: &lt;none&gt; for url: https://archiveofourown.org/users/login'(525)

calibre 8.4  embedded-python: True
Windows-10-10.0.19045-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19045')
Python 3.11.12
Windows: ('10', '10.0.19045', 'SP0', 'Multiprocessor Free')
Interface language: None
EXE path: C:\Program Files\Calibre2\calibre.exe
Successfully initialized third party plugins: DeDRM (7, 2, 1) && Action Chains (1, 20, 4) && Audit Log (1, 0, 19) && Count Pages (1, 13, 6) && EpubMerge (2, 18, 0) && Extract People Other Metadata (1, 0, 0) && FanFicFare (4, 52, 0) && Favourites Menu (1, 3, 1) && Find Duplicates (1, 10, 9) && Goodreads (1, 8, 1) && Goodreads Sync (1, 16, 5) && Job Spy (1, 0, 239) && Save Composite Custom Columns (1, 1, 1) && View Manager (1, 10, 3)
Traceback (most recent call last):
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetchers\fetcher_requests.py", line 128, in request
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FFNfare.zip\requests\models.py", line 943, in raise_for_status
requests.exceptions.HTTPError: 525 Server Error: <none> for url: https://archiveofourown.org/users/login

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 712, in get_urls_from_page_menu
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 735, in get_urls_from_page
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\geturls.py", line 44, in get_urls_from_page
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\base_adapter.py", line 507, in get_urls_from_page
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\base_otw_adapter.py", line 661, in before_get_urls_from_page
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\base_otw_adapter.py", line 141, in performLogin
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\requestable.py", line 105, in post_request
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetchers\base_fetcher.py", line 127, in post_request
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetchers\cache_basic.py", line 123, in fetcher_do_request
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetchers\decorators.py", line 107, in fetcher_do_request
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetchers\base_fetcher.py", line 109, in do_request
  File "C:\Users\User\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetchers\fetcher_requests.py", line 149, in request
fanficfare.exceptions.HTTPErrorFFF: HTTP Error in FFF '525 Server Error: <none> for url: https://archiveofourown.org/users/login'(525)

The latter worked better with "use_archiveofourown_gay:true" but the other errors persisted, sometimes even worsened.

I just tried switching to using browser data which works fine, fortunately. I originally wrote this comment due to an issue I had with it but I figured it out. I'll leave it up in case you wanted to look at the other error.

Thank you for the wonderful extension!

[JimmXinu]

Originally Posted by estherflails

Recently I've been having some trouble with Ao3 HTTP Errors, mostly "HTTP Error in FFF '403 Client Error: Forbidden for url". Sometimes I even get an error when using the "Get Story Urls from Web Page" function (I feel like it was a different error I saw before but this is the one I managed to get just now) ...

It sounds like you've already found all of the possible solutions we have.

I really don't have anything more than what's already documented in https://github.com/JimmXinu/FanFicFa...oblematicSites

[steltek]

@JimmXinu,

Is there a setting that tells FFF to ignore embedded image files in fics so they aren't downloaded?

I have an author with fics that I follow on AO3 where the author adds a note at the end of each chapter. I don't mind the notes, but they also stick an embedded animated GIF or image file in lots of the notes.

As a result, every time I update one of their fics, FFF proceeds to re-download every single one of those animated GIF and/or other various image files that it subsequently appends to the fic in Calibre as JPG files. In addition to causing the fic to take up a lot more storage space, it also causes FFF to take forever to update because it has to open like 20 different linked webpages all over the place outside AO3 to get the GIFs and/or images.

I can provide the link to an example story if it is needed.

[texasnightowl]

@steltek

I don't know if it is exactly the same problem, but recently there was a fic I tried to download and then I realized it was taking forever and a day. And sure enough, it was a fic with a boatload of unnecessary animated gifs.

So, for that specific story, I changed my personal.ini.

[epub]

## include images from img tags in the body and summary of
## stories.  Images will be converted to jpg for size if possible.
## include_images is *only* available in epub and html output format.
## You can also set include_images:coveronly to exclude images in the
## story text but still include specific cover images or
## default_cover_image, force_cover_image, or use_old_cover cover
## images.
## true by default, uncomment and set false to not include images.
#include_images:false

So I uncommented the final line so that include_images was set to false for just the one story and then readded the # to go back to "normal."

As per the comment info, you can set it to coveronly instead of false.

Worked for me on the story I was having an issue with anyway.

But afaik, no way to otherwise choose it for a specific story?

the other option would be to use the exclude_notes option in the AO3 section...but I haven't really played with that.

[JimmXinu]

Originally Posted by steltek

Is there a setting that tells FFF to ignore embedded image files in fics so they aren't downloaded?

include_images:false as texasnightowl said. More details below.

Originally Posted by steltek

As a result, every time I update one of their fics, FFF proceeds to re-download every single one of those animated GIF and/or other various image files that it subsequently appends to the fic in Calibre as JPG files. In addition to causing the fic to take up a lot more storage space, it also causes FFF to take forever to update because it has to open like 20 different linked webpages all over the place outside AO3 to get the GIFs and/or images.

I can provide the link to an example story if it is needed.

Unless you manually removed the images from your epubs, FFF should not be re-downloading them on update; just downloading the new ones.

As a general rule, I always want a story URL. Maybe there is something weird going on in that story that forces image re-download? But without the story URL, I can only speak in general.

Originally Posted by texasnightowl

I don't know if it is exactly the same problem, but recently there was a fic I tried to download and then I realized it was taking forever and a day. And sure enough, it was a fic with a boatload of unnecessary animated gifs.
...
As per the comment info, you can set it to coveronly instead of false.
...
But afaik, no way to otherwise choose it for a specific story?

FYI, coveronly is not helpful with AO3 stories, because AO3 doesn't have official cover images.

You absolutely can make personal.ini settings specific to a story. Example:

[https://archiveofourown.org/works/27670172]
include_images:false

[steltek]

Originally Posted by texasnightowl

@steltek

I don't know if it is exactly the same problem, but recently there was a fic I tried to download and then I realized it was taking forever and a day. And sure enough, it was a fic with a boatload of unnecessary animated gifs.

So, for that specific story, I changed my personal.ini.

[epub]

## include images from img tags in the body and summary of
## stories.  Images will be converted to jpg for size if possible.
## include_images is *only* available in epub and html output format.
## You can also set include_images:coveronly to exclude images in the
## story text but still include specific cover images or
## default_cover_image, force_cover_image, or use_old_cover cover
## images.
## true by default, uncomment and set false to not include images.
#include_images:false

So I uncommented the final line so that include_images was set to false for just the one story and then readded the # to go back to "normal."

As per the comment info, you can set it to coveronly instead of false.

Worked for me on the story I was having an issue with anyway.

But afaik, no way to otherwise choose it for a specific story?

the other option would be to use the exclude_notes option in the AO3 section...but I haven't really played with that.

That was pretty much what I was looking for. Somehow I missed that option when reading through the wiki.

I can live without the cover image so long as I don't have to wait through 20 different web pages opening to move on. If I decide I need the cover image, I am willing to wait until the fic is complete (this author does tend to finish theirs), re-download it, and manually delete all the GIFs.

Originally Posted by JimmXinu

include_images:false as texasnightowl said. More details below.


Unless you manually removed the images from your epubs, FFF should not be re-downloading them on update; just downloading the new ones.

As a general rule, I always want a story URL. Maybe there is something weird going on in that story that forces image re-download? But without the story URL, I can only speak in general.



FYI, coveronly is not helpful with AO3 stories, because AO3 doesn't have official cover images.

You absolutely can make personal.ini settings specific to a story. Example:

[https://archiveofourown.org/works/27670172]
include_images:false

I sent you a PM with an example of a story URL from this particular author as it is marked as explicit on AO3 (so technically NSFW). I could live with a single GIF download per update, but not 20 or more.

As far as my setup is concerned, I'm not deleting the GIF images in between fic updates and I do have FFF set to update fics and not overwrite them.

I had a thought that maybe it has something to do with FFF using AO3's download mechanism that was causing it, but just wasn't sure.

[JimmXinu]

Originally Posted by steltek

I can live without the cover image so long as I don't have to wait through 20 different web pages opening to move on.

...

I sent you a PM with an example of a story URL from this particular author as it is marked as explicit on AO3 (so NSFW). I could live with a single download per update, but not 20 or more.

Fair enough, thanks for respecting MR's NSFW rules.

I don't see anything unusual with the images in that story. Existing images are re-used as expected in my tests. Unless the author is changing them between updates, which I wouldn't see in my testing right now.

OTOH, I'm only getting 9 images in the entire story, not 20.

You said "wait through 20 different web pages opening" -- are you using BrowserCache? Because that's an important detail--I'm using use_archiveofourown_gay:true.

When I try with BrowserCache, the images are all already found in cache because I viewed the full-work page.


Another idea, you could exclude the post-chapter notes entirely with exclude_notes:

## AO3 chapters can include several different types of notes.  We've
## traditionally included them all in the chapter text, but this allows
## you to customize which you include.  Copy this parameter to your
## personal.ini and list the ones you don't want.
#exclude_notes:authorheadnotes,chaptersummary,chapterheadnotes,chapterfootnotes,authorfootnotes,inspiredlinks

[steltek]

Originally Posted by JimmXinu

Fair enough, thanks for respecting MR's NSFW rules.

I don't see anything unusual with the images in that story. Existing images are re-used as expected in my tests. Unless the author is changing them between updates, which I wouldn't see in my testing right now.

OTOH, I'm only getting 9 images in the entire story, not 20.

You said "wait through 20 different web pages opening" -- are you using BrowserCache? Because that's an important detail--I'm using use_archiveofourown_gay:true.

When I try with BrowserCache, the images are all already found in cache because I viewed the full-work page.


Another idea, you could exclude the post-chapter notes entirely with exclude_notes:

## AO3 chapters can include several different types of notes.  We've
## traditionally included them all in the chapter text, but this allows
## you to customize which you include.  Copy this parameter to your
## personal.ini and list the ones you don't want.
#exclude_notes:authorheadnotes,chaptersummary,chapterheadnotes,chapterfootnotes,authorfootnotes,inspiredlinks

Yes, I am using the Browser Cache feature. I previously stopped using use_archiveofourown_gay:true due to the same problems I was seeing with the main site, so maybe it is time to turn it back on again.

And, when I say 20 pages, I literally mean it. FFF often opens the same ones multiple times. And, this is even after I have viewed the full work page. As a test (before I applied the code you supplied above), I downloaded the fic again and it opened 21 browser tabs.

All that being said, I do habitually clear the web browser cache more or less weekly as for some reason I have had some issues with FFF finding Fanfiction.net fics in the browser cache if I let it go longer than that.

In any event, the exclude_notes code you provided above solves the problem for me as long as I exclude chapterfootnotes and authorfootnotes. I also found that downloading the ePub using Ficlab also excludes them (though, I do prefer to use FFF as it is both more convenient and also limits consumed download bandwidth on the hosting sites).

I appreciate the assistance!

[JimmXinu]

Originally Posted by steltek

And, when I say 20 pages, I literally mean it. FFF often opens the same ones multiple times. And, this is even after I have viewed the full work page. As a test (before I applied the code you supplied above), I downloaded the fic again and it opened 21 browser tabs.

I'm not sure what would cause that. I would try setting slow_down_sleep_time higher, default is 4, maybe try 6?

[texasnightowl]

Originally Posted by JimmXinu

You absolutely can make personal.ini settings specific to a story. Example:

[https://archiveofourown.org/works/27670172]
include_images:false

oh! nice! I guess I hadn't looked deeper than per site. Thanks!