[GUI Plugin] FanFicFare

[JimmXinu]

Have you already reviewed the two available work around solutions?

Namely, the Browser Cache Feature implemented directly in FFF and NSAPA's (experimental) proxy system.

FYI, I continue to not see any problems with light ffnet downloads and updates. I think the people seeing issues are on networks that, for whatever reason, CF has flagged for additional checking.

Based on past experience, if everybody were having problems, we'd see a lot more people complaining.

[Science Friction]

Thanks, I haven't tried either the browser cache or captcha proxy so far, since up til now I've been running fff on a headless server (but NOT for very heavy downloading). I've since tried on 3 residential networks and a mobile network and gotten v2 challenge every time. I guess I can try the browser cache or proxy approach though having to manually load dozens or 100+ chapters of a long fic sounds like a nuisance, and someone upthread says cloudflare can now tell if selenium is running the browser. I might try my hand at writing a firefox extension to get the chapters sometime. But I feel like it is more promising in the long run to keep looking for ways to bypass ffnet entirely.

Unrelated issue: I ran "pip install fanficfare" and it installed the fff python code (written in python 3) but tries to run it under python2, which fails badly. I had to uninstall and reinstall with pip3. Probably best to disable the python2 installation somehow.

(Does anyone know how many posts I need before I stop getting the recaptcha on every post, if it ever stops)?

[Ascello]

I had same issue recently (as I mentioned in previous post) but it got resolved the next day, everytime. I just assume they temporarily block ppl who download too much too fast.

I don't use browser cache. If the issue ever bother me, I will, but so far I prefer to just save a link and try again the next day.

[ownedbycats]

@Science Friction - I believe the webtoepub plugin will spin through all the chapters and load them into the cache, if you'd like to try that.

[Chrystalline]

Originally Posted by JimmXinu

Question for Plugin Users

1. Output Format
2. Update Mode
3. Update Calibre Metadata
4. Update EPUB Cover
5. Background Metadata (updates only)

I do sometimes switch Output Format, because while my overwhelming preference is plain HTML, the AO3 stories that have a bazillion embedded images or are only images don't work unless I switch it to epub.

I switch Update Mode depending on whether I'm updating stories to which I've subscribed or downloading someone's list of recommendations (when I'll skip existing rather than update).

I leave the Calibre Metadata option always checked and the EPUB cover always unchecked.

[Science Friction]

Originally Posted by Ascello

I had same issue recently (as I mentioned in previous post) but it got resolved the next day, everytime. I just assume they temporarily block ppl who download too much too fast. I don't use browser cache. If the issue ever bother me, I will, but so far I prefer to just save a link and try again the next day.

Interesting. Over the past several days I tried from at least 7 different networks (3 residential, 1 mobile, and several data center) and consistently got v2 captcha, but today I got a successful first chapter download followed by problem reading AIUI delay on all the subsequent chapters. Unfortunately fanficfare --debug doesn't save or show the text that didn't parse (there is just a null regex result exception). I may try modifying the code to log this and possibly use some reasonable default delay.

Anyway, you're right, this does seem to be non-deterministic and it resets (or at least changes) after a while. I wonder whether cloudflare detects heavy downloading towards the entire site, rather than from a specific IP or network, in deciding to put up the v2 challenge.

@JimmXinu, another suggestion: when it's an AO3 download, could fff possibly use the "entire work" link to get the whole story in one operation, instead of retrieving 1 chapter at a time? Thanks.

[ownedbycats]

It already does.

## AO3 offers a page to download the entire work at once.  This can
## save a lot of network traffic if there are many chapters.
## True by built-in default.
#use_view_full_work:true

[Science Friction]

Originally Posted by ownedbycats

It already does.

Thanks! I wonder if that is new. I am pretty sure it didn't do that when I tried some time back, but it does now. I also hadn't noticed that option before. I only recently started messing with fff again.

Meanwhile I did a git clone and am finding it very confusing to even run the repo version of fff. After "python setup.py build", trying to retrieve a fic gets some kind of cache configuration error (I ran cli.py with -i example.ini). I can probably debug this but it would be helpful to have some simple instructions (maybe in the faq) for running this way. My simple workaround for now will probably be to use "setup.py install" in a separate VM to run my modified versions.

[JimmXinu]

Originally Posted by Science Friction

...

use_view_full_work has been in for ~4 years.

I test and maintain CLI FFF installed via pip. I probably don't follow all the 'correct' python conventions otherwise.

This particular forum is specific to FFF Calibre plugin, BTW.

[Science Friction]

Thanks, no idea about ao3 single page then. I guess I should break down and enroll a microsoft (i.e. github) account for CLI stuff.

I used the setup.py install approach in a VM to run debugging builds and that worked. Interestingly I was able to download from ffnet on that VM while I hadn't been able to from several other VM's. A very minor suggestion: at line 447 of fetcher.py where there is a debug message reporting the response code, could you also include the length of the response content? I always find that useful to indicate whether a web retriever is seeing the right thing. To look into the IAUA error I'll have to dump the content as well, but before I can do that I'll have to reproduce it.

I'm a good python coder but I don't know the packaging conventions and so on either. I wonder if there is a good one-stop place to read about those.

[ackomb]

@JimmXinu - thank for the help the other day, I already had in in my .ini file but once again I had figured the order of things the other way around. Either way it's working.

New thing though, I've been fiddling a bit with extra valid entries based om some stuff I read on this forum as well as on Github. I've gotten quite far but I'm running into the following:

Same story: https://archiveofourown.org/works/28121997

Using the following code:

## [defaults] section applies to all formats and sites but may be overridden at several levels.  
[defaults]
##Chrome Cache
Browser_cache_path:C:\Users\kombr\AppData\Local\Google\Chrome\User Data\Default\Cache

is_adult:true
include_titlepage: true

[www.archiveofourown.org] 
username:***
password:***

use_view_full_work:true 

add_to_extra_valid_entries:,ships2chars,relationships,crossovers
include_in_ships2chars:ships
include_in_relationships:ships
include_in_crossovers:genre

## Tells which tags from the site to include in calibre
include_subject_tags:extratags
include_in_characters:characters,ships2chars
include_in_genre:freeformtags, ao3categories,crossovers
include_in_category:fandoms

add_to_replace_metadata:
 ships2chars=>[ ]*/[ ]*=>/
 ships2chars=>[ ]*&[ ]*=>/
 ships2chars=>([^/]+)/([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3\,\4
 ships2chars=>([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3
 ships2chars=>([^/]+)/(.+)=>\1\,\2

add_to_include_subject_tags:,tagsfromtitle.SPLIT

## Tells which tags from the site to include on the titlepage
titlepage_entries: series,category,genre,characters,status,datePublished,dateUpdated,rating,numChapters,numWords,site,description

## Tells which tags goes to which customs column
custom_columns_settings:
 genre=>#genre,a
 category=>#show,a
 crossovers=>#show,a
 crossovers=>#crossover,a
 characters=>#m_char,a
 relationships=>#m_pair,a

## Excludes entire tags before being processed further
exclude_metadata_pre:
 freeformtags,genre==Other Additional Tags to Be Added
 warnings==Creator Chose Not To Use Archive Warnings
 warnings==No Archive Warnings Apply

replace_metadata:
## Regular cleanup
## Replaces & with / in 'ships' and 'genre'
 genre,category,characters=> & =>/
 genre,category,characters=>&=>/ 
 ships,genre,category,characters=> \| =>/

then there is a bunch of changing characters, genres etc. I won't spam you with that, main thing to get out of it is that I amend the tags:

 genre=>^Minor crossover with Private Practice characters$=>Crossover.Private Practice
 genre=>^Minor crossover with Grey's Anatomy character=>Crossover.Grey's Anatomy

And added them to categories for which I use a customs column "#show"

So far so good, but here's what's getting to me, from the extra valid entry I only want the ones with "Crossover" in then, all the other I want to toss, and I thought below would work, but unfortunately it doesn't.

include_metadata_post:
 crossovers=>.*Crossover.*

It's probably some stupid thing again I'm overlooking or I'm just interpreting the use of this code wrong. Either way,

[Xyla]

Hi, I've tried downloading fics from spikeluver.com but all I get is: 'NoneType' object has no attribute 'a'

Example fic: https://spikeluver.com/SpuffyRealm/v...y.php?sid=1424

[JimmXinu]

Originally Posted by ackomb

...

You have Browser_cache_path instead of browser_cache_path. Unrelated to current problem.

I don't know exactly what was going wrong, but this works for me:

Spoiler:

[www.archiveofourown.org] 

use_view_full_work:true 

add_to_extra_valid_entries:,ships2chars,relationships,crossovers
include_in_ships2chars:ships
include_in_relationships:ships
include_in_crossovers:genre

## Tells which tags from the site to include in calibre
include_subject_tags:extratags
include_in_characters:characters,ships2chars
include_in_genre:freeformtags, ao3categories,crossovers
include_in_category:fandoms

add_to_replace_metadata:
 ships2chars=>[ ]*/[ ]*=>/
 ships2chars=>[ ]*&[ ]*=>/
 ships2chars=>([^/]+)/([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3\,\4
 ships2chars=>([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3
 ships2chars=>([^/]+)/(.+)=>\1\,\2
 genre,category,characters=> & =>/
 genre,category,characters=>&=>/ 
 ships,genre,category,characters=> \| =>/

add_to_include_subject_tags:,tagsfromtitle.SPLIT

## Tells which tags from the site to include on the titlepage
titlepage_entries: series,category,genre,characters,status,datePublished,dateUpdated,rating,numChapters,numWords,site,description,ships2chars,relationships,crossovers

## Tells which tags goes to which customs column
custom_columns_settings:
 genre=>#genre,a
 category=>#show,a
 crossovers=>#show,a
 crossovers=>#crossover,a
 characters=>#m_char,a
 relationships=>#m_pair,a

## Excludes entire tags before being processed further
exclude_metadata_pre:
 freeformtags,genre==Other Additional Tags to Be Added
 warnings==Creator Chose Not To Use Archive Warnings
 warnings==No Archive Warnings Apply

include_metadata_post:
 crossovers=>Crossover

replace_metadata:
 genre=>^Minor crossover with Private Practice characters=>Crossover.Private Practice
 genre=>^Minor crossover with Grey's Anatomy character=>Crossover.Grey's Anatomy

Originally Posted by Xyla

Hi, I've tried downloading fics from spikeluver.com but all I get is: 'NoneType' object has no attribute 'a'

Example fic: https://spikeluver.com/SpuffyRealm/v...y.php?sid=1424

I'm not seeing this problem. Are you using an old version? There was a change back in January for that site.

FYI, I have limited free time until mid-next week; expect slow responses.

[ownedbycats]

Question:

I run from source and noticed this in console:

FFF: DEBUG: 2021-05-09 03:15:16,091: calibre_plugins.fanficfare_plugin.fff_plugin(2324): No metadata value for "N", setting custom column(#read) literally to N.

Is that expected?

Also, aside from watching the console, is there any way to see what chapter a job is currently on?

[Xyla]

Originally Posted by JimmXinu

I'm not seeing this problem. Are you using an old version? There was a change back in January for that site.

No, I'm using the latest test version.