[GUI Plugin] FanFicFare

[theladyoffangorn]

Originally Posted by ernand

A problem that suddenly started happening to me is that FFF is asking for my AO3 login information to update my fics. I already have that information on the .ini, and I have not changed it.

I keep getting the Error: Failed to Login for URL: https://archiveofourown.org/works/57...iew_adult=true) with username: (Ernand)

I am having the same issue, started this evening. I have checked and I am using latest version of both Calibre and FFF, and have checked my personal.ini and it is correct (and worked for the last month).

[theladyoffangorn]

This is the entire error message I'm getting. Tried logging out from AO3 on any other devices
calibre, version 7.26.0
ERROR: Unhandled exception: <b>FailedToLogin</b>:Failed to Login for URL: (https://archiveofourown.org/users/Th...marks?page=156) with username: (TheLadyOfFangorn)

calibre 7.26 embedded-python: True
Windows-10-10.0.19045-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19045')
Python 3.11.5
Windows: ('10', '10.0.19045', 'SP0', 'Multiprocessor Free')
Interface language: None
EXE path: C:\Program Files\Calibre2\calibre.exe
Successfully initialized third party plugins: FanFicFare (4, 42, 0)
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 711, in get_urls_from_page_menu
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 734, in get_urls_from_page
File "C:\Users\rick\AppData\Roaming\calibre\plugins\Fan FicFare.zip\fanficfare\geturls.py", line 44, in get_urls_from_page
File "C:\Users\rick\AppData\Roaming\calibre\plugins\Fan FicFare.zip\fanficfare\adapters\base_adapter.py", line 474, in get_urls_from_page
File "C:\Users\rick\AppData\Roaming\calibre\plugins\Fan FicFare.zip\fanficfare\adapters\base_otw_adapter.p y", line 605, in before_get_urls_from_page
File "C:\Users\rick\AppData\Roaming\calibre\plugins\Fan FicFare.zip\fanficfare\adapters\base_otw_adapter.p y", line 127, in performLogin
fanficfare.exceptions.FailedToLogin: Failed to Login for URL: (https://archiveofourown.org/users/Th...marks?page=156) with username: (TheLadyOfFangorn)

[readerboy7]

I am unable to download the following fanfic from AO3: https://archiveofourown.org/works/49500382 (locked to registered users only)

Calibre 7.26, FanFicFare v4.42.0
Personal.ini does contain my username and password

Error message when run through calibre is
'NoneType' object has no attribute 'get_text'

Error message when run via
calibre-debug.exe --run-plugin FanFicFare -- https://archiveofourown.org/works/49500382
is

Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 83, in <module>
  File "site.py", line 78, in main
  File "site.py", line 50, in run_entry_point
  File "calibre\debug.py", line 289, in main
  File "calibre_plugins.fanficfare_plugin.__init__", line 146, in cli_main
  File "C:\Users\{Name}\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\cli.py", line 346, in main
  File "C:\Users\{Name}\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\cli.py", line 322, in dispatch
  File "C:\Users\{Name}\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\cli.py", line 437, in do_download
  File "C:\Users\{Name}\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\base_adapter.py", line 342, in getStoryMetadataOnly
  File "C:\Users\{Name}\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\base_adapter.py", line 446, in doExtractChapterUrlsAndMetadata
  File "C:\Users\{Name}\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\base_otw_adapter.py", line 199, in extractChapterUrlsAndMetadata
  File "C:\Users\{Name}\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\htmlcleanup.py", line 77, in stripHTML
AttributeError: 'NoneType' object has no attribute 'get_text'

I have been able to download other fanfics from AO3 recently, but I don't think any of them were registered users only.

(Edit: windows 10 computer, if that matters)

[grimorie]

Originally Posted by ernand

A problem that suddenly started happening to me is that FFF is asking for my AO3 login information to update my fics. I already have that information on the .ini, and I have not changed it.

I keep getting the Error: Failed to Login for URL: https://archiveofourown.org/works/57...iew_adult=true) with username: (Ernand)

I have the same problem with downloading from Ao3 asking for login, I'm using a mac, if that helps.

[madame_newbie]

I ran into the same problem as well as off today. It keeps asking me to log in even for works that aren't behind a lock. I do think it could on AO3's side because downloading has been slow for me (my downtime's set to 6) but it was slow even for one medium length fic.

[Pixiestyxs]

I've also been having the `'NoneType' object has no attribute 'get_text'` error, it seems like every 1 out of 5 urls from AO3 give me this error. It just started yesterday it seems like. I'm on Mac.

I have more examples but all of these are freshly giving me the error this morning:
https://archiveofourown.org/works/8130674
https://archiveofourown.org/works/5519189
https://archiveofourown.org/works/6455950
https://archiveofourown.org/works/7194323
https://archiveofourown.org/works/8941855

[JimmXinu]

Instead of the usual "This work is only available to registered users of the Archive." or "Sorry, we couldn&#x27;t find the work you were looking for.", the site is returning a page that says:

Reminder:
This site is in beta. Things may break or crash without notice.

There's a notice linked on the front page about "Intermittent site slowness and errors" including:

We estimate the new Elasticsearch servers will be in place by early April. Until then, you might run into the following issues, especially during busy periods:

• all pages loading more slowly
• Elasticsearch-powered pages like search results and work and bookmark listings taking longer to update
• error pages
• automated checks from Cloudflare's Under Attack mode
• stricter rate limiting
• issues with services like the Wayback Machine or Tumblr RSS accounts that rely on bots, scrapers, or other automated tools, which we have deprioritized in favor of traffic from users

FFF likely is affected by one or more of these. And note that part about early April. This may be going on for a some time.

"Fix" (but not really)

• I can add code to detect the "in beta" page and report it instead of parsing failing.
• I can increase slow_down_sleep_time in defaults.ini for AO3.

I honestly don't know of anything else I can do.

Workaround

If you are already logged in, in your browser, the Browser Cache feature appears to work:

[archiveofourown.org]
always_login:false
## comment out username at least, otherwise Get Stories From Page will try to login
# username:XXX
# password:XXX
use_browser_cache:true
use_browser_cache_only:true
open_pages_in_browser:true

[Pixiestyxs]

Thanks! I'll try the workaround and just count down the days until April. I appreciate all the work you do on this.

[JimmXinu]

New Test Version Posted

2025-02-24
- base_otw_adapter: Detect & report 'This site is in beta' page
- AO3: Double default slow_down_sleep_time

[JimmXinu]

FYI, AO3 view full work is also failing. The automatic fall back to per-chapter download works, but it takes longer.

[Amalthia]

Thanks for posting the work around. I'll give it a shot. Using a VPN really messes up the rate limit on cloudflare...hopefully browser cache is sufficient.

[Chrystalline]

I can't get the browser cache to work anymore. It's ignoring the path I gave it and trying to use the default browser instead, which is a browser I don't use. You can't set portable apps to be a default. It worked a week ago; I don't understand what's changed.

I have the path under defaults in personal ini, and use_browser_cache:true and use_browser_cache_only:true under the site specific settings for AO3 and FFN.

[JimmXinu]

If FFF is looking in the wrong cache directory, search your personal.ini, maybe you have more than one browser_cache_path setting.

If you have open_pages_in_browser:true, and the pages FFF wants cached are opening in the default browser, that's how it works.

You could try changing Calibre's openers_by_scheme tweak, but the last report I had from some one trying that, it didn't work.

[fishing_forever]

I've been trying to use the workaround and set the browser_cache_path in defaults. It opens the fic link in my default browser (Firefox), but then I get this error:

HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428)

I tried setting the browser_cache_age_limit to -1 but that didn't do anything.

When I try to update the same fic again with FFF, it doesn't open it in my browser a second time.

[Chrystalline]

Ah, I tried adding open pages in browser, but when I took it back out, it was stuck. Closing and restarting got the FFN working again. I've never gotten it working with AO3.

Does it make a difference which URL is used? I've tried:
https://archiveofourown.org/works/57...full_work=true which is how it appears when I open the page in the browser.
https://archiveofourown.org/works/57094942 is how it links in my notifications.
https://archiveofourown.org/works/57...56639#workskin is the third chapter if I change it to view chapter by chapter in the browser.

I admit I've copy/pasted a lot of ini code from other people, because I don't fully understand what does what, so I don't know if something in here is causing the problem.

Spoiler:

[defaults]
## [defaults] section applies to all formats and sites but may be
## overridden at several levels.  Example:

## [defaults]
## titlepage_entries: category,genre, status
## [www.whofic.com]
## # overrides defaults.
## titlepage_entries: category,genre, status,dateUpdated,rating
## [epub]
## # overrides defaults & site section
## titlepage_entries: category,genre, status,datePublished,dateUpdated,dateCreated
## [www.whofic.com:epub]
## # overrides defaults, site section & format section
## titlepage_entries: category,genre, status,datePublished
## [overrides]
## # overrides all other sections
## titlepage_entries: category
keep_in_order_characters:true
keep_in_order_category:true
keep_in_order_freeformtags:true
keep_in_order_genre:true
keep_in_order_fandoms:true
keep_in_order_ships:true
characters_label:Characters
ships_label:Relationships
series_label:Series
seriesUrl_label:Series URL
extratags_label:Extra Tags
replace_hr: false
include_images:true
no_image_processing: true
is_adult:true


## As a work around for certain sites blocking automated downloads,
## FFF now offers the ability to look for pages in your Chrome(or
## Chromium-derived) or Firefox browser's cache.  Requires both
## use_browser_cache:true and browser_cache_path to be set.
##
## browser_cache_path needs to be set to the location of YOUR browser
## cache.  Here are a few examples for different OS.  Note that all
## have YOUR user name in them somewhere as well as a Profile name,
## frequently Default.  Make sure you have personalized (and
## uncommented) ONE browser_cache_path setting.

## Note also that browser_cache_path goes under [defaults] because
## browser cache is read once and shared between all downloads for
## performance.

## Further note that there is a time cost to first reading the browser
## cache on each download session (not Calibre session)
## windows:
### Chrome:
#browser_cache_path:C:\Users\YourUser\AppData\Local\Google\Chrome\User Data\Default\Cache
#browser_cache_path:C:\Users\YourUser\AppData\Local\Google\Chrome\User Data\Profile 1\Cache
### Firefox
#browser_cache_path:C:\Users\YourUser\AppData\Local\Mozilla\Firefox\Profiles\ZjwI7Fo4.default\cache2

### FFDL
# browser_cache_path:C:\Users\YourUser\AppData\Local\fanficdownloader\cache\QtWebEngine\Default\Cache

### Opera:
browser_cache_path:C:\Users\YourUser\Desktop\Opera\Data\profile\data\Default\Cache\Cache_Data

## It's common for browser cached files to be kept for several hours
## or more.  You can limit the age of cached files FFF will use from
## browser_cache_path with browser_cache_age_limit.  Only cached files
## that were downloaded within 'browser_cache_age_limit' hours will be
## used.  If set to -1, all cached files will be used.
browser_cache_age_limit:-1

[archiveofourown.org]

## In order to get bookmarktags and bookmarksummary, you need to login
## all the time. This defaults to off to save time and network
## traffic. Requires valid AO3 username and password when true.
always_login:false

use_browser_cache:true
use_browser_cache_only:true

## AO3 offers a page to download the entire work at once. This can
## save a lot of network traffic if there are many chapters.
## True by built-in default.
#use_view_full_work:true

## AO3 allows users to archive stories they didn't write in certain
## cases.  These are indicated by showing a byline such as:
## 'Orig Author [archived by Archivist Author]'
## If use_archived_author is set true (and there's only one author
## listed), author will be set to 'Orig Author' instead of 'Archivist
## Author'.  authorUrl will still point to the Archivist Author's page.
use_archived_author:true

## Some adapters collect additional meta information beyond the
## standard ones. They need to be defined in extra_valid_entries to
## tell the rest of the FanFicFare system about them. They can be
## used in include_subject_tags, titlepage_entries,
## extra_titlepage_entries, logpage_entries, extra_logpage_entries,
## and include_in_* config items. You can also add additional entries
## here to build up composite metadata entries.
## archiveofourown.org, for example, fills genre (a standard
## entry) as the composite offreeformtags, ao3categories in
## include_in_genre. If there's ever more than 4 series, add
## series04,series04Url etc.
extra_valid_entries:fandoms, freeformtags, freefromtags, ao3categories, comments, chapterslashtotal, chapterstotal, kudos, hits, bookmarks, collections, byline, bookmarked, bookmarktags, bookmarksummary, bookmarkprivate, bookmarkrec, restricted, series00, series01, series02, series03, series00Url, series01Url, series02Url, series03Url, series00HTML, series01HTML, series02HTML, series03HTML
fandoms_label:Fandoms
freeformtags_label:Freeform Tags
freefromtags_label:Freeform Tags
ao3categories_label:AO3 Categories
comments_label:Comments
chapterslashtotal_label:Chapters/Total Chapters
chapterstotal_label:Total Chapters
kudos_label:Kudos
hits_label:Hits
collections_label:Collections
## Count of bookmarks on story by all users
bookmarks_label:Bookmarks
## Tags & Summary from *your* bookmark on the story. Only collected
## if always_login:true
#bookmarked_label:I Bookmarked Story
#bookmarktags_label:My Bookmark Tags
#bookmarksummary_label:My Bookmark Summary
#bookmarkprivate_label:My Bookmark Private
#bookmarkrec_label:My Bookmark Rec
#restricted_label:Restricted to Registered Users
#series00HTML_label:Series
#series01HTML_label:Series2
#series02HTML_label:Series3
#series03HTML_label:Series4

## Assume entryUrl, apply to "<a class='%slink' href='%s'>%s</a>" to
## make entryHTML.
make_linkhtml_entries:series00,series01,series02,series03

## AO3 doesn't have anything it calls 'genre'.  The adapter used to be
## hardcoded to include the site specific metadata freeformtags &
## ao3categories in the standard metadata field genre.  By making it
## configurable, users can change it.
include_in_genre: freeformtags, ao3categories

## AO3 uses the word 'category' differently than most sites.  The
## adapter used to be hardcoded to include the site specific metadata
## fandom in the standard metadata field category.  By making it
## configurable, users can change it.
include_in_category:fandoms

## freeformtags was previously typo'ed as freefromtags.  This way,
## freefromtags will still work for people who've used it.
include_in_freefromtags:freeformtags

## chapterslashtotal contains 1/3 or 1/1 or 3/? etc as reported by AO3.
## chapterstotal is just the total chapters part after the /
include_in_chapterstotal:chapterslashtotal.NOREPL
add_to_replace_metadata:
chapterstotal=>^([0-9])+/(.+)$=>\2

## adds to titlepage_entries instead of replacing it.
extra_titlepage_entries: fandoms,freeformtags,ao3categories,comments,chapterslashtotal,chapterstotal,kudos,hits,bookmarks,bookmarked,bookmarktags,bookmarksummary,series01HTML,series02HTML,series03HTML,byline

## adds to include_subject_tags instead of replacing it.
extra_subject_tags:fandoms,freeformtags,ao3categories

## extra_valid_entries defines valid custom metadata entries.
## Need to use add_to_ because defaults.ini already defines several
## under [archiveofourown.org].  Yes, the leading , is needed
## See https://github.com/JimmXinu/FanFicFare/wiki/INI-File#add_to_keyword
add_to_extra_valid_entries:,platships,romships

## Labels used if included in titlepage_entries or logpage_entries
platships_label:Platonic Relationships
romships_labels:Romantic Relationships

## essentially copy ships to platships and romships.  
## **After** replace_metadata and in/exclude on ships so
## and edits to ships will be applied to both still.
## Use ships.NOREPL to not get ships edits.
include_in_platships:ships
include_in_romships:ships

## Include  **only** platships containing & and romships containing /
## & html entity is used for & in FFF entries for historical reasons.
include_metadata_pre:
 platships=~&
 romships=~/

## not necessary, but can be useful to see this part is working if nothing else
add_to_titlepage_entries:,platships,romships

## put platships & romships into their custom columns.
## Quite a few ini parameters are documented primarily in
## the defaults.ini file
## See https://github.com/JimmXinu/FanFicFare/blob/main/calibre-plugin/plugin-defaults.ini#L345
add_to_custom_columns_settings:
platships=>#platships
romships=>#romships
kudos=>kudos
hits=>hits
bookmarks=>bookmarks
comments=>comments

## AO3 chapters can include several different types of notes.  We've
## traditionally included them all in the chapter text, but this allows
## you to customize which you include.  Copy this parameter to your
## personal.ini and list the ones you don't want.
#exclude_notes:authorheadnotes,chaptersummary,chapterheadnotes,chapterfootnotes,authorfootnotes

## AO3 is blocking people more aggressively.  If you download fewer
## stories less often you can likely get by with reducing this sleep.
slow_down_sleep_time:20

## AO3 allows users to archive stories they didn't write in certain
## cases. These are indicated by showing a byline such as:
## 'Orig Author [archived by Archivist Author]'
## If use_archived_author is set true (and there's only one author
## listed), author will be set to 'Orig Author' instead of 'Archivist
## Author'. authorUrl will still point to the Archivist Author's page.
#use_archived_author:false

## AO3 chapters have 'date' available for each chapter.
## These can be used with custom output (see
## https://github.com/JimmXinu/FanFicFa...i/CustomOutput ) or with
## chapter_title_*_pattern settings.
## Examples for html & epub:
#[html]
#tocpage_entry:
# <a href="#section${index04}">${chapter}</a> ${date}<br />
#[epub]
#tocpage_entry:
# <a href="file${index04}.xhtml">${chapter}</a> ${date}<br /><br />

## The 'date' value for chapters mentioned above can be formated with
## datechapter_format. Otherwise it will default to
## datePublished_format
#datechapter_format:%%Y-%%m-%%d

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.) There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'. 'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only. (Default is 'r'.)

## Literal strings can be set into custom columns using double quotes.
## Each metadata=>column mapping must be on a separate line and each
## needs to have one space at the start of each line.

## 'r_anthaver' and 'n_anthaver' can be used to indicate the same as
## 'r' and 'n' for normal downloads, but to average the metadata for
## the differents story in an anthology before setting in integer and
## float type custom columns. This can be useful for a averrating
## column, for example. Default is to sum the values of all stories,
## and numChapters and numWords are always summed.

custom_columns_settings:
series01=>#series2
series02=>#series3
series03=>#series4
# cliches=>#acolumn
# themes=>#bcolumn,a
# timeline=>#ccolumn,n
# "FanFiction"=>#collection
# averrating=>#averrating,r_anthaver

[www.fanfiction.net]

## browser cache will only be used if use_browser_cache:true and ONLY
## for fanfiction.net right now.  Requires a browser_cache_path set in
## [defaults].
use_browser_cache:true

## use_browser_cache_only:true prevents FFF from falling through to
## request from the server when a page isn't in the browser cache.
## You can turn this on to be sure you're only using the browser
## cache.
use_browser_cache_only:true

## Pre-existing ffnet INI settings you might want to think about
## changing with use_browser_cache:

## If you want covers to work correctly with use_browser_cache, you'll
## need to click the cover image at least once for each story.  Or
## turn cover off.
never_make_cover:true

## If you want skip_author_cover to work correctly with
## use_browser_cache, you'll need to click the author page at least
## once for each story.  Or turn skip_author_cover off.
##
## From defaults.ini:
## April 2015, ffnet changed their story and user pictures urls to be
## the same.  Now the only way to know if the story image is really
## the author image is to go get the author image and check.
skip_author_cover:false

## If you turned this on before, you probably want it off while using
## browser cache.
##
## From defaults.ini:
## ffnet used to have a tendency to send out update notices in email
## before all their servers were showing the update on the first
## chapter.  It generates another server request and doesn't seem to
## be needed lately, so now default it to off.
check_next_chapter:true