[GUI Plugin] FanFicFare

[JimmXinu]

Putting (?i) at the start of the regexp makes the entire expression case insensitive. The Python docs on it are very limited.

The example I tested was:

add_to_replace_metadata:
 title=>(?i)TEST=>teeest!

Which replaced Test only when (?i) was present.

[JimmXinu]

Originally Posted by shaybella

The followup question then: why is FFF blocked but not my browser?

Because many sites now employ specialized software and services (Cloudflare is the only one I know by name) to specifically detect and block access when there isn't a full browser with a user sitting at it. Any site that has ads has a profit motive to make sure your eyes see the ads.

To bring it back full circle, why you and not everybody; applying that level of blocking to everybody slows everything down and burns more CPU cycles and electricity. So it's also common that different levels of detection and blocking are applied to different lists of IPs.

[thenico]

On the Calibre provided by Fedora 33 (that's Calibre 4.23 running on Python 3.9.5), you need remove the included_dependencies/bs4 folder before running makeplugin.py.
I supposed removing the bs4 folder from the zip would also works.

If you don't do that, Calibre' python3 will try to load the FanFicFare' embedded version of Beautiful Soup that is not compatible with Python3.
The error message will be the following:

calibre, version 4.23.0 (linux, embedded-python: False)
Failed to Download Stories: Échoué: Download 1 FanFiction Book(s)

Download 1 FanFiction Book(s)
FFF: INFO: 2021-06-05 02:01:49,507: calibre_plugins.fanficfare_plugin.jobs(49): CPUs:3
FFF: INFO: 2021-06-05 02:01:49,507: calibre_plugins.fanficfare_plugin.jobs(52): FanFicFare v4.3.2
FFF: INFO: 2021-06-05 02:01:49,507: calibre_plugins.fanficfare_plugin.jobs(68): Launch background process for site archiveofourown.org:
https://archiveofourown.org/works/29157849
FFF: INFO: 2021-06-05 02:01:50,439: calibre_plugins.fanficfare_plugin.jobs(130): 

================================================================================ site:(archiveofourown.org)
Traceback (most recent call last):
  File "/usr/bin/calibre-parallel", line 20, in <module>
    sys.exit(main())
  File "/usr/lib64/calibre/calibre/utils/ipc/worker.py", line 208, in main
    result = func(*args, **kwargs)
  File "/usr/lib64/calibre/calibre/utils/ipc/worker.py", line 145, in arbitrary_n
    return func(*args, **kwargs)
  File "calibre_plugins.fanficfare_plugin.jobs", line 186, in do_download_site
  File "calibre_plugins.fanficfare_plugin.jobs", line 199, in do_download_for_worker
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "/usr/lib64/calibre/calibre/customize/zipplugin.py", line 184, in load_module
    exec(compiled, mod.__dict__)
  File "calibre_plugins.fanficfare_plugin.dialogs", line 69, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/home/nico/.config/calibre/plugins/FanFicFare.zip/fanficfare/geturls.py", line 33, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 982, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1414, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1388, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1369, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 423, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 716, in spec_from_file_location
  File "<frozen zipimport>", line 191, in get_filename
  File "<frozen zipimport>", line 713, in _get_module_code
  File "<frozen zipimport>", line 647, in _compile_source
  File "/home/nico/.config/calibre/plugins/FanFicFare.zip/bs4/__init__.py", line 52
    'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work.'<>'You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
                                                                                                      ^
SyntaxError: invalid syntax

Traceback (most recent call last):
  File "/usr/bin/calibre-parallel", line 20, in <module>
    sys.exit(main())
  File "/usr/lib64/calibre/calibre/utils/ipc/worker.py", line 208, in main
    result = func(*args, **kwargs)
  File "/usr/lib64/calibre/calibre/utils/ipc/worker.py", line 145, in arbitrary_n
    return func(*args, **kwargs)
  File "calibre_plugins.fanficfare_plugin.jobs", line 134, in do_download_worker
TypeError: 'NoneType' object is not iterable

[JimmXinu]

Originally Posted by thenico

On the Calibre provided by Fedora 33 (that's Calibre 4.23 running on Python 3.9.5), you need remove the included_dependencies/bs4 folder before running makeplugin.py.
I supposed removing the bs4 folder from the zip would also works.

If you don't do that, Calibre' python3 will try to load the FanFicFare' embedded version of Beautiful Soup that is not compatible with Python3.

The included version of BeautifulSoup(bs4 AKA BS) is for Calibre v2 only and when FFF (as I release it) is run using Kovid's builds of Calibre it is ignored except when needed for the older versions. Calibre bundles a copy of BS in newer versions.

Calibre before v5 is itself officially not Python3 compatible, so if they (or you) are some how running Calibre 4.23 on Python 3.9, that's already unsupported to the best of my knowledge.

And yes, I do care more about keeping FFF running on Calibre 2 more than on everybody's Linux distro versions--it (reportedly) still has more users.

From https://calibre-ebook.com/dynamic/calibre-usage

Usage share by major version
5	62.0%
4	17.4%
3	14.0%
2	4.5%
1	1.1%
0	0.9%

Operating systems calibre runs on
win	82.6%
osx	14.0%
oth	3.4%

[thenico]

Originally Posted by JimmXinu

Calibre before v5 is itself officially not Python3 compatible, so if they (or you) are some how running Calibre 4.23 on Python 3.9, that's already unsupported to the best of my knowledge.

It look like they begun patching Calibre for running under Python3 two years ago.
Currently, Fedora have very little patch.
Anyway, they are going to upgrade to Calibre v5 very-soon(r)(tm)

Originally Posted by JimmXinu

And yes, I do care more about keeping FFF running on Calibre 2 more than on everybody's Linux distro versions--it (reportedly) still has more users.

That's why I was documenting and not sending a PR

[JimmXinu]

It's come up before, so I think there's a known solution if it isn't already incorporated in Calibre 5.

[DavidBrookes]

I am trying to update my FanFiction.Net library in Calibre using the FanFicFare plugin but I am being blocked updating via using URLS due to CloudFlare. I tried doing this via emails but are getting the following:

calibre, version 5.20.0
ERROR: Unhandled exception: <b>error</b>:command: UID =&gt; got more than 1000000 bytes

calibre 5.20 [64bit] embedded-python: True is64bit: True
Windows-10-10.0.18362 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 3.8.5
Windows: ('10', '10.0.18362', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: FanFicFare (4, 3, 0) && Wiki Reader (2, 2, 1)
Traceback (most recent call last):
imaplib.IMAP4.error: got more than 1000000 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
imaplib.IMAP4.error: command: UID => got more than 1000000 bytes

Since I do this once a month, (first weekend of the month), please help.

[thenico]

imaplib is saying that an email in your mailbox have an Unique ID that is over 1 megabyte in size.
That very surprising since the spec say that it should be a 32 bit value.

What is your email provider?

[DavidBrookes]

Originally Posted by thenico

imaplib is saying that an email in your mailbox have an Unique ID that is over 1 megabyte in size.
That very surprising since the spec say that it should be a 32 bit value.

What is your email provider?

iiNet here in Australia. I have checked the mailbox folders and there are no emails in any folders that size.

[JimmXinu]

Originally Posted by DavidBrookes

I am trying to update my FanFiction.Net library in Calibre using the FanFicFare plugin but I am being blocked updating via using URLS due to CloudFlare. I tried doing this via emails but are getting the following:

If you read the last few pages of this thread, ffnet appears to be blocking more vigorously again. You might have to use a semi-automated process as outlined here. Using Email isn't going to make any difference to that.

(Or skip it and try again next month. They have increased and then decreased blocking before.)

Originally Posted by DavidBrookes

Traceback (most recent call last):
imaplib.IMAP4.error: command: UID => got more than 1000000 bytes

No idea what this is about. Do you have FFF checking a folder with a lot of old emails in it? How many stories are you trying to get updates for?

If you run Calibre in debug mode and try it again, the log may give us more information.

[jklly12]

I'm kind of confused on how to set up the browser cache option? any help in how to do that? or a step by step sort of tutorial i used the one provided but its not working

[leckadams]

I'm not sure what happened. I was trying to update AO3 fics from my email, like normal and got this message and it won't do it.

calibre, version 5.20.0
ERROR: Unhandled exception: <b>gaierror</b>:[Errno 11001] getaddrinfo failed

calibre 5.20 embedded-python: True is64bit: False
Windows-10-10.0.19041 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 11, 0) && EpubMerge (2, 11, 0) && FanFicFare (4, 3, 2) && Find Duplicates (1, 8, 5)
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 572, in get_urls_from_imap_menu
File "C:\Users\lecka\AppData\Roaming\calibre\plugins\Fa nFicFare.zip\fanficfare\geturls.py", line 187, in get_urls_from_imap
File "imaplib.py", line 1297, in __init__
File "imaplib.py", line 198, in __init__
File "imaplib.py", line 1310, in open
File "imaplib.py", line 303, in open
File "imaplib.py", line 1300, in _create_socket
File "imaplib.py", line 293, in _create_socket
File "socket.py", line 787, in create_connection
File "socket.py", line 918, in getaddrinfo
socket.gaierror: [Errno 11001] getaddrinfo failed


could someone direct me on how to fix this? TIA

[JimmXinu]

Originally Posted by leckadams

I'm not sure what happened. I was trying to update AO3 fics from my email, like normal and got this message and it won't do it.

calibre, version 5.20.0
ERROR: Unhandled exception: <b>gaierror</b>:[Errno 11001] getaddrinfo failed

That error (usually) indicates a failure to get an IP address for a DNS name. Most common cause, using the wrong address name.

If it worked before and you haven't changed anything, it might be a transient lookup failure/timeout and it might if you try again later.

[leckadams]

Originally Posted by JimmXinu

That error (usually) indicates a failure to get an IP address for a DNS name. Most common cause, using the wrong address name.

If it worked before and you haven't changed anything, it might be a transient lookup failure/timeout and it might if you try again later.

Thank you so much. I appreciate it

[linuxstony]

Hi,

I am sorry to bother you, but i try to setup the browser-cache feature for my wife because of a cloudflare 2 error, but apparantly my times with linux are so way back, that i am too stupid to setup the browser-cache-path correctly.

I have the following set in the personal.ini

Spoiler:

## This is an example of what your personal configuration might look
## like. Uncomment options by removing the '#' in front of them.

[defaults]
## [defaults] section applies to all formats and sites but may be
## overridden at several levels. Example:
#browser_cache_path:'/home/kathrin/.cache/mozilla/firefox/cxmz1llt.default/cache2'
#browser_cache_path:/home/kathrin/.cache/mozilla/firefox/cxmz1llt.default/cache2
browser_cache_path:/home/kathrin/.cache/mozilla/firefox/cxmz1llt.default/cache2

[www.fanfiction.net]
use_browser_cache:true
browser_cache_path:/home/kathrin/.cache/mozilla/firefox/cxmz1llt.default/cache2
## browser cache will only be used if use_browser_cache:true and ONLY
## for fanfiction.net right now. Requires a browser_cache_path set in
##[defaults]
## [defaults]
## titlepage_entries: category,genre, status
## [www.whofic.com]
## # overrides defaults.
## titlepage_entries: category,genre, status,dateUpdated,rating
## [epub]
## # overrides defaults & site section
## titlepage_entries: category,genre, status,datePublished,dateUpdated,dateCreated
## [www.whofic.com:epub]
## # overrides defaults, site section & format section
## titlepage_entries: category,genre, status,datePublished
## [overrides]
## # overrides all other sections
## titlepage_entries: category

## Some sites also require the user to confirm they are adult for
## adult content. Uncomment by removing '#' in front of is_adult.
#is_adult:true

## Don't like the numbers at the start of chapter titles on some
## sites? You can use strip_chapter_numbers to strip them off. Just
## want to make them all look the same? Strip them off, then add them
## back on with add_chapter_numbers. Don't like the way it strips
## numbers or adds them back? See chapter_title_strip_pattern and
## chapter_title_add_pattern.
#strip_chapter_numbers:true
#add_chapter_numbers:true

## Add this to genre if there's more than one category.
#add_genre_when_multi_category: Crossover

[epub]
## include images from img tags in the body and summary of stories.
## Images will be converted to jpg for size if possible. Images work
## in epub format only. To get mobi or other format with images,
## download as epub and use Calibre to convert.
#include_images:true

## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
#keep_summary_html:true

## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
#make_firstimage_cover:true

## Resize images down to width, height, preserving aspect ratio.
## Nook size, with margin.
#image_max_size: 580, 725

## Change image to grayscale, if graphics library allows, to save
## space.
#grayscale_images: false


## Most common, I expect will be using this to save username/passwords
## for different sites. Here are a few examples. See defaults.ini
## for the full list.

[www.twilighted.net]
#username:YourPenname
#password:YourPassword
## default is false
#collect_series: true

[ficwad.com]
#username:YourUsername
#password:YourPassword

[www.adastrafanfic.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content.
#is_adult:true

[www.twcslibrary.net]
#username:YourName
#password:yourpassword
#is_adult:true
## default is false
#collect_series: true

[www.fictionalley.org]
#is_adult:true

[www.harrypotterfanfiction.com]
#is_adult:true

[www.fimfiction.net]
#is_adult:true
#fail_on_password: false

[www.tthfanfic.org]
#is_adult:true
## tth is a little unusual--it doesn't require user/pass, but the site
## keeps track of which chapters you've read and won't send another
## update until it thinks you're up to date. This way, on download,
## it thinks you're up to date.
#username:YourName
#password:yourpassword


## This section will override anything in the system defaults or other
## sections here.
[overrides]
## default varies by site. Set true here to force all sites to
## collect series.
#collect_series: true

I wasn't sure where exactly to put the path ....

But i still get the error message, that /home/kathrin is no valid browser-cache-directory.

Spoiler:

Status
Titel
Autor
Kommentar
URL
Fehler
Unknown
Unknown
Directory does not contain a known browser cache type: '/home/kathrin'
https://www.fanfiction.net/s/1380727...t-of-the-Minds

So i assume that the path isn't set correctly as it searches in /home/kathrin and not the mentioned cache path.

I am very sorry if this problem is basic - usually i am pretty good with the error solving when calibre is acting up - but i am at a loss atm.

I appreciate the work you put into this plugin, this helped my wife tremendously building a huge database of fanfiction she likes.

Thanks for any advice that you can give.

Stefan