[GUI Plugin] FanFicFare

[dakota1979]

I don't actually know which link is the problem, is not listed in the error message and I refresh a lot of links in one go... I'll check again if there will be a new error message

[Tanjamuse]

I thought you said it was every link?

[JimmXinu]

Originally Posted by rishidiams

I'm currently importing fics into Calibre by "get story URLs from web page". Once it gathers the URLs if the story already exists, I want it to Update Calibre Metadata from Web Site. I've noticed that if I choose this and a book appears within an anthology story I already have, it will give me the skip option, but otherwise I will end up with duplicates entries in my database, same author name, same title, same URL.

There is a bug with the way FFF handles 'Update Calibre Metadata from Web Site' when given a URL list that contains URL(s) that aren't already in the library. Rather than changing to 'Add New Book' for only that book, it changes for the whole list.

I'm need to think about how to address that correctly. It may require significant changes to the internal architecture.

In the meantime, your best option is probably to use 'Update EPUB Always' instead. That will update metadata (both in Calibre and inside epubs), and add stories that don't already exist. It will, however, re-download existing ffnet epubs from AO3 if you choose to let it change URL.

Originally Posted by rishidiams

If I choose Update EPUB if New Chapters, it will skip stories with the same author name, same title, and same URL, but even though Update Calibre metadata is checked, the metadata for existing books is not updated.

The 'Update Calibre Metadata?' checkbox only turns off Calibre metadata update when an ebook is updated. It does not turn on Calibre metadata update when the ebook is not updated. (It's a bit of legacy from the early days when some users didn't trust FFF to update their metadata.)

Originally Posted by rishidiams

Also, some of the stories in my database are from Fanfiction.net and I would prefer to have AO3's metadata, so if given the option to update the URL to AO3's I always choose yes, but the stories are not being updated with the new URL either.

I cannot duplicate this in isolation.

If it's happening at the same time as when doing 'Update Calibre Metadata from Web Site' with a URL list containing new story URLs, then it's part of that issue.

[JimmXinu]

New Test Version Posted

2018-10-15
- Fix for Calibre Metadata update called with URLs not in library.

This should address rishidiams' issues.

[dakota1979]

no, not all of them, some of the m..

calibre 3.32 [64bit] embedded-python: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.12+
Windows: ('7', '6.1.7601', 'SP1', u'Multiprocessor Free')
Interface language: hu
Successfully initialized third party plugins: Find Duplicates (1, 6, 3) && Reading List (1, 6, 6) && FanFicFare (3, 0, 0)
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.dialogs", line 688, in do_loop
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf3 in position 38: invalid continuation byte

here is an example

<urlopen error EOF occurred in violation of protocol (_ssl.c:590)>
https://www.fanfiction.net/s/11983272/1/
'' https://www.fanfiction.net/s/5302747/1/
<urlopen error EOF occurred in violation of protocol (_ssl.c:590)> https://www.fanfiction.net/s/10498808/1/
<urlopen error EOF occurred in violation of protocol (_ssl.c:590)> https://www.fanfiction.net/s/9466720/1/
'' https://www.fanfiction.net/s/7337176/1/
<urlopen error EOF occurred in violation of protocol (_ssl.c:590)> https://www.fanfiction.net/s/3852537/1/
<urlopen error EOF occurred in violation of protocol (_ssl.c:590)> https://www.fanfiction.net/s/12813751/1/
'' https://www.fanfiction.net/s/11849217/1/

[JimmXinu]

@dakota1979

Those don't look like the same error, but could be related, I suppose.

The most common SSL error we see has to do with certs being out of date on Windows. Usually that can be fixed by visiting the site using the OS browser, MSIE or Edge.

There was a known OpenSSL bug for a while that may be related. There's (mostly) undocumented feature you can use to work around that bug, it may help you, too:

[www.fanfiction.net]
use_ssl_unverified_context:true

[firefoxxy]

Originally Posted by JimmXinu

@dakota1979

Those don't look like the same error, but could be related, I suppose.

The most common SSL error we see has to do with certs being out of date on Windows. Usually that can be fixed by visiting the site using the OS browser, MSIE or Edge.

There was a known OpenSSL bug for a while that may be related. There's (mostly) undocumented feature you can use to work around that bug, it may help you, too:

[www.fanfiction.net]
use_ssl_unverified_context:true

Hi Jimm,
I have a similar problem. Unfortunately, the work around you suggested did not help.
I tried downloading six stories from ff.net. Three were okay, three had an error. The error message is still the same, even after adding the code you suggested:

Spoiler:

Fehler
Time Immemorial
FawkesyLady
<urlopen error [Errno 10054] Eine vorhandene Verbindung wurde vom Remotehost geschlossen>
https://www.fanfiction.net/s/12495244/1/
(the error translates to: an existing connection was closed by the remote host)
Fehler
A Failure to Communicate
Otter's Corner
''
(the error description is just the two '')
https://www.fanfiction.net/s/12648418/1/
Fehler
They Didn't Know We Were Seeds
LucyLuna
<urlopen error EOF occurred in violation of protocol (_ssl.c:590)>
https://www.fanfiction.net/s/12386916/1/

Edit: I just saw that one of the stories I did download successfully today returned an error yesterday. I'm not sure if that helps.

[JimmXinu]

Originally Posted by firefoxxy

Hi Jimm,
I have a similar problem. Unfortunately, the work around you suggested did not help.
I tried downloading six stories from ff.net. Three were okay, three had an error. The error message is still the same, even after adding the code you suggested:

<urlopen error [Errno 10054] Eine vorhandene Verbindung wurde vom Remotehost geschlossen>
https://www.fanfiction.net/s/12495244/1/
(the error translates to: an existing connection was closed by the remote host)

Edit: I just saw that one of the stories I did download successfully today returned an error yesterday. I'm not sure if that helps.

Those errors are much more likely to be the site server experiencing congestion, or possibly even blocking you if you've been downloading a lot. See FAQ.

[firefoxxy]

Hm, it's probably not blocking me. Today I tried downloading those six stories and before that only three, yesterday it was four and then three again.
So, you say it's just a matter of patience? I can live with that. Thank you!

[JimmXinu]

New Version Released

Version 3.1.0 - 17 Oct 2018
- Fix for Calibre Metadata update called with URLs not in library.
- Fix adapter_wuxiaworldco date format for 24 hour clock. Closes #345
- Change adapter_lightnovelgatecom to adapter_novelonlinefullcom for site change. Closes #346
- Add max_zalgo feature.
- Handle stories without cover correctly adapter_wwwlushstoriescom. Closes #344
- Add cover images for adapter_wwwlushstoriescom
- Rename adapter_royalroadcom appropriately.
- Fix for royalroad.com warnings.
- Put SV(base_xenforo) prefix spans from title into 'genre', comment out some debugs.
- Fix for issues with single-chapter stories not getting correct chapter title after chapter meta refactor Jul 2018.
- Add fix calibre title/author cases options, move force title/author sort values to Standard Columns config Tab.
- Add 'date' chapter metadata to AO3. Closes #336
- Move mobi TOC back to after title page. Requested by jxxtan.

[ILB]

Hi, I updated to Version 3.1.0 and now I get an error message for https://www.fanfiction.net/s/1259739...ty-Eighth-Life

Spoiler:

calibre, version 3.32.0
ERROR: Unhandled exception: <b>KeyError</b>:u'collision'

calibre 3.32 [64bit] embedded-python: True is64bit: True
Windows-10-10.0.17134-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.17134')
Python 2.7.12+
Windows: ('10', '10.0.17134', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 8, 2) && Kobo Books (1, 3, 0) && EpubMerge (2, 4, 0) && Copy Cover To Device (2, 0, 0) && KoboTouchExtended (3, 0, 2) && Manage Series (1, 2, 8) && Similar Stories (1, 0, 58) && Find Duplicates (1, 6, 3) && Generate Cover (1, 5, 21) && Reading List (1, 6, 6) && FanFicFare (3, 1, 0) && Quick Preferences (1, 4, 1) && Read MP3 AudioBook metadata (1, 0, 79) && SmartEject (2, 1, 0) && Kobo Utilities (2, 6, 0) && Annotations (1, 8, 0) && Kindle Collections (1, 7, 21)
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 996, in prep_downloads
File "calibre_plugins.fanficfare_plugin.dialogs", line 615, in LoopProgressDialog
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 1486, in start_download_job
KeyError: u'collision'

The story is on my reject list but another story on this list goes without this error message.

[JimmXinu]

New Test Version Posted

2018-10-18
- Fix for problem with new collision code and rejected URL.

I will probably do a point release for this fix before too long, but let's see if anyone finds more problems first.

Originally Posted by ILB

Hi, I updated to Version 3.1.0 and now I get an error message for https://www.fanfiction.net/s/1259739...ty-Eighth-Life

Spoiler:

calibre, version 3.32.0
ERROR: Unhandled exception: <b>KeyError</b>:u'collision'

calibre 3.32 [64bit] embedded-python: True is64bit: True
Windows-10-10.0.17134-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.17134')
Python 2.7.12+
Windows: ('10', '10.0.17134', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 8, 2) && Kobo Books (1, 3, 0) && EpubMerge (2, 4, 0) && Copy Cover To Device (2, 0, 0) && KoboTouchExtended (3, 0, 2) && Manage Series (1, 2, 8) && Similar Stories (1, 0, 58) && Find Duplicates (1, 6, 3) && Generate Cover (1, 5, 21) && Reading List (1, 6, 6) && FanFicFare (3, 1, 0) && Quick Preferences (1, 4, 1) && Read MP3 AudioBook metadata (1, 0, 79) && SmartEject (2, 1, 0) && Kobo Utilities (2, 6, 0) && Annotations (1, 8, 0) && Kindle Collections (1, 7, 21)
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 996, in prep_downloads
File "calibre_plugins.fanficfare_plugin.dialogs", line 615, in LoopProgressDialog
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 1486, in start_download_job
KeyError: u'collision'

The story is on my reject list but another story on this list goes without this error message.

[ILB]

Originally Posted by JimmXinu

New Test Version Posted

2018-10-18
- Fix for problem with new collision code and rejected URL.

Thanks. Now I get the normal "Story on Reject List" message.

[JimmXinu]

New Version Released

Plugin version only.

Version 3.1.1 - 19 Oct 2018
- Fix for problem with new collision code and rejected URL.

[ilovejedd]

Originally Posted by JimmXinu

I did look at this, both applying one of the existing rule sets and re-using the infrastructure (aka tag_mapper) to create additional rule sets.

And finally, only Tanjamuse and (maybe?) ilovejedd showed any interest.

So I'm shelving this proposed change for the time being. A more general tool, like CALM may be useful.

Update: The Job Spy plugin recently added a tag scrubbing utility.

I'm quite happily using a combination of Excel/LibreOffice Calc (to maintain the tag wrangling rules table) + Job Spy to handle AO3's freeform tags.

Current workflow is:
#freeform -> #tmptags (run scrub against this cc) -> #{relationships|characters|tropes}