[GUI Plugin] FanFictionDownLoader - Page 250

JimmXinu · 02-16-2015, 01:07 PM

Quote:

Originally Posted by BookJunkieLI

I'm trying to setup a custom column to download AO3's kudos into and I keep getting the following error message:
...
Any idea what I'm doing wrong?

I just tested that story with a float column named #float with display format {0:,.2f}.

In personal.ini, I put:

Code:

[archiveofourown.org]
add_to_custom_columns_settings:
 kudos=>#float

...and it worked fine. So, no, I don't have any idea. I don't even know where you got that error from.

BookJunkieLI · 02-16-2015, 01:40 PM

Er, yeah, so I may not have added the

Code:

add_to_custom_columns_settings:

part to the settings.

It was picking up the 15.0 from the setting listed right above it.

mehetabelo · 02-16-2015, 03:59 PM

That latest update seems to have worked through that error, I got all the old ones ... or most of them.

Something I've noticed when I did a mass-download... about 5-10% of them don't make it onto my list. When I started working with this, I decided I wanted to get a list of every single update I had, filter it down to stories only, and then filter it further (comparing current stuff I have to stuff on the list to see what I need to get that's old).

So, instead of downloading roughly 16k I broke it into 1k chunks marked the messages in those 1k as unread, and then threw the plugin at it. I copied what it popped up into excel, marked the next 1k as unread... and just looped. What I noticed is that there was usually 920-950 url's that came out of 1k every time. This last time I had 2280 messages, so I just did them all, but I came up with ~2100 URL's.

I don't care all that much, as it did (primarily) what I wanted it to do and with a little replacement magic in word, and sort/filter magic in excel... well I got a list. I did want to make you aware of it, and am curious if you know a reason for it.

JimmXinu · 02-16-2015, 06:27 PM

Quote:

Originally Posted by mehetabelo

...
What I noticed is that there was usually 920-950 url's that came out of 1k every time. This last time I had 2280 messages, so I just did them all, but I came up with ~2100 URL's.
...

If you can give me specific emails that have problems, I'll look into it.

But I'm more likely to put a in 100 email-at-a-time cap than I am to worry about what happens with you try to give it thousands. This feature is intended for getting new updates, not data mining.

mehetabelo · 02-16-2015, 10:18 PM

Quote:

Originally Posted by JimmXinu

If you can give me specific emails that have problems, I'll look into it.

But I'm more likely to put a in 100 email-at-a-time cap than I am to worry about what happens with you try to give it thousands. This feature is intended for getting new updates, not data mining.

When I get back home (maybe this weekend) I'll see if I can work on identifying it. I didn't tell you because I needed it fixed, I only mentioned it because I thought you should know. It might be a problem with the number that's being downloaded, and it might be something else. *shrug* I don't know until I try it.

I'll probably do something like 50 at a time and see if it misses any when it does that. Then move up to 100, just to experiment. Maybe run the same set a couple times, to see if it gets any different results. I know it wasn't intended for data-mining, but it was useful instead of spending several hours copying them by hand. I have almost all the stories anyway, I was looking for ones I hadn't downloaded in the past, or identify ones that I had read and had disappeared from ff.net (over 150 from my lists) to see if I could figure out if I could find said story elsewhere.

It really irritates me to lose something I read and may want to read again.... I am still trying to find some old fanfics by Master Eldryn that I had started right at the time he deleted them. I *really* wish I had deffnet'd them (I'm pretty sure it was available then) or copied them into a word file as I'd done a lot of fanfics pre deffnet. I had one or two that I found later when they made their way around the internet, but never the ones I wanted.

Ah well, someday. Anyway, thanks, I'll try and remember to do those checks later.

Tanjamuse · 02-18-2015, 11:16 AM

I'm unsure if this is the right place to post it, but I'm having trouble understanding the searching terms in "Search within results" under a specific fandom.

If anyone can help me, please send me a PM.

cryzed · 02-20-2015, 08:54 AM

It really is unrelated to FFDL, but it most likely means further narrowing down search results from a previous search. Imagine searching for the "Buffy" character in the set of all stories -- the result is a subset of all stories. Now you can narrow down the result even further by searching for "Spike". This should be equivalent to searching for both: "Buffy" and "Spike" within a single search, but this multi-pass search allows you more control and flexibility.

EDIT: I also just noticed that you asked to be sent a PM, sorry about that. Anyways, in case someone has a similar question they might now find this thread (and the answer) via Google.

tricklem · 02-20-2015, 08:38 PM

Below is error that I get when trying to download from fictionpad. Anyone know what this is about or how to fix it?

Spoiler:

PeterT · 02-20-2015, 08:42 PM

You don't have the most current Microsoft Root certificate update installed. Please see http://support.microsoft.com/kb/931125 and install the Microsoft Update listed for YOUR O/S.

tricklem · 02-21-2015, 08:08 AM

Quote:

Originally Posted by tricklem

Below is error that I get when trying to download from fictionpad. Anyone know what this is about or how to fix it?

Spoiler:

pic Review (Newest First)
Today 01:42 AM
PeterT You don't have the most current Microsoft Root certificate update installed. Please see http://support.microsoft.com/kb/931125 and install the Microsoft Update listed for YOUR O/S.

So I went to and did this and I am back to getting this error:

Spoiler:

JimmXinu · 02-21-2015, 11:04 AM

Quote:

Originally Posted by tricklem

pic Review (Newest First)
Today 01:42 AM
PeterT You don't have the most current Microsoft Root certificate update installed. Please see http://support.microsoft.com/kb/931125 and install the Microsoft Update listed for YOUR O/S.

So I went to and did this and I am back to getting this error:

Spoiler:

Hmmm. I'm seeing this error for the first time now. (We lumped you in the same category as another recent user who had a similar problem without researching your problem very deeply. Sorry about that.)

It happens actually on s3.amazonaws.com urls. I can see it when I try to fetch that URL with a simple urllib2 call in Python 2.7.9 CLI, too.

It looks like this is a bug in OpenSSL's cert chain checking--or possibly just Python 2.7.9's use of it. Either way, it's up stream of both FFDL and calibre.

There is a 'monkeypatch' that can be used to bypass the issue by disabling the cert chain check. For minimal impact, I've added it as a personal.ini option that will only effect the background jobs.

With the attached version try this in your personal.ini:

Code:

[fictionpad.com]
use_ssl_unverified_context:true

IMPORTANT NOTE -- The first version of calibre 2.20.0 released had a bug that prevented some plugins (including FFDL) from saving changes to their configuration to the db.

Kovid made a fixed update and put it online but did not change the version number. So I'm not sure how you're supposed to know if you need the update or not--but I needed it.

I strongly suggest that if you updated to calibre 2.20 before the time of this posting you should download and install it again.

JimmXinu · 02-21-2015, 11:53 AM

Attached is (yet another) test version with potionsandsnitches.net -> potionsandsnitches.org changes. Plus all previous test changes, as usual.

tricklem · 02-21-2015, 04:29 PM

Quote:

Originally Posted by JimmXinu

Hmmm. I'm seeing this error for the first time now. (We lumped you in the same category as another recent user who had a similar problem without researching your problem very deeply. Sorry about that.)

It happens actually on s3.amazonaws.com urls. I can see it when I try to fetch that URL with a simple urllib2 call in Python 2.7.9 CLI, too.

It looks like this is a bug in OpenSSL's cert chain checking--or possibly just Python 2.7.9's use of it. Either way, it's up stream of both FFDL and calibre.

There is a 'monkeypatch' that can be used to bypass the issue by disabling the cert chain check. For minimal impact, I've added it as a personal.ini option that will only effect the background jobs.

With the attached version try this in your personal.ini:

Code:

[fictionpad.com]
use_ssl_unverified_context:true

IMPORTANT NOTE -- The first version of calibre 2.20.0 released had a bug that prevented some plugins (including FFDL) from saving changes to their configuration to the db.

Kovid made a fixed update and put it online but did not change the version number. So I'm not sure how you're supposed to know if you need the update or not--but I needed it.

I strongly suggest that if you updated to calibre 2.20 before the time of this posting you should download and install it again.

This worked!!

jklly12 · 02-22-2015, 11:31 PM

anyone getting Error Updating metadata from TWCS its annoying.

cryzed · 02-23-2015, 08:01 AM

Please provide us with the URL that causes an error.

02-16-2015, 01:40 PM	#3737
BookJunkieLI Evangelist Posts: 435 Karma: 572984 Join Date: Jan 2010 Location: Long Island Device: Kobo Libra 2, Kindle 4, Nook Gl4, Nook STR, REB 1100, Ebookwise 1500,	Er, yeah, so I may not have added the Code: add_to_custom_columns_settings: part to the settings. It was picking up the 15.0 from the setting listed right above it.

02-20-2015, 08:38 PM	#3743
tricklem Connoisseur Posts: 60 Karma: 10 Join Date: Feb 2015 Device: none	Below is error that I get when trying to download from fictionpad. Anyone know what this is about or how to fix it? Spoiler: calibre, version 2.20.0 (win32, isfrozen: True) Failed to Download Stories: Failed: Download FanFiction Book Download FanFiction Book FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(36):FanFictionDownLoader v2.1.3 FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(40):Adding jobs for URLs: FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(42):https://fictionpad.com/author/tufano79/stories/823 FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(78):Logfile for book ID 659 (La Cantante) FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(79):urlhttps://fictionpad.com/author/tufano79/stories/823) id659) FFDLEBUG:calibre_plugins.fanfictiondownloader_pl ugin.jobs(131):cookiejar:<LWPCookieJar[<Cookie _fictionpad_session=b0IrUUxrLzg0akZmRDlzeGwrcGZmSm htZ2VyR0YzTW0rcW5ZM1IwUEx0SmhBQ1BZT1Z2NHJMcUNCVlpP djJOMXhndDlsZ2tsV1owcXVhaE51cTVEeTBRL3RPQ1Bjc3ViMn M3QUZ2SEZXV1ROUDdadjdKZlRlc25JenM5d0NGK1pTeExqaHI4 THE1K2dKeFBWckhFbWsvRGtESi9zeXpxRTBuR083L1dndzlET0 F1MzV0ZnhTcldBV2NqTkxVRGk5LS1UanpYak4xL3VFbEtlR0NI V0hqaE13PT0%3D--3b275d680c3ee5606567c956b83b21a48cee6b48 for fictionpad.com/>, <Cookie guid=1d611a72-a272-47c5-975c-61e98887f1f2 for fictionpad.com/>, <Cookie remember_token=BAhJIhtaM25EZS1DZ0RqS2lGeTFPUFpSRG1 BBjoGRVQ%3D--fbd01ddca791556084a9c50f2de5cc2a9a82a019 for fictionpad.com/>]> FFDLEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.adapter_fictionpadc om(112):URL: https://fictionpad.com/author/tufano79/stories/823 FFDLEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(292):# #################################### pagecache MISS: https://fictionpad.com/author/tufano79/stories/823 story:{u'published_version_id': 42632, u'collection_ids': [], u'type': u'Story', u'note_ids': [15741, 15613, 15592, 15571, 15330, 15080, 14965, 14916, 14821, 14716, 14541, 14335, 14277, 14225, 14161, 14068, 13794, 12194, 10568, 8181, 7238, 6779, 6679, 6218, 5635, 4150, 3264, 2434, 2138, 1730, 218], u'is_liked_by_current_user': False, u'followed_type': u'1', u'story_collections_count': 59, u'first_published_at': u'2014-01-07T01:30:10Z', u'id': 823, u'is_authored_by_current_user': False, u'current_user_furthest_reading_position_chapter_n umber': None, u'views_count': 1041, u'published_at': u'2015-02-18T22:20:39Z', u'draft_version_id': None, u'comments_count': 274, u'user_id': 202, u'current_user_furthest_reading_position_index': None, u'followers_count': 346, u'is_private_following': False, u'likes': 48.0, u'is_followed_by_current_user': True, u'is_public_following': True} FFDLEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(292):# #################################### pagecache MISS: https://s3.amazonaws.com/attachments...1379183725.jpg FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.fanficdownloader.story(815):Failed to load or convert image, skipping: https://s3.amazonaws.com/attachments...1379183725.jpg Exception: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)> FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(204)o update - epub(65) vs url(69) FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(205):write to C:\Users\Miller\AppData\Local\Temp\calibre_pvox2t\ fanfictiondownloader_32jv0m\La Cantante-tufano79-ovswzp.epub FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.fanficdownloader.writers.base_writer(208):Save directly to file: C:\Users\Miller\AppData\Local\Temp\calibre_pvox2t\ fanfictiondownloader_32jv0m\La Cantante-tufano79-ovswzp.epub FFDL:ERROR:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(540):E rror parsing HTML, probably poor input HTML. 'NoneType' object has no attribute 'next_element' FFDL:ERROR:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(540):E rror parsing HTML, probably poor input HTML. 'NoneType' object has no attribute 'next_element' FFDLEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.adapter_fictionpadc om(184):Getting chapter text from: https://s3.amazonaws.com/attachments...1423886472.htm FFDLEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(292):# #################################### pagecache MISS: https://s3.amazonaws.com/attachments...1423886472.htm FFDL:WARNING:calibre_plugins.fanfictiondownloader_ plugin.fanficdownloader.adapters.base_adapter(334) :Caught an exception reading URL: https://s3.amazonaws.com/attachments...1423886472.htm sleeptime(0) Exception <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>. FFDLEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(292):# #################################### pagecache MISS: https://s3.amazonaws.com/attachments...1423886472.htm FFDL:WARNING:calibre_plugins.fanfictiondownloader_ plugin.fanficdownloader.adapters.base_adapter(334) :Caught an exception reading URL: https://s3.amazonaws.com/attachments...1423886472.htm sleeptime(0.5) Exception <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>. Python function terminated unexpectedly 'NoneType' object has no attribute '__getitem__' (Error Code: 1) Traceback (most recent call last): File "site.py", line 132, in main File "site.py", line 109, in run_entry_point File "site-packages\calibre\utils\ipc\worker.py", line 193, in main File "site-packages\calibre\utils\ipc\worker.py", line 133, in arbitrary_n File "calibre_plugins.fanfictiondownloader_plugin.jobs" , line 83, in do_download_worker File "calibre_plugins.fanfictiondownloader_plugin.jobs" , line 83, in <lambda> TypeError: 'NoneType' object has no attribute '__getitem__' Last edited by theducks; 02-20-2015 at 09:54 PM. Reason: Wrapped long paste in Spoiler

02-21-2015, 11:53 AM	#3747
JimmXinu Plugin Developer Posts: 7,125 Karma: 5005503 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	Attached is (yet another) test version with potionsandsnitches.net -> potionsandsnitches.org changes. Plus all previous test changes, as usual. Last edited by JimmXinu; 02-25-2015 at 12:05 PM. Reason: Remove obsolete test versions - replaced by newer test or released version.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Count Pages	kiwidude	Plugins	1874	10-17-2025 03:22 PM
[GUI Plugin] Resize Cover	kiwidude	Plugins	100	07-10-2025 08:50 AM
[GUI Plugin] Find Duplicates	kiwidude	Plugins	1124	04-18-2025 09:19 AM
[GUI Plugin] Open With	kiwidude	Plugins	404	02-21-2025 05:42 AM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

02-16-2015, 03:59 PM	#3738
mehetabelo e-Bibliophile Posts: 60 Karma: 10 Join Date: Jun 2009 Location: California Device: Paperwhite 1-3, Kobo AuraHD, Boox Afterglow2	That latest update seems to have worked through that error, I got all the old ones ... or most of them. Something I've noticed when I did a mass-download... about 5-10% of them don't make it onto my list. When I started working with this, I decided I wanted to get a list of every single update I had, filter it down to stories only, and then filter it further (comparing current stuff I have to stuff on the list to see what I need to get that's old). So, instead of downloading roughly 16k I broke it into 1k chunks marked the messages in those 1k as unread, and then threw the plugin at it. I copied what it popped up into excel, marked the next 1k as unread... and just looped. What I noticed is that there was usually 920-950 url's that came out of 1k every time. This last time I had 2280 messages, so I just did them all, but I came up with ~2100 URL's. I don't care all that much, as it did (primarily) what I wanted it to do and with a little replacement magic in word, and sort/filter magic in excel... well I got a list. I did want to make you aware of it, and am curious if you know a reason for it.

02-18-2015, 11:16 AM	#3741
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	I'm unsure if this is the right place to post it, but I'm having trouble understanding the searching terms in "Search within results" under a specific fandom. If anyone can help me, please send me a PM.

02-20-2015, 08:54 AM	#3742
cryzed Evangelist Posts: 408 Karma: 1050547 Join Date: Mar 2011 Device: Kindle Oasis 2	It really is unrelated to FFDL, but it most likely means further narrowing down search results from a previous search. Imagine searching for the "Buffy" character in the set of all stories -- the result is a subset of all stories. Now you can narrow down the result even further by searching for "Spike". This should be equivalent to searching for both: "Buffy" and "Spike" within a single search, but this multi-pass search allows you more control and flexibility. EDIT: I also just noticed that you asked to be sent a PM, sorry about that. Anyways, in case someone has a similar question they might now find this thread (and the answer) via Google.

02-20-2015, 08:42 PM	#3744
PeterT Grand Sorcerer Posts: 13,693 Karma: 79983758 Join Date: Nov 2007 Location: Toronto Device: Libra H2O, Libra Colour	You don't have the most current Microsoft Root certificate update installed. Please see http://support.microsoft.com/kb/931125 and install the Microsoft Update listed for YOUR O/S.

02-22-2015, 11:31 PM	#3749
jklly12 Member Posts: 21 Karma: 10 Join Date: Feb 2015 Device: Kindle	anyone getting Error Updating metadata from TWCS its annoying.

02-23-2015, 08:01 AM	#3750
cryzed Evangelist Posts: 408 Karma: 1050547 Join Date: Mar 2011 Device: Kindle Oasis 2	Please provide us with the URL that causes an error.