View Full Version : [GUI Plugin] FanFictionDownLoader


Pages : 1 2 3 4 5 6 7 8 9 [10] 11 12 13 14 15

Firedancer885
12-11-2013, 02:07 PM
I downloaded the beta again, installed it, restarted Calibre and still got the notice. The version you posted registers as 1.8.1.

JimmXinu
12-11-2013, 02:10 PM
No. I got a notice because the version I installed registered as 1.8.1. I had 1.8.2 installed previously. Please check.
name = 'FanFictionDownLoader'
description = _('UI plugin to download FanFiction stories from various sites.')
supported_platforms = ['windows', 'osx', 'linux']
author = 'Jim Miller'
version = (1, 8, 03)
minimum_calibre_version = (1, 13, 0)

Are you sure you're installing the same file you downloaded? I've done that before, download to one dir, but be looking somewhere else for it.

Firedancer885
12-11-2013, 02:14 PM
Are you sure you're installing the same file you downloaded? I've done that before, download to one dir, but be looking somewhere else for it.

:smack: Thanks :)

Depletion
12-11-2013, 11:14 PM
Got it. The problem is the extra_valid_entries line in your personal.ini. It's overriding the one in defaults.ini and removing groupsURL. Remove that line and it will work.

Removed it and it works. :) And went ahead and removed the extra as you suggested. Thanks so much!

Firedancer885
12-12-2013, 09:15 AM
Jimm,

I managed to install the proper plugin yesterday, but it still doesn't work. I put the update hook line into my personal ini. Still got 2 appendix chapters.

JimmXinu
12-12-2013, 01:26 PM
I managed to install the proper plugin yesterday, but it still doesn't work. I put the update hook line into my personal ini. Still got 2 appendix chapters.

http://archiveofourown.org/works/972937/navigate

The chapter list says that that is the most recently updated chapter:
"238. Appendix (2013-12-12)"

I assume you did "Overwrite Always" yesterday, and there aren't any chapters marked 2013-12-11. That suggests that the chapter dates aren't accurate.

And if that's true, there's nothing further I can do to detect new out of order chapters. You'll just have to manually watch for out of order updates and overwrite when they happen.

JimmXinu
12-12-2013, 03:58 PM
Version 1.8.04 - 12 Dec 2013
(v1.8.03 was a testing only version)

Raise minimum calibre version to 1.13.0 due to changes in identifier:url searching
Alternate URL efiction.trekiverse.org for trekiverse.org/efiction
Add do_update_hook to AO3 for updated chapters in the middle
Reorder defaults.ini files to put user/pass is_adult first in all sections
Graceful fail for 'k'HTML when no 'k'Url

redscarlett
12-12-2013, 07:53 PM
I've installed the latest update, and now any archiveofourown.org fics won't download - it just says that "local variable 'chapter' referenced before assignment". I've tried several urls, so it isn't one specific fic. Any ideas please? I'm on the newest version of everything, and I used it fine yesterday.

JimmXinu
12-12-2013, 07:58 PM
I've installed the latest update, and now any archiveofourown.org fics won't download - it just says that "local variable 'chapter' referenced before assignment". I've tried several urls, so it isn't one specific fic. Any ideas please? I'm on the newest version of everything, and I used it fine yesterday.

Oops. New bug that affects AO3 stories with only one chapter. New version shortly.

redscarlett
12-12-2013, 07:59 PM
Oops. New bug that affects AO3 stories with only one chapter. New version shortly.

Thank you!!

JimmXinu
12-12-2013, 08:08 PM
Version 1.8.05 - 12 Dec 2013

Fix issue with AO3 single chapter stories introduced in 1.8.04


Sorry about that.

Depletion
12-16-2013, 12:56 AM
Hi guys, since having my problem fixed last week I've been experimenting with my personal.ini more.. I've gotten it to automatically fill a few custom columns I'd been doing manually which is rather exciting. Now I'm trying out different things that I've wanted but calibre didn't start with. However, I've run into a bit of trouble with mixing custom columns and site-specific metadata.

Specifics:
On fimfiction.net, I'm trying to create/populate a custom column for Likes. I have a specific custom column set up, restarted calibre, and went to the custom columns tag under FFDL. It'd not there though so I'm pretty sure I'm missing something in my personal.ini. Probably somewhat related to these lines if I had to guess:

#custom_columns_settings:
# cliches=>#acolumn,r
# themes=>#bcolumn,a
# timeline=>#ccolumn,n

I've been reading the wiki and the FAQs which helped some of my other problems but not this one. Anyway, advice on how or where to read to get to the next step would be appreciated. :)

FaceDeer
12-16-2013, 02:41 AM
Those lines are commented out (the preceding # does that), so I'm not sure how they could have an effect on anything.

I collect Likes metadata from FiMFiction myself. Here's what I have in the [www.fimfiction.net] section of my personal.ini:

custom_columns_settings:
likes=>#likes,r
dislikes=>#dislikes,r
views=>#views,r
total_views=>#total_views,r
short_description=>#short_description,r
groups=>#groups,r
prequelHTML=>#prequel,r
sequelsHTML=>#sequels,r

The #likes custom column is set to "Integers" type, as is the #dislikes, #views, and #total_views column. #short_description is a "Long text" type column, and #groups, #prequel and #sequels are all "Comma separated text" types.

JimmXinu
12-16-2013, 11:34 AM
...
Specifics:
On fimfiction.net, I'm trying to create/populate a custom column for Likes. I have a specific custom column set up, restarted calibre, and went to the custom columns tag under FFDL. It'd not there though so I'm pretty sure I'm missing something in my personal.ini. Probably somewhat related to these lines if I had to guess:
...

Site specific metadata don't appear on the custom columns tab, only the regular ones.

FaceDeer's answer is correct, for site specific metadata you need to use custom_columns_settings under [www.fimfiction.net] section.



[www.fimfiction.net]

# lines starting with # are comments only.

custom_columns_settings:
likes=>#likes,r

# 'likes' and other lines to be included in custom_columns_settings
# must each start with a space.

JimmXinu
12-16-2013, 04:20 PM
Test version with new site: literotica.com adapter mostly by de3sw2aq1. This is very much an adult site.

UPDATE Dec 17, 2013 -- Remove obsolete beta versions
(Look for a newer version in more recent posts.)

Depletion
12-17-2013, 02:28 AM
I was using the commented out info just for examples. And thanks Facedeer, your answer helped me see where I was going wrong. I was going into configure FFDL and the custom column tag when I should have just updated the metadata. Silly mistake in hindsight but I appreciate both your and JimmXinu's help. :)

davidfor
12-17-2013, 08:13 AM
Test version with new site: literotica.com adapter mostly by de3sw2aq1. This is very much an adult site.

I had to look :)

The formatting of the epubs produced has a problem. This is because the story text is wrapped in a single p tag. It uses br tags to create the impression of paragraphs. The adapter passes the text through utf8FromSoup and will use replace_br_with_p if the users options have it set. But, because the text is wrapped in a single set of paragraph tags, nothing is changed. To get it working right, I had to strip the tags off.

At line 200 is:

story1 = soup1.find('div', 'b-story-body-x')
storytext = self.utf8FromSoup(url, story1)

I changed this to:

story1 = soup1.find('div', 'b-story-body-x')
story1 = str(story1.find('div').p)
story1 = story1[len('<p>'):len(story1) - len('</p>')]
story1 = bs.BeautifulSoup(story1)
storytext = self.utf8FromSoup(url, story1)


In the page loop, line 215 is the same.

This of course doesn't take into account having the "replace_br_with_p" set to false. In that case, the paragraph tag probably should be put back.

JimmXinu
12-17-2013, 12:00 PM
The formatting of the epubs produced has a problem. This is because the story text is wrapped in a single p tag. It uses br tags to create the impression of paragraphs. The adapter passes the text through utf8FromSoup and will use replace_br_with_p if the users options have it set. But, because the text is wrapped in a single set of paragraph tags, nothing is changed. To get it working right, I had to strip the tags off.
...

Good catch. Here's a version that addresses the issue.

This version also adds more sleep time between hits to ffnet for metadata. I think that may help people who are getting errors from ffnet; I suspect it's because they are doing too many downloads/updates at once.

However, that also means it takes longer for FFDL to get metadata before pushing downloads into the background. I'm not convinced that this is worth the extra time it will cost people who aren't attempting mass downloads/updates. Let me know what you think.

UPDATE Dec 17, 2013 -- Remove obsolete beta versions
(Look for a newer version in more recent posts.)

rlh3
12-17-2013, 03:08 PM
I was doing some random testing of the Literotica adapter and encountered this error.


'NoneType' object has no attribute '__getitem__'
http://www.literotica.com/s/you-dont-choose-love

I have encountered this on several stories, but this is the only one that I remembered to write down.

Also, I cannot seem to get the "are you an adult" dialogue to go away. My personal.ini entry is (username and password deleted):

[Literotica.com]
username:XXXXXX
password:XXXXXX
collect_series: true
is_adult:true

Am I doing something wrong here?

JimmXinu
12-17-2013, 08:36 PM
'NoneType' object has no attribute '__getitem__'
http://www.literotica.com/s/you-dont-choose-love

That's a real problem caused by bad parsing when it's part of a 'series' (which are being treated as chapters of a single story) when it's the last series in the list.

Also, I cannot seem to get the "are you an adult" dialogue to go away. My personal.ini entry is (username and password deleted):

[Literotica.com]
username:XXXXXX
password:XXXXXX
collect_series: true
is_adult:true

[literotica.com] -- case matters.

This adapter doesn't use user/pass or the collect_series option.

The decision to treat 'series' as multi-chapter stories was made by the original adapter author. Since they all seem to be name XXX Ch 01, etc, it seems the right call to me.

UPDATE Dec 17, 2013 -- Remove obsolete beta versions
(Look for a newer version in more recent posts.)

rlh3
12-17-2013, 09:27 PM
Seems to work just fine now. The case change in the personal.ini fixed that right up. Whatever you did on the other error seems to have solved it. I have only had time for some quick and dirty testing, but I will give it a better test tonight or in the morning, and if anything else comes up I will be sure to mention it.

thanks

davidfor
12-17-2013, 10:22 PM
It works, but, for the replace_br_with_p to work, it needs the paragraph tag that wraps the text removed. The conversion to string and then the substring was because I couldn't work out a way to get BeautifulSoup to give me the contents of the paragraph as a BeautifulSoup object that utf8FromSoup would work on.

JimmXinu
12-17-2013, 10:30 PM
It works, but, for the replace_br_with_p to work, it needs the paragraph tag that wraps the text removed. The conversion to string and then the substring was because I couldn't work out a way to get BeautifulSoup to give me the contents of the paragraph as a BeautifulSoup object that utf8FromSoup would work on.

Can you provide a story URL and what you think is wrong?

I tested both with and without replace_br_with_p and both looked fine to me.

davidfor
12-17-2013, 11:36 PM
I picked a story from the author that rlh3 posted: http://www.literotica.com/s/down-drinking-at-the-bar. The choice was simply that it had a short first paragraph.

When I look at the code your version is producing, at the start I see:

<h3>Down Drinking at the Bar</h3>
<div>
<p>Joe Morrison wasn't particularly happy. His wife, who had been much too fond of drink had developed Korsakov, and she'd been in a care centre for years. He usually visited her in the afternoons.<br />
<br />
Life was lonely at home; it had been lonely for as long as he could remember. There were a few comforts he'd stuck to, all those years: his collection of poetry, string quartets, a dram before bed and the barmaid at the Rose and Crown.<br />
<br />
.
.
.
"Didn't you hear? He drove into tree. Must've been dead at once."<br />
<br />
"Oh no," Polly said. She slumped down with her arms on the bar, put her head on her arms and cried.</p>
</div>

When I strip the paragraph tag, I get:

<h3>Down Drinking at the Bar</h3>
<div>
<p>Joe Morrison wasn't particularly happy. His wife, who had been much too fond of drink had developed Korsakov, and she'd been in a care centre for years. He usually visited her in the afternoons.</p>
<p>Life was lonely at home; it had been lonely for as long as he could remember. There were a few comforts he'd stuck to, all those years: his collection of poetry, string quartets, a dram before bed and the barmaid at the Rose and Crown.</p>
.
.
.
<p>"Didn't you hear? He drove into tree. Must've been dead at once."</p>
<p>"Oh no," Polly said. She slumped down with her arms on the bar, put her head on her arms and cried.</p>
</div>

The first one always has the blank lines when reading. The latter can be styled however you want it.

When I look at the method replace_br_with_p, I found the following comment:

# Find all bexisting blocks with p, pre and blockquote tags, we need to shields break tags inside those.

That's what made me strip the paragraph tag.

JimmXinu
12-18-2013, 12:46 AM
I picked a story from the author that rlh3 posted: http://www.literotica.com/s/down-drinking-at-the-bar. The choice was simply that it had a short first paragraph.
...

This version changes the p tag to a div tag and adds another br explicitly to each page. It appears to work for both replace_br_with_p: true and false.

However, I don't have a big enough sample set to know if all literotica stories use <br><br> paragraph breaks.

If this still doesn't satisfy, we'll have to get Asbjørn to look at changing replace_br_with_p--that's his baby and I treat it like a black box. I'm not willing to put conditionals in individual adapters for replace_br_with_p.

UPDATE Jan 4, 2013 - Remove obsolete beta versions

davidfor
12-18-2013, 02:01 AM
That looks good. And from a look around, they all seem to be formatted in exactly the same way.

And I can't blame you about replace_br_with_p. I had a quick look and it has far to many and complex regex's for my liking. Luckily the comments gave me an idea of what was happening.

rlh3
12-18-2013, 03:24 PM
In continued testing of the Literotica adapter, I have found that the "get story URLs to download from web page" function does not work. I just discovered this function, and thought to try it.

This is the error message I get:

calibre, version 1.15.0
ERROR: Unhandled exception: <b>TypeError</b>:unbound method getSiteExampleURLs() must be called with LiteroticaSiteAdapter instance as first argument (got nothing instead)

calibre 1.15 isfrozen: True is64bit: False
Windows-8-6.2.9200 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '8', '6.2.9200')
Python 2.7.5
Windows: ('8', '6.2.9200', '', 'Multiprocessor Free')
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 399, in get_urls_from_page_menu
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 415, in get_urls_from_page
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.geturls", line 37, in get_urls_from_page
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.__init__", line 171, in getAdapter
TypeError: unbound method getSiteExampleURLs() must be called with LiteroticaSiteAdapter instance as first argument (got nothing instead)


If this in functionality that is not going to be supported, then no big deal, but I thought that I would mention it.

JimmXinu
12-18-2013, 04:54 PM
In continued testing of the Literotica adapter, I have found that the "get story URLs to download from web page" function does not work. I just discovered this function, and thought to try it.

This is the error message I get:
...

Good catch. It will be fixed in the next version.

cryzed
12-18-2013, 08:35 PM
Hey there Jimm!
Just letting you know that I still enjoy your plugin and love that you are so actively developing it. I'm really thankful for your continued work.

Greetings!

JimmXinu
12-18-2013, 09:36 PM
Just letting you know that I still enjoy your plugin and love that you are so actively developing it. I'm really thankful for your continued work.

Thanks, and you're welcome. :thumbsup:

hevreka
12-21-2013, 11:24 AM
Hiya i am having some problems with FFDL when i try to download a storie.:help:

i Get This From FFDL :

Status // Title // Author // Comment // URL
Error // Harry Potter: The Lone Traveller // The Professional // 'utf8' codec can't decode byte 0xf6 in position 41: invalid start byte //
https://www.fanfiction.net/s/2673584/1/

the "//" i put in the Error text to help see the string and not make it a blurrr.

I Would Liket to say that FFDL save me a lot of hassle when flagfic whent down for ff.net.:thanks:

My Best Regards
Hevreka

Feynor
12-21-2013, 03:37 PM
Hiya i am having some problems with FFDL when i try to download a storie.:help:

i Get This From FFDL :

Status // Title // Author // Comment // URL
Error // Harry Potter: The Lone Traveller // The Professional // 'utf8' codec can't decode byte 0xf6 in position 41: invalid start byte //
https://www.fanfiction.net/s/2673584/1/

the "//" i put in the Error text to help see the string and not make it a blurrr.

I Would Liket to say that FFDL save me a lot of hassle when flagfic whent down for ff.net.:thanks:

My Best Regards
Hevreka

I know this is going to be asked. Which version of Calibre and FFDL are you using, and on which OS?

Calibre 1.16 and FFDL 1.8.5 on Windows 7 have no problems with that URL.

JimmXinu
12-21-2013, 09:15 PM
I know this is going to be asked. Which version of Calibre and FFDL are you using, and on which OS?

Calibre 1.16 and FFDL 1.8.5 on Windows 7 have no problems with that URL.

Yep. Works fine for me, too. Make sure you (hevreka) have the latest version of both plugin and calibre.

Amalthia
12-22-2013, 05:12 PM
Hi, I was curious to know if the Angst Archive (http://amothea.slashcity.com/db/index.php)could be added to the list of archives FFDL plugin supports? It's an efiction archive so I'm not sure if I'd need to make code changes on the archive to make this happen?

JimmXinu
12-22-2013, 10:03 PM
Hi, I was curious to know if the Angst Archive (http://amothea.slashcity.com/db/index.php)could be added to the list of archives FFDL plugin supports? It's an efiction archive so I'm not sure if I'd need to make code changes on the archive to make this happen?

663 stories and no updates on their Recent list. Plus, from the front page:

The Angst Archive
otherwise known as the issues of consent archive
...
If you do not like stories containing physical and/or psychological abuse, and non-consensual sex. Please find another archive.

If somebody else wants to write an adapter for it, I'll include it. But I'm not interested in doing it.

Amalthia
12-23-2013, 12:14 AM
Thanks for giving it consideration. It's an older archive.

Feynor
12-23-2013, 03:18 AM
FF.net have made a little change that breaks the adapter.
(The URL for fictionratings in line 160 of adapter_fanfictionnet.py needs to be changed to https)

Jim, I pushed the change on Hg. I hope you don't mind.

Happy Holidays

Edit: Removed obsolete build

Feynor
12-23-2013, 03:44 AM
Is there a way to "enhance" FFDL to only download new chapters?
FF.net is getting very slow when downloading something like this: https://www.fanfiction.net/s/5681042/1/
(75 chapters)
Caveat: This would, of course, mean that existing chapters wouldn't be updated.

aleyx
12-23-2013, 04:51 AM
Is there a way to "enhance" FFDL to only download new chapters?
FF.net is getting very slow when downloading something like this: https://www.fanfiction.net/s/5681042/1/
(75 chapters)
Caveat: This would, of course, mean that existing chapters wouldn't be updated.

I think that's what the "Update EPUB if new chapters" option is for.

Feynor
12-23-2013, 06:03 AM
I think that's what the "Update EPUB if new chapters" option is for.

Hmmm, I'll have to have a look at that option next time I run into a long story. I have seen it of course, but read the "if" literally.

BaconButty
12-23-2013, 09:21 AM
FF.net have made a little change that breaks the adapter.
(The URL for fictionratings in line 160 of adapter_fanfictionnet.py needs to be changed to https)

Jim, I pushed the change on Hg. I hope you don't mind.

Happy Holidays

This resolved the problem for me. :thumbsup:
Thank you :)

leckadams
12-23-2013, 02:49 PM
So I am getting this after attempting to add a fic to the library (brand new not in library at all) from fanfiction.net


Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
'NoneType' object has no attribute 'string'
https://www.fanfiction.net/s/4854949/1/


And I get the same error when trying to update an existing file

Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
'NoneType' object has no attribute 'string'
https://www.fanfiction.net/s/6568400/1/

McHunter
12-23-2013, 06:28 PM
Replace your existing plugin with the zip-archive above from Feynor, FanFiction has tweaket ist code, again....

Would be nice if there was an update of the versions number so calibre installs the new version itselfs, I have searched where the plugin is stored, i Tought ist was somewhere in the caliber order, it wasnt, it was somewehere in the settigs directory of the User.

leckadams
12-23-2013, 06:41 PM
I know that in an earlier post it shows you how to do so, but I odn' tknow where that is...how do i update with the zip drive?

JimmXinu
12-23-2013, 07:11 PM
Version 1.8.06 - 23 Dec 2013

Fix for latest ffnet change.
Additional sleep added to ffnet metadata downloads.
New site: literotica.com. Thanks to de3sw2aq1.
Tweak to htmlheuristics for literotica.com (unicode vs str).


I was going to wait until next week for this update, but ffnet had other ideas.

After 3 years, one of my favorite stories has finished: In Flight by gabriel blessing (https://www.fanfiction.net/s/6556187/1/)
Highly recommended.

Robin42069
12-23-2013, 08:29 PM
thank you for the fix on the plugin i'm new to calibre so i was looking to find out what i did to mess it up. i thought i had killed my new golden goose. i find out fanfiction.net had. so thanks for making it work again cuz i hadnt a clue what was going on:thanks: :) :thumbsup:

Amalthia
12-24-2013, 12:55 AM
Hmmm, I'll have to have a look at that option next time I run into a long story. I have seen it of course, but read the "if" literally.

Were you able to test out this option? Does it really only add new chapters to the story? I've been reading the "if" literally as well. If it had said, "Update Epub 'with' new chapters" that may have been less confusing.

Firedancer885
12-24-2013, 05:09 AM
Were you able to test out this option? Does it really only add new chapters to the story? I've been reading the "if" literally as well. If it had said, "Update Epub 'with' new chapters" that may have been less confusing.

It only adds new chapters, and keeps the existing ones. I had a problem with one of the stories I'm following, because of the author's update method. I always ended up with two identical last chapters when I used the "update if new chapters" method. I'm using "update if" as default and the others only if necessary.

hevreka
12-25-2013, 10:40 AM
I know this is going to be asked. Which version of Calibre and FFDL are you using, and on which OS?

Calibre 1.16 and FFDL 1.8.5 on Windows 7 have no problems with that URL.

Sry that my Reply Took so LOOOONNNNNGG Been a Hectic Week as a busdriver.

Calibre 1.16 and FFDL 1.8.5 Windows 7 Ultimate before the updates Today

After Updates Calibre 1.17.0 and FFDL 1.8.6 same OS

Feynor
12-27-2013, 07:57 AM
Sry that my Reply Took so LOOOONNNNNGG Been a Hectic Week as a busdriver.

Calibre 1.16 and FFDL 1.8.5 Windows 7 Ultimate before the updates Today

After Updates Calibre 1.17.0 and FFDL 1.8.6 same OS

I've been unable to reproduce the error you are mentioning in your original post.

Do you still get it?

If so, please repeat it while Calibre is in Debug mode (Click the Preferences drop down arrow and select "Restart in Debug Mode") and just update this one story, then close Calibre after you get the error.
Save the text file that comes up afterwards and attach it to a response.

TrulyInnovative
12-27-2013, 11:46 AM
The storiesonline.net plugin seems to have some trouble with special characters. For example, the dash symbol (–) is replaced by – and é is replaced by é.

Feynor
12-27-2013, 12:31 PM
The storiesonline.net plugin seems to have some trouble with special characters. For example, the dash symbol (–) is replaced by – and é is replaced by é.

Do you have a link? (To a specific story that is causing this)

Edit:
Just a note, it looks like the utf-8 encoding is not being handled correctly in the adapter for the site. The Site itself does seem to send the correct encoding header.

JimmXinu
12-27-2013, 06:53 PM
Do you have a link? (To a specific story that is causing this)

Edit:
Just a note, it looks like the utf-8 encoding is not being handled correctly in the adapter for the site. The Site itself does seem to send the correct encoding header.

Sometimes individual stories will be uploaded with one encoding while the site assumes another. Without the story URL it's hard to say.

The encoding used by FFDL can be changed for each site in personal.ini:

[storiesonline.net]

website_encodings: utf8, Windows-1252

# or instead:

website_encodings: Windows-1252, utf8

TrulyInnovative
12-28-2013, 11:44 AM
Do you have a link? (To a specific story that is causing this)

Here's an example in the second paragraph of Chapter 1: http://storiesonline.net/s/51245:56696

Oilily Sepperl
12-28-2013, 05:43 PM
Good evening! At first I want to say a big Thank You to the maker of the ffdl plug-in for Calibre. It's fantastic and I love using it!
I have problems lately though and hope somebody can help me.
I run Calibre version 1.17.0 and the FFDL plug-in version 1.8.6.
I haven't downloaded or updated fics for a while and wanted to do that the last two days. But I have trouble downloading or updating more than 5 or 6 fics at once. One single fic works fine and 5 or 6 fics work fine too, but I tried to update for example 14 fics and it wouldn't complete the task. There wasn't any error message and Calibre didn't freeze, but it took such a long time and wouldn't process further.
Same thing with about 100 fics or more. Often it would complete until 97 % and then simply wouldn't process any further for an hour.
I felt this is unusual and ended the tasks and only did about 5 fics.
It's unnerving though, because I have a lot of fics in-progress which I like to update regularly.
Did anyone here have this problem already or does anybody know what I could do or where the problem lies? Thanks in advance!

davidfor
12-28-2013, 08:58 PM
Here's an example in the second paragraph of Chapter 1: http://storiesonline.net/s/51245:56696

I can see the problem. Adding "website_encodings: utf8, Windows-1252" to the personal.ini fixes it. The page info in Firefox tells me it is "UTF-8"

Looking at the code, it has:

self.decode = ["Windows-1252",
"utf8"]

From a quick look, most of the adapters have them in the same order. The base adapter is the other way around. As the page reports UTF-8, either the order in the adapter should be changed, or it should be removed as there is no real need to override the parent class.

But, there is another error. The summary/comments has "A Story in the Bec - Lost in the Maze Universe" rather than the story description. That should be the series. And it is also missing the author notes and disclaimer that is on the index page of the story. I'll have a look to fix this, but it might not be until I get back home in a week or so. At the moment, I don't have a full time connection.

cryzed
12-29-2013, 07:43 PM
Hey there, I've been using the fanficdownloader module as part of the scripts running on my Raspberry Pi for quite a while now. Recently I've started getting this error:

cryzed ~/fanfiction >>> python update.py
(30.12.2013) 00:38:50: + Checking story...
Traceback (most recent call last):
File "update.py", line 205, in <module>
main()
File "update.py", line 160, in main
if is_up_to_date(epub_path, adapter):
File "update.py", line 74, in is_up_to_date
remote_chapter_count = int(adapter.getStoryMetadataOnly().getMetadata('nu mChapters'))
File "/home/cryzed/fanfiction/fanficdownloader/adapters/base_adapter.py", line 243, in getStoryMetadataOnly
self.extractChapterUrlsAndMetadata()
File "/home/cryzed/fanfiction/fanficdownloader/adapters/adapter_fanfictionnet.py", line 161, in extractChapterUrlsAndMetadata
rating = a.string
AttributeError: 'NoneType' object has no attribute 'string'


I suppose this is a simple error in the fanfiction.net adapter module, due to recent changes of their site-layout. I run a script once daily to retrieve the latest release (https://code.google.com/p/fanficdownloader/downloads/list) of the module, but it hasn't been updated for a bit apparently. Do you guys think it would be possible to once again update the downloadable module on the site?

JimmXinu
12-29-2013, 07:57 PM
...
I suppose this is a simple error in the fanfiction.net adapter module, due to recent changes of their site-layout. I run a script once daily to retrieve the latest release (https://code.google.com/p/fanficdownloader/downloads/list) of the module, but it hasn't been updated for a bit apparently. Do you guys think it would be possible to once again update the downloadable module on the site?

Google Code will not support creating new downloads starting January 15th, 2014. The last download on the 'download' page includes this notification, which was also published on the FFDL Google group:

NEW CLI DOWNLOAD URL

Google Code is blocking new downloads in the download area next month. This is a simple alternative, so let's see how it works out for everyone.

The newest version of the CLI will be available at this URL every release:

http://fanficdownloader.googlecode.com/hg/fanficdownloader.zip

Back versions will be available by browsing the repository for that file:
http://code.google.com/p/fanficdownloader/source/browse/fanficdownloader.zip

JimmXinu
12-29-2013, 08:06 PM
I can see the problem. Adding "website_encodings: utf8, Windows-1252" to the personal.ini fixes it. The page info in Firefox tells me it is "UTF-8"

Looking at the code, it has:

self.decode = ["Windows-1252",
"utf8"]

From a quick look, most of the adapters have them in the same order. The base adapter is the other way around. As the page reports UTF-8, either the order in the adapter should be changed, or it should be removed as there is no real need to override the parent class.

It's saddeningly common for sites to report one encoding while using the other more commonly. Or to use both on different stories.

I don't read storiesonline.net. If someone who does tells me that utf8 is correct more often than Windows-1252, I'm willing to change it.

But, there is another error. The summary/comments has "A Story in the Bec - Lost in the Maze Universe" rather than the story description. That should be the series. And it is also missing the author notes and disclaimer that is on the index page of the story. I'll have a look to fix this, but it might not be until I get back home in a week or so. At the moment, I don't have a full time connection.

I don't have the energy to look at this right now either. Maybe tomorrow.

JimmXinu
12-29-2013, 08:16 PM
...
But I have trouble downloading or updating more than 5 or 6 fics at once. One single fic works fine and 5 or 6 fics work fine too, but I tried to update for example 14 fics and it wouldn't complete the task. There wasn't any error message and Calibre didn't freeze, but it took such a long time and wouldn't process further.
Same thing with about 100 fics or more.
...

If you're trying to update that many stories at once, you're probably running into throttling on the server. Especially if they are fanfiction.net stories.

I've written before (http://www.mobileread.com/forums/showthread.php?p=2701816#post2701816) about why doing mass downloads/updates is a bad idea.

davidfor
12-29-2013, 08:29 PM
It's saddeningly common for sites to report one encoding while using the other more commonly. Or to use both on different stories.

I don't read storiesonline.net. If someone who does tells me that utf8 is correct more often than Windows-1252, I'm willing to change it.

A quick random click around and all are UTF-8. I would expect only the one encoding as all pages on the site is generated. And whatever is generating it is custom written for this and the related sites.

I don't have the energy to look at this right now either. Maybe tomorrow.

Don't worry to much. I'll probably get bored in the next few days and have a go. If not, definitely sometime next week.

cryzed
12-30-2013, 01:29 AM
This might also be interesting for corner-cases: In case the decoding fails, one could fallback to the chardet (https://pypi.python.org/pypi/chardet) module, and try to detect the real encoding. And even if then there are still errors or the encoding couldn't be detected properly, just do a .decode(..., 'ignore'/'replace') possibly instead of crashing? I'm not sure what the module currently does.

Oilily Sepperl
12-30-2013, 06:29 AM
If you're trying to update that many stories at once, you're probably running into throttling on the server. Especially if they are fanfiction.net stories.

I've written before (http://www.mobileread.com/forums/showthread.php?p=2701816#post2701816) about why doing mass downloads/updates is a bad idea.

Thank you so much for your answer, JimmXinu. Should have searched first, but I didn't know which words to use.
I assumed something like that already - was surprised though, because it functioned quite a while for me -, so I'll simply change my personal updating system.
Thank you again!

davidfor
12-30-2013, 07:10 AM
I have fixed the problems noted in the last few days plus a couple of other things. These are:

- Site uses UTF-8. As the base adapter uses this first, I have removed the setting from this adapter.

- If the story was in a series or universe, the summary was just that text and didn't include the story description.

- While looking at the above, I realised that when a story was in a universe, that was being lost. So, when there is a universe, the series is set to the universe. One problem with this is that with a universe, there isn't an order, so the series index is set to zero.

The other thing I mentioned the other day was the notice on some story index pages. This can be a disclaimer or some sort or some setup for the story. http://storiesonline.net/s/51245 is the story that started this round of changes, and it has a notice. Extracting the notice was easy. But, then I couldn't quickly work out how to display it in the story. It probably should be added after the story details, but, I don't immediately see how to do that. I have put the extract code in. If someone can tell me where to look, I can add it to the actual story.

Also, the finestories.com should have the change for the character set. I had a look at a few pages and they are being generated as UTF-8. I think it should have the universe/series change as well, but I didn't have a close look at that.

JimmXinu
12-30-2013, 09:52 PM
I have fixed the problems noted in the last few days plus a couple of other things. These are:

- Site uses UTF-8. As the base adapter uses this first, I have removed the setting from this adapter.
...

Okay.

- While looking at the above, I realised that when a story was in a universe, that was being lost. So, when there is a universe, the series is set to the universe. One problem with this is that with a universe, there isn't an order, so the series index is set to zero.

That's not a good solution. I've changed it to use the site specific metadata universe and universeUrl. The datum universeHTML will be automatically created. I haven't tested it because I don't have any story URLs with universe. You'll need to add universeHTML to extra_titlepage_entries to see it.

The other thing I mentioned the other day was the notice on some story index pages. This can be a disclaimer or some sort or some setup for the story. http://storiesonline.net/s/51245 is the story that started this round of changes, and it has a notice. Extracting the notice was easy. But, then I couldn't quickly work out how to display it in the story. It probably should be added after the story details, but, I don't immediately see how to do that. I have put the extract code in. If someone can tell me where to look, I can add it to the actual story.

Added in another new site specific metadatum, notice.

Also, the finestories.com should have the change for the character set. I had a look at a few pages and they are being generated as UTF-8. I think it should have the universe/series change as well, but I didn't have a close look at that.

I've changed the encoding, but I'm not going to add the universe code with an example story URL to test against.

Attached is a test version with the changes discussed.

UPDATE Jan 4, 2013 - Remove obsolete beta versions

JimmXinu
12-30-2013, 10:05 PM
This might also be interesting for corner-cases: In case the decoding fails, one could fallback to the chardet (https://pypi.python.org/pypi/chardet) module, and try to detect the real encoding. And even if then there are still errors or the encoding couldn't be detected properly, just do a .decode(..., 'ignore'/'replace') possibly instead of crashing? I'm not sure what the module currently does.

There's already a copy of chardet in FFDL--it can be invoked by putting 'auto' in the website_encodings parameter:
website_encodings: auto, utf8, Windows-1252

When I added it, chardet did a very poor job of differentiating utf8 from iso8859-1 or windows-1252. So while I left it in, I didn't use it much.

I should either update it so it works better or remove it outright, I suppose.

davidfor
12-31-2013, 06:54 AM
That's not a good solution. I've changed it to use the site specific metadata universe and universeUrl. The datum universeHTML will be automatically created. I haven't tested it because I don't have any story URLs with universe. You'll need to add universeHTML to extra_titlepage_entries to see it.

The story that started all this, http://storiesonline.net/s/51245, is in a universe, but it 42 chapters. I had a look through the list of universes and found the the "BOFH" universe - BOFH fanfic, so it is apt for this plugin and proof that there is fanfic for everything. One of the stories is http://storiesonline.net/s/64407.

I can see why, but, I'm not sure I agree. The site has both universes and series (and series can be ordered or be unordered). Universes can hold individual stories or series. But, there is no way to tell from a story in a series that the series is part of a universe. That means a story will only have one. For me, I suppose I want to get whichever the story has into the series column. Can I do this through the preferences or personal.ini?

For curiosity, I had a look at what is in one of the site generated epubs. For the BOFH story, http://storiesonline.net/s/64407, the OPF has:
<meta name="calibre:series" content="BOFH"/>

For a book that is in a series, it also has the series index. So, the site is at least partly treating the universe the same as a series.


Added in another new site specific metadatum, notice.

Using http://storiesonline.net/s/51245 (which is also in a universe), I had an error that seems to be related to this. The traceback is:
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 590, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 1068, in prep_download_loop
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.story", line 542, in formatFileName
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.story", line 419, in getAllMetadata
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.story", line 330, in getMetadata
TypeError: sequence item 0: expected string or Unicode, Tag found

Looking at the code, the value for the notice needed to be converted to a string. That was my mistake.

I've changed the encoding, but I'm not going to add the universe code with an example story URL to test against.

That's fair enough. I'll have a look and see if I can find something suitable for testing.

JimmXinu
01-02-2014, 05:30 PM
davidfor,

So in summary, a storiesonline.net story can be a member of:

an ordered series;
an unordered series, or;
a universe.

Additionally, a series (ordered or not) can also be a member of a universe.

And you'd like 'universe' to be populated into the standard 'series' metadatum when a story is a direct member of a universe, since it can't be a member of a series then.

I suppose that's not unreasonable, but I'd also like to be able to populate 'universe' for stories that are members of series that are in turn members of a universe.

Have you found example stories of each of these cases? I haven't found any unordered series or series that are a member of a universe. I suspect the existing code will break on an unordered series.

...
Looking at the code, the value for the notice needed to be converted to a string. That was my mistake.

Fixed for next version.

TrulyInnovative
01-03-2014, 12:08 AM
Have you found example stories of each of these cases? I haven't found any unordered series or series that are a member of a universe. I suspect the existing code will break on an unordered series.


Here are examples of series that are members of a universe:
http://storiesonline.net/library/universe.php?id=4
http://storiesonline.net/library/universe.php?id=659

Here are unordered series (presumably called collections):
http://storiesonline.net/library/show_series.php?id=177
http://storiesonline.net/library/show_series.php?id=604

Incidentally, it's possible that authors make a mistake in numbering a series. This series has three books all marked #1 and one book marked #2 in the series:
http://storiesonline.net/library/show_series.php?id=113

davidfor
01-03-2014, 02:47 AM
davidfor,

So in summary, a storiesonline.net story can be a member of:

an ordered series;
an unordered series, or;
a universe.

Additionally, a series (ordered or not) can also be a member of a universe.

Yes to all of that.

And you'd like 'universe' to be populated into the standard 'series' metadatum when a story is a direct member of a universe, since it can't be a member of a series then.

I suppose that's not unreasonable, but I'd also like to be able to populate 'universe' for stories that are members of series that are in turn members of a universe.

Yes. I think the lowest level of universe/series membership should be in the series column. Populating the universe would also be good, but, there isn't a direct link. From what I can see, to get the universe you need to go back to the author page, choose the "Universes" link and then see if the story or the series it belongs to is there.

Actually that might not be that bad. The authors universe page has "&type=uni" added to the authors URL. Then it is simply a matter of finding an a tag that refers to the story page. I'll have a play later.

Have you found example stories of each of these cases? I haven't found any unordered series or series that are a member of a universe. I suspect the existing code will break on an unordered series.

TrulyInnovative posted examples, but here are a couple that I found while clicking through the universes page. http://storiesonline.net/library/universe.php?id=19 is a universe with a series. The series looks to be ordered. And http://storiesonline.net/library/universe.php?id=377 has a series that doesn't seem to be ordered. Though from the names it should have been.

The unordered series is broken. I just tested to from stories from the above. The series name and link is put into the description and the description is lost. I'll have a play with that as well.

davidfor
01-03-2014, 04:11 AM
The unordered series is broken. I just tested to from stories from the above. The series name and link is put into the description and the description is lost. I'll have a play with that as well.

The following seems to work:

try:
a = lc4.find('a', href=re.compile(r"/library/show_series.php\?id=\d+"))
if a:
series_contents = a.parent.contents
i = 0 if len(series_contents) == 1 else series_contents[1].strip(' ()')
self.setSeries(stripHTML(a), i)
self.story.setMetadata('seriesUrl','http://'+self.host+a['href'])
desc = lc4.contents[2]
except:
pass
For unordered series, I'm not sure what the best series index is. Zero seems to be the safest. But, maybe it should be 1. I'm tempted to use zero for universes and 1 for series as they can be inside a universe. But, that might be complicating things.

aleyx
01-03-2014, 06:21 AM
Maybe taking advantage of hierarchical series?

series = Series Name [0] (no universe, unordered series)
series = Series Name [n] (no universe, ordered series)
series = Universe [0] (universe, no series)
series = Universe.Series Name [0] (universe, unordered series)
series = Universe.Series Name [n] (universe, ordered series)

davidfor
01-03-2014, 06:42 AM
That does make sense. I'm not sure if I am a fan of the "Universe.Series", but I can see that some people would like it. Maybe an option in the personal.ini?

aleyx
01-03-2014, 06:58 AM
Well, it's the separator character for hierarchical items in the tag browser; I don't believe it's configurable. Obviously if the tag browser is not a consideration (along with all the search- and filter-related features), any string would do.

davidfor
01-03-2014, 07:11 AM
No, I meant configure what gets put into the series field if the book is in series that is in a universe. I think I would be happy with just the series, but I can understand of someone wanted both.

fidofido300
01-04-2014, 01:27 PM
Hi;
While you are playing with the code can you look
http://storiesonline.net/s/52707
I tried to download it but I was forced to stop after 571 and change minutes later :)
Thanks

JimmXinu
01-04-2014, 03:44 PM
Hi;
While you are playing with the code can you look
http://storiesonline.net/s/52707
I tried to download it but I was forced to stop after 571 and change minutes later :)
Thanks

That story has 329 chapters and 1.8 million words--it just takes a long time to download, especially if you have slow_down_sleep_time set or a slow Internet connection.

JimmXinu
01-04-2014, 03:51 PM
Attached is a test version with:

Fix for 'Update' with no updates when 'Mark' option is selected.
Fix for storiesonline.net non-ordered series.
Include 'universe' as series for storiesonline.net stories with universe. Optional, defaults on.


For the case of a story in a series in a universe, I don't see anyway to get the universe from the story.

UPDATE Jan 5, 2014 -- Remove obsolete beta versions

davidfor
01-04-2014, 07:08 PM
Attached is a test version with:

Fix for 'Update' with no updates when 'Mark' option is selected.
Fix for storiesonline.net non-ordered series.
Include 'universe' as series for storiesonline.net stories with universe. Optional, defaults on.


For the case of a story in a series in a universe, I don't see anyway to get the universe from the story.

I seem to have lost a post. I wrote a post last night with a version of the adapter with the series and universe changes. Including a solution for tha last problem. I'll test your change and see if I can merge mine in.

davidfor
01-04-2014, 07:18 PM
Hi;
While you are playing with the code can you look
http://storiesonline.net/s/52707
I tried to download it but I was forced to stop after 571 and change minutes later :)
Thanks

Is that still going? As JimmXinu said, that is big and will take ages. Especially if the images are downloaded. Last time I saw that story, there was a web site for it. It is probably worthwhile looking for it and seeing if it has an e-book version available. The authors blog at storiesonline has the URL and that there are updates at the site.

JimmXinu
01-04-2014, 09:19 PM
That story has 329 chapters and 1.8 million words--it just takes a long time to download, especially if you have slow_down_sleep_time set or a slow Internet connection.

Make that 2.3 million words. And 2000+ images. :eek:

davidfor
01-05-2014, 02:36 AM
As threatened earlier, here is yet another version of the storiesonline.net adapter.

This adds the fetch of the universe name and URL for stories that are in a series that is part of a universe. I thought I had posted this last night, but something happened.

There is of course another change. While testing this, I realised the series and universe name displayed differently depending on where it is. As an example, for the story http://storiesonline.net/s/51121, the universe is shown as:

- "A story set in the post-sickness world of Phoenicia" on the story index and the author page.
- "Phoenicia — A Universe from the Mind of Shakes Peer2B" on the universe page
- "Universe: Phoenicia" on the authors universe page
- "Phoenicia" on the universe list page

The first one is good on the title page, but I think the last is the correct one to use in the metadata. I've added code to retrieve this for both the series and the universes.

I think these or similar changes should also be made to the finestories.com adapter. I haven't had a chance to look at the site yet. I'll probably do it later in the week and make any updates necessary.

JimmXinu: I have left a few debug statements in this section of the code. I'm happy for them to be removed, but, they might be useful for the next little while.

fidofido300
01-05-2014, 10:55 AM
Thanks
Ok I will search for the Authors site.
Actually the main question was does the adapter downloads illustrated stories images or not?
Thanks again :)

JimmXinu
01-05-2014, 12:10 PM
As threatened earlier, here is yet another version of the storiesonline.net adapter.
...
Attached is a test version with davidfor's latest changes.

I think these or similar changes should also be made to the finestories.com adapter. I haven't had a chance to look at the site yet. I'll probably do it later in the week and make any updates necessary.
...
Sounds good.

UPDATE Jan 7, 2014 - Remove obsolete beta versions

JimmXinu
01-07-2014, 09:00 PM
New test version, now with French translation. Thanks, Serge!

UPDATE Jan 8, 2014 - Remove obsolete beta versions

JimmXinu
01-08-2014, 04:58 PM
New test version with some fixes for the new French translation. Thanks again to Serge.

UPDATE Jan 13, 2014 - Remove obsolete beta versions

citac
01-10-2014, 10:49 PM
Hi all, I have a question on downloading an ff.net story. I've just been recced https://www.fanfiction.net/s/8466122/1/The-Long-Game and would like to download it for later. Unfortunately the story cannot be downloaded, and the log shows the following:

Status Title Author Comment
Bad Unknown Unknown 'NoneType' object has no attribute 'string'

URL
http://www.fanfiction.net/s/8466122/1/

The pop up then asks if I want to update my library error column. How can I download this story, and how do I set up the error column? I tried searching the calibre forums, but I'm mostly getting results for "error" on its own. I'd appreciate any advice.

JimmXinu
01-10-2014, 11:18 PM
It downloaded fine for me.

Make sure you're using the most recent version of the plugin.

If it still doesn't work, it may be ffnet blocking you. They've taken to doing that to people who do 'too many' requests to their servers too quickly. How many is 'too many' is unknown.

You can create a text custom column in calibre and at the bottom of the Custom Columns tab of the FFDL config, you can set FFDL to populate error messages there. This only works for updates/overwrites of existing stories, not new downloads--there's no book to write the error too for them.

Hi all, I have a question on downloading an ff.net story. I've just been recced https://www.fanfiction.net/s/8466122/1/The-Long-Game and would like to download it for later. Unfortunately the story cannot be downloaded, and the log shows the following:

Status Title Author Comment
Bad Unknown Unknown 'NoneType' object has no attribute 'string'

URL
http://www.fanfiction.net/s/8466122/1/

The pop up then asks if I want to update my library error column. How can I download this story, and how do I set up the error column? I tried searching the calibre forums, but I'm mostly getting results for "error" on its own. I'd appreciate any advice.

chrnno
01-12-2014, 01:14 PM
Hi, having a problem here. Been downloading from tthfanfic without problems so far but got this problem in the two stories of a series.

Error
XSGCOM: Goa'uld Defence
Hotpoint
'utf8' codec can't decode byte 0xe3 in position 51: invalid continuation byte
http://www.tthfanfic.org/Story-16092

Error
XSGCOM: Terra from the Deep
Hotpoint
'utf8' codec can't decode byte 0xe3 in position 51: invalid continuation byte
http://www.tthfanfic.org/Story-21213

chrnno
01-12-2014, 11:00 PM
I can't seem to find the edit button(?) so posting again to add:

Error
OMAKES for In Flight
chronodekar
'utf8' codec can't decode byte 0xe3 in position 51: invalid continuation byte
https://www.fanfiction.net/s/6957555/1/
Error
Escaping Fate
Arashi Leonhart
'utf8' codec can't decode byte 0xe3 in position 51: invalid continuation byte
https://www.fanfiction.net/s/6960576/1/

Also hpfanficarchive has made some major changes to the site so while I have no tested it myself problems are likely to arise there.

And lastly I would like to say thanks for this amazing plugin and to how much you do to keep it to update. It is very much appreciated.

Jester11
01-12-2014, 11:49 PM
I have started getting this error from the hpfanficarchive:

time data '' does not match format '%m %d, %Y'

JimmXinu
01-13-2014, 04:07 PM
...
Error
XSGCOM: Goa'uld Defence
Hotpoint
'utf8' codec can't decode byte 0xe3 in position 51: invalid continuation byte
http://www.tthfanfic.org/Story-16092
...
Error
Escaping Fate
Arashi Leonhart
'utf8' codec can't decode byte 0xe3 in position 51: invalid continuation byte
https://www.fanfiction.net/s/6960576/1/

Also hpfanficarchive has made some major changes to the site so while I have no tested it myself problems are likely to arise there.

Those are working fine for me.

The fact you're getting the same error in the same place for two different sites suggests that perhaps you're getting some kind of error/auth page injected by your provider. Like when you're using a hotel's wi-fi and they want you to click through their web page first.

hpfanficarchive.com has changed and I have fixes for it. Posted separately.

JimmXinu
01-13-2014, 04:08 PM
New test version with fixes for hpfanficarchive.net changes.

UPDATE Jan 13, 2014 - Remove obsolete beta versions

chrnno
01-13-2014, 05:14 PM
Those are working fine for me.

The fact you're getting the same error in the same place for two different sites suggests that perhaps you're getting some kind of error/auth page injected by your provider. Like when you're using a hotel's wi-fi and they want you to click through their web page first.

hpfanficarchive.com has changed and I have fixes for it. Posted separately.
Hum, from home here and I am not aware of the connection here having anything like that. Another weird part is that I have also downloaded a dozen fics from tth and several dozen from ff.net without any problem. Will give a look at the other sites I frequent to see if any comes up. For the time being any suggestions?

Also tested an update and a new fic from hpfanfic and it worked correctly.

JimmXinu
01-13-2014, 09:17 PM
Version 1.8.07 - 13 Jan 2014


Change default encoding for finestories.com.
Change default encoding and improve metadata for storiesonline.net, thanks davidfor.
Fixes for hpfanficarchive.com changes.
Fix for 'update' with no updates and 'mark' enabled.
Now with French translation, thanks Ptitprince.

citac
01-16-2014, 06:15 AM
It downloaded fine for me.

Make sure you're using the most recent version of the plugin.

If it still doesn't work, it may be ffnet blocking you. They've taken to doing that to people who do 'too many' requests to their servers too quickly. How many is 'too many' is unknown.

You can create a text custom column in calibre and at the bottom of the Custom Columns tab of the FFDL config, you can set FFDL to populate error messages there. This only works for updates/overwrites of existing stories, not new downloads--there's no book to write the error too for them.

Thank you, for the suggestion - it seems that it is ffnet blocking me, since I had updated both calibre and the plugin before trying to download the fic. I'll try it again now that I have some more time on my hands. And thanks for the clarification of the error column, I'll do that right now!

Amalthia
01-17-2014, 03:13 AM
I was looking at the default settings in the personal.ini tab and I have "margin: 1%%" as my margin setting, but I have no clue what the percentages mean.

This is what is pasted in the personal.ini file as a guide for users but I'm not sure why the margin needs to escape and I've seen the percentages used in other areas??
## Make sure to keep at least one space
## at the start of each line and to escape % to %%

And to be clear margins aren't the same as padding in ebooks? I tend to get these two confused.

JimmXinu
01-17-2014, 12:29 PM
I was looking at the default settings in the personal.ini tab and I have "margin: 1%%" as my margin setting, but I have no clue what the percentages mean.

The defaults.ini file specifies for epubs:
body { ...
margin: 2%%;
...
}


When I first started working on FFDL, the original author included 5% margins all the way around. I later thought that was too much wasted space on my 1st generation Nook and reduced it to 2%. That was honestly the last time I really thought about how big they should be.

This is what is pasted in the personal.ini file as a guide for users but I'm not sure why the margin needs to escape and I've seen the percentages used in other areas??

The % sign is doubled because the ini file parser included in Python uses % to indicate special characters and doubling it escapes past that behavior, leaving one % in the output.

And to be clear margins aren't the same as padding in ebooks? I tend to get these two confused.

EPUB format books are made using a subset of (X)HTML and CSS. The difference between padding and margin is that padding is space inside the border (none in this case) and margin is outside. So for this case, there's no real difference.

W3schools has a good page about it (http://www.w3schools.com/css/css_boxmodel.asp).

Amalthia
01-17-2014, 03:54 PM
The defaults.ini file specifies for epubs:
body { ...
margin: 2%%;
...
}



The % sign is doubled because the ini file parser included in Python uses % to indicate special characters and doubling it escapes past that behavior, leaving one % in the output.

EPUB format books are made using a subset of (X)HTML and CSS. The difference between padding and margin is that padding is space inside the border (none in this case) and margin is outside. So for this case, there's no real difference.

W3schools has a good page about it (http://www.w3schools.com/css/css_boxmodel.asp).

Thank you for the explanation. It sounds like I may not need to use padding in my css code at all. I'll try adjusting my margins and remove the padding. I got a new ebook reader and it seems like my previous settings don't quite work the same way on the new device. I figured if I'm going to change the settings I'd like to do it right.

Thank you for the link! I'll read it and maybe print a copy. :)

Amalthia
01-17-2014, 07:35 PM
I had another question, does FFDL use any part of the Calibre conversion settings when it converts the ebooks? Like margins, font-size, css override, line height, and etc...

I ask because as I change my settings I want to make sure I don't confuse the two areas if they are separate.

JimmXinu
01-17-2014, 08:46 PM
I had another question, does FFDL use any part of the Calibre conversion settings when it converts the ebooks? Like margins, font-size, css override, line height, and etc...
No. Because FFDL also has CLI and web service versions, it uses it's own code to generate epub, mobi, html and text output.

However, I also acknowledge that FFDL's mobi output is marginal and recommend that Kindle users download to epubs and use calibre to convert to azw3 or mobi for their devices. It's better output and can use the epub update feature. That's what I do for my Kindle.

In that case, yes, the calibre conversion settings matter.

And in case you didn't notice it, calibre remembers conversion settings per book. So changing settings for one book will be used and remembered for that book, but not others.

Amalthia
01-17-2014, 09:17 PM
No. Because FFDL also has CLI and web service versions, it uses it's own code to generate epub, mobi, html and text output.


Thanks for the explanation. I use .ePub so my conversions so far have been great.

I tend to try not to change the conversion settings book-by-book but instead use the default conversion settings I've selected.

zraith
01-18-2014, 12:48 AM
Not sure if this is an error or not, but I have recently encountered problems with downloading new stories off fanfiction.net (tried it with several other sites, including fictionpress and no problems). The strange thing is that I can still update existing stories that I have had previously downloaded, the only issue is with new ones.

When trying to download a new story, the add-on gets to this part and just stays there instead of continuing:
Download FanFiction Book
FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(44):FanFictionDownLoader v1.8.7
FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(48):Adding jobs for URLs:
FFDL:INFO:calibre_plugins.fanfictiondownloader_plu gin.jobs(50):https://www.fanfiction.net/s/8601250/1/

I tried it with different length stories and still get the same problem, it doesn't matter if the story is 100k+ words over 40 chapters or a 1 chapter clip with <1k words.

JimmXinu
01-18-2014, 01:16 PM
...
When trying to download a new story, the add-on gets to this part and just stays there instead of continuing:

I tried it with different length stories and still get the same problem, it doesn't matter if the story is 100k+ words over 40 chapters or a 1 chapter clip with <1k words.

Evidence continues to mount that ffnet has instituted some sort of throttling system that significantly slows down some requests once they decide that you've made too many. We don't know what their criteria are, but it does seem to be slow downs rather than outright blocking.

The only advice I can give is don't do 'too many' downloads or update at once, or in a day. The value of 'too many' is unknown, but I'd guess it's lower than 100.

riothamus
01-19-2014, 08:56 PM
I'm trying to figure out how to get FFDL to populate a custom column. Specifically, I want to populate the "Collection" column for Kindle Collections to read. I tried doing it through Reading List, but without knowing how FFDL can call it, I've had no luck. Any help would be appreciated. Otherwise I love FFDL and it works great.

JimmXinu
01-20-2014, 12:52 AM
@riothamus - What information are you trying to get FFDL to put in the custom column?

If it's one of the standard metadatum collected for stories from most sites, just select it for that column on the Custom Columns tab of FFDL's config.

zraith
01-20-2014, 01:31 PM
Evidence continues to mount that ffnet has instituted some sort of throttling system that significantly slows down some requests once they decide that you've made too many. We don't know what their criteria are, but it does seem to be slow downs rather than outright blocking.

The only advice I can give is don't do 'too many' downloads or update at once, or in a day. The value of 'too many' is unknown, but I'd guess it's lower than 100.I am curious as to the criteria for that throttling system as there isn't any reasonable rhythm to the throttling. It appears to be on the number of requests but then why would it also prevent downloading new stories that only have 1 chapter each (just pondering out loud, not expecting anyone to be able to answer this).

As for the number of requests, I'm seeing it throttle when trying to update an existing story that I have that has had ~30 new chapters added (~300 words each) since the last time I've updated it. What is weird is that just after stopping this job, I tried updating a story that only had 1 new chapter added and had no problem with updating it.

The other weird thing is that I've tried 2 other different services for downloading stories, 1 is a web service and while it took a while was able to download (https://www.fanfiction.net/s/8601250/1/A-Dead-World) without any problems. The other service was one that went by a similar name, Fanfictiondownloader (found at fanfictiondownloader.net), and was a stand-alone program that required to be installed on the PC. This 2nd one I tried it with 2 different stories, the aforementioned one and another bigger one that is over 500k words spread over 73 chapters, no problem with downloading them (for now I guess).

This is a little telling that it might have to be with how the FanFictionDownloader plugin is retrieving/requesting the stories off FFnet that is causing the problem.

On a side note is there a way to increase the timer between sending requests?

riothamus
01-20-2014, 01:36 PM
@riothamus - What information are you trying to get FFDL to put in the custom column?

If it's one of the standard metadatum collected for stories from most sites, just select it for that column on the Custom Columns tab of FFDL's config.

I'm trying to populate a Collection column with the text "Fanfics" so they go to the Fanfics collection on the kindle. It's the only collection I have in this library.

JimmXinu
01-20-2014, 01:41 PM
...
The other weird thing is that I've tried 2 other different services for downloading stories, 1 is a web service and while it took a while was able to download (https://www.fanfiction.net/s/8601250/1/A-Dead-World) without any problems. The other service was one that went by a similar name, Fanfictiondownloader (found at fanfictiondownloader.net), and was a stand-alone program that required to be installed on the PC. This 2nd one I tried it with 2 different stories, the aforementioned one and another bigger one that is over 500k words spread over 73 chapters, no problem with downloading them (for now I guess).

fanfictiondownloader.net is unrelated to this project.

http://fanfictiondownloader.appspot.com/ is the web service version of this FFDL project.

This is a little telling that it might have to be with how the FanFictionDownloader plugin is retrieving/requesting the stories off FFnet that is causing the problem.

It's possible the User-Agent used is an issue, as well. That can be configured in personal.ini with user_agent.

On a side note is there a way to increase the timer between sending requests?

slow_down_sleep_time

JimmXinu
01-20-2014, 03:39 PM
I'm trying to populate a Collection column with the text "Fanfics" so they go to the Fanfics collection on the kindle. It's the only collection I have in this library.

It's possible, but it's enough of a corner case that I've never make it easy. But since you're the third(?) person to ask for it in the last two years, let's change that.

Attached is a version that allows literal strings to be set using custom_columns_settings.

You have to have "Allow custom_columns_settings from personal.ini to override" checked on the Custom Columns tab and in personal.ini, add:


custom_columns_settings:
"FanFics"=>#collection


The full spec is:

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.) There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'. 'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only. (Default is 'r'.)
## Literal strings can be set into custom columns using double quotes.
## Each metadata=>column mapping must be on a separate line and each
## needs to have one space at the start of each line.

#custom_columns_settings:
# cliches=>#acolumn
# themes=>#bcolumn,a
# timeline=>#ccolumn,n
# "FanFiction"=>#collection


If this works well, it will be in the next release.

UPDATE Jan 21, 2014 - Remove obsolete beta versions

riothamus
01-21-2014, 08:37 PM
It's possible, but it's enough of a corner case that I've never make it easy. But since you're the third(?) person to ask for it in the last two years, let's change that.

Attached is a version that allows literal strings to be set using custom_columns_settings.

You have to have "Allow custom_columns_settings from personal.ini to override" checked on the Custom Columns tab and in personal.ini, add:


custom_columns_settings:
"FanFics"=>#collection


If this works well, it will be in the next release.

I tried this in several ways last night with no luck. I tried the following custom column types: Text show in tag browser and Text fixed values. I do have the checkbox marked on the Custom Column tab and the code in the personal.ini tab. I also made sure to install the attached version and restarted caliber. The column just refuses to update with anything.

JimmXinu
01-21-2014, 09:44 PM
I tried this in several ways last night with no luck. I tried the following custom column types: Text show in tag browser and Text fixed values. I do have the checkbox marked on the Custom Column tab and the code in the personal.ini tab. I also made sure to install the attached version and restarted caliber. The column just refuses to update with anything.

Works fine for me with a "text show in tag browser" column.

What section of personal.ini did you add it to? [defaults]? Or a site specific section?

And you do understand that it won't be populated for a book until you download or update that book again?

riothamus
01-21-2014, 10:30 PM
Works fine for me with a "text show in tag browser" column.

What section of personal.ini did you add it to? [defaults]? Or a site specific section?

And you do understand that it won't be populated for a book until you download or update that book again?

I do indeed realize it only populates when I add or update via the FFDL plugin.

I first put it in the overrides section at the bottom. I then moved it to just above the overrides section which might have been considered a site specific section. I just moved it to after all the default and before the start of the site specific settings and yep, that fixed it. I didn't realize the site specific sections worked like that.

Thanks! That works great. I only wish I'd asked over a year ago when I first started using the FFDL plugin. This and the Reading List plugin are the best plugins for my use of calibre.

JimmXinu
01-21-2014, 10:46 PM
Version 1.8.08 - 21 Jan 2014

Allow literal strings in custom_columns_settings to set columns.

Firedancer885
01-22-2014, 08:37 AM
Version 1.8.08 - 21 Jan 2014

Allow literal strings in custom_columns_settings to set columns.


Jimm, I didn't see the alert for the new version in Calibre so I downloaded the zip you put into the first post of the thread. I got the following error when I tried to install it:

calibre, version 1.20.0
HIBA: Ismeretlen hiba történt: <b>SyntaxError</b>:invalid token (calibre_plugins.fanfictiondownloader_plugin.__ini t__, line 45)

calibre 1.20 isfrozen: True is64bit: False
Linux-3.11.0-15-generic-i686-with-debian-wheezy-sid Linux ('32bit', 'ELF')
('Linux', '3.11.0-15-generic', '#23-Ubuntu SMP Mon Dec 9 18:16:27 UTC 2013')
Python 2.7.5
Linux: ('debian', 'wheezy/sid', '')
Traceback (most recent call last):
File "site-packages/calibre/gui2/preferences/plugins.py", line 308, in add_plugin
File "site-packages/calibre/customize/ui.py", line 375, in add_plugin
File "site-packages/calibre/customize/ui.py", line 53, in load_plugin
File "site-packages/calibre/customize/zipplugin.py", line 195, in load
File "site-packages/calibre/customize/zipplugin.py", line 175, in load_module
File "calibre_plugins.fanfictiondownloader_plugin.__init __", line 45
version = (1, 8, 08)
^
SyntaxError: invalid token

JimmXinu
01-22-2014, 12:13 PM
Version 1.8.9 - 22 Jan 2014

Leading zeros in version number makes it octal. Remove and bump to 1.8.9. :smack:

JimmXinu
01-22-2014, 12:55 PM
Testing a fix for a typo in French translation.

UPDATE Jan 26, 2014 - Remove obsolete beta versions

Etorak
01-22-2014, 05:50 PM
Hi,

I'm using Calibre in French. The last two version of the FanFictionDownLoader plugin didn't work for me. During the download phase, some errors would occurs.
However, with the version posted today everything works fine now. Not sure what you were trying to fix, but looks good to me !

JimmXinu
01-22-2014, 06:05 PM
I'm using Calibre in French. The last two version of the FanFictionDownLoader plugin didn't work for me. During the download phase, some errors would occurs.
However, with the version posted today everything works fine now. Not sure what you were trying to fix, but looks good to me !

Sorry about that. That issue was reported today (http://code.google.com/p/fanficdownloader/issues/detail?id=70)--it was a typo in the French translation file.

julianabr
01-22-2014, 09:24 PM
I got some different errors when trying to update to 1.8.9 on my Mac.

When I tried to load by file:

calibre, version 1.20.0
ERROR: Unhandled exception: <b>InvalidPlugin</b>:The plugin in u'/Users/kindle/Downloads/FanFictionDownLoaderPlugin-1.zip' is invalid. It does not contain a top-level __init__.py file

calibre 1.20 isfrozen: True is64bit: True
Darwin-13.0.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '13.0.0', 'Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64')
Python 2.7.5
OSX: ('10.9.1', ('', '', ''), 'x86_64')
Traceback (most recent call last):
File "site-packages/calibre/gui2/preferences/plugins.py", line 308, in add_plugin
File "site-packages/calibre/customize/ui.py", line 375, in add_plugin
File "site-packages/calibre/customize/ui.py", line 53, in load_plugin
File "site-packages/calibre/customize/zipplugin.py", line 188, in load
File "site-packages/calibre/customize/zipplugin.py", line 282, in _locate_code
InvalidPlugin: The plugin in u'/Users/kindle/Downloads/FanFictionDownLoaderPlugin-1.zip' is invalid. It does not contain a top-level __init__.py file


When I tried to use 'get new plugin' button:

calibre, version 1.20.0
ERROR: Install Plugin Failed: A problem occurred while installing this plugin. This plugin will now be uninstalled. Please post the error message in details below into the forum thread for this plugin and restart Calibre.

Traceback (most recent call last):
File "site-packages/calibre/gui2/dialogs/plugin_updater.py", line 711, in _install_clicked
File "site-packages/calibre/customize/ui.py", line 383, in add_plugin
OSError: [Errno 13] Permission denied: '/Users/kindle/Library/Preferences/calibre/plugins/FanFictionDownLoader.zip'

JimmXinu
01-22-2014, 10:21 PM
I got some different errors when trying to update to 1.8.9 on my Mac.
....


It installs fine for me on Windows. I don't have a Mac to test, but the error messages suggest that the plugin zip file didn't download correctly.

I suggest trying to install it again, maybe by hand as per the instructions under How do I install a plugin manually? (http://www.mobileread.com/forums/showthread.php?t=118680)

julianabr
01-23-2014, 01:26 AM
I did try installing manually, both the new version and my last working version. I also tried just copying the file directly into the Library folder where calibre plugins live. It copied, but did not activate.

I have a feeling it is the Mavericks update to OSX behaving badly about permissions to install things. Even though I'm signed in as an admin, Mac has always asked for the password again before installing things. It seems like mavericks is taking permissions to a stricter level, however usually it will offer the username/password if it needs permissions during an install. Now I'm kicking myself because I removed the old version and now I'm without anything. :(

If you want me to test anything on my macs, I would be happy to help. Your plugin is great and I'd love to contribute.

Do you know of any other tools ffnet hasn't shut down yet?

JimmXinu
01-23-2014, 12:16 PM
I did try installing manually, both the new version and my last working version. I also tried just copying the file directly into the Library folder where calibre plugins live. It copied, but did not activate.

...

Do you know of any other tools ffnet hasn't shut down yet?

I would check the permissions on the plugin zip file and change them to be readable if they're not:

OSError: [Errno 13] Permission denied: '/Users/kindle/Library/Preferences/calibre/plugins/FanFictionDownLoader.zip'

I'd also check the contents of that zip file.

As for other tools, FFDL also has a CLI version (http://fanficdownloader.googlecode.com/hg/fanficdownloader.zip) (requires Python) and a web service (http://fanfictiondownloader.appspot.com/).

JimmXinu
01-24-2014, 10:26 PM
I don't know if it's always behaved this way, or if it's changed along the way, but when if I wait and let books be automatically converted to azw3 when Reading List is syncing my Kindle, it sends the metadata after every book now instead of just once for all of them.

So I've added an auto-convert feature to FFDL that, when checked, will automatically call calibre's Convert on update/download. The output format will be the one select in calibre's Preferences->Behavior.

Give it a try, see what you think.

UPDATE Feb 7, 2014 - Remove obsolete beta versions

redscarlett
01-26-2014, 11:25 AM
Hi

Got a bit of a problem with anthology updates. It's cropped up a few times with a few different fics (I can't remember which unfortunately) in the past few months, but I eventually just re-downloaded the fic. Now, though, I just want to see if I can fix it!

The series that I'm struggling with is at url:http://archiveofourown.org/series/46314 and it downloaded fine into an anthology only a couple of weeks ago. I've updated it since the initial download too, and that had no problems.

I've got the latest versions of both calibre and FFDL, and the debug said:

calibre 1.21 isfrozen: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '7', '6.1.7601')
Python 2.7.5
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Starting up...
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(155):Plugin F
anFictionDownLoader macmenuhack file_path:C:\Users\Sarah\AppData\Roaming\calibre
\plugins\fanfictiondownloader_macmenuhack.txt
macmenuhack file_path:C:\Users\Sarah\AppData\Roaming\calibre\p lugins\fanfictiond
ownloader_macmenuhack.txt
Started up in 5.69 seconds with 3172 books
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(564):tdir:
C:\Users\Sarah\AppData\Local\Temp\calibre_sueomw\f fdl_anthology_cmxws8

Can anyone help please?

riothamus
01-26-2014, 11:44 AM
So I've added an auto-convert feature to FFDL

Give it a try, see what you think.

The auto-convert call seems to work fine. I turned it back off as I'm not sending to kindle right now. I haven't tested the metadata issue yet.

JimmXinu
01-26-2014, 11:47 AM
Got a bit of a problem with anthology updates. It's cropped up a few times with a few different fics (I can't remember which unfortunately) in the past few months, but I eventually just re-downloaded the fic. Now, though, I just want to see if I can fix it!

And what exactly is the problem that you've seen?

The series that I'm struggling with is at url:http://archiveofourown.org/series/46314 and it downloaded fine into an anthology only a couple of weeks ago. I've updated it since the initial download too, and that had no problems.

I've got the latest versions of both calibre and FFDL, and the debug said:
...


If it downloaded fine, and it updated fine, what is it you're struggling with? It downloaded fine for me.

There are no errors in the debug output you posted. Just normal informational debug info.

redscarlett
01-26-2014, 02:51 PM
And what exactly is the problem that you've seen?



If it downloaded fine, and it updated fine, what is it you're struggling with? It downloaded fine for me.

There are no errors in the debug output you posted. Just normal informational debug info.

Sorry, I thought I'd been clearer, but with hindsight, I wasn't. When I click for an update, an error springs up saying that it cannot update the anthology, because it isn't an FFDL anthology or there is no FFDL urls contained within.

I'm awful with computers, so the debug just goes over my head, but I just wanted to see if anything was given away by it as to why the anthology won't update.

JimmXinu
01-26-2014, 03:08 PM
Sorry, I thought I'd been clearer, but with hindsight, I wasn't. When I click for an update, an error springs up saying that it cannot update the anthology, because it isn't an FFDL anthology or there is no FFDL urls contained within.

Have you been converting, polishing, editing, tweaking, etc the anthology epubs? If you have, it probably disturbed the structure and metadata FFDL relies on to recognize an anthology.

If you have one of the epubs that exhibit the problem that I could look at, that would help.

redscarlett
01-26-2014, 03:31 PM
Have you been converting, polishing, editing, tweaking, etc the anthology epubs? If you have, it probably disturbed the structure and metadata FFDL relies on to recognize an anthology.

If you have one of the epubs that exhibit the problem that I could look at, that would help.

I haven't touched it since FFDL downloaded it and merged it into an anthology, except 'generate cover' added. I haven't edited or polished, converted, etc it at all.

The only one I know of that is behaving like this at the moment is from this url:http://archiveofourown.org/series/46314 but if you prefer I'll pm you a copy of the epub I have.

JimmXinu
01-26-2014, 05:31 PM
I'll email it you now.

The file names inside that epub have all been changed--the same way calibre's epub->epub convert changes them. file0001.xhtml becomes something like file0001_u12.xhtml.

I might have been able to adjust for the file renames, but the convert also throws away metadata files that FFDL (EpubMerge called by FFDL technically) needs to 'unmerge' the anthology epub before updating each book inside it.

Maybe you ran convert to embed the cover image, not realizing that it would change more than that--that used to be the recommended way to embed covers. Now, however, you can embed the cover image using either the Polish tool or the Modify Epub plugin. They don't break anthology update.

If you have an ORIGINAL_EPUB format in that book, you could restore it and then anthology update will probably work.

redscarlett
01-26-2014, 06:01 PM
The file names inside that epub have all been changed--the same way calibre's epub->epub convert changes them. file0001.xhtml becomes something like file0001_u12.xhtml.

I might have been able to adjust for the file renames, but the convert also throws away metadata files that FFDL (EpubMerge called by FFDL technically) needs to 'unmerge' the anthology epub before updating each book inside it.

Maybe you ran convert to embed the cover image, not realizing that it would change more than that--that used to be the recommended way to embed covers. Now, however, you can embed the cover image using either the Polish tool or the Modify Epub plugin. They don't break anthology update.

If you have an ORIGINAL_EPUB format in that book, you could restore it and then anthology update will probably work.

Yeah I do have the original epub - I'll do that, thank you!

edit: Ooops - it was the original epub I had for another fic :S never mind, I'll just re-download it. Thanks for the help!

TrulyInnovative
01-27-2014, 11:23 AM
I'm having trouble with a particular story on Storiesonline.net: http://storiesonline.net/s/69562

Portions of the text including the cast of characters and the protagonist's thoughts (the very beginning of chapter 1, for example) do not show up on the epub.

JimmXinu
01-27-2014, 12:26 PM
I'm having trouble with a particular story on Storiesonline.net: http://storiesonline.net/s/69562

Portions of the text including the cast of characters and the protagonist's thoughts (the very beginning of chapter 1, for example) do not show up on the epub.

davidfor wrote the adapter for that site, so I'd like to hear from him before doing a permanent solution.

I have attached a version that includes that content, but it might include other stuff that isn't wanted, too.

UPDATE Feb 7, 2014 - Remove obsolete beta versions

davidfor
01-28-2014, 09:26 AM
Looking at the code, I can't remember exactly why I did that. I think I was removing something in the header or footer, but I'm not sure what now. Removing is correct. But, I noticed something else in my testing.

To get rid if the headings at top if the story, I was just stripping the heading tags. But, some use subheadings through the story, and those were also being removed. I'll fix this and post an update.

Emmi
01-28-2014, 01:56 PM
Hi,
I just came by to let you know, that fanfiction.net appears to have broken down again. I get a 'NoneType' object has no attribute '__getitem__' error message when trying to update existing or adding new stories. Other sites seem to be uneffected.

Thank you for your continued work,
Emmi

JimmXinu
01-28-2014, 03:17 PM
Hi,
I just came by to let you know, that fanfiction.net appears to have broken down again.

It's working fine for me.

Do you have specific story URLs it's not working for?

Debug output (run from command line as calibre-debug -g) would also be helpful.

Emmi
01-28-2014, 03:34 PM
It happens with all stories I've tried so far.
As for the debuglog:
calibre Debug log
calibre 1.21 [64bit] isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.5
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Starting up...
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(155):Plugin FanFictionDownLoader macmenuhack file_path:C:\Users\Emmi\AppData\Roaming\calibre\pl ugins\fanfictiondownloader_macmenuhack.txt
Started up in 22.34 seconds with 764 books
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(722):FanFictionDownLoader v1.8.9
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(757):url:https://www.fanfiction.net/s/8968909/1/
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.adapter_fanfictionn et(88):URL: http://www.fanfiction.net/s/8968909/1/
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.adapter_fanfictionn et(121):=Trying newer chapter: https://www.fanfiction.net/s/8968909/2/
FFDL:WARNING:calibre_plugins.fanfictiondownloader_ plugin.fanficdownloader.adapters.base_adapter(182) :Caught an exception reading URL: https://www.fanfiction.net/s/8968909/2/ Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>.
FFDL:WARNING:calibre_plugins.fanfictiondownloader_ plugin.fanficdownloader.adapters.base_adapter(182) :Caught an exception reading URL: https://www.fanfiction.net/s/8968909/2/ Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>.
FFDL:WARNING:calibre_plugins.fanfictiondownloader_ plugin.fanficdownloader.adapters.base_adapter(182) :Caught an exception reading URL: https://www.fanfiction.net/s/8968909/2/ Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>.
FFDL:WARNING:calibre_plugins.fanfictiondownloader_ plugin.fanficdownloader.adapters.base_adapter(182) :Caught an exception reading URL: https://www.fanfiction.net/s/8968909/2/ Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>.
FFDL:ERROR:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(184):G iving up on https://www.fanfiction.net/s/8968909/2/
FFDL:ERROR:__init__.py(1182):<urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.base_adapter", line 179, in _fetchUrl
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.base_adapter", line 168, in _fetchUrlRaw
File "urllib2.py", line 404, in open
File "urllib2.py", line 422, in _open
File "urllib2.py", line 382, in _call_chain
File "urllib2.py", line 1222, in https_open
File "urllib2.py", line 1184, in do_open
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
FFDL:ERROR:calibre_plugins.fanfictiondownloader_pl ugin.dialogs(600):Exception: {u'comment': u"'NoneType' object has no attribute '__getitem__'", u'publisher': u'www.fanfiction.net', u'begin': None, u'good': False, u'end': None, u'author': [u'rightforlife'], u'url': u'https://www.fanfiction.net/s/8968909/1/', u'listorder': 0, u'author_sort': u'rightforlife', u'comments': u'', u'calibre_id': 872, u'added': False, u'title': u'A Wrinkle in Time'}:'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 590, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 823, in prep_download_loop
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.base_adapter", line 243, in getStoryMetadataOnly
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.adapter_fanfictionnet", line 132, in extractChapterUrlsAndMetadata
TypeError: 'NoneType' object has no attribute '__getitem__'

JimmXinu
01-28-2014, 04:01 PM
It happens with all stories I've tried so far.
As for the debuglog:
...
Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
...

That's an SSL error that indicates the response you're getting isn't even valid SSL, let alone HTTP or HTML.

Were you downloading/updating a large (more than 12) number of stories? Because we have seen ffnet throttling requests from users who make too many hits to their servers. (The value of 'too many' is not known.) But I thought they were doing it more gracefully.

Any other causes I try to speculate would only be guesses. In your place, I'd try making sure I could read the stories from a browser, rebooting, and trying from a different network.

Robin42069
02-01-2014, 03:01 AM
ok i've tried to download this story http://storiesonline.net/s/69347 and i get this message
Status
Title
Author
Comment
URL
Error
The Earl's Man
FantasyLover
list index out of range
http://storiesonline.net/s/69347


i've had no problem downloading anything else really but this one story. can someone explain why it wont work.

heres the debug log as well to help you to tell me if its my problem or just a bug of some sort

davidfor
02-01-2014, 03:43 AM
ok i've tried to download this story http://storiesonline.net/s/69347 and i get this message
Status
Title
Author
Comment
URL
Error
The Earl's Man
FantasyLover
list index out of range
http://storiesonline.net/s/69347


i've had no problem downloading anything else really but this one story. can someone explain why it wont work.

heres the debug log as well to help you to tell me if its my problem or just a bug of some sort

It is an error in Chapter 12 (http://storiesonline.net/s/69347:125063). There is an extra "</div>" just before the story text. If you open the page, you can see what is happening. The background colour for the story text and the margins are different to the other pages.

I'm not sure how to fix this. I'll have a think and see if I can come up with a solution. Of course, reporting it to the author or the site admin might get the page fixed so that the problem goes away :)

And for the problem reported earlier in the week, I have fix. But, I haven't had a chance to test it enough. Hopefully tonight.

Robin42069
02-01-2014, 08:15 PM
well damn i didnt think i would get a reply back so soon. and you figured it out quick as well. thanks for the explanation as well. i see what you mean on chapter 12 with the margins and such. i'll drop a email to the site manager to see if they will fix it not sure if they would see the need or not but doesn't hurt to try though. but thank you for the help.

davidfor
02-03-2014, 08:55 AM
davidfor wrote the adapter for that site, so I'd like to hear from him before doing a permanent solution.

I have attached a version that includes that content, but it might include other stuff that isn't wanted, too.

I have finally had a chance to finish this and test it. The page retrieval code has been rewritten. The problems it fixes include:

- Headings within the story where being stripped
- For multi-page stories, the second page and further were in a div inside the main story div. This worked but it looked wrong.
- Spans used for styles in the story were stripped along with the text they containted.
- Any br tags inside the story were also stripped.

This mainly meant some structure was being lost. The spans were probably the worst bit as if they were used, they tended to be used a lot.

I haven't made any attempt to fix the problem reported on the weekend where there was a "</div>" in the wrong place.

The finestories.com plugin probably should have the same changes, but it will be a few days before I can look at it.

JimmXinu
02-03-2014, 07:05 PM
Here's a test version with davidfor's changes included.

UPDATE Feb 7, 2014 - Remove obsolete beta versions

isubs111
02-04-2014, 01:11 PM
Hi,

Would it be possible to have a button made for your browser that would grab the current page address and start the download of fanfiction in Calibre without you having to go and copy and paste the address and add the download separately? The button could be saved onto your Favourites bar in Firefox for example. You could maybe also have all your options set and pre-saved so you won't have to go into Calibre to say 'yes' to add to library, etc.

Thanks & kind regards,

Nic

TrulyInnovative
02-04-2014, 04:00 PM
Here's a test version with davidfor's changes included.

Thanks again to both you and davidfor. Based on a very brief test, this seems to solve the problems I had. The only minor nitpick I have is that there are no colors on the epub, but I don't even know if it's possible to include them. Besides, I mostly read on a black & white reader so that's really not a big issue.

Robin42069
02-04-2014, 07:29 PM
well damn man. you fixed that one problem with the story margins and now it downloaded the story i wanted so thank you very much. great work!

JimmXinu
02-04-2014, 08:47 PM
Would it be possible to have a button made for your browser that would grab the current page address and start the download of fanfiction in Calibre without you having to go and copy and paste the address and add the download separately? The button could be saved onto your Favourites bar in Firefox for example. You could maybe also have all your options set and pre-saved so you won't have to go into Calibre to say 'yes' to add to library, etc.

Yeah, that would be nice, wouldn't it. But it's not that easy.

Calibre uses the GUI thread as the lock on the database. So there's no way (that I know of) to properly invoke a plugin in the GUI, from outside the GUI.

You could make a bookmarklet that used the FFDL web service version and save it to calibre's auto-add directory. Or use a browser add-on to call the FFDL CLI (or plugin CLI) to download and save it to calibre's auto-add directory. But that wouldn't help with updates and couldn't call the other plugins that FFDL integrates with.

davidfor
02-04-2014, 09:56 PM
Thanks again to both you and davidfor. Based on a very brief test, this seems to solve the problems I had. The only minor nitpick I have is that there are no colors on the epub, but I don't even know if it's possible to include them. Besides, I mostly read on a black & white reader so that's really not a big issue.

The colours are in the stylesheet. If you use the storiesonline stylesheet, it will show them. You can copy that into your personal.ini, but it then gets used for all sites.

davidfor
02-04-2014, 10:56 PM
well damn man. you fixed that one problem with the story margins and now it downloaded the story i wanted so thank you very much. great work!

Sorry, but it didn't. It just didn't show throw any errors. All the chapters that are in "Book III" are wrong. The first only has the second page. The others are blank. They all have the same error in the page source.

I haven't looked a how to handle this error yet.

JimmXinu
02-05-2014, 12:01 AM
Sorry, but it didn't. It just didn't show throw any errors. All the chapters that are in "Book III" are wrong. The first only has the second page. The others are blank. They all have the same error in the page source.

I haven't looked a how to handle this error yet.

Unless it's an error that's common on the site, I wouldn't feel any obligation to work too hard correcting for one author's bad html.

Robin42069
02-05-2014, 12:15 AM
i guess since it looked like it worked i didnt look into it more thoroughly.thanks for the heads up. i'm sure i would had figured it out later and wondered what the hell was wrong with it. lol

davidfor
02-05-2014, 01:23 AM
Unless it's an error that's common on the site, I wouldn't feel any obligation to work too hard correcting for one author's bad html.

I tend to agree. I haven't seen another error like it. I don't think it is the site code adding the </div> in the wrong place. It looks like the author has done a cut-and-paste from somewhere and included an extra line. Detecting that the chapter text is empty after stripping the site navigation stuff isn't hard. I think the best that would be to either write a message into the text that something went wrong or throw an exception. The first means that any good parts of the story are downloaded, the second means that nothing is downloaded but the user finds out immediately. I'm not sure which would be the preferred behaviour.

i guess since it looked like it worked i didnt look into it more thoroughly.thanks for the heads up. i'm sure i would had figured it out later and wondered what the hell was wrong with it. lol

Yes, when you got to the blank chapter!

Amalthia
02-05-2014, 01:24 AM
Hi,

I had two ebooks fail to download and I got these error messages:

Failed to initialize plugin: u'C:\\Users\\Amalthia\\AppData\\Roaming\\calibre\\ plugins\\MobiDeDRM.zip'
Failed to initialize plugin: u'C:\\Users\\Amalthia\\AppData\\Roaming\\calibre\\ plugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\Amalthia\\AppData\\Roaming\\calibre\\ plugins\\Goodreads Metadata.zip'

I tried to uninstall the listed plugins, however I could not find these plugins in my settings...I'm not sure why fanfiction.net is trying to initialize these plugins?

I'd appreciate any and all help.
Thanks,
Amalthia

french25
02-05-2014, 04:56 PM
Hi,
I've used this plugins for months now without problem and since a couple of days, i've got the following message when trying to update stories from fanfiction.net :
calibre, version 1.22.0
ERREUR : Exception non gérée: <b>TypeError</b>:not enough arguments for format string

calibre 1.22 isfrozen: True is64bit: False
Windows-8-6.2.9200 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '8', '6.2.9200')
Python 2.7.5
Windows: ('8', '6.2.9200', '', 'Multiprocessor Free')
Traceback (most recent call last):
File "site-packages\calibre\gui2\__init__.py", line 393, in dispatch
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 1298, in download_list_completed
TypeError: not enough arguments for format string

JimmXinu
02-05-2014, 08:53 PM
Hi,
I've used this plugins for months now without problem and since a couple of days, i've got the following message when trying to update stories from fanfiction.net :
calibre, version 1.22.0
ERREUR : Exception non gérée: <b>TypeError</b>:not enough arguments for format string


There's an error in the French translation. There's a test version posted (http://www.mobileread.com/forums/showthread.php?p=2755142&postcount=2394) that fixes it.

(I should really release a new version--I was waiting for some other changes that have been delayed, but it's been a couple weeks now for this issue.)

JimmXinu
02-05-2014, 09:06 PM
Version 1.8.10 - 5 Feb 2014

Fix French translation typo that prevents download.
Add auto-convert on download/update feature.
Fix divide by zero error in replace_br_with_p on chapters with no content.
Change to storiesonline.net by davidfor to address overzealous culling in text.

BaconButty
02-06-2014, 05:49 AM
Hi there,

I came across an error message when trying to update 3 fics form The Writers' Coffee Shop (TWCS) today. The error message is the same for the 3 fics:

'utf8' codec can't decode byte 0xe9 in position 58: invalid continuation byte


http://www.thewriterscoffeeshop.com/library/viewstory.php?sid=9626
http://www.thewriterscoffeeshop.com/library/viewstory.php?sid=9429
http://www.thewriterscoffeeshop.com/library/viewstory.php?sid=9534

I have tried "updating if new chapters" (my default) and "Creating new book", without any success...

Thanks for your help with this problem... :)

ETA: I'm on V1.8.9 and I haven't been prompted to update yet...
Edit 2: the problem disappeared about a Calibre restart. Pls ignore :)

JimmXinu
02-06-2014, 05:01 PM
I had two ebooks fail to download and I got these error messages:
...
I tried to uninstall the listed plugins, however I could not find these plugins in my settings...I'm not sure why fanfiction.net is trying to initialize these plugins?

I don't think it's anything to do with FFDL directly. I think you have some outdated file input plugins.

I think it's a similar problem someone had a while back. See this posting (http://www.mobileread.com/forums/showthread.php?p=1915724#post1915724).

Since the error messages give you the path to the problem plugin zip files, I'd just go (re)move them.

Amalthia
02-07-2014, 03:07 AM
I don't think it's anything to do with FFDL directly. I think you have some outdated file input plugins.

I think it's a similar problem someone had a while back. See this posting (http://www.mobileread.com/forums/showthread.php?p=1915724#post1915724).

Since the error messages give you the path to the problem plugin zip files, I'd just go (re)move them.

Thanks for the link. I'll see what I can do to find and remove these plugins.

richreads
02-07-2014, 06:55 PM
Hi.
Is it possible to have FFDL call the Modify ePub plugin or Polish books after it finished downloaded like it does for the Generate Cover plugin?

I have the Generate Cover plugin run but that does not insert the cover into the epub so I then manually run Modify ePub.

Unless there is another method to ensure the generated cover gets inserted that i am missing :) .

ThanX

JimmXinu
02-07-2014, 11:39 PM
Is it possible to have FFDL call the Modify ePub plugin or Polish books after it finished downloaded like it does for the Generate Cover plugin?

At first I was going to say no. But then I remembered I already call Polish to do Smarten Punctuation. :)

Attached is a test version that adds an additional option on the Generate Cover page to call Polish to inject/update the cover inside the ebook, but only for EPUBs.

I'd also like to hear if somebody has better ideas for what the option and the tooltop for it could say.

UPDATE Feb 16, 2014 - Remove obsolete beta versions

uten
02-08-2014, 04:52 PM
Hi there,

I'm having problems with FanFiction.net. I have sorta solved it, but my solution makes no sense.

It was working fine a few days ago, but now, everytime I try to update or add a story, calibre hangs. I use a Mac, and I have to Force Quit calibre.

I also have tried to get a debug log, but it is empty. However, the terminal screen shows this:

FFDL: DEBUG:calibre_plugins.fanfictiondownloader_plugin. ffdl_plugin(155):Plugin FanFictionDownLoader macmenuhack file_path:/Users/uten/Library/Preferences/calibre/plugins/fanfictiondownloader_macmenuhack.txt
FFDL: DEBUG:calibre_plugins.fanfictiondownloader_plugin. ffdl_plugin(722):FanFictionDownLoader v1.8.10
2014-02-08 15:33:54.353 Calibre[8325:507] deliverNotificationWithTitle()
FFDL: DEBUG:calibre_plugins.fanfictiondownloader_plugin. ffdl_plugin(757):url:https://www.fanfiction.net/s/10075318/1/
FFDL: DEBUG:calibre_plugins.fanfictiondownloader_plugin. fanficdownloader.adapters.adapter_fanfictionnet(88 ):URL: http://www.fanfiction.net/s/10075318/1/
Terminated: 15

The website itself seems to work fine. I use AT&T DSL.

To fix it, I turned on my VPN. It now works fine. I found that out by accident.

Why was it crashing?? And why would just turning on a VPN fix that?

Thanks,
uten

P.S. After further testing, it seems to have only fixed it some of the time..

richreads
02-08-2014, 05:33 PM
At first I was going to say no. But then I remembered I already call Polish to do Smarten Punctuation. :)

Attached is a test version that adds an additional option on the Generate Cover page to call Polish to inject/update the cover inside the ebook, but only for EPUBs.

I'd also like to hear if somebody has better ideas for what the option and the tooltop for it could say.

Worked - :)
I think the tooltip and description is clear enough.
Thank-you for the amazing quick work and for the plugin.
:thanks:

JimmXinu
02-09-2014, 03:01 PM
I'm having problems with FanFiction.net. I have sorta solved it, but my solution makes no sense.

I've made an FAQ for this issue (https://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderFAQs#Why_am_I_having_errors_do wnloading_from_fanfiction.net). It says:

Why am I having errors downloading from fanfiction.net?

The site fanfiction.net has apparently implemented a throttling system that blocks requests from users who make too many hits to their servers. The value of 'too many' is not known, but appears to be around 12-15 downloads in short period of time.

Adding slow_down_sleep_time in your personal.ini for fanfiction.net can help keep you from being blocked at the expensive of slower downloads. We don't know anyway to get unblocked but to wait.

[fanfiction.net]
slow_down_sleep_time:2

Switching networks probably got you around the blocking, then you did a big stack of updates again and got throttled on the new network, too.

kneon
02-09-2014, 07:58 PM
Hi,

I have not been able to update or even download new from Fanfic.com for a few days. I am updated on all updates on Calibre

Thank you


Status
Title
Author
Comment
URL

Bad
Gardenia
Judy1998
<urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
https://www.fanfiction.net/s/9855608/1/

DawnB2
02-09-2014, 09:59 PM
I've been using FFDL for a few months. At random times when I use the Update if New Chapters it will download the newest chapters & for some unknown reason the existing chapters will become blank. I don't get an error & it happens entirely random. I have download chapters for different stories fine 4 times then the 5th will have this problem. Its not specific stories either. I can download a new chapter for 1 story fine, then the next time it will blank out old chapters, then the next time it will be fine again etc...

I only download from FanFiction.net & only 1 story at a time/per day.

Today I downloaded Chapter 7 for a story & when I opened the epub chapters 1-6 were now blank. I had to re-download the whole story again.

Is there anyway to fix this issue? I'd rather not have to download the whole story each time a chapter is added.

Firedancer885
02-10-2014, 09:17 AM
Attached is a test version that adds an additional option on the Generate Cover page to call Polish to inject/update the cover inside the ebook, but only for EPUBs.


Thank you very much :) I had to manually call polish books every time I downloaded new books and it always popped up the options window. This is much much better.

JimmXinu
02-10-2014, 11:59 AM
I've been using FFDL for a few months. At random times when I use the Update if New Chapters it will download the newest chapters & for some unknown reason the existing chapters will become blank. I don't get an error & it happens entirely random. I have download chapters for different stories fine 4 times then the 5th will have this problem. Its not specific stories either. I can download a new chapter for 1 story fine, then the next time it will blank out old chapters, then the next time it will be fine again etc...

I only download from FanFiction.net & only 1 story at a time/per day.

Today I downloaded Chapter 7 for a story & when I opened the epub chapters 1-6 were now blank. I had to re-download the whole story again.

Is there anyway to fix this issue? I'd rather not have to download the whole story each time a chapter is added.

That's rather odd. Are you doing anything to the epubs in between? Converting, Polishing, editing? Have you customized your CSS or HTML in FFDL's personal.ini?

There are a few things you could do to help figure this out.

Next time you go to do updates, run calibre from a command line as "calibre-debug -g", then after you see this problem again, copy the debug output from the command line window and post it here.

It would be helpful for me to see one of the epubs with blank chapters. Next time it happens, please save the epub and post it here or some where else I can see it.

It would be even more helpful for me to see the epub before the problem happens, too. But to do that, you'd have to save a copy before updating every time.

uten
02-10-2014, 01:06 PM
Having problems with getting fanfiction.net to use their 'covers'.

I added this to personal.ini

[www.fanfiction.net]
never_make_cover: false

What other settings need to be ticked/unticked? No matter what, I'm getting a cover that is just the title and summary text.

Thanks.

JimmXinu
02-10-2014, 03:31 PM
Having problems with getting fanfiction.net to use their 'covers'.

I added this to personal.ini

[www.fanfiction.net]
never_make_cover: false

Looks like that's broken right now. The attached version fixes it.

You also need to check "Include images in EPUBs" on the FFDL Basic tab or add the equivalent lines in personal.ini (see the tooltip).

UPDATE Feb 16, 2014 - Remove obsolete beta versions

uten
02-11-2014, 03:38 AM
Thanks for the fix. It works great now!

Eternal Link
02-12-2014, 03:16 PM
i can no longer download anything from fanfiction.net, i have the latest release and everything, could you please help me?


Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
<urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
https://www.fanfiction.net/s/6035532/1/

JimmXinu
02-12-2014, 03:32 PM
i can no longer download anything from fanfiction.net, i have the latest release and everything, could you please help me?


Lots of people have been asking that lately. I've made an FAQ for this issue (https://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderFAQs#Why_am_I_having_errors_do wnloading_from_fanfiction.net). It says:

Why am I having errors downloading from fanfiction.net?

The site fanfiction.net has apparently implemented a throttling system that blocks requests from users who make too many hits to their servers. The value of 'too many' is not known, but appears to be around 12-15 downloads in short period of time.

Adding slow_down_sleep_time in your personal.ini for fanfiction.net can help keep you from being blocked at the expensive of slower downloads. We don't know anyway to get unblocked but to wait.

[fanfiction.net]
slow_down_sleep_time:2

The next version is going to contain a longer value for slow_down_sleep_time for fanfiction.net by default.

Eternal Link
02-12-2014, 07:20 PM
im sorry to be posting again, but i tried your suggestion and it downloaded the metadata but after 50 minutes on the only book i wanted to download it failed again, this is the error,
Status
Title
Author
Comment
URL
Error
Souls Abound
robst
<urlopen error [Errno 110] Connection timed out>
https://www.fanfiction.net/s/6035532/1/
sorry if i am being a nusiance

JimmXinu
02-12-2014, 08:14 PM
im sorry to be posting again, but i tried your suggestion and it downloaded the metadata but after 50 minutes on the only book i wanted to download it failed again, this is the error,

This isn't a quick fix. Once ffnet is blocking you, it will take an unknown length of time (possibly days) before you are unblocked.

The additional sleep time is to keep you from getting blocked in the first place.

Eternal Link
02-13-2014, 08:41 PM
thanks for the info, ill wait a few days then try again, until then

tykobrian
02-15-2014, 11:19 AM
I'm sorry if this has been discussed before but how can I download the original book covers along with the story from the site fanfiction.net? So far only the epub downloads but no cover page. :help:

PS: I have the generate cover plugin installed

JimmXinu
02-15-2014, 11:36 AM
I'm sorry if this has been discussed before but how can I download the original book covers along with the story from the site fanfiction.net? So far only the epub downloads but no cover page. :help:

PS: I have the generate cover plugin installed

Well, if you'd read up the page a bit, you'd have found this posting (http://www.mobileread.com/forums/showthread.php?p=2760993&postcount=2421) that explains it.

Generate Cover doesn't matter for downloaded covers.

FYI, I plan to release a new version with that fix today if you don't want to figure out the manual plugin install.

tykobrian
02-15-2014, 11:56 AM
Well, if you'd read up the page a bit, you'd have found this posting (http://www.mobileread.com/forums/showthread.php?p=2760993&postcount=2421) that explains it.

Generate Cover doesn't matter for downloaded covers.

FYI, I plan to release a new version with that fix today if you don't want to figure out the manual plugin install.

Thanks for the help. I entered the lines in persona.ini and now a cover is provided with the ebook. But the thing is it's a custom cover, not the cover provided in the ffn site with the story. Please see the attached image.

Is it possible to download the original book cover which is provided in the site (ffn)using this plugin? I remember the site flagfic could attach the original cover back when they were allowed to download from ffn. Donno if they used this plugin though.

JimmXinu
02-15-2014, 11:56 AM
Version 1.8.11 - 15 Feb 2014

Fix for fanfiction.net cover download. This is turned off by default in CLI and plugin due to how many stories in ffnet show the author's picture. In section [www.fanfiction.net] add never_make_cover: false to turn covers on.

Add feature to automatically inject cover from Generate Cover into epub using calibre's Polish feature.

Don't do auto-convert when Updating Calibre Metadata Only.


This version doesn't have translations in German & French for the new cover feature yet.

firefoxxy
02-16-2014, 01:09 PM
Hi again,

I have difficulties to get the images from Fanfiction.com to become the cover of my epubs. They do appear on the inside of the epub just fine - just not as "outside"-bookcover. How do I extract them? Do you need any further information?

Your help is appreciated! Thanks.

tykobrian
02-16-2014, 01:22 PM
Hi again,

I have difficulties to get the images from Fanfiction.com to become the cover of my epubs. They do appear on the inside of the epub just fine - just not as "outside"-bookcover. How do I extract them? Do you need any further information?

Your help is appreciated! Thanks.

I'm facing the same problem. I've installed the latest version of the plugin.

JimmXinu
02-16-2014, 04:47 PM
@firefoxxy and @tykobrian,

It's working fine for me.

Make sure that you have the 'Update Calibre cover from ebook' option turned on.

JimmXinu
02-16-2014, 05:02 PM
Attached is a new test version with some fixes and changes to FiMFiction's site-specific metadata. Thanks FaceDeer for tracking these.

It also increases the string checked for for 'is_adult' due to a story comment containing the previous, shorter string.

UPDATE Feb 17, 2014 - Remove obsolete beta versions

DawnB2
02-16-2014, 08:31 PM
That's rather odd. Are you doing anything to the epubs in between? Converting, Polishing, editing? Have you customized your CSS or HTML in FFDL's personal.ini?

There are a few things you could do to help figure this out.

Next time you go to do updates, run calibre from a command line as "calibre-debug -g", then after you see this problem again, copy the debug output from the command line window and post it here.

It would be helpful for me to see one of the epubs with blank chapters. Next time it happens, please save the epub and post it here or some where else I can see it.

It would be even more helpful for me to see the epub before the problem happens, too. But to do that, you'd have to save a copy before updating every time.

I think I found this issue, if the ebub has been re-converted by calibre it changes the body tag from <body> to <body class="calibre">. When you use Update existing fanfiction book it adds the <body> tag back so now there are 2 open body tags but only 1 close tag

so it looks like this:

<body>
</head>
<body class="calibre">

I'm not sure why it adds the open body tag before the close head tag or why it only sometimes blanks out the old chapters.

I did confirm that before adding the new chapter there is only <body class="calibre"> & no extra body tag, after adding the new chapter all the existing chapters have the extra body tag.

I'm not sure if this is something you can fix in your plugin or if its a calibre itself issue.

I'm going through all Fanfiction books & changing all the <body class="calibre"> to <body>. I'll let you know if the issue happens again after all the body tags have been fixed/changed.

JimmXinu
02-17-2014, 01:40 AM
I think I found this issue, if the ebub has been re-converted by calibre it changes the body tag from <body> to <body class="calibre">. When you use Update existing fanfiction book it adds the <body> tag back so now there are 2 open body tags but only 1 close tag

Attached is a version that corrects that issue. I believe if you run an 'update always' on a story that has the problem, this fix will correct it, even if there aren't any new updates.

UPDATE Feb 17, 2014 - Remove obsolete beta versions

RenoA
02-17-2014, 03:52 AM
Did fanfiction.net change something again? It's been a while since I've tried to download a story and I just tried again, and now whenever I try to download or update a story from there, Calibre freezes when trying to fetch the metadata. I updated to the latest version of the plugin, but it's still having the problem.

I'm on a Mac (Mountain Lion) if that makes any difference. I also don't think it's because of FFN's throttling, since I only ever tried one story at a time.

JimmXinu
02-17-2014, 12:06 PM
Another test version really containing FaceDeer's changes.

@RenoA - I'm not seeing any problems like that. How long is it taking when it 'freezes'? It's not uncommon for it take a few minutes anymore.

UPDATE Feb 21, 201 4- Remove obsolete beta versions

RenoA
02-17-2014, 12:30 PM
Oh, wow. Okay, I thought it had been freezing because I got Apple's infamous spinning beach ball after just a couple of seconds when using it and it became unresponsive, but waiting for about a minute seems to prove that it does just take a while. Sorry about that, I guess I just didn't give it enough time.

JimmXinu
02-17-2014, 12:33 PM
Oh, wow. Okay, I thought it had been freezing because I got Apple's infamous spinning beach ball after just a couple of seconds when using it and it became unresponsive, but waiting for about a minute seems to prove that it does just take a while. Sorry about that, I guess I just didn't give it enough time.

Due to the number of people getting themselves throttled, the default sleep time for ffnet has been hiked up considerably.

People who limit their number of downloads themselves can get away with setting the sleep shorter in their setup. I've had no problems running with:
[www.fanfiction.net]
slow_down_sleep_time:1
... in my personal.ini.

FaceDeer
02-17-2014, 01:21 PM
Since this issue has been coming up a lot lately with regards to ff.net, might it be possible to add some sort of visual indicator to the GUI that there's a sleep timer running? FFDL would be saying "yes I'm running slow right now, that's deliberate" rather than "guess whether I've crashed." :)

paronomania
02-19-2014, 07:12 PM
Per instructions from Calibre upon failed install of plugin update:

calibre, version 1.24.0
ERROR: Install Plugin Failed: A problem occurred while installing this plugin. This plugin will now be uninstalled. Please post the error message in details below into the forum thread for this plugin and restart Calibre.

Traceback (most recent call last):
File "site-packages\calibre\gui2\dialogs\plugin_updater.py", line 677, in _install_clicked
File "site-packages\calibre\customize\ui.py", line 361, in add_plugin
File "site-packages\calibre\customize\ui.py", line 53, in load_plugin
File "site-packages\calibre\customize\zipplugin.py", line 183, in load
InvalidPlugin: The plugin at C:\Users\Leisa\AppData\Local\Temp\calibre_cuzlpq\v tggby.zip needs a version of calibre >= 1.13.0

JimmXinu
02-19-2014, 10:12 PM
Per instructions from Calibre upon failed install of plugin update:

calibre, version 1.24.0
ERROR: Install Plugin Failed: A problem occurred while installing this plugin. This plugin will now be uninstalled. Please post the error message in details below into the forum thread for this plugin and restart Calibre.

Traceback (most recent call last):
File "site-packages\calibre\gui2\dialogs\plugin_updater.py", line 677, in _install_clicked
File "site-packages\calibre\customize\ui.py", line 361, in add_plugin
File "site-packages\calibre\customize\ui.py", line 53, in load_plugin
File "site-packages\calibre\customize\zipplugin.py", line 183, in load
InvalidPlugin: The plugin at C:\Users\Leisa\AppData\Local\Temp\calibre_cuzlpq\v tggby.zip needs a version of calibre >= 1.13.0
It installs fine on 1.24 for me. I think your calibre is confused somehow.

martinjh99
02-20-2014, 05:50 AM
Did fanfiction.net change something again? It's been a while since I've tried to download a story and I just tried again, and now whenever I try to download or update a story from there, Calibre freezes when trying to fetch the metadata. I updated to the latest version of the plugin, but it's still having the problem

Same problem here with the latest calibre and plugin versions... It just hangs after fetching the metadata with the Jobs icon going round and round...

firefoxxy
02-20-2014, 05:20 PM
@firefoxxy and @tykobrian,

It's working fine for me.

Make sure that you have the 'Update Calibre cover from ebook' option turned on.

Hi again,

thank you for your quick reply. I still only get the covers to show up inside of the ebook.

Has this perhaps something to do with the options on the generate cover tab? There are three checkboxes and I have none of them activated.

Should I only choose the option for the cover in the personal.ini OR in the general tab (both are set to allow covers)?

Thanks again in advance for any hints.

JimmXinu
02-21-2014, 09:09 PM
When setting bool custom columns, look for t/true/yes/y/1 and f/false/no/n/0 in any case and set true and false accordingly.

UPDATE Feb 24, 2014 - Remove obsolete beta versions

JimmXinu
02-22-2014, 11:00 PM
This test version is a bit more 'test' than usual.

It implements a feature to count how many ffnet stories you're trying to download and adjusts the sleep time between requests higher for more and lower for less from a couple seconds with just 1 to a max of 5 seconds with 10 or more.

The default sleep time for the background processing remains 5 seconds between requests.

This feature is about reducing the 'Fetching metadata' step for people doing small numbers of downloads.

UPDATE Feb 24, 2014 - Remove obsolete beta versions

FaceDeer
02-23-2014, 11:50 AM
Neat, I'll try this out. But I've set my ff.net sleep time to 30 seconds by default anyway, since I don't generally care how long it takes to download stuff - I set it running when I'm about to go do something away from the computer, and it's done when I get back. Will this override that?

JimmXinu
02-23-2014, 12:03 PM
Yes, for the initial meta data collection. Not for the background processing.

This is a first stab at it, if it works it will be smarter in future.

JimmXinu
02-23-2014, 05:46 PM
Attached is another test version where the reduction in metadata fetch sleep is configurable. Keep in mind that this applies only to fanfiction.net downloads.

[www.fanfiction.net]

## fanfiction.net is blocking people more aggressively. If you
## download fewer stories less often you can likely get by with
## reducing this sleep.
slow_down_sleep_time:4

## ffnet is sensitive to too many hits. Users are sensitive to long
## waits during the initial metadata collection in the foreground.
## When used, these settings will speed up metadata downloads in the
## foreground linearly.
tweak_fg_sleep:true
min_fg_sleep:1.0
max_fg_sleep:4.0
max_fg_sleep_at_downloads: 10


UPDATE Feb 24, 2014 - Remove obsolete beta versions

JimmXinu
02-24-2014, 11:24 PM
Version 1.8.12 - 24 Feb 2014

Fixes for FimFiction change, thanks FaceDeer.

Set Boolean custom columns from custom_columns_settings, looking for true/t/yes/y/1 vs false/f/no/n/0.

Add a feature to reduce sleep between ffnet requests in initial metadata collection for small download sets.

Make ffnet look-for-next chapter feature optional and default to off.

jesterod
02-25-2014, 01:49 AM
i just updated and i get an error when trying to update
calibre, version 1.25.0
ERROR: Unhandled exception: <b>KeyError</b>:u'site'

calibre 1.25 Portable isfrozen: True is64bit: False
Windows-XP-5.1.2600-SP3 Windows ('32bit', 'WindowsPE')
('Windows', 'XP', '5.1.2600')
Python 2.7.5
Windows: ('XP', '5.1.2600', 'SP3', 'Uniprocessor Free')
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 607, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 614, in do_when_finished
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 699, in update_dialog_finish
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 722, in prep_downloads
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 722, in <lambda>
KeyError: u'site'

JimmXinu
02-25-2014, 01:54 AM
i just updated and i get an error when trying to update

What were you trying to update?

jesterod
02-25-2014, 01:55 AM
i ment i updated the plugin and when i tried to update the stories with it i got that error

JimmXinu
02-25-2014, 01:58 AM
i ment i updated the plugin and when i tried to update the stories with it i got that error

What story were you updating that gave the error? If I don't know how the error happened, it's hard to fix.

jesterod
02-25-2014, 01:59 AM
317 different stories.... my in-progress list

JimmXinu
02-25-2014, 02:02 AM
317 different stories.... my in-progress list

And people wonder why ffnet has started throttling.

jesterod
02-25-2014, 02:03 AM
there not all on ff.net alot are on other sites and the addon worked before the update

jesterod
02-25-2014, 02:13 AM
i figured it out there's 2 stories in there that had no url and for some reason that messed it up when it didn't before

FicReader81
02-25-2014, 08:34 AM
i just updated and i get an error when trying to update

Hi. I am getting a similar error message while trying to update stories (The number doesn't matter nor which stories I try to update. Could be 5, could be 50.) after I updated the plugin on calibre today. Can someone please suggest a solution.

JimmXinu
02-25-2014, 11:01 AM
Hi. I am getting a similar error message while trying to update stories (The number doesn't matter nor which stories I try to update. Could be 5, could be 50.) after I updated the plugin on calibre today. Can someone please suggest a solution.

Don't try to update books that don't have story URLs.

When you are looking at the update list, remove the ones that say 'No story url found' before hitting OK.

fatalerror328
02-25-2014, 11:08 AM
I was interested if there are any plans on improving mobi generating? Like TOC and fixing unreadable parts? If not, then maybe I can try to help? Is there any prefered method to help you with development?

FicReader81
02-25-2014, 03:39 PM
Don't try to update books that don't have story URLs.

When you are looking at the update list, remove the ones that say 'No story url found' before hitting OK.

Thank you for the reply and the solution. It worked. I had quite a few stories that had no URLs or those that had been deleted on FF.net and once I got rid of those I stopped receiving the error.

However now their is a different issue. Whenever I try to update the stories (Again the no doesn't matter, I have tried it with groups numbering 5 to 30 of different stories from my collection of 400 odd), the plugin starts fetching metadata for the stories but it doesn't finish. Calibre ends up freezing somewhere during this step of the updating and when I try to retrieve it "Windows Explorer" gives me the error message that Calibre will now be closing. Calibre hasn't been able to successfully finish updating the stories I select (As I mentioned I have tried different size bunches from the collection) ever since I was able to get rid of the earlier error.

I have been using the plugin on Calibre for 4-5 months now and this is the first time I have experienced any issues. It seems these issues have surfaced right after I updated the plugin today.

I would appreciate if you could look into the problems and suggest a solution again. Would truly appreciate it.

JimmXinu
02-25-2014, 08:33 PM
I was interested if there are any plans on improving mobi generating? Like TOC and fixing unreadable parts? If not, then maybe I can try to help? Is there any prefered method to help you with development?

I don't have any plans to, no.

The Google code project we keep it in has a facility for people to clone the repository (not the whole project) for their own use.

I'd suggest starting with a clone and then once you've made some progress let me know and I'll review it for inclusion.

JimmXinu
02-25-2014, 08:37 PM
Attached is a test version with a potential fix for the KeyError: u'site' problem.

I'd appreciate it if a couple of the people who saw that problem would give it a try.

Thanks.

UPDATE Feb 28, 2014 -- Remove obsolete beta versions

JimmXinu
02-25-2014, 08:48 PM
However now their is a different issue. Whenever I try to update the stories (Again the no doesn't matter, I have tried it with groups numbering 5 to 30 of different stories from my collection of 400 odd), the plugin starts fetching metadata for the stories but it doesn't finish. Calibre ends up freezing somewhere during this step of the updating and when I try to retrieve it "Windows Explorer" gives me the error message that Calibre will now be closing. Calibre hasn't been able to successfully finish updating the stories I select (As I mentioned I have tried different size bunches from the collection) ever since I was able to get rid of the earlier error.

If you just leave it alone for several minutes does it eventually finish?

To prevent throttling on ffnet, the sleep time between requests was significantly increased version before the last. Long enough that calibre was entering the 'not responding' state according to windows.

The last version included code to reduce those sleep times for small update sets.

Quoting myself from an earlier post:

Checking for updates on hundreds* of stories at once causes excessive traffic on the source site. fanfiction.net in particular has been known to block people's IP address if they generate too much traffic.

Instead, I highly recommend that you 'follow' stories and use the site's email notifications to inform you about updates and then update those specific stories. I also recommend using Update EPUB, not Overwrite, unless you have reason to believe that a particular story has changes in previous chapters.
* The original post said thousands.

Throttling by ffnet is no longer a theory, but an established fact.

FicReader81
02-26-2014, 08:08 AM
Attached is a test version with a potential fix for the KeyError: u'site' problem.

I'd appreciate it if a couple of the people who saw that problem would give it a try.

Thanks.

Hi. I would like to help but I don't know what to do with the contents of the zip file. If you could explain, perhaps I may be able to help.

FicReader81
02-26-2014, 08:21 AM
If you just leave it alone for several minutes does it eventually finish?

To prevent throttling on ffnet, the sleep time between requests was significantly increased version before the last. Long enough that calibre was entering the 'not responding' state according to windows.

The last version included code to reduce those sleep times for small update sets.

Quoting myself from an earlier post:


* The original post said thousands.

Throttling by ffnet is no longer a theory, but an established fact.

It doesn't matter how long I allow the process to continue, it just doesn't finish. As an example I take a sample of 5 fics from FF.net in my library and try to update them. The process would start and the plugin would try to fetch the metadata for the stories. Sometimes it would get completed, but most of the times it would just get stuck at lets say 3rd out of the 5 stories (I would say out of 10 tries, 2 would succeed and 8 would freeze) and when I try to retrieve the calibre program, "Windows Explorer" would come up with the message that calibre is not responding and it would then close the program. Prior to this update I could update my entire library in one go, but after failing a couple of times yesterday, I tried to update in batches of 10-20 fics. But as I said, 8 out of 10 times, calibre would just freeze (No matter how long I allowed it) and then I would have to restart the program and start again.

I usually update my fics once in a week or 10 days and haven't encountered any problems prior to this update (Although the update of the entire library did take a long time), but trying to select 10 and going from there is a bit tedious, esp if you don't know whether the update would succeed or not. I have made the changes you suggested wrt choosing update instead of overwrite, but the issue is the process doesn't go beyond the stage of fetching metadata more often than not.

Hope this would be of some use to you. Thank you once again for the reply and hope you can provide a solution.

JimmXinu
02-26-2014, 10:23 AM
Hi. I would like to help but I don't know what to do with the contents of the zip file. If you could explain, perhaps I may be able to help.

See this "Introduction to plugins" posting (http://www.mobileread.com/forums/showthread.php?t=118680) and scroll down to the section titled "How do I install a plugin manually?"

JimmXinu
02-26-2014, 10:37 AM
It doesn't matter how long I allow the process to continue, it just doesn't finish. As an example I take a sample of 5 fics from FF.net in my library and try to update them.
...

You're probably on ffnet's throttle list. They've been blocking (errors) more than hanging lately, but they've done both in past.

Evidence from other users is that it generally takes a couple days for the throttle to expire. After that, you're generally okay again as long as you don't do too many updates at once. Personally, I define 'too many' as about 10.

AmberDragon
02-27-2014, 02:29 AM
Evidence from other users is that it generally takes a couple days for the throttle to expire. After that, you're generally okay again as long as you don't do too many updates at once. Personally, I define 'too many' as about 10.

Interesting. I got caught in the throttle problem a while back and was confused because I try and keep my updating number to three or less at time. I chalked it up to FFNet's general flakiness, but I've just checked through my FFNet anthologies and found one that has eight stories in it. Could an update of a large anthology trigger the throttle?

JimmXinu
02-27-2014, 11:31 AM
Interesting. I got caught in the throttle problem a while back and was confused because I try and keep my updating number to three or less at time. I chalked it up to FFNet's general flakiness, but I've just checked through my FFNet anthologies and found one that has eight stories in it. Could an update of a large anthology trigger the throttle?

Under the covers, updating an anthology is really an update of each individual story in the anthology. So yes, updating anthologies could if the sleep times aren't long enough.

Robin42069
02-28-2014, 02:17 AM
Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
Story does not exist: (http://www.fanfiction.net/s/5904185/1/)
https://www.fanfiction.net/s/5904185/1/


I've been seeing this since i tried updating or even adding new ones to my library. could this mean i'm on they're throttle list or something cuz its not the only story that says that and i've then checked on them and the story is there?

FicReader81
02-28-2014, 10:06 AM
See this "Introduction to plugins" posting (http://www.mobileread.com/forums/showthread.php?t=118680) and scroll down to the section titled "How do I install a plugin manually?"

Hi I installed the plugin as per the instructions 2 days ago. I have been updating and adding new fics as usual. Their doesn't seem to be any problems other than the program hanging as I described in my earlier posts. (Which you suggested could be from the FF.net servers throttling. Although for me this has happened for the first time after the recent update. I have always updated fics once a week or 10 days and didn't face this issue before.)

Is their anything specific you want me to check for or a test to run? If so, please let me know.

JimmXinu
02-28-2014, 11:10 AM
Story does not exist: (http://www.fanfiction.net/s/5904185/1/)
https://www.fanfiction.net/s/5904185/1/


I've been seeing this since i tried updating or even adding new ones to my library. could this mean i'm on they're throttle list or something cuz its not the only story that says that and i've then checked on them and the story is there?

Ffnet had an outage itself - I'm going to chalk these reports up to that unless it continues to happen now.

From http://blog.fictionpress.com/ :

28 Feb

Our network provider has fixed the upstream issue that caused the service outage earlier today. We are currently working around the clock to make sure there are no linger effects and are in working with the provider to make sure this will not happen again.

Upstream Network Outage
27 Feb

[Update: Resolved]

Our upstream network service provider is experiencing an outage at this moment. We will keep everyone updated as the issue is being worked on.

Tanjamuse
02-28-2014, 12:02 PM
Do you take requests for additional sites?

Deheart
02-28-2014, 12:27 PM
Ffnet had an outage itself - I'm going to chalk these reports up to that unless it continues to happen now.

From http://blog.fictionpress.com/ :

Just to let you know, I'm still getting the error

By the way, I pretty much only read stuff I download with this plugin anymore, thank you very much! :)

JimmXinu
02-28-2014, 01:10 PM
Do you take requests for additional sites?

We do take requests, but I confess we haven't been acting on them very much lately. You might check the supported sites page (http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderSupportedsites) to see if it's been suggested before.

JimmXinu
02-28-2014, 01:15 PM
Just to let you know, I'm still getting the error

You're getting 'The story does exist' errors?

It's a possibility that ffnet has again changed their throttling behavior.

If it still happens after a couple days of only doing small downloads, then we'll look at it more closely.

Tanjamuse
02-28-2014, 01:59 PM
I would love it if you could add Spuffyrealm.com

Robin42069
02-28-2014, 03:57 PM
ok i got no problem.you were right it was fanfiction.net and not the plugin. i assumed because i could get to the story on the site itself it had to be the plugin i'm glad i was wrong. other than that small problem that doesnt exist i love the plugin so great job everyone who works on it. and thank you for helping me out.

JimmXinu
02-28-2014, 04:32 PM
Testing fix for ffnet 'Story does not exist' problem.

Turns out that ffnet made a change that revealed a problem in FFDL.

While FFDL changed to record and allow https urls for ffnet, it wasn't actually using them. Ffnet has started throwing 404 errors on http (not https) urls.

This version seems to work, please let me know how it does for you.

UPDATE Feb 28, 2014 -- Remove obsolete beta versions

Deheart
02-28-2014, 04:46 PM
Edit: Nevermind, it worked for 4 stories and just now when I tried again it didn't, downloading update :)

I had just tried again and the older version now is working, but thank you for the fast update :)

BaconButty
02-28-2014, 05:06 PM
... the older version now is working...

Just to say the old version isn't working for me at the moment.
Will try the updated version and report back...

Update: new version uploaded and it works fine. Thank you! :)

JimmXinu
02-28-2014, 07:18 PM
Version 1.8.13 - 28 Feb 2014

Fix for https vs http on ffnet (Story does not exist) problem.

Fix for KeyError:'site' problem.


Guys--updating your old posts doesn't trigger new notification emails.

If you have new information, make a new post or the odds are good I'll never see it.

Tanjamuse
03-01-2014, 11:20 AM
I just updated the Plug-in to 1.8.13 and I got this error:

Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
'NoneType' object has no attribute '__getitem__'
https://www.fanfiction.net/s/4521560/1/

JimmXinu
03-01-2014, 11:26 AM
I just updated the Plug-in to 1.8.13 and I got this error:
...
'NoneType' object has no attribute '__getitem__'
https://www.fanfiction.net/s/4521560/1/

From ffnet's blog (http://blog.fictionpress.com/):
Network Upgrade Notice
28 Feb

Our upstream network provider will be performing critical upgrades during a window tonight between 9PM and tomorrow 9AM. During the window, there may be a chance of service disruption of 10-15 minutes. The upgrade will resolve and prevent the re-occurrence of yesterday’s service disruption.


They don't mention what timezone that is, but it could well be now.

Ffnet is currently throwing a new kind of error message I haven't seen from them before and FFDL isn't handling it gracefully.

Tanjamuse
03-01-2014, 11:28 AM
I'll continue trying, do you know if there's a limit for how many stories I can download a day?

JimmXinu
03-01-2014, 12:00 PM
I'll continue trying, do you know if there's a limit for how many stories I can download a day?

I don't know of any daily limit, but ffnet does have a throttling system of some kind that can block your access for a couple days if they think you're hitting their servers too much. 'Too much' seems to be around 12 downloads at a time.

Tanjamuse
03-01-2014, 12:02 PM
Okay.

Thanks, I don't think the server is online at the moment :( Nothing happens when I try to open a link.

julianabr
03-01-2014, 02:12 PM
A while ago you got a request to support this site:

http://meryton.com/aha/index.php?showforum=47
(this is the link directly to the forum that holds the completed fanfics)

But since it was password protected and a forum-based fanfic site, you declined. I was wondering if you would reconsider, based on the following information:

bugmenot has this common username/password combo
JAReader/loveausten

Yes, it is a forum, but authors create their own locked topics, where they post a chapter at a time. No one other than the author is allowed to post on this topic. There is a TON of Jane Austen fanfic there that is nowhere else. Currently, the only way to get it on my kindle in any kind of reasonably formatted way, is to copy and paste into word, and try and search and replace all the extra crap that the 'copy' lifted from the page.

Given this, would you take another look at supporting this site as well?

Thanks for such a great tool, here online and as the calibre plug in

juliana

FaceDeer
03-01-2014, 03:26 PM
Currently, the only way to get it on my kindle in any kind of reasonably formatted way, is to copy and paste into word, and try and search and replace all the extra crap that the 'copy' lifted from the page.


I can't speak toward FFDL's support for this site, but as a better workaround than copying and pasting perhaps you could take a look at the GrabMyBooks (http://www.grabmybooks.com/) plugin? It allows you to build up an ebook from one or more web pages by grabbing just certain elements of the page, and can remember the "rules" you set up for any given website so you can quickly grab similarly-formatted stuff. I've found it excellent for making ebooks out of any old random article I come across.

JimmXinu
03-01-2014, 05:08 PM
Test version with a fix for potionsandsnitches.net.

Just a reminder, there are instructions on how to manually install plugins in the plugins introduction post (http://www.mobileread.com/forums/showthread.php?t=118680).

UPDATE Mar 08, 2014 -- Remove obsolete beta versions

distaltaper
03-01-2014, 06:07 PM
Is there a way to configure FanfictionDownloader so not all <span> element style attributes get stripped out? Is there a way for me to preserve style information enclosed in a span element in the original html?

Enjoying the plugin so far. Very slick. :)

JimmXinu
03-01-2014, 06:15 PM
Is there a way to configure FanfictionDownloader so not all <span> element style attributes get stripped out? Is there a way for me to preserve style information enclosed in a span element in the original html?

Enjoying the plugin so far. Very slick. :)

Glad you like it.

In personal.ini you can set the keep_style_attr setting for all sites in the [defaults] section or for a particular site by adding a section for it.

## If set true, any style attributes on tags in the story HTML will be
## kept. Useful for keeping extra colors & formatting from original.
keep_style_attr: true

bianca.slate
03-01-2014, 09:59 PM
[QUOTE=JimmXinu;2776532]Test version with a fix for potionsandsnitches.net.


I used the command line install procedure. The test new story: No One Noticed (http://potionsandsnitches.net/fanfiction/viewstory.php?sid=3033) downloaded fine.

When I tried to check for new chapters to a previously downloaded story, A Travesty (http://www.potionsandsnitches.net/fanfiction/viewstory.php?sid=3030), the story file updated, but the number of chapters in the calibre preview pane says 5, and the story now has 11 chapters.

Is that a Calibre error, not updating the preview of the EPUB 1st page?

JimmXinu
03-01-2014, 11:34 PM
I used the command line install procedure. The test new story: No One Noticed (http://potionsandsnitches.net/fanfiction/viewstory.php?sid=3033) downloaded fine.

When I tried to check for new chapters to a previously downloaded story, A Travesty (http://www.potionsandsnitches.net/fanfiction/viewstory.php?sid=3030), the story file updated, but the number of chapters in the calibre preview pane says 5, and the story now has 11 chapters.

Is that a Calibre error, not updating the preview of the EPUB 1st page?

Does the title page inside the epub should show the correct chapter count? If it's only the cover image in calibre that's out of date, make sure that you have the option in FFDL config for updating the cover turned on. See attached image.

Linwe
03-06-2014, 04:11 PM
So I finally got around to updating FFDL (I was a couple versions behind >_<) and found this wonderful little addition "Automatically convert new/update books?" Yay!

My only problem now is that it only works on newly added books. Am I mistaken that it should also convert updated books? My basic page is below, should I change something to make it work better?

119928

:thanks:

JimmXinu
03-07-2014, 02:18 AM
So I finally got around to updating FFDL (I was a couple versions behind >_<) and found this wonderful little addition "Automatically convert new/update books?" Yay!

My only problem now is that it only works on newly added books. Am I mistaken that it should also convert updated books? My basic page is below, should I change something to make it work better?

Updates should also kick off the convert. They do for me...