[GUI Plugin] FanFictionDownLoader

[Robin42069]

thank you for the fix on the plugin i'm new to calibre so i was looking to find out what i did to mess it up. i thought i had killed my new golden goose. i find out fanfiction.net had. so thanks for making it work again cuz i hadnt a clue what was going on

[Amalthia]

Originally Posted by Feynor

Hmmm, I'll have to have a look at that option next time I run into a long story. I have seen it of course, but read the "if" literally.

Were you able to test out this option? Does it really only add new chapters to the story? I've been reading the "if" literally as well. If it had said, "Update Epub 'with' new chapters" that may have been less confusing.

[Firedancer885]

Originally Posted by Amalthia

Were you able to test out this option? Does it really only add new chapters to the story? I've been reading the "if" literally as well. If it had said, "Update Epub 'with' new chapters" that may have been less confusing.

It only adds new chapters, and keeps the existing ones. I had a problem with one of the stories I'm following, because of the author's update method. I always ended up with two identical last chapters when I used the "update if new chapters" method. I'm using "update if" as default and the others only if necessary.

[hevreka]

Originally Posted by Feynor

I know this is going to be asked. Which version of Calibre and FFDL are you using, and on which OS?

Calibre 1.16 and FFDL 1.8.5 on Windows 7 have no problems with that URL.

Sry that my Reply Took so LOOOONNNNNGG Been a Hectic Week as a busdriver.

Calibre 1.16 and FFDL 1.8.5 Windows 7 Ultimate before the updates Today

After Updates Calibre 1.17.0 and FFDL 1.8.6 same OS

[Feynor]

Originally Posted by hevreka

Sry that my Reply Took so LOOOONNNNNGG Been a Hectic Week as a busdriver.

Calibre 1.16 and FFDL 1.8.5 Windows 7 Ultimate before the updates Today

After Updates Calibre 1.17.0 and FFDL 1.8.6 same OS

I've been unable to reproduce the error you are mentioning in your original post.

Do you still get it?

If so, please repeat it while Calibre is in Debug mode (Click the Preferences drop down arrow and select "Restart in Debug Mode") and just update this one story, then close Calibre after you get the error.
Save the text file that comes up afterwards and attach it to a response.

[TrulyInnovative]

The storiesonline.net plugin seems to have some trouble with special characters. For example, the dash symbol (–) is replaced by – and é is replaced by é.

[Feynor]

Originally Posted by TrulyInnovative

The storiesonline.net plugin seems to have some trouble with special characters. For example, the dash symbol (–) is replaced by – and é is replaced by é.

Do you have a link? (To a specific story that is causing this)

Edit:
Just a note, it looks like the utf-8 encoding is not being handled correctly in the adapter for the site. The Site itself does seem to send the correct encoding header.

[JimmXinu]

Originally Posted by Feynor

Do you have a link? (To a specific story that is causing this)

Edit:
Just a note, it looks like the utf-8 encoding is not being handled correctly in the adapter for the site. The Site itself does seem to send the correct encoding header.

Sometimes individual stories will be uploaded with one encoding while the site assumes another. Without the story URL it's hard to say.

The encoding used by FFDL can be changed for each site in personal.ini:

[storiesonline.net]

website_encodings: utf8, Windows-1252

# or instead:

website_encodings: Windows-1252, utf8

[TrulyInnovative]

Originally Posted by Feynor

Do you have a link? (To a specific story that is causing this)

Here's an example in the second paragraph of Chapter 1: http://storiesonline.net/s/51245:56696

[Oilily Sepperl]

Good evening! At first I want to say a big Thank You to the maker of the ffdl plug-in for Calibre. It's fantastic and I love using it!
I have problems lately though and hope somebody can help me.
I run Calibre version 1.17.0 and the FFDL plug-in version 1.8.6.
I haven't downloaded or updated fics for a while and wanted to do that the last two days. But I have trouble downloading or updating more than 5 or 6 fics at once. One single fic works fine and 5 or 6 fics work fine too, but I tried to update for example 14 fics and it wouldn't complete the task. There wasn't any error message and Calibre didn't freeze, but it took such a long time and wouldn't process further.
Same thing with about 100 fics or more. Often it would complete until 97 % and then simply wouldn't process any further for an hour.
I felt this is unusual and ended the tasks and only did about 5 fics.
It's unnerving though, because I have a lot of fics in-progress which I like to update regularly.
Did anyone here have this problem already or does anybody know what I could do or where the problem lies? Thanks in advance!

[davidfor]

Originally Posted by TrulyInnovative

Here's an example in the second paragraph of Chapter 1: http://storiesonline.net/s/51245:56696

I can see the problem. Adding "website_encodings: utf8, Windows-1252" to the personal.ini fixes it. The page info in Firefox tells me it is "UTF-8"

Looking at the code, it has:

self.decode = ["Windows-1252",
                       "utf8"]

From a quick look, most of the adapters have them in the same order. The base adapter is the other way around. As the page reports UTF-8, either the order in the adapter should be changed, or it should be removed as there is no real need to override the parent class.

But, there is another error. The summary/comments has "A Story in the Bec - Lost in the Maze Universe" rather than the story description. That should be the series. And it is also missing the author notes and disclaimer that is on the index page of the story. I'll have a look to fix this, but it might not be until I get back home in a week or so. At the moment, I don't have a full time connection.

[cryzed]

Hey there, I've been using the fanficdownloader module as part of the scripts running on my Raspberry Pi for quite a while now. Recently I've started getting this error:

cryzed ~/fanfiction >>> python update.py
(30.12.2013) 00:38:50: + Checking story...
Traceback (most recent call last):
  File "update.py", line 205, in <module>
    main()
  File "update.py", line 160, in main
    if is_up_to_date(epub_path, adapter):
  File "update.py", line 74, in is_up_to_date
    remote_chapter_count = int(adapter.getStoryMetadataOnly().getMetadata('numChapters'))
  File "/home/cryzed/fanfiction/fanficdownloader/adapters/base_adapter.py", line 243, in getStoryMetadataOnly
    self.extractChapterUrlsAndMetadata()
  File "/home/cryzed/fanfiction/fanficdownloader/adapters/adapter_fanfictionnet.py", line 161, in extractChapterUrlsAndMetadata
    rating = a.string
AttributeError: 'NoneType' object has no attribute 'string'

I suppose this is a simple error in the fanfiction.net adapter module, due to recent changes of their site-layout. I run a script once daily to retrieve the latest release of the module, but it hasn't been updated for a bit apparently. Do you guys think it would be possible to once again update the downloadable module on the site?

[JimmXinu]

Originally Posted by cryzed

...
I suppose this is a simple error in the fanfiction.net adapter module, due to recent changes of their site-layout. I run a script once daily to retrieve the latest release of the module, but it hasn't been updated for a bit apparently. Do you guys think it would be possible to once again update the downloadable module on the site?

Google Code will not support creating new downloads starting January 15th, 2014. The last download on the 'download' page includes this notification, which was also published on the FFDL Google group:

NEW CLI DOWNLOAD URL

Google Code is blocking new downloads in the download area next month. This is a simple alternative, so let's see how it works out for everyone.

The newest version of the CLI will be available at this URL every release:

http://fanficdownloader.googlecode.c...downloader.zip

Back versions will be available by browsing the repository for that file:
http://code.google.com/p/fanficdownl...downloader.zip

[JimmXinu]

Originally Posted by davidfor

I can see the problem. Adding "website_encodings: utf8, Windows-1252" to the personal.ini fixes it. The page info in Firefox tells me it is "UTF-8"

Looking at the code, it has:

self.decode = ["Windows-1252",
                       "utf8"]

From a quick look, most of the adapters have them in the same order. The base adapter is the other way around. As the page reports UTF-8, either the order in the adapter should be changed, or it should be removed as there is no real need to override the parent class.

It's saddeningly common for sites to report one encoding while using the other more commonly. Or to use both on different stories.

I don't read storiesonline.net. If someone who does tells me that utf8 is correct more often than Windows-1252, I'm willing to change it.

Originally Posted by davidfor

But, there is another error. The summary/comments has "A Story in the Bec - Lost in the Maze Universe" rather than the story description. That should be the series. And it is also missing the author notes and disclaimer that is on the index page of the story. I'll have a look to fix this, but it might not be until I get back home in a week or so. At the moment, I don't have a full time connection.

I don't have the energy to look at this right now either. Maybe tomorrow.

[JimmXinu]

Originally Posted by Oilily Sepperl

...
But I have trouble downloading or updating more than 5 or 6 fics at once. One single fic works fine and 5 or 6 fics work fine too, but I tried to update for example 14 fics and it wouldn't complete the task. There wasn't any error message and Calibre didn't freeze, but it took such a long time and wouldn't process further.
Same thing with about 100 fics or more.
...

If you're trying to update that many stories at once, you're probably running into throttling on the server. Especially if they are fanfiction.net stories.

I've written before about why doing mass downloads/updates is a bad idea.