[GUI Plugin] FanFictionDownLoader

[vikarti]

Issues with fictionmania and literotica:
- some stories on fictionnamia.tv are 'Story with Images' like http://www.fictionmania.tv/stories/r...93490826853422 - FFDL plugin says 'no valid story URL entered' (
downloads for some literotica's stories (even ones which worked before) are getting stuck with 'Fetching metadata for stories'.
for some other it results in "FFDLEBUG:calibre_plugins.fanfictiondownloader_p lugin.ffdl_plugin(761):url:http://www.literotica.com/s/the-story-of-odilia-ch-03
DEBUG:calibre_plugins.fanfictiondownloader_plugin. ffdl_plugin:url:http://www.literotica.com/s/the-story-of-odilia-ch-03
FFDLEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.adapter_literotica( 86):first page URL: http://www.literotica.com/s/the-story-of-odilia-ch-03
DEBUG:calibre_plugins.fanfictiondownloader_plugin. fanficdownloader.adapters.adapter_literotica:first page URL: http://www.literotica.com/s/the-story-of-odilia-ch-03
FFDLEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.adapter_literotica( 135):last page number: 2
DEBUG:calibre_plugins.fanfictiondownloader_plugin. fanficdownloader.adapters.adapter_literotica:last page number: 2
FFDL:ERROR:calibre_plugins.fanfictiondownloader_pl ugin.dialogs(600):Exception: {u'comment': u"'NoneType' object has no attribute 'findAll'", u'begin': None, u'good': False, u'end': None, u'author': [u'Unknown'], u'url': u'http://www.literotica.com/s/the-story-of-odilia-ch-03', u'listorder': 24, u'author_sort': [u'Unknown'], u'site': 'www.literotica.com', u'comments': u'', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:'NoneType' object has no attribute 'findAll'
ERROR:calibre_plugins.fanfictiondownloader_plugin. dialogs:Exception: {u'comment': u"'NoneType' object has no attribute 'findAll'", u'begin': None, u'good': False, u'end': None, u'author': [u'Unknown'], u'url': u'http://www.literotica.com/s/the-story-of-odilia-ch-03', u'listorder': 24, u'author_sort': [u'Unknown'], u'site': 'www.literotica.com', u'comments': u'', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:'NoneType' object has no attribute 'findAll'
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 590, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 839, in prep_download_loop
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 258, in getStoryMetadataOnly
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.adapter_literotica", line 204, in extractChapterUrlsAndMetadata
AttributeError: 'NoneType' object has no attribute 'findAll'"

[JimmXinu]

Originally Posted by vikarti

Issues with fictionmania and literotica:
- some stories on fictionnamia.tv are 'Story with Images' like http://www.fictionmania.tv/stories/r...93490826853422
...
http://www.literotica.com/s/the-story-of-odilia-ch-03
...

The literotica issue is one that I thought I'd already fixed, but I guess not well enough. Attached is a version that addresses that issue.

I didn't write the adapter for fictionnamia.tv, but it looks like that type of story just isn't supported.

[xvector]

Downloading from fanfiction.net takes about 4 minutes per novel-length book! And about 2 minutes per book if I have a couple in the list.

Is this normal? Applications like the fanfiction downloader application take much less time (~1 second per chapter). Am I doing something wrong? Is it because this plugin for Calibre downloads more metadata, tags, etc? I noticed that the fanfiction dowloader application doesn't download nearly as much metadata (tags, etc) as this.

[JimmXinu]

Originally Posted by xvector

Downloading from fanfiction.net takes about 4 minutes per novel-length book! And about 2 minutes per book if I have a couple in the list.

Is this normal? ....

Pretty much.

ffnet will start blocking your IP address if you download too many chapters too fast. To avoid that, especially for the people who try to update their whole library at once, there are several sleep timers built-in to FFDL when downloading ffnet stories.

So rather than risking users being blocked for several days or a week for over doing it, FFDL runs slower for everyone.

[Jade Aislin]

I know I mentioned this problem before, but I'm still having issues when I try to update a story with a different URL. If my library already has the story, it does not change the identifier.

For example, my library already has the story:

https://www.fanfiction.net/s/9065703/1/

Then I try to download:

http://archiveofourown.org/works/706971

I get asked if I want to change the story url and I say yes. Almost all the metadata changes, including the author url. However, the identifier or story url does not change. So if I try to select the story and update, it tries to update from the wrong site.

Any idea why the story url isn't changing?

[JimmXinu]

Originally Posted by Jade Aislin

I know I mentioned this problem before, but I'm still having issues when I try to update a story with a different URL. If my library already has the story, it does not change the identifier.

For example, my library already has the story:

https://www.fanfiction.net/s/9065703/1/

Then I try to download:

http://archiveofourown.org/works/706971

I get asked if I want to change the story url and I say yes. Almost all the metadata changes, including the author url. However, the identifier or story url does not change. So if I try to select the story and update, it tries to update from the wrong site.

Any idea why the story url isn't changing?

I just tested it with those two exact stories. It continues to work fine for me. Some possibilities:

1. The URL isn't updated until it's finished downloading from the new source and you've clicked Yes to 'Proceed with updating your library'. Are you waiting until after that to look?
2. It's not going to update the URL if you have Ids set to 'New Only' on the Standard Columns tab.
3. It's not going to update if you have it set to not update calibre data after download..

[JimmXinu]

Attached is a test version that should work for both the existing 1.X calibre and the upcoming 2.X version calibre. 2.x will use PyQt5 which requires changes to virtually all plugins.

This version needs to be tested with the existing calibre.

Brave souls to test with with the beta 2.X version would also be appreciated. Links to the 2.X beta and discussion of the transition (for plugins) can be see at: Attn plugin developers: calibre moving to Qt 5.

[Jade Aislin]

Originally Posted by JimmXinu

I just tested it with those two exact stories. It continues to work fine for me. Some possibilities:

1. The URL isn't updated until it's finished downloading from the new source and you've clicked Yes to 'Proceed with updating your library'. Are you waiting until after that to look?
2. It's not going to update the URL if you have Ids set to 'New Only' on the Standard Columns tab.
3. It's not going to update if you have it set to not update calibre data after download..

1. Yes, I am. I wait until it's gone through the Count Pages plugin before I check.

3. I have all the update options checked, including update always on existing stories.

2. Ooops. This was my problem. I haven't been on the standard column in so long, I didn't remember I had some set.

Thanks for figuring out my problem. It's working fine now.

[xvector]

Originally Posted by JimmXinu

Pretty much.

ffnet will start blocking your IP address if you download too many chapters too fast. To avoid that, especially for the people who try to update their whole library at once, there are several sleep timers built-in to FFDL when downloading ffnet stories.

So rather than risking users being blocked for several days or a week for over doing it, FFDL runs slower for everyone.

Thanks.

Is it possible to download all books for a certain search result or tag selection on fanfiction.net?

For example, would it be possible to automatically download all ~300 books from https://www.fanfiction.net/book/Harr...100&c1=1&_c1=6 ?

[JimmXinu]

Originally Posted by xvector

Thanks.

Is it possible to download all books for a certain search result or tag selection on fanfiction.net?

For example, would it be possible to automatically download all ~300 books from https://www.fanfiction.net/book/Harr...100&c1=1&_c1=6 ?

No. There's a feature you can use to gather story URLs from a single page, but it doesn't walk through lists of pages for you. Mass downloads like that is what makes ffnet cranky.

[Tanjamuse]

Do you know if I can download a whole page of links (25) at a time from FF.net without getting "kicked out" by the website?

Thanks

[cryzed]

Yes you can, just use the default sleep settings (which you do as far as I know). The plugin was specifically changed to allow this, it might just take a while.

[Tanjamuse]

@Cryzed

Default sleep settings, where do I find those ?

[cryzed]

Tanja, if you didn't change your personal.ini much from last I've seen it you don't need to worry. The default sleep settings are contained in the default configuration -- you would have to specifically overwrite them in your personal.ini (which you didn't). It would be the entry "slow_down_sleep_time" in your "fanfiction.net" section, which I'm pretty sure isn't there.

[Tanjamuse]

Thanks so much ... That'll make it that much easier I've been downloading only a couple at a time, and it takes sooooo long