[GUI Plugin] FanFictionDownLoader

[crimsonsnow]

Hello, is there any way to use FFDL if my ISP ban fanfiction.net? Do I need to set my own proxy server and set it into Calibre? I can open it in a browser using proxy extension/add-on but it doesn't extend to Calibre.

[aleyx]

Originally Posted by JimmXinu

I think you'll find the word kaiju pre-dates Pacific Rim by several decades. Pacific Rim is an homage to the entire kaiju movie genre.

Indeed ^^ I think I've first heard about "real" Kaiju in ~1987 (I was ten), when I first saw a bad VHS copy of 1954 Godzilla. That's when I understood that Bioman wasn't so great after all.

Originally Posted by BookJunkieLI

Everyone has a preference. I don't do daily checks for updates, weekly to every other week depending on when I remember.

Sure; my own preferences tend towards doing the least amount of work by myself. By spending a minute or two sorting through my fic notices maildir, then updating only the stories I want to read that day, I spare myself and the hosting sites much more pain than by bulk-updating, _then_ deciding which ones I'd want to read. To each their own, sure, but "think about the environment" also kind of apply here too, in a way.

I've though about adding some mail-parsing ability to FFDL, so I'd have to work even less, but JimmXinu (I think?) noted that giving an addon access to one's mail wasn't perhaps a very smart thing to do. I'm still trying to think about a way around that that wouldn't require an in-house mail server and/or a dedicated account.

Originally Posted by BookJunkieLI

Plus not all sites I have stories from have email notifications.

That's strange; usually any site mature enough (code-wise) to be handled by FFDL also has a feature set complete enough to include mail notifications... Maybe they chose RSS or something? Can you give some links?

[JimmXinu]

Originally Posted by crimsonsnow

Hello, is there any way to use FFDL if my ISP ban fanfiction.net? Do I need to set my own proxy server and set it into Calibre? I can open it in a browser using proxy extension/add-on but it doesn't extend to Calibre.

Really? That's surprising. I'd think about switching ISPs.

There's a
calibre FAQ about using proxies that might help. I've never tested it, so I don't know if proxy settings work with FFDL or not.

Another option is to use the FFDL web service instead of the calibre plugin: http://fanfictiondownloader.appspot.com

[ILB]

Originally Posted by JimmXinu

Here's a new test version that still has the Qt5 changes, but also adds a new 'Reject Silently' option. Look at the bottom right of the Basic config tab.

I'd also like to hear if anybody has better ideas for what text to use for the label and tooltip for that checkbox.

I like the "Reject Silently" option. I think it will be very useful.
Maybe naming it "Reject without confirmation"?

For the tooltip: "Always URLs on the Reject List without stopping and asking" - is it just me or is a "reject" missing after "Always"?

[JimmXinu]

Originally Posted by ILB

I like the "Reject Silently" option. I think it will be very useful.
Maybe naming it "Reject without confirmation"?

For the tooltip: "Always URLs on the Reject List without stopping and asking" - is it just me or is a "reject" missing after "Always"?

Good suggestions, thanks.

Now that FFDL has translations in other languages, I can't change the text it shows as casually as I used to.

[chrnno]

Any and all stories from all sites I ever fail on the get metadata part and give "bogus escape (end of line)" as comment. I don't recall doing anything so I presume it was the calibre update that did it, either way what I should do now?

[JimmXinu]

Originally Posted by chrnno

Any and all stories from all sites I ever fail on the get metadata part and give "bogus escape (end of line)" as comment. I don't recall doing anything so I presume it was the calibre update that did it, either way what I should do now?

I'm not seeing that at all.

You can try adding/updating a story while running calibre in debug mode (see the calibre Preferences dropdown for 'Restart in debug mode') and post it here if there's anything interesting in the debug log.

[crimsonsnow]

Originally Posted by JimmXinu

Really? That's surprising. I'd think about switching ISPs.

There's a
calibre FAQ about using proxies that might help. I've never tested it, so I don't know if proxy settings work with FFDL or not.

Another option is to use the FFDL web service instead of the calibre plugin: http://fanfictiondownloader.appspot.com

It was perfect for other purposes though, I think the benefit isn't worth much. I can also use FFDL at work, but not at home.

I will try for setting the proxy this weekend, maybe. Thanks for the suggestion for the web version. I forget about that! Can't bulk update though.

[chrnno]

Originally Posted by JimmXinu

I'm not seeing that at all.

You can try adding/updating a story while running calibre in debug mode (see the calibre Preferences dropdown for 'Restart in debug mode') and post it here if there's anything interesting in the debug log.

Here:

calibre Log de Debug
calibre 1.44 [64bit] isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.5
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Starting up...
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(156):Plugin FanFictionDownLoader macmenuhack file_path:C:\Users\Felipe\AppData\Roaming\calibre\ plugins\fanfictiondownloader_macmenuhack.txt
macmenuhack file_path:C:\Users\Felipe\AppData\Roaming\calibre\ plugins\fanfictiondownloader_macmenuhack.txt
Started up in 16.04 seconds with 2725 books
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(726):FanFictionDownLoader v1.8.26
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(761):url:https://www.fanfiction.net/s/4730390/1/
FFDL:ERROR:calibre_plugins.fanfictiondownloader_pl ugin.dialogs(600):Exception: {u'comment': u'bogus escape (end of line)', u'begin': None, u'good': False, u'end': None, u'author': [u'Unknown'], u'url': u'https://www.fanfiction.net/s/4730390/1/', u'listorder': 0, u'author_sort': [u'Unknown'], u'site': 'www.fanfiction.net', u'comments': u'', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:bogus escape (end of line)
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 590, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 821, in prep_download_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ util", line 42, in get_ffdl_adapter
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.__init__", line 187, in getAdapter
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.adapter_fanfictionnet", line 40, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 87, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.story", line 315, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.story", line 425, in setReplace
File "re.py", line 190, in compile
File "re.py", line 242, in _compile
error: bogus escape (end of line)
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(761):url:https://www.fanfiction.net/s/10085240/1/
FFDL:ERROR:calibre_plugins.fanfictiondownloader_pl ugin.dialogs(600):Exception: {u'comment': u'bogus escape (end of line)', u'begin': None, u'good': False, u'end': None, u'author': [u'Unknown'], u'url': u'https://www.fanfiction.net/s/10085240/1/', u'listorder': 1, u'author_sort': [u'Unknown'], u'site': 'www.fanfiction.net', u'comments': u'', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:bogus escape (end of line)
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 590, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 821, in prep_download_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ util", line 42, in get_ffdl_adapter
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.__init__", line 187, in getAdapter
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.adapter_fanfictionnet", line 40, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 87, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.story", line 315, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.story", line 425, in setReplace
File "re.py", line 190, in compile
File "re.py", line 242, in _compile
error: bogus escape (end of line)
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(761):url:https://www.fanfiction.net/s/7415078/1/
FFDL:ERROR:calibre_plugins.fanfictiondownloader_pl ugin.dialogs(600):Exception: {u'comment': u'bogus escape (end of line)', u'begin': None, u'good': False, u'end': None, u'author': [u'Unknown'], u'url': u'https://www.fanfiction.net/s/7415078/1/', u'listorder': 2, u'author_sort': [u'Unknown'], u'site': 'www.fanfiction.net', u'comments': u'', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:bogus escape (end of line)
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 590, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 821, in prep_download_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ util", line 42, in get_ffdl_adapter
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.__init__", line 187, in getAdapter
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.adapter_fanfictionnet", line 40, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 87, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.story", line 315, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.story", line 425, in setReplace
File "re.py", line 190, in compile
File "re.py", line 242, in _compile
error: bogus escape (end of line)

Hoping really hard it is something obvious... :(

[JimmXinu]

Originally Posted by chrnno

Here:

...
File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.story", line 315, in __init__
File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.story", line 425, in setReplace
File "re.py", line 190, in compile
File "re.py", line 242, in _compile
error: bogus escape (end of line)
...

There's a mistake in your replace_metadata in personal.ini. Probably something to do with a '\' character at the end of a line.

If you have trouble finding it, you can post your replace_metadata here. Make sure you look at all the sections that could be effecting it; [defaults] and [overrides] as well as site specific.

[chrnno]

That is odd I don't recall having made any alterations to it after I last updated... Damn way too young to have memory problems here...

Anyway thank you very much for the quick answer, it took a while since there were 6 extra '\' scattered and my replace has several hundred lines but now it is working again.

[JimmXinu]

Originally Posted by chrnno

That is odd I don't recall having made any alterations to it after I last updated... Damn way too young to have memory problems here...

Anyway thank you very much for the quick answer, it took a while since there were 6 extra '\' scattered and my replace has several hundred lines but now it is working again.

[MononWalker]

Hi there! Long time user, first time caller

There are a couple of suggestions that I have that I hope you would consider.

1) I noticed that when I run the plugin and tell it to not update the metadata, that it does not update any of the metadata including the date. Shouldn't you update that at least so that it shows the last date imported/packaged?

2) The reason I noticed this is because the Calibre default for author names that contain two or more names is to generate a "lastname, firstname" for the author sort. For example, "chem prof" gets an author sort of "prof, chem" and when my device sorts on author name I have to look for it down in the P's when I want to find it in the C's. So I run the plugin on this particular author (as well as others) with the "Update Metadata" unchecked, which in turn does not update the (packaged) date. Is there a way you can either override the generation of the author sort so that the space either becomes a "hard" space so that the author name and author sort are the same value? Or maybe replace it an underscore?

3) This one's kind of a long shot ... you report on the title page the number of chapters. Is there a metadata equivalent that can be updated so that I can see it in the Calibre grid?

Love all the work you have done ... this has been a major time (and life) saver. Thanks so much for all your effort!

[BookJunkieLI]

Originally Posted by MononWalker

3) This one's kind of a long shot ... you report on the title page the number of chapters. Is there a metadata equivalent that can be updated so that I can see it in the Calibre grid?

I can't comment on your first two point but this one is easy.

Create a custom column with column type as Integers. Doesn't need any special formatting. Then go to the Configure Plugin menu for FFDL, then the Custom Columns tab. Select your chapter column and select the Chapters listing.
Next time you update it will add the number of chapters.

[JimmXinu]

Originally Posted by MononWalker

1) I noticed that when I run the plugin and tell it to not update the metadata, that it does not update any of the metadata including the date. Shouldn't you update that at least so that it shows the last date imported/packaged?

I suggest that you look at calibre's built-in Modified(last_modified) column. That one is updated when you update a book in anyway, even with 'update metadata' unchecked. But I don't believe it's shown by default.

Other than that, I submit that the Date is metadata and you told FFDL not to update metadata.

Originally Posted by MononWalker

2) The reason I noticed this is because the Calibre default for author names that contain two or more names is to generate a "lastname, firstname" for the author sort. For example, "chem prof" gets an author sort of "prof, chem" and when my device sorts on author name I have to look for it down in the P's when I want to find it in the C's. So I run the plugin on this particular author (as well as others) with the "Update Metadata" unchecked, which in turn does not update the (packaged) date. Is there a way you can either override the generation of the author sort so that the space either becomes a "hard" space so that the author name and author sort are the same value? Or maybe replace it an underscore?

On the FFDL Config dialog, in the Basic tab, hover your mouse over the option 'Force Author into Author Sort?' and the tooltip will explain:
"If checked, the author(s) as given will be used for the Author Sort, too.
If not checked, calibre will apply it's built in algorithm which makes 'Bob Smith' sort as 'Smith, Bob', etc."

Calibre quietly uses 'Author Sort' lots of places that you might otherwise expect it to use 'Author'. Such as the author field when sending books to most devices. Try it, I bet that does what you want.

As for point 3, BookJunkieLI explained it perfectly well.

(The only custom columns I personally populate from FFDL are Updated date, Words and Chapters.)