[GUI Plugin] FanFictionDownLoader

[JimmXinu]

Here's another test version that improves on the Email story url retrieval.

This version will save your email password if you want, or save it only for a session, or not save it at all.

I've also added some verbiage to explain the risk a bit more. I'm open to suggestions on wording.

[cryzed]

For the tech-savvy guys: In case you find yourself in the situation of suddenly wanting to follow all your stories on fanfiction.net, you can export the URLs via the FFDL icon in Calibre and then use this script. You need Python 3, the selenium module and the Mozilla Firefox browser (although with slight modifications other browsers are supported of course too). Then just call the script with a path to a file containing the exported list of URLs. Again, this is for the tech-savvy people, and I know this makes me seem like a douche, but I really don't want to be responsible for giving support for this throw-away script. Good luck!

@Jimm: I think it would be nice if you could extend the default action settings in the "GUI Options" menu of the configuration dialogue. I would like to be able to get FFDL to check the emails by simply clicking on it, instead of defaulting to the add- or update-dialogue.

EDIT: Also the current code will raise an UnicodeDecodeError if the message isn't using the UTF-8 encoding (since that is what Python in Calibre defaults to if you just call unicode, I think). You somehow need to figure out the encoding on a per-email basis and decode to Unicode accordingly. Maybe this helps, but I never really worked with imaplib in Python myself.

[mehetabelo]

Since cryzed remarked on it, I figured I'd note that I noticed the unicode error too, but it was on some really, really old fanfiction stuff that I had wanted to get a bulk list for, then clean it up and get the stories I had on my list but hadn't read in a really long time. Mostly it was annoying because when it caused the error everything that was being downloaded was 'lost' and I had to start again, and it had already marked anything before the error as read.

When it didn't work I just put it off, since there was no real need to support it for ff.net stuff that old. However, now that I think about it, there may be other site update emails that will cause the problem. If it matters, for fanfiction.net stuff is pre mid-Sept 2006 and was basically nailing errors on things like: ' or " or * along with some of the odder characters (IE: Æ) when they were in the text.

Edit:Also, Cryzed, thanks for the code. I'm going to figure it out and probably use it. At one point there was a simple html that you could insert the story ID into and use that as a direct link, as long as you were logged into ff.net. I know I have seen it mentioned before. I don't know what it is though. But it will be damn useful to be able to do it in bulk again when I get time to sit down for a while to use your code.

[cryzed]

I'm glad it's useful to someone besides me, and yes there's probably some easier way to do it but this is what came to mind first.

Again, using it should be relatively easy: Install Python 3 (make sure to enable "Add python.exe to search path"), execute "pip install selenium" on the command-line and then run the script with "python path-to-script.py path-to-urls.txt". A Firefox instance should open and automatically navigate to the fanfiction.net login page, prompting you to login. Afterwards simply press enter when the command-prompt is focused and it should start doing its thing while you can just minimize both, the Firefox instance and the command-prompt.

[JimmXinu]

FYI, I don't look at the thread except when I get notices about new posts. So when you edit an old post, odds are I'm not going to see it. Like happened this time until mehetabelo replied.

I need to see the unicode error, or preferably have an email that causes it, before I can do anything about it.

@Jimm: I think it would be nice if you could extend the default action settings in the "GUI Options" menu of the configuration dialogue. I would like to be able to get FFDL to check the emails by simply clicking on it, instead of defaulting to the add- or update-dialogue.

I use hot-keys for Add, Update, Config, Get from Email, etc. I suggest trying that.

[cryzed]

Thanks for the hotkey tip, I suppose that is what I will be doing. Regarding the traceback: I didn't have the error myself, but I am almost entirely sure it happens in line 218 in fanficdownloader.geturls, when calling get_urls_from_text and then finally in line 125 where you use unicode(). I remember seeing "UnicodeDecodeError 0x4e is not a valid continuation byte" or some such (I gave support to someone via TeamViewer) -- clear signs that the encoding didn't match (UTF-8). According to the docs get_payload returns:

[...] If some other encoding is used, or Content-Transfer-Encoding header is missing, or if the payload has bogus base64 data, the payload is returned as-is (undecoded).

So that might be what's happening. The e-mail that caused the error for my friend was an official (german) email by Google IIRC. But yeah, let's wait for more complete tracebacks (and possibly entire e-mails) for better debugging.

[Tanjamuse]

Can I create a custom column that picks up the kudos and hits from AO3?

Thanks

[JimmXinu]

Originally Posted by cryzed

...
I didn't have the error myself, but I am almost entirely sure it happens in line 218 in fanficdownloader.geturls, when calling get_urls_from_text and then finally in line 125 where you use unicode(). I remember seeing "UnicodeDecodeError 0x4e is not a valid continuation byte" or some such
...

That helps. I can put a try/except around the get_urls_from_text(part.get_payload(...)) code, at least.

Attached is a test version that adds that plus some other tweaks to Email fetch--mostly changes to verbiage.

[JimmXinu]

Originally Posted by Tanjamuse

Can I create a custom column that picks up the kudos and hits from AO3?

Yes. But they have to be separate columns and you have to set them using custom_columns_settings in personal.ini. AIR, you can't put two metadata into a single column.

[aehli]

Hi all, I don't know if it is due to the update or something else - I have recently downloaded several files and found when reading on my kindle (paperwhite) I am unable to continue to the next page (it doesn't allow me to press to next page). This has happened on 4 separate occasions.

- I tried pressing manually the next page number and it shows garbled text (and a lot of overlapping text), then it repeats the paragraph and I'm still unable to continue. (e.g. I'm stuck on page 199 so I manually go to page 200).
- Around 5 pages later it starts working as normal, however 5 pages of text are skipped. (Page 205 is normal)
- I've tried recreating the mobi and same thing happens
- I've tried creating epub then converting to mobi and it doesn't work
- the file seems to work fine on the computer (calibre).

I'm not sure if this is an issue with:
- my kindle
- the plugin
- the file itself

If anyone knows anything about this and a fix that would be great.
Thank you

EDIT: Found a similar situtation in past post so will try it.
EDIT 2: When I convert from epub to mobi, the converted file on kindle is just a blank white screen.
EDIT 3: When I convert from epub to AZW - success!

[Tanjamuse]

How do I get rid of the tag that says Fanfiction in every story?

I've tried looking through the settings but I can't find it.

Edit:

I've also added the following in my personal.ini but the tag still shows up.

extra_tags:

[chrnno]

Originally Posted by Tanjamuse

How do I get rid of the tag that says Fanfiction in every story?

I've tried looking through the settings but I can't find it.

Edit:

I've also added the following in my personal.ini but the tag still shows up.

There is no _ in the code so just correct it under defaults to 'extratags' and it should work.

[Tanjamuse]

I can still see the Fanfiction Tag.

Edit:

It works now

Do you by any chance know how to make an url clickable?

Like the one I have in the bottom for the story?

[cryzed]

I think this question came up a few times before, and it's currently not possible in Calibre IIRC.

[BookJunkieLI]

I'm trying to setup a custom column to download AO3's kudos into and I keep getting the following error message:

Bad
Al Hamdu li-llah (All praise is due to Allah)
fresne
invalid literal for float(): 15.0 kudos=>#kudos
http://archiveofourown.org/works/37820

I've tried it as an integer column and as a floating point number column but always the same error message. I've tried no number formatting and I've tried {0:,} and {0:f}. I looked at the python documentation page linked in Calibre's custom column setup page but honestly it made zero sense to me.

This is the line I added to my personal.ini under AO3:
kudos=>#kudos

Any idea what I'm doing wrong?