[GUI Plugin] FanFictionDownLoader

[cryzed]

I don't think it's their intention to work well with Calibre, and the RSS feeds used to work a lot better a few years ago, until they castrated them entirely during some update.

The email idea sounds great.

[aleyx]

Originally Posted by cryzed

I don't think it's their intention to work well with Calibre, and the RSS feeds used to work a lot better a few years ago, until they castrated them entirely during some update.

Yeah, that was really just wishful thinking...

Originally Posted by cryzed

The email idea sounds great.

And has the added benefit of working with basically every site out there.

People have been predicting the death of email since Myspace, then Facebook, then Twitter. Feh, I say. Feh. Email will bury them all, in time. ^_^

N.

[jklly12]

Need some Calibre help, please. I'm trying to download a fic from FictionPad (which I have done before without any problems). I'm now getting this error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

[JimmXinu]

Originally Posted by jklly12

Need some Calibre help, please. I'm trying to download a fic from FictionPad (which I have done before without any problems). I'm now getting this error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

SSL is failing to verify the site's certificate--which works fine for me.

So either something is wrong with your SSL connection, like a man-in-the-middle attack, or you don't have the appropriate certificates to compare to.

Calibre, and therefore FFDL, use your OS's base certificate store. So if you're getting certificate verify failed errors, it's likely that your OS hasn't been updated in a while and doesn't have the correct certs.

If your OS is already up to date, I don't know of anything else I could do to help you.

[jklly12]

Got you I will do that now thanks again

[JimmXinu]

Okay, attached is a new test version with code to fetch story URLs from your email.

First, you have to go into FFDL config on the new Email Settings tab and set your IMAP server, user and folder. You can also check whether you want FFDL to automatically make those emails read.

Your email account must allow IMAPS (IMAP with SSL). I've only tested with Gmail (a Google Apps account, actually).

The folder must already exist. Use INBOX for your... inbox, or the name of the label for a Gmail label.

Then there's a new menu option for "Get Story URLs to Download from Email". It works very nearly the same as "Get Story URLs to Download from Web Page". Errors are not handled terribly gracefully yet, nor is there any 'busy' indicator while it's working.

It asks for your email password the first time you use it each session. The password is discarded when you quit calibre or when you switch libraries, since different libraries have different settings.

WARNING There's very little data protect in Python--once you've given FFDL your password, any plugin or other part of calibre could read it if it knows to look for it.

[aleyx]

Originally Posted by JimmXinu

Okay, attached is a new test version with code to fetch story URLs from your email.

Of course it had to happen when I'm cut from Internet at home ;_;

N.

[mehetabelo]

The new email download portion seems to work well... after dealing with all the loops that Google forced me to go through to allow it to work. Enabling access, then going to the link of the error they gave.

In order to be a little safer on my end, I simply have all my fanfiction forwarding to an email account I use for some very specific things, and even then it was rarely used anyway, so it's one that I'm not worries about if something happens to it. Gmail make it ... difficult to get additional accounts anymore, but I have a dozen or so, and I'm not too worries about losing one if it comes down to it.

[tricklem]

Originally Posted by jklly12

Need some Calibre help, please. I'm trying to download a fic from FictionPad (which I have done before without any problems). I'm now getting this error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

Originally Posted by JimmXinu

SSL is failing to verify the site's certificate--which works fine for me.

So either something is wrong with your SSL connection, like a man-in-the-middle attack, or you don't have the appropriate certificates to compare to.

Calibre, and therefore FFDL, use your OS's base certificate store. So if you're getting certificate verify failed errors, it's likely that your OS hasn't been updated in a while and doesn't have the correct certs.

If your OS is already up to date, I don't know of anything else I could do to help you.

Originally Posted by jklly12

Got you I will do that now thanks again

Hey Jklly12 did this work for you? I updated my windows os but it didn't make any difference. I still get this error for fictionpad downloads. Does anyone know how I can manually update the certificate myself or how to bypass it?

[aleyx]

Originally Posted by JimmXinu

Okay, attached is a new test version with code to fetch story URLs from your email.

Testing with my local Postfix/Dovecot mail stack (so, mainstream IMAP protocol, no GMail surprises). Works very well.

One strange thing: I have a space in my folder name. For mail.select() to work, I had to put the folder's name inside quotes in the new preferences panel.

Originally Posted by JimmXinu

WARNING There's very little data protect in Python--once you've given FFDL your password, any plugin or other part of calibre could read it if it knows to look for it.

Maybe calling EmailPassDialog directly in get_urls_from_imap's parameters? Sure, the user would have to reenter his password each time they click the menu, but I don't think the function would be used at such a frequency that it would be a hassle.

I haven't tested it, mind.

Perhaps even ask the user: "You have tried to retrieve your mail three times in less than three minutes. Do you want me to remember your password until you restart Calibre? Y/N (Warning: this can be a security risk)". Or something like that.

N.

[cryzed]

I think reentering the password, even if I have to do it only once after starting Calibre, would qualify as a hassle for me. I'd prefer if I could specify my password somewhere permanently and FFDL would use it, similarly to how the login process for fanfiction sites is currently facilitated (which is even more insecure, and somehow no one is worried?).

Somehow I doubt that someone would tailor a plugin to specifically exploit FFDL's temporary, or even permanent, saving of a password. How is Calibre handling it in the "Sharing books by email" dialogue? Does it use some kind of system keychain or a simple (and obviously open-source) method of encryption for the password?

And although I haven't taken a closer look at the new code yet, executing EmailPassDialog in the parameters won't prevent passwords from being stolen if someone really wants to -- I could just replace the class entirely at runtime, or alternatively modify/decorate parts of it and grab the password from a different plugin (I did something similar a few months ago to forcefully overwrite the hardcoded sleep delay in FFDL for some sites).

[kovidgoyal]

There is no secure way to reversibly store a secret on disk against an attacker that can run local code.

1) Even if you encrypt the stored secret with x-doo-dah-super-dooper encryption, based on a passphrase from the user, all the attacker has to do is spoof the program and ask the user for the passphrase. Or use a privilege escalation exploit to gain root and read the password from process memory. Or run a dictionary/brute force attack on the passphrase. And really, if you are saving the secret to disk, then that means you dont want the user to enter the password every time, which means storing it in a keyring. And all keyrings I know of are completely insecure against attackers that can run code locally.

2) There is no way to secure a secret safely in RAM against an attacker that can either run code in the calibre process or run code as root in the machine.

3) All operating systems have many local privilege escalation expolits.

Basically, once an attacker can run arbitrary code on your machine, you are hosed.

So about the only case where you can (somewhat) hope to store a password securely is against an attacker that can read arbitrary files on your system, but not run arbitrary code. That is a very small subset of attackers, unless you are running afile server.

[JimmXinu]

Originally Posted by kovidgoyal

There is no secure way to reversibly store a secret on disk against an attacker that can run local code.
...

I agree. Which is why I've been reluctant to add this feature despite it's obvious utility.

However, I hadn't really thought about the fact calibre can already be holding onto your email user/pass for sending email, setting a precedent for it.

Right now I'm considering giving additional options and letting the user decide between:

1. Saving password in FFDL config (by library)
2. Entering password once per session/library switch (same as last test version)
3. Entering password everytime
4. Not using email url fetching

[FaceDeer]

Maybe also add a "these are the risks" notification screen that the user has to go through to get to the email settings. You could strongly advise people to create an email account solely for use by FFDL, that way when it gets hacked the worst that can happen is the attacker screwing with your fanfiction updates.

Of course, some people will click through it without reading and use their private super important email addresses anyway. But there's nothing you can do to prevent *everyone* from being dumb, and if you prevent enough of them you'll at least make it less worth a hacker's while to try this roundabout method of hackery.

[cryzed]

@kovidgoyal: Yeah, I feared that was the case, that's why I mentioned the encryption being open-source for example as well -- Firefox's and Chrome's password store (without a master password) can be easily decrypted too IIRC, although grabbing a potential master password there should pose more of a problem.

And I agree with FaceDeer, creating a separate, dedicated email account might be the safest solution.