[GUI Plugin] FanFictionDownLoader

[JimmXinu]

Originally Posted by ilovejedd

Question, is there a debug option available for FFDL to save the original source HTML files in a user-defined directory (e.g. "/debug/SiteAbbrev/StoryID") as well as intermediate files? If not, would it be possible to add this option?

To what end? What do you want to do with this?

[rlh3]

If this is not in the correct place, I apologize, but I was wondering if there had ever been any interest in supporting the sites www.storiesonline.net and www.Finestories.com?

[iatheia]

Originally Posted by rlh3

If this is not in the correct place, I apologize, but I was wondering if there had ever been any interest in supporting the sites www.storiesonline.net and www.Finestories.com?

I'd rather not. Wouldn't be impossible, but it isn't really fanfiction, and in the past we were asked to stop the support of one of the website that hosted purely original work.

Not to mention that both of them by the looks of it already give you an ability to download ebooks.

[rlh3]

They do have a means for downloading epubs, but I was hoping to automate my updates. (I have about 45 stories that I am following and would love to be able to get the updates quickly.)

Thanks for looking (and the really fast response!)

[iatheia]

How about that - if you ask the admins of those sites and they okay it, I'll consider it =)

[rlh3]

Email sent to the Admin of SOL, we will see what the response is.

[ilovejedd]

Originally Posted by JimmXinu

To what end? What do you want to do with this?

Debugging and educational purposes. I've wanted to learn Python for a while now but this is the first project that I'm interested in that actually appears to be within my reach (the site adapters, at least). Would like to create a site adapter for AFF (adultfanfiction.net). If it's not allowed to be included with FFDL, then I'd at least like to write it for my own use. Thanks!

[iatheia]

It's not that it isn't allowed, it is just it's too complicated to do - the site has really overdid itself on the legal warnings, and just a tad bit too ugly to attempt. Plus it appears that they are undergoing a rewrite. Not really ideal for the first adapter, but if you want to try it - go ahead.

The way you going around is is a tad bit ineffective, in my opinion. You will learn very little at staring at the result of someone else's complicated code. The best way to learn how to code is by doing So take some adapter (adapter_castlefansorg.py is as good of a way to start as any given its documentation) and start playing around with it - modifying a part there, a part here to fit certain purposes. Look at the html code of that example website, too, to see why it was coded like that. That's how I've learned, at least. AFF is not efiction of course, which makes a matter tad bit complicated, but at least it will show you what are all the components that go into it.

And if upon the compiling you generate an error, and upon staring at it for a couple of minutes (and checking the code at the line specified in the error) you cannot figure the solution - try googling it. If that still doesn't work - feel free to ask us, we'll help to the best of our abilities.

And while I'm at it, here is a helpful bit of advise before you do anything. Python fundamentally depends on the proper formatting of the code. Other languages may allow everything to be flushed to the edge, but python would be very confused and yell at you if that were the case. So do keep the same formatting as the code you're modifying - four spaces (not tabs!) per indent. This is something that gave me hell to figure out when I've just started.

[JimmXinu]

@ilovejedd - That would be a bit of a hefty feature to add just for playing with.

I think iatheia's advice is better. And I would add:

It's much easier to try different things and work on the code in command line than it is in the plugin. In command line, every edit you save is used immediately the next time you run it. In plugin, you have to create the updated plugin zip, install it and restart calibre for every single change.

There's a command line interface(CLI) version of FFDL available. If you don't want to install Python separately, after getting and unzipping the CLI version you can use "calibre-debug -e download.py" to run it using calibre's built-in python.

When debugging plugins, when I want to see what the HTML from the site looks like before any processing, I find the fetch call (typically "data = self._fetchUrl(url)") and then add a "print(data)" call. Consider that your first step in learning python.

Then run the CLI downloader and redirect it's output to a file. Something like this:

c:\Python27\python.exe downloader.py "http://test1.com?sid=4" > output

Editing downloader.py and changing where it says "logging.basicConfig(level=logging.INFO,..." to "logging.basicConfig(level=logging.DEBUG,..." is also useful.

When testing FFDL output or settings, you can use the dummy site test1.com with story URLs like test1.com?sid=123 That will generate output, but not waste time and bandwidth going out to a real site.

[JimmXinu]

Originally Posted by JimmXinu

Almost certainly has something to do with weird (or flat out wrong) HTML from the original author. It's going to be a couple days before I have time to look at it in depth.

As soon as I hit submit on that, I realized I was taunting Murphy.

It's the blockquote tags. We have two versions of HTML parser that we use for FFDL. One more featured and one more bare bones. We generally prefer to use the bare bones one--it makes fewer mistakes. But one of the things it doesn't do is blockquote tags.

I've changed the wraithbait.com adapter to use the more featured html parser and attached it to the earlier post.

Try it out and see if a) it fixes that story and b) if it breaks any others.

Jim

[ilovejedd]

Originally Posted by iatheia

It's not that it isn't allowed, it is just it's too complicated to do - the site has really overdid itself on the legal warnings, and just a tad bit too ugly to attempt. Plus it appears that they are undergoing a rewrite. Not really ideal for the first adapter, but if you want to try it - go ahead.

The way you going around is is a tad bit ineffective, in my opinion. You will learn very little at staring at the result of someone else's complicated code. The best way to learn how to code is by doing So take some adapter (adapter_castlefansorg.py is as good of a way to start as any given its documentation) and start playing around with it - modifying a part there, a part here to fit certain purposes. Look at the html code of that example website, too, to see why it was coded like that. That's how I've learned, at least. AFF is not efiction of course, which makes a matter tad bit complicated, but at least it will show you what are all the components that go into it.

And if upon the compiling you generate an error, and upon staring at it for a couple of minutes (and checking the code at the line specified in the error) you cannot figure the solution - try googling it. If that still doesn't work - feel free to ask us, we'll help to the best of our abilities.

I've been able to create an AFF adapter before for FLAG (PHP). It's possible to ignore the redirect to the warning page (at least using curl by setting CURLOPT_FOLLOWLOCATION to false). I think I'll try to do the proper thing with the birthday/signature stuff (as a user-configurable option) but if that fails, I can try to find the python equivalent to the above php+curl option.

I've been viewing a couple of FFDL adapters and dokuga.com seems to be the closest match. As for being ineffective, well it works for me. Viewing code examples and seeing what they do is generally how I learn a language. Needless to say, Calibre's code is too complicated for me but the FFDL code doesn't appear to be too daunting. This is why I'd prefer to be able to see step by step output.

Originally Posted by iatheia

And while I'm at it, here is a helpful bit of advise before you do anything. Python fundamentally depends on the proper formatting of the code. Other languages may allow everything to be flushed to the edge, but python would be very confused and yell at you if that were the case. So do keep the same formatting as the code you're modifying - four spaces (not tabs!) per indent. This is something that gave me hell to figure out when I've just started.

Thanks. I'm already familiar with Python indenting rules (from writing custom Calibre template functions, etc) and Notepad++ makes this very simple.

Originally Posted by JimmXinu

@ilovejedd - That would be a bit of a hefty feature to add just for playing with.

I figured that would be the case after I looked at the code for some of the site adapters (hadn't checked them before I made the request). To implement it, each site adapter would need to be edited. On the upside, it appears I can easily add the option to whatever site adapter I'm working with and comment it out later.

Originally Posted by JimmXinu

It's much easier to try different things and work on the code in command line than it is in the plugin. In command line, every edit you save is used immediately the next time you run it. In plugin, you have to create the updated plugin zip, install it and restart calibre for every single change.

No need to restart but yeah, need to go through the hassle of creating the updated plugin and then installing it to calibre.

Originally Posted by JimmXinu

There's a command line interface(CLI) version of FFDL available. If you don't want to install Python separately, after getting and unzipping the CLI version you can use "calibre-debug -e download.py" to run it using calibre's built-in python.

When debugging plugins, when I want to see what the HTML from the site looks like before any processing, I find the fetch call (typically "data = self._fetchUrl(url)") and then add a "print(data)" call. Consider that your first step in learning python.

Then run the CLI downloader and redirect it's output to a file. Something like this:

c:\Python27\python.exe downloader.py "http://test1.com?sid=4" > output

Editing downloader.py and changing where it says
"logging.basicConfig(level=logging.INFO,..." to
"logging.basicConfig(level=logging.DEBUG,..." is also useful.

When testing FFDL output or settings, you can use the dummy site test1.com with story URLs like test1.com?sid=123 That will generate output, but not waste time and bandwidth going out to a real site.

Thanks for the tips! Another question I'd like to ask, short of editing the lmhosts file or creating a proxy, is there a way I can get FFDL to read a previously download copy of a webpage instead of always having to retrieve it from the internet? This would be useful when working offline on my laptop in places without internet access.

Thanks!

[JimmXinu]

Originally Posted by ilovejedd

...
No need to restart but yeah, need to go through the hassle of creating the updated plugin and then installing it to calibre.

Due to the way calibre loads plugins, you do have to restart calibre to pick up changes. Unless it's changed in the last version or two.

Originally Posted by ilovejedd

...
Thanks for the tips! Another question I'd like to ask, short of editing the lmhosts file or creating a proxy, is there a way I can get FFDL to read a previously download copy of a webpage instead of always having to retrieve it from the internet? This would be useful when working offline on my laptop in places without internet access.

Just the test1.com adapter.

I suppose you could add some kind of caching mechanism to the _fetch routines in adapters/base_adapter.py. Not sure how accurately it could replicate real sites, though.

[JimmXinu]

Originally Posted by ilovejedd

Would like to create a site adapter for AFF (adultfanfiction.net). If it's not allowed to be included with FFDL, then I'd at least like to write it for my own use. Thanks!

Originally Posted by iatheia

It's not that it isn't allowed, it is just it's too complicated to do - the site has really overdid itself on the legal warnings, and just a tad bit too ugly to attempt. Plus it appears that they are undergoing a rewrite. Not really ideal for the first adapter, but if you want to try it - go ahead.

Ugliness is part of--we have common code already for asking 'are you an adult' and collecting user/pass and I didn't want to add code for collecting DOB and 'signature'.

A lot of it was just that I didn't want to deal with a site that felt the need to be that lawyered up. To the point that I've never gone past the scary DOB/'signature' page that includes warnings such as:

Providing a false declaration under the penalties of perjury is a criminal offense. This document constitutes an un-sworn declaration under federal law.

[ilovejedd]

Originally Posted by JimmXinu

Ugliness is part of--we have common code already for asking 'are you an adult' and collecting user/pass and I didn't want to add code for collecting DOB and 'signature'.

A lot of it was just that I didn't want to deal with a site that felt the need to be that lawyered up. To the point that I've never gone past the scary DOB/'signature' page that includes warnings such as:

In fairness to AFF, the site basically started after FanFiction.Net got sued for having NC-17 content in 2002. Since practically all content on AFF is adult material and they don't require registration to access adult material, that's basically the reason for all the legal warnings. They're really just trying to protect themselves lest they be a target for lawsuits.

[JimmXinu]

Yep. Me too. If they feel it necessary to put that much legalese on it, it makes me uncomfortable to provide a way to bypass it.