[GUI Plugin] FanFictionDownLoader

[Snowman]

Originally Posted by JimmXinu

Ffnet has started blocking the default python user agent, it appears. It may be a while before I can do anything about it.

Yes - confirmed. I still have proxomitron lying around, so I fired it up and configured it to spoof a different user agent, then pointed calibre at it:

Connect via proxomitron proxy:

Spoiler:

+++GET 66+++
GET /s/8045114/60/ HTTP/1.1
Host: www.fanfiction.net
User-Agent: Mozilla/4.61 [ja] (X11; I; Linux 2.2.13-33cmc1 i686)
Accept-Encoding: gzip, deflate
Connection: keep-alive

+++RESP 66+++
HTTP/1.1 200 OK
Vary: Accept-Encoding
Last-Modified: Fri, 06 Sep 2013 22:59:54 GMT
X-xss-protection: 1; mode=block
X-frame-options: SAMEORIGIN

Select proxomitron bypass:

Spoiler:

+++GET 70+++
GET /s/8045114/1/ HTTP/1.1
Host: www.fanfiction.net
User-Agent: Python-urllib/2.7
Accept-Encoding: gzip
Connection: keep-alive

+++RESP 70+++
HTTP/1.1 403 Forbidden
Date: Fri, 06 Sep 2013 23:09:53 GMT
Accept-Ranges: bytes
Server: LiteSpeed
Cache-Control: private, no-cache, max-age=0
Pragma: no-cache
Content-Type: text/html
Content-Length: 381
+++CLOSE 70+++

Snowman

[shesgottaread]

I couldn't figure out how to follow what Snowman had done, but figured I'd see if a little googling might help me figure out another way around.

Following up on mentions of a workaround that Jimm made a while back for yourfanfiction, I tracked down the code and added it to my install of the calibre ffdl plugin.

In the file adapter_fanfictionnet.py:

Original code:

        BaseSiteAdapter.__init__(self, config, url)
        self.story.setMetadata('siteabbrev','ffnet')

the adaptor code now reads:

        BaseSiteAdapter.__init__(self, config, url)

	###  sgr's attempt to workaround the 403 blocking:
	### copied from http://code.google.com/p/fanficdownloader/source/browse/fanficdownloader/adapters/adapter_yourfanfictioncom.py?spec=svn7a2fb1721cc56a125d5713e26a4eec6604d2e32e&r=7a2fb1721cc56a125d5713e26a4eec6604d2e32e
		
        # fanfiction.net blocking the default user-agent.  
        # Changing the default user-agent value from: 
        # User-Agent: Python-urllib/2.7
        # to a new FFDL-specific user-agent value:
        self.opener.addheaders = [('User-agent', 'FFDL/1.6')]

	###  /\ sgr's attempt to workaround the 403 blocking
		
        self.story.setMetadata('siteabbrev','ffnet')

Quick and dirty, but it has worked for me so far. Note that I should have probably updated the 1.6 to the current version number of FFDL, but I've yet to update since just before calibre went to 1.0.0.

Hope this is helpful to others!
--sgr

P.S. Sorry for not having the spoilers thingie working. still learning the editor.

[Firedancer885]

Originally Posted by shesgottaread

I couldn't figure out how to follow what Snowman had done, but figured I'd see if a little googling might help me figure out another way around.

Hope this is helpful to others!
--sgr

Thank you, it works

Firedancer

[BaconButty]

Sorry, but I have to ask...

how/where do I find adapter_fanfictionnet.py to add the code?

[FaceDeer]

It's in the FanFictionDownLoader.zip addin bundle in your Calibre installation's plugin folder. On my Windows machine it's in C:\Users\[my user name]\AppData\Roaming\calibre\plugins\FanFictionDownLoa der.zip\fanficdownloader\adapters, I don't know where it would be in a Linux install.

After messing with the contents of this zip you'll need to restart Calibre to make it reload it, similar to how you have to restart Calibre after updating a plugin. Make a backup if you've never messed with python before. If you make a syntax error it can cause the plugin to not load at all, you'll see this if your FFDL toolbar button isn't there when you start Calibre up.

[Firedancer885]

I use Ubuntu linux.
Calibre preferences - Miscellaneous - Open Calibre configuration library. This way you don't have to know the direct path to the plugin folder.

[Depletion]

Having a spot of trouble with a particular story on fimfiction.com. It updates the .epub file as well as word count, but not the metadata.

http://www.fimfiction.net/story/8932...bit-hard-on-me

calibre, version 1.1.0
ERROR: Error Updating Metadata: <p>An error has occurred while FFDL was updating calibre's metadata for <a href='http://www.fimfiction.net/story/89324/'>Love's Been a Little Bit Hard on Me</a>.</p>The ebook has been updated, but the metadata has not.

Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1113, in update_books_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1442, in update_metadata
  File "site-packages\calibre\db\legacy.py", line 458, in set_metadata
  File "site-packages\calibre\db\cache.py", line 56, in ans
  File "site-packages\calibre\db\cache.py", line 1115, in set_metadata
  File "site-packages\calibre\db\cache.py", line 1110, in set_field
  File "site-packages\calibre\db\cache.py", line 914, in set_field
  File "site-packages\calibre\db\backend.py", line 1335, in windows_check_if_files_in_use
  File "site-packages\calibre\utils\filenames.py", line 340, in __init__

[BaconButty]

Originally Posted by FaceDeer

It's in the FanFictionDownLoader.zip addin bundle in your Calibre installation's plugin folder. On my Windows machine it's in C:\Users\[my user name]\AppData\Roaming\calibre\plugins\FanFictionDownLoa der.zip\fanficdownloader\adapters.

After messing with the contents of this zip you'll need to restart Calibre to make it reload it, similar to how you have to restart Calibre after updating a plugin. Make a backup if you've never messed with python before. If you make a syntax error it can cause the plugin to not load at all, you'll see this if your FFDL toolbar button isn't there when you start Calibre up.

Found it, thank you.
I edited the file with Notepad, but it's asking me to save it, and I cannot seem to be able to save it in the same location.
What am I doing wrong?

I feel like a cleaner being asked to perform surgery!

[FaceDeer]

Originally Posted by Depletion

Having a spot of trouble with a particular story on fimfiction.com. It updates the .epub file as well as word count, but not the metadata.

Yay, another FiMFictioner!

That story downloaded fine for me. The error arose in windows_check_if_files_in_use, which makes me think perhaps you had some other program accessing the story's file at the time, preventing Calibre from writing to it. Do you have your library in a Dropbox folder, for example? There was some stuff about that causing trouble a page or two ago.

[FaceDeer]

Originally Posted by BaconButty

Found it, thank you.
I edited the file with Notepad, but it's asking me to save it, and I cannot seem to be able to save it in the same location.
What am I doing wrong?

The file is inside a .zip file, which is a compressed archive. You may need to get the file out of there for Notepad to be able to do its thing. The easiest way to do that on Windows would probably be to just drag the file out of the .zip to some other folder (or the desktop), make your edits, then drag it back into the place it came from and tell Windows to overwrite the old one with the edited one when it asks if that's what you want to do.

Note, if you've never worked with the Python language before, that the code's indentation level is important. And that the indentation has to be done with the same characters throughout the file (so if the python file uses spaces to indent the other existing lines, your new line needs to use spaces too). This is an easy thing to mess up and Calibre's error message may not be particularly informative if it happens.

This little fanfiction crisis is still in very early days, if editing this file continues to prove difficult you could just wait a little while and see if either ffnet removes this blockage or JimmXinu releases an updated version to work around it. I'm betting this wasn't a deliberate effort to block FFDL but rather just an oversight that'll get sorted out in the fullness of time.

[BaconButty]

Originally Posted by FaceDeer

The file is inside a .zip file, which is a compressed archive. You may need to get the file out of there for Notepad to be able to do its thing. The easiest way to do that on Windows would probably be to just drag the file out of the .zip to some other folder (or the desktop), make your edits, then drag it back into the place it came from and tell Windows to overwrite the old one with the edited one when it asks if that's what you want to do.

Note, if you've never worked with the Python language before, that the code's indentation level is important. And that the indentation has to be done with the same characters throughout the file (so if the python file uses spaces to indent the other existing lines, your new line needs to use spaces too). This is an easy thing to mess up and Calibre's error message may not be particularly informative if it happens.

This little fanfiction crisis is still in very early days, if editing this file continues to prove difficult you could just wait a little while and see if either ffnet removes this blockage or JimmXinu releases an updated version to work around it. I'm betting this wasn't a deliberate effort to block FFDL but rather just an oversight that'll get sorted out in the fullness of time.

Thank you.

I'll bear your suggestion in mind and sit on my hands for a while...

[Depletion]

Originally Posted by FaceDeer

Yay, another FiMFictioner!

That story downloaded fine for me. The error arose in windows_check_if_files_in_use, which makes me think perhaps you had some other program accessing the story's file at the time, preventing Calibre from writing to it. Do you have your library in a Dropbox folder, for example? There was some stuff about that causing trouble a page or two ago.

Yup, that site is actually what prompted me to start using calibre and subsequently FFDL. Couldn't keep up otherwise.

And I do, I forgot I moved it to Dropbox. It's funny that I updated around 30 FiM stories this past week and that's been the only consistent problem. But I relocated the library folder and the story was able to be updated.

[Sefiriot]

As recent threads around here have shown, calibre, along with its database and related files, particularly for versions 1.0 and higher absolutely hates and can't tolerate having two different programs accessing files at the same time. Thus having your calibre library in Dropbox, and then doing things while Dropbox's syncing files apparently has similar effects as having two calibre installs accessing the database at the same time-- not pretty.

If you are using Dropbox to backup/sync calibre, pausing Dropbox sync while calibre does its thing should make sure everything works well though.

@BaconButty: when you extract the file to edit the plugin, it might also be a good idea to put the extracted file into a non-system folder: I've noticed that trying to edit things in \Roaming doesn't always work, which seems to be because Windows is terrified of the user modifying things in system folders.

[leckadams]

Holy crow! So I consider myself fairly adapt at learning things, but I have NO clue how to do what was just explained to fix the solution.

Is there a how to fix Calibre extension for dummies version?

thanks

[BaconButty]

Originally Posted by leckadams

Holy crow! So I consider myself fairly adapt at learning things, but I have NO clue how to do what was just explained to fix the solution.

Is there a how to fix Calibre extension for dummies version?

thanks

Well, I tried ... and failed
But I'm not usually one to look under the bonnet to find out how things work!
See if you can do better (shouldn't be difficult at all!)

(With Windows)
Open Calibre Maintenance, Advanced, Miscellaneous, Open Calibre Configuration Directory.

From there, open as follows: plugins, FanFictionDownLoader.zip, fanficdownloader, adapters.

Find the file called adapter_fanfictionnet.py and copy it into your docs.
Rename this copy adapter_fanfictionnet (original).

Create another copy of the same file in the same location.
Open this file with Notepad and find the section of code shown in the repair suggested in shesgottaread's post above.
Replace that section of code with the suggested repair code, making sure you respect the spaces and syntax.
Before you save and close the file, ensure you update the FFDL version number in the code with the current version (mine says 1.7.39).
Save and close.

Drag and drop the updated file into the list of adapters opened via Calibre.
When prompted by Windows, select to replace the existing file.
Close the list of adapters etc.
Restart Calibre.

If the FFDL icon is missing from its usual place in the toolbar, something has gone wrong.
But do not despair, you still have the original adapter_fanfictionnet.py file.
Simply go back to your Docs, rename the new adapter_fanfictionnet as adapter_fanfictionnet-repair.
Rename the file called adapter_fanfictionnet (original) as adapter_fanfictionnet.

Open Calibre Maintenance... as before, then drag and drop the file called adapter_fanfictionnet into the list of adapters.
When prompted by Windows, select to replace the existing file.
Close the list of adapters etc.
Restart Calibre and it should be back to normal, albeit still without a FFN fix...

Well, that's how I did it anyway!


Maybe someone more used to this can highlight where I went wrong when I tried to fix it...