[GUI Plugin] FanFicFare

[scarlettruin]

Originally Posted by JimmXinu

FFF isn't able to resolve the host name. Could be a problem with your network, your proxy settings (or proxy), or a firewall/security program blocking calibre from making outbound connections.

If you can still see the story page on the same computer using your browser (and don't use a proxy), a firewall program is the most likely cause.

The strange thing is that it just started happening - there's been no changes at all, so I can't figure out what it could be. Do you have any tips for anyway I could sort this or anything to try please?

[JimmXinu]

I'd try restarting calibre and rebooting your computer. And try downloading a story from a different site--it's possible you've been blocked and the error message is misleading.

And it has become common for OSes and antivirus/'protection' software to self-update without notifying the user. Just because you didn't change anything doesn't mean nothing changed...

[tichkiir]

I'm having the same getaddrinfo failed issue. It looks like maybe with just AO3?

It loads fine in browser, I tried rebooting, don't see any issues with firewall or anything.

If I try and update an existing Anthology from AO3, it can pull up the URLs for the stories in it fine, including a new one that wasn't there before - so at least sometimes it's able to get to the site fine, it can even fetch the metadata ok maybe. But when it goes to pull the actual stories down, error on 5/6 of them, which is very odd. Tried getting same new file 5 times, none worked.

Ahh, found some generic python debugging and this is what I got:

C:\Users\charris>python
Python 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib, urllib2, httplib
>>>
>>> url = 'https://archiveofourown.org/works/11656197?view_adult=true'
>>> httplib.HTTPConnection.debuglevel = 1
>>>
>>> print "urllib"
urllib
>>>
>>> data = urllib.urlopen(url);
send: 'GET /works/11656197?view_adult=true HTTP/1.0\r\nHost: archiveofourown.org\r\nUser-Agent: Python-urllib/1.17\r\n\r\n'
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
header: Server: nginx/1.13.7
header: Content-Type: text/html; charset=utf-8
header: Connection: close
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: Location: https://unicorn_story/works/11656197/chapters/26225727
header: Set-Cookie: _otwarchive_session=cjQ3dkRyQkxyT0RNRGp6U2l0T3lhbz d2TFl3ditIZ1R1alp1aHdzRU5tdmNXajM5UHJwejJBbktUUmtn V3VQUXpoTzZXMzRVNWg4Qit4eFpRbXdoc2hhRzBHeXNSK2ZSeF JVRkxBaE9PWklqenpLSlY4bTlOWkJxMWd0b3FkQzJlZ09nZDVx amlGL1l6ekUrTWdHY1h0bCtLbHRpbmJybFRjczBMYkZlU3FRPS 0tanJEb3ozbi9DeUFhWGNoQUNvamxrUT09--2dd0ad9f1aae2f
97e314a9c946288070acb523ad; path=/; expires=Sun, 11 Mar 2018 10:21:36 -0000; HttpOnly
header: X-Request-Id: 667fb26a-2116-4cc8-9aeb-3e3133a15983
header: X-Runtime: 0.028923
header: X-Aooo-Debug1: Archive Unicorn
header: X-Clacks-Overhead: GNU Terry Pratchett
header: Potential_upstream: unicorn_story
header: X-Hostname: ao3-front04
header: Date: Sun, 25 Feb 2018 10:21:36 GMT
header: X-Page-Speed: 1.13.35.1-0
header: Cache-Control: max-age=0, no-cache
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python27\lib\urllib.py", line 208, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 451, in open_https
return self.http_error(url, fp, errcode, errmsg, headers)
File "C:\Python27\lib\urllib.py", line 372, in http_error
result = method(url, fp, errcode, errmsg, headers)
File "C:\Python27\lib\urllib.py", line 635, in http_error_302
data)
File "C:\Python27\lib\urllib.py", line 661, in redirect_internal
return self.open(newurl)
File "C:\Python27\lib\urllib.py", line 208, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 437, in open_https
h.endheaders(data)
File "C:\Python27\lib\httplib.py", line 991, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 844, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 806, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1194, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno 11004] getaddrinfo failed
>>>

[scarlettruin]

Originally Posted by tichkiir

I'm having the same getaddrinfo failed issue. It looks like maybe with just AO3?

It loads fine in browser, I tried rebooting, don't see any issues with firewall or anything.

If I try and update an existing Anthology from AO3, it can pull up the URLs for the stories in it fine, including a new one that wasn't there before - so at least sometimes it's able to get to the site fine, it can even fetch the metadata ok maybe. But when it goes to pull the actual stories down, error on 5/6 of them, which is very odd. Tried getting same new file 5 times, none worked.

Ahh, found some generic python debugging and this is what I got:

C:\Users\charris>python
Python 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib, urllib2, httplib
>>>
>>> url = 'https://archiveofourown.org/works/11656197?view_adult=true'
>>> httplib.HTTPConnection.debuglevel = 1
>>>
>>> print "urllib"
urllib
>>>
>>> data = urllib.urlopen(url);
send: 'GET /works/11656197?view_adult=true HTTP/1.0\r\nHost: archiveofourown.org\r\nUser-Agent: Python-urllib/1.17\r\n\r\n'
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
header: Server: nginx/1.13.7
header: Content-Type: text/html; charset=utf-8
header: Connection: close
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: Location: https://unicorn_story/works/11656197/chapters/26225727
header: Set-Cookie: _otwarchive_session=cjQ3dkRyQkxyT0RNRGp6U2l0T3lhbz d2TFl3ditIZ1R1alp1aHdzRU5tdmNXajM5UHJwejJBbktUUmtn V3VQUXpoTzZXMzRVNWg4Qit4eFpRbXdoc2hhRzBHeXNSK2ZSeF JVRkxBaE9PWklqenpLSlY4bTlOWkJxMWd0b3FkQzJlZ09nZDVx amlGL1l6ekUrTWdHY1h0bCtLbHRpbmJybFRjczBMYkZlU3FRPS 0tanJEb3ozbi9DeUFhWGNoQUNvamxrUT09--2dd0ad9f1aae2f
97e314a9c946288070acb523ad; path=/; expires=Sun, 11 Mar 2018 10:21:36 -0000; HttpOnly
header: X-Request-Id: 667fb26a-2116-4cc8-9aeb-3e3133a15983
header: X-Runtime: 0.028923
header: X-Aooo-Debug1: Archive Unicorn
header: X-Clacks-Overhead: GNU Terry Pratchett
header: Potential_upstream: unicorn_story
header: X-Hostname: ao3-front04
header: Date: Sun, 25 Feb 2018 10:21:36 GMT
header: X-Page-Speed: 1.13.35.1-0
header: Cache-Control: max-age=0, no-cache
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python27\lib\urllib.py", line 208, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 451, in open_https
return self.http_error(url, fp, errcode, errmsg, headers)
File "C:\Python27\lib\urllib.py", line 372, in http_error
result = method(url, fp, errcode, errmsg, headers)
File "C:\Python27\lib\urllib.py", line 635, in http_error_302
data)
File "C:\Python27\lib\urllib.py", line 661, in redirect_internal
return self.open(newurl)
File "C:\Python27\lib\urllib.py", line 208, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 437, in open_https
h.endheaders(data)
File "C:\Python27\lib\httplib.py", line 991, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 844, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 806, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1194, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno 11004] getaddrinfo failed
>>>

I get the same thing - i don't know what the unicorn story bit is about either?

[JimmXinu]

Originally Posted by tichkiir

I'm having the same getaddrinfo failed issue. It looks like maybe with just AO3?

It loads fine in browser, I tried rebooting, don't see any issues with firewall or anything.
...
Ahh, found some generic python debugging and this is what I got:
...
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
...
header: Location: https://unicorn_story/works/11656197/chapters/26225727
...

First, great job finding that debug code. I didn't know trick and it gives us some important details.

When you two do it, you are getting redirected by the web server to the domain name 'unicorn_story' which clearly does not exist (no TLD).

When I do it from my home network (and from my server in the cloud), I get redirected to the same URL path, but at archiveofourown.org. Also, your redirects are to https and mine to http--I don't know why that would be.

Also, I'm seeing some different headers:

header: Server: nginx/1.6.2
header: Location: http://archiveofourown.org/works/11656197/chapters/26225727
header: X-Aooo-Debug: The Lion
header: X-Hostname: ao3-front02
header: X-Page-Speed: 1.9.32.3-4448

While you are seeing:

header: Server: nginx/1.13.7
header: Location: https://unicorn_story/works/11656197/chapters/26225727
header: X-Aooo-Debug1: Archive Unicorn
header: X-Hostname: ao3-front04
header: X-Page-Speed: 1.13.35.1-0

So I think you are hitting a different set of servers that are misconfigured. Maybe because of network optimization that sends us each to a 'closer' server, maybe because they are upgrading--your debug is showing higher version numbers.

What I can't explain as easily is why it would continue to work in your browser. They may be differentiating by user-agent, but another possibility is they are deliberately sending you two in particular in a different direction. That could happen if you've been doing a ton of downloads and tripped some blocking behavior.

Here are a few suggestions:

• You could report the problem to AO3 and see if they can correct it. If it were me, I'd send them the python debug output as proof to start the conversation. It's much more informative than if you start with 'FanFicFare isn't working for me on AO3'.
• Your browser may continue to work because of user-agent specific handling. You could try configuring FFF to use the same user-agent as your browser. In personal.ini set user_agent:--sites like http://getright.com/useragent.html can tell your browser's user-agent.
• You could download using the FFF web service until the problem is resolved. That has its own issues as the web service has daily resource limits that can be exhausted and can't populate your Calibre columns.
• AO3 is one of the sites that offers ebook formats directly for download. You can download their EPUB files and use those instead of using FFF.

[tichkiir]

Originally Posted by JimmXinu

First, great job finding that debug code. I didn't know trick and it gives us some important details.

When you two do it, you are getting redirected by the web server to the domain name 'unicorn_story' which clearly does not exist (no TLD).

When I do it from my home network (and from my server in the cloud), I get redirected to the same URL path, but at archiveofourown.org. Also, your redirects are to https and mine to http--I don't know why that would be.

Also, I'm seeing some different headers:

header: Server: nginx/1.6.2
header: Location: http://archiveofourown.org/works/11656197/chapters/26225727
header: X-Aooo-Debug: The Lion
header: X-Hostname: ao3-front02
header: X-Page-Speed: 1.9.32.3-4448

While you are seeing:

header: Server: nginx/1.13.7
header: Location: https://unicorn_story/works/11656197/chapters/26225727
header: X-Aooo-Debug1: Archive Unicorn
header: X-Hostname: ao3-front04
header: X-Page-Speed: 1.13.35.1-0

So I think you are hitting a different set of servers that are misconfigured. Maybe because of network optimization that sends us each to a 'closer' server, maybe because they are upgrading--your debug is showing higher version numbers.

What I can't explain as easily is why it would continue to work in your browser. They may be differentiating by user-agent, but another possibility is they are deliberately sending you two in particular in a different direction. That could happen if you've been doing a ton of downloads and tripped some blocking behavior.

Here are a few suggestions:

• You could report the problem to AO3 and see if they can correct it. If it were me, I'd send them the python debug output as proof to start the conversation. It's much more informative than if you start with 'FanFicFare isn't working for me on AO3'.
• Your browser may continue to work because of user-agent specific handling. You could try configuring FFF to use the same user-agent as your browser. In personal.ini set user_agent:--sites like http://getright.com/useragent.html can tell your browser's user-agent.
• You could download using the FFF web service until the problem is resolved. That has its own issues as the web service has daily resource limits that can be exhausted and can't populate your Calibre columns.
• AO3 is one of the sites that offers ebook formats directly for download. You can download their EPUB files and use those instead of using FFF.

Hahah, I did change the user-agent and it worked! Yay! And then I tried running the python again and it worked too. So I'm thinking your idea about us getting routed to some broken hosts might be correct. Either way, it's working now, and maybe the user-agent will keep it working in the future.

Thanks so much for looking into this and responding with suggestions.

[JimmXinu]

New Test Version Posted

2018-02-26
- Update translations
- Calculate number of chapters using start-end range when doing updates.

Unless something else comes up, this will be the last test version before the next release. FYI.

[JimmXinu]

New Version Released

Version 2.23.0 - 28 Feb 2018
- Update translations.
- Calculate number of chapters using start-end range when doing updates.
- Make Get URLs from Page work better with TtH is_adult.
- Fix rating, warnings, add ships to adapter_harrypotterfanfictioncom
- Update adapter_efpfanficnet to use https and remove www. by default.
- *Don't* include fandoms in category for fimfiction.net by default.
- Handle new VIP chapter types in adapter_webnovelcom (#263) (Thanks cryzed)

[alexeric123]

I was wondering if you guys could consider adding novelupdates and bakatsuki to your plugin, but so far I really like it!

[Atherton25]

@everyone downloading stuff from Wuxiaworld
Based on a post I ran across today, I figured others might want to be notified that wuxiaworld will be "be transitioning to v1.0 of the new site" and though they're not specific, they basically say there's a lot of change in the underlying structure and they'll be slowly transitioning it over the next few weeks. This will likely break the plugin for that site.

I figure I'd notify people so they can update now anything they'll probably be reading the next few weeks. I don't know who maintains the plugin, I'd have to go double check (cryzed probably), but with the multiple/continuing changes they might not fix it until all is done, if at all. Again, just a heads-up.

@alexeric123
Novelupdates is not a novel site, rather it shows links to updates on various sites. It has been discussed, and I don't believe anyone can do it well due to the varying sites. Though it'd be nice particularly not to have to update each new book I add with the metadata from novelupdates, I don't think it'll happen.

As for bakatsuki, I don't recall much talk of it, though I believe it has been mentioned at least once. It's probably a bit too much for a basic rip, plus it generally provides an easy full text download in one form or another, and so likely wouldn't be even a normal priority for most people working on the plugin. I don't use the site much myself, but I've downloaded a few things from there.

[leckadams]

I have searched the posts, but I am not finding the relative posts. Is there a reason that I can't use the author download function or website download (I used to use it for bookmarks) on AO3 any more?

Thanks in advance!

[JimmXinu]

I am not aware of any issues with AO3, nor do I see any problems with story URL lists from an author page or my own bookmarks page using the latest version.

You're going to have to be more specific if you're having a problem somewhere else. IE, include page URLs and precisely what you are or aren't seeing.

[leckadams]

Originally Posted by JimmXinu

I am not aware of any issues with AO3, nor do I see any problems with story URL lists from an author page or my own bookmarks page using the latest version.

You're going to have to be more specific if you're having a problem somewhere else. IE, include page URLs and precisely what you are or aren't seeing.

I apologize. This is what I keep getting when I try to pull bookmarks or author stuff on AO3. I have all my logins in the configuration.

calibre, version 3.18.0
ERROR: Unhandled exception: <b>FailedToLogin</b>:Failed to Login for URL: (https://archiveofourown.org/users/leckadams/bookmarks) with username: (leckadams)

calibre 3.18 embedded-python: True is64bit: False
Windows-8.1-6.3.9600-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '8.1', '6.3.9600')
Python 2.7.12+
Windows: ('8.1', '6.3.9600', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Find Duplicates (1, 6, 3) && EpubMerge (2, 1, 1) && EpubSplit (2, 3, 0) && Count Pages (1, 8, 0) && Goodreads (1, 1, 13) && FanFicFare (2, 23, 0)
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 548, in get_urls_from_page_menu
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 568, in get_urls_from_page
File "calibre_plugins.fanficfare_plugin.fanficfare.getu rls", line 61, in get_urls_from_page
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.adapter_archiveofourownorg", line 126, in performLogin
FailedToLogin: Failed to Login for URL: (https://archiveofourown.org/users/leckadams/bookmarks) with username: (leckadams)

[JimmXinu]

Well, clearly the error is that it's failed to login. Mine logged in fine using the same versions.

The most obvious question is: Do you have the correct AO3 password set in personal.ini? But you said you do.

You've apparently figured out how to log errors, what do you get if you also run calibre in debug mode? There should be some additional details output that way.

[sonofden]

I think I found a glitch with the webnovel.com script. It seems to be ignoring words that appear between <>. As an example, chapter 15 of MMORPG: Martial Gamer https://www.webnovel.com/book/906713...fair-Advantage the name of the game <Rebirth> as well as names of the items are all missing.