[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by davidfor

adapter_finestoriescom is subclassed from this. Does it need the personal.ini file related changes as well?

I dunno. Does finestories also have different chapter titles in chapter pages?

[davidfor]

Originally Posted by JimmXinu

I dunno. Does finestories also have different chapter titles in chapter pages?

I assumed so as the sites use the same code and are run by the same person. By luck, the first of my test cases I checked, showed that it does the same thing. This is http://finestories.com/s/10610.

[JimmXinu]

Originally Posted by davidfor

I assumed so as the sites use the same code and are run by the same person. By luck, the first of my test cases I checked, showed that it does the same thing. This is http://finestories.com/s/10610.

Right you are, thanks for pointing that out. I've added it and pushed the code.

I'm going to leave it for the next test version, since you can use it now with finestories.com if you want, it will just give a warning. Unless there's an outcry for it or something.

[Atherton25]

Alright, I had this really long explanation written out as I figured it out, but after way too long narrowing it down to exactly what’s occurring, I think I can rewrite this and make it easier to understand and replicate.

First, this has been happening for a long time. It was just annoying to figure out and so I never could understand why it was happening until yesterday it suddenly occurred what it might be. I honestly thought it was a pure calibre problem for a while.

Second, this is a problem with the FanFicFare Update (Date) column in my case it's named FFFCheck. Calibre Column Type is set to Date. The problem is an error I get a lot. It's a calibre error, now that I've had my epiphany and spent time narrowing it down, it looks like it comes from the interaction from FFF.

…So here we go.

I've got a calibre test database (and program) stripped down. Just running Calibre x64 and FanFicFare, both latest version. My test database has 2 files in it. a FFF story, in this case I just grabbed something tiny off ff.net. The second is a non FFF epub.

The search I'm using in a database is: #fffcheck:<5daysago first time no problem.

Highlight both, update. Of course the second file doesn't work. In Update Existing Dialog it says No Story URL Found. I update the files. Since the second has nothing to do with FFF, and there's no URL, I've never considered it might cause a problem.

Update - doesn't matter if the Fanfic updates or not, as long as both files metadata is updated in the fffcheck column.

Search again: #fffcheck:<5daysago ...... and Error: ValueError: astimezone() cannot be applied to a naive datetime (full error below)

Spoiler:

calibre, version 4.12.0
ERROR: Unhandled exception: <b>ValueError</b>:astimezone() cannot be applied to a naive datetime

calibre 4.12 [64bit] Portable embedded-python: True is64bit: True
Windows-10-10.0.18362-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 2.7.16+
Windows: ('10', '10.0.18362', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 9, 0) && FanFicFare (3, 17, 3)
Traceback (most recent call last):
  File "site-packages\calibre\gui2\library\views.py", line 1237, in search_proxy
  File "site-packages\calibre\gui2\library\models.py", line 491, in search
  File "site-packages\calibre\db\view.py", line 272, in search
  File "site-packages\calibre\db\view.py", line 307, in search_getting_ids
  File "site-packages\calibre\db\cache.py", line 62, in call_func_with_lock
  File "site-packages\calibre\db\cache.py", line 1015, in search
  File "site-packages\calibre\db\search.py", line 883, in __call__
  File "site-packages\calibre\db\search.py", line 925, in _do_search
  File "site-packages\calibre\db\search.py", line 490, in parse
  File "site-packages\calibre\utils\search_query_parser.py", line 342, in parse
  File "site-packages\calibre\utils\search_query_parser.py", line 363, in _parse
  File "site-packages\calibre\utils\search_query_parser.py", line 371, in evaluate
  File "site-packages\calibre\utils\search_query_parser.py", line 417, in evaluate_token
  File "site-packages\calibre\utils\search_query_parser.py", line 421, in _get_matches
  File "site-packages\calibre\db\search.py", line 581, in get_matches
  File "site-packages\calibre\db\search.py", line 207, in __call__
  File "site-packages\calibre\utils\date.py", line 226, in dt_as_local
ValueError: astimezone() cannot be applied to a naive datetime

I've tried dozens of variations. This has no problem whatsoever when everything is a FFF story (with URL), but anything missing a URL effects this as far as I can tell. Even files that were FFF downloaded and had the URL removed because of say rewrite (and removal of all the old chapters to do it, so I kept the old version). That's probably why I couldn't narrow this problem down and it just seemed to randomly occur, I rarely ever get non FFF stories selected when I try and update FFF stories. Still, it's an error that's happened a lot for me, and I'd love to see it fixed.

[JimmXinu]

New Test Version Posted

2020-03-24
- Make sure all timestamps are using Calibre's local_tz.

@Atherton25,

I've tried to follow your steps exactly, but I cannot get an error from doing a #fffcheck:<5daysago search with anything I try.

Are you talking about FFF's Last Checked Column feature? That is not the same thing as just setting a date column to the updated date metadata, which is what you appear to describe above.

Because I can get an astimezone() cannot be applied to a naive datetime error while trying to update a non-FFF story while Last Checked Column is enabled, but it only appears in the debug log where it is otherwise ignored.

It's very easy to argue that FFF shouldn't be setting anything for books that aren't from FFF, even if the user tries to. But there are cases when a book may have a had a valid story URL in past, but the site isn't supported anymore. And then I'd be left with only leaving out books with no URL at all, which is even more chaotic...

On one occasion while working on this, I was able to get Calibre to give me an astimezone() cannot be applied to a naive datetime error while editing metadata after trying various updates.

And now I cannot get that error trying to replicate it either way above. (sigh).

So what I've done is looked at the few places in FFF that wasn't already explicitly assigning the same timezone as Calibre (a few datetime.now() calls) and forced them to Calibre's TZ also.

That's my best guess, and it seems logical. So hopefully it will help.

[Atherton25]

Originally Posted by JimmXinu

New Test Version Posted

2020-03-24
- Make sure all timestamps are using Calibre's local_tz.

I think that did it! Whatever it is that you changed, it seems to be working on my test database. I'll be running it on my main one to see if it ever throws that error again and then (should it happen again) I'll try to narrow it down as best as possible.

I was prepared to zip a copy my test database with stripped down personal.ini and and offer it for checking, and then I was finally able to narrow it down very specifically to discover what was doing it. I'm sorry my instructions weren't quite clear enough, but you saw it at least once. I probably missed a step when I was rewriting the instructions/situation. Yes, I was definitely using the Last Checked Column feature. I thought I'd said that right, but apparently I hadn't.

And I understand the frustration. I've been having this error off and on for well over a year, probably closer to 2, if not longer. It seemed that, just randomly, I couldn't sort by a date column. And once I tried to sort by the wrong column (I think it was always my FFFCheck column) I got errors thrown anytime I tried to sort by another column I'd have to change databases, or restart Calibre. With the large database I work in getting back in takes a minute or two no matter how I did it, so it was *really* annoying particularly when I wanted to just upload the books onto my kindle and walk away.

Thanks for the fix.

[vikarti]

Calibre's FanFicFare plugin appears to stop working with ficbook.net
Both existing and new urls results in errors like this

Статус (=Status)
Название (=Name)
Автор (=Author)
Comment
URL
Ошибка (=Error)
Mass Effect: Human Revolution
Lightstorm
'NoneType' object has no attribute 'text'
https://www.ficbook.net/readfic/8483426

Site itself appears to work fine from my computer.
Attemp to use ./calibre-debug --run-plugin FanFicFare https://www.ficbook.net/readfic/8483426 results in

Python function terminated unexpectedly: initial_value must be unicode or None, not str
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 187, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 121, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/debug.py", line 327, in main
File "calibre_plugins.fanficfare_plugin.__init__", line 139, in cli_main
File "calibre_plugins.fanficfare_plugin.fanficfare.cli" , line 308, in main
File "calibre_plugins.fanficfare_plugin.fanficfare.cli" , line 356, in do_download
File "calibre_plugins.fanficfare_plugin.fanficfare.cli" , line 557, in get_configuration
TypeError: initial_value must be unicode or None, not str

[JimmXinu]

Originally Posted by vikarti

Calibre's FanFicFare plugin appears to stop working with ficbook.net
Both existing and new urls results in errors like this
...

Looks like the site has changed. I'll look into it.

FYI, that second error with the Calibre CLI is unrelated and should be fixed in the latest test version.

[JimmXinu]

New Test Version Posted

2020-03-25
- Fixes for ficbook.net site changes.
- Correct comments for universe_as_series setting in defaults.ini.

Note that the ficbook.net changes include putting all 'Tags' into 'genre' because they're all mixed together. Now including ships!

[vikarti]

Originally Posted by JimmXinu

Looks like the site has changed. I'll look into it.

FYI, that second error with the Calibre CLI is unrelated and should be fixed in the latest test version.

Thanks.
Last test version mostly works but
https://www.ficbook.net/readfic/8857583 results in

Ошибка
Unknown
Unknown
'NoneType' object has no attribute 'find_next'
https://www.ficbook.net/readfic/8857583

[JimmXinu]

New Test Version Posted

2020-03-25
- Fixes for ficbook.net site changes.
- Correct comments for universe_as_series setting in defaults.ini.
- Fix for adapter_ficbooknet when no ships/characters.

[vikarti]

Looks like handling of utopiastories.com urls is not fully correct in Calibre's FanFicFare plugin.
only http://www.utopiastories.com/code/show_story.asp/recid/<story_id_number> URLs are handled but https://www.utopiastories.com/code/show_story.asp/recid/<story_id_number> are considered invalid by fanficfare even while same content is available at both URLs.

[JimmXinu]

Originally Posted by vikarti

Looks like handling of utopiastories.com urls is not fully correct in Calibre's FanFicFare plugin.
only http://www.utopiastories.com/code/show_story.asp/recid/<story_id_number> URLs are handled but https://www.utopiastories.com/code/show_story.asp/recid/<story_id_number> are considered invalid by fanficfare even while same content is available at both URLs.

HTTPS doesn't work for me on www.utopiastories.com in Python from CLI or from in Calibre. Checking with an SSL analyzer site, their SSL implement is rated 'F' due to only supported known insecure protocols.

I will add code in the next test version to accept https:// URLs for utopiastories.com, but FFF can't use https:// for downloads for that site at this time.

[MidwestJen]

I've been using the "get from email" feature for a long time now. It seems Google updated how the security is for 3rd party access to your account and now you have to turn on a flag to allow the "less secure" in (which I did). Even with that, I'm still getting an error. I haven't changed my credentials with gmail, and I did re-enter them in FFF just in case they were corrupted. I've restarted and I continue to get the same error.

calibre, version 4.13.0
ERROR: Unhandled exception: <b>error</b>:[AUTHENTICATIONFAILED] Invalid credentials (Failure)

calibre 4.13 [64bit] embedded-python: True is64bit: True
Windows-10-10.0.18362-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 2.7.16+
Windows: ('10', '10.0.18362', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 9, 0) && NormComment (0, 0, 2) && Generate Cover (1, 5, 22) && Barnes & Noble (1, 2, 15) && FanFicFare (3, 17, 0) && FictionDB (1, 2, 0)
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 493, in get_urls_from_imap_menu
File "calibre_plugins.fanficfare_plugin.fanficfare.getu rls", line 223, in get_urls_from_imap
File "imaplib.py", line 523, in login
error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)

This is a dedicated account for FF.net emails and has 187 unread notifications. I'd try lessening the amount to see if it's overloading, but my guess is that it's failing before it even gets to that.

Thoughts on what could be the problem?

[JimmXinu]

No idea, unfortunately. You are correct that the error is occurring during login, before it gets to fetching emails.

Back in Feb 2020 somebody else had the same or a similar issue that mysteriously went away.

All I can say is that's working fine for me and I too use Google account for it.