[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by thatblondelady

...
Edit: The new version does not seem to have solved the problem. Both the getting stuck at 30% issue and the anthology problems still exist.

Both things function properly in debug mode.

Edit 2: To be clear, this was with the beta version of Calibre, and the test version of FFF from above.

Is that 30% of downloading metadata? Or 30% done in the background job?

If it's 30% that suggests you're doing several downloads--a single download would jump from 0% to 100% at once. Do single downloads work?

Can you get a screen shot of it when it gets stuck?

[thatblondelady]

Originally Posted by JimmXinu

Is that 30% of downloading metadata? Or 30% done in the background job?

If it's 30% that suggests you're doing several downloads--a single download would jump from 0% to 100% at once. Do single downloads work?

Can you get a screen shot of it when it gets stuck?

Yes. 30% of downloading metadata. It gets stuck there every time. Individual downloads work just fine, and I can update the stories it gets stuck on. Works just fine on Windows and in debug mode.

(I don't normally update all my in progress at once as there are 30, so I wait for an email, but I did so for purposes of testing here.)

[thatblondelady]

Interesting. It just went through, but now I have the same problem that I did in anthologies. The download happens, and I get the popup saying 1 good update and however many bad, click ok, but the next step of updating the book does not happen. I'm going to go upstairs and test it on windows to get the URL of the story that's not updating in case that's relevant.

This was not happening this morning under the release versions of calibre and FFF.

Edit: the story is http://archiveofourown.org/works/6149728. It will not update if I do a group update of all my stories on mac unless I'm in debug mode. It does on windows. I can update it individually and that works.

The good news is I'm no longer getting stuck at 30%, so there is that.

[JimmXinu]

After some more tests that all worked, I did have it stop at 12 of 18 downloads on OS X, without debug, and then succeeded with debug.

And then a bunch of other tests all worked fine both with and w/o debug. So I did see the problem. Once. And now I can't replicate it.


I'll look at it some more, maybe tomorrow. But for now the answer for Mac users is to run in debug mode and hope Kovid or somebody more knowledge figures something out.

[kovidgoyal]

It sounds like the ASL service (which is where print statements go in normal mode) is failling sporadically, probably when there is a lot of output to stdout.

The simple solution is probably just not to print lots of stuff when not in debug mode. Or alternatively use a wrapper for the print function that ignores failures silently.

[edeniz]

I have a question about downloading from spacebattles and similar forums.

I thought fanficfare was supposed to use posting dates for the published/updated dates? Did I misunderstand it? Because in both cases I get the date of download for it. It's alright if that's how it's supposed to be, I just want to make sure I'm not making a mistake somewhere. Though to be fair, I did not change anything from the defaults.ini in this section of the personal.ini, aside from adding a description_limit.

Example urls:
https://forums.spacebattles.com/threads/341476/
https://forums.spacebattles.com/threads/374200/

[JimmXinu]

Originally Posted by kovidgoyal

It sounds like the ASL service (which is where print statements go in normal mode) is failling sporadically, probably when there is a lot of output to stdout.

The simple solution is probably just not to print lots of stuff when not in debug mode. Or alternatively use a wrapper for the print function that ignores failures silently.

I agree that that has been a problem in past. However, I've already replaced all of the prints in FanFicFare with logger.debug() calls.

I also use logger.info()/error() calls. I assume that the overall process logger level is set such that none of them are output when calibre isn't running in debug mode?

[kovidgoyal]

What logger are we talking about here? IIRC plugin base classes have no default logger defined? If you want to use a DEBUG level dependent logger, the way to do it is like this:

from calibre.utils.logging import ThreadSafeLog

log = ThreadSafeLog(level=ThreadSafeLog.DEBUG if DEBUG else ThreadSafeLog.WARN)


That means that log.info()/log.debug() will be no-ops unless DEBUG is defined.

[JimmXinu]

Originally Posted by edeniz

I have a question about downloading from spacebattles and similar forums.

I thought fanficfare was supposed to use posting dates for the published/updated dates? Did I misunderstand it? Because in both cases I get the date of download for it. It's alright if that's how it's supposed to be, I just want to make sure I'm not making a mistake somewhere. Though to be fair, I did not change anything from the defaults.ini in this section of the personal.ini, aside from adding a description_limit.

Example urls:
https://forums.spacebattles.com/threads/341476/
https://forums.spacebattles.com/threads/374200/

Not quite. When using threadmarks for the chapter list, the Published and Updated dates are taken from the oldest and newest threadmark dates. (Example)

When taking the chapter list from a post, the Updated date is taken from the last edit timestamp of the chapter list post.

(The Packaged date will always been when the ebook was created/updated. That's the value used to set Calibre's date field, if that's what you're looking at.)

Downloading those two stories, I got the dates I expected.

[JimmXinu]

Originally Posted by kovidgoyal

What logger are we talking about here? IIRC plugin base classes have no default logger defined? If you want to use a DEBUG level dependent logger, the way to do it is like this:

from calibre.utils.logging import ThreadSafeLog

log = ThreadSafeLog(level=ThreadSafeLog.DEBUG if DEBUG else ThreadSafeLog.WARN)

That means that log.info()/log.debug() will be no-ops unless DEBUG is defined.

Then that's probably the source of the problem. I've been using:

import logging
logger = logging.getLogger(__name__)

I will try that out today. Thanks for your help.

[edeniz]

Originally Posted by JimmXinu

Not quite. When using threadmarks for the chapter list, the Published and Updated dates are taken from the oldest and newest threadmark dates. (Example)

When taking the chapter list from a post, the Updated date is taken from the last edit timestamp of the chapter list post.

(The Packaged date will always been when the ebook was created/updated. That's the value used to set Calibre's date field, if that's what you're looking at.)

Downloading those two stories, I got the dates I expected.

Well, yes, I understand that about the packaged date. But it's create/publish date that I meant. But basically at least the update timestamp for this example url you chose should be March 6, 2016, right? And not today's date? Take a look at how the titlepagentries look like for me for said example please, if you would, since I'm bad at explaining.

This is my usual titlepage_enties list:

Spoiler:

titlepage_entries: storyId,seriesHTML,category,genre,language,characters,ships,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,numWords,site,description

plus relevant are extratags for xenforo in the defaults:
add_to_extra_titlepage_entries:,tagsfromtitle,forumtags

Is the output list in the epub really what I can expect from the default settings for xenforo and my usual titlepage list?

[JimmXinu]

Originally Posted by edeniz

Well, yes, I understand that about the packaged date. But it's create/publish date that I meant. But basically at least the update timestamp for this example url you chose should be March 6, 2016, right? And not today's date? Take a look at how the titlepagentries look like for me for said example please, if you would, since I'm bad at explaining.

This is my usual titlepage_enties list: ...

Is the output list in the epub really what I can expect from the default settings for xenforo and my usual titlepage list?

I download the same story using the default settings and I get:
Published: 2015-05-24 17:22:00
Updated: 2016-03-06 22:58:00
Packaged: 2016-04-22 09:16:41

Given that your epub is not showing times (only dates) for the dates, you definitely have some personal.ini config that is effecting dates--the question is how much. I can also see that you're overriding the defaults.ini [base_xenforoforum] setting for at least strip_chapter_numbers.

So until I see what's in your personal.ini, my inclination is to blame unintended configuration.

[edeniz]

Originally Posted by JimmXinu

I download the same story using the default settings and I get:
Published: 2015-05-24 17:22:00
Updated: 2016-03-06 22:58:00
Packaged: 2016-04-22 09:16:41

Given that your epub is not showing times (only dates) for the dates, you definitely have some personal.ini config that is effecting dates--the question is how much. I can also see that you're overriding the defaults.ini [base_xenforoforum] setting for at least strip_chapter_numbers.

So until I see what's in your personal.ini, my inclination is to blame unintended configuration.

Well, yes, I got rid of the hour/minutes/seconds for dates in personal.ini. Don't really like it being that precise. Have attached relevant personal.ini (that is to say, without replace_metadata so it wouldn't be too large a file to look over, but otherwise complete).

[JimmXinu]

Originally Posted by edeniz

Well, yes, I got rid of the hour/minutes/seconds for dates in personal.ini. Don't really like it being that precise. Have attached relevant personal.ini (that is to say, without replace_metadata so it wouldn't be too large a file to look over, but otherwise complete).

Using the ini file provided, I get the proper dates:

Spoiler:

The Knights Errant by Zam
Story ID: 374200
Status: Unknown
Published: 2016-01-24
Updated: 2016-04-08
Packaged: 2016-04-22

The Commission by Xomniac
Story ID: 341476
Status: Unknown
Published: 2015-05-24
Updated: 2016-03-06
Packaged: 2016-04-22

The only two things that come to mind are that you are changing the dates in replace_metadata some how (or another part of the ini you didn't post), or if you've added user/pass for forums.spacebattles.com it might conceivably be using a different that parses differently. (FYI, you don't need user/pass for forums.spacebattles.com.)

[edeniz]

Originally Posted by JimmXinu

Using the ini file provided, I get the proper dates:

Spoiler:

The Knights Errant by Zam
Story ID: 374200
Status: Unknown
Published: 2016-01-24
Updated: 2016-04-08
Packaged: 2016-04-22

The Commission by Xomniac
Story ID: 341476
Status: Unknown
Published: 2015-05-24
Updated: 2016-03-06
Packaged: 2016-04-22

The only two things that come to mind are that you are changing the dates in replace_metadata some how (or another part of the ini you didn't post), or if you've added user/pass for forums.spacebattles.com it might conceivably be using a different that parses differently. (FYI, you don't need user/pass for forums.spacebattles.com.)

Very strange. Because, no, I'm not changing dates in replace_metadata. Whyever (or even how?) would someone do that? And I'm not even a member of spacebattles, so I don't have user/pass to add to personal.ini. I mean it, there is nothing in the truncuated parts but replace_metadata for mostly ao3, tth, and ffnet. Even stranger: I too have tried to download with the very same truncuated ini, and the result was same as the epub I previously attached.