[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by edeniz

1) "Overwrite If Newer" does not seem to be working anymore when the format in question is "html". At least not on ffnet and ao3 both. The error message is,

Not Overwriting, web site is not newer.

...
If necessary, I could attach an example fic each for ffnet and ao3 or the debug log.

Yes. Please always include story URLs when reporting any problem. Debug log, when you have it, is also always a good idea.

Originally Posted by edeniz

2) tth seems to be on the fritz yet again.
...
And to clarify: I'm using the latest plugin version, but not the test version.

This continues to be a problem that doesn't happen when I try. The test version includes additional debug info that might help if you can capture a debug log while it happens.

Frankly, from the complete lack of debug logs showing the problem posted, I had assumed it had gone away. (And if what I suspect is happening is true, there isn't anything I can do about it...)

Originally Posted by edeniz

EDIT:
I forgot to mention something else as well, so better add it here instead of making another post, right?

I don't know what the official mobileread is policy on double posts, but I can tell you that I often don't see changes to past posts if I read them before they are changed. You got lucky this time.

Originally Posted by edeniz

exclude_metadata_post is supposed to be executed after replace_metadata, right? Well. It doesn't seem to work that way for me.

http://archiveofourown.org/works/246947[1]
The Jenny Code (Ch 1-1)
Authors: Domenika Marzione (domarzione)
...
Again, no idea if it is something new or if it's been this way for a while. It's not that it's supposed to be this way and I simply misunderstood what exclude_metadata_post is supposed to do, right?

The problem is that the order of application is in/exclude_pre, then replace_metadata, then in/exclude_pre per metadata entry.

So when the replace_metadata line for characters fetches the metadata for genre to check against 'Genii', it invokes in/exclude_pre, then replace_metadata, then in/exclude_pre for genre before doing the comparison.

Changing that now would involve a huge amount of testing that I'm not interested in doing for one corner case.

You can accomplish what you want with this instead:

[archiveofourown.org]

add_to_extra_valid_entries:,rawgenre
include_in_rawgenre:genre.NOREPL

replace_metadata:
 characters=>(.*)=>\1\,Genii&&rawgenre=>Genii

exclude_metadata_post:
 genre==Genii

If it's a common problem people are running into, I'm willing to consider adding code to allow for entry.NOREPL more widely to avoid having to add rawgenre.

[eschwartz]

Originally Posted by JimmXinu

I don't know what the official mobileread is policy on double posts, but I can tell you that I often don't see changes to past posts if I read them before they are changed. You got lucky this time.

Most people are the same.

I am fairly positive that MobileRead policy is "make a new post if you want".
And I DO want to post a new post if more than ~10 minutes has passed and I have significant new information to add.

[Firedancer885]

Jimm,

I downloaded 3 different stories from tth today and had no problems. I had no tth problems for the last 6 months. (If this helps any.)

[JimmXinu]

Originally Posted by Firedancer885

I downloaded 3 different stories from tth today and had no problems. I had no tth problems for the last 6 months. (If this helps any.)

It confirms I'm not the only one seeing no problems, thanks.

[edeniz]

Sorry for the late answer. Couldn't get to my computer before now.

Originally Posted by JimmXinu

Yes. Please always include story URLs when reporting any problem. Debug log, when you have it, is also always a good idea.

Ok, will do. Just didn't want to clutter should it turn out to be unnecessary. Attached is everything relevant.

Originally Posted by JimmXinu

This continues to be a problem that doesn't happen when I try. The test version includes additional debug info that might help if you can capture a debug log while it happens.

Frankly, from the complete lack of debug logs showing the problem posted, I had assumed it had gone away. (And if what I suspect is happening is true, there isn't anything I can do about it...)

Well, the tth thing at least turned out to be my own mistake. Apparently I had this line under the tthfanfic.org part of the personal.ini, smack dab under my username and password:

Spoiler:

 centeredcat_to_characters:true
 pairingcat_to_characters_ships:true
 romancecat_to_characters_ships:true

And as you can see, with freaking indent (no idea when that mistake happened), so it thought they were a part of the username/password process too. Since it's in the defaults I deleted them, of course, problem solved. (I'd put that there in December, when you first introduced these new changes, and forgotten to delete them now that it's turned on by default. But, did you know that those lines are marked red in the ini files as if they were an error?)

Originally Posted by JimmXinu

I don't know what the official mobileread is policy on double posts, but I can tell you that I often don't see changes to past posts if I read them before they are changed. You got lucky this time.

Oops. Thanks. I'll remember next time. I keep confusing in which forums it's okay to double post and in which it's frowned on.

Originally Posted by JimmXinu

The problem is that the order of application is in/exclude_pre, then replace_metadata, then in/exclude_pre per metadata entry.

So when the replace_metadata line for characters fetches the metadata for genre to check against 'Genii', it invokes in/exclude_pre, then replace_metadata, then in/exclude_pre for genre before doing the comparison.

Changing that now would involve a huge amount of testing that I'm not interested in doing for one corner case.

You can accomplish what you want with this instead:

[archiveofourown.org]

add_to_extra_valid_entries:,rawgenre
include_in_rawgenre:genre.NOREPL

replace_metadata:
 characters=>(.*)=>\1\,Genii&&rawgenre=>Genii

exclude_metadata_post:
 genre==Genii

If it's a common problem people are running into, I'm willing to consider adding code to allow for entry.NOREPL more widely to avoid having to add rawgenre.

Dear me, no, by all means. I didn't mean it like that. No need to change it on my behalf, especially not if it's a lot of work. I just assumed that I was probably doing something wrong, and couldn't figure out what I was missing. It's not really a problem. The solution idea you offered is more than enough. (It wouldn't even be an issue in the first place if ao3 authors didn't keep adding character or ship tags into the genre field. Half the time it's a good thing that ao3 has no limitations on the tags, the other half of the time it's a headache.)


Thank you for all your help

[JimmXinu]

Originally Posted by edeniz

Ok, will do. Just didn't want to clutter should it turn out to be unnecessary. Attached is everything relevant.
<attached files for Overwrite in HTML issue>

I don't see anything that jumps out. It's working fine for me, but I don't usually keep HTML format stories around.

When doing "Overwrite if Newer", FFF looks at the file system last-modified datetime(mtime from os.stat()) for the zip file calibre puts HTML in and compares it to the 'updated' date from the site.

However, if for any reason the modified time on the zip has been updated since, it can throw things off. I don't know of any reason why calibre would do so, but it could happen for other reasons.

I've attached a test version that includes some debug output to report the file and site times found.

(The file mtime includes time of day while for most sites 'updated' is just a date. So if updated date == mtime date and there's no updated time, "Overwrite if Newer" will assume the site is newer and overwrite. There can also be issues when the timezone on your computer and on the site differ.)

BTW, if you are receiving story update notices by email, there's no reason not to use Overwrite Always when you know there is new content.

Originally Posted by edeniz

... But, did you know that those lines are marked red in the ini files as if they were an error?

Thanks for pointing that out. I found a couple other ini/highlighting errors to fix too.

[edeniz]

Hmm. Strange, that. Even stranger that of the two ao3 fics I provided, one posed no problem, whereas the other did.

But, since you mentioned calibre's modified date throwing things off: would moving the file from one calibre library to another count then? Because that I did. (Also true that timezones differ, I think. Mine is gmt+1, not sure what the timezones of ao3 and ffnet are, probably some American timezone.)

In fact, because you mentioned it, I've tested it out: I moved a few files that have new chapters that I didn't download yet into a new library, and when I tried to update, even the epub file would not allow it unless I overwrote. So yeah, that might just be it as far as problem source goes. But, I'll install and use the test version anyway, and get back to you with its debug file.

But you're right, I do get email update notices, so might as well use overwrite always, no reason not to. As long as I can update and download at all, I'm happy.

[JimmXinu]

Originally Posted by edeniz

But, since you mentioned calibre's modified date throwing things off: would moving the file from one calibre library to another count then? Because that I did.

Absolutely. Calibre doesn't do move--it does copy and delete which would definitely reset the mod time.

Originally Posted by edeniz

In fact, because you mentioned it, I've tested it out: I moved a few files that have new chapters that I didn't download yet into a new library, and when I tried to update, even the epub file would not allow it unless I overwrote.

For epubs I recommend 'Update if New Chapters'. In fact, I generally recommend downloading/updating epubs and then converting to other formats. That's what I do for my Kindle.

[edeniz]

Originally Posted by JimmXinu

Absolutely. Calibre doesn't do move--it does copy and delete which would definitely reset the mod time.

Well, turns out that was it after all. I made a debug log with the test version you attached (see attachment), and according to it, the move to a different library made all the difference. I'll know better in future.

Originally Posted by JimmXinu

For epubs I recommend 'Update if New Chapters'. In fact, I generally recommend downloading/updating epubs and then converting to other formats. That's what I do for my Kindle.

Of course. Makes no sense to overwrite an epub when it can simply add new chapters. It takes much less time that way. I meant rather, if I make a boneheaded move and forget not to copy/move files before updating them. Because in that case, overwriting it is the only way, after all. As I said, now I know better, so hopefully it won't be an issue again.

Again, for all your help.

[DavidBrookes]

Not sure if anyone is having the same issue but I have had problems over the last two months with downloading stories from Sink Into Your Eyes. I have been able to sometimes download new stories but updating current ones has been unsuccessful. I have attached the log files from my last attempt to this post. Please help.

[JimmXinu]

Originally Posted by DavidBrookes

Not sure if anyone is having the same issue but I have had problems over the last two months with downloading stories from Sink Into Your Eyes. I have been able to sometimes download new stories but updating current ones has been unsuccessful. I have attached the log files from my last attempt to this post. Please help.

First off, you're trying to update 692 stories at once. That's waaaaaaaaay more than I recommend or condone.

That said, I've spot checked a few of the stories you had errors with:

Those with error 'NoneType' object has no attribute '__getitem__' are stories that no longer exist on the site. (Example). I will update the adapter to report those as StoryDoesNotExist.

Of those that gave you error HTTP Error 400: Bad Request and urlopen error, the 6 that I spot checked all worked fine. I believe you either were blocked by the site for excessive traffic or caused the site server to fail.

In short, don't try to update them all at once.

If you really think you must, try adding this to your personal.ini first so you don't kill the server:

[www.siye.co.uk]
slow_down_sleep_time:4

[susanpr]

I am attempting to update a few stories on AO3, and since this morning I am getting this statement in the update function in the Fanficfare plugin for Calibre.


Status
Title
Author
Comment
URL
Bad
The Kiss You Deserve
JackofSomeTrades
<urlopen error [Errno 10061] No connection could be made because the target machine actively refused it>
http://archiveofourown.org/works/5350130

When I try to use the fanficfare app on the web, it will grab the story. I have tried reinstalling the plugin, but no change. Can you help me get my plugin working again?

[JimmXinu]

Originally Posted by susanpr

I am attempting to update a few stories on AO3, and since this morning I am getting this statement in the update function in the Fanficfare plugin for Calibre.

<urlopen error [Errno 10061] No connection could be made because the target machine actively refused it>
http://archiveofourown.org/works/5350130

When I try to use the fanficfare app on the web, it will grab the story. I have tried reinstalling the plugin, but no change. Can you help me get my plugin working again?

I was able to download that story too. The site may have been having problems at the time.

Alternatively, how many updates have you tried to do recently? Because that refused connection sounds like your IP is being blocked by the site and excessive traffic is the only reason I know of they'd do that.

[susanpr]

Originally Posted by JimmXinu

I was able to download that story too. The site may have been having problems at the time.

Alternatively, how many updates have you tried to do recently? Because that refused connection sounds like your IP is being blocked by the site and excessive traffic is the only reason I know of they'd do that.

I haven't updated more than I normally do, maybe 10 or 15 stories, and no more than 5 at a time. I'm still getting the message. Maybe I'll try tomorrow

[Chrystalline]

I have a couple of might-be-silly questions:

Is it possible to export the reject URL list to transfer to another computer? I see where URLs can be added, but I don't see an easy way to export the rejected ones. I'm thinking about moving my entire Calibre database, and I know that someday down the road I'll junk this machine for a newer one, so being able to preserve the reject filter seems like a good idea to me. (and now I'm wondering if there's a way to use the list outside of FFF/Calibre to do the filtering in advance, to speed things up - not that I know how to go about that...)

Is it possible to set an option to "always do this" or "do this for any other conflicts in this session" with the "Change URL?" and "Download as New Story?" dialogs? I like to set FFF to downloading in the background while I'm doing something else, and I don't always notice that it's waiting for input, which makes me feel really stupid when I come back to it expecting it to have finished downloading, and it's still stuck on whether to change the URL. There is much facepalming on those days

Please forgive me if this is a simple thing I've missed; there's so much magic in this plugin, I don't know if I'll ever fully master it.