[GUI Plugin] FanFictionDownLoader

[Snowman]

That's good to hear. Bug hunting is so much easier with something tangible to work with.

And thank-you again for such a useful plugin.

Snowman

[JimmXinu]

Originally Posted by Snowman

That's good to hear. Bug hunting is so much easier with something tangible to work with.

Very much so. I've figured out what's happening, and I have a suspicion as why.

You're ending up with tags Fanfiction and FanFiction (note the second f/F). calibre 1.0 doesn't like that.

I've opened a calibre bug about it. The attached screen shot shows how to cause it outside FFDL.

Your existing books have tag Fanfiction. I suspect you changed the tag FanFiction to Fanfiction at some point across your library. In past, calibre silently folded them together, but now it's not.

In the meantime, if you go into your personal.ini and add:

[defaults]
extratags: Fanfiction

...that will change it to Fanfiction for you.

Unfortunately, I can't promise that the same fix will work for everyone, because if you have other tags in your library that have the same letters but different case as the sites you're downloading from, you'll still have the same issue.

I'm looking at ways to work around the bug in FFDL.

[Snowman]

Awesome - that did the trick. I made the amendment, then chose a random book, blanked some columns, including wordcount, then did a force update. It worked as expected.

Incidentally,I do not remember changing any tag. I rarely do; could it be that ffn itself has various versions of Fanfiction, FanFiction, etc in its own db?

What is interesting is that I then selected all book, went to bulk-edit->tag editor, and looked for Fanfiction. I expected to see both versions, but still only the one was in the total list of tags.

So, to summarise - if a tag exists somewhere, and an book update (not just ffdl) attempts to add a tag with a different case, then a partial update will (silently) occur. This is will probably be noticed during individual updates, but is likely to be overlooked during a bulk update.

I have a feeling that I saw something similar in the "backend db testers" thread...

The post in "backend db testers" is #227-230. In this case, a changed case in a tag could cause an update to unrelated books. So similar, but not the same.

best regards

Snowman

[JimmXinu]

Attached is a version of FFDL that works around the calibre 1.0 tag case bug.

I believe this will get around the ConstraintError issue for FFDL users until Kovid gets it fixed.

Please let me know both if it works and if it doesn't for you.

UPDATE Aug 25, 2013 - Remove obsolete beta versions

[Snowman]

Originally Posted by JimmXinu

Attached is a version of FFDL that works around the calibre 1.0 tag case bug.

I believe this will get around the ConstraintError issue for FFDL users until Kovid gets it fixed.

Please let me know both if it works and if it doesn't for you.

Yes - this also worked. I removed the ini 'extratags' entry, then installed the plugin update and restarted calibre. I then tried blanking columns for several books, and both an individual book update and a multiple book update worked as expected.

regards

Snowman

[Sefiriot]

Just tested the released update to fix ConstraintError problems: seems to have worked on the anthology I forced updated-- downloaded word count is updated and custom columns are updated, no error messages shown. I'll see later if it's resumed normal service with Reading List again.

[JimmXinu]

Originally Posted by JimmXinu

...

You're ending up with tags Fanfiction and FanFiction (note the second f/F). calibre 1.0 doesn't like that.

I've opened a calibre bug about it. The attached screen shot shows how to cause it outside FFDL.

Your existing books have tag Fanfiction. I suspect you changed the tag FanFiction to Fanfiction at some point across your library. In past, calibre silently folded them together, but now it's not.

The good news is Kovid has acknowledged and fixed the problem. You can click the bug link above to see it.

The bad news is next update of calibre isn't due out until Friday next week.

Due to the fact that FFDL *doesn't* throw and error out for the user to see (I'm not entirely sure why that is), I am going to release the version with the work around for the benefit of users who may not otherwise notice the problem.

(FYI, you don't need to have changed FanFiction to Fanfiction on every book. If you ever changed it on one book, it would have changed for all (assuming 0.9 treated that part the same), or if you already had a single book with tag Fanfiction before using FFDL.)

[JimmXinu]

Version 1.7.37 - 24 Aug 2013
Work around minor bug (1216398) in calibre 1.0.0. Tags 'A' and 'a' (for example) conflict and prevent metadata update.

[DavidBrookes]

Originally Posted by JimmXinu

Okay, I know what the problem is. It's something we've bumped into before when sites allow HTML tags in the title.

Up until now, I've be changing it for one site as a time as it becomes a problem. This time I've gone through and fixed it for all sites. But that ends up touching the adapters for 75 different sites.

So here's a test version for people to use for a bit to make sure I didn't break anything in the process. :-)

Just to let you know that I tried again with the updated plugin and Calibre 1.0.0 - no problems now - thank you

[Jade Aislin]

I've been working on placing the general replace_metadata in the defaults section. However, I'm now trying to make sure the title page is cleaned up as well.

Near the beginning of the defaults section, I have this code:

add_to_extra_valid_entries:,tcategory,tcharacters,tships
include_in_tcategory:category
include_in_tcharacters:characters
include_in_tships:ships


tcategory_label:Fandom
tcharacters_label:Character(s)
tships_label:Pairing(s)
 
add_to_replace_metadata:
 #take out the big sub headings to make the title page look neat
 tcategory=>(Book/Comics\.|Crossover\.|TV\.|Movie\.|Anime-Manga\.|Cartoon\.)=>
 tcharacters=>(Book/Comics\.|Crossover\.|TV\.|Movie\.|Anime-Manga\.|Cartoon\.)=>
 tships=>(Book/Comics\.|Crossover\.|TV\.|Movie\.|Anime-Manga\.|Cartoon\.)=>

titlepage_entries: seriesHTML,tcategory,tcharacters,tships,genre,warnings,status,numChapters,numWords,rating,datePublished,dateUpdated,description,site

This was supposed to delete the hierachical grouping, making the title page cleaner.

Further down, I have my usual add_to_replace_metadata for the categories, characters, etc. Now the above code is not working as it did before.

I thought it might not be working because it was before the rest of the replace_metadata. I tried placing this replace_metadata. However when I did that, my other replace_metadata code (for the category, characters, etc) stopped working.

I then tried to change the code to tcategory_LIST, but ran into the same problem.

I'm not sure what to try next. Any help would be appreciated.

[Sefiriot]

Is anyone else getting a little strange behaviour updating or adding fics already in calibre? When I update a single fic, whether anthology or not, for some reason the last_modified value on every other fic by the same author is getting changed as well, despite nothing being done to any other fic.

I updated a story and its last_modified value shows as <date>T03:01:02. The 46 other fics by the same author I have in calibre then show up with last_modified values <date>T03:01:01-- despite my not having done anything to actively change a thing about their metadata or book file! Nothing seems to be out of the ordinary in the logs. Something strange is happening here. Did I screw up updating my copy of the plugin somehow?

[JimmXinu]

Originally Posted by Jade Aislin

I've been working on placing the general replace_metadata in the defaults section. However, I'm now trying to make sure the title page is cleaned up as well.

Near the beginning of the defaults section, I have this code:

add_to_extra_valid_entries:...

Further down, I have my usual add_to_replace_metadata for the categories, characters, etc. Now the above code is not working as it did before.
...

You have more than one add_to_replace_metadata etc in the same section(defaults)? Because while they will add *between* sections, when in the same section only one of each key word will apply.

[FaceDeer]

This is interesting. FFDL is hanging at "updating calibre for FanFiction stories" whenever I try updating or adding stories to libraries on my laptop (only way out is to kill the program), but it seems to be working fine on my desktop. Unfortunately I've got to head out of the house for a bit right now, but I'll play around a bit tonight with debug logging and whatnot and see if I can figure out what's causing the difference. Anyone else see something like this with the most recent updates?

[JimmXinu]

Originally Posted by Sefiriot

Is anyone else getting a little strange behaviour updating or adding fics already in calibre? When I update a single fic, whether anthology or not, for some reason the last_modified value on every other fic by the same author is getting changed as well, despite nothing being done to any other fic.

I updated a story and its last_modified value shows as <date>T03:01:02. The 46 other fics by the same author I have in calibre then show up with last_modified values <date>T03:01:01-- despite my not having done anything to actively change a thing about their metadata or book file! Nothing seems to be out of the ordinary in the logs. Something strange is happening here. Did I screw up updating my copy of the plugin somehow?

On each add/update, FFDL updates the author's 'link field' in calibre. Commenting out that code shows that that is why the last_modified date is changed for all books by that author when any of them are updated.

Considering last_modified isn't normally visible in the GUI (at least I couldn't find it except for from CLI), does it really matter? And one could argue that it is accurate.

Looking in my library at older books that haven't been updated since calibre 1.0.0, I think that's always been the behavior.

[FaceDeer]

Heh. I actually do make use of the last_modified date, myself. I've got a library of in-progress stories that I'm interested in reading when finished but not actively following as they're written, and since it's uncouth to use FFDL to "check for updates" on a regular basis I created a custom column that uses last_modified to flag stories that haven't been checked in certain intervals for update (with an exponentially-increasing gap between check times, so that frequently-updated fanfics get "pinged" as quickly as once a week and long-quiescent ones only get checked every six months or more). This ensures that if a story stops updating it will only be checked a few extraneous times before being automatically "given up on." It never occurred to me that updating a fanfic would result in last_modified being changed on other documents that hadn't been flagged for updating.

Since generally each author will only have one or two stories in-progress simultaneously the situation may not have occurred frequently enough to be noticeable for me, and even if it had happened it wouldn't really have made much difference (it would simply "renew hope" on a fanfic that should be heading out into once-a-year land and check it a few extra times before it goes back out there), but it's a use-case if you were curious to have one so I figured I'd mention it.

If you want to be more rigorous anyway, perhaps make it so that FFDL doesn't modify other fanfics if the update wouldn't actually change the contents of the fanfic's author link field to a new value?