[GUI Plugin] FanFicFare

[BookJunkieLI]

Originally Posted by JimmXinu

So I'm working on a new feature that would be helpful for adding custom columns and populating them from existing epubs without having to go out to the sites again.

Specifically, a feature to:

• First, invisibly save all the metadata FanFicFare collects in the epub--even if you don't use it all, and;
• Second, let you update calibre metadata from that saved metadata in the epub without going back out to the site.

Saving the metadata isn't too hard--calibre already does it whenever you save or send an epub.

But what's a good way to integrate part 2 into the FanFicFare PI user interface? So far, I'm thinking about:

1. A new menu option at the same level as 'Download for URLs' and 'Update Existing FanFiction Books'?
2. A new 'If Story already Exists' option? In addition to 'Update EPUB if New Chapters', etc, and 'Update Calibre Metadata Only' add 'Update Calibre Metadata from EPUB'?
3. Change the behavior of the existing 'Update Calibre Metadata Only' option to use the epub metadata whenever available and only revisit the site when it's not available. (Also insert metadata into epub when it has to revisit?)
4. An checkbox option, either in config or on the add/update dialog that modifies 'Update Calibre Metadata Only' to either use existing epub metadata or goes out to the site?

Any suggestions? Opinions?

I'm greatly in favor of this idea. I also like @ILB's suggestion

Originally Posted by ILB

How about to change 'Update Calibre Metadata Only' to 'Update Calibre Metadata from webpage' and add 'Update Calibre Metadata from EPUB'?

This way if someone is looking to do a bulk update they don't hit the website on the first go around at all. FFF does it's thing, you get a list back of the ones that updated and the ones that didn't and can then go back and work on updating through the site just the ones that need to be. Easy peasy if you have a FFF Log column.

As for @eschwartz's suggestion I can't really say one way or the other which would be better - stored to the epub or a cache db. I will say that I personally rarely use the Embed Metadata or Polish Books features and I've never used it on more than the book I'm currently working on.

[JimmXinu]

Version 2.2.4 - 29 Apr 2015

• More fixes for storiesonline.net site changes.
• Exclude FimF emoticons as covers by default.
• Fall back to smaller cover image in FimF when there's a problem with the larger linked-to cover.
• Detect and reject animated gifs in Calibre. They don't work well with the included Calibre image lib. FimF's new fall back will be activated.
• Fix for an error in Spanish translation that caused an exception.
• Make 'include_images' on by default, remove config checkbox.
• Enhance/Clarify Calibre Cover update and Cover Generation.
• Add feature to use Calibre's built-in Generate cover.
• Improve Reject listing.

[Tanjamuse]

I've come across a very odd problem. This search term no longer works for me:

#complete:"False"

As far as I can see I haven't done anything wrong, I've attached screenshots of my custom column and how it's configured in FFF.

Here's a link for a file it doesn't recognize as being incomplete:

http://archiveofourown.org/works/3697334

I also have a problem with some of my replaced metadata.

I have this in my personal.ini:

category=>The Kane Chronicles - Rick Riordan$=>The Kane Chronicles
category=>Kane Chronicles$=>The Kane Chronicles
category=>Kane Chronicles - Rick Riordan$=>The Kane Chronicles
category=>Kane Cronicles$=>The Kane Chronicles

But when I download stories, it says: The The Kane Chronicles.

Another one is:

characters=>Gil Grisson$=>Gil Grissom
characters=>Gill Grissom$=>Gil Grissom
characters=>Grissom$=>Gil Grissom

This one changes to: Gil Gil Grissom.

What can I have done wrong?

[chrnno]

Changing the lines to these should work if I what I think is happening is true.

category=>Kane Chronicles$=>The Kane Chronicles
category=>The Kane Chronicles - Rick Riordan$=>The Kane Chronicles
category=>Kane Chronicles - Rick Riordan$=>The Kane Chronicles
category=>Kane Cronicles$=>The Kane Chronicles

characters=>Grissom$=>Gil Grissom 
characters=>Gil Grisson$=>Gil Grissom
characters=>Gill Grissom$=>Gil Grissom

Basically this is a simple ordered replacement so when it reaches your last line and replaces "Grissom" for "Gil Grissom" there is already a "Gil" before that.

You can be lazy too and simply add at the end a replacement for "Gil Gil Grissom" to turn into "Gil Grissom". I have done it at times while I figured out how to do otherwise and there is probably a few still left here.

And Jimn I don't think I posted yet so great work on the transition and thanks again for maintaining and adding to this great plugin, really awesome how useful it is and how you keep to it though all this time.

[Tanjamuse]

How do you explain this one:

category=>The Lord of The Rings \(Movies\)$=>Lord of The Rings (Movies)

It still places the "The" in front, even though it shouldn't.

[chrnno]

That should work properly so presumably there is another replacement somewhere doing it. I presume that if you remove it does it show as "The The"? If yes then use search to track down the other replacement.

[Tanjamuse]

category=>The Lord of The Rings \(Movies\)$=>Lord of The Rings (Movies)
category=>Heart's Journey$=>Lord of The Rings (Movies)
category=>Lord of the Rings \(2001 2002 2003\)$=>Lord of The Rings (Movies)
category=>Lord of the Rings \(2001\)$=>Lord of The Rings (Movies)
category=>Lord of the Rings \(Jackson\)$=>Lord of The Rings (Movies)
category=>Lord of the Rings \(Movies\)$=>Lord of The Rings (Movies)
category=>Lord of the Rings - FPF$=>Lord of The Rings (Movies)
category=>Lord of the Rings: Return of the King$=>Lord of The Rings (Movies)
category=>Lord of the Rings: The Two Towers \(Movie\)$=>Lord of The Rings (Movies)
category=>Lord of the Rings: Two Towers$=>Lord of The Rings (Movies)
category=>LOTR FPF$=>Lord of The Rings (Movies)
category=>The Lord of the RIngs - Movies$=>Lord of The Rings (Movies)
category=>The Lord Of The Rings Jackson movies$=>Lord of The Rings (Movies)

I don't understand removing the "The" in the first part, because it doesn't pick up that metadata and replace it then.

[chrnno]

I meant removing the entire line but this was presuming the problem was the same as previous so disregard it.

Now that you posted your entire code though I suggest moving the first line to the end as that will allow for the possibility of any of the lines 3 to 10 being preceded by a 'The'* and might be the problem here though even if it isn't then at least covers for the future. If not then what story are you using so I can see the original metadata? Or just post that works too.

*Basically if the original metadata is, for example, "The Lord of the Rings: Return of the King" As you have written it then the result would be "The Lord of The Rings (Movies)".

[JimmXinu]

Originally Posted by Tanjamuse

I've come across a very odd problem. This search term no longer works for me:

As far as I can see I haven't done anything wrong, I've attached screenshots of my custom column and how it's configured in FFF.

Here's a link for a file it doesn't recognize as being incomplete:

http://archiveofourown.org/works/3697334

It correctly comes up as In-Progress for me, on the title page inside, in tags and in a y/n custom column. Does it show status correctly on the title page inside the ebook? That will tell if it's something in replace_metadata or after that.

Originally Posted by Tanjamuse

I also have a problem with some of my replaced metadata.

I have this in my personal.ini:



But when I download stories, it says: The The Kane Chronicles.

Another one is:



This one changes to: Gil Gil Grissom.

What can I have done wrong?

(Please use \[code\] markup around your ini blocks. When you use quote they are removed when others quote your message.)

chrnno is correct that the ordering is tripping you up.

However, I believe I've explained before that you can use the real power of regular expressions to reduce the number of replacement lines you have considerably. For example, I think these two lines will match all of the cases you showed:

 category=>^.*Kane Ch?ronicles.*$=>The Kane Chronicles

 characters=>^.*Grisso[mn]$=>Gil Grissom

If you're planning to continue adding a line to exactly match every variation you come across, I suggest using a leading '^' (beginning of string marker) as well as the '$' (end of string marker) that you're already using. That should help with your 'Gil Gil' and 'The The' issues regardless of order.

[JimmXinu]

Originally Posted by chrnno

And Jimn I don't think I posted yet so great work on the transition and thanks again for maintaining and adding to this great plugin, really awesome how useful it is and how you keep to it though all this time.

Thanks.

[chrnno]

Originally Posted by JimmXinu

However, I believe I've explained before that you can use the real power of regular expressions to reduce the number of replacement lines you have considerably. For example, I think these two lines will match all of the cases you showed:

 category=>^.*Kane Ch?ronicles.*$=>The Kane Chronicles

 characters=>^.*Grisso[mn]$=>Gil Grissom

If you're planning to continue adding a line to exactly match every variation you come across, I suggest using a leading '^' (beginning of string marker) as well as the '$' (end of string marker) that you're already using. That should help with your 'Gil Gil' and 'The The' issues regardless of order.

On that note was thinking of how to get every reasonable variation of say Harry Potter for example to replace to that, would this generally work?

 category=>(^.*Harry .*Pott?er.*$|^.*HP.*$)=>Harry Potter

And then realized people also use author's name so...?

 category=>^.*Rowling.*$=>Harry Potter

And because once I started thinking on it I couldn't help it. To get all variations of Lord of the Rings and The Hobbit when including Tolkien, book or novel change to Lord of the Rings(Book) and The Hobbit(Book) but otherwise to Lord of the Things(Movies) and The Hobbit(Movies) would this work?

 category=>^.*Hobbit.*(Tolkien|Book|Novel).*$=>The Hobbit(Book)
 category=>^.*Lord of the Rings.*(Tolkien|Book|Novel).*$=>Lord of the Rings(Book)
 category=>^.*Hobbit.*$=>The Hobbit(Movie)
 category=>^.*Lord of the Rings.*$=>Lord of the Rings(Movie)

[JimmXinu]

Originally Posted by chrnno

On that note was thinking of how to get every reasonable variation of say Harry Potter for example to replace to that, would this generally work?

 category=>(^.*Harry .*Pott?er.*$|^.*HP.*$)=>Harry Potter

If you're willing to assume there aren't other HP titles you'll ever want. That would also catch 'Maximizing HP', for example. (Hit points or horse power--take your pick.)

Originally Posted by chrnno

And then realized people also use author's name so...?

 category=>^.*Rowling.*$=>Harry Potter

Again, with the caveat that not all Rowlings are JK. Just most.

Originally Posted by chrnno

And because once I started thinking on it I couldn't help it. To get all variations of Lord of the Rings and The Hobbit when including Tolkien, book or novel change to Lord of the Rings(Book) and The Hobbit(Book) but otherwise to Lord of the Things(Movies) and The Hobbit(Movies) would this work?

 category=>^.*Hobbit.*(Tolkien|Book|Novel).*$=>The Hobbit(Book)
 category=>^.*Lord of the Rings.*(Tolkien|Book|Novel).*$=>Lord of the Rings(Book)
 category=>^.*Hobbit.*$=>The Hobbit(Movie)
 category=>^.*Lord of the Rings.*$=>Lord of the Rings(Movie)

That particular sequence isn't going to do what you want because category=>^.*Hobbit.*$=>The Hobbit(Movie) will also apply to the The Hobbit(Book) you just changed to above it. Ditto for movie.

The best way to find out what regex will do is try them. And FanFicFare will help you out there. In personal.ini you can define test stories and give them whatever metadata you want to try out:

[teststory:1003]
title:Some Other Title
category_list:Thor (Movies),Marvel Cinematic Universe,The Avengers (Marvel Movies),Captain America (Movies),xyz,Harry Potter,a little bit fluff,angst and fluff,Hurt/Comfort,Percy Jackson and the Olympians & Related Fandoms - All Media Types
characters_list:Fred Weasley,George,Frigga (mom),Wesley W.P.,Wesley WaP.
genre_list:Action,Romance
chaptertitles:Prologue,Chapter 1\, Xenos on Cinnabar,Chapter 2\, Sinmay on Kintikin,3. Chapter 3,Chapter 4,Chapter 5,Chapter 6,Chapter 7,Chapter 8
ships_list:Thor/Jane,Bob
rating:G

Then add/update-always/update-calibre with story url test1.com?sid=1003. Just make sure the replace_metadata you're trying to test are in [defaults] or [test1.com].

[chrnno]

Originally Posted by JimmXinu

If you're willing to assume there aren't other HP titles you'll ever want. That would also catch 'Maximizing HP', for example. (Hit points or horse power--take your pick.)

Again, with the caveat that not all Rowlings are JK. Just most.

Hum, true. Suppose I can do away with that and use more specific ones.

That particular sequence isn't going to do what you want because category=>^.*Hobbit.*$=>The Hobbit(Movie) will also apply to the The Hobbit(Book) you just changed to above it. Ditto for movie.

Spent a while in how to write that and then forgot why I was writing it to begin with... At least it is obviously easy to deal with.

The best way to find out what regex will do is try them. And FanFicFare will help you out there. In personal.ini you can define test stories and give them whatever metadata you want to try out:

[teststory:1003]
title:Some Other Title
category_list:Thor (Movies),Marvel Cinematic Universe,The Avengers (Marvel Movies),Captain America (Movies),xyz,Harry Potter,a little bit fluff,angst and fluff,Hurt/Comfort,Percy Jackson and the Olympians & Related Fandoms - All Media Types
characters_list:Fred Weasley,George,Frigga (mom),Wesley W.P.,Wesley WaP.
genre_list:Action,Romance
chaptertitles:Prologue,Chapter 1\, Xenos on Cinnabar,Chapter 2\, Sinmay on Kintikin,3. Chapter 3,Chapter 4,Chapter 5,Chapter 6,Chapter 7,Chapter 8
ships_list:Thor/Jane,Bob
rating:G

Then add/update-always/update-calibre with story url test1.com?sid=1003. Just make sure the replace_metadata you're trying to test are in [defaults] or [test1.com].

Yeah I know, going to do that now as the number I will be doing pretty much guarantees a mistake somewhere. Just wanted to make sure I could use it this way and any weird metadata that resulted was me not considering properly what I wrote instead of using regexp wrong.

And thank you for the answer and so quickly at that.

[zraith]

Receiving a NoneType error ('NoneType' object has no attribute '__getitem__') for stories from FicWad. This is with both the previous 2.2.3 version and after updating today also with the newer 2.2.4 version.

[JimmXinu]

Originally Posted by zraith

Receiving a NoneType error ('NoneType' object has no attribute '__getitem__') for stories from FicWad. This is with both the previous 2.2.3 version and after updating today also with the newer 2.2.4 version.

ficwad.com has changed the URLs they use for authors. New test version with fix has been posted.