[GUI Plugin] FanFictionDownLoader

[Jade Aislin]

Originally Posted by JimmXinu

There's no way I can think of to do that with the regex replacement feature. But since I've been playing with some additional metadata config options anyway, I'll try adding this.

Attached is a version with new options:

• sort_ships option: Reorder ships so b/a and c/b/a become a/b and a/b/c. Only separates on '/', so use replace_metadata to change separator first if needed. Something like: ships=>[ ]*(/|&|&)[ ]*=>/ You can use ships_LIST to change the / back to something else if you want.

add_to_replace_metadata:
  ships=>[ ]*(/|&|&)[ ]*=>/
# Change ships with ' & ' and ' / ' to '/' separated.

# sort the ships.
sort_ships:true

I already had code to convert anything that isn't a '/' in ships. It's probably not as clean as your code is, but it worked for what I needed.

replace metadata:
 ships=> (&|and) (.*?)$=>/\2 (Friendship)
 ships=>&(.*?)$=>/\1 (Friendship) 
 ships=> (&|and) =>/
 ships=>&amp=>/
 ships=>(( )?(x)( )?)=>/

The sorting worked perfectly. I had been worried that it would sort the ships before making any regex changes. However, I noticed it incorrectly sorted a ship which had a character with more than one name that I had changed to use a '/' to separate them. Once I got rid of the '/' in the character, it worked great.

Thanks for making this change, it's going to greatly reduce my personal ini.

- Jade Aislin

[Dylan Tomorrow]

Dear JimmXinu,

I have now more than 300 fanfics in my calibre collection, all downloaded with FFDL. It's probably among my most used calibre plug-ins at this point. I still love how easy this makes getting fanfics onto my Kindle.

Yet there is still one irk I've been wondering if you could fix (dunno how hard to implement this would be):

Almost all the fanfics I download don't have proper typography (and I am pretty OCD about that! ), so I routinely manually apply Smarten Punctuation via the great Polish Books feature calibre has had for a few months now. This can get pretty tedious, especially with updating still ongoing stories where if I forget to do this just once, the resulting AZW3/KF8 file will have the positions of all my notes and marks disarranged (The notes are positioned by absolute byte counts unfortunately. Srsly, Amazon? ).

So could you integrate the Smarten Punctuation feature (and maybe others from Polish Books if there is a need with other users?) into FFDL like you did with the plug-ins Generate Cover, Reading List and Count Pages?

So that I could, say, make FFDL routinely auto-apply Smarten Punctuation after every new download and/or overwrite-if-newer update of a fanfic ?

That would rock!

Cheers,
MaxManuka.

[JimmXinu]

Dylan, are you running Polish Book before or after converting to AZW3/KF8?

There's a good chance I can call Polish Book from FFDL, but not after Convert.

[JimmXinu]

This version can call Smarten Punctuation on each EPUB download/update. Look for the option on the Basic config tab.

It mildly messes up the Update Log for story summaries--the smartened summary may not match the actual summary and therefore it will be included every update.

Let me know how this works for people.

UPDATE Jul 18, 2013 - Remove obsolete beta versions

[Dylan Tomorrow]

Yes, I do apply Smarten Punctuation directly on the EPUB before converting.

I should have checked back on this thread earlier I guess! Sorry I didn't clarify.

I shall test this immediately. Incidentally, one of the fanfics I follow has a new update out today. Let's see …

… aaand it worked flawlessly \o/. No errors in the download log and the punctuation is smartened just like when I apply it manually. Jimm, thank you a lot for implementing this, and so quickly as well! :-)

[JimmXinu]

Great. Oh, I should also mention, that test version (2 posts back from this) has another new feature I've been playing with:

You can drag and drop story urls, text containing story urls, or even emails (like from Thunderbird) containing story urls onto the FFDL plugin button. And drag and drop existing books in the library onto it for update.

It's very similar to how you d-n-d those onto the edit box in the Add dialog, but skips the step of opening the Add dialog first.

[JimmXinu]

Version 1.7.31 - 18 Jul 2013

• Additional drag and drop. You can drag and drop story urls, text containing story urls, or even emails (like from Thunderbird) containing story urls onto the FFDL plugin button. And drag and drop existing books in the library onto it for update. It's very similar to how you d-n-d those onto the edit box in the Add dialog, but skips the step of opening the Add dialog first.
  
• Smarten Punctuation - FFDL can run the calibre Polish Book's Smarten Punctuation on new and updated epubs automatically. Only on calibre 0.9.39 and newer.
  
• sort_ships option: Reorder ships so b/a and c/b/a become a/b and a/b/c. Only separates on '/', so use replace_metadata to change separator first if needed. Something like: ships=>[ ]*(/|&|&)[ ]*=>/ You can use ships_LIST to change the / back to something else if you want.
  
• join_string_<entry> option: FFDL list entries are comma separated by default. You can use this to change that. For example, if you want authors separated with ' & ' instead, use join_string_calibre_author:\s&\s. (\s == space)
  
• keep_in_order_<entry> options: FFDL sorts list entries by default (except for author/authorUrl/authorId). But if you want to use an extra entry derived from author, it ends up sorted. For example, if you added calibre_author: keep_in_order_calibre_author:true
  
• replace_metadata <entry>_LIST options: FFDL replace_metadata lines operate on individual list items for list entries. But now if you want to do a replacement on the joined string for the whole list, you can by using <entry>_LIST. Example, if you added calibre_author: calibre_author_LIST=>^(.{,100}).*$=>\1

[JimmXinu]

Naturally, there's a new problem reported right after I post a new version.

thehexfiles.net allows HTML tags in story titles. Epub/calibre/etc don't allow that.

Attached version strips out HTML from story titles (and author names) on thehexfiles.net.

UPDATE Jul 23, 2013 - Remove obsolete beta versions

[Jade Aislin]

I seem to be having problems with the testing feature now. I copied the example Jim gave in a previous post. First I added the [test story:defaults], then I added the [test.com].

My code is:

[test1.com]


replace_metadata:
 ships=>(( )?(-)?( )?[Bb]rother(s)?(ly )?(love)?|( )?(-)?( )?[Ff]amil(y)?(ial)?( Relationship)?)=>\1 (Family)
 

titlepage_entries: seriesHTML,tcategory,characters,ships,genre,warnings,status,numChapters,numWords,rating,datePublished,dateUpdated,description,site

[teststory:1002]
ships_list:brotherly love Spock/Kirk, Sess/Rin family, Ori/Dori Familial Relationship, Bilbo/Balin - Brothers

However, when I created a story using http://test1.com?sid=1002, many of the columns did not get populated (category, characters, ships, series, warnings). Most of those tags were specified in the defaults section. Am I doing it incorrectly? Although I am testing the ships out, I thought the other columns would fill with what was already given.

[JimmXinu]

Jade,

Looks like I forgot to get all of the entries into the 'valid_entries' list on the default. Including ships. I'll add it to the next version, but in the meantime, you can add one more line to your config to tell it to allow ships in test stories:

[teststory:1002]

add_to_valid_entries:,ships_list

[Jade Aislin]

Originally Posted by JimmXinu

Jade,

Looks like I forgot to get all of the entries into the 'valid_entries' list on the default. Including ships. I'll add it to the next version, but in the meantime, you can add one more line to your config to tell it to allow ships in test stories:

[teststory:1002]

add_to_valid_entries:,ships_list

That worked. Unfortunately, I'm having trouble placing the end tag with the ship sorting. I also ran into one instance where it did not sort.

This is my code with my testing entries:

[test1.com]

replace_metadata:
 ships=>(( )?(-)?( )?[Bb]rother(s)?(ly )?(love)?|( )?(-)?( )?[Ff]amil(y)?(ial)?( Relationship)?)=> (Family)
 ships=>(\(Family\)|\(Friendship\))( )?(.*?)$=>\3 \1
 

[teststory:1002]
add_to_valid_entries:,ships_list

ships_list:brotherly love Spock/Kirk, Sess/Rin family, Ori/Dori Familial Relationship, Bilbo/Balin - Brothers, Thorin/Bilbo

Before the second replace ships would have placed the (Family) at the end of the string.

However when I tried it now, I got this:
Ships: Balin (Family)/Bilbo, Bilbo/Thorin, Dori (Family)/Ori, Rin (Family)/Sess, Spock/Kirk (Family)

It looks as though when the keywords for my tag is at the end, it is able to sort the ships. The only ship that didn't get sorted had keywords before the ship, although the tag did get placed at the end. When the ships sorted, it sorted the tag as well.

Is there any way to get the end tag (Family) at the end, even when the ship is sorted?

[JimmXinu]

Originally Posted by Jade Aislin

Before the second replace ships would have placed the (Family) at the end of the string.

However when I tried it now, I got this:
Ships: Balin (Family)/Bilbo, Bilbo/Thorin, Dori (Family)/Ori, Rin (Family)/Sess, Spock/Kirk (Family)

It looks as though when the keywords for my tag is at the end, it is able to sort the ships. The only ship that didn't get sorted had keywords before the ship, although the tag did get placed at the end. When the ships sorted, it sorted the tag as well.

Is there any way to get the end tag (Family) at the end, even when the ship is sorted?

The Spock/Kirk isn't becoming Kirk/Spock because you've left a space in front of Spock and space sorts before 'K'.

Yes, your family/etc tags are being included in the sort. It's just a string as far as the code is concerned; "Rin family" or "Rin (Family)" is no different from "Rin Smith".

However, you should be able to put your 'tags' at the end using replace_metadata with ships_LIST. Match "(Family|etc) name/other name" and swap it around.

Keep in mind that you're dealing with the whole list at that point, you can match on the commas between and end of string. If there are commas in any of the names it will cause problems, but could that be handled by changing join_string_ships and reverting it afterwards with another ships_LIST regex. (Or by removing commas before all your processing probably a better solution for calibre tag-like columns.)

You'll also need additional complexity for 3, 4, 5, etc ways if you want those to work the same.

[Jade Aislin]

Originally Posted by JimmXinu

The Spock/Kirk isn't becoming Kirk/Spock because you've left a space in front of Spock and space sorts before 'K'.

Okay. I figured out that problem. I changed the code to this:

replace_metadata:
 ships=>(( )?(-)?( )?[Bb]rother(s)?(ly )?(love )?|( )?(-)?( )?[Ff]amil(y)?(ial)?( Relationship)?( )?)=> (Family)
 ships=> \(Family\)(.*?)$=>\1 (Family)

Now the one that would not sort is sorting.

Originally Posted by JimmXinu

However, you should be able to put your 'tags' at the end using replace_metadata with ships_LIST. Match "(Family|etc) name/other name" and swap it around.

Keep in mind that you're dealing with the whole list at that point, you can match on the commas between and end of string. If there are commas in any of the names it will cause problems, but could that be handled by changing join_string_ships and reverting it afterwards with another ships_LIST regex. (Or by removing commas before all your processing probably a better solution for calibre tag-like columns.)

You'll also need additional complexity for 3, 4, 5, etc ways if you want those to work the same.

I think I finally figured it out.

I now have this code:

replace_metadata:
 #Convert keywords to end tags.
  ships=>(( )?(-)?( )?[Bb]rother(s)?(ly )?(love )?|( )?(-)?( )?[Ff]amil(y)?(ial)?( Relationship)?( )?)=> (Family)
  
#Place the end tag at the end of the ship.

 ships=> \(Family\)(.*?)$=>\1 (Family)
 

  #sort the names in the ships (code is in the default for all sites)
 
 #Place the end tag at the end of the ship using ships_LIST
 ships_LIST=>(\(Family\)|\(Friendship\))( )?(.*?),=>\3 \1,
 ships_LIST=>,(.*?)( \(Family\)|\(Friendship\))/(.*?)$=>,\1/\3\2


 #get rid of space before the '/'

  ships_LIST=> /=>/

It took me a while to figure out what you meant by commas. Then I had trouble figuring out how to deal with the last ship in the string.

Next up, reworking the code to separate ships with more than one end tag into their own ships (ie: A/B (Friendship) (Past), becomes: A/B (Friendship), A/B (Past).)

I'm hoping it will be as simple as using the code I had for ships with a comma at the beginning of the regex. However, I'll work on that another day.

I'm actually proud of myself. I rewrote this message three or four times because I figured out how to fix my problems with putting the end tag at the end.

Thanks for all the help.

[JOJO1]

Hello

I have a little problem.
We like to get the stories from "fanfiktion.de" with the calibre plugin.
But since about two weeks the plugin doesn't accept the URLs from the german site.
(no problems with fanfiction.net)

I have installed calibre and the plugin 1.7.31 on a new win7 pc und there is the same problem.

Perhaps you can help us.

[Firedancer885]

Jimm,

please check this:
http://www.fanfiktion.de/s/4e74e42f0001c5040650ea60/1

I got a 'NoneType' object has no attribute 'string' error