[GUI Plugin] FanFictionDownLoader

[FaceDeer]

That's correct, the links are one-way from group to story. Stories can be added to groups without the involvement of the author so it wouldn't really be proper for them to appear on the story page itself, the author might not appreciate it if the StoriesThatSuck group added it or whatever (there actually was a problem with a group along those lines on FiMFiction for a while until it got shut down by the administrators).

I was also trying to make sure this suggestion was useful in contexts other than just FiMFiction, too, and I figured it was hard to get more generic than "here's a list of fanfic links, apply this category to them when you download/update them."

Anyway, having a temporary mark applied to the most recent batch of downloads would be a good middle-ground option, I think. It's an extra step but it would allow me to add group tags to a completely different special column devoted entirely to that rather than mixing it in with the generic tags, which would make the tagging safe from harm should I ever accidentally or intentionally take off the "keep existing tags" option.

[JimmXinu]

Here's a test version that uses calibre's 'marked' feature to mark all added, updated and even failed books.

There's a new config option "Show added/updated books when finished". When checked, FFDL will automatically display all successfully added/update books when it finishes. (They will display in the order they were given--convenient for series or series like sets.)

FFDL will always mark the books, even if you don't have that checked, so you can also manually search for 'marked:ffdl_success'. 'marked:ffdl_failed' is also available, or search 'marked:ffdl' for both.

calibre only has one 'marked' field, so these are available only until you run FFDL again or something else that uses it. View List from Reading List, Quality Check, etc.

UPDATE Mar 13 13 - Remove obsolete beta versions

[FaceDeer]

Works great, thanks!

[JimmXinu]

Here's another, newer version with the 'marked' feature.

This version actually implements keeping the story order correctly and 'marked:ffdl_success' and 'marked:ffdl_failed' in all cases instead of just most cases.

However, it also makes several changes to the way things are handled internally, so there's more risk of bugs in previously solid features.

UPDATE Mar 14, 2013 - Remove obsolete beta versions

[JimmXinu]

Yet Another Test Version--this one performs the error column library update on all-failed more correctly. Which changes how the all-failed case is reported to everybody, not just the ones who use error column. But it also makes it the same as the successful case, so I'll call it an improvement.

As always, please give it a try and report if you have any problems.

Thanks.

UPDATE Mar 15, 2013 - Remove obsolete beta versions

[JimmXinu]

Version 1.7.13 - 15 Mar 2013

• Remove BBCode--fimfiction.net adapter doesn't use it anymore.
• Add "add_to_" feature to ini config. Allow higher priority sections to *add* to any ini param rather than replace it.
• Add calibre CLI.
• Add "marked:ffdl_success/failed" to added/updated/failed books.
• Add "Show added/updated" pref using above.
• Preserve log page in epub across overwrites as well as updates.
• Drag and drop emails from Thunderbird on FFDL URL box to add URLs.

Command Line Interface(CLI)

Starting with calibre 0.9.23, calibre offers a command-line interface (CLI) for calling plugins.

This plugin supports that option and allows the user to call the plugin's basic functionality from outside calibre without having to install Python or get the plugin's files separately.

(Note that at this time, FFDL via calibre CLI will use your personal.ini settings from your library, but does not add to or update your library.)

Details:

Usage: calibre-debug.exe --run-plugin FanFictionDownLoader -- [options] <storyurl>

Options:
  -h, --help            show this help message and exit
  --library-path=LIBRARY_PATH, --with-library=LIBRARY_PATH
                        Path to the calibre library. Default is to use the
                        path stored in the settings.
  -f FORMAT, --format=FORMAT
                        write story as FORMAT, epub(default), mobi, text or
                        html
  -c CONFIG, --config=CONFIG
                        read config from specified file(s) in addition to
                        calibre plugin personal.ini,
                        ~/.fanficdownloader/personal.ini, and ./personal.ini
  -b START, --begin=START
                        Begin with Chapter START
  -e END, --end=END     End with Chapter END
  -o NAME=VALUE, --option=NAME=VALUE
                        set an option NAME=VALUE
  -m, --meta-only       Retrieve metadata and stop.  Or, if --update-epub,
                        update metadata title page only.
  -u, --update-epub     Update an existing epub with new chapters, give epub
                        filename instead of storyurl.
  --update-cover        Update cover in an existing epub, otherwise existing
                        cover (if any) is used on update.  Only valid with
                        --update-epub.
  --force               Force overwrite of an existing epub, download and
                        overwrite all chapters.
  -l, --list            Get list of valid story URLs from page given.
  -d, --debug           Show debug output while downloading.

[seabream]

I’m having three categories of difficulty:
reviews/favs/follows >3 digits long, strings containing “()” and strings containing “&”.

I’m using Mac OS 10.7.5, calibre 0.9.22, and Fanfiction Downloader 1.7.12

1) I’ve got custom columns for reviews/favs/follows on Fanfiction.net. I formatted them the same way that the instructions for the Count Pages plugin tells us to format the custom columns for “words” and “pages” i.e.: {0:,}
When the number of reviews (or favs or follows) is shorter than four digits, this works fine, when there are more, I get : “invalid literal for int() with base 10: ‘1,276’” where 1,276 is the number of reviews in question. Does FanFiction Downloader handle numbers differently than Count Pages does? Apologies if this counts as a silly question to someone who knows what the formatting means. I looked in http://docs.python.org/2/library/string.html, and failed to find an answer.
Sample link for a shortish story: http://www.fanfiction.net/s/6483644/1/
Also note, that other custom columns settings after reviews were aborted. Meaning, that the numerical ones showed undefined, despite being shorter than four digits, and assorted replacements that would normally happen didn’t.

2) With round brackets in fandoms and character names on AO3, I haven’t been able to get anything to work successfully. I’ve tried \(, [(] and even .* for the segment containing the brackets e.g.:
characters=>(^Henry.*$)=>Henry Mills&&category=>Once Upon a Time.*
to try to get character and variants: “Henry (Once Upon a Time)” on stories in fandom: “Once Upon a Time (TV)”
In all cases, it’s as though the line doesn’t exist.
Here’s a sample story link: http://archiveofourown.org/works/545044

3) I’m doing a character format replacement that works fine in fandoms without an &, and weirdly for one that does.
So here’s a working example with site: fanfiction.net, fandom: Pitch Perfect, character Chloe B., and a non-working one with fanfiction.net, fandom: Rizzoli & Isles, character: Jane R.

I have four places I want characters to show up: 1) Tags, 2) #characters custom column, 3) Titlepage, 4) logpage.

In Tags (1), I want the format “ch fandom: character” (e.g.: ch Pitch Perfect: Chloe Beale), in the other three I want “fandom.character” (e.g.: Pitch Perfect.Chloe Beale)

I’ve got this to work by having:

extra_valid_entries:edchars
edchars_label: Characters by fandom
include_in_edchars:characters

custom_columns_settings:
edchars=>#characters,n

replace_metadata:
characters=>(^Chloe B\.$)=>Chloe Beale&&category=>Pitch Perfect
characters=>(.+)=>ch Pitch Perfect: \1&&category=>Pitch Perfect
edchars=>(ch Pitch Perfect: )=>Pitch Perfect.
characters=>J\. Rizzoli=>Jane Rizzoli&&category=>Rizzoli & Isles
characters=>(.+)=>ch Rizzoli & Isles: \1&&category=>Rizzoli & Isles
edchars=>(ch Rizzoli & Isles: )=>Rizzoli & Isles.

and putting edchars into tittlepage_entries and extra_logpage_entries

For fandoms without an & this works fine. For some reason, with Rizzoli & Isles, it doesn’t.

I can’t get the four to match up correctly.
a) if edchars=>(ch Rizzoli & Isles: )=>Rizzoli & Isles.
I get 1, 3, 4, right, and 2 wrong, i.e.: 1 and 2 prints “ch fandom: character”, and 3 and 4 shows “fandom.character”
b) if edchars=>(ch Rizzoli & Isles: )=>Rizzoli & Isles.
I get 1, 2, 3, 4 right, but the title page doesn’t show in the book details, or if there is already a title page, it doesn’t update properly.
c) if edchars=>(ch Rizzoli & Isles: )=>Rizzoli & Isles.
I get 1 and 2 right, and 3 and 4 wrong. i.e.: 1, 3, 4, as “ch fandom: character” and 2 as “fandom.character”
d) if edchars=>(ch Rizzoli & Isles: )=>Rizzoli & Isles.
I get the same results as c.

Why? And how do I fix it?

Here’s a sample link for Rizzoli & Isles: http://www.fanfiction.net/s/8813042/1/
Here’s a sample link for Pitch Perfect: http://www.fanfiction.net/s/8770999/

[JimmXinu]

Originally Posted by seabream

When the number of reviews (or favs or follows) is shorter than four digits, this works fine, when there are more, I get : “invalid literal for int() with base 10: ‘1,276’” where 1,276 is the number of reviews in question.

FFDL doesn't treat favs, etc as numbers--it treats them as strings. So when calibre calls int('456') it works, but not on int('1,234').

I suppose I should change those to a number--or at least strip non-digits--for users who want to use non-US-en number formatting.

In the mean time, you could use replace_metadata to strip ',' from those entries.

Originally Posted by seabream

2) With round brackets in fandoms and character names on AO3, I haven’t been able to get anything to work successfully. I’ve tried \(, [(] and even .* for the segment containing the brackets e.g.:
characters=>(^Henry.*$)=>Henry Mills&&category=>Once Upon a Time.*
to try to get character and variants: “Henry (Once Upon a Time)” on stories in fandom: “Once Upon a Time (TV)”
In all cases, it’s as though the line doesn’t exist.
Here’s a sample story link: http://archiveofourown.org/works/545044

I cannot duplicate this problem. Parans '()' in both the character and category work fine for me. Unless there's a difference between Mac and Windows, I'd suspect you're somehow not applying, maybe? Missing space at the start of a line or something?

I don't even understand the question yet for your part 3. I'll need more time to read that again a couple times.

[JimmXinu]

@seabream -

Attached is a version that fixes your problems 1 & 3.

I cannot replicate your problem 2.

Run these fake story URLs in FFDL. They demonstrate issues 2 & 3 without having to hit real servers:
http://test1.com?sid=80 [Rizzoli & Isles]
http://test1.com?sid=81 [Pitch Perfect]
http://test1.com?sid=82 [Henry (Once Upon a Time), Once Upon a Time (TV)]

This is the replace_metadata code I was testing with:

extra_valid_entries:edchars
edchars_label: Characters by fandom
include_in_edchars:characters

add_to_custom_columns_settings:
 edchars=>#characters

replace_metadata:
 characters=>(^Henry.*$)=>Henry Mills&&category=>Once Upon a Time.*
 characters=>(^Chloe B\.$)=>Chloe Beale&&category=>Pitch Perfect
 characters=>(.+)=>ch Pitch Perfect: \1&&category=>Pitch Perfect
 edchars=>(ch Pitch Perfect: )=>Pitch Perfect.
 characters=>(^J\. Rizzoli$)=>Jane Rizzoli&&category=>Rizzoli & Isles
 characters=>(.+)=>ch Rizzoli & Isles: \1&&category=>Rizzoli & Isles
 edchars=>(ch Rizzoli & Isles: )=>Rizzoli & Isles.

You do have to use '&' rather than just '&' for now obscure reasons related to the early days of FFDL. But at least it's consistent now.

Do you realize that your category.character system doesn't work for crossover stories? Fake story http://test1.com?sid=83 demos a Rizzoli & Isles X Pitch Perfect story. Each character ends up with both categories: "Rizzoli & Isles.Pitch Perfect.Chloe Beale, Rizzoli & Isles.Pitch Perfect.J. Rizzoli"

If you're already creating lines for each character name filtered by category, that's probably where you need to add the category to the character, too.

UPDATE Mar 16, 2013 - Remove obsolete beta versions

[JimmXinu]

Here's a new test version that also address some new fimfiction.net changes.

UPDATE Mar 18 2013 -- Remove obsolete beta versions

[Linwe]

I don't know if this is a problem related to something on my end, or if it's just something to deal with every update, but my personal.ini settings are being changed back to the default settings the last three times I've updated FFDL. I've had to keep a txt file on my computer to copy and paste to/from. I thought that the personal.ini wasn't supposed to be affected during updates? Have I misunderstood somewhere or is calibre glitching my settings somehow?

I updated calibre to 0.9.23 before I updated FFDL to 1.7.13.

[JimmXinu]

@Linwe,

When you say update, do you mean calibre update or FFDL update?

What is your OS type and version? Are you using 32 or 64bit calibre? Are you doing anything atypical, like a USB stick install?

FFDL keeps personal.ini and all it's settings in the calibre database for each library. Many other plugins store their config the same way. You should not need to re-enter them after updates.

[Linwe]

@Jimm,

I updated both calibre and FFDL, but it's been the last few FFDL updates that's reverted the personal settings. I'm using Windows 7 64-bit; 64bit calibre. I have calibre portable on a USB stick, but it's the non-portable version on my computer that I use primarily for everything (portable is more for the organization when I travel between places) and that has reset itself.

I didn't think I needed to re-enter them, I hadn't for other updates just these last couple that have been off for me.

[JimmXinu]

That's the same setup I use. Are you counting the beta versions I post when you say the last couple updates, or just official releases?

How FFDL stores it's prefs hasn't changed, but I did make a change to where in the code it does it. But that's only just been released in 1.7.13.

Looking back, do I see in your past posts that you used to edit the contents of the plugin zip file? Might you have ended up with more than one copy of the zip file?

[Linwe]

I can only assume it'd be the official releases - I usually don't update until calibre tells me there are plugin updates waiting.

I use to edit the zip yes, because silly me couldn't figure out how to use the personal.ini. Once I did I uninstalled and then reinstalled to prevent "cross contamination" so-to-speak. Maybe that's what I need to do again, just to clean the slate. :/