[GUI Plugin] FanFictionDownLoader

[davidfor]

Originally Posted by JimmXinu

Have you tried the obvious? Something like:

replace_metadata:
 characters=>Bob Smith=>Bob, Smith

It will still be considered one string "Bob, Smith" for any replace_metadata lines included after it, but when it hits calibre or is included in title_page, it will just be another comma separated item.

That was the first thing I tried. But, the comma is replaced by a semi-colon. Actually, it half works. I hadn't looked before. The title page has the comma separated list. The semi-colon is only in the tag.

[JimmXinu]

Originally Posted by davidfor

That was the first thing I tried. But, the comma is replaced by a semi-colon. Actually, it half works. I hadn't looked before. The title page has the comma separated list. The semi-colon is only in the tag.

Hmm. That works for custom columns because they are set as a string. Tags are kept as list objects in python. So calibre, when it sees a ',' in a tag entry is changing it to a ';'.

So there isn't an easy way to do it.

There is, however, a complicated way. You can make two copies of category in new entries, modify those, and then replace the original category with the two new ones. It's not intuitive, but because of the order things get included and replaced it does work.

extra_valid_entries:splitcat1,splitcat2
include_in_splitcat1:category
include_in_splitcat2:category
include_in_category:splitcat1,splitcat2

replace_metadata:
 splitcat1=>Magical Girl Lyrical Nanoha=>Lyrical Nanoha
 splitcat2=>Magical Girl Lyrical Nanoha=>Magical Girl
 splitcat1=>Puella Magi Madoka Magica/魔法少女まどか★マギカ=>Puella Magi Madoka Magica
 splitcat2=>Puella Magi Madoka Magica/魔法少女まどか★マギカ=>魔法少女まどか★マギカ

[JimmXinu]

The attached version adds three new config options:

1. Option to turn off the 'Story URL will be changed' check.
2. Option to force Author(s) name to be the same in author sort.
3. Option to force Title to be the same in title sort.

Let me know how this works for you.

UPDATE Aug 20 2013 -- Remove obsolete beta versions

[davidfor]

Originally Posted by JimmXinu

Hmm. That works for custom columns because they are set as a string. Tags are kept as list objects in python. So calibre, when it sees a ',' in a tag entry is changing it to a ';'.

So there isn't an easy way to do it.

There is, however, a complicated way. You can make two copies of category in new entries, modify those, and then replace the original category with the two new ones. It's not intuitive, but because of the order things get included and replaced it does work.

extra_valid_entries:splitcat1,splitcat2
include_in_splitcat1:category
include_in_splitcat2:category
include_in_category:splitcat1,splitcat2

replace_metadata:
 splitcat1=>Magical Girl Lyrical Nanoha=>Lyrical Nanoha
 splitcat2=>Magical Girl Lyrical Nanoha=>Magical Girl
 splitcat1=>Puella Magi Madoka Magica/魔法少女まどか★マギカ=>Puella Magi Madoka Magica
 splitcat2=>Puella Magi Madoka Magica/魔法少女まどか★マギカ=>魔法少女まどか★マギカ

Ugly, but that should work for what I want. I have only seen a couple that I wanted to do this wit, o it won't be a problem. Of course, the programmer in me says to find the code and change it

On a separate note, have you considered dynamically loading the adapters from a directory? I was playing with one, and each time I made a change, I had to restart calibre. It occurred to me that loading adapters at runtime would make that a lot easier. I had a very brief look at it, and I know it isn't that simple. But, I just remembered that the Annotations plugin is doing this, so that might serve as an example.

[JimmXinu]

Originally Posted by davidfor

Ugly, but that should work for what I want. I have only seen a couple that I wanted to do this wit, o it won't be a problem. Of course, the programmer in me says to find the code and change it

There's a link to the project home page in the first post. I'm open to accepting code contributions and to giving regular contributors write perms to the source repository.

Originally Posted by davidfor

On a separate note, have you considered dynamically loading the adapters from a directory? ...

Before the plugin version, when I first re-wrote FFDL, it did exactly that. However, I couldn't find a way to dynamically find and load the files from the plugin zip within calibre. So when the plugin version came along, that went away.

It's been a couple years since then and I haven't looked at the Annotations plugin. But dynamically loading files would only improve the development loop if it also detected and reloaded updated files. And the number of users who would benefit from it can be counted on one hand, I expect. (I use the CLI when developing new adapters, which side steps the problem neatly.)

I have been toying with the idea of adding some sort of user-added code system to allow python code to be entered similar to personal.ini and then called at different points in FFDL's execution to change the story. Pre/post processors for chapter text, metadata, etc. Up to now, however, none of the users wanting the most specific levels of control have had any python skills.

It continues to amaze me how much effort people will put into massaging their metadata. By contrast, except for usernames/passwords, this is the entirety of my personal.ini:

Spoiler:

# Library

[defaults]
is_adult: true

include_subject_tags: extratags, genre, category, status

strip_chapter_numbers:true
add_chapter_numbers:true

[epub]
include_tocpage: false
include_titlepage: true
include_logpage: smart
logpage_entries: fulldownload,dateCreated,datePublished,dateUpdated,numChapters,numWords,status,version,title,storyUrl,series,author,category,genre,rating,warnings,description

include_images:true
keep_summary_html:true
make_firstimage_cover:true

# so overwrite downloads have some marker.  
add_to_extra_valid_entries:,fulldownload
include_in_fulldownload:title
fulldownload_label:New Download

add_to_replace_metadata:
 fulldownload=>^.*$=>Yes

add_to_generate_cover_settings:
 ${anthology}=>true=>FFDL

[www.fanfiction.net]
never_make_cover: true
replace_metadata:
 author=>Bobmin356=>Bobmin

[davidfor]

Originally Posted by JimmXinu

There's a link to the project home page in the first post. I'm open to accepting code contributions and to giving regular contributors write perms to the source repository.

I had a suspicion that would be the answer We'll see how annoyed I get by it.

Before the plugin version, when I first re-wrote FFDL, it did exactly that. However, I couldn't find a way to dynamically find and load the files from the plugin zip within calibre. So when the plugin version came along, that went away.

It's been a couple years since then and I haven't looked at the Annotations plugin. But dynamically loading files would only improve the development loop if it also detected and reloaded updated files. And the number of users who would benefit from it can be counted on one hand, I expect. (I use the CLI when developing new adapters, which side steps the problem neatly.)

I had a look at the Annotations plugin a while ago to add support for the Kobo devices. It loads classes for the devices from both the zip file and a configurable directory. But, you're right, it wouldn't be used by many people. The CLI is probably the way to go the next time I'm playing.

I have been toying with the idea of adding some sort of user-added code system to allow python code to be entered similar to personal.ini and then called at different points in FFDL's execution to change the story. Pre/post processors for chapter text, metadata, etc. Up to now, however, none of the users wanting the most specific levels of control have had any python skills.

User exists are always nice, but a pain for the support person.

It continues to amaze me how much effort people will put into massaging their metadata. By contrast, except for usernames/passwords, this is the entirety of my personal.ini:

# Library
.
.
.
replace_metadata:
 author=>Bobmin356=>Bobmin

Short and sweet. I suspect that mine will get complicated and then in six months I'll look at it and wonder why it has all that rubbish.

[shesgottaread]

Jimm,

Thanks, glad that my requests/questions weren't too outrageous. I've been trying since yesterday to reply. Keep getting a timed out token. Will cross my fingers this time it will work.

Otherwise, I see you posted a new beta version with this added. I've downloaded and will give it a try.
Cheers!
--sgr

[shesgottaread]

Jimm,

I've tried the new version out with Check for changed story URL? unchecked, and
Force Author into Author_Sort? checked. Everything seems to be working. No prompts about the story already being in the library and A Magiluna Stormwriter's name didn't get corrected when AuthorSort was populated.

Thanks!
--sgr

[Firedancer885]

Originally Posted by shesgottaread

1. Author sort: is it possible for me to tell FFDL to override the Calibre author_sort value with author? I keep fanfic in a separate library, and think of authors by name not last name, so sorting by author can be frustrating when "B of Ericaland" author_sort becomes "Ericaland, B of".

There is a tweak in "Calibre Preferences - Tweaks - Author sort name algorithm". Why would you need to tell FFDL to override anything? Change the tweak to "copy" and you are set.

Firedancer

[JimmXinu]

Originally Posted by Firedancer885

There is a tweak in "Calibre Preferences - Tweaks - Author sort name algorithm". Why would you need to tell FFDL to override anything? Change the tweak to "copy" and you are set.

True. Except that Tweaks apply to all libraries and shesgottaread specifically said she uses two libraries and only wants to change the fanfiction library.

[JimmXinu]

Version 1.7.36 - 20 Aug 2013

• Make check for Story URL change configurable.
• Add configurable options to keep Title/Author the same in Title/Author Sort.

[FaceDeer]

Aw man, why did I never think to suggest that author/author sort option myself? That's been minorly bugging me for as long as I've been fanficking with this plugin, the vast majority of author pseudonyms with spaces in them aren't really meant to be treated as name and surname.

Now to update the author sort column for my existing downloads to keep everything consistent. Regexes ahoy, it seems.

[glos]

Hi Jimm for the plugin! I keep my library only for Fanfic, basically, so this is very very helpful
Would the replace metadata not work for, say, publisher?
I wanted to do this, but it seems I'm not getting any changes, or maybe I'm not doing it right.

replace_metadata:
 publisher=>archiveofourown.org=>Archive of Our Own

the second is the publisher you get by directly using AO3's download options on their site and then adding the book to Calibre. Most of my 1 chapter fic are like that, while I keep the copy I get from FFDL for the multi-chaptered ones because those epub downloaded from AO3 are always missing chapter notes (HUGE bug and they know it °_° ) while with FFDL I get all the notes (Why am I keeping those with 1 chapter as they are from ao3? because I like their "first page" formatting *see pic* and I try to keep it in as many fics as I can, not because I don't like what your plugin does!)

[JimmXinu]

Originally Posted by glos

Hi Jimm for the plugin! I keep my library only for Fanfic, basically, so this is very very helpful

Glad you like it.

Originally Posted by glos

Would the replace metadata not work for, say, publisher?
I wanted to do this, but it seems I'm not getting any changes, or maybe I'm not doing it right.

While it's labeled 'Publisher', the metadata entry is actually 'site'.
Do this instead:

replace_metadata:
 site=>archiveofourown.org=>Archive of Our Own

[glos]

Thank you so much!
Hah, I thought it was something easy like that, I had taken for granted it was publisher