View Single Post
Old 09-22-2012, 12:03 PM   #861
seabream
Enthusiast
seabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheese
 
Posts: 48
Karma: 1056
Join Date: Jul 2012
Location: Central Canada
Device: Sony PRS-T1, RIM Playbook & Z10, Samsung Note, MacOSX
1.6.10 feature test

As one of the people who requested the inclusion of the new categories for FanFiction.net, thanks for the new functionality. I respect the amount of work that it must have taken, and appreciate the flexibility in how they have been implemented.

Tested on a Macbook Pro running Mac OS X 10.7.4, Calibre 0.8.64 and the beta of FFDL provided in post #844 (updated with notes on 1.6.10), with fics from Fanfiction.net using the following settings in personal.ini:

Spoiler:
is_adult:true

extra_valid_entries:reviews,favs,follows

custom_columns_settings:
reviews=>#reviews,r
favs=>#favs,r
follows=>#follows,r

extra_logpage_entries: reviews,favs,follows
titlepage_entries: series,category,genre,language,characters,status,d atePublished,dateUpdated,dateCreated,rating,warnin gs,numChapters,numWords,site,reviews,favs,follows, description


All of it works as described with multiple fics.

However, Get Story URLs from Web Page seems to now be broken for Fanfiction.net. Attempting results in:
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 251, in get_urls_from_page
NameError: global name 'url' is not defined
Attempted with: http://www.fanfiction.net/u/1728624/ and http://www.fanfiction.net/u/2469838/twitchy-witch
By accident, I tried it with a non-link and got the same result.

I donít actually read at dramione.org, but since you apparently havenít gotten many replies on it, I decided to test that too. I made the custom columns listed in custom_columns_settings and used the following basically default settings mentioned in the described personal.ini:
Spoiler:

extra_valid_entries:themes,hermiones,dracos,timeli ne,cliches

include_in_genre:genre, themes
include_in_cliches:hermiones,dracos

themes_label:Themes
hermiones_label:Hermiones
dracos_labelracos
timeline_label:Timeline
cliches_label:Character Cliches

extra_titlepage_entries: themes,timeline,cliches
extra_logpage_entries: themes,timeline,cliches
extra_subject_tags: themes,timeline,cliches

custom_columns_settings:
cliches=>#cliches,r
themes=>#themes,a
timeline=>#timeline,n
hermiones=>#hermiones,a
dracos=>#dracos,a

Picking a random popular author with short stories (ThatRomantic) I tried And What If They Were Lovers ? http://dramione.org/viewstory.php?sid=533 and Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which worked except for Genres. I eventually realized the mistake, but you may want to add to the notes that in order for include_in_genre:genre,themes to work you need to add genre=>#genre under custom_column_settings too, because from the description, at first glance, it looks as though one doesnít need to do anything in addition. For reference:
## For another example, you could, by uncommenting this line, include
## themes in with genre metadata.
#include_in_genre:genre, themes

Testing with a few more stories by the same author: And What If Karma Caught Up With Them? http://dramione.org/viewstory.php?sid=535 and Battlefield: http://dramione.org/viewstory.php?sid=1479 and I got this from the Updates completed, final status window:
Comments: sequence item 0: expected string or Unicode, NoneType found

If I commented #include_in_genre:genre, themes the comment went away.
Commented or not, Hermiones, Dracos, Timeline and Themes donít populate with the above two stories. Genre doesnít give the composite column, but does populate, strangely, with ďFanFictionĒ, which isnít included with the themes in the Genre column for the story that worked. (I have Genre set to populate with Extra Tags in the GUI Custom Column tab in the FFDL preferences, but when I put include_in_genre:genre,themes,extra_tags and try http://dramione.org/viewstory.php?sid=533 ďFanFictionĒ still doesnít show, so Iím not sure where itís coming from)

Not related to new features, but noticed that for Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which of the stories I checked is the only story with an image, even though as shown below I have make_firstimage_cover:false, both under [epub] and [defaults], the epub uses the banner as the title page anyway.
Spoiler:

[epub]
include_images:true
keep_summary_html:true
make_firstimage_cover:false

## This is an example of what your personal configuration might look
## like.

[defaults]
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
#is_adult:true

## include images from img tags in the body and summary of
## stories. Images will be converted to jpg for size if possible.
include_images:true

## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
keep_summary_html:true

## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
make_firstimage_cover:false

While I was writing this up, 1.6.10 was released, so I retested on dramione and FanFiction.net. The parts that didnít work still didnít. I didnít check all of the parts that did, but Iíll let you know if I notice anything new.

Also not new to this version, but something I hadnít tried before. If I call for the FFDL version in the log or title page, with 1.6.10 it shows version 4.4. Same with 1.6.09. Is this intentional?

I can confirm that the relationships meta information is working for me for AO3.

I canít find the setting for adding a keyboard shortcut for FFDL configuration. Calibre > Preferences > Keyboard > FanFictionDownLoader only has the following options
[Add New FanFiction Book(s) from URL(s), FanFictionDownloader, Get Story URLs from Web Page, Get URLs from Selected Books, Update Existing FanFiction Book(s)] and it doesnít seem to be anywhere in the FFDL preferences. Is it somewhere else?

Also, Iím not sure exactly when this is was but at least from 1.6.3, fics from FanFiction.net are downloading with left justified text (as displayed on the website) showing up as justified in the epub. I can change it with Convert Books, but that makes everything left justified.

Question about tag substitution. Though I havenít actually implemented it, in principle I know how to automate tag substitution on import in personal.ini globally, or on a site by site basis with replace_metadata and regexp. But is there a way to have FFDL take a given tag and use different rules for replacement depending on the source category and/or destination column? e.g.: input type: Characters, entry ďHan S.Ē output to Tags as ďchar Star Wars: Han SoloĒ and to Characters as ďStar Wars.Han SoloĒ? Designating which metadata category a rule applies to is mostly needed for fandoms where the fandom name is also that of a main character, or a main characterís name tends to show up in story titles, but also because I donít want to automatically change author names when in fan works, they can contain strings that may also appear in just about any other category. I can see how to do something sort of like it after the fact using Bulk Metadata Edit, though itís two steps, but Iíd need to redo it as I add to the library, and as the library gets larger, running it would take progressively longer, and Iíd need to do it separately for every single tag I want to replace (which total in the four digits by now).

Reason being that I use the custom column with heirarchical tags in Calibre, but not all my mobile readers support tags (in three cases, not at all, and in a limited way with the Sony i.e.: it gets very slow above a few hundred tags, and itís not always accurate, FB Reader on the Samsung is good, but itís not my phone), which means that Iím using the output of Calibreís Generate Catalog, which doesnít include tags from custom columns.

By the way, generally feel free to P.M. me if you have things you want tested. I donít check the forum every day, or week, and I wonít guarantee that Iíll have time to test immediately, but Iím happy to help when thereís time.

Last edited by seabream; 09-22-2012 at 12:13 PM. Reason: typos and adding spoiler tags for length
seabream is offline   Reply With Quote