[GUI Plugin] FanFictionDownLoader

[evamaria]

This seems to fix it - thanks so much for your patience and help!

Originally Posted by JimmXinu

If you haven't already, go create a file named /Users/evamaria/Library/Preferences/calibre/plugins/fanfictiondownloader_macmenuhack.txt

See http://code.google.com/p/fanficdownl...rCalibrePlugin regarding "Mac User Crashes"

[Firedancer885]

Hi Jim,

is there a way to make FFDL strip the extra indentation of the comments for the AO3 stories without also stripping all the other html?

[JimmXinu]

Originally Posted by Firedancer885

is there a way to make FFDL strip the extra indentation of the comments for the AO3 stories without also stripping all the other html?

Can you give an example?

I know that Ida wrote the AO3 adapter and went to some effort to include the leading and trailing author notes and chapter summaries. I'd probably have just left them off.

[iatheia]

Ah, those things. AO3 formats all the notes and summaries as blockquote. When including them I haven't changed that tag to anything else. Shouldn't be terribly hard to change it to p or div instead, but personally I think it is more charming this way.

[Firedancer885]

Originally Posted by iatheia

Ah, those things. AO3 formats all the notes and summaries as blockquote. When including them I haven't changed that tag to anything else. Shouldn't be terribly hard to change it to p or div instead, but personally I think it is more charming this way.

Could you or Jimm add something to either FFDL or the specific adapter to let people choose? I don't like the blockquotes. I don't mind them in the chapter summaries, only in the comments.

[JimmXinu]

Originally Posted by Firedancer885

Could you or Jimm add something to either FFDL or the specific adapter to let people choose? I don't like the blockquotes. I don't mind them in the chapter summaries, only in the comments.

How about giving yourself a personalized output_css and adding:

blockquote {margin-left: 0px; margin-right: 0px;}

[AmberDragon]

I think I just ran into either a bug or an unforseen case on AO3. Apparently, on AO3, a story can be tagged with multiple series tags, but FFDL only picks up the first series listed when you try to grab the story. You can find a series with several stories that have secondary series tags at http://archiveofourown.org/series/25185 for testing.

[JimmXinu]

Originally Posted by AmberDragon

I think I just ran into either a bug or an unforseen case on AO3. Apparently, on AO3, a story can be tagged with multiple series tags, but FFDL only picks up the first series listed when you try to grab the story. You can find a series with several stories that have secondary series tags at http://archiveofourown.org/series/25185 for testing.

As I recall, there are a few sites that allow a story to be in multiple series. But what would FFDL do with them?

Calibre defines one standard series column. You can add custom series columns, the issue is populating them. Even if FFDL had a feature for an extra series, can one story have three series? Four?

And even if the 'second' series for a story is put in a custom series column, if the next story in the same two series list them in the opposite order, it's still not useful. You could end up with:

Series     Custom Series
A[1]        A[2]
B[2]        B[1]

You couldn't sort correctly on either column.

That said, I am willing to add some AO3 specific extra_valid_entries for additional series. But you'd have to use custom_columns_settings to get them into your secondary custom series column(s). (Another alternative would be to include them in tags or tags-like custom column.)

Does that sound useful to anyone?

[AmberDragon]

I didn't even think to check whether Calibre supported multiple series or not, I just assumed it did. Sorry. If it's a Calibre thing, I'll just keep on using the Manage Series plugin to keep everything neat and tidy.

[JimmXinu]

I could use some help testing a new version again.

This one is mostly internal changes and minor fixes rather than new features for plugin users.

• Changes to logging to make more logging visible in PI and the same across web/CLI/PI.
• Don't number chapter when only one chapter even if add_chapter_numbers is on.
• Fix a bug where cover could still be partially set to an image in the cover_exclusion_regexp list.
• Add include_images support for HTML format. (I can't imagine that anyone uses HTML output with calibre--this is for CLI users.)
• Preserve id and class attributes on the HTML tags. (This is for users who want more control from output_css. I don't *think* it will hurt anything for other users...)

If I could get a few users to use this version for a few days, I'd appreciate it, thanks.

Update Oct 15, 2012: There's a newer test version with one change a few posts down.

[iatheia]

Originally Posted by JimmXinu

I could use some help testing a new version again.[*]Don't number chapter when only one chapter even if add_chapter_numbers is on.

What about adding new chapters to a recently published one-chaptered ongoing story? Would that add the number to the first chapter?

[ilovejedd]

Found an issue with grangerenchanted.com. For some reason, it's not getting the chapter text for chapters that have an image at the beginning of the chapter.

Here's one of the affected fics:
http://grangerenchanted.com/enchant/...y.php?sid=2956

the following chapters to be specific:
http://grangerenchanted.com/enchant/...956&chapter=42
http://grangerenchanted.com/enchant/...956&chapter=44
http://grangerenchanted.com/enchant/...956&chapter=51
http://grangerenchanted.com/enchant/...956&chapter=54

This is using the latest version of FFDL (1.6.14). Issue also exists on older versions I've tried (1.6.08, 1.6.09, 1.6.12 & 1.6.13).

[JimmXinu]

Originally Posted by iatheia

What about adding new chapters to a recently published one-chaptered ongoing story? Would that add the number to the first chapter?

Yes. Update actually reads the chapter text from each existing chapter and uses it as if it had just been downloaded. So the chapter titles are rewritten, adding numbers when there's more than one.

[JimmXinu]

Originally Posted by ilovejedd

Found an issue with grangerenchanted.com. For some reason, it's not getting the chapter text for chapters that have an image at the beginning of the chapter.

Try this version. It uses a different, more forgiving version of the HTML parser.

Update: There's a newer test version later on incorporating this change.

[Dylan Tomorrow]

Hi everyone,

as before, I've been reading loads of fanfics and mostly been very happy with this plugin and the fabulous job it is doing at downloading them, but two issues remained when reading the MOBI files on my Kindle Touch, for which I can at least share how I fixed them using two other tools (Howto at the end):

1. Broken indexing: This only happened to some of the MOBIs created by FFDL—most of them do index without hassle—but it was very annoying as the Kindle tries to index those files into eternity, draining the battery while doing so. This one I got to the bottom of—for 3 files—in trying to solve it. I figured it must be some non.validating HTML tag soup responsible for it and turned out, that was the case for those three. I let FFDL download the EPUB and then converted that to MOBI with the official KindleGen command line tool.

It was the three fictionalley stories I mentioned before, all of them had not properly closed <p> tags which KindleGen corrected (well, for two of them. One of them it just gave me an "Enhanced Mobi building failure" ). None of the other files gave me errors, but most of the MOBIs KindleGen put out indexed where they did not before, no matter what I tried (adding only one unindexed file at a time, restarting the kindle, etc.). So I figure they must have had some invalid syntax somewhere.

2. Missing Table of Contents: This is an issue with all of the MOBIs created by FFDL. This one is also peculiar. The MOBIs FFDL creates evidently have TOCs, as Calibre can navigate through them in the sidebar of its EbookViewer. But they never showed up in the Go to menu. You could only jump to the TOC that was part of the ebook itself via Go to, which made it more cumbersome.

But the TOC does show up in the Go to menu for ebooks converted with KindleGen. To be more exact, it does in the KF8 part. You see, MobiGen creates a hybrid MOBI file that contains both the old Mobipocket/mobi7 format and the new KF8/AZW3 format. Because there were still some files refusing to index, I used MobiUnpack to split the MOBIs into .mobi (mobi7) and .azw3 (KF8) files. The .azw3 files are the ones where the TOC works as expected, and all of them index without a problem and also superfast.

Summary/Conversion Workaround Howto:
So to anyone else who has problems with indexing/missing TOCs on Kindle eReaders (Warning: This will probably break your previous notes and marks from the old MOBIs, so I'd advise to only do it for the non-indexing files and files without notes/marks):

Preparations (only necessary before first conversion):

• configure duplicates automerge: Go to Preferences -> Import/Export -> Adding Books: Activate checkbox Automerge added books if they already exist in the calibre library, choose Create new record for each duplicate format in the dropdown menu, click Apply.
• Download & Install KindleGen [mr wiki page] (install under linux: execute sudo cp kindlegen/ /opt/kindlegen/ ; sudo ln -s /opt/kindlegen/kindlegen /usr/local/bin/kindlegen), download & unpack MobiUnpack [mr wiki page] (use the python script in Mobi_Unpack_v054.zip, as the calibre plugin can unfortunately not work on more than one file at a time), copy contents of /lib/ subfolder into your conversion folder

The conversion workaround itself:

1. Download Fanfics as EPUB with FFDL
2. Export EPUBs (and MOBIs, if any) to conversion directory with calibre's save to disk in a single directory action (delete OPF files (you can keep the old MOBIs, if any, as backup, but move them to another directory so that KindleGen won't overwrite them))
3. batch convert EPUBs with kindlegen; e.g. in the Linux bash console:

   for b in *.epub ; do kindlegen "$b" -c2 -verbose && echo "$b converted successfully." ; done

4. batch split .azw3s from .mobis, collect .azw3s from MobiUnpack-created subfolders in conversion folder and delete leftover MU subfolders; e.g. in the Linux bash console:

   for b in *.mobi ; do python mobi_unpack.py -s -d "$b" && echo "$b converted successfully." ; done
   mv */*.azw3 .
   rm -rf ./*/

5. delete any left-over old MOBIs from the fanfics with the remove files of a specific format from selected books action
6. import .azw3 files from your conversion directory into calibre via drag&drop or the Add Books menu entry, they are automagically added to the right entries
7. Put files onto Kindle, experience working TOC and indexing
8. PROFIT!

EDIT:
I did not save the KindleGen log for the first conversion batch I did, so I just ran the corresponding FFDL created EPUBs through Epubcheck 3. Almost all of them had similar HTML syntax errors (mostly unclosed <p> tags, also faulty element nesting, missing attributes, inline elements in the <body> outside of an enclosing block element). ALL of the fictionalley fanfics produced errors, also some from AO3 and fanfiction.net.

So I assume those kinds of syntax errors explain why the files did not index. If only all fanfic sites enforced proper XHTML/HTML5!