[GUI Plugin] FanFictionDownLoader

[cryzed]

Oh, so I misunderstood you . I already posted it in the correct section, I'm not sure how the whole process works but with a bit of luck you might see it there soon or possibly even directly integrated into Calibre, we'll see. Here is the thread.

Also in case someone here starts using it, a little feedback would be great, meaning: does it work for you or not?

[trianglekitty]

After downloading some 3000 works, I'm coming across an occasional issue in some of them. This may have already been brought up, but I'm getting basically just the metadata page and a title page. The actual text of the fic is just not there.

Here's an example of a fic that downloads this way:

http://www.fanfiction.net/s/10165/1/

I think it's only happening on fanfiction.net and it seems to happen more often with fics that have any kind of fancy font or text. The particular one I'm using as an example has a smaller than typical font size. Others might have fancy fonts or strange formatting, so I suspect it's throwing something off.

I've tried erasing and redownloading and uploading/overwriting. It isn't happening often...maybe thirty, forty fics out of the lot.

Also, just as a suggestion...it would be useful to have a way to select what metadata is updated. If I'm messed around and changed some things in custom columns that are filled with metadata automatically, I might not want all of the column to be overwritten if I do an update.

[seabream]

@Jim Thanks! I’m trying to get the Story ID (and Favs and Follows) to appear on the autogenerated cover (the default FFDL one, not the default one from Calibre), and the standard calibre IDs column doesn’t. When it appeared as a character it did. Though it wasn’t ideal it was easy to tell what it was. I was hoping that if I had it as a custom column it might, though having tried it, it apparently doesn’t work that way. Now I’m trying to find the setting for the defaults for what metadata goes into the cover.

Other changed behaviour issues: I’m a beta reader, so sometimes I want to have different historical versions of a story in my library for comparison. I used to do this at occasional break points (for AO3 and FF.net anyway) by downloading a new copy, and appending “, version number” or something, to the title of the old one. Subsequent updates via URL would update the one with the original title. Now this doesn’t work. When I try using ‘If Story Already Exists? Update EPUB if New Chapters’ the log says “More than one identical book--can’t tell which to update/overwrite” despite the fact that they don’t have the same title. Is there somewhere else that I can need to add a difference in order for it to be able to tell which one to update? I’ve tried this with several different stories by different authors, so I don’t think that it’s particular to one. I can workaround by putting the versions in a separate Calibre library, but that means that I can’t have the current and old versions open at the same time because Calibre always crashes right after switching libraries, and can’t be restarted until all open stories have been closed. Alternately, I can re-download the story as new each time, but that just seems like bad practice because of the increased load on the host. Also, I’d need to re-do any custom metadata (of which there’s often a fair bit).

Separately, Calibre reliably crashes in the following cases:
- I right click a story and choose Fanfiction Downloader > Update Existing Story (this is why I use the method in the second paragraph for updating stories)
Thanks! I’m trying to get the Story ID (and Favs and Follows) to appear on the autogenerated cover (the default FFDL one, not the default one from Calibre), and the standard calibre IDs column doesn’t. When it appeared as a character it did. Though it wasn’t ideal it was easy to tell what it was. I was hoping that if I had it as a custom column it might, though having tried it, it apparently doesn’t work that way. Now I’m trying to find the setting for the defaults for what metadata goes into the cover.

Other changed behaviour issues: I’m a beta reader, so sometimes I want to have different historical versions of a story in my library for comparison. I used to do this at occasional break points (for AO3 and FF.net anyway) by downloading a new copy, and appending “, version number” or something, to the title of the old one. Subsequent updates via URL would update the one with the original title. Now this doesn’t work. When I try using ‘If Story Already Exists? Update EPUB if New Chapters’ the log says “More than one identical book--can’t tell which to update/overwrite” despite the fact that they don’t have the same title. Is there somewhere else that I can need to add a difference in order for it to be able to tell which one to update? I’ve tried this with several different stories by different authors, so I don’t think that it’s particular to one. I can workaround by putting the versions in a separate Calibre library, but that means that I can’t have the current and old versions open at the same time because Calibre always crashes right after switching libraries, and can’t be restarted until all open stories have been closed. Alternately, I can re-download the story as new each time, but that just seems like bad practice because of the increased load on the host. Also, I’d need to re-do any custom metadata (of which there’s often a fair bit).

Separately, Calibre reliably crashes in the following cases:
- I right click a story and choose Fanfiction Downloader > Update Existing Story (this is why I use the method in the second paragraph for updating stories)
- I click on either the FFDL icon in the toolbar, or FanFictionDownloader in the menu bar, and then follow that by almost anything that switches away from the main Calibre window. e.g.: directly by selecting Configure Plugin, or clicking away somewhere else in Calibre to dismiss the dropdown, then switching to another window, whether by clicking another window, the Dock, opening a story (which automatically switches to another window), opening the tag manager, or using Command + Tab to switch applications. It seems to be fine with my doing stuff until switching to another window - I can resort by different tags/authors/etc.., I can use the dropdowns for saved searches and finding items in the tag browser, I can retitle something via two spaced out clicks. Anything that switches away from the main Calibre window after clicking either one though and about a second later and “calibre quit unexpectedly”. If the other window was a Calibre one that requires opening, it may or may not complete. Opening stories usually does, About Plugin likewise, the Tag Manager doesn’t. This is not the case with other menu bar or toolbar items.

It’s not a huge thing, since I can get to configuring via Calibre > Preferences > Advanced > Plugins > User Interface Plugins > FFDL, and Add via URL (using the keyboard shortcut), it’s just cumbersome, annoying, and I’m sure isn’t supposed to work that way. I’m running Calibre 0.8.62, FFDL 1.6.0 on Mac OS 10.7.4. The other user plugins I have installed are Count Pages 1.6.0, Epub Merge 1.2.0, Generate Cover 1.5.4, and Webscription 1.0.2. I have the system generated crash report, but it’s long so I suspect I shouldn’t put the whole thing up here. Is there a different one I should send? Or should I send it, but not to the forum?

It’s not a huge thing, since I can get to configuring via Calibre > Preferences > Advanced > Plugins > User Interface Plugins > FFDL, and Add via URL (using the keyboard shortcut), it’s just cumbersome, annoying, and I’m sure isn’t supposed to work that way. I’m running Calibre 0.8.62, FFDL 1.6.0 on Mac OS 10.7.4. The other user plugins I have installed are Count Pages 1.6.0, Epub Merge 1.2.0, Generate Cover 1.5.4, and Webscription 1.0.2. I have the system generated crash report, but it’s long so I suspect I shouldn’t put the whole thing up here. Is there a different one I should send? Or should I send it, but not to the forum?

[seabream]

Originally Posted by trianglekitty

Also, just as a suggestion...it would be useful to have a way to select what metadata is updated. If I'm messed around and changed some things in custom columns that are filled with metadata automatically, I might not want all of the column to be overwritten if I do an update.

@trianglekitty I can't help with the first part, but for the second, I think that you can maybe do that depending on what you’re going for, though it might be tedious if your needs change significantly from fic to fic. In the Basic tab of the FFDL configuration menu, there's a checkbox for “Keep Existing Tags when Updating Metadata” which you should check, then in your personal.ini under [defaults], do what’s described in post #635 in this thread (page #43) and put “include_subject_tags: extratags, genre, category, status, characters” etc… omitting the ones that you don’t want to be overwritten from or added to. Whether this works depends on which custom columns you’re dealing with though. Like if you have a Relationships column, or Fandom column (as I do) that populates from tags or taglike data, it should work. I don’t think that it’ll work for a custom column populated from AuthorID or other non-tag metadata because I don't have AuthorID in the list on mine, and it still populates (which is fine with me). I'm not sure exactly which bits of metadata are tags or not. I think I may have read a list somewhere in the documentation, but I haven't got it memorised.

[JimmXinu]

Originally Posted by trianglekitty

After downloading some 3000 works, I'm coming across an occasional issue in some of them. This may have already been brought up, but I'm getting basically just the metadata page and a title page. The actual text of the fic is just not there.

Here's an example of a fic that downloads this way:

http://www.fanfiction.net/s/10165/1/

Those are going to be because of weird/unusual/illegal things in the HTML uploaded by the author.

That particular story has a full HTML header in the story text:

Spoiler:

<div class='storytextp ' id='storytextp' align=center style='padding:0 0.5em 0 0.5em;'>
<div class='storytext xcontrast_txt' id='storytext'><HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=windows-1252"> <META NAME="Generator" CONTENT="Microsoft Word 97"> <TITLE>Déjà vu~ an ATF story</TITLE> </HEAD> <BODY LINK="#0000ff" VLINK="#800080"> <FONT FACE="Futura Md BT"><P>Title- Déjà vu</P> <DL> <DT>Author- Maria Villa</DT> </DL> <P>Disclaimer- They're not mine, blah, blah, blah. The ATF universe was started by Mog and went from there</P> <P>Notes- a Chris angst fic.</P> <P> </P>

I believe I can correct for that problem, but if you could post some more example URLs it would be helpful. That would tell me if the same problem occurs in several stories, or if there are other HTML artifacts causing problems.

Originally Posted by trianglekitty

Also, just as a suggestion...it would be useful to have a way to select what metadata is updated. If I'm messed around and changed some things in custom columns that are filled with metadata automatically, I might not want all of the column to be overwritten if I do an update.

I could possibly add a feature to let you choose which custom columns should always be updated and which should only be populated for new books. The Generate Cover feature has a similar option.

[JimmXinu]

Originally Posted by seabream

Other changed behaviour issues: I’m a beta reader, so sometimes I want to have different historical versions of a story in my library for comparison. I used to do this at occasional break points (for AO3 and FF.net anyway) by downloading a new copy, and appending “, version number” or something, to the title of the old one. Subsequent updates via URL would update the one with the original title. Now this doesn’t work.
When I try using ‘If Story Already Exists? Update EPUB if New Chapters’ the log says “More than one identical book--can’t tell which to update/overwrite” despite the fact that they don’t have the same title.

When the feature to allow multiple authors was added, I found that searching for large numbers of authors slowed down searching considerably. So as an efficiency measure, I changed FFDL to search for matching stories by the URL in the standard Identifiers column. Only if that comes up empty does it search for title/author now.

However, if you select a book and tell FFDL to update Update Existing, it will update that book ignoring any others that have the same URL or title/authors.

(hide long quote)

Spoiler:

Originally Posted by seabream

Separately, Calibre reliably crashes in the following cases:
- I right click a story and choose Fanfiction Downloader > Update Existing Story (this is why I use the method in the second paragraph for updating stories)
- I click on either the FFDL icon in the toolbar, or FanFictionDownloader in the menu bar, and then follow that by almost anything that switches away from the main Calibre window. e.g.: directly by selecting Configure Plugin, or clicking away somewhere else in Calibre to dismiss the dropdown, then switching to another window, whether by clicking another window, the Dock, opening a story (which automatically switches to another window), opening the tag manager, or using Command + Tab to switch applications. It seems to be fine with my doing stuff until switching to another window - I can resort by different tags/authors/etc.., I can use the dropdowns for saved searches and finding items in the tag browser, I can retitle something via two spaced out clicks. Anything that switches away from the main Calibre window after clicking either one though and about a second later and “calibre quit unexpectedly”. If the other window was a Calibre one that requires opening, it may or may not complete. Opening stories usually does, About Plugin likewise, the Tag Manager doesn’t. This is not the case with other menu bar or toolbar items.

It’s not a huge thing, since I can get to configuring via Calibre > Preferences > Advanced > Plugins > User Interface Plugins > FFDL, and Add via URL (using the keyboard shortcut), it’s just cumbersome, annoying, and I’m sure isn’t supposed to work that way. I’m running Calibre 0.8.62, FFDL 1.6.0 on Mac OS 10.7.4. The other user plugins I have installed are Count Pages 1.6.0, Epub Merge 1.2.0, Generate Cover 1.5.4, and Webscription 1.0.2. I have the system generated crash report, but it’s long so I suspect I shouldn’t put the whole thing up here. Is there a different one I should send? Or should I send it, but not to the forum?

Crashing calibre that much--especially with the other crashes you mentioned when switching libraries--sounds like it's some problem outside FFDL.

It's possible that FFDL is doing something subtly wrong that Mac doesn't like, but I don't have a Mac to test with. Nor have I had an reports of problems from other Mac users.

The only thing I can suggest is that I recall hearing from someone once that turning on calibre's 'Enable system tray icon' feature made calibre flaky on Mac or Linux.

[seabream]

@JimmXinu - re: Identifiers: Ah! Okay, that’s fine then. Thanks!

Re: crashing: I’m leaning towards it being an interaction with FFDL, because the library switching crashing is the only non-FFDL crash that Calibre exhibits regularly. It doesn’t react that way to doing the same thing with any of the other toolbar or menubar items. Actually, I just tested, and if I remove FFDL, the library switching crash stops happening. Reinstall it and the behaviour returns. All the crash reports, whether library or menu/window share:

“Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000008“

I understand about not having a system to test it on being a problem, but if you think that you might get something useful out of a crash report, I’d be happy to send one to you.

Re: Reports: I’ve been using FFDL with Calibre for a few months. There’s a threshold of usability below which it didn’t seem worth it to ask. Until I worked out enough of a pattern of things to avoid in order find that it was possible to have a functional library it didn’t seem stable enough that it would be useful to pursue. If I’d given up in frustration before I’d gotten there, I wouldn’t have reported.

Originally Posted by JimmXinu

The only thing I can suggest is that I recall hearing from someone once that turning on calibre's 'Enable system tray icon' feature made calibre flaky on Mac or Linux.

I read the same thing, so I never turned “Enable System tray icon” on in the first place.

Is there a different thread or forum I should try asking about the crashing in?

[cryzed]

@seabream, you might try just posting in the general Calibre forum, many people seem to do it and have similar problems.

[JimmXinu]

Originally Posted by seabream

I’m leaning towards it being an interaction with FFDL, because the library switching crashing is the only non-FFDL crash that Calibre exhibits regularly. It doesn’t react that way to doing the same thing with any of the other toolbar or menubar items. Actually, I just tested, and if I remove FFDL, the library switching crash stops happening. Reinstall it and the behaviour returns.

Yeah, that points to FFDL pretty clearly.

It reminds me of a problem that we had 3 months ago with Python garbage collecting the menu objects because Python didn't have any references to them even though the windowing system did. There's a fix in place for that already, but it does point me in that direction.

Kovid was able to ID that problem pretty quickly from the error log. Might want to try doing the same.

[seabream]

@JimmXinu I remember that error. At that stage FFDL was completely unusable for me too. I'm glad it was fixed.

I’m wary of just posting to the Calibre forum since it seems like it doesn’t belong there, and that would make work for a mod who would have to move it like they did for the thread you referenced. When you say “might want to try doing the same”, do you mean post the error log in this thread and ask Kovid to take a look at it (via PM)? Or is putting it in the Plugins forum better? I haven’t read around enough on this forum to know the etiquette.

[JimmXinu]

<shrug> Nobody's going to flame or ban you if you get it a bit wrong. I'd post a new thread to Plugins with the full error log.

Posting the full error log here and PMing Kovid would also work, I expect, but a new thread may garner attention from other devs in case Kovid's busy.

[seabream]

Thanks JimmXinu. Done. And Kovid's replied. Apparently it is the sort of thing that was there in April.

Originally Posted by JimmXinu

I could possibly add a feature to let you choose which custom columns should always be updated and which should only be populated for new books.

That would actually be quite useful for me too. Generally for updates, I only need Dates, Status, Chapters, and Words. I've usually got all the other bits sorted out to my liking by then.

[Zauberlehrling!]

Although I use a working Proxy with my Calibre there are some issues with updating stories.

I tried to run update the following address:
http://www.fanfiktion.de/s/4829d2e900008dc806519a28

and get this error message.

HTTP Error 502: Proxy Error (The parameter is incorrect.)

I noticed this error with another story from fanfiktion.de.
Story-download and -update from ksarchive.com is fine.

Therefore I assume that it doesn't have anything to do with our internal Proxy. Maybe someone could have a look? Thanks.

[JimmXinu]

@seabream - I've combed through my code and FFDL does save explicit references to all it's menu updates--twice, in fact.

I did, however, find one icon image that might conceivably be the problem. (But I doubt it.) Please try this version and let me know if works any better.

[trianglekitty]

Originally Posted by JimmXinu

Those are going to be because of weird/unusual/illegal things in the HTML uploaded by the author.



I could possibly add a feature to let you choose which custom columns should always be updated and which should only be populated for new books. The Generate Cover feature has a similar option.

Here's some more URLs...unfortunately I had deleted the majority of them.

http://www.fanfiction.net/s/327204/1/

Most of the other fics by the above author download fine

http://www.fanfiction.net/s/5630/1/

All of the above author's won't download (at least in the Magnificent Seven category)

http://www.fanfiction.net/s/5581/1/The_Wages_of_Sin

This author has two fics in M7, neither download, both appear to have 'special' html

Thinking about it more, I'm not sure how useful only updating some columns would be to other people. I mostly need it just because I screwed up big time when first downloading all of my fics...I didn't realize I could assign tags and metadata to custom columns and wasted a lot of time entering data by hand. Now that I've got everything set to download correctly, being able to preserve my prior work would be nice but not essential and again probably only useful to me.

The only way I could see it being useful is in a situation like this: sometimes I use the fic's summary as downloaded and sometimes I write in my own. So if I could choose whether to update that column, I'd still be able to update the rest without losing my sometimes-custom-summary-column.

There may be other ways already to do this that I'm just not aware of.