[GUI Plugin] FanFictionDownLoader

[Feynor]

Originally Posted by Amalthia

Hi,

I've noticed this is happening in more stories that I've downloaded. Authors are adding empty paragraph tags between the actual paragraphs.

...

I would appreciate any and all help.

Thanks!

The problem is detecting and deciding when to remove the empty paragraphs. Sometimes they are added as scene/section breaks, others, like you describe, they are between every paragraph.
We humans can easily see which scenario we are looking at, but telling a computer how top make the distinction is a little harder.

Jim, I'll try to work on this. I'd say the function should be in a different method than replace_br though, but I think the counting trick I used for the breaks should work well here.

[JimmXinu]

Version 1.8.00 - 18 Nov 2013

• Internationalization/Localization code added. If people volunteer to do translations, they'll be able to use FFDL in their own languages.
• More fanfiction.net fixes. Don't included Updated/Published as characters, don't include excess Misc/Movies/etc categories on crossovers, start using https instead of http.
• Improved error messaging for ficwad & squidge.org/peja

[JimmXinu]

I'm looking for volunteers to translate FanFictionDownLoader to other languages. I have a volunteer for French, but no other languages at this time.

If you're interested, POEdit is a free tool that's pretty easy to use. I've uploaded a default.po for FanFictionDownLoader. Just open it in POEdit, add the translations for your language and send it to me here or by email (my address is in the file). And, of course, tell me what language it is because I don't known them all.

Thanks!

[leckadams]

I have a question. This only started happening the last day or so, but I keep my files in order by date modified. It is now grouping all the authors together and listing them all as modified if only 1 has been. Any ideas on why this is happening?

thanks

[JimmXinu]

Originally Posted by leckadams

I have a question. This only started happening the last day or so, but I keep my files in order by date modified. It is now grouping all the authors together and listing them all as modified if only 1 has been. Any ideas on why this is happening?

fanfiction.net is changing their URLs to use https instead of just http. The latest version of FFDL uses the https URLs--including for the author URL.

Calibre saves author URLs in its database by author, not by book. So when the author URL changes, it changes for all books by that author. But calibre also considers that a change to each book by that author, which updates the last_modified date for each.

Personally, I sort by date, not last_modified for that very reason--last_modified can change when you didn't expect it.

[leckadams]

Originally Posted by JimmXinu

fanfiction.net is changing their URLs to use https instead of just http. The latest version of FFDL uses the https URLs--including for the author URL.

Calibre saves author URLs in its database by author, not by book. So when the author URL changes, it changes for all books by that author. But calibre also considers that a change to each book by that author, which updates the last_modified date for each.

Personally, I sort by date, not last_modified for that very reason--last_modified can change when you didn't expect it.

Ughhh FFN.... ok I'll just change the way i sort. Thanks!

[Amalthia]

Originally Posted by Feynor

The problem is detecting and deciding when to remove the empty paragraphs. Sometimes they are added as scene/section breaks, others, like you describe, they are between every paragraph.
We humans can easily see which scenario we are looking at, but telling a computer how top make the distinction is a little harder.

Jim, I'll try to work on this. I'd say the function should be in a different method than replace_br though, but I think the counting trick I used for the breaks should work well here.

I ran across another story that used empty paragraphs to signify section breaks but Calibre stripped it out...I'm not sure what exactly changed between now and then.

But you're right this does sound like something only a human can figure out by looking. I really wish authors would stick to normal posting practices. It would make life so much easier.

Thanks for giving my question consideration.
Amalthia

[seabream]

Originally Posted by Amalthia

I ran across another story that used empty paragraphs to signify section breaks but Calibre stripped it out...I'm not sure what exactly changed between now and then.

But you're right this does sound like something only a human can figure out by looking. I really wish authors would stick to normal posting practices. It would make life so much easier.

Thanks for giving my question consideration.
Amalthia

Well, on "normal" posting practices, it's not just a matter of author choice. It also depends on how the story got to the site. If it was written in one program, then exported to one site, then imported to a second one from the first, and then to a third, or if they used a crossposting script, and each site handles styles differently, you can easily end up with formatting artifacts like the ones you describe, even if the original was decently structured. Depending on how long it's been, the original may no longer exist. Not all sites give much detail about what their import engines do to a story when you import them. It can be as frustrating, or more, to an author as it is to a reader to go through a long story and try to format it just the way they want it to be seen, and then have it turn out differently. And depending on the tools the author has available, it can be a lot of work to re-format a story to look the way they want it on multiple sites, given that there aren't fully standardised import processes, and even a given site changes its import engine from time to time, and as you've observed, it can't be fully automated.

[FaceDeer]

Originally Posted by leckadams

Ughhh FFN.... ok I'll just change the way i sort. Thanks!

I ran into this difficulty myself a little while back and I found a solution that should fit your desired workflow. Make a new custom date column and set FFDL to put the "Created" value in there. "Created" is a bit of an odd word to describe it, but that value is basically the date that FFDL created the current existing epub, which means it's the date that FFDL last modified it. It doesn't get changed by anything else in Calibre.

[leckadams]

Originally Posted by FaceDeer

I ran into this difficulty myself a little while back and I found a solution that should fit your desired workflow. Make a new custom date column and set FFDL to put the "Created" value in there. "Created" is a bit of an odd word to describe it, but that value is basically the date that FFDL created the current existing epub, which means it's the date that FFDL last modified it. It doesn't get changed by anything else in Calibre.

I'll try that...

Do you have any suggestions on how to find duplicate listings?

[FaceDeer]

Originally Posted by leckadams

I'll try that...

Do you have any suggestions on how to find duplicate listings?

There's a "Find Duplicates" Calibre plugin that's quite handy for this sort of thing, it can search for duplicate stories with a variety of degrees of strictness and various different algorithms for finding similar names. FFDL will also notice if you're downloading a new fanfic that has the same title and author as an existing fanfic and ask if you want to update the old fanfic's unique identifier to match the new one (essentially replacing the old one with the new one). That won't eliminate existing duplicates, though.

[leckadams]

Originally Posted by FaceDeer

There's a "Find Duplicates" Calibre plugin that's quite handy for this sort of thing, it can search for duplicate stories with a variety of degrees of strictness and various different algorithms for finding similar names. FFDL will also notice if you're downloading a new fanfic that has the same title and author as an existing fanfic and ask if you want to update the old fanfic's unique identifier to match the new one (essentially replacing the old one with the new one). That won't eliminate existing duplicates, though.

I've found that it alerts me, but somehow ive still got some...thank you for your help. I really appreciate it

[Amalthia]

Originally Posted by seabream

Well, on "normal" posting practices, it's not just a matter of author choice. It also depends on how the story got to the site. If it was written in one program, then exported to one site, then imported to a second one from the first, and then to a third, or if they used a crossposting script, and each site handles styles differently, you can easily end up with formatting artifacts like the ones you describe, even if the original was decently structured. Depending on how long it's been, the original may no longer exist. Not all sites give much detail about what their import engines do to a story when you import them. It can be as frustrating, or more, to an author as it is to a reader to go through a long story and try to format it just the way they want it to be seen, and then have it turn out differently. And depending on the tools the author has available, it can be a lot of work to re-format a story to look the way they want it on multiple sites, given that there aren't fully standardised import processes, and even a given site changes its import engine from time to time, and as you've observed, it can't be fully automated.

You have a good point about stories being imported multiple times and how the formatting can become rather odd after time.

I've uploaded my own stories and I always checked the formatting to make sure the paragraph spacings make sense and that nothing was lost with the upload.

To be fair, I understand know how to use html/css moderately well. I generally knkow what to look for if something does go wrong with a story upload. I don't think many authors have this technical knowledge and so if something did go wrong I'm not sure they'd know what to do to fix it.

Amalthia

[ILB]

Originally Posted by JimmXinu

I'm looking for volunteers to translate FanFictionDownLoader to other languages. I have a volunteer for French, but no other languages at this time.

If you're interested, POEdit is a free tool that's pretty easy to use. I've uploaded a default.po for FanFictionDownLoader. Just open it in POEdit, add the translations for your language and send it to me here or by email (my address is in the file). And, of course, tell me what language it is because I don't known them all.

Thanks!

I can translate it to german if you still need this language.

[JimmXinu]

Originally Posted by ILB

I can translate it to german if you still need this language.

I have volunteers for French and Russian so far, so German is all yours--thanks!