Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Formats > Workshop

Notices

Reply
 
Thread Tools Search this Thread
Old 06-08-2011, 02:03 AM   #286
octarineblues
Member
octarineblues began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Jul 2010
Device: none
Erm, two things I've just noticed. One, FFN added links to share stories on various sites, and the parser is picking up the text portion of that, leading to the word share being stuck at the beginning of every chapter. Two, I only just noticed that all centered text is being rendered left-aligned for some reason. Neither of these things is really a huge deal, but I thought I'd point them out.
octarineblues is offline   Reply With Quote
Old 06-08-2011, 02:21 AM   #287
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by octarineblues View Post
Erm, two things I've just noticed. One, FFN added links to share stories on various sites, and the parser is picking up the text portion of that, leading to the word share being stuck at the beginning of every chapter. Two, I only just noticed that all centered text is being rendered left-aligned for some reason. Neither of these things is really a huge deal, but I thought I'd point them out.
Thanks for letting me know .

The 'share' links appear to have been introduced with ff.net's latest site update - I have updated FLAG's grabber to remove them. Edit: Looks like fictionpress also added these links - FLAG now removes those as well .

The centering is being stripped by the markup tidying process - this is a slightly more complicated fix, but it should be done sometime shortly.

Last edited by erayd; 06-08-2011 at 02:36 AM.
erayd is offline   Reply With Quote
Old 06-08-2011, 02:23 AM   #288
IndecisiveMe
Addict
IndecisiveMe doesn't litterIndecisiveMe doesn't litter
 
IndecisiveMe's Avatar
 
Posts: 276
Karma: 106
Join Date: Jan 2010
Location: Melbourne, Australia
Device: Pocketbook 360, Sony PRS-T1
I don't use FLAG, but if it's possible to customise the templates then I'd seriously consider using it. I've been using Graffer for ages and as of today, with FF.net's stupid share icons, it's not picking up the chapter text at all. It looks like the developer has given up and has closed his website.

So, that leaves me without a downloader at the moment. I know there are a few out there, but none that I've found are anywhere near as customisable as Graffer.

My ideal downloader would:

- Allow me to customise both the index.html and any layout output pages, including css,

- Allow me to create multiple templates to address the way some authors title their fics, for instance, in Graffer I have the ability to create mulitple index.html files and then pick them from a drop down menu on the download window. This solves chapter titling issues, for instance some authors like to give their chapter real titles, while others don't. Then you also have some who like prologues and then name each subsequent chapter by number. So if the download lets you pull chapter titles you could end up with "Chapter 1 : Prologue" and "Chapter 2 : Chapter 1", which drives me insane.

I'm not sure if FLAG can do any of that, but if it's possible either now, or in future releases, then I'd jump ship.
IndecisiveMe is offline   Reply With Quote
Old 06-08-2011, 02:28 AM   #289
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by IndecisiveMe View Post
I don't use FLAG, but if it's possible to customise the templates then I'd seriously consider using it. I've been using Graffer for ages and as of today, with FF.net's stupid share icons, it's not picking up the chapter text at all. It looks like the developer has given up and has closed his website.

So, that leaves me without a downloader at the moment. I know there are a few out there, but none that I've found are anywhere near as customisable as Graffer.

My ideal downloader would:

- Allow me to customise both the index.html and any layout output pages, including css,

- Allow me to create multiple templates to address the way some authors title their fics, for instance, in Graffer I have the ability to create mulitple index.html files and then pick them from a drop down menu on the download window. This solves chapter titling issues, for instance some authors like to give their chapter real titles, while others don't. Then you also have some who like prologues and then name each subsequent chapter by number. So if the download lets you pull chapter titles you could end up with "Chapter 1 : Prologue" and "Chapter 2 : Chapter 1", which drives me insane.

I'm not sure if FLAG can do any of that, but if it's possible either now, or in future releases, then I'd jump ship.
You're the second Graffer user in two days who has approached me about custom templates .

I'm currently in the process of adding an 'advanced' download form that will allow custom templates & CSS - this requires a few changes to the way FLAG manages things internally, but is currently on track to go live sometime this month.

I haven't had anyone ask for the ability to customise chapter titles before; could you expand a bit on what you're after here?
erayd is offline   Reply With Quote
Old 06-08-2011, 02:52 AM   #290
IndecisiveMe
Addict
IndecisiveMe doesn't litterIndecisiveMe doesn't litter
 
IndecisiveMe's Avatar
 
Posts: 276
Karma: 106
Join Date: Jan 2010
Location: Melbourne, Australia
Device: Pocketbook 360, Sony PRS-T1
Sure can and I'm guessing the other person who contacted you is a friend of mine.

The chapter titles is a big one for me. As it is now, there are three different ways it seems to work..

1. The author gives their chapters real titles, for instance "In the Beginning". Which I had set up for Graffer to grab as "Chapter 1 : In the Beginning", so chapter number and chapter title.

2. The author has a prologue and then numbers each subsequent chapter, for instance "Prologue" then "Chapter One". Which normally would render as "Chapter 1 : Prologue" then "Chapter 2 : Chapter One", so I set the template up to only grab the chapter titles and not the chapter numbers in that one.

3. The author didn't name any chapters and they only had chapter numbers. In this instance a normal template would grab "Chapter 1 : Chapter 1", so I set it up to only grab chapter numbers and no chapter titles.

I like knowing the number of the chapter I'm on and also the title, without going to the TOC.

In Graffer, I have the ability to create mulitple templates, so I have three that deal with the titling issue. From the drop down menu on the download window, I just choose which one I want to use, based on each fic itself, and that's it.

I think part of the problem I'm having with other fic downloaders, namely Fanfiction Downloader is that I'm not sure what tags they're using. They are different than the ones in Graffer and I can't find information on them anywhere. So that would be something I'd like made available as well. It makes customising much easier if you know what you're working with
IndecisiveMe is offline   Reply With Quote
Old 06-08-2011, 04:24 AM   #291
octarineblues
Member
octarineblues began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Jul 2010
Device: none
Huh, never noticed this before, but that's probably because hardly anyone uses macrons in their stories. I would assume this effects all letters with macrons, but at the very least, o with a macron is rendered as a question mark in both titles and story text. Here's the story I noticed it in: http://www.fanfiction.net/s/5686039/...ekyo_Sharingan
octarineblues is offline   Reply With Quote
Old 06-08-2011, 04:38 AM   #292
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by octarineblues View Post
Huh, never noticed this before, but that's probably because hardly anyone uses macrons in their stories. I would assume this effects all letters with macrons, but at the very least, o with a macron is rendered as a question mark in both titles and story text. Here's the story I noticed it in: http://www.fanfiction.net/s/5686039/...ekyo_Sharingan
Have you noticed any other characters that do this? FLAG does some slightly funky things internally to work around some problems PHP has with character encoding, so I'd expect the odd one, but they should be rare, and hopefully the most uncommon ones.

If it's getting in the road too much, let me know and I'll see if I can come up with a different approach.
erayd is offline   Reply With Quote
Old 06-08-2011, 06:02 AM   #293
octarineblues
Member
octarineblues began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Jul 2010
Device: none
Quote:
Originally Posted by erayd View Post
Have you noticed any other characters that do this? FLAG does some slightly funky things internally to work around some problems PHP has with character encoding, so I'd expect the odd one, but they should be rare, and hopefully the most uncommon ones.

If it's getting in the road too much, let me know and I'll see if I can come up with a different approach.
Hunting around, the only special characters I've seen problems with are ones with macrons. Accents, umlauts, and the eszett from German all seem to work fine. Just out of curiosity, I also tested a Japanese story, since it uses a completely different character set, and that didn't work at all, but I didn't really expect it to, and I'm probably just about the only person who would care about that anyway. As far as the macrons go, I don't really care if they're rendered properly if you could get them to at least render as the letter without the macron. Just showing up as a question mark is a bit annoying.

Also, I've just noticed that while you've taken care of the "Share" at the beginning of every chapter in multi-chapter fics, it still shows up in fics with only one chapter.

I'll let you know if I come across anything else strange. Thanks for all the hard work you've put into this!
octarineblues is offline   Reply With Quote
Old 06-08-2011, 06:14 AM   #294
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by octarineblues View Post
Hunting around, the only special characters I've seen problems with are ones with macrons. Accents, umlauts, and the eszett from German all seem to work fine. Just out of curiosity, I also tested a Japanese story, since it uses a completely different character set, and that didn't work at all, but I didn't really expect it to, and I'm probably just about the only person who would care about that anyway. As far as the macrons go, I don't really care if they're rendered properly if you could get them to at least render as the letter without the macron. Just showing up as a question mark is a bit annoying.
With the current approach, anything that can't be crammed into ISO-8859-15 is transliterated into the closest possible match that PHP's iconv functions know about - which for some reason in your case seems to be question marks (except for Japanese; you won't have *any* luck fitting that in!). I'll have another crack at getting things working end-to-end in UTF-8, but that may take me a while - PHP has a lot of rather strange problems with that approach that needs to be worked around. Currently about two thirds of the app uses UTF-8, and the other third uses ISO-8859-15.

Quote:
Also, I've just noticed that while you've taken care of the "Share" at the beginning of every chapter in multi-chapter fics, it still shows up in fics with only one chapter.
Thank you - now nuked in single-chapter fics too .

Quote:
I'll let you know if I come across anything else strange.
Thanks; feedback is always useful - it's great to hear about problems / suggestions / requests etc, and every little bit helps me make FLAG a better tool .

Quote:
Thanks for all the hard work you've put into this!
You're most welcome .
erayd is offline   Reply With Quote
Old 09-13-2011, 12:50 AM   #295
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
Hello again! Congratulations on all the improvements! It's the first time I've tried the FLAG web service since a couple years back and it's looking much, much nicer than the older version. The ePub output looks very professional. By the looks of things, you've also gotten UTF-8 working just fine now.

A few comments and suggestions:
  • The top left and right border graphics don't translate very well on small screens such as the iPhone. The images end up overlapping some of the text in Stanza. Perhaps you could use a single graphic top center instead. e.g.:
  • Perhaps you could use a Unicode font for the cover image? The title of the following story just shows up as boxes in the cover: http://www.fanfiction.net/s/7246882/. In the same vein, some operating systems don't play nice with unusual characters in filenames. This was downloaded as "panneler-san - 私にうそをつく.epub" on my Windows 7 PC. Not sure if XP would have handled it as well.
  • For FanFiction.Net, add the category/ies as subject. Assuming you're still using the same ffnet.source.php, adding the following lines to ffnet_get_meta() should do the trick:
    Code:
    preg_match("/var cat_title = '(.+)';/Ui", $page, $matches);
    $meta['category'] = $matches[1];
  • Use the ff.net url as identifier in content.opf, e.g.:
    Code:
    <dc:identifier xmlns:dc="http://purl.org/dc/elements/1.1/"
                   id="storyid"
                   opf:scheme="URI">http://www.fanfiction.net/s/1234567/</dc:identifier>
    You might also want to change the content.opf header to reflect the URL as the unique indentifier:
    Code:
    <package xmlns="http://www.idpf.org/2007/opf" version="2.0" unique-identifier="storyid">

On a side note, what zip class are you using to make the epubs? Conversion via Calibre is pretty slow so I'd prefer to create the epubs directly in PHP but the built-in zip class just isn't working for me.

Again, thank you very much for all your work! I'm using a customized, "webified" version of your original CLI script and it has made archiving fanfics and reading them on my smartphones, tablets and e-ink readers so much easier. Heck, if not for the ease of fanfic -> epub conversion that FLAG has afforded me, I probably wouldn't have considered buying an e-ink reader at all.

Last edited by ilovejedd; 09-13-2011 at 12:58 AM.
ilovejedd is offline   Reply With Quote
Old 09-13-2011, 02:33 AM   #296
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by ilovejedd View Post
Hello again! Congratulations on all the improvements! It's the first time I've tried the FLAG web service since a couple years back and it's looking much, much nicer than the older version. The ePub output looks very professional. By the looks of things, you've also gotten UTF-8 working just fine now.
Thanks - I'm glad you like the changes .

Yes, UTF-8 is now fully supported - was a bit of a headache to work around all the bits of PHP that couldn't cope with it, but thankfully that's now complete!

Quote:
The top left and right border graphics don't translate very well on small screens such as the iPhone. The images end up overlapping some of the text in Stanza. Perhaps you could use a single graphic top center instead. e.g.:
That's a great idea. I'm intending to get rid of the existing graphics anyway, as I've had a few complaints about them - they're apparently a bit too ornate for many people's taste.

Quote:
Perhaps you could use a Unicode font for the cover image? The title of the following story just shows up as boxes in the cover: http://www.fanfiction.net/s/7246882/.
It already does use a unicode font! Thanks for bringing this to my attention - looks like it's an issue with GD not handling the UTF-8 input correctly (which interestingly enough is contrary to how the documentation says it should be behaving). I'll need to take a more detailed look to figure out what's going on here.

Quote:
In the same vein, some operating systems don't play nice with unusual characters in filenames. This was downloaded as "panneler-san - 私にうそをつく.epub" on my Windows 7 PC. Not sure if XP would have handled it as well.
Good idea - output filenames are now sanitised to alphanumeric characters, and a restricted set of punctuation.

Quote:
For FanFiction.Net, add the category/ies as subject.
Good idea - I'll add this.

Quote:
Assuming you're still using the same ffnet.source.php...
I'm not - I completely rewrote the entire application earlier this year. It now takes a nice ~20-30 line XML definition for each supported site, rather than the mess of PHP & regex that the older version required. It also parses the DOM directly for most things, rather than using regex (although it does use regex in some places to narrow things down a bit) - overall, I've found using the DOM to be much more precise, and less inclined to break when the source sites change.

Quote:
Use the ff.net url as identifier in content.opf, e.g.:
Code:
<dc:identifier xmlns:dc="http://purl.org/dc/elements/1.1/"
               id="storyid"
               opf:scheme="URI">http://www.fanfiction.net/s/1234567/</dc:identifier>
Good idea - done.

Quote:
You might also want to change the content.opf header to reflect the URL as the unique indentifier:
Code:
<package xmlns="http://www.idpf.org/2007/opf" version="2.0" unique-identifier="storyid">
Also done.

Quote:
On a side note, what zip class are you using to make the epubs? Conversion via Calibre is pretty slow so I'd prefer to create the epubs directly in PHP but the built-in zip class just isn't working for me.
I'm using PHP's standard zip class (ZipArchive) - so it definitely works; if you're having problems with it, then it must be something you're doing wrong.

Quote:
Again, thank you very much for all your work! I'm using a customized, "webified" version of your original CLI script and it has made archiving fanfics and reading them on my smartphones, tablets and e-ink readers so much easier. Heck, if not for the ease of fanfic -> epub conversion that FLAG has afforded me, I probably wouldn't have considered buying an e-ink reader at all.
No worries, you're most welcome . I must say I find it fascinating you still use the old script though - it lacked a *lot* of functionality!
erayd is offline   Reply With Quote
Old 09-13-2011, 10:42 AM   #297
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
Quote:
Originally Posted by erayd View Post
It already does use a unicode font! Thanks for bringing this to my attention - looks like it's an issue with GD not handling the UTF-8 input correctly (which interestingly enough is contrary to how the documentation says it should be behaving). I'll need to take a more detailed look to figure out what's going on here.
From experience, GD appears to handle UTF-8 just fine. Perhaps the font just doesn't have the necessary characters? I've yet to see a single font which includes the whole UTF-8 character set.

Quote:
Originally Posted by erayd View Post
I completely rewrote the entire application earlier this year. It now takes a nice ~20-30 line XML definition for each supported site, rather than the mess of PHP & regex that the older version required. It also parses the DOM directly for most things, rather than using regex (although it does use regex in some places to narrow things down a bit) - overall, I've found using the DOM to be much more precise, and less inclined to break when the source sites change.
That sounds very interesting indeed. Would you be willing to post a zip/tar.gz of the new php scripts on this thread? Thanks!

Quote:
Originally Posted by erayd View Post
I'm using PHP's standard zip class (ZipArchive) - so it definitely works; if you're having problems with it, then it must be something you're doing wrong.
Could be a weirdness in Windows, too. Hmm, I'll have to revisit this. I've actually forgotten the reason why I switched to using exec() with 7z CLI. I remember getting a working output using the standard zip class at one point in time.

Quote:
Originally Posted by erayd View Post
I must say I find it fascinating you still use the old script though - it lacked a *lot* of functionality!
I'm using a highly customized version, though. The code may be a big mess and modifications a big headache but it's highly functional and works with the fanfic sites I use.
ilovejedd is offline   Reply With Quote
Old 09-13-2011, 11:35 AM   #298
Jellby
frumious Bandersnatch
Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.
 
Jellby's Avatar
 
Posts: 6,196
Karma: 4800739
Join Date: Jan 2008
Location: Spaniard in Sweden
Device: Cybook Orizon, Kobo Aura
Quote:
Originally Posted by ilovejedd View Post
From experience, GD appears to handle UTF-8 just fine. Perhaps the font just doesn't have the necessary characters? I've yet to see a single font which includes the whole UTF-8 character set.
Code2000 is close, but the character set is Unicode, UTF-8 is the encoding.
Jellby is online now   Reply With Quote
Old 09-13-2011, 11:50 AM   #299
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
Quote:
Originally Posted by Jellby View Post
Code2000 is close, but the character set is Unicode, UTF-8 is the encoding.
Thanks for the clarification. I get a bit confused with those.
ilovejedd is offline   Reply With Quote
Old 09-13-2011, 12:09 PM   #300
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by ilovejedd View Post
From experience, GD appears to handle UTF-8 just fine. Perhaps the font just doesn't have the necessary characters? I've yet to see a single font which includes the whole UTF-8 character set.
It's possible I guess - the font I'm using is DejaVu Sans though, which should cover most things it's likely to run across.

Quote:
That sounds very interesting indeed. Would you be willing to post a zip/tar.gz of the new php scripts on this thread? Thanks!
No, because it's not opensource (this is partly because I'm sick of supporting people's issues with installing & updating stuff, and the new version has some requirements that most people will be lacking) - but if you flick me a note with your email address and let me know what you want, I'll see what I can do.

Quote:
Could be a weirdness in Windows, too. Hmm, I'll have to revisit this. I've actually forgotten the reason why I switched to using exec() with 7z CLI. I remember getting a working output using the standard zip class at one point in time.
That's possible, although unlikely. I don't develop on Windows at all though (I don't even have Windows installed on my PC), so you're in a better position than me to figure that one out .

Quote:
I'm using a highly customized version, though. The code may be a big mess and modifications a big headache but it's highly functional and works with the fanfic sites I use.
If it does what you want it to do, then it fits the bill perfectly.
erayd is offline   Reply With Quote
Reply

Tags
converter, fanfiction, fanfiction.net, grabber, lrf

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Fanfiction.net on Kindle forkyfork Amazon Kindle 26 08-07-2011 08:42 AM
bookmarks/notes grabber Reader2 Android Developer's Corner 0 10-02-2010 09:24 AM
EASY fanfiction grabber? sherryg Workshop 19 01-08-2010 03:13 AM
FLAG (Fanfiction.net Lightweight Automated Grabber) and Calibre? malkie13 Calibre 1 02-10-2009 05:43 PM


All times are GMT -4. The time now is 08:05 AM.


MobileRead.com is a privately owned, operated and funded community.