Markdown conversion: line spacing & lists

Snorkledorf · 11-07-2011, 12:34 AM

I'm getting a group of odd results with calibre's conversion of Markdown-formatted text files. Is this a bug worthy of filing, or do I just have something set wrong somewhere?

Here's a sample Markdown source file:

Code:

# This is a Markdown test


## List tests

1. This is item 1

2. This is item 2

Here's some text as a spacer.

1. This is item 1, with 2 spaces following it to scrunch the lines together  
2. This is item 2

More spacer text

* Here's a bullet point

* Another bullet point

Spacer text...

* Two spaces after this one  
* Another bullet point

## Linefeed tests

These lines should be  
close together but not joined.

These are three lines that
should
flow smoothly together.

Here's BBEdit 10's preview, which looks fine:

After doing a Txt --> Mobi conversion in calibre, this is what the Mobi viewer shows:

And here's what it looks like on the Kindle 3:

I see these problems:

List items in the converted book get shifted down from their bullets/numbers (on Kindle only)
Line spacing isn't being correctly controlled when 2 spaces are added at the end of the first of two sequential lines
Sequential lines of text aren't being joined together the way Markdown syntax says they should (calibre viewer & Kindle)
(Also, calibre's mobi viewer seems to be adding extra vertical spacing at the top of a list; not quite the same topic though)

Should I officially report these? Or report myself, for lack of knowing what I'm doing?

The images are duplicated below: BBEdit / calibre's mobi viewer / kindle screenshots

kovidgoyal · 11-07-2011, 01:02 AM

Make sure you have set format to markdown in the TXT input settings.

Agama · 11-07-2011, 02:24 AM

Quote:

Originally Posted by Snorkledorf

I'm getting a group of odd results with calibre's conversion of Markdown-formatted text files. Is this a bug worthy of filing, or do I just have something set wrong somewhere?

I have taken your example code and converted with calibre, using:

Paragraph style off and Formatting style markdown, and the results look exactly as expected, so it does look like your conversion settings.

Snorkledorf · 11-07-2011, 03:49 AM

Definite progress, so thanks! Still have one issue remaining.

Formatting style was set to Markdown in my TXT Input section, but Paragraph Style was set to auto so it was applying its own rules to the (mostly) otherwise-correct Markdown formatting.

Now when I convert it I get the same correct preview that you got, Agama. But when I actually send it to the Kindle, this is what I get:

The extra vertical spacing remains, but now only for the lists whose members don't have the extra two spaces at the end. Weird. Other than that it's perfect. I don't know what the Kindle (and Kindle Previewer app too) is seeing differently from the calibre mobi previewer.

DoctorOhh · 11-07-2011, 03:56 AM

Quote:

Originally Posted by Snorkledorf

Other than that it's perfect. I don't know what the Kindle (and Kindle Previewer app too) is seeing differently from the calibre mobi previewer.

Calibre doesn't have a Mobi previewer, calibre is an ePub viewer at heart. Everything is essentially converted to ePub prior to being displayed in the calibre viewer.

No idea if it will work, but you might try converting to ePub then Mobi.

Also maybe the folks that refined this section will take a closer look at the mobi conversion and fill us in on what the solution might be.

Snorkledorf · 11-07-2011, 05:52 AM

Interesting tidbit about it being an ePub viewer.

Converted from Markdown TXT to ePub, and it looks good in Sigil.

Converted Markdown TXT to Mobi (results on Kindle as above) and then from that Mobi to ePub, and it again looks good in Sigil.

So it's not like the file is irreparably damaged, it just seems to quite predictably come out as a file that Kindle Previewer and the actual Kindle don't fully like.

Agama · 11-07-2011, 08:20 AM

What about Markdown TXT to ePub and then
ePub to Mobi (as dwanthny suggested)?

Snorkledorf · 11-07-2011, 05:18 PM

Tried that one too and got the same results.

ldolse · 11-07-2011, 09:30 PM

Try converting to ePub using Calibre, and then use Kindlegen instead of Calibre to convert from ePub to mobi - see if that retains the formatting. In some cases Kindlegen does a better job than Calibre, this could be one.

I can't quite tell what the issue is as you're not posting the pics here on Mobileread - never understood the attraction for people to use imageshack as most people can't see those pics... The forum will let you attach pics directly, fyi.

Snorkledorf · 11-08-2011, 06:21 PM

I used the attachments button to duplicate the inline images for my posts. Hope that helps you see what's going on.

Slight sidetrack, relating to the forum software:

I didn't even know MobileRead allowed you to attach images until now; the "Insert Image" button just asks you for a URL so that's what I've been giving it. I've always thought that the attachments button would be for .zip files and the like, since there's already an Image button right there that's "obviously" what you're supposed to use. I'm guessing that the other people who link to external images are thinking the same thing, that they have no other choice.

The "Insert Image" method does have one advantage: being able to manually place the images in specific locations in the post, rather than all jammed together at the end. Doesn't compensate for the inconvenience of having had to upload them elsewhere each time though; I'll be using the attachment window from now on, so thanks for the head's up.

For that matter, why do you think you're not able to see Imageshack images? I haven't heard of that before.

ldolse · 11-08-2011, 10:00 PM

I can't ever see anyone's imageshack images (it just shows me a frog and says the domain isn't registered - this is everywhere, not just mobileread) - I think only people with imageshack accounts can see images, and I'm not one for going through the hassle of signing up for that just to see other people's pics.

If the bullet lists look like that even after getting the ePub right I think it might be a bug - I would assume Kindle should be able to display that type of formatting.

Snorkledorf · 11-08-2011, 11:08 PM

KindleGen gave the same result as calibre's conversion from ePUb.

But when I edited the calibre-produced HTML to remove the (superfluous?) paragraph tags around the list items, I finally got a good mobi file! It converts correctly whether converted by calibre or KindleGen.

For example, I changed this:

Code:

<ol class="calibre3">
<li class="calibre4">
<p class="calibre5">This is item 1</p>
</li>
<li class="calibre4">
<p class="calibre5">This is item 2</p>
</li>
</ol>

to this:

Code:

<ol class="calibre3">
<li class="calibre4">This is item 1</li>
<li class="calibre4">This is item 2</li>
</ol>

...and everything worked well. It looks like the paragraph tags on list items is what's messing things up in this case.

I'll whip up a RegEx in BBEdit to batch-process my files until this gets taken care of. Now maybe this is a bug worth filing!

(off-topic zone follows)

Interesting about Imageshack not displaying for you. Actually I don't have an account or anything, I just go to the site, upload the image, and copy the URL it gives me. Theoretically I could sign up in order to have a list of the images I've uploaded, but I don't bother with that.

Maybe it's an IP address thing? It'd be over the top to do just to check this, but trying something that changes your IP like Hotspot Shield might give different results.

ldolse · 11-09-2011, 01:57 AM

Yes, I'd definitely consider that a bug, though it might be considered a borderline FR as the mobi code is all reverse engineered. Regardless if you post these details and your test case to the bug tracker I would expect it to be addressed.

Don't know what the reason with Imageshack is precisely, but I regularly travel around the world and see the frog/error wherever I go, so IP address wouldn't be the reason. I assume if I have the problem then other users must too, but maybe it's not so common.

Agama · 11-09-2011, 02:12 AM

I'm not convinced that it's a bug, but possibly a rendering error. If you try the official Markdown test page, (http://daringfireball.net/projects/markdown/dingus) using:

Code:

## List tests

1. This is item 1

2. This is item 2

then the output is

Code:

<h2>List tests</h2>

<ol>
<li><p>This is item 1</p></li>
<li><p>This is item 2</p></li>
</ol>

Therefore calibre seems to be working in line with the Markdown specification. So before fixing, it may be worth testing that this fix won't cause problems in other rendering systems.

kovidgoyal · 11-09-2011, 02:34 AM

Markdown is supposed to produce <li><p>, as such this is not a bug, but an artifact of the way the Kindle chooses to render <li><p> constructs. Arguably, calibre could compensate, but calibre deals with input from lots of sources not just markdown, so adding a special workaround for this is likely to break other things.

11-07-2011, 12:34 AM	#1
Snorkledorf Blue. Not sad...just blue Posts: 218 Karma: 1267018 Join Date: Oct 2009 Location: Japan Device: Ridibooks Paper Pro	Markdown conversion: line spacing & lists I'm getting a group of odd results with calibre's conversion of Markdown-formatted text files. Is this a bug worthy of filing, or do I just have something set wrong somewhere? Here's a sample Markdown source file: Code: # This is a Markdown test ## List tests 1. This is item 1 2. This is item 2 Here's some text as a spacer. 1. This is item 1, with 2 spaces following it to scrunch the lines together 2. This is item 2 More spacer text * Here's a bullet point * Another bullet point Spacer text... * Two spaces after this one * Another bullet point ## Linefeed tests These lines should be close together but not joined. These are three lines that should flow smoothly together. Here's BBEdit 10's preview, which looks fine: After doing a Txt --> Mobi conversion in calibre, this is what the Mobi viewer shows: And here's what it looks like on the Kindle 3: I see these problems: List items in the converted book get shifted down from their bullets/numbers (on Kindle only) Line spacing isn't being correctly controlled when 2 spaces are added at the end of the first of two sequential lines Sequential lines of text aren't being joined together the way Markdown syntax says they should (calibre viewer & Kindle) (Also, calibre's mobi viewer seems to be adding extra vertical spacing at the top of a list; not quite the same topic though) Should I officially report these? Or report myself, for lack of knowing what I'm doing? The images are duplicated below: BBEdit / calibre's mobi viewer / kindle screenshots Attached Thumbnails Last edited by Snorkledorf; 11-08-2011 at 06:03 PM.

11-07-2011, 03:49 AM	#4
Snorkledorf Blue. Not sad...just blue Posts: 218 Karma: 1267018 Join Date: Oct 2009 Location: Japan Device: Ridibooks Paper Pro	Definite progress, so thanks! Still have one issue remaining. Formatting style was set to Markdown in my TXT Input section, but Paragraph Style was set to auto so it was applying its own rules to the (mostly) otherwise-correct Markdown formatting. Now when I convert it I get the same correct preview that you got, Agama. But when I actually send it to the Kindle, this is what I get: The extra vertical spacing remains, but now only for the lists whose members don't have the extra two spaces at the end. Weird. Other than that it's perfect. I don't know what the Kindle (and Kindle Previewer app too) is seeing differently from the calibre mobi previewer. Attached Thumbnails Last edited by Snorkledorf; 11-08-2011 at 06:04 PM.

11-08-2011, 11:08 PM	#12
Snorkledorf Blue. Not sad...just blue Posts: 218 Karma: 1267018 Join Date: Oct 2009 Location: Japan Device: Ridibooks Paper Pro	KindleGen gave the same result as calibre's conversion from ePUb. But when I edited the calibre-produced HTML to remove the (superfluous?) paragraph tags around the list items, I finally got a good mobi file! It converts correctly whether converted by calibre or KindleGen. For example, I changed this: Code: <ol class="calibre3"> <li class="calibre4"> <p class="calibre5">This is item 1</p> </li> <li class="calibre4"> <p class="calibre5">This is item 2</p> </li> </ol> to this: Code: <ol class="calibre3"> <li class="calibre4">This is item 1</li> <li class="calibre4">This is item 2</li> </ol> ...and everything worked well. It looks like the paragraph tags on list items is what's messing things up in this case. I'll whip up a RegEx in BBEdit to batch-process my files until this gets taken care of. Now maybe this is a bug worth filing! (off-topic zone follows) Interesting about Imageshack not displaying for you. Actually I don't have an account or anything, I just go to the site, upload the image, and copy the URL it gives me. Theoretically I could sign up in order to have a list of the images I've uploaded, but I don't bother with that. Maybe it's an IP address thing? It'd be over the top to do just to check this, but trying something that changes your IP like Hotspot Shield might give different results.

11-09-2011, 02:12 AM	#14
Agama Guru Posts: 776 Karma: 2751519 Join Date: Jul 2010 Location: UK Device: PW2, Nexus7	I'm not convinced that it's a bug, but possibly a rendering error. If you try the official Markdown test page, (http://daringfireball.net/projects/markdown/dingus) using: Code: ## List tests 1. This is item 1 2. This is item 2 then the output is Code: <h2>List tests</h2> <ol> <li><p>This is item 1</p></li> <li><p>This is item 2</p></li> </ol> Therefore calibre seems to be working in line with the Markdown specification. So before fixing, it may be worth testing that this fix won't cause problems in other rendering systems.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Line Spacing Issue after Conversion	anleva	Calibre	3	11-07-2011 12:32 AM
indents & line spacing - ems, %, pts ?	cybmole	Sigil	6	03-23-2011 12:33 PM
Markdown / Blockquote conversion giving unexpected results	Agama	Conversion	23	02-23-2011 11:16 AM
7.34 line spacing???	JSWolf	Calibre	18	12-19-2010 11:09 PM
Line Spacing on PDF to Epub conversion	poodlemama	Calibre	2	05-03-2010 08:28 PM

11-07-2011, 01:02 AM	#2
kovidgoyal creator of calibre Posts: 43,856 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Make sure you have set format to markdown in the TXT input settings.

11-07-2011, 05:52 AM	#6
Snorkledorf Blue. Not sad...just blue Posts: 218 Karma: 1267018 Join Date: Oct 2009 Location: Japan Device: Ridibooks Paper Pro	Interesting tidbit about it being an ePub viewer. Converted from Markdown TXT to ePub, and it looks good in Sigil. Converted Markdown TXT to Mobi (results on Kindle as above) and then from that Mobi to ePub, and it again looks good in Sigil. So it's not like the file is irreparably damaged, it just seems to quite predictably come out as a file that Kindle Previewer and the actual Kindle don't fully like.

11-07-2011, 08:20 AM	#7
Agama Guru Posts: 776 Karma: 2751519 Join Date: Jul 2010 Location: UK Device: PW2, Nexus7	What about Markdown TXT to ePub and then ePub to Mobi (as dwanthny suggested)?

11-07-2011, 05:18 PM	#8
Snorkledorf Blue. Not sad...just blue Posts: 218 Karma: 1267018 Join Date: Oct 2009 Location: Japan Device: Ridibooks Paper Pro	Tried that one too and got the same results.

11-07-2011, 09:30 PM	#9
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Try converting to ePub using Calibre, and then use Kindlegen instead of Calibre to convert from ePub to mobi - see if that retains the formatting. In some cases Kindlegen does a better job than Calibre, this could be one. I can't quite tell what the issue is as you're not posting the pics here on Mobileread - never understood the attraction for people to use imageshack as most people can't see those pics... The forum will let you attach pics directly, fyi.

11-08-2011, 06:21 PM	#10
Snorkledorf Blue. Not sad...just blue Posts: 218 Karma: 1267018 Join Date: Oct 2009 Location: Japan Device: Ridibooks Paper Pro	I used the attachments button to duplicate the inline images for my posts. Hope that helps you see what's going on. Slight sidetrack, relating to the forum software: I didn't even know MobileRead allowed you to attach images until now; the "Insert Image" button just asks you for a URL so that's what I've been giving it. I've always thought that the attachments button would be for .zip files and the like, since there's already an Image button right there that's "obviously" what you're supposed to use. I'm guessing that the other people who link to external images are thinking the same thing, that they have no other choice. The "Insert Image" method does have one advantage: being able to manually place the images in specific locations in the post, rather than all jammed together at the end. Doesn't compensate for the inconvenience of having had to upload them elsewhere each time though; I'll be using the attachment window from now on, so thanks for the head's up. For that matter, why do you think you're not able to see Imageshack images? I haven't heard of that before.

11-08-2011, 10:00 PM	#11
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	I can't ever see anyone's imageshack images (it just shows me a frog and says the domain isn't registered - this is everywhere, not just mobileread) - I think only people with imageshack accounts can see images, and I'm not one for going through the hassle of signing up for that just to see other people's pics. If the bullet lists look like that even after getting the ePub right I think it might be a bug - I would assume Kindle should be able to display that type of formatting.

11-09-2011, 01:57 AM	#13
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Yes, I'd definitely consider that a bug, though it might be considered a borderline FR as the mobi code is all reverse engineered. Regardless if you post these details and your test case to the bug tracker I would expect it to be addressed. Don't know what the reason with Imageshack is precisely, but I regularly travel around the world and see the frog/error wherever I go, so IP address wouldn't be the reason. I assume if I have the problem then other users must too, but maybe it's not so common.

11-09-2011, 02:34 AM	#15
kovidgoyal creator of calibre Posts: 43,856 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Markdown is supposed to produce <li><p>, as such this is not a bug, but an artifact of the way the Kindle chooses to render <li><p> constructs. Arguably, calibre could compensate, but calibre deals with input from lots of sources not just markdown, so adding a special workaround for this is likely to break other things.

Advert

Advert