feature request: better handling of books with hard wrapping

robertc99 · 09-14-2012, 03:28 AM

I'd like to suggest an improvement for calibre conversion.
Better handling of documents with hard line wrapping.
Apologies if I've missed something obvious, that would deal with this issue.

I have a bunch of books in .lit format that I want to read on a kindle.
Unfortunately, the paragraphs are all hard wrapped.
That is if a line reaches 65-70 character, the creation program inserted a carriage return
to wrap the line.
So using the obvious lit to mobi conversion, when you read it on the kindle, you tend to get a line of text followed by a short line, then another full line etc.

Turning on heuristics didnt seem to help.

Now, I can force a better conversion by converting to ascii.
Then doing an ascii to mobi conversion, with settings in the txt input to treat it as unformatted paragaphs.

But it would be much more convenient if the heuristics for lit could handle hard wrapping better.

Some idea's for an approriate heuristic. Allow people to set an unwrap line length range eg 65-75.
If a line is spotted with length in that range, followed by a single carriage return, then another back to text. Delete the carriage return.

theducks · 09-14-2012, 09:20 AM

Lower the 'Unwrap Factor' number a small amount in the conversion settings. (Mine is now set about point45 )

I always us the same exact 'Master' format when retrying a different conversion setting (In other words, don't try and figure the NEW best setting on a file that has been converted after/during the previous setting. Restore the File if needed and use a modified setting..

Calibre remembers the Last Conversion settings used on that book (The Preference is the Default for books never converted (or not reset) ).Calibre also will hint input format type (which may NOT be the one you started with and needs to be changed back to the 'Master')

robertc99 · 10-07-2012, 04:40 AM

Quote:

Originally Posted by theducks

Lower the 'Unwrap Factor' number a small amount in the conversion settings. (Mine is now set about point45 )

I always us the same exact 'Master' format when retrying a different conversion setting (In other words, don't try and figure the NEW best setting on a file that has been converted after/during the previous setting. Restore the File if needed and use a modified setting..

Calibre remembers the Last Conversion settings used on that book (The Preference is the Default for books never converted (or not reset) ).Calibre also will hint input format type (which may NOT be the one you started with and needs to be changed back to the 'Master')

Maybe I'm doing something wrong, but the Unwrap stuff in the heuristics didnt seem to help.

I tried unwrap factors between .1 and .9 and it just didnt seem to touch this wrapping at all.

theducks · 10-07-2012, 07:59 AM

Quote:

Originally Posted by robertc99

Maybe I'm doing something wrong, but the Unwrap stuff in the heuristics didnt seem to help.

I tried unwrap factors between .1 and .9 and it just didnt seem to touch this wrapping at all.

Are you sure the words (lines) are not picture of the original document page?
Calibre does not change word wrap in pictures.

ldolse · 10-08-2012, 12:52 AM

It's possible that the text is using an uncommon hard-wrapping technique that isn't covered by heuristics (e.g. <br /> everywhere). Simplest solution for those types of books is to convert to text - use markdown or textile if there is any formatting worth preserving.

Then convert the text back to an ePub or mobi using heuristics.

robertc99 · 10-12-2012, 08:00 AM

Its certainly not in an image format.
The original is a lit. So its difficult to see what the line breaks are exactly.

But I must be doing something wrong.
Even if I convert the lit to txt.
Then process the txt, I can't get the unwrap heuristic to do anything useful.

I can effect the breaks by tweaking the "txt input" settings.
But I can't seem to get the "heuristic unwrap" to do anything.

I'm a little confused by the "line unwrap factor".
I find the description unenlightening..
If you want to unwrap more lines do you decrease or increase the unwrap factor.

If 99% of the lines in the file are the same length.
Does that mean I need to set the unwrap factor to .01 to effect them?

I've tried both 0 and 1 as the unwrap factor and neither seems to have any effect whatsoever.

theducks · 10-12-2012, 11:19 AM

I adjust the unwrap in small amounts: .5 => .48 => .46 till I get good results.
Note Preferences Unwrap applies as the DEFAULT.
Once a book has been converted, the default is NOT used unless you use the reset button on the conversion screen.

Try making the changes inside the conversion screen, then you can set the default for use with other books.

robertc99 · 10-12-2012, 09:00 PM

Well, I havent been changing the preferences default. I've been changing the setting
in the conversion screen.

And its not a matter of tweaking it until I get a "good" result.
I have never seen any result. I've never seen the unwrap heuristic have any effect whatsover on anything. And I've tried settings from 0 to 1 in .1 increments.

If other people werent assuring me that it works for them, I would simply assume that the
unwrap heuristics code was simply completely broken.
Actually I've never seen any of the heuristics have any effect on anything.

But I have the ticks set for heuristics and unwrap heuristics in the conversion screen.
I don't suppose theres a setting somewhere that says "never apply heuristics even if its enabled in the conversion screen".
Because thats about what I'm seeing.

robertc99 · 10-12-2012, 09:49 PM

Now I'm really confused. I'm getting inconsistent results.
I cut the input .txt down to a small fragment to make testing easier.
And I'm getting unwrapping happening in the fragment.
But not in the original document.

I need to do more testing and work out exactly whats going on.

09-14-2012, 03:28 AM	#1
robertc99 Junior Member Posts: 5 Karma: 10 Join Date: Sep 2012 Device: kindle toucg	feature request: better handling of books with hard wrapping I'd like to suggest an improvement for calibre conversion. Better handling of documents with hard line wrapping. Apologies if I've missed something obvious, that would deal with this issue. I have a bunch of books in .lit format that I want to read on a kindle. Unfortunately, the paragraphs are all hard wrapped. That is if a line reaches 65-70 character, the creation program inserted a carriage return to wrap the line. So using the obvious lit to mobi conversion, when you read it on the kindle, you tend to get a line of text followed by a short line, then another full line etc. Turning on heuristics didnt seem to help. Now, I can force a better conversion by converting to ascii. Then doing an ascii to mobi conversion, with settings in the txt input to treat it as unformatted paragaphs. But it would be much more convenient if the heuristics for lit could handle hard wrapping better. Some idea's for an approriate heuristic. Allow people to set an unwrap line length range eg 65-75. If a line is spotted with length in that range, followed by a single carriage return, then another back to text. Delete the carriage return.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Feature Request] Drag and drop books out of calibre	phil_ga	Calibre	5	08-27-2012 10:36 AM
Get Books - Feature Request - Save Searches	nynaevelan	Calibre	5	05-14-2011 04:23 PM
PRS-600 To Sony Feature Request, Last # Books Read.	dzcowart	Sony Reader	10	05-28-2010 04:07 AM
Feature request: show recently opened e-books	yegorich	Calibre	1	01-18-2010 11:35 AM
How to deal with irregular hard-wrapping on a large scale?	Robotech_Master	Workshop	7	04-27-2009 08:06 PM

09-14-2012, 09:20 AM	#2
theducks Well trained by Cats Posts: 30,905 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Lower the 'Unwrap Factor' number a small amount in the conversion settings. (Mine is now set about point45 ) I always us the same exact 'Master' format when retrying a different conversion setting (In other words, don't try and figure the NEW best setting on a file that has been converted after/during the previous setting. Restore the File if needed and use a modified setting.. Calibre remembers the Last Conversion settings used on that book (The Preference is the Default for books never converted (or not reset) ).Calibre also will hint input format type (which may NOT be the one you started with and needs to be changed back to the 'Master')

10-08-2012, 12:52 AM	#5
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	It's possible that the text is using an uncommon hard-wrapping technique that isn't covered by heuristics (e.g. <br /> everywhere). Simplest solution for those types of books is to convert to text - use markdown or textile if there is any formatting worth preserving. Then convert the text back to an ePub or mobi using heuristics.

10-12-2012, 08:00 AM	#6
robertc99 Junior Member Posts: 5 Karma: 10 Join Date: Sep 2012 Device: kindle toucg	Its certainly not in an image format. The original is a lit. So its difficult to see what the line breaks are exactly. But I must be doing something wrong. Even if I convert the lit to txt. Then process the txt, I can't get the unwrap heuristic to do anything useful. I can effect the breaks by tweaking the "txt input" settings. But I can't seem to get the "heuristic unwrap" to do anything. I'm a little confused by the "line unwrap factor". I find the description unenlightening.. If you want to unwrap more lines do you decrease or increase the unwrap factor. If 99% of the lines in the file are the same length. Does that mean I need to set the unwrap factor to .01 to effect them? I've tried both 0 and 1 as the unwrap factor and neither seems to have any effect whatsoever.

10-12-2012, 11:19 AM	#7
theducks Well trained by Cats Posts: 30,905 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	I adjust the unwrap in small amounts: .5 => .48 => .46 till I get good results. Note Preferences Unwrap applies as the DEFAULT. Once a book has been converted, the default is NOT used unless you use the reset button on the conversion screen. Try making the changes inside the conversion screen, then you can set the default for use with other books.

10-12-2012, 09:00 PM	#8
robertc99 Junior Member Posts: 5 Karma: 10 Join Date: Sep 2012 Device: kindle toucg	Well, I havent been changing the preferences default. I've been changing the setting in the conversion screen. And its not a matter of tweaking it until I get a "good" result. I have never seen any result. I've never seen the unwrap heuristic have any effect whatsover on anything. And I've tried settings from 0 to 1 in .1 increments. If other people werent assuring me that it works for them, I would simply assume that the unwrap heuristics code was simply completely broken. Actually I've never seen any of the heuristics have any effect on anything. But I have the ticks set for heuristics and unwrap heuristics in the conversion screen. I don't suppose theres a setting somewhere that says "never apply heuristics even if its enabled in the conversion screen". Because thats about what I'm seeing.

10-12-2012, 09:49 PM	#9
robertc99 Junior Member Posts: 5 Karma: 10 Join Date: Sep 2012 Device: kindle toucg	Now I'm really confused. I'm getting inconsistent results. I cut the input .txt down to a small fragment to make testing easier. And I'm getting unwrapping happening in the fragment. But not in the original document. I need to do more testing and work out exactly whats going on.