PDF samples

[kovidgoyal]

Hi all,

I'm starting work on a new PDF conversion engine for calibre that will hopefully handle header and footer extraction and multiple column extraction as well.

I'm asking for a few sample PDF files that I can use as a test corpus. I'd appreciate it if you could just extract a few pages with different typographical features and make a new PDF file with them.

Note that this new engine will not handle mathematics/tables/vector diagrams, etc. so don't provide samples for those.

Also this is a bit of a long term project, so don't expect results too quickly.

[Elfwreck]

Would you prefer a single PDF with extracts from different works, or separate PDFs?

[kovidgoyal]

separate PDFs as the algorithm is likely to take into account overall document structure as well. I just dont want very large PDFs and also if they're copyrighted, its best to just extract a small subsection.

[Elfwreck]

Sample PDFs, all Creative Commons or or other ok-to-distribute files.

They range from simple mainstream novel PDFs to nightmarish magazine formatting. Some have pictures; some have links.

• ANAT annual report 2008: Colored text, multi-column layout. (CC)
• James Boyle's The Public Domain: Changes in margins & leading; lists (CC)
• Helen Keller's essay I Learned to Speak: Non-crucial font & margin changes; should convert well. (PD text)
• Lenz v Universal: standard legal document layout (PD)
• Lowry Pei's For Adam: Novel with irregular page breaks & nonstandard headers; should convert well, might be worth noting how the spacing carried over. (CC)
• TWC-1: Journal; columns with pictures (permission to share)
• Wick's Houses of the Blooded: RPG game book; nightmarish; not expected to convert well at all. (permission to share)

Metadata's likely all over the place. Some have it; some don't.

[kovidgoyal]

Thanks, will come in handy.

[neilmarr]

Hello there, Kovid: I just sent you an email, but in case it gets lost; I head up the editorial team at a small indie publishing house that's covered all its paperback novels with PDF versions for the past eight years (we have about 120 titles at www.bewrite.net). If you think it would be of any help, please drop me a line and I'll send as many PDF ebooks as you might need to experiment with. Most are straight text, all well-formatted, some use more than one font and a few carry inside illustration. All are front covered.

Cheers and good luck. Neil

[darkmonk]

Here my contribution, a book I've tried several times to convert using xpath to remove headers/footers. Just left a few sections with different things. If you want the full book, it's here. It's legal to distribute, too, I think.
I'm really happy you're finally trying to do this, but I'd still like it if you kept the option to specify what header/footer to remove.

[Pablo]

Originally Posted by darkmonk

Here my contribution, a book I've tried several times to convert using xpath to remove headers/footers. Just left a few sections with different things. If you want the full book, it's here. It's legal to distribute, too, I think.
I'm really happy you're finally trying to do this, but I'd still like it if you kept the option to specify what header/footer to remove.

What an interesting book! I wonder if it is legal to reformat and redistribute it. I doesn't include any license information.

[kovidgoyal]

@darkmonk: Well the new engine wont use text content, but rather position on page to detect headers and footers. And I will probably remove the current remove header/footer option and replace it with a generic "remove content" option.

[user_none]

Originally Posted by kovidgoyal

@darkmonk: Well the new engine wont use text content, but rather position on page to detect headers and footers. And I will probably remove the current remove header/footer option and replace it with a generic "remove content" option.

It might be a good idea to keep it. Just re-define it as remove content since it can actually match any text in the document. This could be helpful for poor PDF conversion for instance where the headers and footers are interspersed in the middle of the text in say Mobi or Epub files.

[acidzebra]

Originally Posted by kovidgoyal

And I will probably remove the current remove header/footer option and replace it with a generic "remove content" option.

I hope there will still be an "show advanced/all features" button or something like that

[kovidgoyal]

Originally Posted by user_none

It might be a good idea to keep it. Just re-define it as remove content since it can actually match any text in the document. This could be helpful for poor PDF conversion for instance where the headers and footers are interspersed in the middle of the text in say Mobi or Epub files.

Yeah that's what remove content will do

[aleks]

Hi Kovid,

Here is something with columns for you to work on.

Kozak

[mrmikel]

Here is a pdf from the UK, the early history of their air force. It have tried and tried to convert this and everything I did was unsuccessful without endless manual work, so good luck!