REGEX to Remove Embedded Header/Footer in the Text from a PDF?

[enuddleyarbl]

Well, that title probably makes no sense. I've got the text from a PDF file that has the header and footer information embedded within it (so the header/footer looks just like actual text). Something like:

blah blah blah

[page number] <== ex Footer

[title] <== ex Header

blah blah blah.

I'd like to convert that text to something passably readable as an EPUB via Calibre. But, before conversion, I need to get rid of those header/footer combinations. My REGEX knowledge is only microscopically above the zero point, and the best I could figure out as a way to find those headers/footers is:

\s+\d+\s+TITLE\s+

For my own future knowledge, I'll put what I think those codes mean in here:

\s means to match whitespace
+ means to match 1 or more
\d means to match a digit
TITLE is the title of the document that stuck in what used to be a header

So, it looks like that REGEX should grab from the start of the whitespace before the page number and run through the title to the end of the whitespace where the actual text picks up again. Probably not the best bit of REGEX, but it seems to work.

If the text before that header/footer combination is the end of a paragraph, that's fine. But, if the header/footer combination occurs right in the middle of a sentence, then removing it will result in the continuation "paragraph" being smashed right up against the paragraph that was before the header/footer.

For instance:

Lit lognued in one of the gseut criahs in N’kcis ofcife, his lnog lges spilwarng
far asorcs the rgu. He was attauneted rehtar tahn bgi. Too mcuh of his chohdliod


11



TITLE

had been snept in fere flla. Now he cluod not fit itno a stadnard prussere siut
or sparcecaft cniba; and whvereer he sta, he lekood lkie he was tniyrg to tkae orev.

would be transmogrified to:

Lit lognued in one of the gseut criahs in N’kcis ofcife, his lnog lges spilwarng
far asorcs the rgu. He was attauneted rehtar tahn bgi. Too mcuh of his chohdliodhad been snept in fere flla. Now he cluod not fit itno a stadnard prussere siut
or sparcecaft cniba; and whvereer he sta, he lekood lkie he was tniyrg to tkae orev.

Can anyone come up with a better way to strip out all those headers/footers?

EDIT: I guess if I replace the selection with a CR LF (/r/n), that would work reasonably. It doesn't look like it would be any worse than all the other lines ending with CR LF. I'll have to check and see if Calibre's conversion routine gets rid of those.

[retiredbiker]

You are on the right track. As you have seen, sometimes the header or footer will be on its own, sometimes mixed in with other text. Also, maybe not in your current book but very often, you will find varied spellings in the headers/footers. III, roman 3, often appears as Ill, and so on. Titles can be just anything. It all depends on the OCR that created the text, if the text was done with OCR.

To the point where, for example, if I want an epub of something from Internet Archive, I usually find doing my own OCR is quicker and easier than trying to correct their horrible epub, where rotten headers and footers abound, along with other stuff.

The result is, there is no magic expression, regex or plain, that will do the job completely. So do it in the editor, rather than at conversion. Usually with one of these I go through with several expressions, and still find the junk when scrolling slowly through. Several simple expressions are much better than trying to find one magical one to do everything. And depending on how consistent, or not, the book is, "replace all" may never be the best choice.

As to the badly split paragraphs (once the headers are out of the way), some simple regex can help there. Try searching for

([a-z])</p>\s+<p class="indent">([a-z])

and replace with

/1 /2

Adjust the "indent" bit with whatever your main paragraph style is. This, and some small variations of it, can quickly do a lot of fixing.

Edit: You don't need CR/LF, just add a space character to your replace string so it appears between the words. (And here you would only need a newline character, \n, anyway.)

[enuddleyarbl]

Thanks for the help. I had used Notepad++ to remove the header/footer stuff from the pure text. Since your regex was searching for HTML tags, I imported my text into a Calibre book, converted it to EPUB and edited it from within Calibre. Easy peasy.

Your replacement code should be \1 \2 instead of /1 /2. But, other than that, it seems to have worked very nicely. I'm sure there might be an instance or two where a "paragraph" ended with something like a comma instead of a lower case letter (so it wouldn't be caught). But, I'll have to read through to find them. I'd say those edits got me to a nicely readable text.

Thanks, again.

[retiredbiker]

Yeah, typing off the cuff, too fast. Sorry for the /1 /2

[Shohreh]

Before converting it to eg. EPUB, a simpler solution than regexes to get rid of headers+footers from a PDF is to… remove them from the PDF before calling Calibre :-)

Here's a script to mark those sections as "redaction annotations" and remove them.