53,432 page PDF

[ahi]

I thought this might be of interest even to people other than Sony PRS-505 owners and individuals who want Wikipedia as an eBook... so I'm starting this thread here, with much the same information I posted in an earlier thread.

I have just successfully loaded a PDF with the following attributes:

- 53,432 pages
- 133.2 MB
- at least 15,000 items in external table of contents (2540 first level items, and each of those has 4 - 15+ subitems)
- all pages basically full of text (content is the complete Summa Theologica repeated 5 times over)

When loading it from my old-ish SD card, I've observed the following:

- There is a one-time 7-8 minute loading time.
- Page turning is slower than usual... about 3 seconds, but regardless whether one is going to the next page, or 30,000 pages further in the text.

When loading it from main memory:

- There is a one-time 4 minute loading time.
- Page turning seems consistently about 3 seconds still.

Warning: Don't ever try to actually use an *external* table of contents that has over a hundred thousand entries. My PRS-505 spent several "hours" formatting before I was somehow able to get it to reset. (Initially it was even ignoring my poking the reset button in the back... though otherwise it seemed like it was actually working/processing, not frozen.

---------------

I should mention that, in contrast, a 10,000 page PDF has less than 1 minute one-time loading time and pretty snappy page-turning.

Also, the reason I went for a 50,000 page or so PDF is because LaTeX memory limits (with all settings maxed out) still seem to kick in a few hundred pages before 60,000--albeit this isn't a page limit, but a text (i.e.: string) limit.

If anyone manages to generate a similarly content-heavy PDF with over 100,000 pages, please post what you did.

- Ahi

[Jellby]

Originally Posted by ahi

anyone manages to generate a similarly content-heavy PDF with over 100,000 pages, please post what you did.

You could maybe try appending the same file several times with pdftk or similar.

[RobbieClarken]

How big is the LaTeX source? If it is not too big, would you mind uploading it or emailing it to me? I wouldn't mind generating the PDF to see how it performs on the DR1000S.

[ahi]

Originally Posted by Jellby

You could maybe try appending the same file several times with pdftk or similar.

That's true, actually. I might do so just to so...

The reason I did not try it is because (even if different files are appended) each segment would be necessarily self-contained with no ability to cross-link into other sections.

Still... it is a reasonable way to see at what point it becomes virtually unusable.

Originally Posted by RobbieClarken

How big is the LaTeX source? If it is not too big, would you mind uploading it or emailing it to me? I wouldn't mind generating the PDF to see how it performs on the DR1000S.

I will upload it (somewhere else) tonight. It shouldn't be too big once zipped. It will be nice to hear about how such a large file performs on other devices.

- Ahi

[ahi]

Actually, RobbieClarken, given that the LaTeX sources require XeLaTeX and the memory settings maxed out, you might have an easier time with the PDF file itself. (Which is only a 130 MB, as stated above.)

I'll post both the source and the PDF though.

- Ahi

[RobbieClarken]

Originally Posted by ahi

Actually, RobbieClarken, given that the LaTeX sources require XeLaTeX and the memory settings maxed out, you might have an easier time with the PDF file itself. (Which is only a 130 MB, as stated above.

Sure thing, as long as you don't mind uploading 130 MB.

[ahi]

Originally Posted by RobbieClarken

Sure thing, as long as you don't mind uploading 130 MB.

I don't, but I haven't manage to do it yet.

In the meantime, the LaTeX source (just click on "Ingyenes", then after on "Letöltés indítása"):
http://data.hu/get/1628766/summa_theologica.zip.html

I'll post again once I have the PDF up somewhere too.

- Ahi

[Jellby]

Do you actually need to explicitly duplicate the text? Can't you have a single copy of the text and \input it several times.

[Jellby]

By \input-ing the same text seven times and increasing some settings I could create this PDF (86830 pages, 170MB) with pdflatex. It could be larger, I guess. (source, 3.4MB)

[ahi]

You are obviously right... I'm not sure why that didn't occur to me at the time.

At what page limit does PDFLaTeX start to crap out on you? Perhaps that's what I should be working with for such large files... maybe XeLaTeX (unintuitively enough) has a lower maximum memory limit that PDFLaTeX.

And how does the 86830 page monstrum work on your Cybook Gen3?

- Ahi

[Jellby]

Originally Posted by ahi

At what page limit does PDFLaTeX start to crap out on you?

I don't know... and I'm not sure I want to try I had to raise max_strings, pool_size and save_size in /usr/share/texmf/web2c/texmf.cnf (I'm using tetex, it may be different in other TeX distributions), and it seems the .out file (for pdf bookmarks) was the most problematic part.

And how does the 86830 page monstrum work on your Cybook Gen3?

I don't know either, but I'm afraid it could crash (reboot), it does that with another largish (and not particularly complex) PDF I created. It could be different with the new firmware when it comes out

[ahi]

Hmmm.... are you saying you generated the 80,000 page PDF with the default memory settings?

I only got about 5,000 pages with those... can't even generate the 10,000 page (single Summa Theologica) file without raising those limits.

Can you tell me more about the .out file stuff? Also, do you know how to prevent PDFLaTeX from generating any PDF bookmarks? For the Sony, with PDF links inside the content working, the PDF bookmarks are unnecessary and (I think) mainly responsible for the minutes of one-time initial load time.

- Ahi

[Jellby]

Originally Posted by ahi

Hmmm.... are you saying you generated the 80,000 page PDF with the default memory settings?

I only got about 5,000 pages with those... can't even generate the 10,000 page (single Summa Theologica) file without raising those limits.

Hmm... it depends what you mean with "memory settings". I had to change what I said above to generate the 80,000 page PDF, otherwise it aborts at around 60,000 pages, but nothing else, I didn't have to re-create formats or anything else. It may be that I had already modified something a couple of years ago, but I don't think so, this system's tetex should be pretty much out-of-the-box.

Can you tell me more about the .out file stuff? Also, do you know how to prevent PDFLaTeX from generating any PDF bookmarks?

The .out file is where hyperref stores the information for the bookmarks. I noticed that when pdflatex aborts, if I try to recompile, it then aborts again right at the beginning, when reading the .out file, that's why I think the stuff in the .out file might be one of the culprits.

To disable the bookmarks try passing bookmarks=false to the hyperref package.

[ahi]

Originally Posted by Jellby

Hmm... it depends what you mean with "memory settings". I had to change what I said above to generate the 80,000 page PDF, otherwise it aborts at around 60,000 pages, but nothing else, I didn't have to re-create formats or anything else. It may be that I had already modified something a couple of years ago, but I don't think so, this system's tetex should be pretty much out-of-the-box.

Oh, sorry. I misread what you had written.

Originally Posted by Jellby

The .out file is where hyperref stores the information for the bookmarks. I noticed that when pdflatex aborts, if I try to recompile, it then aborts again right at the beginning, when reading the .out file, that's why I think the stuff in the .out file might be one of the culprits.

To disable the bookmarks try passing bookmarks=false to the hyperref package.

Thanks for that!

I don't suppose you want to go for the gold and try to make a 200,000 page version?

If not, I'll try my hand at it in the next day or two.

- Ahi

[ahi]

In trying to think for some use for such mad-sized PDFs, I've been thinking of a KJV bible with appropriately linked bible commentary and strong's words.

The internal linking would have to be pretty extensive and (while simple to use) pretty complicated...

I think on each page, the verse numbers would have to be links to where the verse appears by itself, with commentary below and on subsequent pages, and each word that has a corresponding STRONG's entry also a link to said entry. Also probably a variety of navigation links (back to main portion, previous verse, next verse, et cetera).

Do you think something like that would be doable, Jellby?

- Ahi