Which format is best for long-term storage?

[eschwartz]

Originally Posted by jandrew

Thank you. I have been a software developer for a couple decades.

Yes html (and sgml, LaTeX, etc) is considered formatted text or rich text, but still plain text. In the context of a discussion of preserving content for the future, the distinction is important. Some people in this thread seem confused to the point of thinking that a text file stripped of all html is more future proof than the html file itself. This is a misconception that only amounts to a loss of information.

Why thank you for alerting me to that fact. Because I wasn't blatantly making fun of you.

(Hint: I am well aware that anyone who casually discusses LaTeX and rdoc doesn't need that explanation unless they are being deliberately obtuse.)

Oh, yes. And while some people around here were pointing out that .txt (not plain text, .txt) is a bad choice because you should do something normal and use HTML, you felt the need to point out that "html is plain text too", so who do you suppose was carrying out the real misconception, Professor Clarity?

[jandrew]

Originally Posted by eschwartz

so who do you suppose was carrying out the real misconception, Professor Clarity?

Sigh. You are correct. I was not trying to be deliberately obtuse, but my point was not well made and did not add to the discussion in the manner I had intended.

[eschwartz]

Originally Posted by jandrew

Sigh. You are correct. I was not trying to be deliberately obtuse, but my point was not well made and did not add to the discussion in the manner I had intended.

Forgiven.

[RichL]

Originally Posted by HarryT

Plain text files have no formatting. You'd lose a great deal of information by storing a plain text file. That really wouldn't be a terribly good idea.

All my books have been converted to text, what information have I lost that is needed to understand the story?

I would have to agree with you if we're talking about tech manuals, pictures books and etc. somehow or tuther I think typewriter art diagrams just wouldn't quite cut it.

[eschwartz]

bold/underlined/italic text missing, wrecked navigation, all those extensive maps of westeros and the realms --- and the book cover -- disappeared off the face of the earth... for starters.

What benefits do you now have over EPUB? (Or even OEB -- open ebook folder full of HTML and resource files. If you are afraid the ZIP implementation will be lost. )

[RichL]

Originally Posted by eschwartz

bold/underlined/italic text missing, wrecked navigation, all those extensive maps of westeros and the realms --- and the book cover -- disappeared off the face of the earth... for starters.

What benefits do you now have over EPUB? (Or even OEB -- open ebook folder full of HTML and resource files. If you are afraid the ZIP implementation will be lost. )

For me it actually started with 5-1/4 inch floppy days. I got tired of trying to keep up with the 'latest' in formating so simply changed all documents to text. I have simply continued on since then. I'm also one of those people that simply chop of everything in a book (trees or silicon) prior to chapter one and after 'The End'.

Damn, sound like a formatting Luddite or some such don't I.

[HarryT]

Originally Posted by RichL

All my books have been converted to text, what information have I lost that is needed to understand the story?

Formatting can significantly change the meaning of a sentence. Eg, consider:

"I think it's a good idea, but you don't agree."

compared to:

"I think it's a good idea, but you don't agree."

The italics totally alter the stress and the significance of the sentence.

[bla]

It's not really important which file format you use html and epub are very similar.. you could zip up the files to save a few bytes, if that is an issue and to make handling a bit easier I zip my ebooks up for storage by type .. burning to dvd is probably the safest/cheapest for long term storage a standard dvd will hold about 4.5 gigs... personally I use a usb drive to back up all my books you can get an 8 gig drive for < $20 in the end what way you do it is more a matter of personal preference.. I have a 32 gig sd card on my sony t1 all my books are loaded on the card .. I can mount the card in my android tablet or use a usb adapter to back the card up to my pc,,,,

[Mivo]

If you are really worried about long-term accessibility, then just store the books in multiple formats, including a plain text one. But on a practical level, it really doesn't matter much. If we are talking very, very long term storage, then readability of the media is a bigger concern (both the format and the material).

My Calibre library is backuped to my tablet, to one of my e-readers, and to an external drive. I also keep incremental backups in case it gets corrupted and I only notice later (those get synced with an external drive as well).

[shalym]

Originally Posted by HarryT

Formatting can significantly change the meaning of a sentence. Eg, consider:

"I think it's a good idea, but you don't agree."

compared to:

"I think it's a good idea, but you don't agree."

The italics totally alter the stress and the significance of the sentence.

Harry, I understand what you're saying, but I still save all of my books as plain text files (as well as mobi and sometimes epub). My reasoning is that plain text with no tags or anything else, is much more likely to be able to be read easily far into the future than anything else. I was cleaning out an old desk at my father-in-law's house the other day, and I found a CD simply labelled "Stuff" I looked at it, and there were lots of very very old files on there. The newest were Microsoft Works documents. The oldest were very old WordStar files. These files had been moved from computer to computer and media format to media format since the early 1980s. All of them were able to be opened using notepad, and once I removed the extraneous header info and formatting tags (I assume that's what they were, at least) I could read them very easily. I'm sure that they were very nicely formatted, but unfortunately, since I didn't have the original program, I couldn't see any of it...I could only see the text. I know how to strip the extra stuff so that I can see just the plain text--my children and grandchildren may not. I want them to be able to open my books and read them--if there is some formatting missing, they will still be able to read the words. I know that html is nothing more than text with extra tags added to it, but if I open an html file in a text editor and show it to my mother, she won't be able to read it easily. I also know that there is a very good chance that html capable applications will be around for a very long time to come, BUT...I also know that, back in the day, everyone thought that Lotus 123 and WordStar would be around (and be used regularly) for a very long time.

For a shortened version...I keep things in plain text as a backup of my backup, as well as keeping them in their original format. I KNOW that the plain text will be able to be read easily, without having to do any tweaking to see the words.

Shari

[eschwartz]

Maybe you should convert to markdown.

[Katsunami]

Originally Posted by shalym

Harry, I understand what you're saying, but I still save all of my books as plain text files (as well as mobi and sometimes epub). My reasoning is that plain text with no tags or anything else...

Well... it's possible to run a lot of 20 year old software, down to stuff created for Windows NT 3.1 (1993) on a 64-bit installation of Windows 8.1, which was released in 2013. If you need even older software, you can use DOSBox, or even a virtual machine to run software back to 1975 or so, also on Windows 8.1.

That is 40 (!) year old software that can be run *now*, often without having access to the source code.

Now, enter EPUB2, which is an format with an open specification, and calibre implements reading and writing it, and is able to convert almost any format in existence to it. Calibre is open source as well, so all code is available.

Even if calibre ceases to be developed for whatever reason, someone will at least lift the EPUB2 read/write routines out of it (at minimum), porting them to another programming language if necessary, so EPUB2 files can be read, and thus subsequently written into a newer format.

As EPUB2 is still in full swing at this time, it being only 7 years old or so, with millions and millions (billions?) of e-books sold in it, I expect it to be fully supported for at least another 10-15 years. I'll be in my late forties at that time.

Assuming the EPUB2 read routine survives another 30-40 years after that, I assume someone will write an accompanying routine to write into a new relevant format; if not open source, it will be in a paid program.

I've started a topic about this in the past actually, but I've become convinced that if I keep good, clean EPUB2 books, I'll be safe for the next 40 to 50 years or so, at the very least being able to convert to newer formats.

(I feel the same about FLAC: that music file format is almost perfect. It has had an update in 2012 to fix a tiny bug which sometimes prevented it to convert 4+ GB files on Windows, but it essentially has not changed since 2007. If another lossless format comes into vogue, converting to it from FLAC will be possible.)

[shalym]

Originally Posted by Katsunami

Well... it's possible to run a lot of 20 year old software, down to stuff created for Windows NT 3.1 (1993) on a 64-bit installation of Windows 8.1, which was released in 2013. If you need even older software, you can use DOSBox, or even a virtual machine to run software back to 1975 or so, also on Windows 8.1.

That is 40 (!) year old software that can be run *now*, often without having access to the source code.

Now, enter EPUB2, which is an format with an open specification, and calibre implements reading and writing it, and is able to convert almost any format in existence to it. Calibre is open source as well, so all code is available.

Even if calibre ceases to be developed for whatever reason, someone will at least lift the EPUB2 read/write routines out of it (at minimum), porting them to another programming language if necessary, so EPUB2 files can be read, and thus subsequently written into a newer format.

As EPUB2 is still in full swing at this time, it being only 7 years old or so, with millions and millions (billions?) of e-books sold in it, I expect it to be fully supported for at least another 10-15 years. I'll be in my late forties at that time.

Assuming the EPUB2 read routine survives another 30-40 years after that, I assume someone will write an accompanying routine to write into a new relevant format; if not open source, it will be in a paid program.

I've started a topic about this in the past actually, but I've become convinced that if I keep good, clean EPUB2 books, I'll be safe for the next 40 to 50 years or so, at the very least being able to convert to newer formats.

(I feel the same about FLAC: that music file format is almost perfect. It has had an update in 2012 to fix a tiny bug which sometimes prevented it to convert 4+ GB files on Windows, but it essentially has not changed since 2007. If another lossless format comes into vogue, converting to it from FLAC will be possible.)

My point is that I want my children and grandchildren to be able to open my stuff *without* having to find copies of old software, and then try to figure out how to make it work on new hardware. If someone who only knows the basics about computers was confronted with those files I found, they would have just thrown up their hands and said "Oh well, they can't be read!". If they were in plain text, they would have opened easily.

Shari

[eschwartz]

Your grandchildren will have an AI to do all their thinking for them.

[Katsunami]

Originally Posted by shalym

My point is that I want my children and grandchildren to be able to open my stuff *without* having to find copies of old software, and then try to figure out how to make it work on new hardware. If someone who only knows the basics about computers was confronted with those files I found, they would have just thrown up their hands and said "Oh well, they can't be read!". If they were in plain text, they would have opened easily.

Shari

First: If you just keep an eye out and move your library to the next relevant open format as soon as EPUB2 is in danger of becoming obsolete, your grandchildren won't have a problem. You might need to do this 1-3 times in the time span of 40-60 years.

I can live with that, to be honest.

Second: If there are still people only able to use the very basics of computers in 40 years time, they're in trouble. Even now I see people of around my age (early to mid-35) who use the computer like a monkey: "Click this button to do that." They actually don't *understand* what is happening. The consequence of that is that they have great difficulty leading their daily lives.

"Could you send me that picture by e-mail?" (When not behind their own computer.)
"You can order this at store ...." (When not ordering at their usual store.)
"Please that file onto the desktop of that computer..." (on a Mac instead of Windows 7)

All impossible for some people. You can't be like that in another 40 years, because it's like being illiterate now; maybe even worse.