Text files blanked

Halk · 05-15-2009, 07:26 PM

I can't be sure that Calibre was the cause of this - and no real damage has been done, since I had the files elsewhere...

I've just downloaded Calibre and installed it to see if I can get my head around using it to organise and reformat the books I have.

I plugged in the Cybook, and Calibre generated an error. Similar to this one. I closed Calibre and restarted it, and this time it picked up the Cybook correctly, and read in the files on the Cybook (none) and on the SD card (600 or so books).

I thought I might as well empty the Cybook so I was starting fresh, and from there only put on books that were in a proper state. So I selected all of them and went for save to disk, and then specified a directory on my hard drive. It processed the request for about a minute and a half on the task manager, and then reported it was complete. No files appeared on the hard drive. I then repeated but instead selected the save to disk all in one folder. Same thing happened, and again no files on the disk.

Having Calibre send files to the Cybook would be nice, but it's not exactly required since I can do this using Windows explorer.

I then took the SD card out and put it in a card reader to copy all the files off... and then noticed that all the .txt files were 0KB in length. Every .txt file had been erased. Other formats were fine.

Since I last read one of the text files the only thing I've done is throw a rather large PDF file on the Cybook, which it didn't like and displayed blank pages. So I'm suspecting that something may have gone wrong with Calibre.

In any case I don't want to point any fingers around, but I thought I'd mention it in case there's some kind of glitch.

Edit : All .txt files are blank. As are all .prc files, and some (but not all) HTML files.

Edit 2 : It now looks more likely that it was Calibre, as when I first connected the Cybook it showed 1.6GB free on the card, it now shows 1.7GB on the card. I've done nothing else to the card other than copy all the files from it. So the files vanished between the point where I connected the Cybook and it showed on Calibre and when I disconnected and put the SD card into the card reader.

sirbruce · 05-15-2009, 07:33 PM

Wow, I hope you had backups...

I've never heard of using "Save to Disk" from the device page before. I'm not saying it's not supposed to work; I just don't know of anyone who uses it that way. Generally Save to Disk is for your Calibre database, not copying files off your device.

Halk · 05-15-2009, 07:38 PM

Not directly backups as such.

I have everything on it in various places across my hard drives - I've just redid my PC storage, and I now have a single 2.7TB volume, with everything that was on roughly a dozen drives collected together. So everything is there somewhere.

What was on the Cybook was basically books that I'd bought, free books, and some fanfic that I'd had on my hard drive and various other things that I'd just thrown onto the Cybook over time. So while I don't have an exact backup of the stuff, I do have it all somewhere. The collection of stuff on it wasn't exactly organised or particularly formatted or anything either, so nothing of value has been lost.

I needed to start from scratch anyway, and only put on books that were in a good state.

kovidgoyal · 05-15-2009, 07:40 PM

You're saying that a SAve to disk operation on the files in your SD card caused them to be replaced with zero size files? the only way I can see that happening is if you tried to save the files on top of the existing files.

Halk · 05-15-2009, 07:44 PM

Yes, not all of the files though. The directory I attempted to save to twice was eBooks on my E:\ drive, I'm certain it was the E:\ drive and not the SD card drive because I was careful not to select the eReading directory below... had I selected the SD card drive then it wouldn't have shown the eReading directory.

Halk · 05-15-2009, 07:57 PM

I've been able to replicate it.

. Drop a single .txt file into the SD card.
. Start up Calibre. Wait until it finds the Cybook and reads it, and the SD card.
. Switch to the SD card listing in Calibre
. Select a the text document
. Right click and from the context menu select "Save to disk > Save to disk"
. In the file browser window select e:\eBooks and select OK
. To verify the file is now blank either view it in Calibre, or view it in Explorer to see it's 0 bytes in length.

kovidgoyal · 05-15-2009, 08:21 PM

What you are doing is trying to save the original file over itself. That's what is causing this. Do the following:

Connect your cybook
In calibre select the SD card
Choose the book
click save to disk and choose c:\Temp (or some other directory on c:\)

Now open the directory in windows explorer and you will see tat the file is saved fine

Halk · 05-15-2009, 08:26 PM

No, I'm not trying to save the file over itself. E: is a hard drive array, not the SD card. I've been very careful to ensure that.

However to ensure it's not an issue saving to anything but the C: drive I did the same thing, and aimed at C:\temp. The result was an erased .txt file.

Edit : To be clear, nothing appears on the E:\eBooks directory, or the C:\temp directory. My guess at what appears to be happening is similar to what you're saying. The file is indeed writing over itself, however it's not doing that because I selected the wrong place to put it, it seems to be doing it because regardless of where I tell Calibre to put it, Calibre tries to put it on top of itself.

kovidgoyal · 05-15-2009, 08:36 PM

Ah i was suspicious because eBooks is the name of the folder under which the Cybook stores its ebook files as well. I'm going to have to pass this to user_none, since the Cyook support is his baby.

Halk · 05-15-2009, 08:40 PM

Sure. It's not a big deal for me, since I now know to avoid that. However it looks to me that it might not be Cybook specific. You might want to try to replicate it on a Sony to see if it does the same thing, or if it is indeed down to some part of the Cybook code. The Cybook code, I would guess, is just reading from the device, the generic code in Calibre would be the part doing the writing, and that's the part that doesn't seem to be pointing in the right place. That's pure guesswork though.

If you'd like me to try other things to help sort out this problem I'm more than happy to do so. Just drop me a PM and I'll get an instant email. I also hope I don't seem ungrateful about Calibre, I very much appreciate you putting so much effort into it.

user_none · 05-16-2009, 08:17 AM

The closest I've been able to reproduce is if you choose the same directory the file is in on the SD card. That will cause the file to be truncated. I will be committing a fix to prevent this soon. But to other directories and drives it works correctly for me.

The generic driver code is what is doing the reading and it only opens the file in read only mode. It then copies the file from the device to the destination opened as write only. Since the file is opened in read only mode Calibre cannot write to the file at all. If it were to try to write to the file Python would not allow it and throw an exception. Unless they are the same location in which case it currently truncates.

Can you try using save to disk with a book on the Cybooks main memory. This will help to deduce if you have a bad memory card.

Halk · 05-16-2009, 10:23 AM

I've just checked and...If I do it from the Cybook's main memory, it does the same thing.

From what you've described the code doing, the simple explanation seems to be that Calibre is ignoring where I tell it to write the file to, and writing on top of itself.

I'm using Calibre 5.11 on Vista 64 SP1 (UAC off), with all drivers up to date, and all patches applied.

Halk · 05-16-2009, 11:49 AM

I thought I'd make a video, so you can be sure I'm not trying to save it on top of itself.

Have a look.

sirbruce · 05-16-2009, 01:05 PM

Can you confirm, after that video, that the txt file on the device is actually zeroed out, and not simply the file that was saved to disk? Check the device via USB on the disk, not via Calibre.

Halk · 05-16-2009, 01:38 PM

The video confirms that. It shows me opening the file on the SD card. Then it shows me telling Calibre to save it to the C:\ drive (nothing appears there) and then it shows that the file has been overwritten with an empty file. The video does not show me looking on the C:\ drive.

The point of the video is to show that I have not told Calibre to send the file on top of itself by accident.

05-15-2009, 07:26 PM	#1
Halk Fanatic Posts: 534 Karma: 469999 Join Date: Feb 2008 Location: Scotland Device: Sony PRS-650 (PRS+ alpha - thanks Kartu!)	Text files blanked I can't be sure that Calibre was the cause of this - and no real damage has been done, since I had the files elsewhere... I've just downloaded Calibre and installed it to see if I can get my head around using it to organise and reformat the books I have. I plugged in the Cybook, and Calibre generated an error. Similar to this one. I closed Calibre and restarted it, and this time it picked up the Cybook correctly, and read in the files on the Cybook (none) and on the SD card (600 or so books). I thought I might as well empty the Cybook so I was starting fresh, and from there only put on books that were in a proper state. So I selected all of them and went for save to disk, and then specified a directory on my hard drive. It processed the request for about a minute and a half on the task manager, and then reported it was complete. No files appeared on the hard drive. I then repeated but instead selected the save to disk all in one folder. Same thing happened, and again no files on the disk. Having Calibre send files to the Cybook would be nice, but it's not exactly required since I can do this using Windows explorer. I then took the SD card out and put it in a card reader to copy all the files off... and then noticed that all the .txt files were 0KB in length. Every .txt file had been erased. Other formats were fine. Since I last read one of the text files the only thing I've done is throw a rather large PDF file on the Cybook, which it didn't like and displayed blank pages. So I'm suspecting that something may have gone wrong with Calibre. In any case I don't want to point any fingers around, but I thought I'd mention it in case there's some kind of glitch. Edit : All .txt files are blank. As are all .prc files, and some (but not all) HTML files. Edit 2 : It now looks more likely that it was Calibre, as when I first connected the Cybook it showed 1.6GB free on the card, it now shows 1.7GB on the card. I've done nothing else to the card other than copy all the files from it. So the files vanished between the point where I connected the Cybook and it showed on Calibre and when I disconnected and put the SD card into the card reader. Last edited by Halk; 05-15-2009 at 07:33 PM. Reason: 1. Accuracy 2. Further info

05-15-2009, 08:26 PM	#8
Halk Fanatic Posts: 534 Karma: 469999 Join Date: Feb 2008 Location: Scotland Device: Sony PRS-650 (PRS+ alpha - thanks Kartu!)	No, I'm not trying to save the file over itself. E: is a hard drive array, not the SD card. I've been very careful to ensure that. However to ensure it's not an issue saving to anything but the C: drive I did the same thing, and aimed at C:\temp. The result was an erased .txt file. Edit : To be clear, nothing appears on the E:\eBooks directory, or the C:\temp directory. My guess at what appears to be happening is similar to what you're saying. The file is indeed writing over itself, however it's not doing that because I selected the wrong place to put it, it seems to be doing it because regardless of where I tell Calibre to put it, Calibre tries to put it on top of itself. Last edited by Halk; 05-15-2009 at 08:29 PM. Reason: Addition

05-16-2009, 10:23 AM	#12
Halk Fanatic Posts: 534 Karma: 469999 Join Date: Feb 2008 Location: Scotland Device: Sony PRS-650 (PRS+ alpha - thanks Kartu!)	I've just checked and...If I do it from the Cybook's main memory, it does the same thing. From what you've described the code doing, the simple explanation seems to be that Calibre is ignoring where I tell it to write the file to, and writing on top of itself. I'm using Calibre 5.11 on Vista 64 SP1 (UAC off), with all drivers up to date, and all patches applied. Last edited by Halk; 05-16-2009 at 10:32 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
help with formatting text files	chooky	Workshop	2	11-26-2009 04:16 AM
Text files with # and ##	p3aul	Calibre	6	10-13-2009 03:10 PM
Do need help editing text files?	Nate the great	Workshop	3	04-01-2009 01:18 PM
Text tool for formatting Gutenberg text files	bob_ninja	Workshop	5	11-13-2007 12:28 PM
Reading Text Files ?	grayfox	iRex	8	10-17-2007 11:20 PM

05-15-2009, 07:33 PM	#2
sirbruce Provocateur Posts: 1,859 Karma: 505847 Join Date: Feb 2009 Location: Columbus, OH Device: Kindle Touch, Kindle 2, Kindle DX, iPhone 3GS	Wow, I hope you had backups... I've never heard of using "Save to Disk" from the device page before. I'm not saying it's not supposed to work; I just don't know of anyone who uses it that way. Generally Save to Disk is for your Calibre database, not copying files off your device.

05-15-2009, 07:38 PM	#3
Halk Fanatic Posts: 534 Karma: 469999 Join Date: Feb 2008 Location: Scotland Device: Sony PRS-650 (PRS+ alpha - thanks Kartu!)	Not directly backups as such. I have everything on it in various places across my hard drives - I've just redid my PC storage, and I now have a single 2.7TB volume, with everything that was on roughly a dozen drives collected together. So everything is there somewhere. What was on the Cybook was basically books that I'd bought, free books, and some fanfic that I'd had on my hard drive and various other things that I'd just thrown onto the Cybook over time. So while I don't have an exact backup of the stuff, I do have it all somewhere. The collection of stuff on it wasn't exactly organised or particularly formatted or anything either, so nothing of value has been lost. I needed to start from scratch anyway, and only put on books that were in a good state.

05-15-2009, 07:40 PM	#4
kovidgoyal creator of calibre Posts: 43,842 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You're saying that a SAve to disk operation on the files in your SD card caused them to be replaced with zero size files? the only way I can see that happening is if you tried to save the files on top of the existing files.

05-15-2009, 07:44 PM	#5
Halk Fanatic Posts: 534 Karma: 469999 Join Date: Feb 2008 Location: Scotland Device: Sony PRS-650 (PRS+ alpha - thanks Kartu!)	Yes, not all of the files though. The directory I attempted to save to twice was eBooks on my E:\ drive, I'm certain it was the E:\ drive and not the SD card drive because I was careful not to select the eReading directory below... had I selected the SD card drive then it wouldn't have shown the eReading directory.

05-15-2009, 07:57 PM	#6
Halk Fanatic Posts: 534 Karma: 469999 Join Date: Feb 2008 Location: Scotland Device: Sony PRS-650 (PRS+ alpha - thanks Kartu!)	I've been able to replicate it. . Drop a single .txt file into the SD card. . Start up Calibre. Wait until it finds the Cybook and reads it, and the SD card. . Switch to the SD card listing in Calibre . Select a the text document . Right click and from the context menu select "Save to disk > Save to disk" . In the file browser window select e:\eBooks and select OK . To verify the file is now blank either view it in Calibre, or view it in Explorer to see it's 0 bytes in length.

05-15-2009, 08:21 PM	#7
kovidgoyal creator of calibre Posts: 43,842 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	What you are doing is trying to save the original file over itself. That's what is causing this. Do the following: Connect your cybook In calibre select the SD card Choose the book click save to disk and choose c:\Temp (or some other directory on c:\) Now open the directory in windows explorer and you will see tat the file is saved fine

05-15-2009, 08:36 PM	#9
kovidgoyal creator of calibre Posts: 43,842 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Ah i was suspicious because eBooks is the name of the folder under which the Cybook stores its ebook files as well. I'm going to have to pass this to user_none, since the Cyook support is his baby.

05-15-2009, 08:40 PM	#10
Halk Fanatic Posts: 534 Karma: 469999 Join Date: Feb 2008 Location: Scotland Device: Sony PRS-650 (PRS+ alpha - thanks Kartu!)	Sure. It's not a big deal for me, since I now know to avoid that. However it looks to me that it might not be Cybook specific. You might want to try to replicate it on a Sony to see if it does the same thing, or if it is indeed down to some part of the Cybook code. The Cybook code, I would guess, is just reading from the device, the generic code in Calibre would be the part doing the writing, and that's the part that doesn't seem to be pointing in the right place. That's pure guesswork though. If you'd like me to try other things to help sort out this problem I'm more than happy to do so. Just drop me a PM and I'll get an instant email. I also hope I don't seem ungrateful about Calibre, I very much appreciate you putting so much effort into it.

05-16-2009, 08:17 AM	#11
user_none Sigil & calibre developer Posts: 2,488 Karma: 1063785 Join Date: Jan 2009 Location: Florida, USA Device: Nook STR	The closest I've been able to reproduce is if you choose the same directory the file is in on the SD card. That will cause the file to be truncated. I will be committing a fix to prevent this soon. But to other directories and drives it works correctly for me. The generic driver code is what is doing the reading and it only opens the file in read only mode. It then copies the file from the device to the destination opened as write only. Since the file is opened in read only mode Calibre cannot write to the file at all. If it were to try to write to the file Python would not allow it and throw an exception. Unless they are the same location in which case it currently truncates. Can you try using save to disk with a book on the Cybooks main memory. This will help to deduce if you have a bad memory card.

05-16-2009, 11:49 AM	#13
Halk Fanatic Posts: 534 Karma: 469999 Join Date: Feb 2008 Location: Scotland Device: Sony PRS-650 (PRS+ alpha - thanks Kartu!)	I thought I'd make a video, so you can be sure I'm not trying to save it on top of itself. Have a look.

Advert

Advert

05-16-2009, 01:05 PM	#14
sirbruce Provocateur Posts: 1,859 Karma: 505847 Join Date: Feb 2009 Location: Columbus, OH Device: Kindle Touch, Kindle 2, Kindle DX, iPhone 3GS	Can you confirm, after that video, that the txt file on the device is actually zeroed out, and not simply the file that was saved to disk? Check the device via USB on the disk, not via Calibre.

05-16-2009, 01:38 PM	#15
Halk Fanatic Posts: 534 Karma: 469999 Join Date: Feb 2008 Location: Scotland Device: Sony PRS-650 (PRS+ alpha - thanks Kartu!)	The video confirms that. It shows me opening the file on the SD card. Then it shows me telling Calibre to save it to the C:\ drive (nothing appears there) and then it shows that the file has been overwritten with an empty file. The video does not show me looking on the C:\ drive. The point of the video is to show that I have not told Calibre to send the file on top of itself by accident.