Glo [Glo HD] Strange files (parsed extention) in kobo-images directory

Nick_1964 · 10-01-2015, 05:47 PM

Hi..
Had some problems with the search.. for parsed and files it gives numerous hits.
In the kobo-images directory there are a big number of sub directory's without names.
Like FAkvhJeZMUqoRwrralLQQA - N3_LIBRARY_GRID.parsed
It was easy to find out that this are the cover files and perhaps the little tile files from the frontpage and maybe the index.
But,the books that belongs to those covers are deleted a long time ago.
Why does this files stay, they consume some space..
I did some database checks but the database is ok.
It is almost impossible to look by hand what pictures they are, but why do they stay there and more important, how can I avoid that they stay behind when a book is deleted and how can I clean them up ?

I tried with the kobo utility's with the option: Clean images directory of extra cover images, but they keep existing..
My "downstairs" reader contains only 8 books, the Kobo-images directory contains 191 directory's and 349 files !!!

I guess that when I connected the device first it did a sync with the bol.com bookshop and somehow synced all the covers of the books I buyed ??
So now what.. delete them all by hand ??
With the bol.com bookshop it is for as far as I know,impossible to set which book must be synced or not ( I know it is possible in the kobo shop )

And I am afraid that my "upstairs reader" contains at least the same amount of files.. ( bought more then 500 books...)
Anyone who can help ?
When I do a factory reset I am sure all books are deleted, but then it starts with the initial setup again..

The only thing I know is delete them all, and go into calibre, use the tab of books on the device,and then select them all,and use the option to send selected book covers to the device ??
But then,when I delete a book again,will the cover be deleted to ??
So many questions, so many possibilities, so many options,but which one to choose ?

davidfor · 10-01-2015, 07:53 PM

Yes, as you discovered the .parsed files are the cover images. For a book, there can be up to four different sizes depending on the firmware version. The recent firmware puts the files in the .kobo-images directory. Under there are two levels of numbered directories with the images in the lowest level. Exactly which directory the images are in is done from a hash based on the image file name. All cover images for a book will have the same name followed by a suffix depending on what it is for. The suffix starts with the "-". The file name is stored in the database against each book.

The example you gave is a the cover for a kepub. Or it could be for a recommendation or other books shown through some of the bookstore functions on the device. The name is a random looking string of letters and numbers. For sideloaded books, it is the full path of the book with certain characters changed to underscores.

Generally, the firmware is good at cleaning up the images as books are removed. This includes when recommendations and other store relate books are removed. In the past, when I found extra cover images, I could find a cause such as a crash or restoring the database (cover images downloaded after the backup was taken are orphaned). For sideloaded books, the only problem I know of is if there is a single quote in the name. In this case, the cover images are not removed when the books are deleted through the firmware. I put a fix in calibre recently to sanitise the file names when sending books so that this didn't happen.

Deleting a book from calibre should remove all the cover images. I have checked this fairly recently, so I am confident it works. If you find a case that it doesn't, then report it.

The image cleanup in the utilities plugin should also work. But, I don't run it often and haven't sat through a full check to make sure for a while. Nothing has changed in the database or the image file naming that would have affected. The cleanup removes all cover images that do not have a reference in the database. That means it will there will be more cover images than actual books on the device. The extras will be for recommendations, related books, wishlist items and previews. If you browse the store, I'm not sure how those covers are handled. Hopefully they are cached in memory while in the store and then discarded.

If there are cover images being left behind by the cleanup or deleting from calibre, then I'd have to look into it. I would of course need some information. A calibre debug log when the cleanup or delete was done would be the starting point. Plus the names of the cover images left behind. And the full path to these files. And there is a good chance I would need a copy of the database to look into why these files were not removed.

If you just want to remove them all, it is reasonably safe to just delete the ".kobo-images" directory or its contents. The cover images for sideloaded books will be regenerated the next time they are needed. I'm not sure what happens for downloaded books. My feeling has always been that the cover images are downloaded rather than generated on the device, but I haven't done much testing of this. If they are generated as needed, everything will be OK. If they are downloaded, you will probably see the default text cover for these books.

Nick_1964 · 10-01-2015, 08:14 PM

The files are left behind, but I really don't know if the books are deleted by hand or by calibre but most of them are from June the days that I bought both readers.. They are left behind, the books are not on the reader, but I only sideloaded by calibre and I only delete by the reader, so I don't add or delete books by using a filemanager, also the database is compressed almost 1.5 megabytes for just 6 books.. The plugin doesn't delete the files, some directory's are empty.. It looks like calibre or the plug-in handles the recent books correctly, but leaves the old ones untouched... So I delete them or do you want something as examples, then please tell me what you want me to do for your examination?

davidfor · 10-01-2015, 08:57 PM

Quote:

Originally Posted by Nick_1964

The files are left behind, but I really don't know if the books are deleted by hand or by calibre but most of them are from June the days that I bought both readers.. They are left behind, the books are not on the reader,

And was June the last time you synced the device and received recommendations from Kobo? If so, the cover images are probably for those books

Quote:

but I only sideloaded by calibre and I only delete by the reader, so I don't add or delete books by using a filemanager, also the database is compressed almost 1.5 megabytes for just 6 books.. The plugin doesn't delete the files, some directory's are empty..

The firmware doesn't delete the empty directories when books or cover images are removed. Doing this from calibre will remove empty directories. And yes, a "clean device" function would be handy in the utilities plugin to remove empty directories and and run the image cleanup and annotations file removal in one go.

Quote:

It looks like calibre or the plug-in handles the recent books correctly, but leaves the old ones untouched...

The image cleaning scans the .kobo-images directory and removes all cover images that do not have a reference in the database. So, unless there is a bug in how that is done, "the old ones" are still being referred to. The question is how. The only other reason I can think of for them not being removed is if the files were set to read-only.

Quote:

So I delete them or do you want something as examples, then please tell me what you want me to do for your examination?

If you think there is an error, then I need examples before I can fix it. I will need a copy of the database plus the names and paths to the cover images that you think should have been removed. Either send me a PM with a location I can download them from, or I can send you an email address (though you might already have that).

If you want to check yourself, you can look in the database. If you have an SQLite database manager, you can open the database and the following SQL statement will check for the cover image you mentioned in the first post.

Code:

SELECT * 
FROM content
WHERE ImageID = 'FAkvhJeZMUqoRwrralLQQA'

The value in the quotes is the part of the file name before the dash.

If running the query returns a row, then there is a book on the device for the cover image and the image cleanup should not touch it. If it doesn't return a row, then the image cleanup should remove the file. If you run the cleanup and the file isn't removed, then there is a bug.

Nick_1964 · 10-01-2015, 10:15 PM

I don't want to mess with the database, if u can send me a pm with your mail address I can send it later today and try to find out if it is also on the other reader, the covers are definitely from purchased books, as far as I can see from the first time I have used the reader, because that is the only sync I ever did, they are not from recommendation.. I never used the shop on both devices.. It is possible that I react late, it is 4 am now here, I have to go at 1 pm with 5 cat's to the vet and then I have to go to work. I can zip the whole content of the directory, maybe from both devices completely with databases if it is not to much work for you.. Then I leave it untouched.. Good night!

davidfor · 10-02-2015, 01:22 AM

Quote:

Originally Posted by Nick_1964

I don't want to mess with the database, if u can send me a pm with your mail address I can send it later today and try to find out if it is also on the other reader, the covers are definitely from purchased books, as far as I can see from the first time I have used the reader, because that is the only sync I ever did, they are not from recommendation.. I never used the shop on both devices.. It is possible that I react late, it is 4 am now here, I have to go at 1 pm with 5 cat's to the vet

I've taken two cats to a vet a few times, and that was fun. Getting the first into the carry box wasn't that hard. But, the second one knew what was going on. I can't imaging trying to corral five cats at once.

Quote:

and then I have to go to work. I can zip the whole content of the directory, maybe from both devices completely with databases if it is not to much work for you.. Then I leave it untouched.. Good night!

I have sent you my email address. A copy of the database and the full paths for the files you think should have been removed is what I need. Or, you can do a fill directory listing for the .kobo-images directory and put that into a file. This can be done from a command line by:

Code:

dir /s x:\.kobo-images > imageslist.txt

You need to replace the "x:" with whatever the drive letter for you Kobo is. And the file "imageslist.txt" will get the results of the dir command. Of course, that is Windows, if you are using something else, you will have to use the appropriate commands for that OS.

Nick_1964 · 10-02-2015, 11:01 AM

Quote:

Originally Posted by davidfor

I've taken two cats to a vet a few times, and that was fun. Getting the first into the carry box wasn't that hard. But, the second one knew what was going on. I can't imaging trying to corral five cats at once.

oops missed this message..
3 of them are totally blind, one partially, so the first thing to do is try to isolate them,then catch the one who can see,because when he sees me catching the others he is hanging on the ceiling,then try to keep them silent as long as possible so they can't hear each other because when one starts screaming, it is over !
Well.. in my next life i want to be a vet, over 300 euro's for checkup,vaccination and deworming and some extra to keep the supplies in my first aid kit up to date

Quote:

I have sent you my email address. A copy of the database and the full paths for the files you think should have been removed is what I need. Or, you can do a fill directory listing for the .kobo-images directory and put that into a file. This can be done from a command line by:

Code:

dir /s x:\.kobo-images > imageslist.txt

damn.. so stupid.. forgot this old dos command.. normally I needed the /p option to prevent it from stopping and ask me to press any key..
Device 2 has a staggering 54Megabytes of pictures.. and also a lot of empty directory's of old books.. looks like the shelves names are not deleted when removing a book.. I gonna dive into that.
i sended you some mail with path and names made by hand, if you need the full patch lists, let me know.

davidfor · 10-03-2015, 03:03 AM

OK, I have had a look at the database and the cover file name sent. The database has record of 133 books. These break down to:

6 sideloaded books
99 books are purchased from Kobo but are not on the device. Just the details are in the database. Five of these were downloaded on 02/06/2015 and then removed.
28 are recommendations that were downloaded 02/06/2015.

There will be at least one cover image file on the device for each of these books. There should be somewhere between 133 and 432 cover image files. The .kobo-images directory has some other images for thing like the awards.

Based on what you have said, I assume 02/06/2015 is the date you set this device up. Part of that was a sync that downloaded the details of all books you have bought from Kobo and the actual book for five of them. And the recommendations were also downloaded at this time. That is part of the set up of the device. As you haven't done a sync since then, they haven't been updated, but haven't been removed either.

From the sample cover image file names, all of them were referenced in the database. Hence, the image cleaning will not delete them.

The empty directories are because the firmware does not delete empty directories when deleting the books and the cover images. To me, this is the correct behaviour. If you are sideloading books into a directory, then you own the directory, not the firmware. Removing the directory would be the firmware making an assumption about the directory. For the image directories, these are generated as needed. Because of the way this is done, the directory can have cover images for more than one book. And as they will be reused, there is no real reason to remove them. The utilities plugin does remove them, but that was my decision when I wrote the code.

I have run an image clean on my Glo HD and one of the Touch's I have. In both cases, extra images were found. When I checked a few of these, there were no references to them in the database.

As far as I can tell, there is no problem here.

Nick_1964 · 10-03-2015, 06:14 AM

Quote:

Originally Posted by davidfor

OK, I have had a look at the database and the cover file name sent. The database has record of 133 books. These break down to:

6 sideloaded books
99 books are purchased from Kobo but are not on the device. Just the details are in the database. Five of these were downloaded on 02/06/2015 and then removed.
28 are recommendations that were downloaded 02/06/2015.

There will be at least one cover image file on the device for each of these books. There should be somewhere between 133 and 432 cover image files. The .kobo-images directory has some other images for thing like the awards.

Based on what you have said, I assume 02/06/2015 is the date you set this device up.

I think so, I really don't know when purchased the glo HD, it was at release date.

Quote:

Because of the way this is done, the directory can have cover images for more than one book. And as they will be reused, there is no real reason to remove them. The utilities plugin does remove them, but that was my decision when I wrote the code.

So I have to remove them manually ?? it is just that I don't like stuff on the internal card that doesn't "belong" there.
Because I don't re-sync the images would be there all time ?? it now takes the space of about 54 ebooks..
I am not a "utility writer" but I would love a utility that would scan the book directory's and remove images of covers that are not on the reader.
When I manually remove them, would the database will be upgraded ??
Thank you for your work, I think the best way is to "now and then" run the directory's and clean what is no longer there, but I see also a problem with that.. now the dates are simple to find back, all the same of the second of june.. but in later progress dates will be more different and more close to the real date.
I just don't like it that space is been occupied with something there is not longer there.
Also I don't see any possibility to stop sending reccomendations.. I just synced to see if that would remove some not used bookcovers and update the database,but in stead of that, 48 new recommendations are sended..

I am still looking to a way to force rebuilding of the database,does it grows more and more when entry's stay in it for books that are no longer there ??

davidfor · 10-03-2015, 09:10 AM

Quote:

Originally Posted by Nick_1964

I think so, I really don't know when purchased the glo HD, it was at release date.

So I have to remove them manually ?? it is just that I don't like stuff on the internal card that doesn't "belong" there.
Because I don't re-sync the images would be there all time ?? it now takes the space of about 54 ebooks..
I am not a "utility writer" but I would love a utility that would scan the book directory's and remove images of covers that are not on the reader.
When I manually remove them, would the database will be upgraded ??
Thank you for your work, I think the best way is to "now and then" run the directory's and clean what is no longer there, but I see also a problem with that.. now the dates are simple to find back, all the same of the second of june.. but in later progress dates will be more different and more close to the real date.
I just don't like it that space is been occupied with something there is not longer there.
Also I don't see any possibility to stop sending reccomendations.. I just synced to see if that would remove some not used bookcovers and update the database,but in stead of that, 48 new recommendations are sended..

I am still looking to a way to force rebuilding of the database,does it grows more and more when entry's stay in it for books that are no longer there ??

Firstly, the device is working as designed. All the cover images you are finding on the device are supposed to be there. They are for books that the device knows about. Whether the book file is there, doesn't really matter.

Those 99 books you bought will be displayed in the library list if you go in to the Library settings uncheck the "Only show downloaded items" option. If these really bug you, delete them. They will still be in your account, but in the trash. You can restore them out of the trash if you want, but they will be downloaded the next time you do a sync.

Recommendations are updated whenever you do a sync. If you don't sync, they won't change, but they won't go away. They will just be there occupying a tiny amount of space in the database and a little more space in the .kobo-images directory.

I have no idea why you are worried about space. You have a device with about 3.2GB of usable space. You have stated you only keep five or six books on it at a time. That leaves so much space going to waste that a few extra files just won't really be noticed.

If these files really bug you, just delete the .kobo-images directory and restart the device. That will quickly get rid of them and the next time a cover is needed for one of you books, the directory will created and the cover images generated. As long as you never do anything to make the device display the other covers, there should be no problems. And if you want really want to delete them manually, just do so. You know what books you have put on the device, anything else is extra and you can remove.

Nick_1964 · 10-03-2015, 10:42 AM

Quote:

Originally Posted by davidfor

Firstly, the device is working as designed. All the cover images you are finding on the device are supposed to be there. They are for books that the device knows about. Whether the book file is there, doesn't really matter.

I understand, I really do..

Quote:

Recommendations are updated whenever you do a sync. If you don't sync, they won't change, but they won't go away.

Thanks to clarify that !

Quote:

I have no idea why you are worried about space.

That is a "tick" maybe but on my harddisks in my computer I have about 8 terrabyte space, I even empty the windows temp directory.
I think it is from the time with my first computer, a 8086 with only 360K floppy's, I had 2 drives in it, A was running dos 2 and the B drive was a 360K floppy disk..
Used space for something that was not in use,was a luxury I don't had.
After that I got a really big harddrive with 10 meg. space running stacker on it.
I don't know why this is an issue for me, when I download a firmware update, I copy it to the patch directory.. but then delete it from the download directory.
It's my problem and I don't know why i am doing it.
I also remove temp directory's of programs,and most of the time with a de-installation there are empty directory's left.. I remove them all.
Has nothing to do with privacy.. it is a habit that "moved in" and never "moved out" again.
I even use "treesize professional" to see if some big files are left in my system...

10-01-2015, 05:47 PM	#1
Nick_1964 Bookworm Posts: 975 Karma: 768585 Join Date: Aug 2010 Location: Netherlands Device: Sony prs-650, Kobo Glo HD (2x), Kobo Glo	[Glo HD] Strange files (parsed extention) in kobo-images directory Hi.. Had some problems with the search.. for parsed and files it gives numerous hits. In the kobo-images directory there are a big number of sub directory's without names. Like FAkvhJeZMUqoRwrralLQQA - N3_LIBRARY_GRID.parsed It was easy to find out that this are the cover files and perhaps the little tile files from the frontpage and maybe the index. But,the books that belongs to those covers are deleted a long time ago. Why does this files stay, they consume some space.. I did some database checks but the database is ok. It is almost impossible to look by hand what pictures they are, but why do they stay there and more important, how can I avoid that they stay behind when a book is deleted and how can I clean them up ? I tried with the kobo utility's with the option: Clean images directory of extra cover images, but they keep existing.. My "downstairs" reader contains only 8 books, the Kobo-images directory contains 191 directory's and 349 files !!! I guess that when I connected the device first it did a sync with the bol.com bookshop and somehow synced all the covers of the books I buyed ?? So now what.. delete them all by hand ?? With the bol.com bookshop it is for as far as I know,impossible to set which book must be synced or not ( I know it is possible in the kobo shop ) And I am afraid that my "upstairs reader" contains at least the same amount of files.. ( bought more then 500 books...) Anyone who can help ? When I do a factory reset I am sure all books are deleted, but then it starts with the initial setup again.. The only thing I know is delete them all, and go into calibre, use the tab of books on the device,and then select them all,and use the option to send selected book covers to the device ?? But then,when I delete a book again,will the cover be deleted to ?? So many questions, so many possibilities, so many options,but which one to choose ?

10-03-2015, 03:03 AM	#8
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	OK, I have had a look at the database and the cover file name sent. The database has record of 133 books. These break down to: 6 sideloaded books 99 books are purchased from Kobo but are not on the device. Just the details are in the database. Five of these were downloaded on 02/06/2015 and then removed. 28 are recommendations that were downloaded 02/06/2015. There will be at least one cover image file on the device for each of these books. There should be somewhere between 133 and 432 cover image files. The .kobo-images directory has some other images for thing like the awards. Based on what you have said, I assume 02/06/2015 is the date you set this device up. Part of that was a sync that downloaded the details of all books you have bought from Kobo and the actual book for five of them. And the recommendations were also downloaded at this time. That is part of the set up of the device. As you haven't done a sync since then, they haven't been updated, but haven't been removed either. From the sample cover image file names, all of them were referenced in the database. Hence, the image cleaning will not delete them. The empty directories are because the firmware does not delete empty directories when deleting the books and the cover images. To me, this is the correct behaviour. If you are sideloading books into a directory, then you own the directory, not the firmware. Removing the directory would be the firmware making an assumption about the directory. For the image directories, these are generated as needed. Because of the way this is done, the directory can have cover images for more than one book. And as they will be reused, there is no real reason to remove them. The utilities plugin does remove them, but that was my decision when I wrote the code. I have run an image clean on my Glo HD and one of the Touch's I have. In both cases, extra images were found. When I checked a few of these, there were no references to them in the database. As far as I can tell, there is no problem here.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Glo Kobo Glo for Copy Files from USB to SD?	loreno	Kobo Reader	0	04-19-2015 07:48 AM
*.parsed files?	techie42	Kobo Reader	16	03-21-2014 01:08 AM
Glo What images are available for the Kobo Glo?	jamawg	Kobo Developer's Corner	6	11-15-2013 08:06 AM
PNG Images in ePUB Files not rendered on Kobo	DragonflyKing	ePub	12	10-12-2013 08:28 PM
Glo deDRMing kindle files for kobo glo	andamooka	Kobo Reader	5	01-02-2013 09:30 AM

10-01-2015, 07:53 PM	#2
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	Yes, as you discovered the .parsed files are the cover images. For a book, there can be up to four different sizes depending on the firmware version. The recent firmware puts the files in the .kobo-images directory. Under there are two levels of numbered directories with the images in the lowest level. Exactly which directory the images are in is done from a hash based on the image file name. All cover images for a book will have the same name followed by a suffix depending on what it is for. The suffix starts with the "-". The file name is stored in the database against each book. The example you gave is a the cover for a kepub. Or it could be for a recommendation or other books shown through some of the bookstore functions on the device. The name is a random looking string of letters and numbers. For sideloaded books, it is the full path of the book with certain characters changed to underscores. Generally, the firmware is good at cleaning up the images as books are removed. This includes when recommendations and other store relate books are removed. In the past, when I found extra cover images, I could find a cause such as a crash or restoring the database (cover images downloaded after the backup was taken are orphaned). For sideloaded books, the only problem I know of is if there is a single quote in the name. In this case, the cover images are not removed when the books are deleted through the firmware. I put a fix in calibre recently to sanitise the file names when sending books so that this didn't happen. Deleting a book from calibre should remove all the cover images. I have checked this fairly recently, so I am confident it works. If you find a case that it doesn't, then report it. The image cleanup in the utilities plugin should also work. But, I don't run it often and haven't sat through a full check to make sure for a while. Nothing has changed in the database or the image file naming that would have affected. The cleanup removes all cover images that do not have a reference in the database. That means it will there will be more cover images than actual books on the device. The extras will be for recommendations, related books, wishlist items and previews. If you browse the store, I'm not sure how those covers are handled. Hopefully they are cached in memory while in the store and then discarded. If there are cover images being left behind by the cleanup or deleting from calibre, then I'd have to look into it. I would of course need some information. A calibre debug log when the cleanup or delete was done would be the starting point. Plus the names of the cover images left behind. And the full path to these files. And there is a good chance I would need a copy of the database to look into why these files were not removed. If you just want to remove them all, it is reasonably safe to just delete the ".kobo-images" directory or its contents. The cover images for sideloaded books will be regenerated the next time they are needed. I'm not sure what happens for downloaded books. My feeling has always been that the cover images are downloaded rather than generated on the device, but I haven't done much testing of this. If they are generated as needed, everything will be OK. If they are downloaded, you will probably see the default text cover for these books.

10-01-2015, 08:14 PM	#3
Nick_1964 Bookworm Posts: 975 Karma: 768585 Join Date: Aug 2010 Location: Netherlands Device: Sony prs-650, Kobo Glo HD (2x), Kobo Glo	The files are left behind, but I really don't know if the books are deleted by hand or by calibre but most of them are from June the days that I bought both readers.. They are left behind, the books are not on the reader, but I only sideloaded by calibre and I only delete by the reader, so I don't add or delete books by using a filemanager, also the database is compressed almost 1.5 megabytes for just 6 books.. The plugin doesn't delete the files, some directory's are empty.. It looks like calibre or the plug-in handles the recent books correctly, but leaves the old ones untouched... So I delete them or do you want something as examples, then please tell me what you want me to do for your examination?

10-01-2015, 10:15 PM	#5
Nick_1964 Bookworm Posts: 975 Karma: 768585 Join Date: Aug 2010 Location: Netherlands Device: Sony prs-650, Kobo Glo HD (2x), Kobo Glo	I don't want to mess with the database, if u can send me a pm with your mail address I can send it later today and try to find out if it is also on the other reader, the covers are definitely from purchased books, as far as I can see from the first time I have used the reader, because that is the only sync I ever did, they are not from recommendation.. I never used the shop on both devices.. It is possible that I react late, it is 4 am now here, I have to go at 1 pm with 5 cat's to the vet and then I have to go to work. I can zip the whole content of the directory, maybe from both devices completely with databases if it is not to much work for you.. Then I leave it untouched.. Good night!