Mini kobo mini with 4993 WORKS -YIPPEEEEE.HOW I DID IT - Page 3

davidfor · 05-23-2013, 12:01 AM

One of the performance problems is opening the shelf list. As far as I can tell it is purely about building that list. It doesn't matter where the books are, how big they are or whether they have covers. The time taken increases with the number of shelves, and the number of books on any shelf.

The shelf list shows all the shelves and a count of the books on them. At first glance it is simple to write an SQL statement to fetch the list. But, there are some complications.

The main complication is the SD card. If you insert an SD card, all the books on it are added to the database. You can then put these on shelves. If you remove the SD card, the books are not removed from the database. But, they are no longer listed in the library. And the count of books on the shelves does not include them.

Even with that complication, it takes to long. I had a play with this earlier in the week. I can produce the appropriate results with the following query:

Code:

SELECT name, COUNT(c.contentID)
FROM shelf s LEFT OUTER JOIN shelfContent sc ON s.name = sc.shelfname
LEFT OUTER JOIN content c ON c.contentid = sc.contentid
WHERE (c.contentType = 6 or c.contentType  IS NULL)
AND (externalId IS NULL OR externalid = '90164EF8' OR externalid = '')
AND s._IsDeleted = 'false'
AND (sc._IsDeleted = 'false' OR sc._IsDeleted is NULL)
GROUP BY name
ORDER BY name

That takes no time at all on my PC. That is the version when the SD card is inserted. Remove the second test on externalId when it isn't.

I also had a play with the indices. I added a couple that I thought might help, but didn't get any performance improvements.

From the above, I can only assume that someone has written some bad code that iterates over the database and manually counts everything. Or maybe the device is low enough powered to not be able to execute the query quickly. I would love to look at the code and see what they did. Alternatively, I might have missed something in my query and there is a legitimate reason for the slow speed.

And for the interested, "externalId" appears to be an id for the external SD card. I don't know where it comes from, but any book with this set, is on the SD card. The value is also in the "Kobo Reader.conf" in a line "90164EF8=true".

TechniSol · 05-23-2013, 12:08 AM

I had forgotten that they didn't store covers for books on the SDHC in past versions(and seem to have dropped creation again as the implementation was not the proper resolution as noted by some forum members).

Maybe it's asking a lot, but it seems to me that they'd be better off by at least a factor of 1:3 by just including the high resolution version of the cover in the epub like Calibre does in its processing. But, then I guess that would only be a kepub concern for Kobo. It'd be nice if they could do it for both kepubs and epubs.

TechniSol · 05-23-2013, 12:15 AM

I have a hard time believing that it should take anywhere near 90 seconds to assemble and start displaying a list even with 5000 items with a 1GHz processor and anything resembling efficient code, but as David has pointed out without a look under the hood, who knows?

davidfor · 05-23-2013, 12:18 AM

Quote:

Originally Posted by Uschiekid

The reason I bring my case up though, is that I orginally didn't have "create covers" in calibre checked. then i did. so only some were sent to the glo before the kobo had a chance to create them. but the covers that the kobo supposedly had already created, would seem to recreate every time. on the main memory. and lag EVERYTHING.

I felt like what might have made the big difference for my case was that when i took all my epubs/pdfs off my main memory and put them on a storage card, that the storage card started fresh and only had book related info on it. i really don't know if this is the reason, but when i looked at all the images saved on the internal memory...well there are something like 5000. it's like every time i had recommended books come at me, those were saved, everytime i searched in the kobo store, those were saved, etc, etc, etc. insane. but some other images that might be important...so i haven't just gone and deleted them all.
whereas my external drive has exactly 3x#of books on card.

You might have been hit by a bug in the driver. In one version, it sent covers to the main memory when books were sent to the SD card. If the image files on the main memory start with "file____mnt_sd", you can delete them. If you don't sideload any books to the main memory, you can also delete any images that start with "file____mnt_onboard".

Images for kepubs and recommendations and things from Kobo, will have a 22 character id. From what I have seen, these are cleaned up properly. If you delete these, the device will still be OK, but a blank or a dot will be shown for the cover of these books.

PeterT · 05-23-2013, 12:42 AM

Quote:

Originally Posted by davidfor

Code:

SELECT name, COUNT(c.contentID)
FROM shelf s LEFT OUTER JOIN shelfContent sc ON s.name = sc.shelfname
LEFT OUTER JOIN content c ON c.contentid = sc.contentid
WHERE (c.contentType = 6 or c.contentType  IS NULL)
AND (externalId IS NULL OR externalid = '90164EF8' OR externalid = '')
AND s._IsDeleted = 'false'
AND (sc._IsDeleted = 'false' OR sc._IsDeleted is NULL)
GROUP BY name
ORDER BY name

I've browsed most of the executables and libraries in the firmware, and I will admit I don't see any elaborate queries there involving shelves.

Uschiekid · 05-24-2013, 09:22 AM

Quote:

Originally Posted by davidfor

You might have been hit by a bug in the driver. In one version, it sent covers to the main memory when books were sent to the SD card. If the image files on the main memory start with "file____mnt_sd", you can delete them. If you don't sideload any books to the main memory, you can also delete any images that start with "file____mnt_onboard".

Images for kepubs and recommendations and things from Kobo, will have a 22 character id. From what I have seen, these are cleaned up properly. If you delete these, the device will still be OK, but a blank or a dot will be shown for the cover of these books.

The reason I was so blown away by the number of images was that a lot are books I've definitely never had on my glo. Some are books I've never even heard of. Or authors I wouldn't dream of reading.

When I did a simple search of files starting with "file", the only thing that came up was 7 images for 3 library books I currently have sideloaded on the main memory (2 I haven't opened yet, so don't have the biggest image file).

It takes my computer ages just to open all the thumbnails in the folder. So it's a bit time consuming to look at exactly what all is there. The properties say 4,959 items, totalling 176.7 MB. I had a few more when I had my sideloaded epubs on main memory, but calibre seems to have cleaned it up fine.

But I have only about 220 kepubs on my main memory (plus the 3 library books and 1 txt file). So even if I had all three images for all of them, it shouldn't be more than around 660 images. Even if calibre has also sent images for the kepubs that match an epub in my library (which I doubt it does) that shouldn't add more than 110x3 images, so still under 1000 images total. And based on what you're saying I shouldn't have most of those 4000 extra files?

I haven't gone into the kobo bookstore on my glo in ages (months) and I haven't gotten recommendations for almost as long (one of the firmwares stopped me getting images since my account has no bought books, now when I go into the bookstore screen it just shows a screen telling me that if i buy my first book, I get a discount...). So I'm assuming these images were collected quite some time ago.

They all have the 22 character ID which you describe, so it's pretty impossible to weed out ones I don't have the books for. But I'm not keen on having all my kepubs lose their covers, so I don't want to just delete them all. Any ideas?

speakingtohe · 05-24-2013, 10:40 AM

Quote:

Originally Posted by Uschiekid

The reason I was so blown away by the number of images was that a lot are books I've definitely never had on my glo. Some are books I've never even heard of. Or authors I wouldn't dream of reading.

When I did a simple search of files starting with "file", the only thing that came up was 7 images for 3 library books I currently have sideloaded on the main memory (2 I haven't opened yet, so don't have the biggest image file).

It takes my computer ages just to open all the thumbnails in the folder. So it's a bit time consuming to look at exactly what all is there. The properties say 4,959 items, totalling 176.7 MB. I had a few more when I had my sideloaded epubs on main memory, but calibre seems to have cleaned it up fine.

But I have only about 220 kepubs on my main memory (plus the 3 library books and 1 txt file). So even if I had all three images for all of them, it shouldn't be more than around 660 images. Even if calibre has also sent images for the kepubs that match an epub in my library (which I doubt it does) that shouldn't add more than 110x3 images, so still under 1000 images total. And based on what you're saying I shouldn't have most of those 4000 extra files?

I haven't gone into the kobo bookstore on my glo in ages (months) and I haven't gotten recommendations for almost as long (one of the firmwares stopped me getting images since my account has no bought books, now when I go into the bookstore screen it just shows a screen telling me that if i buy my first book, I get a discount...). So I'm assuming these images were collected quite some time ago.

They all have the 22 character ID which you describe, so it's pretty impossible to weed out ones I don't have the books for. But I'm not keen on having all my kepubs lose their covers, so I don't want to just delete them all. Any ideas?

If it was me I would delete the cover files and resend the books, but I am at times a bit hasty

.

Maybe make a mirror image first or copy books to a new library and use a newly formatted SD card (they are not too expensive) and send books to it.

Helen

davidfor · 05-24-2013, 10:41 AM

I just had a look in my images directory. There are 768 images. So, that's about 256 books. The library list says 102 books. That suggest 252 covers from somewhere else. They probably are for recommendations, as I have bought a few books and do a sync a couple of times a week.

The database has a row for each book. This includes the imageId. You could delete all images that don't have an imageId in the database. That needs a little SQL and a script. And I think I would be lazy. Move all the images out of the directory. And then generate a script to move the images that have an imageId in the database back.

Uschiekid · 05-24-2013, 11:16 AM

Quote:

Originally Posted by speakingtohe

If it was me I would delete the cover files and resend the books, but I am at times a bit hasty

.

Maybe make a mirror image first or copy books to a new library and use a newly formatted SD card (they are not too expensive) and send books to it.

Helen

this is only for my kepubs's images....i have all my epubs on an SD card and they aren't the issue...that images folder only has the books on the card. i don't believe there is a way to force kepubs onto an SD card (other than filling the whole main memory...)

Uschiekid · 05-24-2013, 11:22 AM

Quote:

Originally Posted by davidfor

I just had a look in my images directory. There are 768 images. So, that's about 256 books. The library list says 102 books. That suggest 252 covers from somewhere else. They probably are for recommendations, as I have bought a few books and do a sync a couple of times a week.

The database has a row for each book. This includes the imageId. You could delete all images that don't have an imageId in the database. That needs a little SQL and a script. And I think I would be lazy. Move all the images out of the directory. And then generate a script to move the images that have an imageId in the database back.

Unfortunately beyond me.
but guess i could just delete all my kepubs and then all the images and then sync the kepubs back on....fun fun.

or I could just leave it. it's not really a big deal, it's more that it's annoying when i'm looking to see if a certain book cover is there or the like, .....5000 is a lot to go through.

would logging in and out clean up the images files or only with a factory reset?

i forget what you lose with a log in and out...do you lose reading stats? i like my reading stats

davidfor · 05-24-2013, 11:29 AM

I don't think logging out would help. It replaces the database, but, I don't think it touches the files. It definitely doesn't touch the sideloaded books or cover images. I don't know if it does anything with the purchased kepubs. And you would lose the reading stats for any books not synced to the Kobo servers.

Uschiekid · 05-24-2013, 11:30 AM

Quote:

Originally Posted by davidfor

I don't think logging out would help. It replaces the database, but, I don't think it touches the files. It definitely doesn't touch the sideloaded books or cover images. I don't know if it does anything with the purchased kepubs. And you would lose the reading stats for any books not synced to the Kobo servers.

ok, thanks.

speakingtohe · 05-24-2013, 11:16 PM

Quote:

Originally Posted by Uschiekid

Unfortunately beyond me.
but guess i could just delete all my kepubs and then all the images and then sync the kepubs back on....fun fun.

or I could just leave it. it's not really a big deal, it's more that it's annoying when i'm looking to see if a certain book cover is there or the like, .....5000 is a lot to go through.

would logging in and out clean up the images files or only with a factory reset?

i forget what you lose with a log in and out...do you lose reading stats? i like my reading stats

Hmmmm your reading stats must be more accurate than mine. One kepub said I had only .3 hours to go to read 170 pages. and it always seems to tell me I only read 20-22 minutes at a time. I wiped them all out and read one page and still 22 minutes.

Helen

Uschiekid · 05-25-2013, 12:05 AM

Quote:

Originally Posted by speakingtohe

Hmmmm your reading stats must be more accurate than mine. One kepub said I had only .3 hours to go to read 170 pages. and it always seems to tell me I only read 20-22 minutes at a time. I wiped them all out and read one page and still 22 minutes.

Helen

I wasn't referring to kepubs....but rather my tally on hours read, books read, etc.

But kepubs that are built right give fairly accurate stats. Try Anna Karenina...that's the kepub I read for the longest and when the kepub stats were introduced, I was quite glad to get the stats on that one

vugtitan · 05-29-2013, 01:16 PM

Quote:

Originally Posted by TechniSol

I have a hard time believing that it should take anywhere near 90 seconds to assemble and start displaying a list even with 5000 items with a 1GHz processor and anything resembling efficient code, but as David has pointed out without a look under the hood, who knows?

as i said just using text,rtf,doc seems to be fine on the kobo mini.

i am up to 5000 now and i plan to dump another 4000 as soon as i have converted from mobi to rtf format using calibre.

i just copy/paste.

also i bought a nook glow and wow it does not suffer the slow down problem at all as i have put about 4000 pdfs and epubs on to it using a micro sd card.

it does not read txt,rtf,doc,mobi files.only pdf with reflow and epub.

no slow downs at all.

it must be using a master file indexing table instead of going through all the files which is what causes the problem.

this is called SMART PROGRAMMING.

browsing through the books is very very smooth.

also it has a explorer type folder browser so you can put different authors in different folders.

i.e it has folder support which is the holy grail of any ebook readers.

you ereader makers listening?

who in their right mind wants masochistically to spend years of their lives laboriously creating shelves 1 book by 1 book?

it is better than the kindle which has a 3000 book limit after which it becomes sluggish unusable or freezes/reboots at random.again poor programming and no folder support.

i might even sell the 3g keyboard kindle which cost me £150 plus £50 for the replacement 3g keyboard kindle when the screen developed lines.

05-23-2013, 12:01 AM	#31
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	One of the performance problems is opening the shelf list. As far as I can tell it is purely about building that list. It doesn't matter where the books are, how big they are or whether they have covers. The time taken increases with the number of shelves, and the number of books on any shelf. The shelf list shows all the shelves and a count of the books on them. At first glance it is simple to write an SQL statement to fetch the list. But, there are some complications. The main complication is the SD card. If you insert an SD card, all the books on it are added to the database. You can then put these on shelves. If you remove the SD card, the books are not removed from the database. But, they are no longer listed in the library. And the count of books on the shelves does not include them. Even with that complication, it takes to long. I had a play with this earlier in the week. I can produce the appropriate results with the following query: Code: SELECT name, COUNT(c.contentID) FROM shelf s LEFT OUTER JOIN shelfContent sc ON s.name = sc.shelfname LEFT OUTER JOIN content c ON c.contentid = sc.contentid WHERE (c.contentType = 6 or c.contentType IS NULL) AND (externalId IS NULL OR externalid = '90164EF8' OR externalid = '') AND s._IsDeleted = 'false' AND (sc._IsDeleted = 'false' OR sc._IsDeleted is NULL) GROUP BY name ORDER BY name That takes no time at all on my PC. That is the version when the SD card is inserted. Remove the second test on externalId when it isn't. I also had a play with the indices. I added a couple that I thought might help, but didn't get any performance improvements. From the above, I can only assume that someone has written some bad code that iterates over the database and manually counts everything. Or maybe the device is low enough powered to not be able to execute the query quickly. I would love to look at the code and see what they did. Alternatively, I might have missed something in my query and there is a legitimate reason for the slow speed. And for the interested, "externalId" appears to be an id for the external SD card. I don't know where it comes from, but any book with this set, is on the SD card. The value is also in the "Kobo Reader.conf" in a line "90164EF8=true".

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Mini kobo mini sale!	desert dog	Kobo Reader	0	12-22-2012 12:08 AM
How do you like your Kobo Mini?	bcgirl	Kobo Reader	8	12-14-2012 01:40 PM
Kobo Mini: So, which is it?	twowheels	Kobo Reader	5	12-13-2012 09:19 AM
Mini Kobo Mini	robdrob	Kobo Reader	57	10-31-2012 04:29 PM
Kobo Desktop for PC Only works if you have a Kobo !	Kevin2960	Kobo Reader	10	12-08-2011 12:48 PM

05-23-2013, 12:08 AM	#32
TechniSol GranPohbah-Fezzes r cool! Posts: 1,056 Karma: 3151024 Join Date: Jul 2010 Device: Nook STRs, Kobo Touch, Kobo Glo	I had forgotten that they didn't store covers for books on the SDHC in past versions(and seem to have dropped creation again as the implementation was not the proper resolution as noted by some forum members). Maybe it's asking a lot, but it seems to me that they'd be better off by at least a factor of 1:3 by just including the high resolution version of the cover in the epub like Calibre does in its processing. But, then I guess that would only be a kepub concern for Kobo. It'd be nice if they could do it for both kepubs and epubs.

05-23-2013, 12:15 AM	#33
TechniSol GranPohbah-Fezzes r cool! Posts: 1,056 Karma: 3151024 Join Date: Jul 2010 Device: Nook STRs, Kobo Touch, Kobo Glo	I have a hard time believing that it should take anywhere near 90 seconds to assemble and start displaying a list even with 5000 items with a 1GHz processor and anything resembling efficient code, but as David has pointed out without a look under the hood, who knows?

05-24-2013, 10:41 AM	#38
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	I just had a look in my images directory. There are 768 images. So, that's about 256 books. The library list says 102 books. That suggest 252 covers from somewhere else. They probably are for recommendations, as I have bought a few books and do a sync a couple of times a week. The database has a row for each book. This includes the imageId. You could delete all images that don't have an imageId in the database. That needs a little SQL and a script. And I think I would be lazy. Move all the images out of the directory. And then generate a script to move the images that have an imageId in the database back.

05-24-2013, 11:29 AM	#41
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	I don't think logging out would help. It replaces the database, but, I don't think it touches the files. It definitely doesn't touch the sideloaded books or cover images. I don't know if it does anything with the purchased kepubs. And you would lose the reading stats for any books not synced to the Kobo servers.

Advert

Advert