View Full Version : Firmware Update AKC! - Automatic Kindle Collection Generator


thermopyle
11-28-2010, 09:15 PM
Hey, first post here, just wanted to share a program I wrote for my Kindle 3 with you guys because this place was a great information resource for while I was writing the thing.

AKC! is a console-based application written in C# that generates collections on your Kindle based on your Kindle's directory structure. Running it is as simple as putting it in the root directory of your Kindle, double-clicking on it, and then you're done. No user-input required, though .net 4.0 needs to be installed.

The program basically allows for emulating a "View by Folders" option on the Kindle, which sadly is not a built-in function. With that in mind, this program does nothing more than run through the directory structure of your Kindle's documents folder and adds every file to a collection based on the sub-directories it's in. For example:

x:\documents\Jim Butcher\Dresden Files\Storm Front.mobi

Storm Front is added to the collections Jim Butcher and Dresden files. As I said, it's small. :)

I suggest using this program with Calibre. If you do so, you can create all kinds of collections based on Calibre column values. You can, for example, add a Genre1 and Genre2 field, putting books into multiple categories such as Paranormal and Mystery, or an Read Status field, with values of Read and Unread for easy sorting of which books you've yet to go through. Simply adding the column and including it in the "Send to Disk" template, then running AKC!, will allow you to easily sort your books according to any categories you wish.

Now, as I'm sure most people are aware, there are tricks to get the kindle to display collections alphabetically (choose "Sort by Title" and have a character before the collection name, such as ( or [ to force it to the top) Here are the two I'm using, plus an extra, in from top to bottom as the Kindle will display them (with this program, you have to use symbols that can be used in folder names):

( -- These are my author collections

[ -- These are my series collections

{ -- This would follow the above two. Just FYI if anybody has a use for it.


Here is my Calibre "sending to device" template: ({author_sort}/[{series}/{title} - {authors}

If you look at it, you'll realize that if the author or series is missing, you'll end up with a folder named ( or [, respectively. The program will automatically add files in such folders to (No Collection and [No Collection, respectively. This acts as a catch-all for those books that do not belong in any other collection. For me, standalone books will end up in [No Collection, as a grouping of books that are not in a series. Same with books without a defined author; they'll appear in (No Collection. You can use whatever template you wish, and the program will use those symbols interchangeably with your template.

So, to use the program:

Download the program here (http://freakykillerhats.info/CollectionGenerator.62.zip).
Backup your collections.json file from your your /system/ folder(this program currently does not do so).
Put CollectionGenerator in the root directory of your Kindle.
Double-click on CollectionGenerator.
Reboot your Kindle, and your collections will be there. ^_^

Note: I do not have any DRM books, so I do not know if this will work for them. Let me know if it does, please. If not, I'll try to fix it. :)

Version information: Current version is .62

.6 - Initial Version
.61 - ( now creates a (No Collection collection, similar to [ created [No Series originally. [ now gives the more generic [No Collection to fit other directory templates.
.62 - Fixed a bug that caused partial matches to be dumped together. For example, books in a directory named Dr. Who Adventures would be added instead to the Dr. Who collection, if both are present.

Planned:

.7 - Will add the ability to leave user-defined collections intact. Users can create their own collections with a leading ~, which the program would leave alone when generating the directory-based collections.

daffy4u
11-28-2010, 09:22 PM
Awe-SOME! Thanks!

tmb20x
11-29-2010, 01:15 AM
I wonder if your program makes a new .json file?
Thank for your sharing

thermopyle
11-29-2010, 01:50 AM
I wonder if your program makes a new .json file?
Thank for your sharing

It makes a new collections.json file in the /system folder. Be sure to back up your existing copy first, or rename it, if you want to be able to switch back to it after trying the program out or if you prefer your previous collections file. It will overwrite whatever is already present.

thermopyle
11-30-2010, 03:01 AM
I'm showing a number of downloads for the program, and plenty of views of the thread, but no real feedback so far.

For the people that are viewing but not downloading, why aren't you downloading? Are you already using another collection program and not interested in trying a different one, or does this one not fit what you're looking for (if so, why not?)

For those who have downloaded it...how did it work for you? Any issues? Recommendations?

Feedback is definitely appreciated. :)

tmb20x
11-30-2010, 06:26 AM
I'm showing a number of downloads for the program, and plenty of views of the thread, but no real feedback so far.

For the people that are viewing but not downloading, why aren't you downloading? Are you already using another collection program and not interested in trying a different one, or does this one not fit what you're looking for (if so, why not?)

For those who have downloaded it...how did it work for you? Any issues? Recommendations?

Feedback is definitely appreciated. :)

I'm using KindleCollectionManager_v0.4.1 and I feel good. The only thing that your program is better than KindleCollectionManager_v0.4.1 is that your program is small. I think the most people are interested in is the ability of making sub-collection (collections in collection)

Scott Nielsen
12-01-2010, 03:37 AM
For the people that are viewing but not downloading, why aren't you downloading? Are you already using another collection program and not interested in trying a different one, or does this one not fit what you're looking for (if so, why not?)

Not quite what I'm looking for. I'm still hoping that one of you clever folks will eventually come up with solution that allow the Calibre tags - which I maintain meticulously - to create collections on the Kindle.

thermopyle
12-01-2010, 10:04 AM
Not quite what I'm looking for. I'm still hoping that one of you clever folks will eventually come up with solution that allow the Calibre tags - which I maintain meticulously - to create collections on the Kindle.

You can use this program for that. Simply have your "Send to Disk" format include the Calibre tag column you wish to use as part of the file path output, then after the file is sent to the Calibre, run my program.

For example, set your Calibre save to disk template like this:

#Custom_Field/{title} - {authors}

This would put your files on the Kindle at this path:

x:/documents/Custom_Field/Storm Front - Jim Butcher.mobi (an example).

Then when you run my program, whatever tag value you have in Custom_Field will be used to create the collection, and any books with that particular tag value will be added to it.

At least, this should work. I haven't done it myself (I don't use any tags other than author, title, and series), but as long as you have Calibre output the file with the tag as the path, my program can use the directories it creates accordingly.

Baldrake
12-01-2010, 06:04 PM
That's very cool.

Scott Nielsen
12-02-2010, 12:59 AM
Then when you run my program, whatever tag value you have in Custom_Field will be used to create the collection, and any books with that particular tag value will be added to it.

At least, this should work. I haven't done it myself (I don't use any tags other than author, title, and series), but as long as you have Calibre output the file with the tag as the path, my program can use the directories it creates accordingly.

Ah, interesting. If a tag creates a directory structure, does that mean if a book have, say, three tags assigned to it there will be three separate copies created on the Kindle?

thermopyle
12-02-2010, 09:47 AM
Ah, interesting. If a tag creates a directory structure, does that mean if a book have, say, three tags assigned to it there will be three separate copies created on the Kindle?

No, the only thing the program does is add a book to a collection based on each subdirectory it's in. For example, if your book is sent from Calibre to this user-defined location:

x:/documents/Custom_Field_value/Author/Series/book.mobi

book.mobi will be added to the Custom_Field_value (Read/unread, etc., whatever you output), Author, and Series collections.

dujik
12-02-2010, 10:52 AM
I tried it and its not working but probably because i`m not doing it right so let me ask you this;
Will your program work if i dont use calibre in any way and i named files as i want.
So for example i put two random books in the same folder x:/documents/ebook/
How do i achieve to collection "ebook" appear on my DX with two books under it
Is the file name crucial?
Please just dont mention Calibre in your answer, thanks.

thermopyle
12-02-2010, 11:00 AM
I tried it and its not working but probably because i`m not doing it right so let me ask you this;
Will your program work if i dont use calibre in any way and i named files as i want.
So for example i put two random books in the same folder x:/documents/ebook/
How do i achieve to collection "ebook" appear on my DX with two books under it
Is the file name crucial?
Please just dont mention Calibre in your answer, thanks.

The program does not require Calibre at all, it simply builds off the directory structure you've created on your Kindle (Calibre just makes those directories if you use it.) You can create any directories you want and the program won't know the difference between that and Calibre-generated folders.

Having said that, where are you running the program from? I just want to make sure you have it in the correct location, which is the root directory of your Kindle. Just put it on the Kindle and run it from there (it uses relative pathing and so needs that as its starting location.)

If that doesn't work, is you collections.json file in the system folder? If not, where is it? Does the program crash or give you an error? Does AKC simply not make changes to your collections, with your previous collections staying as they were, or does it wipe those out and leave you with none?

daffy4u
12-02-2010, 11:07 AM
The program does not require Calibre at all, it simply builds off the directory structure you've created on your Kindle (Calibre just makes those directories if you use it.) You can create any directories you want and the program won't know the difference between that and Calibre-generated folders.


So, it seems that if you have one book that would fall into more than one collection (tag, category...), you would have to have more than one copy of the book on the Kindle (which a couple of people were asking).

If I want a book in the "mystery" and also "paranormal" collections, I'd have to have a copy of the book in each folder? And after the book has been read I'd have to add another copy to the "books I've read" collections? Am I understanding this correctly?

reech
12-02-2010, 11:25 AM
Smart, this works very well for me. However, there's still no damnable 'putting your series in the right order' feature in the Kindle! At least with 3G I can always look up ;)

thermopyle
12-02-2010, 11:59 AM
So, it seems that if you have one book that would fall into more than one collection (tag, category...), you would have to have more than one copy of the book on the Kindle (which a couple of people were asking).

If I want a book in the "mystery" and also "paranormal" collections, I'd have to have a copy of the book in each folder? And after the book has been read I'd have to add another copy to the "books I've read" collections? Am I understanding this correctly?

Ah, I understand what you're asking now. The answer is yes and no. Even if books are in separate locations on the Kindle, they'll still end up in the same collection if the directory name is identical. If you have multiple columns you use for genres and read/unread status, you can have those used in the output Calibre uses (or make the directories yourself, which would be a pain.) Here's an example:

In Calibre, if you add and use fields for Genre1, Genre2, Read Status (values being read and Unread), here could be your save to disk template:

({author_sort}/[{series}/{#genre1}/{#genre2}/{#read_status}/{title} - {authors}

Here would be the output of a book whose genres are Mystery and Paranormal, which has been read:

x:/documents/Jim Butcher/Dresden Files/Mystery/Paranormal/Read/Storm Front.mobi

Now, Calibre will not output a variable if there is no value for it. If you have Genre1 set but not Genre2, you'd get this instead:

x:/documents/Jim Butcher/Mystery/Dresden Files/Read/Storm Front.mobi

Now, the key thing here is that when the AKC program is run, books are added to a collection based on the directory name. So, these three will all be added to the same Mystery collection:

x:/documents/Jim Butcher/Dresden Files/Mystery/Paranormal/Read/Storm Front.mobi
x:/documents/Dan Brown/Robert Langton/Mystery/Unread/Da Vinci Code.mobi
x:/documents/Mystery/Unread/Collection of Mystery Stories.mobi

Note that the second and third examples above would also end up in the Unread collection.

Basically, if you can have a directory name in the path, no matter where that directory name is at in the path, (as long as it's a sub-directory of documents, anyway), then the book will appear in that collection, as will every other book with that directory name in its path.

Now, there is one downside to this: If you have two authors with the exact same name, their books will end up in the same collection. Or if you have two series with the same name (two authors happen to write trilogies with the same name, titled Boy Saves World, for example), those will also show up in the same collection. Either case is pretty rare, however. Even if this were to happen, Author1's collection would have the BSW trilogy, as would Author2's, so the books would be separate there.

So, while you can't use the comma-seperated "Tag" field in Calibre to organize books by collection, using unique values in your own columns and then having them as part of the "Save to Disk" path works just fine. Genre1 and Genre2 and Read_Status would be good examples of this.

NOTE: If you DID try to have multiple copies of the same book on your Kindle, in the way you specifically asked about, it would actually create multiple entries if the directories were repeated. For example:

x:/documents/Paranormal/Jim Butcher/Dresden Files/Storm Front.mobi
x:/documents/Mystery/Jim Butcher/Dresden Files/Storm Front.mobi

This would end up putting Storm Front in the Paranormal collection and the Mystery collection, yes, but then you would have two copies of Storm Front in both the Jim Butcher and Dresden Files collections. So, no, you wouldn't want to take this approach. :)

daffy4u
12-02-2010, 12:09 PM
Thanks for explaining that! I was concerned about having multiple copies of the same book on the Kindle. For folks with large libraries (the ones who REALLY REALLY need your software), it would cause major problems with space.

I guess I'll have to spend some of the my holiday time off, adding/fixing tags for my books in Calibre before I can try your software. I'm quite lazy when it comes to managing my ebook collection. :o

Thanks again. :)

thermopyle
12-02-2010, 12:13 PM
Smart, this works very well for me. However, there's still no damnable 'putting your series in the right order' feature in the Kindle! At least with 3G I can always look up ;)

There is, too. At least, if I'm right in what you mean. :)

Newer versions of Calibre have a feature called Plugboard. It allows for some use of variables based on column values in how the metadata of a book is written when the book is sent to the device.

Check out this link for some useful examples of Plugboard expressions: http://www.mobileread.com/forums/showthread.php?t=100949&highlight=series+order.

The one I'm using is the "General Metadata" one in the first post. It makes the title metadata of the book lead with an initialization of the series title, the number of the book in the series in brackets, a dash, and then the name of the book (the author's name trailing on the right). I'll give you example of how it works. If you had these columns in Calibre:

title author series (book number in brackets)

Storm Front - Jim Butcher - Dresden Files [01]
Fool Moon - Jim Butcher - Dresden Files [02]

What would be output (this is what you'd see while reading on your Kindle):

DF [01] - Storm Front ---- Jim Butcher
DF [02] - Fool Moon ---- Jim Butcher

I definitely recommend checking it out. It's awesome and really, really helpful. Just remember to have leading 0's in the book numbers if you're going to have more than 9 book in a series. Otherwise you'll get this:

Series [1] - title ----author
Series [10] - title -----author
Series [11] - title -----author
Series [2] - title ------author

reech
12-02-2010, 12:24 PM
Ohh - I'll try that later, when back in.

Just tried to use AKC on a couple of series (10 books altogether) it worked fine. Sadly, then moving another 100 books on with about 10 series - no joy, and it's removed ALL the collections - will have a look later on... (And yes, did restart both times!)

thermopyle
12-02-2010, 12:25 PM
Thanks for explaining that! I was concerned about having multiple copies of the same book on the Kindle. For folks with large libraries (the ones who REALLY REALLY need your software), it would cause major problems with space.

I guess I'll have to spend some of the my holiday time off, adding/fixing tags for my books in Calibre before I can try your software. I'm quite lazy when it comes to managing my ebook collection. :o

Thanks again. :)

You're definitely welcome. ^_^ I need to load a bunch more books on my Kindle to see how long the AKC! program takes with a full one. With 400 books, my 3-year-old laptop takes less than 2 seconds to create all of the collections necessary. I'll be very impressed with myself if it's less than 20 seconds for 4000.

Now, I will admit there is one annoying issue that would accompany this type of program approach: You'll end up with a LOT of directories in your Kindle. If you're browsing through the Kindle's contents withe file explorer, that could be a pain. Those directories are invisible to Calibre and in the Kindle itself, however, so it should not cause any usage problems for the vast majority of people. Trying to manage the collections by hand in the form of creating these directories yourself would be a huge orderal, and at that point it would probably be better to use another collection management program. AKC! was written primarily to be used with Calibre, as using the two together allows for the maximum amount of automation. I'm a big fan of less work. :)

thermopyle
12-02-2010, 12:30 PM
Ohh - I'll try that later, when back in.

Just tried to use AKC on a couple of series (10 books altogether) it worked fine. Sadly, then moving another 100 books on with about 10 series - no joy, and it's removed ALL the collections - will have a look later on... (And yes, did restart both times!)

Let me know how that goes. At this point I'm not sure if the program has issues with DRM (I may have to buy a book or two with DRM to test). Did any of the initial 10 books you loaded have DRM? If not, how about the 100 books you did afterwards? If it's not working, please send me your collections.json so I can take a look at what AKC! output to that file.

daffy4u
12-02-2010, 12:36 PM
A couple of other questions.

1 - Do you think this will work with Kindle 2?
2 - If not, will the collections sync from K3 to K2 via Amazon?

thermopyle
12-02-2010, 12:41 PM
A couple of other questions.

1 - Do you think this will work with Kindle 2?
2 - If not, will the collections sync from K3 to K2 via Amazon?

If the Kindle 2 uses the same method of handling collections, it should work just fine. What I would suggest doing is renaming/backing up your collections.json file and running it as a quick test. There's no real consequence if it doesn't work, or if the results are not what you wanted. Simply delete the AKC!-generated collections.json and restore the one you were previously using. :)

dujik
12-03-2010, 05:01 AM
The program does not require Calibre at all, it simply builds off the directory structure you've created on your Kindle (Calibre just makes those directories if you use it.) You can create any directories you want and the program won't know the difference between that and Calibre-generated folders.

Having said that, where are you running the program from? I just want to make sure you have it in the correct location, which is the root directory of your Kindle. Just put it on the Kindle and run it from there (it uses relative pathing and so needs that as its starting location.)

If that doesn't work, is you collections.json file in the system folder? If not, where is it? Does the program crash or give you an error? Does AKC simply not make changes to your collections, with your previous collections staying as they were, or does it wipe those out and leave you with none?

Sorry for late answer.
Ok i found a bug,if there is mbp file along with the book the whole thing is
not working.
I manage to create a collection with clean mobi files(this is the only way that is working for me) and when i start reading mbp file creates itself and then later when i add book in the same folder
and reboot kindle the whole collection disappears because of previous created mbp file.

Mixx
12-03-2010, 08:39 AM
So, while you can't use the comma-seperated "Tag" field in Calibre to organize books by collection, using unique values in your own columns and then having them as part of the "Save to Disk" path works just fine. Genre1 and Genre2 and Read_Status would be good examples of this.


Could someone create a piece of code in Calibre to just take the string of "tags" (separated by commas), disassemble it into the list elements (=tags), sort them to be a unique list and turn that list into a directory structure? That way, we could use tags to organize.

Example:

A book has the tags: History, English, Fiction
The tags variable: "History, English, Fiction"
Sorted: "English, Fiction, History
Directory: X:/documents/English/Fiction/History/book.mobi

As far as I understand, that way each book has a unique folder location, has just one copy and ACK can generate the collections from the segments of the path.

What do you think? I'd do it but have almost zero programming skills.

Regards, Mixx

thermopyle
12-03-2010, 10:28 AM
Sorry for late answer.
Ok i found a bug,if there is mbp file along with the book the whole thing is
not working.
I manage to create a collection with clean mobi files(this is the only way that is working for me) and when i start reading mbp file creates itself and then later when i add book in the same folder
and reboot kindle the whole collection disappears because of previous created mbp file.

I don't believe the .mbp file is the problem. The program is written to ignore those when generating the collection, and I've tested it with no problems. I just tested again, just to be sure; I generated the collections, read a book, rebooted the Kindle, everything was fine. Then I ran AKC! again (with the .mbp file in place), rebooted, and things are still okay.

What was the exact process you went through? You said after you started reading you added a book to the same folder; did you move the book you were reading, or simply add a different book? Did you run AKC! again after adding that book (to add it to the collections), or did you leave the collections as-is?

thermopyle
12-03-2010, 10:33 AM
Could someone create a piece of code in Calibre to just take the string of "tags" (separated by commas), disassemble it into the list elements (=tags), sort them to be a unique list and turn that list into a directory structure? That way, we could use tags to organize.

Example:

A book has the tags: History, English, Fiction
The tags variable: "History, English, Fiction"
Sorted: "English, Fiction, History
Directory: X:/documents/English/Fiction/History/book.mobi

As far as I understand, that way each book has a unique folder location, has just one copy and ACK can generate the collections from the segments of the path.

What do you think? I'd do it but have almost zero programming skills.

Regards, Mixx

It's possible that the creator of Calibre could add this type of output creation. He seems pretty open to requests; have you tried asking him to do this?

It's something that could work well. Manual management of the tags field would still be necessary, I think (some of the books I have came with a ridiculous number of tags, which is unhelpful), but some people might like that. Personally I'd rather declare a few columns for tags that I'll use (as described previously), since with individual columns you can sort by content. A Read_Status column, for example; if you click on the column header, you can sort by whether you've read books in your library. Then it's easier to go through and make changes to that status as you read the books. But we all have different preferences, I suppose. :)

dujik
12-03-2010, 12:12 PM
What was the exact process you went through? You said after you started reading you added a book to the same folder; did you move the book you were reading, or simply add a different book? Did you run AKC! again after adding that book (to add it to the collections), or did you leave the collections as-is?

I simply add another book (one mobi file) and run AKC again.
Basically every time i run AKC and there is *.mob in it ,collection disappears or
it wont be created in the first place.
I`m using kindle DX ,software 2.5.5

thermopyle
12-03-2010, 12:16 PM
I simply add another book (one mobi file) and run AKC again.
Basically every time i run AKC and there is *.mob in it ,collection disappears or
it wont be created in the first place.
I`m using kindle DX ,software 2.5.5

Is it a specific book, or any book? Either way, please provide full file name(s) for those that are causing problems.

dujik
12-03-2010, 01:33 PM
Is it a specific book, or any book? Either way, please provide full file name(s) for those that are causing problems.

Small update.
If i create collection with one book with *.mbp file it will work and if i add another one also with mbp file it will not work.

In this link:
http://rapidshare.com/files/434692081/EBOOK.rar
...there are two books that i`m talking about ,(one of them collection works but if you put both it will not)


or if you dont want to download that here is link for their file names.
http://rapidshare.com/files/434685974/FILE_NAMES.txt

thermopyle
12-04-2010, 02:07 AM
Small update.
If i create collection with one book with *.mbp file it will work and if i add another one also with mbp file it will not work.

In this link:
http://rapidshare.com/files/434692081/EBOOK.rar
...there are two books that i`m talking about ,(one of them collection works but if you put both it will not)


or if you dont want to download that here is link for their file names.
http://rapidshare.com/files/434685974/FILE_NAMES.txt

I downloaded the ebook.rar, ran AKC! with that on the Kindle, looked at the collections.json file, and immediately saw the problem. :) I just had two statements in reverse order, so it was creating a partial file entry for a *.mbp file instead of ignoring those completely. I fixed the issue and the corrected file is now posted in place of the original. Please download and re-run that in order to confirm the fix.

Thanks for pointing that out. ^_^

reech
12-04-2010, 04:41 AM
Check out this link for some useful examples of Plugboard expressions: http://www.mobileread.com/forums/showthread.php?t=100949&highlight=series+order.

The one I'm using is the "General Metadata" one in the first post. It makes the title metadata of the book lead with an initialization of the series title, the number of the book in the series in brackets, a dash, and then the name of the book (the author's name trailing on the right)...:

...so this makes the title of the book Dresden Files 01 - Storm Front , with the Authgor still on the Author part - fair enough.

I forgot to do this last night (grrr!) so I'm having a go now. I've found the plugboard section, so I'll let that carry on over there.

I'll then combine this with AKC, and see what happens... (and I'll see if the DRM issue is there as well)

BTW, AKC author, have *you* successfully run it multiple times? As in loaded some on, run it, reset, saw it's okay, added more books, run it, reset, etc... Just want to see if *that's* the issue as well - although I can't see why it would be.

reech
12-04-2010, 05:01 AM
Right, I've done the plugboard naming thing. It works, and it'll help, but it's still not ideal (Having the series name then the number then the title AS the title is a bit untidy for my OCD mind ;>)

Really, Amazon should pull their finger out when it comes to series sorting, after all they do sell series of books, so they should figure out that it's a nightmare when you have a couple of books in a series (that you've bought from them!) - they must understand that you want to view them in series order!

dujik
12-04-2010, 05:05 AM
Yep , everything is working now perfectly.
I must say i like this program more than the previous mention alternatives because
its quick and simple (you don't have to install it and whatnot) .
Only way you could improve it is if reboot wasn't need it but i suppose that`s kindle
limitation.

Mixx
12-04-2010, 10:47 AM
It's possible that the creator of Calibre could add this type of output creation. He seems pretty open to requests; have you tried asking him to do this?


Hi thermopyle,

I have in this thread (http://www.mobileread.com/forums/showthread.php?p=1252717#post1252717) but I don't think I succeeded.

However, the discussion there lead me to this request to you:

Could you please consider an option or a version that would take a path element and if it is a comma separated list, then it would dissect this list and use its elements as collections to which the book is added?

Calibre can easily (via a template) generate a folder pattern that uses the tags (list) as an element. Thus a book ends up in, say, x:/documents/t1, t2, t3, t4/book.mobi.

You could in your C++ code very easily dissect that list and use its elements as names for collections, couldn't you?

Please consider it, I am desperate to find a way to easily use collections on the Kindle for my book (getting close to 1K).

Thanxx, Mixx

thermopyle
12-04-2010, 12:00 PM
I'll then combine this with AKC, and see what happens... (and I'll see if the DRM issue is there as well)

BTW, AKC author, have *you* successfully run it multiple times? As in loaded some on, run it, reset, saw it's okay, added more books, run it, reset, etc... Just want to see if *that's* the issue as well - although I can't see why it would be.

Yes, I was using this on my own Kindle until I got all the bugs I could find out, and continue to do so now. Trust me, there was some back and forth, and it took a few approaches to the problem before I figured out what worked. I also deleted and reloaded the books on my Kindle a few times in order to experiment with folder output options and plugboards. That's actually the reason that I missed the problem with multiple *.mbp files in the same collection; since I deleted the books I had no *.mbp files. When I tested that it was ignoring those correctly I simply opened up a book, read a bit of it, then made sure AKC! skipped the resulting .mbp file. There didn't seem to be any reason to try multiple *.mbp files. :)

Are you still having the issue you described earlier, where AKC! is not generating the collections correctly? If so, please send me your collections.json file so I can take a look. Please also include a listing of the files in your Kindle's document directory. You can do that by typing this at the command prompt while in the documents directory:

dir /s /b > books.txt

That will output a list of all your books to a text file called books.txt.

thermopyle
12-04-2010, 12:01 PM
Hi thermopyle,

I have in this thread (http://www.mobileread.com/forums/showthread.php?p=1252717#post1252717) but I don't think I succeeded.

However, the discussion there lead me to this request to you:

Could you please consider an option or a version that would take a path element and if it is a comma separated list, then it would dissect this list and use its elements as collections to which the book is added?

Calibre can easily (via a template) generate a folder pattern that uses the tags (list) as an element. Thus a book ends up in, say, x:/documents/t1, t2, t3, t4/book.mobi.

You could in your C++ code very easily dissect that list and use its elements as names for collections, couldn't you?

Please consider it, I am desperate to find a way to easily use collections on the Kindle for my book (getting close to 1K).

Thanxx, Mixx

Hm. I could probably do that. Please post the tag listing for a random five of your books to see what kind of setup you're using. The more tags per book the better. ^_^

thermopyle
12-04-2010, 12:03 PM
Yep , everything is working now perfectly.
I must say i like this program more than the previous mention alternatives because
its quick and simple (you don't have to install it and whatnot) .
Only way you could improve it is if reboot wasn't need it but i suppose that`s kindle
limitation.

It feels like I've rebooted mine a million times. It's annoying, yes. I'm sure it's possible somehow to edit the copy of the collection file that's been loaded into the Kindle's memory at the last boot, but honestly I've no idea how that would be approached. :)

thermopyle
12-04-2010, 12:06 PM
Right, I've done the plugboard naming thing. It works, and it'll help, but it's still not ideal (Having the series name then the number then the title AS the title is a bit untidy for my OCD mind ;>)

Before I found out about the plugboards, I had manually gone through all of my books and added 1 - Title, 2 - Title, etc., for each book in each series. Now that the plugboard is doing that automatically, I'm having to go back through and delete that part of the title (I'm having redundant book numbers at the beginning of each book.) The Kindle is fantastic to read on, but I really wish Amazon didn't make us jump through so many hoops for doing so.

NiLuJe
12-04-2010, 12:08 PM
Note that just restarting the framework is enough to reload collections, you don't have to do a full reboot (ie. usbnetwork/launchpad [or the autoreboot feature of the ss/font hack] is your friend).

thermopyle
12-04-2010, 12:28 PM
Note that just restarting the framework is enough to reload collections, you don't have to do a full reboot (ie. usbnetwork/launchpad is your friend).

Nice. Thanks for the tip, I'm checking that out now. :)

reech
12-05-2010, 05:02 PM
Thanks to thermopyle for some help on sorting and fixing an issue I was having - AKC! is definitely a timesaver, and easily complements any type of manual collection-thwapping tool ;)

After resetting my Kindle fully to get rid of an issue I had when I upgraded to the 3.0.3 firmware, AKC! has helped me rebuild the Collections quickly - well done!

Passepartout
12-08-2010, 07:36 AM
@thermopyle,
want to so say big thanks to you.
That was so easy. :thumbsup:

KecajN
12-08-2010, 07:42 PM
I'm showing a number of downloads for the program, and plenty of views of the thread, but no real feedback so far.

For the people that are viewing but not downloading, why aren't you downloading? Are you already using another collection program and not interested in trying a different one, or does this one not fit what you're looking for (if so, why not?)

For those who have downloaded it...how did it work for you? Any issues? Recommendations?

Feedback is definitely appreciated. :)

I have learned about your program only yesterday and tested it immediately. It is something that I has been looking for... Almost...

For me one key feature is missing: It should preserve and refresh existing collections that are not mirrors of folders.

Explanation: Currently I have 720 already loaded to Kindle (and it will be doubled as soon as I will solve collection management). My current collections span across 3 Kindle DX pages and it is not very convenient to access each ebooks directly from collection. So I created "~~~Active" collection to group all ebooks that I am currently reading. I am going to create another one "~~~Exam" to keep all documents needed to prepare to my certification exams at hand, etc. (Note: "~" character keep my collections at the top of the list)

From my point of view your program should only refresh such "special" (i.e. not mapped to a folder) collection by removing all entries of ebooks that no longer exists (i.e. were deleted from Kindle).

thermopyle
12-09-2010, 12:57 AM
I have learned about your program only yesterday and tested it immediately. It is something that I has been looking for... Almost...

From my point of view your program should only refresh such "special" (i.e. not mapped to a folder) collection by removing all entries of ebooks that no longer exists (i.e. were deleted from Kindle).

This isn't really a feasible solution from that angle, as it requires denoting special characters or some other function that allows such collections to be identified. There is, however, the easy possibility of changing the collections to instead use the existing file instead of creating a new one each time. That would work with the approach I've used in programming AKC! There's a bit of a pitfall in that, however, that I'll need to examine to figure out how to resolve. I'll think on it. :)

stanley_lulu
12-09-2010, 10:57 AM
thank you so much

thermopyle
12-09-2010, 02:29 PM
Actually, on second thought, although the idea I had for a different approach to that problem would work logically and be easy enough to program, the workaround for the pitfall would require making AKC! less automatic. So, what I can do is specify a user-defined collection type (preceded by ~ or something) and leave those collections intact when running AKC! They would be completely user-managed, the program would ignore those collections entirely and you could, if you deleted the books in such a collection, end up with an empty collection. You could delete it by hand, of course, using either the Kindle or another collection management program, but that's what user-managed means. :)

gertblij
12-09-2010, 02:42 PM
Love the idea of your program, but have been struggling for two evenings now and am giving up. Here is the problem I have (using Calibre):

- I create a custom field "Genre"
- In Genre I have many books as "Fiction", others as Non Fiction and yet others as "Dutch Fiction"
- The export to the Kindle is perfect. The folder structure is exactly right.

But when I run your program and reboot the Kindle, there is no collection for Fiction and Non Fiction and all the books are in the collection Dutch Fiction.

Btw, the same happens when I set up the Tags like this or split the two words as single words in "Genre1" and "Genre2" .

Any idea what I could be doing wrong?
TIA

gertblij
12-09-2010, 02:48 PM
While the AZW and PRC files are definitely in the right folders, but after your program has generated the collections, these files are not in any collection and are displayed on the Home screen after the collection list.

Any reason for this?
TIA

thermopyle
12-09-2010, 10:01 PM
Love the idea of your program, but have been struggling for two evenings now and am giving up. Here is the problem I have (using Calibre):

- I create a custom field "Genre"
- In Genre I have many books as "Fiction", others as Non Fiction and yet others as "Dutch Fiction"
- The export to the Kindle is perfect. The folder structure is exactly right.

But when I run your program and reboot the Kindle, there is no collection for Fiction and Non Fiction and all the books are in the collection Dutch Fiction.

Btw, the same happens when I set up the Tags like this or split the two words as single words in "Genre1" and "Genre2" .

Any idea what I could be doing wrong?
TIA

gertblij,

This problem has already been fixed, I just forgot to include a notice about it. Please redownload the program and run it again. Reech sent me a PM over the weekend regarding this issue and I figured out what the problem is and swapped in the new version on Sunday. The current version should work fine. :)

thermopyle
12-09-2010, 10:06 PM
While the AZW and PRC files are definitely in the right folders, but after your program has generated the collections, these files are not in any collection and are displayed on the Home screen after the collection list.

Any reason for this?
TIA

Are those DRM files? I haven't had anybody report on how they're working, so I'm unsure if they do.

gertblij
12-10-2010, 11:17 AM
Are those DRM files? I haven't had anybody report on how they're working, so I'm unsure if they do.

No, they are not DRM'd. It is not a big deal, because in my case there are only 4 of them, so adding themmanually is no big deal, but hey, why settle for anything less than perfection

gertblij
12-10-2010, 11:49 AM
gertblij,

This problem has already been fixed, I just forgot to include a notice about it. Please redownload the program and run it again. Reech sent me a PM over the weekend regarding this issue and I figured out what the problem is and swapped in the new version on Sunday. The current version should work fine. :)

Where do I do that?

I clicked on the "here" link in your very first post: ("So, in summary: Download the program here."), but that downloaded the original file (CollectionGenerator.6.exe - 5.12.10 - 12:41PM)

thermopyle
12-10-2010, 11:56 AM
Where do I do that?

I clicked on the "here" link in your very first post: ("So, in summary: Download the program here."), but that downloaded the original file (CollectionGenerator.6.exe - 5.12.10 - 12:41PM)

The file name is the same. 5/12/2010 (or 12/5/2010) is the date that the current version was uploaded. This weekend I'll change the first post, list what fixes/changes have been made since the initial posting, and do a better job of explaining what the program does. I should be able to add the requested ability to ignore user-defined collections as well.

thermopyle
12-10-2010, 11:57 AM
No, they are not DRM'd. It is not a big deal, because in my case there are only 4 of them, so adding themmanually is no big deal, but hey, why settle for anything less than perfection

Hm. Alright. Please post the file names and paths for those four so I can see what AKC! is tripping on.

Like this, please:

x:/documents/folder/folder/filename.ext

gertblij
12-10-2010, 01:40 PM
The file name is the same. 5/12/2010 (or 12/5/2010) is the date that the current version was uploaded. This weekend I'll change the first post, list what fixes/changes have been made since the initial posting, and do a better job of explaining what the program does. I should be able to add the requested ability to ignore user-defined collections as well.

Oopsie, in that case the problems is still the same. Here is what I did: as the directory structure on the Kindle has not changed (it is correct), I just replaced your program on the Kindle with the new download, ran it and rebooted the Kindle. However, it did not solve the problem as described. Nothing changed. Did I miss something?

FF2
12-10-2010, 02:31 PM
I don't know if the version still does not fix your problem or you have the wrong version - clear the cache in your browser and try again. Sometimes if a file appears the same, the browser looks as if it downloaded the new but does not and uses the one it already had in the cache.

thermopyle
12-10-2010, 03:06 PM
I don't know if the version still does not fix your problem or you have the wrong version - clear the cache in your browser and try again. Sometimes if a file appears the same, the browser looks as if it downloaded the new but does not and uses the one it already had in the cache.

Yes, I'd try that. I know it works because once it was pointed out, I saw why the problem was occuring immediately, and tested it on my own system by producing some directories with partial shared names (like /test/ and /testtest). Reech confirmed the issue was resolved for him, and it's the same problem you're having now.

I'll rename the file tonight or tomorrow to reflect the version change and avoid this download issue.

thermopyle
12-12-2010, 01:07 AM
I edited the first post to include the version history and updated the link to the current version. I also made some changes to the content of the post itself, in attempt to be clearer about its capabilities and how to use it for maximum automation and laziness. :)

gertblij: Please let me know if it's working for you now.

dujik
12-15-2010, 09:00 AM
Originally Posted by NiLuJe View Post
Note that just restarting the framework is enough to reload collections, you don't have to do a full reboot (ie. usbnetwork/launchpad is your friend).

Nice. Thanks for the tip, I'm checking that out now. :)

Any update on this? :)

ElizabethN
12-18-2010, 12:00 AM
Thanks for creating and revising AKC.

Ok, I like the concept and the simplicity of AKC but I'm a bit frustrated at the moment.

If I use a variation of the basic send to disk template, {author_sort}/_{series}/{title} - {authors} , then I have no problems. And, I would stop here, but I have 3000+ files that I would like to subdivide into more than just author, series collections.

But, if I try to add user generated columns, I run into problems.

for example, I first tried a column, science_fiction_fantasy, that was broken into SFF_Fant, SFF_SciFi, SFF_UFant (comma separated text, ... shown in tag browser)

using this template, {author_sort}/{#science_fiction_fantasy}/_{series}/{title} - {authors}, resulted in following type of file

x:\documents\Jim Butcher\#science_fiction_fantasy\Dresden Files\Storm Front.mobi

All that calibre apparently "sees" is the heading, not the data within the column.

I then tried just creating a yes/no column entitled SFF. That template, {author_sort}/{#SFF}/_{series}/{title} - {authors}, resulted in following type of file

x:\documents\Jim Butcher\yes\Dresden Files\Storm Front.mobi and "yes" and "no" collections.

My ultimate goal is a collection called "SFF" but I'm not there yet.

Any thoughts? Am I choosing the inappropriate column data format?

I'm doing all this on my old Kindle DX-2 running 2.5.6 since it has far fewer files than my everyday K3.

Thanks for any help.

thermopyle
12-18-2010, 01:03 PM
ElizabethN,

I'm not sure why the comma separated text approach works the way it does. I duplicated what you described and got the same result.

Having said that, however, I think the second approach you were using is the correct one for what you really want, but it needs to be adjusted slightly. If what you're looking for is a way to categorize your books as being Fantasy, SciFi, UFantasy (whatever this one is), the best way to do it is like this:

1. Add column of type "Text, column shown in tag browser" and name it Genre or whatever you want.
2. Set the genre values of your books to SciFi, Fantasy, or whatever you want the folder names to be (SFF, in your example, would also work).
3. Set your "send to disk" template to include /{#genre}/ somewhere in the output path.

That should be it for getting SFF or whatever in your folder output, at which point running AKC! will create the appropriate collection.

Let me know how that works!

ElizabethN
12-18-2010, 08:10 PM
1. Add column of type "Text, column shown in tag browser" and name it Genre or whatever you want.
2. Set the genre values of your books to SciFi, Fantasy, or whatever you want the folder names to be (SFF, in your example, would also work).
3. Set your "send to disk" template to include /{#genre}/ somewhere in the output path.

That should be it for getting SFF or whatever in your folder output, at which point running AKC! will create the appropriate collection.



Worked like a charm with 2 custom columns.

I also read parts of the calibre manual & mobileread threads related to plugboard and template creation and custom and composite column creation. So I played around with all that as well.

Only difficulties now are
1)samples from amazon (which are usually drm-free) will not go into a collection using AKC

2)most books downloaded from amazon will not go into a collection using AKC, although some did. this has actually been a problem with other collection managers that I've tried. will have to see if there's anything in common between them all.

Question - Since AKC appears to re-write collections each time you run it, would any books/files that I move manually into a collection be removed from that collection the next time AKC is run?

Before I devote large quantities of time to manually adding books to collections, I want to find out what happens. I have a few pdfs that I sideloaded onto the kindle that have remained in their original collection through several AKC runs. I'm going to try to manually move some of those uncategorized books and samples into collections. I'll reboot aka update the json file then save it. Then use calibre to put a few more books on the kindle, run AKC, update and see what happens.

Thanks for your help and the program.

ps. ufant = urban fantasy i.e, Patricia Briggs, Ilona Andrews...

pss - rewording - AKC creates the collection but the file that should go into the collection doesn't. collection shows as (0).

thermopyle
12-18-2010, 10:03 PM
Question - Since AKC appears to re-write collections each time you run it, would any books/files that I move manually into a collection be removed from that collection the next time AKC is run?
...

Thanks for your help and the program.


AKC! currently will overwrite any collections in the previous file. The next version will allow for author-maintained collections which will be ignored by the program--probably by the user adding a ~ prefix to the name of the collection, or some other identifying character.

You're welcome. :)

ElizabethN
12-25-2010, 10:24 PM
I noticed that none of my amazon samples and purchased books go into a collection without manually moving them into a collection using the kindle. This has been relatively consistent throughout trials of different kindle collection editors, including AKC!

...2) The weird asin thing I mentioned above. This isn't "fixed" as such, but I've come up with several workarounds, which are described in the now-huge comment at the top of the script file.

Basically, I noticed that all the affected books were ones purchased from Amazon, whether they were stripped of DRM or not. So the simplest workaround is to reconvert them twice: First convert them to epub format and then convert the epubs back to mobi format. That'll strip out that embedded metadata asin and allow them to be managed as normal.
...
http://www.mobileread.com/forums/showthread.php?p=1255247#post1255247 for script file

After reading through various threads, I came across Mornington's suggestion and gave it a try. I converted an azw file to epub then used the epub file to convert to mobi. I deleted the azw file from calibre and the original from the device and then send the new mobi file to the device. AKC! was able to add books to collections if the mobi, not azw, file was used. If I didn't delete the azw file then a revised azw file was sent to the device, even though a mobi file was also present. Without opening calibre, I can't remember if it sends over azw before mobi files if both are present.

For samples specifically, if the azw file was used, the kindle read the file as a "sample" with working links to the kindle store, but the file wouldn't go into a collection except manually. If the mobi file was used, the kindle no longer saw the file as a sample, so kindle store links didn't work, but AKC! was able to put the file into a collection.

So some of those empty collections that AKC! created were finally used.

AKC! currently will overwrite any collections in the previous file. The next version will allow for author-maintained collections which will be ignored by the program--probably by the user adding a ~ prefix to the name of the collection, or some other identifying character.


So, until the next version comes out with above ability to keep my manually tweaked samples in their designated collection, I will avoid messing with samples and just clean up the amazon files that currently won't go into collections. I have plenty of those to keep me busy.

thermopyle
12-27-2010, 10:38 AM
If I didn't delete the azw file then a revised azw file was sent to the device, even though a mobi file was also present. Without opening calibre, I can't remember if it sends over azw before mobi files if both are present.

I believe it does send the azw file by default. Just in case you are not aware of it, however, there is an option to send a particular format to the Kindle. Simply right-click on the books you want to transfer, mouse-over "Send to Device", and then go to "Send Specific Format To" and "Main Memory." When you click on that, it will pop up a screen asking which format you want sent to the Kindle. Simply choose mobi and that should do the trick.

I'll try to get the next version posted this weekend. Unfortunately the holidays have been a bit hectic otherwise I would have gotten that done earlier. :)

dixieknits
12-27-2010, 11:23 AM
I'm considering trying this as I have 10 pages of books to scroll through to find anything. Also each track of an audio book is showing up in the library.
I do not understand this
"No user-input required, though .net 4.0 needs to be installed."
What is .net 4.0?
I do not want to jailbreak, root or modify my Kindle. Will your application work as is. I still have no collections at all as that option is not avaiable for some reason.

thermopyle
12-27-2010, 12:03 PM
I'm considering trying this as I have 10 pages of books to scroll through to find anything. Also each track of an audio book is showing up in the library.
I do not understand this
"No user-input required, though .net 4.0 needs to be installed."
What is .net 4.0?
I do not want to jailbreak, root or modify my Kindle. Will your application work as is. I still have no collections at all as that option is not avaiable for some reason.

You have to register your Kindle in order to get collections to be a selectable option. That's quite easy, however--just enter your info, turn wireless on and connect to a router, and register it. Then turn wireless back off, if you wish.

.net 4.0 is a Windows framework. A lot of programs use .net. It's nothing to do with the Kindle--the .net framework has to be installed on the computer you're connecting the Kindle to, is all. You can download .net from Microsoft here: http://www.microsoft.com/downloads/en/details.aspx?FamilyID=9cfb2d51-5ff4-4491-b0e5-b386f32c0992&displaylang=en.

ElizabethN
12-27-2010, 04:45 PM
I believe it does send the azw file by default. Just in case you are not aware of it, however, there is an option to send a particular format to the Kindle. Simply right-click on the books you want to transfer, mouse-over "Send to Device", and then go to "Send Specific Format To" and "Main Memory." When you click on that, it will pop up a screen asking which format you want sent to the Kindle. Simply choose mobi and that should do the trick.

I'll try to get the next version posted this weekend. Unfortunately the holidays have been a bit hectic otherwise I would have gotten that done earlier. :)

Thanks! I forgot that I can do that with calibre, I've gotten so used to just hitting the button and using my previous default. That way I wouldn't have to lose the original azw file from calibre.

Whenever you get the next version released is fine, I appreciate what you've done so far. I've got plenty of files to clean up on the K3 before running AKC on it. I've just been practicing with AKC on our kindle DX as it has far fewer files.

Mike G
12-28-2010, 06:56 AM
Deleted (sorry, couldn't find the delete post button)

playful
11-11-2011, 09:08 PM
Hello thermopyle,

What a marvelous idea for a program! Exactly the functionality I've been wishing my Kindle had. What's the point of folders if you can't navigate them?

You mentioned that you coded AKC! for your K3. Do you happen to know if it will run on a DX? I'm on a DXG and very keen to try it, but don't want to break anything.

With gratitude,
Wishing you a beautiful day,

Andy

Slizide
01-05-2012, 09:24 PM
I don't know if anyone is still reading this thread but is there a way to use this program to automatically add books to collections that are sent through email? For example if I email a book to my kindle email address it would then automatically add it to a specified collection.

Thanks,
Adam