Website for the Sony Reader: Ajax, LaTeX and RSS - Page 2

Kosst Amojan · 12-05-2006, 04:48 PM

Ok, sounds like a plan.

But on the catagories I'm still not feeling you. Why just not have them nested, fiction>science fiction>hard sci-fi. An easy way to make sure it all rolls together is to have users submit their books to a admin who would double check their formatting and labeling.

Now how about some forums. I was thinking that user input would drive deciding which books get formated first. And on that subject what formats do you want to offer other then PDF? If any?

Hadrien · 12-05-2006, 04:58 PM

Quote:

Originally Posted by Kosst Amojan

Ok, sounds like a plan.

But on the catagories I'm still not feeling you. Why just not have them nested, fiction>science fiction>hard sci-fi. An easy way to make sure it all rolls together is to have users submit their books to a admin who would double check their formatting and labeling.

Now how about some forums. I was thinking that user input would drive deciding which books get formated first. And on that subject what formats do you want to offer other then PDF? If any?

If we receive thousands of books, it'll be really hard to double check everything. I agree that the tag system may not be perfect, but it is also a lot more dynamic than something based on a classification. I'll work a whole lot during the beta and afterwards to improve the search and similar books features of the website, which is in my view a lot more usefull than a plain list of books. Anyway, if people really want some nested tree for the categories, I'll add it too.

User inputs should be the following:
- Comments on almost anything
- The ability to rate books
- We'll log the number of downloads for every book/author
- A favorite system, it'll be a way for you to bookmark things first, but after a while it'll be used as a "taste maker". We'll recommend books based on what's in your favorites etc...

For books in the public domain, it'll be possible for users to send books that are supposed to be in the public domain. We'll then check the title to see if it's the case, and then add it to the website. It could be pretty easy too to add a "most wanted" page where people could ask for a specific book.

We'll offer PDF first, since it's pretty much portable and good looking using LaTeX (we'll always have a table of contents for example). We'll see next what could be the best suited for the website. We'll also offer A4 PDF, because even though it seems that the Sony PRS 500 is doing quite well, there's still a lot more people looking for books to read on their computer than on a portable device.

Kosst Amojan · 12-05-2006, 07:23 PM

OK, one last question...how will the duel language system work? Just to be honest...and I have no idea how large the French market for this is but it looks to me like the English one will be much larger.

BTW: I'm from Southern Louisiana so I understand spoken Cajun french ok but I read it like a 4 year old.

Hadrien · 12-05-2006, 07:48 PM

Quote:

Originally Posted by Kosst Amojan

OK, one last question...how will the duel language system work? Just to be honest...and I have no idea how large the French market for this is but it looks to me like the English one will be much larger.

BTW: I'm from Southern Louisiana so I understand spoken Cajun french ok but I read it like a 4 year old.

When you create a book, you'll just have to say which language it is. When you'll browse or search for a book, you'll see directly which language it is and we could filter out, those that are not in a language that you understand for example...

Hadrien · 12-15-2006, 08:44 AM

Well...
The website is almost ready for beta, should launch next week.
The newspaper/rss won't be open for beta next week, but the rest should be just fine.

I'll post a better video demonstration of the website really soon...

jæd · 12-15-2006, 08:54 AM

Quote:

Originally Posted by Hadrien

Well...
The website is almost ready for beta, should launch next week.
The newspaper/rss won't be open for beta next week, but the rest should be just fine.

That's quick....! Let us all know when we can beta test it...!

alex_d · 12-19-2006, 03:13 AM

Is this a school project or something closer to being commercial? (Sorry to be blunt, but I guess the question is if you're just doing enough to get a grade or a paper or if you really plan to put in the huge amount of time, attention, and love to make it appealling to the masses.)

Also, keep in mind that the Reader can't handle pdf right no matter what the size. (The fonts always come out too faint.) You might consider making available RTF or BBeB format as well. Given that you're setting up a database you should consider giving people a lot of freedom in the file formats and the fonts, font sizes, etc that they download in. Combined with a nice ajax interface (ie, some sort of dynamic preview), this functionality would be very useful and could be the killer app. It seems half the conversations on this part of the forum are about doing this step well. (This will at least draw users to your site, even if you'd rather have them stick around for the social/user content component).

Lastly, if you're looking to go mainstream, remember the hardest part won't be getting it to work but figuring out how to get the servers to handle the tremendous load of distilling pdfs all day.

If you pull the above two things off well, you can rub it in gutenberg.org's nose for years.

Hadrien · 12-19-2006, 06:30 AM

We're trying to do something commercial and not just a school project and we've already spent a huge amount of time (and love, yes) on it.
As for PDF, I've read 10 books in PDF already and the overall experience was nice. We can do pretty much any format, but we're sticking with PDF first, because generating PDF with LaTeX is much more advanced than any other format.
It is true that the toughest part is handling all those PDF requests, but for 90% of the users, I'm pretty sure they'll stick with A4 PDFs for the moment, and we generate these ones only once, they're in the cache afterwards. The beta will give us a lot more details on this problem, that's why we're rushing to release it as soon as possible, and tweak the whole system to work much better.
As of rubbing it in gutenberg.org's nose, it'll take some time for us to get as many books as they have, we can't simply parse the whole site like manybooks or other did, we need something much more complex and divided in parts & chapters. A bot could do it, but I'm not sure that it'll do it quite well. So we won't start with such a huge amount of books, but at the same time, we're not really doing the same thing than what they do. The user content component, and the generated newspapers (tough one, much more PDF generated) are also key components of FeedBooks.

alex_d · 12-19-2006, 06:06 PM

"we're sticking with PDF first, because generating PDF with LaTeX is much more advanced than any other format."

lol, interesting reasoning. I'm still not sure why you have to pick just one format.

"A bot could do it, but I'm not sure that it'll do it quite well."

The best scheme would have bots and humans working together. And the "I'm not sure" sounds like you haven't even sufficiently studied the problem.

Honestly, I'm failing to see the benefit of dynamically generating books from a database if the point isn't to have the freedom to generate them in a thousand different ways.

You're also kind of saying, "we're gonna spend all this energy delivering books dynamically through a database, but what we really care about is user-generated books and rss newspapers. And that is the reason we won't spend more energy on the database." Either focus on the latter and generate the books beforehand (like mobileread or whatnot) or really bring the book database component to its full potential. Quite telling was your response to me of, "well, the way we solve the database load problem is by hoping that 90% of people don't download from the database." A weird compromise between the two is just wasted effort and will look good only to a professor, not the public.

"As for PDF, I've read 10 books in PDF already and the overall experience was nice."
pdf readability depends on the font size... regarding which, it seems, there is zero flexibility and you can't even use the Size button.

Sorry about coming on a little strong, but I find that it's always helpful to tackle the tough questions in the beginning, before it turns out that the potential answers can no longer affect the course of development.

Hadrien · 12-19-2006, 09:38 PM

"lol, interesting reasoning. I'm still not sure why you have to pick just one format."

We're using LaTeX, and with pdflatex we can create great looking PDF, using a software that's been in development for a few years now, that's not the case of BBeB which is far from standard. The whole point of using a database is that we're not stuck in the future with only one format, font size etc...

"The best scheme would have bots and humans working together. And the "I'm not sure" sounds like you haven't even sufficiently studied the problem."

We've studied the problem comparing a few sources like Wikisource and Gutenberg, and while it's a lot faster to parse books this way, the errors are a lot more tougher to find, meaning you have to proof read each and every book. So yes, we're still aiming for a solution using both bots and books submitted by users since it's the best way to deal with this problem, but it's much better to open our database "wikisource-style" than to parse with bots.

And you don't seem to get how the whole thing work, all the books, public domain books or user submitted books are in the database. We have a few generic formatting style for these books: A4, Sony Reader and iLiad. Whenever someone download a book using one of these generic formatting, the book is generated and cached, which mean that the next one won't have to wait the 1 or 2 seconds it takes to generate a book. But if you want to generate a custom pdf, then ok, as long as you're logged in, we'll remember the settings that you're usually generating custom PDF with, and you'll be able to select, the font size, the margin etc... These custom generated books are not stored in cache afterwards, but it's still possible to generate any book that you'd like this way.

As for the size button, on BBeB it's not that great either. There's no hyphenation, meaning that you end up with huge blank spaces between the words. Justified text is nice, but if there's too many blank spaces between the words, it's not really readable, and BBeB can't deal with hyphenation in a nice way. I'd say that the best solution is to generate the right font size for your eyes, if the generic Sony Reader file doesn't suit you.

alex_d · 12-19-2006, 10:18 PM

"Whenever someone download a book using one of these generic formatting, the book is generated and cached, which mean that the next one won't have to wait the 1 or 2 seconds it takes to generate a book. But if you want to generate a custom pdf, then ok, as long as you're logged in, we'll remember the settings that you're usually generating custom PDF with, and you'll be able to select, the font size, the margin etc... These custom generated books are not stored in cache afterwards, but it's still possible to generate any book that you'd like this way."

Ok, I see. That's pretty much all I was arguing for. I probably wasn't reading carefully enough (indeed, i see you did mention it briefly in post 7). I'd still argue, though, that your personal preference for latex notwithstanding, incorporating other formats (even with reduced layout) is a useful thing. You might not like spaces between words, but others just use landscape and have changing preferences for font size with different light levels and hate the page-turn and font-rendering penalty of pdf. I'm just saying that if you build a database you should make the most of it, and I'm glad you're already incorporating some of the possibilties.

About getting content into the system... why do you seem to say it's tougher to correct errors than to just do the whole thing manually? If the interface is nice and responsive (very important), and there's some sort of credit system, I think you could drum up volunteers to correct a lot of books. I think you'll get more books corrected than entered. You'll definately need to organize the community in some sort of concerted effort either way. (Don't just say, if you feel like doing it, go ahead).

Also, typesetting is quite an art, and you might consider offering different pre-made "themes" (fonts, margins, spacings) that users could choose between (and which users could contribute and vote on). This would be a good way to further explore that power of latex that you're talking about.

Lastly, I still say hoping that 90% of people won't generate their own content is just saying that you don't think it's worth generating (and, by extension, worth implementing). I say start thinking about how you could use clustering to distribute the workload and add enough features that 90% of people will _want custom versions of books.

Stingo · 12-19-2006, 11:10 PM

If your site is only using PDF I for one will probably not download from it. PDF is simply not versatile enough on the Sony Reader. Two levels of magnification and no reflowing just does not cut it for my needs. Unfortunately, although it is not my preferred choice, LRF and RTF are the only to reasonable choices on this device at this time. Good luck.

Hadrien · 12-20-2006, 07:07 AM

Quote:

Originally Posted by alex_d

Lastly, I still say hoping that 90% of people won't generate their own content is just saying that you don't think it's worth generating (and, by extension, worth implementing). I say start thinking about how you could use clustering to distribute the workload and add enough features that 90% of people will _want custom versions of books.

We're using FreeBSD and Lighttpd for these reasons: with this OS and lighttpd instead of a really slow Apache web server, it'll be much easier to do load balancing. We're already using different processes for the the part generating books and the rest of the website. As soon as we can make enough money out of the website to buy new servers, we'll add them to the load balancing system, in order to make the whole experience much more smoother, and increase the possibilities on the website.
The bigger our cluster will be, the more we'll add functionalities that require this.

Quote:

Originally Posted by Stingo

If your site is only using PDF I for one will probably not download from it. PDF is simply not versatile enough on the Sony Reader. Two levels of magnification and no reflowing just does not cut it for my needs. Unfortunately, although it is not my preferred choice, LRF and RTF are the only to reasonable choices on this device at this time. Good luck.

RTF can't handle hyperlinks and table of contents, and we're also planning on having tech books for example or books in many different languages. LaTeX + PDF can handle this.
As for LRF, we might use it later on, from what I've seen on the Connect Store, quite a few books look like they got formatted in LaTeX and then transformed into LRF, so I wonder how Sony or the publishers create LRF files. If we find a way to make good looking LRF files (embedded fonts, table of contents, footnotes...) we'll do it. We're using a database separated into chapters/sections etc... for this single reason: this way we can add support for anything in the future.

eyestrain · 12-20-2006, 01:49 PM

It looks and sounds awesome Hadrien, you're doing an amazing job. very impressive development approach as well.

I fervently hope you do find a way of generating good LRFs, maybe some of the others here could help you on that one. PDFs are a little bit too fiddly for the size of the readers screen.

alex_d · 12-20-2006, 05:03 PM

"If we find a way to make good looking LRF files (embedded fonts, table of contents, footnotes...) we'll do it."

That's a self-centered, presumptious attitude (lol, you frenchies). There are clear benefits to rtf/lrf, and you should let the users weigh those against the drawbacks. Incorporating these formats is actually quite easy (in fact due to their simplicity), so that shouldn't be what stops you.

Anyway, I've put in my 2 cents. Can't wait to see what comes out of this.

12-19-2006, 09:38 PM	#25
Hadrien Feedbooks.com Co-Founder Posts: 2,263 Karma: 145123 Join Date: Nov 2006 Location: Paris, France Device: Sony PRS-t-1/350/300/500/505/600/700, Nexus S, iPad	"lol, interesting reasoning. I'm still not sure why you have to pick just one format." We're using LaTeX, and with pdflatex we can create great looking PDF, using a software that's been in development for a few years now, that's not the case of BBeB which is far from standard. The whole point of using a database is that we're not stuck in the future with only one format, font size etc... "The best scheme would have bots and humans working together. And the "I'm not sure" sounds like you haven't even sufficiently studied the problem." We've studied the problem comparing a few sources like Wikisource and Gutenberg, and while it's a lot faster to parse books this way, the errors are a lot more tougher to find, meaning you have to proof read each and every book. So yes, we're still aiming for a solution using both bots and books submitted by users since it's the best way to deal with this problem, but it's much better to open our database "wikisource-style" than to parse with bots. And you don't seem to get how the whole thing work, all the books, public domain books or user submitted books are in the database. We have a few generic formatting style for these books: A4, Sony Reader and iLiad. Whenever someone download a book using one of these generic formatting, the book is generated and cached, which mean that the next one won't have to wait the 1 or 2 seconds it takes to generate a book. But if you want to generate a custom pdf, then ok, as long as you're logged in, we'll remember the settings that you're usually generating custom PDF with, and you'll be able to select, the font size, the margin etc... These custom generated books are not stored in cache afterwards, but it's still possible to generate any book that you'd like this way. As for the size button, on BBeB it's not that great either. There's no hyphenation, meaning that you end up with huge blank spaces between the words. Justified text is nice, but if there's too many blank spaces between the words, it's not really readable, and BBeB can't deal with hyphenation in a nice way. I'd say that the best solution is to generate the right font size for your eyes, if the generic Sony Reader file doesn't suit you. Last edited by Hadrien; 12-19-2006 at 09:41 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Classic G:RSS: Optimized Google Reader (RSS) for the Nook [BETA Testers needed]	Fmstrat	Barnes & Noble NOOK	24	12-28-2010 12:22 PM
G:RSS: Optimized Google Reader (RSS) for the Kindle 3 (and Nook)	Fmstrat	Amazon Kindle	47	12-13-2010 12:20 PM
latex template for sony reader pdf	technicolor	Sony Reader	10	07-25-2008 09:51 AM
Sony's Reader: LaTeX and Linux.	jibanes	Sony Reader	4	02-27-2007 02:41 AM
Can JPluck pull mixed RSS/website	gadgetguru	Reading and Management	1	06-23-2004 02:57 PM

12-05-2006, 04:48 PM	#16
Kosst Amojan Addict Posts: 207 Karma: 19597 Join Date: Apr 2006 Location: Northern Virginia Device: Nook Glowlight Plus	Ok, sounds like a plan. But on the catagories I'm still not feeling you. Why just not have them nested, fiction>science fiction>hard sci-fi. An easy way to make sure it all rolls together is to have users submit their books to a admin who would double check their formatting and labeling. Now how about some forums. I was thinking that user input would drive deciding which books get formated first. And on that subject what formats do you want to offer other then PDF? If any?

12-05-2006, 07:23 PM	#18
Kosst Amojan Addict Posts: 207 Karma: 19597 Join Date: Apr 2006 Location: Northern Virginia Device: Nook Glowlight Plus	OK, one last question...how will the duel language system work? Just to be honest...and I have no idea how large the French market for this is but it looks to me like the English one will be much larger. BTW: I'm from Southern Louisiana so I understand spoken Cajun french ok but I read it like a 4 year old.

12-15-2006, 08:44 AM	#20
Hadrien Feedbooks.com Co-Founder Posts: 2,263 Karma: 145123 Join Date: Nov 2006 Location: Paris, France Device: Sony PRS-t-1/350/300/500/505/600/700, Nexus S, iPad	Well... The website is almost ready for beta, should launch next week. The newspaper/rss won't be open for beta next week, but the rest should be just fine. I'll post a better video demonstration of the website really soon...

12-19-2006, 03:13 AM	#22
alex_d Addict Posts: 303 Karma: 187 Join Date: Dec 2006 Device: Sony Reader	Is this a school project or something closer to being commercial? (Sorry to be blunt, but I guess the question is if you're just doing enough to get a grade or a paper or if you really plan to put in the huge amount of time, attention, and love to make it appealling to the masses.) Also, keep in mind that the Reader can't handle pdf right no matter what the size. (The fonts always come out too faint.) You might consider making available RTF or BBeB format as well. Given that you're setting up a database you should consider giving people a lot of freedom in the file formats and the fonts, font sizes, etc that they download in. Combined with a nice ajax interface (ie, some sort of dynamic preview), this functionality would be very useful and could be the killer app. It seems half the conversations on this part of the forum are about doing this step well. (This will at least draw users to your site, even if you'd rather have them stick around for the social/user content component). Lastly, if you're looking to go mainstream, remember the hardest part won't be getting it to work but figuring out how to get the servers to handle the tremendous load of distilling pdfs all day. If you pull the above two things off well, you can rub it in gutenberg.org's nose for years.

12-19-2006, 06:30 AM	#23
Hadrien Feedbooks.com Co-Founder Posts: 2,263 Karma: 145123 Join Date: Nov 2006 Location: Paris, France Device: Sony PRS-t-1/350/300/500/505/600/700, Nexus S, iPad	We're trying to do something commercial and not just a school project and we've already spent a huge amount of time (and love, yes) on it. As for PDF, I've read 10 books in PDF already and the overall experience was nice. We can do pretty much any format, but we're sticking with PDF first, because generating PDF with LaTeX is much more advanced than any other format. It is true that the toughest part is handling all those PDF requests, but for 90% of the users, I'm pretty sure they'll stick with A4 PDFs for the moment, and we generate these ones only once, they're in the cache afterwards. The beta will give us a lot more details on this problem, that's why we're rushing to release it as soon as possible, and tweak the whole system to work much better. As of rubbing it in gutenberg.org's nose, it'll take some time for us to get as many books as they have, we can't simply parse the whole site like manybooks or other did, we need something much more complex and divided in parts & chapters. A bot could do it, but I'm not sure that it'll do it quite well. So we won't start with such a huge amount of books, but at the same time, we're not really doing the same thing than what they do. The user content component, and the generated newspapers (tough one, much more PDF generated) are also key components of FeedBooks.

12-19-2006, 06:06 PM	#24
alex_d Addict Posts: 303 Karma: 187 Join Date: Dec 2006 Device: Sony Reader	"we're sticking with PDF first, because generating PDF with LaTeX is much more advanced than any other format." lol, interesting reasoning. I'm still not sure why you have to pick just one format. "A bot could do it, but I'm not sure that it'll do it quite well." The best scheme would have bots and humans working together. And the "I'm not sure" sounds like you haven't even sufficiently studied the problem. Honestly, I'm failing to see the benefit of dynamically generating books from a database if the point isn't to have the freedom to generate them in a thousand different ways. You're also kind of saying, "we're gonna spend all this energy delivering books dynamically through a database, but what we really care about is user-generated books and rss newspapers. And that is the reason we won't spend more energy on the database." Either focus on the latter and generate the books beforehand (like mobileread or whatnot) or really bring the book database component to its full potential. Quite telling was your response to me of, "well, the way we solve the database load problem is by hoping that 90% of people don't download from the database." A weird compromise between the two is just wasted effort and will look good only to a professor, not the public. "As for PDF, I've read 10 books in PDF already and the overall experience was nice." pdf readability depends on the font size... regarding which, it seems, there is zero flexibility and you can't even use the Size button. Sorry about coming on a little strong, but I find that it's always helpful to tackle the tough questions in the beginning, before it turns out that the potential answers can no longer affect the course of development.

12-19-2006, 10:18 PM	#26
alex_d Addict Posts: 303 Karma: 187 Join Date: Dec 2006 Device: Sony Reader	"Whenever someone download a book using one of these generic formatting, the book is generated and cached, which mean that the next one won't have to wait the 1 or 2 seconds it takes to generate a book. But if you want to generate a custom pdf, then ok, as long as you're logged in, we'll remember the settings that you're usually generating custom PDF with, and you'll be able to select, the font size, the margin etc... These custom generated books are not stored in cache afterwards, but it's still possible to generate any book that you'd like this way." Ok, I see. That's pretty much all I was arguing for. I probably wasn't reading carefully enough (indeed, i see you did mention it briefly in post 7). I'd still argue, though, that your personal preference for latex notwithstanding, incorporating other formats (even with reduced layout) is a useful thing. You might not like spaces between words, but others just use landscape and have changing preferences for font size with different light levels and hate the page-turn and font-rendering penalty of pdf. I'm just saying that if you build a database you should make the most of it, and I'm glad you're already incorporating some of the possibilties. About getting content into the system... why do you seem to say it's tougher to correct errors than to just do the whole thing manually? If the interface is nice and responsive (very important), and there's some sort of credit system, I think you could drum up volunteers to correct a lot of books. I think you'll get more books corrected than entered. You'll definately need to organize the community in some sort of concerted effort either way. (Don't just say, if you feel like doing it, go ahead). Also, typesetting is quite an art, and you might consider offering different pre-made "themes" (fonts, margins, spacings) that users could choose between (and which users could contribute and vote on). This would be a good way to further explore that power of latex that you're talking about. Lastly, I still say hoping that 90% of people won't generate their own content is just saying that you don't think it's worth generating (and, by extension, worth implementing). I say start thinking about how you could use clustering to distribute the workload and add enough features that 90% of people will _want custom versions of books.

12-19-2006, 11:10 PM	#27
Stingo Fanatic Posts: 559 Karma: 1334691 Join Date: Nov 2006 Location: Miami Device: KH2O, KPW2, KDXG, KPW1, K3, S505	If your site is only using PDF I for one will probably not download from it. PDF is simply not versatile enough on the Sony Reader. Two levels of magnification and no reflowing just does not cut it for my needs. Unfortunately, although it is not my preferred choice, LRF and RTF are the only to reasonable choices on this device at this time. Good luck.

12-20-2006, 01:49 PM	#29
eyestrain Junior Member Posts: 6 Karma: 10 Join Date: Dec 2006 Device: Kindle Keyboard, Kindle Voyage	It looks and sounds awesome Hadrien, you're doing an amazing job. very impressive development approach as well. I fervently hope you do find a way of generating good LRFs, maybe some of the others here could help you on that one. PDFs are a little bit too fiddly for the size of the readers screen.

12-20-2006, 05:03 PM	#30
alex_d Addict Posts: 303 Karma: 187 Join Date: Dec 2006 Device: Sony Reader	"If we find a way to make good looking LRF files (embedded fonts, table of contents, footnotes...) we'll do it." That's a self-centered, presumptious attitude (lol, you frenchies). There are clear benefits to rtf/lrf, and you should let the users weigh those against the drawbacks. Incorporating these formats is actually quite easy (in fact due to their simplicity), so that shouldn't be what stops you. Anyway, I've put in my 2 cents. Can't wait to see what comes out of this.

Advert

Advert