We now have a beta section where our regular members (everyone with 20 or more posts) can download pre-made scoops in iSilo format (Plucker will follow).

These scoops are regularly updated (several times a day). The list of scoops is expanding, and currently includes: The Economist, Reuters, NZZ, The Inquirer.

MobileRead.com Team

Alex, this looks really cool! How did you get the Economist scoop to work?! It looks great! Thanks for this service.

I would love to have a few NYTimes sections up here too, if possible. Maybe National, International,Technology, and Week in Review?

Fantastic work!

Thanks for the compliments :) I will add those NYT sections; are your .site files ready or do we have to optimize them further?

I never finished the final mods to the .site files because I couldn't figure out the proper implementation of the code. But for those sections they should work fine. The problem only arises in sections like Movies or Health where there are commonly articles older than 10 days. In the more "newsy" sections, there are never any posts that old, so there is no problem!

The only note that I would add is that my .site file is designed for one large scoop, containing all the sections a user would want. It is based on a local HTML file. So, if you plan to do it section-wise, you will simply have to have multiple html files, that each point to only one section, and a scoop for each of these sections. However, the rest of the .site file can probably stay the same, with the exception of the pointer to the HTML file. I could do these mods for you, if you like, but not until tomorrow...

and how did you make the Economist scoop work?

Cool, Alexander :D Thanks!

Well, I used the same .site file Morpheus posted earlier. I only added images to it.

Then, I did some changes to the sitescooper itself. As recommended by stobs, I changed the default user agent to something like "Internet Explorer" to hide sitescooper. To do so, you must edit /lib/Sitescooper/Main.pm:

Find (around line 977)
$self->{useragent}->agent ("sitescooper/$VERSION ($self->{home_url}) ".
$self->{useragent}->agent);
and change it to
$self->{useragent}->agent ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");

That did the trick here at least.

Greets

I could do these mods for you, if you like, but not until tomorrow...Lemme think about it and the easiest way how we can do that.

FoxNews scoups would be great. These are a nice touch.

Fox News? Don't you mean, White House State News Service?

Alex, any chance you can set up something like this for Mobipocket? I prefer reading news in landscape mode.

As always, thank you for your work on our behalf.

Gyffes, Mobipocket I am afraid won't be possible because it lacks a linux-compatible console tool to generate ebooks, right? I will try and see what I can do about it.

As recommended by stobs, I changed the default user agent to something like "Internet Explorer" to hide sitescooper. That's a great find. I had understood (though don't know where I read it now...) that you could not spoof with sitescooper. There's a couple of other sites that I've been working on that this may help with. Great work Alex.

That's a great find. I had understood (though don't know where I read it now...) that you could not spoof with sitescooper. There's a couple of other sites that I've been working on that this may help with. Great work Alex.
Np :) I think together we can make Sitescooper even better. For what it's worth, I could also turn 'user-agent' into a .site option.

Any chance of adding PIC mobile (http://www.palminfocenter.com/mobile), CS Monitor (text edition at http://www.csmonitor.com/cgi-bin/redirect.pl?textEdition), and/or Wired (http://www.wired.com/wireless)? I'd love to use this to gather up my morning reading, rather than running iSiloX, because I'm lazy like that. ;)
Plus, I don't think I'm the only one that likes to read those sites regularly.

These look great, Alex, thanks!. Only tried a few, but on The Economist scoop when you click on the "More From (blank)" links you get screens and screens of urls. Was expecting more stories, but see only the list of urls (which aren't links).

Ahh yes, only the articles on the main page are currently available. Otherwise, the download becomes too big. I will think of it again. Either I will try to include it (without images perhaps), or I will at least remove the more from links to avoid confusion.

... I'd love to use this to gather up my morning reading, rather than running iSiloX, because I'm lazy like that. ;)
Plus, I don't think I'm the only one that likes to read those sites regularly.
Sure can do that for you :P Though the primary purpose of this here is to have scoops of non-mobile pages available that aren't easily converted with iSiloX.

Great move Alex - really nice scoops. Also, I agree with Tanker Bob that it'd be nice to have FoxNews.
Gyffes - what's your Palm device?
Since everyone's making requests how about The Daily Telegraph (http://www.telegraph.co.uk/)?

We now have a beta section where our regular members (everyone with 20 or more posts) can download pre-made scoops in iSilo format (Plucker will follow).

These scoops are regularly updated (several times a day). The list of scoops is expanding, and currently includes: The Economist, Reuters, NZZ, The Inquirer.

MobileRead.com Team
Alex,

Where can I find the scoops?

Zire, did you receive a mail? Anyways, check your PM.

melvynadam, Tanker Bob:
Foxnews should be included with the next hourly update. Will try to add more of your requests soon (today is request day, isn't it?) :)

Hi,

Alex,I was also wondering about the access.

I just checked my email and had no message for details.

This sounds like a great idea.Do any of the clips include photos?

Peace,

David

David, not sure why you didn't get the mail, but check your PM now. Photos can be and are included when appropriate.

Alex, thanks so much for adding FoxNews. Whilst you're in a giving mood......

Are there any other Israelis on this list who would be interested in Ha'aretz (http://www.haaretzdaily.com/)? It's too difficult for me to get this on iSilo as it's definitely one of those "non-mobile pages available that aren't easily converted with iSiloX". If there's enough interest maybe it could be added.

Hi,

Alex,simply amazing work and I look forward to seeing what you add in the coming days.

I have one small request,would it be possible to consolidate all of the Reuters stuff in one file For those who may just want a single file?

Peace

David

Alex,
Scoops work great! Wonderful job.

Alex, superior work. Thank you for all of your efforts on the scoops and this site.

John

Alex, many thanks - this is great! I can't wait for NYT sections. Is there any chance of making these available via FTP? That would allow users to write a simple script to download the files we want and throw them into the ISilo install directory for "automated" updating of content. It would also be a lot easier to download files while away from a desktop PC . . . or maybe there's a simpler solution to these that I'm missing?
Either way, your work is much appreciated :)

Is there any chance of making these available via FTP? That would allow users to write a simple script to download the files we want and throw them into the ISilo install directory for "automated" updating of content. It would also be a lot easier to download files while away from a desktop PC . . . or maybe there's a simpler solution to these that I'm missing?
Either way, your work is much appreciated :)

Interesting idea,
perhaps "wget" (http://studwww.ugent.be/~bpuype/wget/) can solve that task (I haven't tried).

To fetch the files while online, there were some discussions in the sitescooper ml.

Has anyone experiences with Palm-browser, that directly can download the pre-scooped files and install them (on card or RAM) for iSilo.

-S.

Just use a download manager with scheduling (e.g. GetRight, Download Boost, ReGet Deluxe, Internet Download Manager, InstantGet, DownloadStudio, HiDownload).
If you're interested in a freeware app that'll do this then try out Fresh Download (http://www.freshdevices.com/).

Hope that's of use.

I'd suggested FTP because it not only solves the scheduled download issue, but there are also some good Palm clients (I use VFSFTP) that would make downloading these with a PDA a lot easier than using NetFront or another browser.
As a bonus, if users received individual logins to the FTP server (same as their login here perhaps?) it would be possible for admins to monitor/cap bandwidth usage.
Just my .02

captainao, I must think about it.

...Only tried a few, but on The Economist scoop when you click on the "More From (blank)" links you get screens and screens of urls...
I cleaned it up, Gary (hope it didn't break the script ;)).

Alexander this is a pretty sweet thing, it seems to work very well. What do you plan going forward with this concept??? Subscription based or ????

Either way its a great idea

Alexander, yes, this is a good idea indeed. I've been searching the internet to find references of a successful Palm and Nokia 6610 GPRS set up using the IR connection. If and when I can make it work, then with this collection, it will be like watching news on the hour every hour.

Alexander this is a pretty sweet thing, it seems to work very well. What do you plan going forward with this concept??? Subscription based or ????

Either way its a great idea
Thanks for the compliments, cactusjack. The service will be free, to everyone who is participating in the forum. So far, everyone with +20 posts or who has been contributing in other ways received a private login. This is not meant to be 'elitist', but to serve two purposes. A) To keep it private (don't want to stumble over copyright issues by making this public), B) To keep traffic to a minimum. As far how we will go forward with this... I am not exactly sure. I will definitely add more scoops to it. We will see how the other parts of the new sites (on which I am still working) will go.

Greets
Alex

Alex, thanks for the update - it all sounds very exciting. Can you tell us which scoops are 'Coming Soon To A PC Near You'?
Also, what new sites are you working on? All this time you spoke of new sites I assumed you meant the move to mobileread from turcic. Is there more PDA related stuff to come?

melvyn,

as you noticed correctly turcic.com became mobileread.com. But beside the name change, I am planning to do a complete site makeover. New features, new layout, etc. I am very excited about these changes and have been working on them for the past few months. I think soon I will be able to make them public.

About the scoops... I am going to add those scoops that are of interest to most of us. I haven't planned any specific ones yet, but NYT is definitely going to be one of them.

Greets

Okay, so I got my Palm 515 working with my 6610 GPRS. Which means, I second Captainao's suggestion for the FTP option. Wouldn't it be nice, Sunday morning, go to your favorite Coffee Shop, fire-up the Palm/6610 contraption, FTP to your site. While the coffee is still brewing, I can already be enjoying the latest hot-off-the-press happenings. Sunday comic strips? Probably asking too much. But, hey, it's great stuff you're doing here. Really appreciate it.

just curious since I don't have a wifi/gprs-capable pda yet... isn't it possible to fetch the scoops also from WWW instead of FTP?

Well, the problem I encountered is that the site requires user and password authentication. Since I'm using a freeware web browser, I don't know, if at all, a way to send the user/pw tokens. I enter the url and it replies with an error message authentication required. The user/pw panel doesn't come up to allow me to enter name/pw. I don't quite understand how to use Palm's Web Clipping, which may be the way to do it.

BTW, I'm using a Nokia 6610 GPRS phone, connected to the M515 via IR. I have WAP/GPRS account with StarHub in Singapore.

Can you try it like this, perhaps it works:

http://username:password@sitename.com

Nope. Doesn't work. The browser I used, Sedaka and Matrix, will throw me to Google when I enter the URL as you suggested. Let browse around, should be a way to pass username and password, or perhaps, I need to install another module.

So what's new with the scoops? Are you planning to add more?

Yes - I was just too preoccupied bringing out the new version of mobileread.com. Now I should have more time again to play with the scoops.

Alex

Okay, this thread is getting stale, time for another request!
Any chance of getting any of the www.cnet.com content scooped up and served daily?

Sorry about this. I just visited our scoops site to see if everything is still working ;)

The time that was required to rebuild mobileread.com, and also some unexpected work at my job are the two reasons why I couldn't work more on the "scoop project". Also, I've found a lot of new interest in RSS feeds and programmed a couple of custom feeds which I plan to make available here soon (like a feed for PalmGear's latest software updates). I promise I will spend more time on the scoops again.

Alex,

Are you crazy?! Don't apologise for not having done something for free for other people - especially when the reason was that you were so busy doing something else for us! I really appreciate all of the time you've put into the mobileread site and also the scoops. There's no hurry to expand them and they're great in their current form. When you eventually have time there are many requests in this thread and it'll be fun to see some implemented.

Doh! Where'd the scoops go?

Melvyn, I was contacted by someone to remove the premade Economist scoop. So I figured it best to remove the entire website since I don't want anyone to get upset.

I am currently very busy at work, but also working on something "new"... don't want to say too much yet because I don't know when it will be ready, but I am confident you will like it! ;)

Alex

Well now I'm thoroughly intrigued! If you can't tell me more than that about the 'what' can you give a better idea about the 'when'?

/me all curious to hear more about it... ;)

If I'd meant "me" I wouldn't have posted the message on the public forum! Of course I meant "us".

Alex, will the new "thing" incorporate any ideas/requests from this discussion thread?

Ok sorry to make such a mystery about it. I just want to avoid people asking me later when it is ready, since I cannot make any promises right now (work keeps me busy).

Basically I am working on a database system, where users can generate their own iSiloX (other formats such as Sunrise should follow) files. Take our Web Channel Collection for example; in future, users can make their own favorite .ixl files and assemble the channels to their liking. It sounds all pretty basic, but it comes with some more extras which should make it really cool for everyone ;)

In addition, and this is probably more interesting for you, I finished an automated system where one can easily convert RSS feeds into Mobile pages, plus use regular expressions to rewrite urls on the fly (to point article urls to printer-friendly urls). See The Register thread (http://www.mobileread.com/forums/showthread.php?t=1957) for an example.

Sounds great. I'm really looking forward to trying this out (as I'm sure are others) but I'll refrain from asking when it's going to be ready.
Thanks again for your hard work.

Any updates for us? (Whilst we wait patiently ;-))

You got me here! Sorry, in future I shouldn't make deadline promises I cannot keep for sure. Unfortunately, due to some events in real life (plus a very relaxing christmas holiday) I haven't been able to work more on this project. Which means I must postpone it to a future date, somewhere in February or latest March I hope.

Again, I am truely sorry for keeping you waiting ;(

I'm not bothered about waiting it's just nice to have details on the development cycle. I'm sure you have regulars who can help but I'd be glad to assist in beta development if needed. I work in QA of software so there's always hope I could bring something useful to the project!

If I can just figure out this darn sitescooper program I'll be joining you guys soon. I want my morning paper on my ebookwise! :(

CINCNORAD, which one is your morning paper?

Washington Post or NY Times. How come?

Washington Post or NY Times. How come?
cincnorad, I believe you can find mobile-friendly links for both sites somewhere here on the board. Give the search feature a try.

Yeah. Been searching the sitescooper threads and any tutorials I can get my hands on...

Have you tried these links:
Washington Post (http://www.mobileread.com/forums/showthread.php?t=2189) (also scroll down that thread)
NY Times (http://www.mobileread.com/forums/showthread.php?t=3269)

Sitescooper of course allows you to scrape any parts you want from the original online version of each newspaper. But it is much more difficult to handle, especially if concepts like RegEx are new to you.

Okay... finally figured sitescooper out. And thanks for the NYTIMES link Alexander -- works great!

Now just playing with batch files to automate this whole info gathering routine every morning on my ebookwise. I know someone on this board had figured out how to get the batch file to also load the resulting html files into the eBookwise Librarian program for conversion/loading onto the device. I gotta figure that one out so I can just hook it up, click, and come back after breakfast to have it all done.

Found another good link. Apologies if it's already been posted:
http://www.wired.com/news_drop/palmpilot/