Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > PocketBook

Notices

Reply
 
Thread Tools Search this Thread
Old 06-14-2011, 05:43 PM   #1
niroma
Member
niroma began at the beginning.
 
Posts: 13
Karma: 12
Join Date: Jun 2011
Location: Karlstad, Sweden
Device: PocketBook 902 Pro
[solved] wget pdf newspaper from site?

Hi all,

I'm new on this forum and posting this since I couldn't find any postings covering this problem.

I recently bought a PB902Pro and the intended use is to download and read a daily newspaper on it. Problem is that the poor webbrowser (midori), used for the downloading, probably will make my wife wanna go back to the real paper format in a near future.

The newspaper is in PDF format and I have found a way to download it using wget on my Ubuntu-PC. Encourraged by this I searched this forum and found the very useful rsh script by review. Sadly enough the wget in the PB902 doesn't seem to handle cookies and thats where I now got stuck.

The site (http://www1.nwt.se/script/cgiip.exe/...nwt/login.html) uses cookies to identify the user after logging in. On the Ubuntu machine I can use the option --load-cookies to load a cookie file containing my username and password, but this doesen't seem to be possible on the PB902.

1. Is there a way to replace the built in wget?
2. Can I use some other available bash command(s) to solve this?
3. Is there a way to make midori store the login information?
4. Is it possible to somehow send keypresses to midori or control it by cli/script?

My aim is to create a bash-script for the PB902 that can logon to the site, find the latest issue of the newspaper pdf and finally download it to the memory card (/mnt/ext2).

Any hints are highly appreciated!

Last edited by niroma; 06-17-2011 at 01:00 PM. Reason: Problem solved
niroma is offline   Reply With Quote
Old 06-14-2011, 10:50 PM   #2
rkomar
Wizard
rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.
 
Posts: 2,977
Karma: 18343081
Join Date: Oct 2010
Location: Sudbury, ON, Canada
Device: PRS-505, PB 902, PRS-T1, PB 623, PB 840, PB 633
Quite a while back, LoneTech put up a self-made SDK which was based on a Debian arm-based distribution running in a qemu emulator. The executables in that Debian system are binary-compatible with the PB Pro series systems. So, you can get a copy of his SDK, copy the wget executable from it onto your PB device along with the cookie files and some shell scripts to run program, and you should be set to go.

I copied the 'find' utility to my PB device that way, and it works like a charm for me.

If this sounds beyond your comfort level, PM me and we'll figure out how to get a copy of wget to you. Getting the whole system yourself is handy if you want to later add other utilities, though.
rkomar is offline   Reply With Quote
Advert
Old 06-15-2011, 08:16 AM   #3
jekvb
Connoisseur
jekvb has learned how to buy an e-book online
 
Posts: 52
Karma: 90
Join Date: May 2011
Device: PocketBook Inkpad 3 Pro; Android: PocketBook & ReadEra
Why not use Calibre for this action?
Calibre fetches the newspaper.
The newspaper waits in Calibre, until a user wants to pull the paper to his (or her) e-reader.
It's a pity there is no cron function in PocketBook; in that case you could fire up a script, which opens a connection and does the newpaper pull automagically.
jekvb is offline   Reply With Quote
Old 06-15-2011, 09:41 AM   #4
J.C.
Addict
J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.J.C. could sell banana peel slippers to a Deveel.
 
Posts: 224
Karma: 3184
Join Date: Feb 2011
Device: ONYX BOOX Note Air3 C
Quote:
Originally Posted by jekvb View Post
Why not use Calibre for this action?
Calibre fetches the newspaper.
The newspaper waits in Calibre, until a user wants to pull the paper to his (or her) e-reader.
It's a pity there is no cron function in PocketBook; in that case you could fire up a script, which opens a connection and does the newpaper pull automagically.
One thing that stands out is a wireless mode of use, and it'll be initiated from the reader.
J.C. is offline   Reply With Quote
Old 06-15-2011, 11:16 AM   #5
jekvb
Connoisseur
jekvb has learned how to buy an e-book online
 
Posts: 52
Karma: 90
Join Date: May 2011
Device: PocketBook Inkpad 3 Pro; Android: PocketBook & ReadEra
Quote:
Originally Posted by J.C. View Post
One thing that stands out is a wireless mode of use, and it'll be initiated from the reader.
You could start a "script.app" manually, and do this anytime as you please so....
jekvb is offline   Reply With Quote
Advert
Old 06-15-2011, 03:59 PM   #6
niroma
Member
niroma began at the beginning.
 
Posts: 13
Karma: 12
Join Date: Jun 2011
Location: Karlstad, Sweden
Device: PocketBook 902 Pro
Thank you all for your fast replies!

rkomar's suggestion sounds like what I was looking for. I have downloaded the fsimage.qcow2 from http://donkey.vernier.se/~yann/pocketbook/armeabi/ where I guess the new wget command should be found, but how can I access it?
Can I mount the fsimage somehow?
I'm not familiar with qemu (yet).
niroma is offline   Reply With Quote
Old 06-15-2011, 05:05 PM   #7
rkomar
Wizard
rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.
 
Posts: 2,977
Karma: 18343081
Join Date: Oct 2010
Location: Sudbury, ON, Canada
Device: PRS-505, PB 902, PRS-T1, PB 623, PB 840, PB 633
You need to install qemu on your system, and download the initrd-img-2.6..., vmlinuz-2.6... and README files from LoneTech, as well. The exact instructions on how to run qemu are in the README file. Once you get qemu running and log in, you can 'scp' the file to your host system:

Code:
scp /usr/bin/wget 10.0.2.2:/tmp/
assuming that you have a sshd daemon running on the host.

'qemu-img' can be used to convert the qcow2 disk image to another format, but I personally haven't tried that route.
rkomar is offline   Reply With Quote
Old 06-15-2011, 05:28 PM   #8
niroma
Member
niroma began at the beginning.
 
Posts: 13
Karma: 12
Join Date: Jun 2011
Location: Karlstad, Sweden
Device: PocketBook 902 Pro
After a lot of googling I did succeed in getting wget out of the fsimage.qcow2 file by installing qemu and nbd-client + following this guide:
http://blog.loftninjas.org/2008/10/2...u-disk-images/
I had to leave the command:
#sudo qemu-nbd fsimage.qcow2
running in one terminal window and accessing the image from another window after issuing:
#nbd-client localhost 1024 /dev/nbd0
and then mounting with:
#mount /dev/nbd0p1 /mnt

But how can I make the new wget command overlay the older one? Where do I put it on the PB902?
niroma is offline   Reply With Quote
Old 06-15-2011, 06:02 PM   #9
niroma
Member
niroma began at the beginning.
 
Posts: 13
Karma: 12
Join Date: Jun 2011
Location: Karlstad, Sweden
Device: PocketBook 902 Pro
Thanks for your help rkomar!

I found out that I (of course!!!) could use the new wget by simply supply its path on the command line. Kind of stupid of me not to know/remember that.

Anyway, now I got a working wget and I can move on to writing the "get my newspaper"-script.
niroma is offline   Reply With Quote
Old 06-15-2011, 06:03 PM   #10
rkomar
Wizard
rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.
 
Posts: 2,977
Karma: 18343081
Join Date: Oct 2010
Location: Sudbury, ON, Canada
Device: PRS-505, PB 902, PRS-T1, PB 623, PB 840, PB 633
Quote:
Originally Posted by niroma View Post
But how can I make the new wget command overlay the older one? Where do I put it on the PB902?
If you're just going to run it via scripts, you can put it pretty well anywhere you have write access. Something like
Code:
/mnt/ext1/system/bin
would make sense. If you put the scripts in
Code:
/mnt/ext1/applications
, then you can start them directly from the Applications tab (as long as they don't need any arguments). Just make sure to use the full path to your version of wget in your scripts.

Or are you trying to replace the version of wget used by an internal application on the device?
rkomar is offline   Reply With Quote
Old 06-15-2011, 06:32 PM   #11
niroma
Member
niroma began at the beginning.
 
Posts: 13
Karma: 12
Join Date: Jun 2011
Location: Karlstad, Sweden
Device: PocketBook 902 Pro
At first I thought that I had to replace the built in wget, but as you said, it works to supply the path to the new one.

I have adopted both your suggested locations.
niroma is offline   Reply With Quote
Old 06-16-2011, 04:06 PM   #12
niroma
Member
niroma began at the beginning.
 
Posts: 13
Karma: 12
Join Date: Jun 2011
Location: Karlstad, Sweden
Device: PocketBook 902 Pro
Script working from rsh but not from applications!?

Hi,

I have now made a script that works without trouble when executed from the remote shell on the PB902 mentioned above.
Now the problem is that when the script is launched from my "go_get_nwt.app" in the /mnt/ext1/applications folder from the PB902 (not remote), then the hourglass is shown for a couple of seconds and after that the script ends, without downloading anything. Why???

This is the code for go_get_nwt.app
Code:
#!/bin/sh
exec /mnt/ext2/nwtscript/get_nwt.app
and this is for the get_nwt.app
Code:
#!/bin/sh

#Get page containing the list of available newspapers
/mnt/ext1/system/bin/wget --load-cookies /mnt/ext2/nwtscript/cookies.txt http://www1.nwt.se/script/cgiip.exe/WService=nwt/etidning.html -P /var/tmp &> /dev/null

#Extract a list containing only the relevant rows
cat /var/tmp/etidning.html | grep 'http://np.netpublicator.com/netpublication' > /var/tmp/lista.txt

#Extract the newest newspapers id number
tidnnr=$(awk -F'=' 'NR==1 {print substr($2, 45, 9) }' /var/tmp/lista.txt)

#Extract its filename
filnamn=$(awk -F'=' 'NR==1 {print substr($4, 2, length($4)-3)".pdf" }' /var/tmp/lista.txt)

#Download the actual newspaper using built in wget.
wget http://np.netpublicator.com/np/$tidnnr/$filnamn -P /mnt/ext2 &> /dev/null

#Clean up 
rm /var/tmp/lista.txt
rm /var/tmp/etidning.html
I have tested the script row by row using rsh to the PB902 and then it works flawlessly.
niroma is offline   Reply With Quote
Old 06-16-2011, 04:26 PM   #13
rkomar
Wizard
rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.
 
Posts: 2,977
Karma: 18343081
Join Date: Oct 2010
Location: Sudbury, ON, Canada
Device: PRS-505, PB 902, PRS-T1, PB 623, PB 840, PB 633
Maybe you're having the same problem that yuryfdr had with his midi music player: the device just goes to sleep if there aren't any UI events in some short amount of time. The way around this was to bracket his script with the lines:

Code:
echo test > /sys/power/wake_lock
<script code goes here>
echo test > /sys/power/wake_unlock
I suspect that it would be better to set up an exit "trap" for the last line to ensure that it always gets called, no matter how the script ends:

Code:
trap 'echo test > /sys/power/wake_unlock' EXIT
Otherwise, battery usage will go up if the device can't sleep during inactivity.

Last edited by rkomar; 06-16-2011 at 04:30 PM.
rkomar is offline   Reply With Quote
Old 06-16-2011, 06:41 PM   #14
niroma
Member
niroma began at the beginning.
 
Posts: 13
Karma: 12
Join Date: Jun 2011
Location: Karlstad, Sweden
Device: PocketBook 902 Pro
That sounded like a good idea. Added the first echo line and the trap line to the beginning of my script but it didn't make any difference. Strange.

I made an debug version of the script that writes text after each commad to a log file on /mnt/ext2. Every command seems to be run according to the log file but still nothing downloaded. The log file gets a text for each command.

Also tried to let the script copy a downloaded version of etidning.html from /mnt/ext2 to /var/tmp if the reason was something wrong with the new wget but got the same result.

In the debug version I added sleep 5 to the beginning and that command seems to be run as well as the echo's, but not the wget.

Might it be related to access rights or the user that calls the script?
niroma is offline   Reply With Quote
Old 06-16-2011, 08:44 PM   #15
rkomar
Wizard
rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.rkomar ought to be getting tired of karma fortunes by now.
 
Posts: 2,977
Karma: 18343081
Join Date: Oct 2010
Location: Sudbury, ON, Canada
Device: PRS-505, PB 902, PRS-T1, PB 623, PB 840, PB 633
Quote:
Originally Posted by niroma View Post
In the debug version I added sleep 5 to the beginning and that command seems to be run as well as the echo's, but not the wget.

Might it be related to access rights or the user that calls the script?
It might be a permissions problem. Try to delete all previously generated files (including the log file) before running the script. If you were a different user when logged in via rsh, it could keep the normal 'reader' user from overwriting existing files.

Also, try using your freshly installed version of wget to download the newspaper if it works in getting the HTML file in the first line.

Finally, I'm not familiar with '&>' in sh scripts. Are you sure you used that when logged in via rsh?
rkomar is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Grabbing pages with wget and using ebook-convert TheLazy1 Conversion 2 05-26-2011 10:40 AM
How to use wget to download an online HTML book amoroso Lounge 11 04-25-2011 05:10 AM
Daily downloads of pdf newspaper depers.nl? dirkdevries Reading and Management 1 11-16-2010 02:39 PM
html tree via wget -> epub (or other format) maynard Workshop 4 05-13-2009 06:05 PM
iLiad wget doesn't work with application/x-www-form-urlencoded queries ajnorth iRex Developer's Corner 5 12-24-2007 05:56 AM


All times are GMT -4. The time now is 10:14 PM.


MobileRead.com is a privately owned, operated and funded community.