Web Browsing from the Ebw-1150 - BETA Testers needed

[derrell]

You might have something there I wonder if there is something different in the html at that site.

The output that I get from there in the eb1150 window is identical to what I get with other sites but it ends and the socket closes after several 4092 byte chunks followed by an odd sized ending chunk. Then the reader displays the page. The other links do the same thing but after the odd sized chunk I start getting those 0 byte length ones and socket never closes.

Off to look at the html from ebookwise.com.

[nrapallo]

Originally Posted by derrell

You might have something there I wonder if there is something different in the html at that site.

The output that I get from there in the eb1150 window is identical to what I get with other sites but it ends and the socket closes after several 4092 byte chunks followed by an odd sized ending chunk. Then the reader displays the page. The other links do the same thing but after the odd sized chunk I start getting those 0 byte length ones and socket never closes.

Off to look at the html from ebookwise.com.

Derrell,

What port does eb1150 connect to? In proxy1150.cpp line 203, it gets set and line 210, it gets printed out. Is it 127.0.0.1:9090?

And impserve is 127.0.0.1:9090?

Just poking to see if there's life...

[derrell]

eb1150 is connecting to 127.0.0.1:9090

I was looking at the code in proxy1150.cpp and this looks like it should prevent any data that is 0 bytes being sent. Once it finds the end of the string it should close the socket, I think. Maybe some of the data coming out of impserve isn't null terminated.

      if (data.length() > 0) {

	// Parse and dispatch to the proxy methods      
	if (data.find("gethostbyname") != string::npos) {

	  gethostbyname(data);

	} else if (data.find("socket") != string::npos) {

	  socket(data);

	} else if (data.find("connect") != string::npos) {
	  
	  connect(data);

	} else if (data.find("send") != string::npos) {

	  send(data);

	} else if (data.find("recv") != string::npos) {
	
	  recv(data);

	} else if (data.find("close") != string::npos) {

	  close(data);

	} 
	
      }

[nrapallo]

Originally Posted by derrell

eb1150 is connecting to 127.0.0.1:9090

I was looking at the code in proxy1150.cpp and this looks like it should prevent any data that is 0 bytes being sent. Once it finds the end of the string it should close the socket, I think. Maybe some of the data coming out of impserve isn't null terminated.

I think there is something really weird going on with your if statement above and impserve! It may be that eb1150 is lacking something in this situation.

I tried to debug eb1150 (by placing debug printf's after each if segment to print the data); this is the code I used:

      if (data.length() > 0) {

	// Parse and dispatch to the proxy methods      
	if (data.find("gethostbyname") != string::npos) {
cout << "1: " << data << " gethostname\n";
	  gethostbyname(data);

	} else if (data.find("socket") != string::npos) {

cout << "2: " << data << " socket\n";
	  socket(data);

	} else if (data.find("connect") != string::npos) {
	  
cout << "3: " << data << " connect\n";
	  connect(data);

	} else if (data.find("send") != string::npos) {

cout << "4: " << data << " send\n";
	  send(data);

	} else if (data.find("recv") != string::npos) {
	
cout << "5: " << data << " recv\n";
	  recv(data);

	} else if (data.find("close") != string::npos) {

cout << "6: " << data << " close\n";
	  close(data);

	}

      }

I don't understand where all those 0 byte(s) items are being sent from. I tried closing the socket once the 0's showed up, but to no avail. Please note that I numerically ordered the printf's and saw that for (some) transfers the order was 2,3,4,5,6,5 with the last 5 after a close (6) and thus would produce a recv errno = 9.

Now where to look further...

BTW, I think I remember seeing all those 0 byte message when first using Linreb with hyperlinks to the internet, so it may very well be a eb1150 (linux usb driver) issue.

Is there a way to use the ETI (windows) usb driver with wine and bypass eb1150?

[ashkulz]

Originally Posted by nrapallo

I think there is something really weird going on with your if statement above and impserve! It may be that eb1150 is lacking something in this situation.

I tried to debug eb1150 (by placing debug printf's after each if segment to print the data); this is the code I used:

      if (data.length() > 0) {

	// Parse and dispatch to the proxy methods      
	if (data.find("gethostbyname") != string::npos) {
cout << "1: " << data << " gethostname\n";
	  gethostbyname(data);

	} else if (data.find("socket") != string::npos) {

cout << "2: " << data << " socket\n";
	  socket(data);

	} else if (data.find("connect") != string::npos) {
	  
cout << "3: " << data << " connect\n";
	  connect(data);

	} else if (data.find("send") != string::npos) {

cout << "4: " << data << " send\n";
	  send(data);

	} else if (data.find("recv") != string::npos) {
	
cout << "5: " << data << " recv\n";
	  recv(data);

	} else if (data.find("close") != string::npos) {

cout << "6: " << data << " close\n";
	  close(data);

	}

      }

I don't understand where all those 0 byte(s) items are being sent from. I tried closing the socket once the 0's showed up, but to no avail. Please note that I numerically ordered the printf's and saw that for (some) transfers the order was 2,3,4,5,6,5 with the last 5 after a close (6) and thus would produce a recv errno = 9.

Now where to look further...

BTW, I think I remember seeing all those 0 byte message when first using Linreb with hyperlinks to the internet, so it may very well be a eb1150 (linux usb driver) issue.

Is there a way to use the ETI (windows) usb driver with wine and bypass eb1150?

I think that the problem might be very subtle, and require lots of time to debug! Essentially, the eb1150 thinks that there is data when actually there is none coming from the content server [hence the 0 size requests]. We need to check if a Content-Length is set which makes the device expect more data. Also Proxy1150::recv ultimately does a Proxy1150::ack that it got the data -- we need to see whether the same response is sent which is done by the original USB driver.

[nrapallo]

Originally Posted by ashkulz

I think that the problem might be very subtle, and require lots of time to debug! Essentially, the eb1150 thinks that there is data when actually there is none coming from the content server [hence the 0 size requests]. We need to check if a Content-Length is set which makes the device expect more data. Also Proxy1150::recv ultimately does a Proxy1150::ack that it got the data -- we need to see whether the same response is sent which is done by the original USB driver.

Subtle indeed! I tried changing some boundary conditions i.e '<' to '<=' in Proxy::recv() to see if it would help; it didn't. I probed Dev1150::read() but didn't see anything obvious. It may be deeper in the module usb_bulk_read() from libusb....

Need to do more debugging...

[nrapallo]

Originally Posted by derrell

eb1150 is connecting to 127.0.0.1:9090

I was looking at the code in proxy1150.cpp and this looks like it should prevent any data that is 0 bytes being sent. Once it finds the end of the string it should close the socket, I think. Maybe some of the data coming out of impserve isn't null terminated.

Derrell:

If you want to see what HTTP header information impserve is getting, try inserting the following debug statements after line 211 in "impserve/__init__.py":

        #temp debug statements for header and body of request
        print self.headers
        print info

[nrapallo]

Originally Posted by nrapallo

Derrell:

If you want to see what HTTP header information impserve is getting, try inserting the following debug statements after line 211 in "impserve/__init__.py":

        #temp debug statements for header and body of request
        print self.headers
        print info

I post some logs I made when my EBW1150 connected to my netbook (ASUS eee 1000H) running linux (ubuntu 8.04.1 - complete re-install onto a 40 GB HD partition as I didn't like the performance hit when installed on my 8 GB SDHC card).

I started eb1150 and the EBW1150 connected flawlessly with the ETI servers (see detailed log below) but when I tried to use impserve everything was fine until I accessed real world urls (see simple log below)!

I don't know why one way works and the other way it sends 0 bytes at the end (continuously) after accessing a link to the internet. Ashish?

The one thing I do see is that the ETI communications were without the "HTTP://" prefix and whereas the "real world" urls have it. The local proxy for *.ebooksytem.net excepted!

Food for thought...

[derrell]

I did a little more digging on this problem before I left it. I didn't save any of the output but if you change proxy1150.cpp so that it is showing whats coming into and out it. The eternal URLs are being processed but its like ashkulz thought. It looks like for some reason even after all the data is finished there seems to be more when there really isn't. All of the html is being sent right down to the last </html> tag. I didn't really try too hard to figure out what is really happening because transfering my books to the ebw-1150 reader works without a hitch so far with the orignal code. For now that is enough for me.

Below is the code that I changed if I remember correctly its been a few days.

// Pass the received data from the content server to the device
void Proxy1150::recv(string &data)
{
  vector<string> data_vector;
  split(data, data_vector, 5);

  int buffer_size = atoi(data_vector[4].c_str());
  char buffer[4092] = "";

  int ret;
  ret = ::recv(_socket, buffer, buffer_size, 0);
  if (ret < 0) {
    cerr << "Error: Recv failed (errno = " << errno << ")\n";
    nak(data_vector[1], "0");
  } else {
    string buf_str;
    buf_str.assign(buffer, ret);

    cout << "Recv: " << ret << " byte(s)\n";

// added the next line so I could see what was being moved around
    cout << "buffer str " << buf_str << " content\n";

    ack(data_vector[1], buf_str);
  }
}

[nrapallo]

Originally Posted by derrell

The eternal URLs are being processed but its like ashkulz thought. It looks like for some reason even after all the data is finished there seems to be more when there really isn't. All of the html is being sent right down to the last </html> tag.

If all the html is sent, what else could there be??? I wonder if there is cookie data being sent; just speculating since ashkulz has now allowed cookie data to be transferred through per his unreleased changes.

Let's hope this can easily be fixed.

[nrapallo]

Great news!

Learned from ashkulz recently that the null bytes error when using the eb1150 usb driver with impserve in linux is a thing of the past. I just verified this for myself using Ubuntu 8.04.

Accessing the internet from the links in your content/index.html now fully work; as it does for the links within a .imp ebook.

Only thing missing is direct .imp creation support; but you can now download any .imp onto your device from the internet!

[ashkulz]

For those interested in the reason why it failed earlier: the problem was that some servers sent a 'Content-Length' which was more than the actual data, and the ebook went on waiting ... I've fixed that in ebookutils-0.7 from a bug report from nrapallo for Munsey's, and it seems to work well everywhere