Virtual frambuffer switching for Kobo

[fastrobot]

Originally Posted by davidfor

I have no idea where it is. The options are in libnickel and I assume it interprets them. Whether the telnet implementation is part of nickel or it simply enables something somewhere else, I don't know and have only a passing interest in finding out. When I'm curious about something, it works.

No problem. As I said, I already compiled a telnet host replacement.

What inconsistency? As I keep saying, and no one seems to believe me, the firmware is identical between the devices.

I didn't mention the inconsistency to you, rather it was to another poster; but I'll answer you anyway.
I know that the updates, which are partial replacements -- are in fact identical; because I know I use the same one for both the AuraHD and the Glo. It's not a matter of disbelief in you personally, it's rather I'm baffled as to why my kernel driver works on one device perfectly but is causing the other device to lock up semi-randomly. If they were truly identical, that shouldn't happen... but it does. There may be other explanations for why it's locking up besides differences in the Glo and AuraHD -- But I just can't think of any that make sense off the top of my head.

And yes the kernel is different and that should account for the rest of the hardware differences but is probably built from a common source.

Hmm... I didn't know that -- and hadn't thought about that, for I only compiled my driver against the source code in KoboLabs/Kobo-Reader/hw/imx507-aurahd. The reason is that I downloaded the git repository a month ago -- and there wan't a directory in my clone of the github labeled KoboGlo. So -- since the updates are identical, I assumed the kernel had to be identical.

I just went back to browse git-hub to point out which directory I compiled from, and to point out that there was only AuraHD and not a Glo directory -- but to my shock, I now notice that a Kobo Glo directory was added 20 days ago called "glohd".

sigh.

https://github.com/kobolabs/Kobo-Reader/tree/master/hw

So -- there's an another example of an inconsistency ... which if that kernel proves to be the Glo's kernel (and maybe it's not); it was a belated posting to KoboLabs which caused me to waste my time .... (and yours in dealing with me...). It would have been nice if I knew for sure which kernel my Glo ran the first time I downloaded the repository.

The thing is that what most of us see when using the device is identical running taking the hardware differences into account. And everything running behind the scenes is the same.

And specifically, the developer options and telnet work on all the Kobo devices I have.

I've never tried the depeloper option; so that may well be true. I'm not contradicting you, just stating my experience with trying to follow other instructions found in these forums.

The same goes for all other options that aren't hardware dependent (no light, no function and only the Aura supports mult-touch).

I read through the driver for the zforce-ir-touch, and basically -- that's just a software driver issue. The hardware reports enough information to do multi-touch in the Aura-HD for sure, and the Glo could do it as well -- though I'm not sure the stock driver for multitouch will work on the Glo.

For the rest of your question, sorry I can't help. I have little experience and knowledge in this area, and to be honest, very little interest.

NO problem. I found a copy of gdbserver binary on my rasberry PI, which is also an ARM processor -- and it just so happens that the server runs on the Kobo AuraHD perfectly. So, I'm able to do remote debugging now, and will be able to find the cause of the bug soon.

Thanks for your time.

[davidfor]

Originally Posted by fastrobot

I didn't mention the inconsistency to you, rather it was to another poster; but I'll answer you anyway.
I know that the updates, which are partial replacements

Why do you think they are partial replacements? They don't include what is on the recovery partition. That has much the same files excluding nickel and related files. It also has a set of the files in the upgrade package as at device release. The factory reset simply reboots, formats the root and user partition and installs it's copy of the firmware files in exactly the same way that the upgrade does. And there has been at least one person who replaces the firmware files on the recovery partition with those from an upgrade package and the factory reset worked perfectly to give them that version.

-- are in fact identical; because I know I use the same one for both the AuraHD and the Glo. It's not a matter of disbelief in you personally, it's rather I'm baffled as to why my kernel driver works on one device perfectly but is causing the other device to lock up semi-randomly. If they were truly identical, that shouldn't happen... but it does. There may be other explanations for why it's locking up besides differences in the Glo and AuraHD -- But I just can't think of any that make sense off the top of my head.

I'm pretty sure the Glo and Aura HD use different kernels. They share the same update package, but there are two uImage files in the package. One has an PCB identifier for the Aura HD as part of the name. They are different sizes, so I assume there is a difference in the code.

Hmm... I didn't know that -- and hadn't thought about that, for I only compiled my driver against the source code in KoboLabs/Kobo-Reader/hw/imx507-aurahd. The reason is that I downloaded the git repository a month ago -- and there wan't a directory in my clone of the github labeled KoboGlo. So -- since the updates are identical, I assumed the kernel had to be identical.

I just went back to browse git-hub to point out which directory I compiled from, and to point out that there was only AuraHD and not a Glo directory -- but to my shock, I now notice that a Kobo Glo directory was added 20 days ago called "glohd".

sigh.

https://github.com/kobolabs/Kobo-Reader/tree/master/hw

So -- there's an another example of an inconsistency ... which if that kernel proves to be the Glo's kernel (and maybe it's not); it was a belated posting to KoboLabs which caused me to waste my time .... (and yours in dealing with me...). It would have been nice if I knew for sure which kernel my Glo ran the first time I downloaded the repository.

Huh? The Glo code has been there for ages. The new one is for the Glo HD. The problem is that the labelling has changed. If you look inside "imx507", the uboot file is described as "kernel and bootloader for Kobo Glo, Mini, and newer Touch". That suggests to me that this is the code used for those three devices. They were released at the same time and probably share more hardware than the other devices. The newer devices have their own package. I think the "imx508" is for the first two versions of the Touch, and the others are for the three versions of the WiFi and the original Kobo.

Looking at these, it does make me question my statement about shared code trees. Hopefully they are snapshots taken for each of the devices from the shared code. Someone would have to compare the contents to see.

[frostschutz]

Never knew about the debug service option, but if it's provided by nickel it would be no good for me since sometimes I like to kill nickel ^_^; although I doubt they'd actually duplicate this functionality when busybox already packs telnet and ftp servers.

It should be visible in pstree? With most telnet mods it would look like this:

|-inetd---busybox---sh---pstree

Thanks for your fbswitch, it inspired me to finally try to talk to the framebuffer directly instead of going through pickel showpic.

I actually use softfloat for the Kobo. I wasn't sure if all Kobo's had hardware floating point support and even though one of my programs uses floating point (pngshow via lodepng) I did not notice a difference in performance.

[embryo]

Originally Posted by fastrobot

I had to give my kids the kobo Glo's I had to experiment with to do their book reading with this week, and that's with the system still locking up every so often if they don't press a key. eg: I ran out of time to find and fix the bug.

Did you already read about the "preventfreeze" hack that the KSM is using for the freezes of Glo?
Its difficult to find you the exact post (using my phone), but it was from the first things that tsering had to do for it to work on the Glo.

[NiLuJe]

IIRC, the switch from softfp to hardf depends on a major FW release (2.0? 3.0?, can't remember, I was only looking at it from afar at the time ).

[fastrobot]

Originally Posted by davidfor

Why do you think they are partial replacements?

Because I have edited some files on the working image; and then an update happens... and some of my files were replaced, some were not: Simple Conclusion, not all files in the working image are being replaced by updated ones, hence the update is a partial system update.

They don't include what is on the recovery partition. That has much the same files excluding nickel and related files.

And that includes the kernel and kernel drivers, such as are used for the ir touch?
Which gives one more possible reason why the KoboGlo's e-paper driver reports correct orientation, but the Kobo AuraHD reports a flipped orientation (180 degrees out from reality). For if the CPU is the same, the build in e-paper hardware must be the same -- and only the kernel driver could be causing the problem. But if the hardware is different they could have the same driver, and still have different operation.

When I get my new Glo, I'll have to check the exact CPU in /proc/cpuinfo -- and download the /proc/config.gz from both the Glo and the AuraHD, so I can run diff on them and see what is reported as different...

It also has a set of the files in the upgrade package as at device release. The factory reset simply reboots, formats the root and user partition and installs it's copy of the firmware files in exactly the same way that the upgrade does.

Wait. Do you mean that there is an upgrade package on the backup/resert partition, which installs those files rather than copying the backup partition in its entirety?
I would have expected it to just clone the backup partition using dd.
For if it merely installs a package, then it might not be replacing the kernel on the working root image -- and if the working root kernel got corrupted, even a restore might not fix the reader.

And there has been at least one person who replaces the firmware files on the recovery partition with those from an upgrade package and the factory reset worked perfectly to give them that version.

I was planning on doing the same thing, except installing a hacked version of the upgrade so that a factory reset brings it back to a working version with my extensions already built in; and probably some modifications so that when a new upgrade is detected, that it checks them to see if they can be patched to include my modifications to /etc/init.d/rcS

I'm pretty sure the Glo and Aura HD use different kernels. They share the same update package, but there are two uImage files in the package. One has an PCB identifier for the Aura HD as part of the name. They are different sizes, so I assume there is a difference in the code.

The bugs I'm seeing ought not happen if the hardware for both machines is an imx507, AND the kernel is identical. So -- there definitely is something wrong somewhere. I'd say the odds are excellent that you are correct, ... eg: different kernel code somewhere.

Huh? The Glo code has been there for ages.

I see. The repository for hardware -- is as a whole -- biased-labeled glohd kernels.
Even though it has the kernels for all the readers up until now, and not just glohd.

Who knows, next year they might add a kindle kernel to the repository and rename the whole directory "kindle kernels." ! (Wouldn't that be helpful?!) Or maybe someone will write a python script to change the directory names once a week to alternate possibilities....

But seriously:
Inside, under imx507 and not labeled GLO as a directory -- which is why I didn't notice it before; is a directory containing a second kernel marked kobo-glo. It's just not labeled as such on the directory which contains it.

A bit of labeling inconsistency which makes doing GPL recompilation work harder.
If it has the Glo and Touch kernel, the directory ought to be lableled imx507-glo+touch+foobaz (whatever other devices use this kernel).

So -- I linked my pseudo zforce-ir-touch driver against the AuraHD kernel, and it works perfectly in the AuraHD -- but occasionally locks up in the Glo. Go figure... and the kernel didn't even complain that it was a mismatched module...

The new one is for the Glo HD. The problem is that the labelling has changed.

Yup. And quite inconsistently... for the whole hw repository is now labeled with 'hd'...

That suggests to me that this is the code used for those three devices.

Assuming the labeling isn't wrong (and I will hope it isn't). But yes.... it does suggest that.
A diff will prove the point, later...

They were released at the same time and probably share more hardware than the other devices. The newer devices have their own package. I think the "imx508" is for the first two versions of the Touch, and the others are for the three versions of the WiFi and the original Kobo.

Looking at these, it does make me question my statement about shared code trees. Hopefully they are snapshots taken for each of the devices from the shared code. Someone would have to compare the contents to see.

I will... but only after I get another Glo to work with in a few days...
Thanks for the pointers.

[frostschutz]

Originally Posted by fastrobot

the Kobo AuraHD reports a flipped orientation (180 degrees out from reality)

Oh. I thought it was (setting) the orientation that flipped it; does it actually have a different meaning when (querying) it? If so, my random screensaver mod (current version using pngshow) might rotate images 180° on the Kobo Glo or something. I need some testers... or more Kobo devices to test with (but I can't afford buying one of each so…).

Do you mean that there is an upgrade package on the backup/resert partition, which installs those files rather than copying the backup partition in its entirety?

The recovery partition is more or less a copy of the rootfs partition. It holds the original firmware and files; it has some extra files too, namely the script that actually performs the recovery.

It's true there are some files missing from the KoboRoot.tgz firmware upgrades, for example the device-specific bootup screen progress bar images? Not sure why they're not including those as well, it only saves 1MB or so. Not sure what else is missing either, I should check sometime.

But apart from such small details it's a full firmware image, so you can go any version -> any other version, you don't have to do incremental updates like 3.12->3.13->3.14->3.15. My old reader (iriver) had such an incremental system, it made for small update files, but it also made for bricks when people manually downloaded and used the wrong ones.

I would have expected it to just clone the backup partition using dd.

dd is only used to install bootloader + kernel into the area before the first partition.

If possible I'd not mess with the recovery partition. It's your insurance against bricks, unless you don't mind opening the device up to handle the SD card directly?

[tshering]

Originally Posted by frostschutz

Oh. I thought it was (setting) the orientation that flipped it; does it actually have a different meaning when (querying) it? If so, my random screensaver mod (current version using pngshow) might rotate images 180° on the Kobo Glo or something. I need some testers... or more Kobo devices to test with (but I can't afford buying one of each so…).

I can confirm that the image is rotated 180 on the Touch.

[fastrobot]

If possible I'd not mess with the recovery partition. It's your insurance against bricks, unless you don't mind opening the device up to handle the SD card directly?

I already bricked it once.

Turning the debugging information on for udev made it so my AuraHD didn't want to boot; and for some reason, didn't want to go into recovery mode either. It shouldn't have affected recovery mode, but I ended up pulling the screen face off, to get to the four corner screws, so I could remove the motherboard -- and also removing the SD card, and re-edit the udev because I couldn't figure out how to get it to enter recovery mode the normal way.

I never expected something as innocent as turning debug mode on in udev to lock up a system...

But that's specifically why I bought the AuraHD and Glo, and an SD card reader; as opposed to a plain Aura, or AuraH20, etc. or even a kindle. Because those other ones either don't advertize having an internal SD card, and I'm a hacker at heart and I know there are risks... (My first thread here was about a Sony reader, which I irreversibly bricked.)

In fact, I'm looking at making a new face-plate for my AuraHD anyway; so taking it apart didn't bother me. (It's glued/taped fairly strongly, so removing it is likely to stress the bezel and I don't recommended doing that for everyone. The Glo's are much safer to take apart. )

The factory made bezel overlaps the edge of the screen slightly, so that the first character when printed at the very edge of the screen is partially obscured by the Bezel. (In fact, when I use 8x12 pixels as a letter size -- it obsucres one whole letter -- I have 30/30 vision -- but you might need a magnifying glass to see something that small. It makes for up to 180x90 character display terminal ) and I find obscured edges annoying -- and my kids didn't wan't the AuraHD -- becuase the extra glossy black plastic shows fingerprints really badly and they thought that makes it look dirty. (Which is fine with me, as the Glos's cost less.)

So -- I'm thinking that I'll make a custom bezel out of a silicone urethane mix for my AuraHD since I don't like the fingerprints much either and then I can thumb my nose at my kids. But I do like the higher resolution screen.

As far as the Orientation, whenever I read the information for the rotation orientation from the framebuffer on the AuraHD -- it returns a number 0..3, which is different from the number written to it in the first place to set the orientation. So the read and the write are simply different numbers; but it's consistently wrong, not random -- which is why If you look in the source code I gave at the beginning of the thread -- I exclusive ored that number with 2, so that 0<-->2 and 1<-->3. eg: it causes an erroneous 180 degree flip to correct so that the screen re-loads in the same orientation it was saved in.
Some devices both read and write the same number, some read one set of numbers but write another.

That was my first clue that there was something quite different between the Glo and the AuraHD.

[frostschutz]

Originally Posted by tshering

I can confirm that the image is rotated 180 on the Touch.

Ahhh, clusterfluffle. What's more, my simple workaround doesn't seem to work when using ioctl. FBIO_PUTVSCREENINFO does not flip rotation for me, unless I tested wrong? Writing to /sys/class/graphics/fb0/rotate flips it. What's up with that? So much for not changing rotation, if I have to change it to detect whether it's flipped or not. Oh well, I guess I'll cheat.

EDIT: I take it back. PUTVSCREENINFO flips it, but GETVSCREENINFO too so after a PUT/GET it was back to the original. I can work with that I guess, although it still introduces a race condition.

EDIT2: 180° rotate hack, thanks for testing...

Originally Posted by fastrobot

The factory made bezel overlaps the edge of the screen slightly, so that the first character when printed at the very edge of the screen is partially obscured by the Bezel.

On the H2O the bezel kills the top 10 pixels. Actually since I opened mine it's the top 9 pixels or so, whenever the Kobo displays all black screen (during update or reset or poweroff) I can see a white stripe at the top.

Not sure why they do that. I want ALL the pixels...

[fastrobot]

Originally Posted by tshering

I can confirm that the image is rotated 180 on the Touch.

Oh?

Under what condition, running the framebuffer switcher as compiled at the start of the thread or did you recompile it to not have the AuraHD bugfix or something altogether different?

Because the switcher at the start of this thread will erroneously cause rotations on some devices....
and fix rotations in others...

-----

So:
Here's a more flexible version of fbswitch .
By default it does NOT rotate the buffer, and assumes the kernel isn't lying about the rotation -- but it will rotate the framebuffer by 180 if you add a command line switch of -r. So you can fix the problem (or not) as you desire.

I haven't tested it, but it's only five lines different than the last program...
Worst case it will always do the opposite of the earlier program, so choose whichever program works correctly for your device.

0bc89b8063c67b0a67f4140d673e9a5e fbswitch.tar.gz

[frostschutz]

@fastrobot: sorry, it seems I inadvertently abducted your thread, I was referring to my own program earlier (which I wrote because your fbswitch inspired me to do so)

Originally Posted by frostschutz

Thanks for your fbswitch, it inspired me to finally try to talk to the framebuffer directly instead of going through pickel showpic.

It's odd, I thought my workaround worked, but it actually stops working after a reboot.

If you write to /sys/class/graphics/fb0/rotate, the value you write is flipped.

And then the PUT/GET_VINFO syscalls flip too!

But if you never wrote to /sys/class/graphics/fb0/rotate at least once, PUT/GET_VINFO doesn't flip!

Huuuh?

Now how to hunt for the rotate flippery flag that's hidden somewhere in the bowels of this ... module ... thing.

[davidfor]

Originally Posted by fastrobot

Because I have edited some files on the working image; and then an update happens... and some of my files were replaced, some were not: Simple Conclusion, not all files in the working image are being replaced by updated ones, hence the update is a partial system update.

But, did you edit executable or configuration files? The latter might be generated at run time by something so we don't want them replaced. If there are executables that are not replaced by an upgrade, that to me is a concern. Or it means they are no longer being used.

And that includes the kernel and kernel drivers, such as are used for the ir touch?

That I don't know. Not all are needed as booting from the recovery partition simply runs a script to reinstall the factory firmware in the root partition. The script is very similar to the one that installs the upgrade, but it formats the root and user partitions first.

Which gives one more possible reason why the KoboGlo's e-paper driver reports correct orientation, but the Kobo AuraHD reports a flipped orientation (180 degrees out from reality). For if the CPU is the same, the build in e-paper hardware must be the same -- and only the kernel driver could be causing the problem. But if the hardware is different they could have the same driver, and still have different operation.

When I get my new Glo, I'll have to check the exact CPU in /proc/cpuinfo -- and download the /proc/config.gz from both the Glo and the AuraHD, so I can run diff on them and see what is reported as different...

The Glo and Aura HD are different hardware. They share some things, such as the CPU, but other elements are different enough that the upgrade package includes a separate kernel image for the Aura HD, but the Glo, N905C Touch and the Mini share one.
[QUOTE]
Wait. Do you mean that there is an upgrade package on the backup/resert partition, which installs those files rather than copying the backup partition in its entirety?
I would have expected it to just clone the backup partition using dd.
For if it merely installs a package, then it might not be replacing the kernel on the working root image -- and if the working root kernel got corrupted, even a restore might not fix the reader.
/QUOTE]
Well, that's why I said "installs it's copy of the firmware files in exactly the same way that the upgrade does". You can't do a dd of the recovery partition to the root partition as there is code to execute the reinstall. You would need to remove that afterwards. In any case, the factory reset does install a copy of the uBoot and kernel. And everything else.

I was planning on doing the same thing, except installing a hacked version of the upgrade so that a factory reset brings it back to a working version with my extensions already built in; and probably some modifications so that when a new upgrade is detected, that it checks them to see if they can be patched to include my modifications to /etc/init.d/rcS

Yes, that should work. Others are fiddling with the rcS and others have replaced the firmware package on the recovery partition with later versions. Getting it right is the important thing. But taking a copy of the SD card will save you if you make a mistake.

I see. The repository for hardware -- is as a whole -- biased-labeled glohd kernels.
Even though it has the kernels for all the readers up until now, and not just glohd.

Who knows, next year they might add a kindle kernel to the repository and rename the whole directory "kindle kernels." ! (Wouldn't that be helpful?!) Or maybe someone will write a python script to change the directory names once a week to alternate possibilities....

I didn't have a clue what you meant by this but I went a directory up and realised. What you are talking about is the comment from the last commit in that area. The columns in the list are the name item in the repository, the comment included in the last commit for that item, or item within a directory, and how long ago that commit was made. Hence, "glohd kernel and bootloader" because that was the last thing changed. And before you say anything, the quality of the commit comment is on par with just about any place I've worked.

But seriously:
Inside, under imx507 and not labeled GLO as a directory -- which is why I didn't notice it before; is a directory containing a second kernel marked kobo-glo. It's just not labeled as such on the directory which contains it.

A bit of labeling inconsistency which makes doing GPL recompilation work harder.
If it has the Glo and Touch kernel, the directory ought to be lableled imx507-glo+touch+foobaz (whatever other devices use this kernel).

So -- I linked my pseudo zforce-ir-touch driver against the AuraHD kernel, and it works perfectly in the AuraHD -- but occasionally locks up in the Glo. Go figure... and the kernel didn't even complain that it was a mismatched module...

Yes, it would have been very good if their crystal ball had told them that they would need multiple versions of this and exactly what they would need to call them.

[fastrobot]

Originally Posted by frostschutz

@fastrobot: sorry, it seems I inadvertently abducted your thread, I was referring to my own program earlier (which I wrote because your fbswitch inspired me to do so)

Oh, I see.

It's odd, I thought my workaround worked, but it actually stops working after a reboot.

If you write to /sys/class/graphics/fb0/rotate, the value you write is flipped.

And then the PUT/GET_VINFO syscalls flip too!

But if you never wrote to /sys/class/graphics/fb0/rotate at least once, PUT/GET_VINFO doesn't flip!

Huuuh?

Now how to hunt for the rotate flippery flag that's hidden somewhere in the bowels of this ... module ... thing.

It would be nice if was compiled as a module... but it's actually absorbed into the kernel monolithic-ally. Only a handful of items appear to be actually compiled as a replace-able module... *sigh*.

The generic framebuffer control stuff is going to be the same in all kernels, because the processor itself has the eink controller built in. So - Since I already know that the generic framebuffer skeleton is in the linux kernel at eg: linux-2.6.35.3/drivers/video/imxfb.c -- that means the individual framebuffers driver programs are going to be in a subdirectory of that file; I think you are going to want the file called: linux-2.6.35.3/drivers/video/mxc/mxc_epdc_fb.c

if you cat /sys/class/graphics/fb0/name
Does it say mxc_epdc_fb ?
If so -- that names the file you want.

There are flags for rotation, and additional flags for flipping (mirroring). So there is a possibility of initializing the display incosistently. For two mirrors may be the same as a rotation...

If you're having trouble with boot inconsistencies, often just reading all the variables and then writing them back exactly the same as read will force initialization to be sane; Or else, just choose a state -- and initialize all variables in the framebuffer to always be in a specific state before nickel boots to guarantee the frame-buffer is in a sane state to begin with. There's no guarantee that Kobo initializes all the variables in the framebuffer, unless you can read their code and check. I been assuming they do initialize the screen, but after all the other inconsistencies ... maybe it is a bad assumption.

[frostschutz]

My goal was to have a program that draws on the display without changing its state (rotation and the like). So you can display an image but if nickel [or whatever] decides to redraw the screen it can do so without trouble.

The program that Kobo provides (pickel showpic) breaks that rule completely, it just rotates the screen and then nickel can't draw anymore until you manually restore the original rotation. In conjunction with the 180°/rotation flip issue it looked like this:

rotate=$(cat /sys/class/graphics/fb0/rotate) # <- x
something pickel showpic
echo "$rotate" > /sys/class/graphics/fb0/rotate # x->x or x->y
cat /sys/class/graphics/fb0/rotate > /sys/class/graphics/fb0/rotate # x->x or y->x

So afterwards, even though pickel changed the rotation, the original rotation is restored regardless whether the device changes the rotation on write or no (write read write cycle gets the original back in all cases).

Or so I thought.

What I'm seeing now (trying to replicate that using the GET/PUT syscalls) seems to be more erratic, in particular the rules change from the moment something was written to /sys/class/graphics/fb0/rotate for the first time.

Before writing to /sys/class/graphics/fb0/rotate: the PUT syscall does not change the rotation. (x->x->x->x->x)

After writing to /sys/class/graphics/fb0/rotate: the PUT syscall does change the rotation (x->y->x->y->x) and the GET syscall returns the inverse value.

Suddenly on one and the same device you can't rely on the rotation return value anymore! My earlier attempt shows picture correctly or turned on its head depending on whether something was ever written to /sys/class/graphics/fb0/rotate before.

I'm too tired right now to puzzle this out completely, will try again tomorrow. I hope it's just a bug in my code.