Aura H2O tps65185_int_func under-voltage on DCDC1 detected !

[DomesticExtremis]

Originally Posted by pazos

I'm wondering what happens if you boot your device without the ribbon cable connected to the screen (do it at your own risk, ofc).

No visible difference, same errors:

U-Boot 2009.08-dirty-svn ( 7月 16 2014 - 17:36:26)


CPU:   Freescale i.MX50 family 1.1V at 800 MHz

mx50 pll1: 800MHz

mx50 pll2: 400MHz

mx50 pll3: 216MHz

ipg clock     : 66666666Hz

ipg per clock : 66666666Hz

uart clock    : 24000000Hz

ahb clock     : 133333333Hz

axi_a clock   : 400000000Hz

axi_b clock   : 200000000Hz

weim_clock    : 100000000Hz

ddr clock     : 266666666Hz

esdhc1 clock  : 80000000Hz

esdhc2 clock  : 80000000Hz

esdhc3 clock  : 80000000Hz

esdhc4 clock  : 80000000Hz

Board: MX50 RDP board

Boot Reason: [POR]

Boot Device: SD

I2C:   ready

DRAM:  512 MB

MMC:   FSL_ESDHC: 0, FSL_ESDHC: 1, FSL_ESDHC: 2

In:    serial

Out:   serial

Err:   serial

[_get_sd_number] g_sd_number:2


MMC read: dev # 2, block # 1023, count 1 partition # 0 ... 

1 blocks read: OK


MMC read: dev # 2, block # 1024, count 1 partition # 0 ... 

1 blocks read: OK

PCB ID is 41

ESDin=0,UPGKey=-1,PWRKey=0

ram p=70000000,size=536870912


MMC read: dev # 2, block # 14335, count 1 partition # 0 ... 

1 blocks read: OK


MMC read: dev # 2, block # 14336, count 13205 partition # 0 ... 

13205 blocks read: OK


MMC read: dev # 2, block # 18431, count 1 partition # 0 ... 

1 blocks read: OK

no "logo" bin header


MMC read: dev # 2, block # 45055, count 1 partition # 0 ... 

1 blocks read: OK

no "logo" bin header

Kernel RAM visiable size=505M->505M

init TPS65185 power ...

Relock PLL1 to 1GHz ...

mx50 pll1: 1000MHz

mx50 pll2: 400MHz

mx50 pll3: 216MHz

ipg clock     : 66666666Hz

ipg per clock : 66666666Hz

uart clock    : 24000000Hz

ahb clock     : 133333333Hz

axi_a clock   : 400000000Hz

axi_b clock   : 200000000Hz

weim_clock    : 100000000Hz

ddr clock     : 250000000Hz

esdhc1 clock  : 80000000Hz

esdhc2 clock  : 80000000Hz

esdhc3 clock  : 80000000Hz

esdhc4 clock  : 80000000Hz

Hit any key to stop autoboot:  0 


MMC read: dev # 2, block # 2047, count 1 partition # 0 ... 

1 blocks read: OK

no kernel image signature !


MMC read: dev # 2, block # 2048, count 8192 partition # 0 ... 

8192 blocks read: OK

## Booting kernel from Legacy Image at 70800000 ...

   Image Name:   r7407_#3027 Aug 13 17:39:39

   Created:      2014-08-13   9:39:41 UTC

   Image Type:   ARM Linux Kernel Image (uncompressed)

   Data Size:    1953688 Bytes =  1.9 MB

   Load Address: 70008000

   Entry Point:  70008000

   Loading Kernel Image ... OK

OK


Starting kernel ...


Uncompressing Linux... done, booting the kernel.
drivers/video/mxc/lk_tps65185.c(556):tps65185_int_func under-voltage on DCDC1 detected !

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=0!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=1!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=2!

1+0 records in
1+0 records out
512 bytes (512B) copied, 0.000220 seconds, 2.2MB/s
cannot open /dev/null
drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=3!

dosfsck 3.0.6, 04 Oct 2009, FAT32, LFN
There are differences between boot sector and its backup.
Differences: (offset:original/backup)
  489:8d/93, 490:ef/03, 491:02/81, 492:d2/03, 493:00/2a
  Not automatically fixing this.
drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=4!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=5!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=6!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=7!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=8!

/dev/mmcblk0p3: 207 files, 6669/3634233 clusters
drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=9!



(none) login: [PROGRESS_BAR-3009] No progess ...

epdfbdc_set_width_height(1439): new DCSize(3342336)>original DCSize(3317760) !

sh: you need to specify whom to kill
killall: fickel: no process killed
killall: udhcpc: no process killed
killall: wpa_supplicant: no process killed
killall: ntpd: no process killed
drivers/video/mxc/lk_tps65185.c(556):tps65185_int_func under-voltage on DCDC1 detected !

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=0!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=1!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=2!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=3!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=4!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=5!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=6!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=7!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=8!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=9!

killall: fickel: no process killed
killall: udhcpc: no process killed
killall: wpa_supplicant: no process killed
killall: ntpd: no process killed
drivers/video/mxc/lk_tps65185.c(556):tps65185_int_func under-voltage on DCDC1 detected !

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=0!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=1!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=2!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=3!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=4!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=5!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=6!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=7!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=8!

drivers/video/mxc/lk_tps65185.c(1352):wait power on timeout lpcnt=9!

Power down.

Which would lends weight to the electronics being at fault, not my screen.
If this turns out to be true, disconnecting the screen might be a good diagnostic test.

Time to try measuring some voltages on those tiny, tiny pins....
Stay tuned.

[DomesticExtremis]

Well, this is going to be tricky. The chip has four power modes correspnding to the Kobo modes as follows:

POWER DOWN   Battery disconnected
SLEEP                 Power down
STANDBY            Sleep
ACTIVE                Powered on

The state diagram is attached, as can be seen, the power rails are only on in the active state, so I can only take measurements during the brief period between powering up and the chip shutting itself down

[DomesticExtremis]

Moreover, there are power rail dependencies, meaning that not everything comes on at once:

8.3.2 Dependencies Between Rails
Charge pumps, LDOs, and VCOM driver are dependent on the positive and inverting buck-boost converters and
several dependencies exist that affect the power-up sequencing. These dependencies are listed below.
• Inverting buck-boost (DCDC2) must be in regulation before positive boost (DCDC1) can be enabled.
Internally, DCDC1 enable is gated by DCDC2 power good.
• Positive boost (DCDC1) must be in regulation before LDO2 (VNEG) can be enabled. Internally LDO2 enable
is gated DCDC1 power-good.
• Positive boost (DCDC1) must be in regulation before VCOM can be enabled. Internally VCOM enable is
gated by DCDC1 power good.
• Positive boost (DCDC1) must be in regulation before negative charge pump (CP2) can be enabled. Internally
CP2 enable is gated by DCDC1 power good.
• Positive boost (DCDC1) must be in regulation before positive charge pump (CP1) can be enabled. Internally
CP1 enable is gated by DCDC1 power good.
• LDO2 must be in regulation before LDO1 can be enabled. Internally LDO1 enable is gated by LDO2 power
good.

So, as well as battery voltage and the input voltage to the chip, we will have to look at the pins associated with DCDC2 as well as the pins for DCDC1.
The top half of the typical application gives a hint as to some useful measuring points and pins of interest.

[DomesticExtremis]

Finally the pinout for the chip (attached), and a list of things to measure:

1. battery voltage 3.3 - 4.2V
2. Pins of interest:


Name Pin Description Voltage(min/typ/max)
--------- ----- ---------------- --------------------------------
VIN 10 Input power supply to general circuitry. 3.0/3.7/6.0

VIN_P 24 Input power supply to inverting buck-boost converter (DCDC2). 3/3.7/6

VN_SW 25 Inverting buck-boost converter switch out (DCDC2). -16V

VN 26 Feedback pin for inverting buck-boost converter (DCDC2) and supply for VNEG LDO and VEE charge pump. 16v(?)

VB_SW 40 Boost converter switch out (DCDC1). 16V

PGND1 41 Power ground for DCDC1. 0V

VB 42 Feedback pin for boost converter (DCDC1) and supply for VPOS LDO and VDDH charge pump. 16V(?)

Simples...

[DomesticExtremis]

Well, I said it was going to be fiddly, and it is.
Not only do I have a very short time window to work with before the chip shuts itself down, but the pins are so tiny that my multimeter probes look like giant's fingers by comparison.

So far I have measured the battery at 4.12V and VIN (pin 10, power to the whole chip) at 4.11V.
Which is good as it tells me we definitely are getting the right amount of voltage through the system and that there is no UVLO in play.

Actualy, a ;ittle more can be gleaned from this - from the datasheet:

8.3.8 Fault Handling And Recovery
The TPS65185x monitors input/output voltages and die temperature. The device will take action if operating conditions are outside normal limits when the following is encountered:
• Thermal Shutdown (TSD)
• Positive Boost Under Voltage (VB_UV)
• Inverting Buck-Boost Under Voltage (VN_UV)
• Input Undervoltage Lockout (UVLO)
it shuts down all power rails and enters STANDBY mode. Shut-down follows the order defined by DWNSEQx registers. The exception is VCOM fault witch leads to immediate shutdown of all rails. Once a fault is detected, the PWR_GOOD and nINT pins are pulled low and the corresponding interrupt bit is set in the interrupt register.
...
(blah)
...
Whenever the TPS65185x encounters undervoltage on VNEG (VNEG_UV), VPOS (VPOS_UV), VEE (VEE_UV)
or VDDH (VDDH_UV), rails are not shut down but the PWR_GOOD and nINT is pulled low with the
corresponding interrupt bit set. The device remains in ACTIVE mode and recovers automatically once the fault
has been removed.

So since we have one of the first four errors, the device goes into STANDBY (not ACTIVE), which means some of the input voltages may still be present. We know we don't have TSD and UVLO, so that only leaves VB_UV and VB_UV. The datasheet is not explicit as whether undervoltage condition applies to the input or output voltages for DCDC2 or DCDC1, I suspect it means the output. To confirm I need to try and measure the output from DCDC2 (pin 25, VN_SW) when the chip is powering up.

[pazos]

Wow, @DomesticExtremis, You did your own research.

Probably you're right on the various low level bits. I hadn't time to check. My suggestion about UVLO was just after a quick read of the datasheet, but given there's an specific error for that on kernel sources I would came to the same conclusion as you: not our problem.

Please keep in mind that the PMIC you're exploring is just for driving power to the screen and shouldn't have the capacity of powering the whole system down. There is another PMIC based on MSP430 that's driven via i2c and that handles the various bits exposed to linux (battery, frontlight, GPIO...) and it is the one that could reset the SoC on demand.

https://github.com/pazos/linux-2.6.3.../mxc/pmic/core

On my Kobo Aura HD it has its own serial header (labeled J2) but it just produces gliberish data connected at standard baud rates from 9600 to 115200.

[pazos]

Originally Posted by DomesticExtremis

Which would lends weight to the electronics being at fault, not my screen.
If this turns out to be true, disconnecting the screen might be a good diagnostic test.

I think I don't follow because you're talking about disconnecting the screen as a good diagnostic test and at the same time you said that you don't find any difference with the screen disconnected?

I'm sorry but my english is not good and this is a bidirectional problem, both Input and Output

[DomesticExtremis]

Originally Posted by pazos

I think I don't follow because you're talking about disconnecting the screen as a good diagnostic test and at the same time you said that you don't find any difference with the screen disconnected?

I'm sorry but my english is not good and this is a bidirectional problem, both Input and Output

Well I didn't explain it too well.
If disconnecting the screen has made a difference to the errors seen, we might conclude that the screen is the problem. It did not change any errors, so I inferred that the problem was with the chip, not the screen.
Really, we need one with a known broken screen to repeat the test and see for sure what happens. (i'm not planning to break my screen just for that though ).

[DomesticExtremis]

Originally Posted by pazos

Wow, @DomesticExtremis, You did your own research.

Probably you're right on the various low level bits. I hadn't time to check. My suggestion about UVLO was just after a quick read of the datasheet, but given there's an specific error for that on kernel sources I would came to the same conclusion as you: not our problem.

I'm no expert, that's why it is slow and ponderous. It makes my head hurt and my eyes go funny, but I'm slowly getting somewhere

Please keep in mind that the PMIC you're exploring is just for driving power to the screen and shouldn't have the capacity of powering the whole system down. There is another PMIC based on MSP430 that's driven via i2c and that handles the various bits exposed to linux (battery, frontlight, GPIO...) and it is the one that could reset the SoC on demand.
https://github.com/pazos/linux-2.6.3.../mxc/pmic/core

Yes I get that, I think it's some other processor, or the code that puts the eReader into 'sleep' mode. The chip puts itself into standby mode based one of a certain set of errors which is equivalent to the eReader being in Sleep mode (when the eReader is 'powered down', the chip is in sleep mode, just to avoid confusion).
ETA: actually the eReader is not in sleep mode because the front light stays on. It just fails to boot fully according to the console messages. However the chip goes into standby, as it would if the eReader were sleeping.

On my Kobo Aura HD it has its own serial header (labeled J2) but it just produces gliberish data connected at standard baud rates from 9600 to 115200.

Yes my Aura h2o has two headers. J5 is the one for the serial console, J1 is near an MSP430 DIP package, but just gives out unreadable characters. I suspect this one might be used for calibrating the screen voltages - apparently that can be done and I think the optimal voltages used change for each screen unit.

[DomesticExtremis]

Phew!
After much fiddling and faffing and going bozzeyed, we have some voltage measurements (see attached table).

Pin (number) and (pin) name should be self explanatory. The TP after the pin name indicates a point on the circuit where is is slightly easier to measure with elephantine multimeter probes. I found these by inspecting the traces, checking continuity to the appropriate pin and checking measurements at both pints. A and K are anode and cathode respectively for a diode. For the capacitors, us the end connected to the pin by inspecting the traces.

Note also I used a digital multimeter which is sluggish in response, so some of the voltages could not be accurately measured before the chip went from active to standby mode.

Just to recap:
Sleep corresponds to eReader powered down
Standby corresponds to eReader in sleep, or after failing to boot (in this case)
Active corresponds to the eReader at power-up and before the chip reverts to standby (after two or three seconds).

Also attached is the application example circuit from the datasheet which, unsurprisingly, is almost identical to that on the board. The test points have been added in for reference.

[DomesticExtremis]

Battery voltage was 4.15 to 4.1 volts after many power cycles to do the measuring.

Other than the stray voltage on VIN, I think we can see that the input power to the chip is fine and as expected, as is the regulated 3.3V to pin 45.

Pins 24,25 and 26 are for the DCDC2 negative voltage boost converter. Here too, the input voltage VIN_P (pin 24) looks fine at near battery voltage and within spec.
Pin 25 is VN_SW, the inverter output and stays resolutely at zero, after several attempts to measure it. However pin 26 does shoot up (perhaps down) to a value near the required -16V. I found it difficult to measure exactly because my meter is sluggish and the window for measuring this value is very brief. I am prepared to accept that it does attain its correct value, just that I am not able to measure before the chip reverts to standby mode.
Quite why VN_SW stays at zero I don't understand, particularly since it feed the other end of the diode to VN, so there should only be a diode drop (min 0.3V) of difference in the voltages of pins 25 and 26 (unless I misunderstand the circuit, which is eminently possible).
It may be that the diode has failed open circuit but that no error is reported because the chip measures VN and not VN_SW.

Anyhoo, moving on to pins 40,41, and 42 for the DCDC1 boost converter which should be supplying the +16V rail, there is something really weird going on. In standby mode there is a small voltage on both VB nd VB_SW. In active mode these values are slightly lower, but nowhere near in spec. Again the two pins are joined by a diode, so their voltages should not be equal. It is possible that the diode is shorted and causing the failure of the boost circuit,
However two failed diodes around the same chip at the same time seems unlikely, more likely is an internal failure in the chip.

[DomesticExtremis]

As Columbo used to say, just one more thing...

I need to test the support components around DCDC2 and DCDC1 to check for failed diodes and/or shorted capacitors. If there are none, we can conclude that the chip is funculated and needs to be changed.
I will need to disconnect the battery and remove the board to do that, so not tonight.
Stay tuned...

[DomesticExtremis]

Originally Posted by MGlitch

It amazes me that you have all the equipment to do everything you've done in various posts on this single topic, and yet you do not have a camera to allow for a second set of eyes on the substrate.

Just for you I have obtained an image of the screen here

[DomesticExtremis]

Right, so , after a brief hiatus where life got in the way of doing something of interest, I'm back.

At this point I'm 90% certain that my screen is not broken and 80% certain that the TPS65285 PMIC chip is faulty based on the boot error messages and the voltage measurements.
However, the chip has a substantial number of external support components - diodes, resistors, capacitors and inductor coils any one or more of which might have failed and be responsible for the chip failing to fully power up.
I can limit myself to just investigating the external components attached to the DCDC2 and DCDC1 subsystems: the former needs to be working prior to powering up the latter. Together these two provide the power rails for the EPD function of the screen.
This exercise is based on the assumption that the typical application circuit in the TPS65185 datasheet is what is implemented on the Kobo motherboard. As it turned out this is almost true, the Kobo also has some extra filter capacitors beyond those indicated on the diagram.

[DomesticExtremis]

The next problem is to find the physical location of the components on the board. Lickily all of the support components ahve only two pins, so the circuit was traced out using the following steps:

1. disconnect battery, screen flat flex and frontlight flat flex
2. with a multimeter in continuity/diode mode (i.e. using beeps) connect one probe to the chip pin of interest and locate the 'nearer' end of the first component by probing around. Note down the silkscreen component identifier.
3. locate the next component(s) by probing the 'far' end of the found component and so on.
4. probe around for extra connected components not shown on the diagram.
5. also probe pins indirectly connected to DCDC1 and DCDC2 by external rails through wich current for the two subsystems might end up flowing.

After inspecting the relevant parts of the diagram we are only interested in a couple of diodes, a few capacitors and some inductors.
Inductors can burn out, so would fail open circuit.
Capacitors usually fail short circuit.
Diodes can fail short circuit or open circuit (aiui).

Measuring the component values in-circuit can give misleading results, so I just tested for open/short circuit.

I mention the above for the benefit of those who might think I know what I am doing.

I don't