Okay, thanks to your clue I took it one step further. But this is making my head hurt!
Spoiler:
We are six, equally divided.
I wonder what my colour is. I see 3 green, 2 blue.
IF I am blue, the blue ones see 1 blue, 4 green.
The blue wonders what his colour is.
If he is green, the other blue should see 5 greens, and leave.
1 day passes, the other blue doesn't leave.
So it would follow that the second blue is actually blue.
The 2nd day passes, and none of the blues leave.
Which means I am blue too!
If I try to do this with bigger numbers, I think my head will break
But even so, all this takes into account that the prisoners only just realized that there are two colours only, and both are represented. But they knew that from the start, didn't they? And if so, shouldn't they have left sooner? What changed?