Giving the noisy TV a button
ICM's encoder is built to throw noise away. Give the agent a button that reshuffles the noise and it stops throwing it away.
The claim
A curious agent rewards itself for being surprised. Point one at a television showing static and the surprise never runs out, because static is unpredictable by construction. The agent stops exploring and farms the TV. This is the standard objection to curiosity as a reward signal, and it is a good one.
ICM answers it by moving the target out of pixel space.1Pathak et al., Curiosity-driven Exploration by Self-supervised Prediction (2017). Reward is the forward model's error inside φ, so anything φ drops becomes invisible to the reward.1 You learn an encoder φ, and you measure surprise inside φ instead of on raw pixels. φ is trained by inverse dynamics: given φ(s) and φ(s′), predict the action taken between them. The reasoning is short. Noise tells you nothing about which action you took. So an encoder trained to recover the action has no reason to keep noise, and a TV that φ throws away cannot generate reward.
There is a premise buried in that sentence. It assumes the noise happens on its own, regardless of what the agent does. Drop the premise and the logic runs backwards. If some action resamples the noise, then the noise is the only thing that identifies that action, and now the loss has every reason to keep it. This experiment changes nothing else.
The ceiling
The world is an eight-cell corridor with a wall at each end. Position p in [0,7] is three
bits of signal. Next to it sits a TV of K = 32 random bits, redrawn at the start of every
100-step episode. An observation is one_hot(p, 8) ++ bits: forty dimensions
holding three bits of signal against thirty-two of noise. Three actions: left, right,
press.
def step(self, a):
if a == LEFT:
self.pos = max(0, self.pos - 1)
elif a == RIGHT:
self.pos = min(N_POS - 1, self.pos + 1)
else:
if self.slot:
self.bits = self.rng.integers(2, size=K_BITS)Two conditions, one branch apart. Under Frozen, press does nothing and the TV is ordinary noise the agent cannot touch. Under Slot, press redraws all thirty-two bits. Everything else is held fixed. φ is 40 → 64 → d with an ELU, the inverse head is 2d → 64 → 3, and the two train together on cross-entropy against the true action, Adam at 3e-3, 15 epochs over 100k transitions from a random policy.
The walls do something specific. Walking into a wall leaves p alone, and so does pressing, so at a wall those two actions look identical from position. Under Frozen they look identical from everything: the observation before and after is the same in all forty dimensions. That puts a hard limit of 0.50 on those transitions. Not a baseline. No model of any size beats it, because the data does not contain the answer. Under Slot, press flips about half the bits, and a redraw landing on the same pattern has probability 2⁻³².2Roughly one in four billion, so a press is always visible. Whatever error Slot has left is optimization and hash collisions, not missing information.2 So the ceiling there is 1.00.
That same argument predicts overall accuracy, which is worth checking because it tells you the setup is wired correctly. The position chain is a lazy random walk with clamping, so its stationary distribution is uniform. A quarter of states sit at a wall. Two of three actions are stationary there. You get half of that residue by guessing. Frozen should land at 1 − 1/12 ≈ 0.9167. It measures 0.9133, 0.9097, 0.9118 and 0.9131 at d of 2, 4, 8 and 32. Flat in d, which says what is stopping it is missing information and not capacity.
On the wall transitions themselves Slot reaches 0.96 against Frozen's 0.51. You can watch the moment it happens. For six epochs the Slot encoder sits at Frozen's ceiling, having already solved everything position can solve. Then it finds the TV.
What it keeps
Beating 0.50 under Slot proves φ is carrying the TV. The question is what form it carries it in. So freeze φ and fit three probes on fresh held-out states, 10k to train and 5k to test. Two of them ask about content: recover TV bit b from φ(s), once with logistic regression across all thirty-two bits and once with a 64×64 MLP across eight of them. A linear probe failing on its own would settle nothing, since the bits could be sitting there nonlinearly tangled. The third probe asks about change, and it has to take the pair.
z = phi(obs) # content: what did the TV say?
z_pair = cat([phi(obs), phi(next_obs)], dim=1) # change: did the TV move?The split matters because of where φ sits in the loss. It runs on each state on its own, and the head does the comparing.
logits = inv(phi(obs[idx]), phi(next_obs[idx]))
loss = ce(logits, act[idx])So φ cannot store the fact that the TV changed. It never sees both states. The only thing the loss can push it toward is sensitivity: send different bit patterns to different places, and let the 64-unit head read the gap.
Detection comes back at 0.9916 ± 0.0039. Content comes back at 0.5489 ± 0.0147 under the MLP and 0.5233 under logistic regression.3Slot's content probes do sit above Frozen's, by about 0.011 on the MLP at d = 2, and positive in all three seeds. The leak is real. It is also 0.01 against detection's 0.45.3 Two classes of decoder, one representation, and a gap of 0.44 between what can be detected and what can be read.
That is a hash. Its output moves reliably when the input moves, and it does not keep the input. It is also exactly what was asked for. The loss never wanted the bits. It wanted to separate three actions, and a scrambled two-number signature does that.
Worth being clear about what d = 2 does and does not force.4Two float32s carry about 48 bits of mantissa against 35 bits of state, so a lossless encoding exists. Gradient descent on a smooth network does not find it, because nothing in the loss pays for it.4 It is not an information bottleneck. Two floats hold the whole state several times over. What d bounds is the measurement, since a linear probe reads at most about d independent directions. The content being gone under a nonlinear probe too is a result, not something the architecture guaranteed.
The conclusion is narrower than saying curiosity fails on noisy televisions. Against noise the agent cannot touch, the original argument holds, and the Frozen arm reproduces it. The probe sits at chance and the encoder gains nothing. What breaks is the case where the agent can trigger the noise itself.
There is also distance between what got measured and what I would like to say. The noisy TV problem is a claim about reward and about behavior. There is no forward model here, no reward, no policy, and no loop where the agent seeks out the TV and reshapes its own data by doing it. What this gives you is a precondition. φ cannot be blind to the TV, and everything downstream of φ inherits that. Going from there to "the reward stays high" is right about direction and says nothing about size. A two-dimensional hash adds some variance to forward-prediction error. Whether that variance is large next to the ordinary error of predicting where the agent went is not measured, and it could easily be small.
So that is next. Attach the forward model, measure prediction error on press against move in both conditions, and normalize it against the baseline error everywhere else. Two other gaps are worth naming. The data comes from a random policy, so the feedback loop that makes the noisy TV pathological in the first place is absent. And this encoder is an MLP over a one-hot vector, where the original argument was made over pixels with a convnet. Whether a comparable signature of television static forms there is untested.
The general shape of it is that a representation trained on a stand-in objective will satisfy that objective the cheapest way it can, and the cheapest way is often not the one the intuition assumed. Inverse dynamics does not learn what the environment is doing. It learns enough to name the action. Usually those are the same thing. A noise source with a button is a case where they come apart.