Why did Rosetta's camera switch off?

The answer - safety.

This is the best kind of image we will get:

Asteroid (2867) Šteins in color
Three color-filter images from the OSIRIS wide-angle camera were combined to produce this highest-resolution color view of the asteroid from the Rosetta flyby on September 5, 2008. Šteins is, essentially, gray. Credit: ESA ©2007 MPS for OSIRIS Team MPS / UPD / LAM / IAA / RSSD / INTA / UPM / DASP / IDA

Tracking such a small object as asteroid Steins on its (slightly) uncertain trajectory and at high relative speed, pushed Rosetta to its limits. The spacecraft was operating totally independently during the manoeuvre and it was vital that some areas of the craft, the so-called cold faces, received as little direct sunlight as possible. The strain on Rosetta's systems unfortunately took the narrow-angle camera outside of its operating range and the camera did the sensible thing. It put itself into safe mode to avoid damage. All other systems worked fine.

The Planetary Society reports:

One notable instrument found the extreme conditions of the flyby to be out of its set safety parameters, and put itself into a protective "safe mode" nine minutes before closest approach, recovering a few hours later. That instrument was the narrow-angle camera on OSIRIS, which would have produced the highest-resolution images of Šteins. OSIRIS principal investigator H. Uwe Keller explained that the safety parameters had been set conservatively on OSIRIS because, as interesting as Šteins is, it is not Rosetta's primary science target.

Rosetta will now loop back around the Earth before flying on to its next encounter, asteroid 21 Lutetia, on July 10th 2010.