Trust me, I have done that multiple times. I've taken him to the window for an hour (his favorite place), out to run on the floor, on my bed, for a bath, to eat, etc. It doesn't matter. The second I put him back in there (no matter the length of time he's been out) he is right back at it.