I've been in and out of a few prisons in my time. And not just the ones with big stone walls, but the kind where you're on a planet with no convenient spaceport. The loudmouths that talk about escaping all the time never make it out. The quiet, planning ones? Those are the ones the guards should really keep an eye on. They're always waiting for an opportunity. You never see it coming.

[In other words, he's implying don't count other people out just because they don't seem to doing anything. Goroh is ready to escape at any time. But a guy with a lot of muscle and a really sharp sword can't do much until an opportunity presents itself.]