Any time when a Town Center is not training villagers or researching, a villager is not working, or a military building is not producing. Idle Town Center time is the main thing that separates good and average economies.
The game shows idle villagers with an icon and has a hotkey to cycle through them. Queuing a few villagers at a time and checking that queue often reduces Town Center idle time.