Not everyone who was addicted to tempo got enough of it in their system to have the forced Spirit Pact come into play. Also note that there wasn't a 1:1 match between tempo users and shadow spirit infused trees used to make the drug. So say there were fifty trees in the grove. That is fifty spirits to 'ride' a few thousand or maybe even tens of thousands of pact members. And the Spirit Pact is not the same as possession. With the spirit pact, when the 'vessel' is asleep or unconscious, the spirit can, if they choose to, take over the body for a while. However, it can only control one body at a time, so it can't control however many hundred are pacted to it simultaneously. It essentially 'jumps into' the person and rides them like a drone before leaving them so they can wake up.