As far as I know it doesn't really matter, it will only be delayed at the beginning since you have to wait before they die, but after that it shouldn't matter much. The only thing is that you can't get the bugs instantly, since you cant kill them, so you would have to summon them earlier (anticipate when you would need them).
It didn't take much.. Just spawn as many cultist masters as possible and keep using the ability. Goes pretty quick.
Edit: Am I missing something... since when can buildings be dazed?