Found the slowness! It was because each account that was created for the tests generated two keypairs for activitypub usage. We don't need those for these tests.

Now they run in like 150ms each, or a 10x increase in speed.