For reference, Fedify makes implementing this kind of fine-grained access control quite straightforward—you can check the Fine-grained access control section in the documentation.

fedify.dev

Access control | Fedify

Fedify provides a flexible access control system that allows you to control who can access your resources. This section explains how to use the access control system.