Signal propagation is currently synchronous and single-threaded. This has worked in the past, where signals were propagated towards objects of low relationship cardinality, from objects of high relationship cardinality - e.g. from roles to users. This could be characterized as 'fan-in' signal propagation. In the org model, signal propagation 'fans-out': i.e. signals are propagated down the org hierarchy, from low cardinality relationships, to high-cardinality relationships. This means that the signal associated with adding a new root org to an existing org hierarchy must be propagated down the entire hierarchy, and to all users in this hierarchy.
One option would be to dispatch this signal propagation in an ExecutorService. VertexStatePropagator#propagateVertexStateChange is the entry point into the signal propagation which will occur in each org in the org hierarchy. This threading must maintain synchronous semantics for the mutation which triggered signal propagation. See the DataMigrationService#AsyncReconciliation class for a model of how to maintain synchronous invocations with ExecutorService dispatch in the context of Promises.
This Jira covers doing initial investigation into threading this signal propagation, choosing the appropriate ExecutorService sizing parameters, and the appropriate location for thread dispatch, implementing a PoC, and determining the performance impact using the org model perf tests.
Part of the investigation should also include determining the feasibility of such an approach, as opposed to an approach which persists a representation of this signal propagation to the repo, distributes these jobs across the cluster, and then threads their consumption. The current assumption is that GCP won't shutdown pods actively engaged in an http request. Thus threading signal propagation in memory and maintaining synchronous semantics won't possibly be terminated by abrupt pod shutdown. This assumption should be verified, and alternatives entertained - i.e. can the in-memory threading approach co-exist, or be transitioned to, a repo-persistence threading model?