Characterisation and calibration of multiversal methods

Multiverse Analysis is a heuristic for robust multiple models estimation where data fit many connected specifications of the same abstract model, instead of a singular or a small selection of specifications. Differently from the canonical application of multimodels, in Multiverse Analysis the probabilities of the specifications to be included in the analysis are never assumed independent of each other. Grounded in this consideration, this study provides a compact statistical characterisation of the process of elicitation of the specifications in Multiverse Analysis and conceptually adjacent methods, connecting previous insights from meta-analytical Statistics, model averaging, Network Theory, Information Theory, and Causal Inference. The calibration of the multiversal estimates is treated with references to the adoption of Bayesian Model Averaging vs. alternatives. In the applications, it is checked the theory that Bayesian Model Averaging reduces both error and uncertainty for well-specified multiversal models but amplifies errors when a collider variable is included in the multiversal model. In well-specified models, alternatives do not perform better than Uniform weighting of the estimates, so the adoption of a gold standard remains ambiguous. Normative implications for misinterpretation of Multiverse Analysis and future directions of research are discussed.