The issue seems to be related to choosing the correct branch of the solution. In the analysis script in s_params analysis group, the effective properties are calculated from the S parameters starting at line 205. The equations for the effective n and z values each have 2 roots, and in lines 211 and 214, conditions are applied to choose one of the roots for each frequency point. These conditions work well for the Smith device which is demonstrated in our online example here:

https://kb.lumerical.com/en/index.html?metamaterials_effective_parameters_smith.html

However, the branch choosing conditions are not general, so they don’t work for all devices. Typically, if you would like to extract the effective refractive index, you could plot both branches of n (n1 and n2 calculated on lines 209-210 of the analysis script) and visually check the plots to see which branch makes sense. In the case of your test simulation file, the branches are 1 and -1, and you would want to extract the branch where n=1.

There have been other branch choosing schemes that have been reported in other publications, and you might want to try using. For example, the Smith reference mentioned that a more stable procedure for retrieving effective parameters can be found in this publication:

X. Chen, T. M. Grzegorczyk, B.-I. Wu, J. Pacheco, and J. A.Kong, Phys. Rev. E 70, 016608 s2004d.