Run test error: FDTD Version mismatch


#1

I installed FDTD Solution 8.20.1634 on a Xeon Phi machine 7210 running on CentOS 7.3.1611.
The application is installed and fetch the floating license on another computer without trouble. The FlexNet license manager is on its latest version, 1.7.1073, and running on a Windows server machine.

Problems arise in the “Resource Configuration” tab.

I can run simulations using the “Local Computer” job launching preset, it passes the the configuration test, but it is very slow (slower than a desktop computer).
Therefore, I tried using MPI, both MPICH2 Nemesis and Intel MPI. In both cases I get errors.

a) For a low process counts I get the error message “FDTD Version mismatch”
Version_mismatch

b) For a high number of process I get a timeout.

I checked that the MPI was properly recognized by FDTD Solution and get the same result with the default and Intel MPI. Here is the MPI configuration, as returned by “fdtd-mpi-status.sh”:

I am not sure of where the “FDTD version mismatch” is coming from: I am using a single machine (with a high number of processors), with only one version of FDTD, so no conflicts between nodes or something.

If anyone has ideas regarding this error message it would be greatly appreciated.

Have a nice day,
Arthur


#2

Arthur,

Did you install FDTD Solutions and FlexNet license manager as shown from our KB installation guide?
https://kb.lumerical.com/en/index.html?install_linux_instructions_fdtd.html

If so, MPICH2 should be installed using the default location.

  1. Try to run an example simulation:
    https://kb.lumerical.com/en/nanowire.fsp
    https://kb.lumerical.com/en/ppc_cavity.fsp
    – save these files into your local user account folder. ie… "~/" or "/home/your_username/"

  2. Set your resource configuration to:

    job launching preset = remote: MPICH2
    #processes = 8, #threads = 1

  3. Otherwise, use “Custom” as your job launching preset and please make sure that the command to execute on the advanced resources points to the correct MPICH2 location for your version of FDTD.

Hope this helps.
Cheers,
Lito


#3

Dear Lito, thank you very much for your swift answer.

About the installation process, yes I followed the KB installation guide.
The FlexNet license v1.7.1073 is installed on another computer of the network, running on Windows server 2012, and used daily by our regular FDTD cluster and desktop computers.
For FDTD Solutions on Linux I followed the guide and installed FDTD Solutions v8.20.1634.

The licensing seems to be properly configured, there is no error when starting the software, it can fetch properly the license.

I confirm that MPICH2 is installed in the default location. “mpiexec” is in this location:
“/opt/lumerical/fdtd/mpich2/nemesis/bin/mpiexec”

Here is the output when I run the configuration test:

I can, however, run the simulation.
The simulation is really slow, 2.7 MNodes/s, compared to my simple desktop, 128.7 MNodes/s.
Here is the log file:
ppc_cavity_p0_xeon_phi_MPICH2.log (9.9 KB)

I tested another MPI, the Intel MPI and got the same output:

In this case the simulation is faster but still underperforming.
Here is the log file:
ppc_cavity_p0_xeon_phi_IMPI.log (9.8 KB)

For the sake of comparison, here are the performance of both my desktop and the Xeon Phi machine using the LINPACK benchmark.
3_LINPACK_results

I understand well that the number of GFlops do not transfer directly into FDTD power but it seems reasonnable to expect the new machine to perform FDTD computations faster.

What does “FDTD version mismatch” exactly means?

I tried to look for it but couldn’t find anything. Now this post is the top answer on Google for this result.


#4

I will forward this to our development team to look into any configuration or settings on a Xeon Phi 7210 system. We will get back to you as soon as possible.

As for the FDTD verison mismatch. The error will sometimes occur when #process is set to more than 1.
To just run the test, please set the #processes to 1 as shown. Then every time you RUN the test it should not anymore give you the FDTD version mismatch.We are also looking into this issue on Linux systems and will try to have this fixed on our upcoming release.

Thanks for the updates.
Lito


#5

Please find attached the result you asked for:

I also ran other combinations I could think of, all returning the same error.

2_configuration_test_multiple

Before doing these tests on CentOS I did some previous tests on Windows. However, on this operating system the system struggled to achieve good performance both on the LINPACK benchmark and in FDTD Solutions. This could have been a configuration problem on my part though.

Thank you for your help,
Arthur


#6

Do you get the version mismatch error everytime you run the test after changing the resources to processes 1 and threads 1 and then saving the settings. Open the resource configuration after and re run the test.

Please save the resource configuration first after doing any changes. Then open again and run the test using the saved settings.

We are looking into this issue with the Run Test option. From our tests, it sometimes give a false error of version mismatch but will also provide a “successful test”.

If the simulation if able to run even with the mismatch error - this should not affect your simulation. For the moment we can ignore this error and will try to have this fixed in the next release.

As what we know, Xeon Phi performs better on Linux systems and we are not sure if Xeon Phi comes pre-installed with windows system.

Another test we can do is use the example file from this KB page:
https://kb.lumerical.com/en/index.html?installation_and_setup_fdtd_simulation_benchmark.html
This should provide some comparison with other systems that we tested.

This guide from Intel might be useful.

https://software.intel.com/en-us/articles/quick-start-guide-for-the-intel-xeon-phi-processor-x200-product-family

Configuration Options
After installing the software stack, you can configure the Cluster and Memory Modes on your system. Please refer to the following article for information related to configuring Cluster and Memory modes supported by the Intel Xeon Phi processor.
Intel® Xeon Phi™ x200 Processor - Memory Modes and Cluster Modes: Configuration and Use Cases