SharpNEAT

SharpNEAT 2.4.2

Download: Release v2.4.2 on github

Overview

This a maintenance release, there are no major changes between this version (2.4.2) and the previous version (2.4.1). See below for a list of changes.

However, the test platform has seen some significant software updates (the test hardware remains unchanged from release 2.4.1), and the efficacy sampling results (below) show how these updates have affected performance of SharpNEAT on the two standard benchmark tasks. Of particular note are the following platform updates/changes:


Changes


Efficacy Sampling Tests

Efficacy sampling was performed on the two standard benchmark tasks and the results compared between this and the previous version (version 2.4.1). The resulting best fitness histograms are shown below. To recap, these histograms show the best fitness achieved on each of a large number of independent SharpNEAT runs, each of which terminates after one minute of execution (clock-time). Histograms are also provided comparing the evaluation counts achieved in each 60 second run.

For this release an additional set of results has been produced to compare performance of SharpNEAT v2.4.2 with Spectre and Meltdown mitigations all disabled versus all enabled, on a Windows 10 PC (version 1809). Various attempts at mitigating these security flaws have been issued as OS security updates, with varying degrees of performance degradation depending on the CPU, OS, workload and changes to how the mitigations work.

Enabling and disabling of Spectre and Meltdown mitigations was performed using InSpectre (Note. When using this tool it is necessary to reboot the PC after the mitigation settings have been changed).

Results and Observations

With spectre/meltdown mitigations disabled there appears to be a very slight performance improvement on both tasks in the form of a slight shift to the right on the evaluation count histograms (see figure 2). This improvement is most likely due to (a) the upgrade to .NET 4.8, and (b) a minor performance improvement to the Generative Sinewave task code (vectorization of the squared error calculation that is the basis of that task's fitness score).

With spectre/meltdown mitigations enabled there is a clear loss in performance on the Binary11 Multiplexer task, and a significant but lesser performance loss on the Generative Sinewave task (see evaluation count histograms in figure 4).

Going forward there will be further software and hardware changes related to Spectre and Meltdown, so this will be an issue to be considered for the foreseeable future when carrying out performance tests for SharpNEAT, and more generally.


Figure 1.
Comparison of best fitness histograms obtained from SharpNEAT v2.4.1 and v2.4.2 (spectre/meltdown mitigations off);
on the Binary11 Multiplexer and Generative Sinewave tasks.

Figure 2.
Comparison of evaluation count histograms obtained from SharpNEAT v2.4.1 and v2.4.2 (spectre/meltdown mitigations off);
on the Binary11 Multiplexer and Generative Sinewave tasks.



SharpNEAT v2.4.2 with Spectre/Meltdown Mitigations off versus on


Figure 3.
Comparison of best fitness histograms obtained from SharpNEAT v2.4.2 with spectre/meltdown mitigations off vs. on;
on the Binary11 Multiplexer and Generative Sinewave tasks.

Figure 4.
Comparison of evaluation count histograms obtained from SharpNEAT v2.4.2 with spectre/meltdown mitigations off vs. on;
on the Binary11 Multiplexer and Generative Sinewave tasks.


Appendix 1: Test Platform Environment Details

OS Name: Microsoft Windows 10 Home SP0.0
Architecture: 64-bit
.NET Framework: 4.8 (CLR 4.0.30319.42000)

CPU
   Brand: GenuineIntel
   Name: Intel Core i7-6700T CPU @ 2.80GHz
   Architecture: x64
   Cores: 4
   Frequency: 2808
   RAM: 16 GB

Appendix 2: Resources

R Scripts
Histogram CSV data

Colin,
April 27th, 2019


Copyright 2019 Colin Green.
This article is licensed under a Creative Commons Attribution 3.0 License