Find the Genevestigator configuration by navigating to one of the folders listed below.
Windows: C:\Program Files\Genevestigator\
MacOS X: /Application/Genevestigator/
Now edit the Genevestigator.vmoptions file and change the configuration by uncommenting the matching part in the example below.
# Java: Enlarge current memory size
Make sure to choose an amount of memory appropriate for your machine. Bear in mind that setting this variable too high can result in Genevestigator no longer being able to start. We recommend choosing an amount up to half of your system's memory total. The value in the example is given in Gigabytes.
After changing and saving the file; restart Genevestigator to have the new higher limit enabled.