announcing MARK 11.2

[cooch]

As of this afternoon, Gary has released 2 new versions of MARK (also mirrored on the PhiDot server). Both versions have the same 'capabilities' (i.e., same data types), but differ in the underlying compilation in an attempt to make MARK 'faster' on newer CPU architectures. Specifically, on newer CPU's (typically, anything >2015), there are 'hard coded' instructions in the CPU that make certain numerical operations (in particular, matrix manipulations, and aspects of parallel processing) much faster.

So, two versions of the MARK installer are now available: setup-v3.exe (for computers with a CPU architecture dated 2015 or later -- by proxy, any machine purchased 'new' in the last ~10 years), and setup-v2.exe for anything older.

The performance gain on newer CPUs can be considerable - on a 'modern' CPU, the setup-v3.exe file will produce results 5-30% faster than setup-v2.exe. Older machines will not be able to run the executable files in setup-v3.exe because they likely do not support AVX2 (Advanced Vector Extenstions) first introduced in CPUs around 2011, but fully implemented by 2015.

So, if you have a machine purchased new within past 10 years, might be worth updating to v3. On the other hand, if you're not sure, try v3. If it crashes and burns, it is probably because your machine is too old for the v3 instruction set. At which point, simply downgrade to the v2 version, which should work. If nothing else, the v2 version will give you a slight performance tweak regardless of your CPU (the gain most noticeable for v3 on newer CPUs).

Note [1]: the bigger the job you're running (lots of occasions, lots of model complexity), the more noticeable the performance gain will be. If you're running something computationally 'trivial' (say, the dipper data), you likely won't notice anything at all.

Note [2]: the current Linux version of the MARK executable uses v2. I'll provide both v2 and v3 within the next day or two.

[cooch]

For example, using a 'big ugly' job that Gary and I routinely use to benchmark things (following all run on the same computer):

Using 'old MARK':

Code: Select all

For this job, CPU Time in mins was 63.252 and System Time in mins was 17.229.

Using 'MARK v2'

Code: Select all

 For this job, CPU Time in mins was 62.052 and System Time in mins was 16.196.

Using 'MARK v3'

Code: Select all

 For this job, CPU Time in mins was 46.994 and System Time in mins was 12.406.

So, reduction from ~17 minutes to ~12.5 minutes.

[cooch]

I just posted Linux versions of the 'new and improved' MARK (11.2). The READ.ME file (below) contained in the .zip archive has been updated to reflect 'changes & choices'.

Note: There is an even more optimized version (which Gary and I call the 'v4' version'), which has more stringent hardware requirements (very modern CPU being one of them), but which will give you an even greater increase than the 'v3' version (said version being more compatible with more CPU setups). If you have a new, number-crunching machine (typically, a desktop box where the CPU is likely *not* a hybrid of P-cores and E-cores -- as many laptops are now), and want to try 'v4', send Gary or I a note offlist.

MARK 'under Linux'....

The .zip archive contains the most recent version of the MARK executable
(64-bit). It was compiled with gfortran 15.x (meaning, it may not run
if you don't have gfortran 15 libraries installed). Also, it was compiled
for an x86_64-compatible processor (typically, Intel or AMD). Meaning,
if you have an ARM-based CPU (some newer laptops, typically), it won't run.

There are 2 different versions of the executable in the archive. If your
machine is approximately <= 10 years old, use the mark64-v3 file. Otherwise,
use the mark64-v2 file. For some of the more complicated data types involving
the multiplication of large matrices, the v3 MARK executable will run 10-30%
faster (yes, that much faster!) than the v2 executable.

To use the executable (either v2 or v3), simply change the name of the file
to mark (or whatever is most appropriate for your environment), and make sure it
is executable (typically, sudo chmod +x mark will suffice...).

[cooch]

Gary just made the 'v4' version referred to (above) available. It is linked on his website, and here (on PhiDot) as well.

Gary has the following notes on his website - make sure you read it:

Download 1 of the 3 setup.exe files to your hard disk and execute to install MARK.  If your computer is less than 10 years old, the setup-v3.exe file will produce results 5-30% faster than setup-v2.exe.  Older machines will not be able to run the executable files in setup-v3.exe because they do not support AVX2 (Advanced Vector Extensions) first introduced in CPUs around 2011.  If you have a really new machine purchased since 2020 and your processor supports AVX512 (copy your processor name from Settings | System | About and ask Copilot), download setup-v4.exe.  Setup-v4.exe can give you another ~10% over setup-v3.exe.

[You can, in fact, ask any of the commonly used AI tools - CoPilot is simply one of a growing number.]

[jlaake]

If you see a message like the following, it is coming from MARK and may mean that you are using a version of mark.exe that will not work with the cpu in your computer. If you see an error like this, use an older version like v2 or v3. Someone using RMark ran into this error. It is not a problem with RMark.


Program received signal SIGILL: Illegal instruction.

Backtrace for this error:
#0 0xfc81c433
#1 0xfc80abf3
#2 0xfc7d7ba0
#3 0x808e156e
#4 0x833a63fe
#5 0x83252326
#6 0x833a5d3d
#7 0xfc596a6b
#8 0xfc84a575
#9 0xfc591365
#10 0xfc591066
#11 0x8211e8d6
#12 0x832cc53b
#13 0xffffffff

[cooch]

Thanks Jeff --

One quick 'addendum' to Jeff's note -- its not that versions 'v2' and 'v3' are older, its just that they can run on 'older CPU' architecture. All 3 versions (v2, v3, v4) do exactly the same thing, and will yield the exact same results. But, if your computer hardware (specifically, the CPU) is 'new enough', and supports some of the newer instruction sets, then 'higher versions of MARK' will give you faster performance (v4 > v3 > v2 > 'classic MARK').

For most people (which, based on publicly available data, means probably 95% of MARK users), v3 should work fine. v3 is 10-30% faster than the MARK that most people probably have installed on their machines (whereas v2 is typically only 3-6% faster). For the remaining 5%, (i) some will have processors too old to support v3 - so, they should use v2; (ii) some will have the 'latest and greatest', at which point, v4 might work (v3 will definitely work). If v4 works, it squeezes an additional 5-10% performance. But again, v4 will crash and burn *if* the CPU can't handle it.

Whether or not your CPU can handle a particular version of MARK is something you can figure out for yourself (AI tools are your friend here). In particular, you need AVX2 support for v3, and that plus AVX512 for v4.

So, for most people, v3 is likely your best bet.