FMS Parameters are scalar values used to control the following:

When you start FMS by calling FMSINI, FMS initializes all Parameters to default values. In most cases, the default values will allow you to get FMS running in your application. However, the FMS Parameters do provide some features that are useful during software development. Therefore you should become familiar with the FMS Parameters to know what options are available.

Setting FMS Parameters

After initialization by FMSINI, the Parameters may be changed in one of the following 3 ways:
  1. CALL FMSIST(NAME, IVALUE)
    CALL FMSRST(NAME, RVALUE)
    CALL FMSCST(NAME, CVALUE)
     These subroutines provide a way to change INTEGER, REAL*8 and CHARACTER parameters respectively. You supply the NAME of the parameter as a character string and the new value as an INTEGER, 8-byte real or character string. FMS checks your value, and if appropriate changes the value of the FMS Parameter.

    This method of setting FMS Parameters is recommended for modifications that become a permanent part of your application. For example, if know your matrix should always be positive definite, you should set the FMS Parameter MPOSDF as follows:

    CALL FMSIST ('MPOSDF', 1)
  2. CALL FMSSET
    Inserting this call in your application will cause FMS to solicit input interactively for viewing or changing FMS Parameters as follows:

    FMSSET>name

    where

    FMSSET> is the prompt displayed by FMS and name is the name of the FMS Parameter whose value you want to display.

    You may change the value of an FMS Parameter as follows:

    FMSSET>name=value

    where

    value is the value you want to assign to the parameter.

    When you have finished changing all the required parameters, you enter the characters RETURN at the FMSSET> prompt,

    FMSSET>RETURN

    and FMS continues on with your application.

    This method of changing FMS Parameters is especially useful during development and tuning. It allows you to print out intermediate results or try out Parameters without them becoming a permanent part of your application.

  3. License File
    You may also change the initial (default) value of FMS Parameters in the License File by adding the following lines between the FMSSET and RETURN records:

    FMSSET
    name=value
    ...
    name=value
    RETURN

    When FMS encounters the FMSSET record in the License File, it switches input to FMSSET for further processing. FMSSET then processes each modification name=value as though it was entered directly from a terminal. When FMSSET encounters the RETURN record, it transfers processing back to FMS.

    You should use this method to set FMS Parameters that are not part of the application, especially parameters that describe your hardware configuration. You may also use this method to try different parameters during development in a similar way to calling FMSSET.

Calling FMSSET, either from within your application or through the License File allows you to change FMS parameters without recompiling and relinking your application.

Reading FMS Parameters

You can obtain the value of FMS Parameters using one of the following methods:
  1. From within your application: 
           CALL FMSIGT(NAME, IVALUE)
       CALL FMSRGT(NAME, RVALUE)
       CALL FMSCGT(NAME, CVALUE)
    These subroutines provide a way to obtain the value of INTEGER, REAL*8 and CHARACTER parameters respectively. You supply the NAME of the parameter as a character string and FMS returns the INTEGER, 8-byte real or character string value.
  2. At the FMSSET> prompt enter the name of the Parameter, 
    FMSSET>name
    where name is the name of the FMS Parameter whose value you want to display. FMS then displays the Parameter name and its value.
  3. Use the SHOW value to display a group of Parameters as follows:
    • From within your application, 
             CALL FMSCST(SHOW, CVALUE)
    • At the FMSSET prompt, 
      FMSSET>SHOW=CVALUE
    where CVALUE is a character string describing the information to display. The following table lists the options and their description.
    Displaying Multiple Parameters and Tables
    CVALUE Description
    PARAMETERS Lists all Parameters and their values
    MODIFIED Lists the Parameters that have been modified since the License File
    MEMORY Displays a table of memory allocation
    MALLOC Displays a table of MALLOCed memory
    FILES Displays a table of open FMS files
    FMSDISK Displays a table of disk usage
    CALLS Displays a table of subroutine calls including the number of times called and the amount of time (CPU and WALL) spent
    PROBLEM Shows the maximum problem size
    SEMAPHORES Lists the semaphores and their current value
    SEMHISTORY Lists the most recent history of semaphore values
    IOQUEUE Shows the status of the I/O queues
    USAGE Lists the usage statistics from the operating system
    (not available for all operating systems)
    PLACEMENT Shows memory placement on NUMA machines
    PLACEMENTSUMMARY Shows a summary of memory placement on NUMA machines
    LIBRARIES Lists the dynamically loaded libraries and their entry addresses
    GPU Lists the details of each GPU
    You can also list the options available in this table at the FMSSET> prompt with
    FMSSET>SHOW
With this background, let's take a quick tour of the FMS Parameters. If you want more information on any Parameter, just click on its name.

Hardware Parameters

NAME Function
MAXCPU Number of processors to use
MYNODE Your process number (0 to MAXCPU-1) (read-only)
MDHW Amount of memory installed (8-byte words)(read-only)
MAXMD Amount of memory to use (8-byte words)
MAXMB Amount of memory to use (Megabytes)
MEMPCT Amount of memory to use (% of MDHW)
MDUSED Amount of memory already used (8-byte words)(read-only)
MEMPTR Pointer to FMS managed memory (read-only)
MDHOW How memory is allocated
MDINIT Memory initialization options
MALIGN Memory alignment (8-byte words)
LPAGE Page size (bytes)
NOPAGE Page file location
ISYNC Type of CPU synchronization
ISCOPE Scope and priority of threads
MDPADD Append prefetch area to memory allocations (8-byte words)
LCACHE Size of cache (bytes)
LSTACK Size of stack (bytes)
ITPCPU Processor type (1=scalar, 2=vector)
LENVRG Vector register length (8-byte words)

These parameters provide a way for FMS to know your hardware configuration. Some operating systems provide hooks to determine the number of installed processors and amount of memory. If available, FMS bases its default values on this information. In other cases this information is not available. You should review what FMS uses for the default values of MAXCPU and MAXMD and modify the value in the License File, if required.

The MYNODE Parameter allows you to obtain the thread number that is currently being used to execute your subroutine. The threads are numbered from 0 (parent) to (MAXCPU-1). Typically you would use this value in subroutines you are running in parallel using the FMS parallel processing routine FMSPAR.

The MDHW Parameter stores the amount of installed memory in 8-byte words. This is a read-only parameter.

You can specify the amount of memory FMS uses in one of 3 ways:
  1. MAXMD specified the amount of memory in 8-byte words;
  2. MAXMB specifies the amount of memory in Megabytes;
  3. MEMPCT specifies the amount of memory as a percentage of the amount of installed memory MDHW.

The MDUSED Parameter allows you to obtain the amount of memory that has been used from the FMS memory pool. You may use this if you are calling FMSROW or FMSCOL to populate multiple matrix files simultaneously, or are using the FMS Memory Management System.

The MEMPTR returns the starting address of the pool of memory allocated by the FMS Memory Manager. It is used in FORTRAN programs to set the POINTER value in the arrays used to access FMS memory. In C applications, it provides a pointer to the start of the FMS memory.

The MDHOW Parameter determines which system services FMS uses to allocate memory. The options include mmap, malloc, shm and a fixed array. Not all options are available on all machines. You should use the default value selected by FMS.

The MDINIT Parameter provides options for initializing the memory in the FMS memory pool. Normally this is not required. However, if you are performing benchmarking and want to set the system page table to a known state, you may do so with this parameter. When NUMA options are used, the default value of this Parameter is automatically changed to explicitly place the memory.

The MALIGN Parameter determines the alignment of FMS memory. Typically this is set to the page size LPAGE.

On most systems, FMS uses the system page file for backing up the memory allocated by FMS. The NOPAGE Parameter provides the option of creating a special page file for FMS. However you should use this option with caution. Some systems will attempt to continuously keep this private page file consistent with the FMS memory pool, resulting in degraded performance.

The ISYNC Parameter is used to control the type of parallel synchronization. The default value of semaphores should almost always be used instead of the optional spin-lock method.

The ISCOPE Parameter controls the priority and scheduling of the threads. FMS automatically sets this Parameter for best performance.

The remaining hardware Parameters MDPADD, LCACHE, LSTACK, ITPCPU and LENVRG control hardware architecture features and should be left at the default values.

NUMA Parameters

The following parameters only apply to machines having nonuniform memory access (NUMA). Currently this includes the SGI Origin series. These parameters may only be specified in the license file. These options are most useful when the machine is dedicated to the application using FMS.
NAME Function
NUMAFL Master flag. Must be set to 1 for any of the following parameters to take effect.
NUMAPR Print output level
MYCPU1 Starting processor to use. 1 is first processor.
NPNODE Number of processors per node
NUMNOD Number of nodes (read-only)
NUMAFX Thread binding to processor options.
MDWHEN When memory is allocated
MAXLMD Stride between nodes for placing memory (bytes)
NUMATP Topology (SGI only)
MMGLUE Combine blocks during matrix multiply
NUMAFL is the main flag that turns all the FMS NUMA features on or off. By default, NUMA features are turned off, providing an environment best suited for small jobs in a multi-user environment. When NUMAFL is set, threads and memory become bound to processors, providing explicit control of the machine environment. This feature is best used for large jobs on a dedicated machine, or part of a machine.

The NUMAPR Parameter controls the amount of printed output that occurs when the NUMA environment is being established. This includes information about thread and memory placement.

When NUMA effects are on, MYCPU1 specifies the starting CPU number for the job. This Parameter must be set explicitly if multiple jobs are being run concurrently.

Within FMS, a "NODE" is defined as a part of the NUMA machine having uniform memory access. For current NUMA machines, this is between two and four processors. NPNODE specifies the number of processors per node.

The NUMAFX Parameter determines how the threads are bound to the processors.

The MDWHEN Parameter determines when memory for the FMS memory pool is allocated from the system. For proper NUMA placement, memory is physically allocated from the system and properly placed as required by FMS or your application.

MAXLMD specifies the stride by which memory is allocated among the nodes in a round-robin fashon. If you are allocating memory from the FMS memory pool by the parent in your application, the memory will be allocated among the nodes using this stride. A zero value will override this feature and allocate all the memory on the node running the parent thread. When a memory request is made by a child thread, all memory is automatically allocated on the node running the child thread. This feature provides the control required for you to maximize local memory references in the non-FMS part of your application.

The remaining NUMA Parameters, NUMATP and MMGLUE, should be left at the default values selected by FMS.

File Parameters

NAME Function
INCORE File location (disk or memory)
IEXIST Use existing file
IFKEEP Keep files
IREST Write restart information
IOKIDS Open for access by children
NUMIOQ Number if I/O queues
NUMSTP Number of stripes per queue
NUMVOL Number of volumes per stripe
MBYTES Number of megabytes per volume
IOTYPE Perform raw or direct I/O
IOSYNC Perform synchronous I/O
LSECTR Disk sector length (8-byte words)
NOOPEN Do not physically open files
LBUFSI Buffer size for submatrix integer records
LBUFSR Buffer size for submatrix real records
LBUFSV Buffer size for submatrix vector records

There are several parameters which provide options for configuring the FMS Database.

You may direct FMS to place files in memory with the INCORE Parameter. This can provide improved performance for small problems.

By default, FMS treats files in the Database as scratch files that are deleted at the end of the job. You may set the IFKEEP Parameter before the file is closed to retain a file. Later, when you want to reopen the file, you can set the IEXIST Parameter to use an existing file, rather than to create a new one.

You may also direct FMS to write a restart file during matrix factoring, or to restart a job in the middle of matrix factoring using a previously written restart file. The FMS Parameter IREST controls this feature.

Normally FMS opens a file for access by the parent. If you plan to read or write an FMS file from threads you created by calling FMSPAR, you should set the IOKIDS Parameter.

The FMS File Striping System is controlled by the NUMIOQ, NUMSTP, NUMVOL and MBYTES Parameters. These Parameters specify the number of queues, stripes per queue, volumes per stripe and capacity of each volume. The FMS File Striping System can significantly improve I/O performance. You should review this option before completing your installation. Because the disk configuration on most systems may change, you should set these parameters in the License File.

On some Unix systems, FMS can bypass the buffer cache and perform I/O directly to the disks using raw or direct I/O. The FMS Parameter IOTYPE discusses these options.

By default, FMS performs transfers to the Database asynchronously. You may turn this feature off and perform transfers synchronously with the IOSYNC Parameter.

The LSECTR specifies the minimum number of 8-byte words which may be transfered to a physical disk dirve. Normally the default value selected by FMS will be correct. However if your file system contains multiple drives striped into a single mount point, the default value may be too low. FMS automatically detects this condition and provides instructions on how to change this Parameter to the correct value.

The NOOPEN Parameter skips the physical opening of files. This feature is used to estimate the disk space that will be required when the files are actually opened.

If you are using submatrices to define the matrix data, FMS will buffer the data as it is written by your application. The LBUFSI, LBUFSR and LBUFSV parameters control the size of the buffers used for the integer, real and vector files.

Data Formatting Input/Output Parameters

NAME Function
Print Code Summary
PRINT Set all print codes
IPRI Initialization print code
IPRA Assembly print code
IPRF Factoring print code
IPRS Solution print code
IPRMV Matrix-vectors multiply print code
IPRVV Vectors-vectors multiply print code
IPRDV Vectors-diagonal-vectors multiply print code
LOGTIM Accumulate statistics on FMS subroutine usage
NEQPR1 Lower limit on equation to print
NEQPR2 Upper limit on equation to print
NROWPG Number of rows per page
NCOLPG Number of columns per page
IFMAT Format for printing integers
RFMAT Format for printing real numbers
CFMAT Format for printing complex numbers
SHOW Display reports
RESET Reset FMS parameters to default values
LUINPT Input unit number
LUPR Output unit number
FMS provides complete control over printed output with the print Parameters. You may select the level of printing at any stage of your application with the IPRI, IPRA, IPRF, IPRS, IPRMV, IPRVV and IPRDV Parameters. If you want to set all these parameters to the same value, you may use PRINT. The Print Code Summary is a useful list of the options that are available with each parameter.

The default print level includes status and timings for each subroutine, which is appropriate for most production applications. However, during installation you may wish to modify these parameters to check intermediate results. The FMSSET utility is especially useful for setting these parameters during installation.

Normally FMS accumulates timing information for subroutines you call. The LOGTIM Parameter provides additional options for timing internal routines. This information may be useful for final tuning. Because the additional low level timings will increase run time, this Parameter should be left at the default value for production.

The amount of data in a matrix or vector may be very large. FMS provides the NEQPR1 and NEQPR2 parameters to control the window of data that is printed.

The NROWPG and NCOLPG control the size of your display. The default values of 24 by 80 is appropriate for most terminals.

The IFMAT, RFMAT and CFMAT control the format for INTEGER, REAL*8 and COMPLEX*16 data. You may adjust these Parameters to control the amount of information printed by each matrix and vector term.

The SHOW Parameter is useful for printing reports on FMS Parameters, subroutine calls, memory usage and the status of the FMS Database.

You may use the RESET Parameter to reset all parameter values to the default values, including any changes you made in the License File.

FMS prints output to file LUPR. FMSSET reads input from file LUINPT.

Matrix File Initialization Parameters

NAME Function
MFMAT Matrix format (PROFILE, BLOCK, SLAB)
LENSDF I/O record size (PROFILE format)
NEQBIO I/O block size (BLOCK & SLAB format)
NEQBLK Compute block size
NEQBLO Minimum compute block size
NEQBHI Maximum compute block size
NEQBV Vector block size
IJSTEP Stride toward diagonal
INSIDE Blocking pattern (PROFILE format)
When you open a matrix file, FMS stores the data in one of three formats: PROFILE, BLOCK or SLAB. You may use the MFMAT Parameter to override the default selection. SLAB format, which provides for full column partial pivoting, must always be selected by setting MFMAT. Because the best format depends only on your matrix, you should call FMSIST in your application to change this value.

The LENSDF and NEQBIO Parameters control the record size that is used for the matrix file. The default values of these parameters should provide the best performance.

The storage within each record is controlled by the NEQBLK, IJSTEP and INSIDE Parameters. The default values of these parameters are carefully selected to match optimized kernels within FMS. FMS selects the value of NEQBLK to be between NEQBLO and NEQBHI. You may experiment with these during final tuning, but the default values should provide the best performance.

Matrix Assembly Parameters

NAME Function
LOWASM First changed equation for nonlinear reassembly
MDATAU Call your subroutine to define or modify matrix data
LUOK Make a nonsymmetric matrix symmetric
The FMS Parameters also provide options for building the matrix. LOWASM may be used during nonlinear analysis to reuse the part of the matrix factors which have not changed. LUOK will convert a nonsymmetric matrix to symmetric. The MDATAU Parameter directs FMS to call subroutines you provide to define or modify the matrix data.

Matrix Factoring Parameters

NAME Function
MPOSDF Positive definite flag
MZERO Zero pivot flag
NBITSZ Number of lost bits on diagonal to call zero
NBITSP Number of lost bits on diagonal to print a warning
RPIVOT Value to set zero pivot reciprocal
PIVMIN Absolute value of minimum pivot
NPIVOT Number of equations pivoted (SLAB format)
Read-only
NUMSCG Number of diagonal sign changes during factoring
NEQSUB First equation of substructure
IALGOR Optional factoring algorithms
MINDIM Minimum dimension to terminate Strassen's algorithm
SAVOPS Number of floating point operations saved (read-only)
CONINV Inverse condition number (read-only)
ATERMS Number of terms in matrix (read-only)
AZEROS Number of zero's in matrix (read-only)
FZEROS Number of zero's in matrix factor (read-only)
IPOLLD CPU synchronization on diagonal elements

There are several options provided for handling pivots during the factoring process. MPOSDF instructs FMS to halt if a negative pivot is found. NBITSP controls warning messages for pivots "close" to zero. NBITSZ sets the tolerance for zero. PIVMIN also sets a limit on the minimum pivot value.

When FMS detects a bad pivot, it halts by default. You may use the MZERO Parameter to override this action. The options include setting the pivot reciprocal to RPIVOT and continuing or calling a subroutine you provide.

The NEQSUB Parameter offers a useful modification to the factoring and solving process for many applications.

For some applications, the algorithm used for matrix multiply and/or complex arithmetic may be altered to improve performance without loss of accuracy. The IALGOR, MINDIM and SAVOPS Parameters provide these options.

After the factoring process, you may call FMSIGT to obtain information about the matrix factors. NUMSCG provides the number of sign changes on the diagonal and NPIVOT provides the number of equations pivoted for SLAB matrices.

You may also call FMSRGT to find out information about how the matrix filled in during the factoring process. ATERMS lists the total number of terms in the matrix, AZEROS lists the number zero terms in the unfactored matrix and FZEROS lists the number of zero terms in the factored matrix.

During factoring the parallel processes must synchronize as they compute terms around the diagonal. The IPOLLD parameter selects between event flags and spin-locks for this synchronization. It is strongly recommended that you leave this parameter at the default value.

Solving Parameters

NAME Function
IPBACK Use dot products during back substitution (PROFILE format)
LUTRAN Solve [A]T{X}={B} instead of [A]{X}={B}

As an option, you may use the IPBACK Parameter to direct FMS to perform back substitution with inner products. For most cases, the default value is preferred.

The LUTRAN Parameter may be used to solve using the transpose of the matrix [A] for nonsymmetric problems. This feature has applications in evaluating derivatives of potential functions.

Multiply Parameters

NAME Function
MMROW Shifts the product of [A][B] by MMROW rows in [C].
MMCOL Shifts the product of [A][B] by MMCOL columns in [C].
MMKA Adds MMKA columns of 0's to the beginning of matrix [A].

When using FMSMM to compute matrix or vector multiply operations, the resulting product is normally stored in the upper-left corner of the output matrix. The FMS Parameters MMROW, MMCOL and MMKA provide a way of shifting where the product is stored.

Error Handling Parameters

NAME Function
NERROR FMS error number
Read-only
IOSTAT I/O status returned from last system call
Read-only
LUERR Unit number for writing error messages
IPRERR Information to print if an error occurs
IERROR Action to take if a fatal error is detected

FMS contains an error monitoring system that provides useful information whenever an error occurs. This information is printed on unit LUERR. The amount of information is controlled by IPRERR. The FMS error number is provided by NERROR. If the error was due to a system service, the system error is available in IOSTAT.

The FMS error reporting system is designed to provide the information you need without modification. If, however, you want to process your own error conditions, you may set the IERROR Parameter and check the status in NERROR after each FMS call.

This completes the quick tour of FMS Parameters. A detail description of each Parameter is listed in Appendix D.