Multi-Category Security (MCS) Guide

Overview

The MCS Plugin is meant to extend the current Slurm functionality related to job node exclusivity and job/node information display privacy.

Slurm's OverSubscribe option controls the ability of a partition to execute more than one job at a time on each resource, no matter "what type" of job. Slurm job submission clients can also use the --exclusive and --oversubscribe parameters to request how the job can be shared. The ExclusiveUser slurm.conf parameter and the --exclusive=user client parameter value modifies the exclusivity functionality. With this parameter enabled, the "type of job" now matters when considering exclusivity, so jobs can share resources based on job users, meaning that only jobs whose user is the same can share resources. This indeed adds a new dimension to how Slurm manages exclusivity. With the introduction of the MCS Plugin, Slurm can now be configured to associate an MCS_label to jobs and optionally ensure that nodes can only be shared among jobs having the same label. This adds even more degrees of freedom to how Slurm manages exclusivity, providing end users with much more flexibility in this area.

Slurm also has the PrivateData slurm.conf parameter, which is used to control what type of information is hidden from regular users. Similar to the exclusivity property, the MCS Plugin also extends the privacy by filtering job and/or node information based on the user's access to their MCS_label. This means that privacy is now less restrictive, and information is not just hidden or not to regular users, but now it is filtered depending on these configurable/requestable labels in coordination with the PrivateData option.

Limitations

Use of MCS restricts the operation of Preemption. Specifically, any job that has an MCS label and requests node exclusivity according to that label will be prevented from preempting or being preempted by any job that doesn't match that label. If MCSParameters=enforced,select is set, these restrictions will apply to all jobs.

Configuration

Two parameters are currently available to configure MCS: MCSPlugin and MCSParameters.

  • MCSPlugin Specifies which plugin should be used. Plugins are mutually exclusive, and the type of label to be associated depends on the loaded plugin.

    • mcs/none is the default and disables MCS labels and functionality.

    • mcs/account MCS labels can only take a value equal to the job's --account. NOTE: this option requires enabling of accounting.

    • mcs/group MCS labels can only take a value equal to the job's user group.

    • mcs/user MCS labels can only take a value equal to the username of the job's --uid.

MCS_labels on jobs can be displayed through 'squeue' with the format option mcslabel or through 'scontrol show job'. Nodes also acquire a MCS_label, which is inherited from the allocated job's MCS_label. The node's label can be displayed with 'scontrol show nodes'. The 'sview' command can also be used to see these MCS_labels.

Users may either request a particular category label for a job (through the --mcs-label option), or use the default value generated by the specific MCS plugin implementation. So labels can be configured to be enforced or set on demand, and the specific MCS Plugin is responsible for checking the validity of these labels. When enforced, the MCS Plugin implementation will always associate a MCS label to a submitted job, either the default value or the one requested by the user (if it's considered correct).

The selection (exclusivity) of nodes can be filtered on MCS labels either on demand (ondemand) or always (select) or never (noselect). Users can force the filter with the --exclusive=mcs option (except if noselect mode is enabled).

Label enforcement, node selection filtering policy, private data based on labels and a list of user groups allowed to be mapped to MCS labels can be configured through MCSParameters option.

  • MCSParameters Specifies the options to pass to the specific MCS Plugin implementation. Options should satisfy the following expression:
    "[ondemand|enforced][,noselect|select|ondemandselect][,privatedata]:[mcs_plugin_parameters]". The defaults are "ondemand,ondemandselect" and no privatedata.

    • ondemand|enforced set MCS label on jobs either on demand (using --mcs-label option) or always
    • noselect|select|ondemandselect select nodes with filter on MCS label: never, always or on demand (using --exclusive=mcs option)
    • privatedata accordingly with PrivateData specific option:
      • If configured with privatedata and PrivateData=jobs, job information is filtered based on their MCS labels.
      • If configured with privatedata and PrivateData=nodes, node information is filtered based on their MCS labels.
    • mcs_plugin_parameters Only mcs/group is currently supporting this option. It can be used to specify the list of user groups (separated by the '|' character) that are allowed to be mapped to MCS labels by the mcs/group plugin.

When enforcing privatedata with mcs/account, users will need to have an association with just the relevant Account to see other jobs in that Account. If a Partition is included in the association, it will be over-constrained and will not allow them to view other jobs in the Account.

Different requests and configurations lead to different combinations of use-cases. The following table is intended to help understand the end user the expected behavior (related to exclusivity) for a subset of these use-cases:

Node filtering: Label enforcement: ondemand
(MCS_label set only if requested.) 		Label enforcement: enforced
(MCS_label is mandatory.)
noselect No filter on nodes even if --exclusive=mcs requested. No filter on nodes even if --exclusive=mcs requested.
select Filter on nodes only if job MCS_label is set. Always filter on nodes.
ondemandselect Filter on nodes only if --exclusive=mcs. Filter on nodes only if --exclusive=mcs.

Some examples

slurm.conf: 

MCSPlugin=mcs/account
MCSParameters=enforced,select,privatedata
or 

MCSPlugin=mcs/group
MCSParameters=ondemand,noselect:groupA|groupB|groupC
or 

MCSPlugin=mcs/user
MCSParameters=enforced,select,privatedata

View MCS parameters

Configured MCS parameters can be viewed using the scontrol command. 

scontrol show config |grep MCS
MCSPlugin          = mcs/group
MCSParameters      = ondemand,noselect:groupA|groupB|groupC

Put an mcs_label on a job using salloc, sbatch or srun 

srun -n10 -t 1000 --mcs-label=groupB ./job &

Put an mcs_label on a job with exclusivity 

srun -n10 -t 1000 --mcs-label=groupB --exclusive=mcs ./job &

Put a different mcs_label on a job with the mcs/account plugin 

srun -n10 -t 1000 -A another_account_than_default ./job &

View MCS parameters in usage

The squeue command can also display the MCS with mcslabel format option 

squeue -O jobid,username,mcslabel
JOBID               USER                MCSLABEL
2                   foo                 groupA
3	            bar                 groupB

The scontrol command will now display the MCS_label 

scontrol show nodes
NodeName=node0001 Arch=x86_64 CoresPerSocket=4
   CPUAlloc=0 CPUTot=8 CPULoad=0.01 Features=(null)
   Gres=(null)
   NodeAddr=noder0001 NodeHostName=node0001 Version=15.08
   OS=Linux RealMemory=23780 AllocMem=0 Sockets=2 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=40 Owner=N/A MCS_label=groupA
   BootTime=2015-08-05T10:14:41 SlurmdStartTime=2015-09-07T13:42:28
   CurrentWatts=0 AveWatts=0

Last modified 21 May 2024