pam_slurm_adopt
The purpose of this module is to prevent users from sshing into nodes that they do not have a running job on, and to track the ssh connection and any other spawned processes for accounting and to ensure complete job cleanup when the job is completed. This module does this by determining the job which originated the ssh connection. The user's connection is "adopted" into the "external" step of the job. When access is denied, the user will receive a relevant error message.
Contents
- Installation
- Slurm Configuration
- SSH Configuration
- PAM Configuration
- Administrative Access Configuration
- pam_slurm_adopt Module Options
- Firewalls, IP Addresses, etc.
- SELinux
- Limitations
Installation
Source:
In your Slurm source directory, navigate to ./contribs/pam_slurm_adopt/ and run
make && make install
as root. This will place pam_slurm_adopt.a, pam_slurm_adopt.la, and pam_slurm_adopt.so in /lib/security/ (on Debian systems) or /lib64/security/ (on RedHat/SuSE systems).
RPM:
The included slurm.spec will build a slurm-pam_slurm RPM which will install pam_slurm_adopt. Refer to the Quick Start Administrator Guide for instructions on managing an RPM-based install.
DEB:
The included debian packaging scripts will build the slurm-smd-libpam-slurm-adopt package which will install pam_slurm_adopt. Quick Start Administrator Guide for instructions on managing an DEB-based install.
Slurm Configuration
PrologFlags=contain must be set in the slurm.conf. This sets up the "extern" step into which ssh-launched processes will be adopted. You must also enable the task/cgroup plugin in slurm.conf. See the Slurm cgroups guide. CAUTION This option must be in place before using this module. The module bases its checks on local steps that have already been launched. Jobs launched without this option do not have an extern step, so pam_slurm_adopt will not have access to those jobs.
LaunchParameters=ulimit_pam_adopt will set RLIMIT_RSS in processes adopted by the external step, similar to tasks running in regular steps.
The UsePAM option in slurm.conf is not related to pam_slurm_adopt.
SSH Configuration
Verify that UsePAM is set to On in /etc/ssh/sshd_config (it should be on by default).
PAM Configuration
Add the following line to the appropriate file in /etc/pam.d, such as system-auth or sshd (you may use either the "required" or "sufficient" PAM control flag):
account required pam_slurm_adopt.so
The order of plugins is very important. pam_slurm_adopt.so should be the last PAM module in the account stack. Included files such as common-account should normally be included before pam_slurm_adopt. You might have the following account stack in sshd:
account required pam_nologin.so account include password-auth ... -account required pam_slurm_adopt.so
Note the "-" before the account entry for pam_slurm_adopt. It allows PAM to fail gracefully if the pam_slurm_adopt.so file is not found. If Slurm is on a shared filesystem, such as NFS, then this is suggested to avoid being locked out of a node while the shared filesystem is mounting or down.
pam_slurm_adopt must be used with the task/cgroup task plugin and the proctrack/cgroup proctrack plugin. The pam_systemd module will conflict with pam_slurm_adopt, so you need to disable it in all files that are included in sshd or system-auth (e.g. password-auth, common-session, etc.).
If you need the user management features from pam_systemd, such as handling user runtime directory /run/user/$UID, you can have the prolog script run 'loginctl enable-linger $SLURM_JOB_USER' and the epilog script disable it again (after making sure there are no other jobs from this user on the node) by running 'loginctl disable-linger $SLURM_JOB_USER'. You will also need to export the XDG_* environment variables if your software requires them. You can see an example of prolog and epilog scripts here:
Prolog:loginctl enable-linger $SLURM_JOB_USER exit 0TaskProlog:
echo "export XDG_RUNTIME_DIR=/run/user/$SLURM_JOB_UID" echo "export XDG_SESSION_ID=$(</proc/self/sessionid)" echo "export XDG_SESSION_TYPE=tty" echo "export XDG_SESSION_CLASS=user"Epilog:
#Only disable linger if this is the last job running for this user.
O_P=0
for pid in $(scontrol listpids | awk -v jid=$SLURM_JOB_ID 'NR!=1 { if ($2 != jid && $1 != "-1"){print $1} }'); do
ps --noheader -o euser p $pid | grep -q $SLURM_JOB_USER && O_P=1
done
if [ $O_P -eq 0 ]; then
loginctl disable-linger $SLURM_JOB_USER
fi
exit 0
You must also make sure a different PAM module isn't short-circuiting the account stack before it gets to pam_slurm_adopt.so. From the example above, the following two lines have been commented out in the included password-auth file:
#account sufficient pam_localuser.so #-session optional pam_systemd.so
Note: This may involve editing a file that is auto-generated. Do not run the config script that generates the file or your changes will be erased.
Administrative Access Configuration
pam_slurm_adopt will always allow the root user access, and will do so even before checking for slurm configuration. If you wish to also allow other admins to the system with their own user accounts, this can be accomplished by stacking other modules along with pam_slurm_adopt.
Stacking pam_access with pam_slurm_adopt is one way to permit administrative access, with two possible implementations with subtle differences in behavor. Both will require editing the pam_access configuration file (/etc/security/access.conf). In the following example, the access.conf file will allow members of the group "wheel" to log in.
+:(wheel):ALL -:ALL:ALL
Then you will need to stack the modules in the /etc/pam.d/sshd file. The order here matters, and each ordering has different implications. In the example below, pam_slurm adopt is listed fist as "sufficient", followed by pam_access. In this configuration when the admin has a job running, their ssh session will be adopted into the job. If they do not, access will be permitted by pam_access, but note that pam_slurm_adopt will still emit the "access denied" message.
account sufficient pam_slurm_adopt.so account required pam_access.so
Flipping this order, with pam_access(sufficient) before pam_slurm_adopt(reqired), members of the administrative group will bypass pam_slurm_adopt entierly.
account sufficient pam_access.so account required pam_slurm_adopt.so
The pam_listfile module is another module that can be stacked with pam_slurm_adopt and achieve similar results. In the following example, it will allow all users in the specified file to log in, skipping the pam_slurm_adopt module. This can also be flipped similar to pam_access, with the same implications.
account sufficient pam_listfile.so item=user sense=allow onerr=fail file=/path/to/allowed_users_file account required pam_slurm_adopt.so
More information about the capabilities and configuration options for pam_access and pam_listfile can be found in their respecive man pages.
pam_slurm_adopt Module Options
This module is configurable. Add these options to the end of the pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd or system-auth):
account sufficient pam_slurm_adopt.so optionname=optionvalue
This module has the following options:
- action_no_jobs
- The action to perform if the user has no jobs on the node. Configurable values are:
- action_unknown
-
The action to perform when the user has multiple jobs on the node and
the RPC does not locate the source job. If the RPC mechanism works properly in
your environment, this option will likely be relevant only when
connecting from a login node. Configurable values are:
-
- newest (default)
- On systems with cgroup/v1 pick the newest job on the node. The "newest" job is chosen based on the mtime of the job's step_extern cgroup; asking Slurm would require an RPC to the controller. Thus, the memory cgroup must be in use so that the code can check mtimes of cgroup directories. The user can ssh in but may be adopted into a job that exits earlier than the job they intended to check on. The ssh connection will at least be subject to appropriate limits and the user can be informed of better ways to accomplish their objectives if this becomes a problem. NOTE: If the module fails to retrieve the cgroup mtime, then the picked job may not be the newest one. On systems with cgroup/v2 the newest is just the job with the greatest id, and thus this does not ensure that it is really the newest job.
- allow
- Let the connection through without adoption.
- deny
- Deny the connection.
-
- action_adopt_failure
- The action to perform if the process is unable to be adopted into any job for whatever reason. If the process cannot be adopted into the job identified by the callerid RPC, it will fall through to the action_unknown code and try to adopt there. A failure at that point or if there is only one job will result in this action being taken. Configurable values are:
- action_generic_failure
- The action to perform if there are certain failures such as the inability to talk to the local slurmd or if the kernel doesn't offer the correct facilities. Configurable values are:
- disable_x11
- Turn off Slurm built-in X11 forwarding support. Configurable values are:
- join_container
- Control the interaction with the job_container/tmpfs plugin. Configurable values are:
- log_level
- See SlurmdDebug in slurm.conf for available options. The default log_level is info.
- nodename
- If the NodeName defined in slurm.conf is different than this node's hostname (as reported by hostname -s), then this must be set to the NodeName in slurm.conf that this host operates as.
- service
- The pam service name for which this module should run. By default it only runs for sshd for which it was designed for. A different service name can be specified like "login" or "*" to allow the module to in any service context. For local pam logins this module could cause unexpected behavior or even security issues. Therefore if the service name does not match then this module will not perform the adoption logic and returns PAM_IGNORE immediately.
Firewalls, IP Addresses, etc.
slurmd should be accessible on any IP address from which a user might launch ssh. The RPC to determine the source job must be able to reach the slurmd port on that particular IP address. If there is no slurmd on the source node, such as on a login node, it is better to have the RPC be rejected rather than silently dropped. This will allow better responsiveness to the RPC initiator.
SELinux
SELinux may conflict with pam_slurm_adopt, but it is generally possible for them to work side by side. This is an example type enforcement file that was used on a fairly stock Debian system. It is provided to give some direction and to show what is required to get this working but may require additional modification.
module pam_slurm_adopt 1.0;
require {
type sshd_t;
type var_spool_t;
type unconfined_t;
type initrc_var_run_t;
class sock_file write;
class dir { read search };
class unix_stream_socket connectto;
}
#============= sshd_t ==============
allow sshd_t initrc_var_run_t:dir search;
allow sshd_t initrc_var_run_t:sock_file write;
allow sshd_t unconfined_t:unix_stream_socket connectto;
allow sshd_t var_spool_t:dir read;
allow sshd_t var_spool_t:sock_file write;
It is possible for some plugins to require more permissions than this. Notably, job_container/tmpfs will require something more like this:
module pam_slurm_adopt 1.0;
require {
type nsfs_t;
type var_spool_t;
type initrc_var_run_t;
type unconfined_t;
type sshd_t;
class sock_file write;
class dir { read search };
class unix_stream_socket connectto;
class fd use;
class file read;
class capability sys_admin;
}
#============= sshd_t ==============
allow sshd_t initrc_var_run_t:dir search;
allow sshd_t initrc_var_run_t:sock_file write;
allow sshd_t nsfs_t:file read;
allow sshd_t unconfined_t:fd use;
allow sshd_t unconfined_t:unix_stream_socket connectto;
allow sshd_t var_spool_t:dir read;
allow sshd_t var_spool_t:sock_file write;
allow sshd_t self:capability sys_admin;
Limitations
Alternate authentication methods such as multi-factor authentication may break process adoption with pam_slurm_adopt.
When using SELinux support in Slurm, the session started via pam_slurm_adopt won't necessarily be in the same context as the job it is associated with.
Last modified 22 March 2024