Changelog for perl-slurm-23.11.5-1.2.x86_64.rpm :

* Mon Mar 25 2024 Christian Goll - removed Keep-logs-of-skipped-test-when-running-test-cases-sequentially.patch as incoperated upstream
* Changes in Slurm 23.02.5
* Add the JobId to debug() messages indicating when cpus_per_task/mem_per_cpu or pn_min_cpus are being automatically adjusted.
* Fix regression in 23.02.2 that caused slurmctld -R to crash on startup if a node features plugin is configured.
* Fix and prevent reoccurring reservations from overlapping.
* job_container/tmpfs - Avoid attempts to share BasePath between nodes.
* Change the log message warning for rate limited users from verbose to info.
* With CR_Cpu_Memory, fix node selection for jobs that request gres and
* -mem-per-cpu.
* Fix a regression from 22.05.7 in which some jobs were allocated too few nodes, thus overcommitting cpus to some tasks.
* Fix a job being stuck in the completing state if the job ends while the primary controller is down or unresponsive and the backup controller has not yet taken over.
* Fix slurmctld segfault when a node registers with a configured CpuSpecList while slurmctld configuration has the node without CpuSpecList.
* Fix cloud nodes getting stuck in POWERED_DOWN+NO_RESPOND state after not registering by ResumeTimeout.
* slurmstepd - Avoid cleanup of config.json-less containers spooldir getting skipped.
* slurmstepd - Cleanup per task generated environment for containers in spooldir.
* Fix scontrol segfault when \'completing\' command requested repeatedly in interactive mode.
* Properly handle a race condition between bind() and listen() calls in the network stack when running with SrunPortRange set.
* Federation - Fix revoked jobs being returned regardless of the -a/--all option for privileged users.
* Federation - Fix canceling pending federated jobs from non-origin clusters which could leave federated jobs orphaned from the origin cluster.
* Fix sinfo segfault when printing multiple clusters with --noheader option.
* Federation - fix clusters not syncing if clusters are added to a federation before they have registered with the dbd.
* Change pmi2 plugin to honor the SrunPortRange option. This matches the new behavior of the pmix plugin in 23.02.0. Note that neither of these plugins makes use of the \"MpiParams=ports=\" option, and previously were only limited by the systems ephemeral port range.
* node_features/helpers - Fix node selection for jobs requesting changeable features with the \'|\' operator, which could prevent jobs from running on some valid nodes.
* node_features/helpers - Fix inconsistent handling of \'&\' and \'|\', where an AND\'d feature was sometimes AND\'d to all sets of features instead of just the current set. E.g. \"foo|bar&baz\" was interpreted as {foo,baz} or {bar,baz} instead of how it is documented: \"{foo} or {bar,baz}\".
* Fix job accounting so that when a job is requeued its allocated node count is cleared. After the requeue, sacct will correctly show that the job has 0 AllocNodes while it is pending or if it is canceled before restarting.
* sacct - AllocCPUS now correctly shows 0 if a job has not yet received an allocation or if the job was canceled before getting one.
* Fix intel oneapi autodetect: detect the /dev/dri/renderD[0-9]+ gpus, and do not detect /dev/dri/card[0
*9]+.
* Format batch, extern, interactive, and pending step ids into strings that are human readable.
* Fix node selection for jobs that request --gpus and a number of tasks fewer than gpus, which resulted in incorrectly rejecting these jobs.
* Remove MYSQL_OPT_RECONNECT completely.
* Fix cloud nodes in POWERING_UP state disappearing (getting set to FUTURE) when an `scontrol reconfigure` happens.
* openapi/dbv0.0.39 - Avoid assert / segfault on missing coordinators list.
* slurmrestd - Correct memory leak while parsing OpenAPI specification templates with server overrides.
* slurmrestd - Reduce memory usage when printing out job CPU frequency.
* Fix overwriting user node reason with system message.
* Remove --uid / --gid options from salloc and srun commands.
* Prevent deadlock when rpc_queue is enabled.
* slurmrestd - Correct OpenAPI specification generation bug where fields with overlapping parent paths would not get generated.
* Fix memory leak as a result of a partition info query.
* Fix memory leak as a result of a job info query.
* slurmrestd - For \'GET /slurm/v0.0.39/node[s]\', change format of node\'s energy field \"current_watts\" to a dictionary to account for unset value instead of dumping 4294967294.
* slurmrestd - For \'GET /slurm/v0.0.39/qos\', change format of QOS\'s field \"priority\" to a dictionary to account for unset value instead of dumping 4294967294.
* slurmrestd - For \'GET /slurm/v0.0.39/job[s]\', the \'return code\' code field in v0.0.39_job_exit_code will be set to
*127 instead of being left unset where job does not have a relevant return code.
* data_parser/v0.0.39 - Add required/memory_per_cpu and required/memory_per_node to `sacct
*-json` and `sacct --yaml` and \'GET /slurmdb/v0.0.39/jobs\' from slurmrestd.
* For step allocations, fix --gres=none sometimes not ignoring gres from the job.
* Fix --exclusive jobs incorrectly gang-scheduling where they shouldn\'t.
* Fix allocations with CR_SOCKET, gres not assigned to a specific socket, and block core distribion potentially allocating more sockets than required.
* gpu/oneapi - Store cores correctly so CPU affinity is tracked.
* Revert a change in 23.02.3 where Slurm would kill a script\'s process group as soon as the script ended instead of waiting as long as any process in that process group held the stdout/stderr file descriptors open. That change broke some scripts that relied on the previous behavior. Setting time limits for scripts (such as PrologEpilogTimeout) is strongly encouraged to avoid Slurm waiting indefinitely for scripts to finish.
* Allow slurmdbd -R to work if the root assoc id is not 1.
* Fix slurmdbd -R not returning an error under certain conditions.
* slurmdbd - Avoid potential NULL pointer dereference in the mysql plugin.
* Revert a change in 23.02 where SLURM_NTASKS was no longer set in the job\'s environment when
*-ntasks-per-node was requested.
* Limit periodic node registrations to 50 instead of the full TreeWidth. Since unresolvable cloud/dynamic nodes must disable fanout by setting TreeWidth to a large number, this would cause all nodes to register at once.
* Fix regression in 23.02.3 which broken x11 forwarding for hosts when MUNGE sends a localhost address in the encode host field. This is caused when the node hostname is mapped to 127.0.0.1 (or similar) in /etc/hosts.
* openapi/[db]v0.0.39 - fix memory leak on parsing error.
* data_parser/v0.0.39 - fix updating qos for associations.
* openapi/dbv0.0.39 - fix updating values for associations with null users.
* Fix minor memory leak with --tres-per-task and licenses.
* Fix cyclic socket cpu distribution for tasks in a step where - -cpus-per-task < usable threads per core.- Changes in Slurm 23.02.4
* Fix sbatch return code when
*
*wait is requested on a job array.
* switch/hpe_slingshot
* avoid segfault when running with old libcxi.
* Avoid slurmctld segfault when specifying AccountingStorageExternalHost.
* Fix collected GPUUtilization values for acct_gather_profile plugins.
* Fix slurmrestd handling of job hold/release operations.
* Make spank S_JOB_ARGV item value hold the requested command argv instead of the srun
*
*bcast value when
*
*bcast requested (only in local context).
* Fix step running indefinitely when slurmctld takes more than MessageTimeout to respond. Now, slurmctld will cancel the step when detected, preventing following steps from getting stuck waiting for resources to be released.
* Fix regression to make job_desc.min_cpus accurate again in job_submit when requesting a job with
*
*ntasks
*per
*node.
* scontrol
* Permit changes to StdErr and StdIn for pending jobs.
* scontrol
* Reset std{err,in,out} when set to empty string.
* slurmrestd
* mark environment as a required field for job submission descriptions.
* slurmrestd
* avoid dumping null in OpenAPI schema required fields.
* data_parser/v0.0.39
* avoid rejecting valid memory_per_node formatted as dictionary provided with a job description.
* data_parser/v0.0.39
* avoid rejecting valid memory_per_cpu formatted as dictionary provided with a job description.
* slurmrestd
* Return HTTP error code 404 when job query fails.
* slurmrestd
* Add return schema to error response to job and license query.
* Fix handling of ArrayTaskThrottle in backfill.
* Fix regression in 23.02.2 when checking gres state on slurmctld startup or reconfigure. Gres changes in the configuration were not updated on slurmctld startup. On startup or reconfigure, these messages were present in the log: \"error: Attempt to change gres/gpu Count\".
* Fix potential double count of gres when dealing with limits.
* switch/hpe_slingshot
* support alternate traffic class names with \"TC_\" prefix.
* scrontab
* Fix cutting off the final character of quoted variables.
* Fix slurmstepd segfault when ContainerPath is not set in oci.conf
* Change the log message warning for rate limited users from debug to verbose.
* Fixed an issue where jobs requesting licenses were incorrectly rejected.
* smail
* Fix issues where e
*mails at job completion were not being sent.
* scontrol/slurmctld
* fix comma parsing when updating a reservation\'s nodes.
* cgroup/v2
* Avoid capturing log output for ebpf when constraining devices, as this can lead to inadvertent failure if the log buffer is too small.
* Fix
*
*gpu
*bind=single binding tasks to wrong gpus, leading to some gpus having more tasks than they should and other gpus being unused.
* Fix main scheduler loop not starting after failover to backup controller.
* Added error message when attempting to use sattach on batch or extern steps.
* Fix regression in 23.02 that causes slurmstepd to crash when srun requests more than TreeWidth nodes in a step and uses the pmi2 or pmix plugin.
* Reject job ArrayTaskThrottle update requests from unprivileged users.
* data_parser/v0.0.39
* populate description fields of property objects in generated OpenAPI specifications where defined.
* slurmstepd
* Avoid segfault caused by ContainerPath not being terminated by \'/\' in oci.conf.
* data_parser/v0.0.39
* Change v0.0.39_job_info response to tag exit_code field as being complex instead of only an unsigned integer.
* job_container/tmpfs
* Fix %h and %n substitution in BasePath where %h was substituted as the NodeName instead of the hostname, and %n was substituted as an empty string.
* Fix regression where
*
*cpu
*bind=verbose would override TaskPluginParam.
* scancel
* Fix
*
*clusters/
*M for federations. Only filtered jobs (e.g.
*A,
* u,
*p, etc.) from the specified clusters will be canceled, rather than all jobs in the federation. Specific jobids will still be routed to the origin cluster for cancellation.
* Mon Jan 29 2024 Egbert Eich - Update to version 23.11.03
* slurmrestd - Reject single http query with multiple path requests.
* Fix launching Singularity v4.x containers with `srun --container` by setting .process.terminal to true in generated `config.json` when step has pseudoterminal (`--pty`) requested.
* Fix loading in `dyanmic/cloud` node jobs after `net_cred` expired.
* Fix cgroup null path error on `slurmd/slurmstepd` tear down.
* `data_parser/v0.0.40` - Prevent failure if accounting is disabled, instead issue a warning if needed data from the database can not be retrieved.
* `openapi/slurmctld` - Prevent failure if accounting is disabled.
* Prevent `slurmscriptd` processing delays from blocking other threads in `slurmctld` while trying to launch various scripts. This is additional work for a fix in 23.02.6.
* Fix memory leak when receiving alias addrs from controller.
* `scontrol` - Accept `scontrol token lifespan=infinite` to create tokens that effectively do not expire.
* Avoid errors when Slurmdb accounting disabled when `--json` or `--yaml` is invoked with CLI commands and `slurmrestd`. Add warnings when query would have populated data from Slurmdb instead of errors.
* Fix `slurmctld` memory leak when running job with `--tres-per-task=gres:shard:#`
* Fix backfill trying to start jobs outside of backfill window.
* Fix oversubscription on partitions with `PreemptMode=OFF`.
* Preserve node reason on power up if the node is downed or drained.
* `data_parser/v0.0.40` - Avoid aborting when invoking a not implemented parser.
* `data_parser/v0.0.40` - Fix how nice values are parsed for job submissions.
* `data_parser/v0.0.40` - Fix regression where parsing error did not result in invalid request being rejected.
* Fix segfault in front-end node registration.
* Prevent jobs using none typed gpus from being killed by the controller after a reconfig or restart.
* Fix deadlock situation in the dbd when adding associations.
* Update default values of text/blob columns when updating from old mysql versions in more situations. This improves a previous fix to handle an uncommon case when upgrading mysql/mariadb.
* Fix rpmbuild in openSUSE/SLES due to incorrect mariadb dependency.
* When upgrading the slurmdbd to 23.11, avoid generating a query to update the association table that is larger than `max_allowed_packet` which would result in an upgrade failure.
* Fix rare deadlock when a dynamic node registers at the same time that a once per minute background task occurs.
* `data_parser/v0.0.40` - Fix enumerated strings in OpenAPI specification not have type field specified.
* Improve `scontrol show job -d` information of used shared gres (`shard/mps`) topology.
* accounting_storage/mysql - Fix usage query to use new lineage column instead of lft/rgt.
* `slurmrestd` - Improve handling of missing parsers when content plugins expect parsers not loaded.
* `slurmrestd` - Correct parsing of StepIds when querying jobs.
* `slurmrestd` - Improve error from parsing failures of lists.
* `slurmrestd` - Improve parsing of singular values for lists.
* `accounting_storage/mysql` - Fix `PrivateData=User` when listing associations.
* Disable sorting of dynamic nodes to avoid issues when restarting with heterogenous jobs that cause jobs to abort on restart.
* Don\'t allow deletion of non-dynamic nodes.
* `accounting_storage/mysql` - Fix issue adding partition based associations.
* Respect non-\"slurm\" settings for `I_MPI_HYDRA_BOOTSTRAP` and `HYDRA_BOOTSTRAP` and avoid injecting the `--external-launcher` option which will cause `mpirun/mpiexec` to fail with an unexpected argument error.
* Fix bug where scontrol hold would change node count for jobs with implicitly defined node counts.
* `data_parser/v0.0.40` - Fix regression of support for \"hold\" in job description.
* Avoid sending KILL RPCs to unresolvable `POWERING_UP` and `POWERED_DOWN` nodes.
* `data_parser/v0.0.38` - Fix several potential NULL dereferences that could cause slurmrestd to crash.
* Add `--gres-flags=one-task-per-sharing`. Do not allow different tasks in to be allocated shared gres from the same sharing gres.
* Add `SelectTypeParameters=ENFORCE_BINDING_GRES` and `ONE_TASK_PER_SHARING_GRES`. This gives default behavior for a job\'s `--gres-flags`.
* Alter the networking code to try connecting to the backup controllers if the DNS lookup for the primary `SlurmctldHost` fails.
* Alter the name resolution to only log at `verbose()` in client commands on failures. This allows for HA setups where the DNS entries are withdrawn for some `SlurmctldHost` entries without flooding the user with errors.
* Prevent `slurmscriptd` PID leaks when running `slurmctld` in foreground mode.
* Open all `slurmctld` listening ports at startup, and persist throughout. This also changes the backup `slurmctld` process to open the `SlurmctldPort` range, instead of only the first.
* Fix backup `slurmctld` shutting down instead of resuming standby duty if it took control.
* Fix race condition that delayed the primary `slurmctld` resuming when taking control from a backup controller.
* `srun` - Ensure processed messages are meant for this job in case of a rapidly-reused TCP port.
* `srun` - Prevent step launch failure while waiting for step allocation if a stray message is received.
* Fix backup `slurmctld` to be able to respond to configless config file requests correctly.
* Fix `slurmctld` crashing when recovering from a failed reconfigure.
* Fix `slurmscriptd` operation after recovering from a failed reconfigure.- Make sure `-std=gnu99` is added to CFLAGS on SLE-12.- Use %%autopatch.
* Fri Jan 12 2024 Christian Goll - Update to 23.11.1 with following major improvements and fixing CVE-2023-49933, CVE-2023-49934, CVE-2023-49935, CVE-2023-49936 and CVE-2023-49937
* Substantially overhauled the SlurmDBD association management code. For clusters updated to 23.11, account and user additions or removals are significantly faster than in prior releases.
* Overhauled `scontrol reconfigure` to prevent configuration mistakes from disabling slurmctld and slurmd. Instead, an error will be returned, and the running configuration will persist. This does require updates to the systemd service files to use the `--systemd` option to `slurmctld` and `slurmd`.
* Added a new internal `auth/cred` plugin - `auth/slurm`. This builds off the prior `auth/jwt` model, and permits operation of the `slurmdbd` and `slurmctld` without access to full directory information with a suitable configuration.
* Added a new `--external-launcher` option to `srun`, which is automatically set by common MPI launcher implementations and ensures processes using those non-srun launchers have full access to all resources allocated on each node.
* Reworked the dynamic/cloud modes of operation to allow for \"fanout\" - where Slurm communication can be automatically offloaded to compute nodes for increased cluster scalability.
* Overhauled and extended the Reservation subsystem to allow for most of the same resource requirements as are placed on the job. Notably, this permits reservations to now reserve GRES directly.- Details of changes:
* Fix `scontrol update job=... TimeLimit+=/-=` when used with a raw JobId of job array element.
* Reject `TimeLimit` increment/decrement when called on job with `TimeLimit=UNLIMITED`.
* Fix issue with requesting a job with `
*licenses` as well as `
*tres-per-task=license`.
* `slurmctld` - Prevent segfault in `getopt_long()` with an invalid long option.
* slurmrestd - Added `/meta/slurm/cluster` field to responses.
* Adjust systemd service files to start daemons after `remote-fs.target`.
* Fix `task/cgroup` indexing tasks in cgroup plugins, which caused `jobacct/gather` to match the gathered stats with the wrong task id.
* `select/linear` - Fix regression in 23.11 in which jobs that requested `
*cpus-per-task` were rejected.
* `data_parser/v0.0.40` - Fix the parsing for `/slurmdb/v0.0.40/jobs` exit_code query parameter.
* If a job requests more shards which would allocate more than one sharing GRES (gpu) per node refuse it unless `SelectTypeparameters` has `MULTIPLE_SHARING_GRES_PJ`.
* Trigger fatal exit when Slurm API function is called before `slurm_init()` is called.
* `slurmd` - Fix issue with `scontrol reconfigure` when started with `-c`.
* `slurmrestd` - Job submissions that result in the following error codes will be considered as successfully submitted (with a warning), instead of returning an HTTP 500 error back: `ESLURM_NODES_BUSY`, `ESLURM_RESERVATION_BUSY`, `ESLURM_JOB_HELD`, `ESLURM_NODE_NOT_AVAIL`, `ESLURM_QOS_THRES`, `ESLURM_ACCOUNTING_POLICY`, `ESLURM_RESERVATION_NOT_USABLE`, `ESLURM_REQUESTED_PART_CONFIG_UNAVAILABLE`, `ESLURM_BURST_BUFFER_WAIT`, ESLURM_PARTITION_DOWN`, `ESLURM_LICENSES_UNAVAILABLE`.
* Fix a `slurmctld` fatal error when upgrading to 23.11 and changing from `select/cons_res` to `select/cons_tres` at the same time.
* `slurmctld` - Reject arbitrary distribution jobs that have a minimum node count that differs from the number of unique nodes in the hostlist.
* Prevent `slurmdbd` errors when updating reservations with names containing apostrophes.
* Prevent message extension attacks that could bypass the message hash. CVE-2023-49933.
* Prevent SQL injection attacks in `slurmdbd`. CVE-2023-49934.
* Prevent message hash bypass in slurmd which can allow an attacker to reuse root-level MUNGE tokens and escalate permissions. CVE-2023-49935.
* Prevent NULL pointer dereference on size_valp overflow. CVE-2023-49936.
* Prevent double-xfree() on error in `_unpack_node_reg_resp()`. CVE-2023-49937.
* For jobs that request `
*cpus-per-gpu`, ensure that the `
*cpus-per-gpu request` is honored on every node in the and not just for the job as a whole.
* Fix listing available `data_parser` plugins for json and yaml when giving no commands to `scontrol` or `sacctmgr`.
* `slurmctld` - Rework `scontrol reconfigure` to avoid race conditions that can result in stray jobs.
* `slurmctld` - Shave ~1 second off average reconfigure time by terminating internal processing threads faster.
* Skip running `slurmdbd -R` if the connected cluster is 23.11 or newer. This operation is no longer relevant for 23.11.
* Ensure `slurmscriptd` shuts down before `slurmctld` is stopped or reconfigured.
* Improve error handling and error messages in `slurmctld` to `slurmscriptd` communications. This includes avoiding potential deadlock in `slurmctld` if slurmscript dies unexpectedly.
* Do not hold batch jobs whose extra constraints cannot be immediately satisfied, and set the state reason to `Constraints` instead of `BadConstraints`.
* Fix verbose log message printing a hex number instead of a job id.
* Upgrade rate limit parameters message from debug to info.
* For `SchedulerParameters=extra_constraints`, prevent `slurmctld` segfault when starting a `slurmd` with `
*extra` for a node that did not previously set this. This also ensures the extra constraints model works off the current node state, not the prior state.
* Fix `
*tres-per-task` assertion.
* Fix a few issues when creating reservations.
* Add `SchedulerParameters=time_min_as_soft_limit` option.
* Remove `SLURM_WORKING_CLUSTER` env from batch and srun environments.
* `cli_filter/lua` - return nil for unset time options rather than the string `2982616-04:14:00` (which is the internal macro `NO_VAL` represented as time string).
* Remove \'none\' plugins for all but auth and cred. scontrol show config will report (null) now.
* Removed `select/cons_res`. Please update your configuration to `select/cons_tres`.
* `mpi/pmix` - When aborted with status 0, avoid marking job/step as failed.
* Fixed typo on `initialized` for the description of `ESLURM_PLUGIN_NOT_LOADED`.
* `cgroup.conf` - Removed deprecated parameters `AllowedKmemSpace`, `ConstrainKmemSpace`, `MaxKmemPercent`, and `MinKmemSpace`.
* `proctrack/cgroup` - Add `SignalChildrenProcesses=` option to `cgroup.conf`. This allows signals for cancelling, suspending, resuming, etc. to be sent to children processes in a `step/job` rather than just the parent.
* Add `PreemptParameters=suspend_grace_time` parameter to control amount of time between `SIGTSTP` and `SIGSTOP` signals when suspending jobs.
* `job_submit/throttle` - improve reset of submitted job counts per user in order to better honor `SchedulerParameters=jobs_per_user_per_hour=#`.
* Load the user environment into a private pid namespace to avoid user scripts leaving background processes on a node.
* `scontrol` show `assoc_mgr` will display Lineage instead of Lft for associations.
* Add `SlurmctldParameters=no_quick_restart` to avoid a new `slurmctld` taking over the old `slurmctld` by accident.
* Fix `--cpus-per-gpu` for step allocations, which was previously ignored for job steps. `
*cpus-per-gpu` implies `--exact`.
* Fix mutual exclusivity of `--cpus-per-gpu` and `--cpus-per-task`: fatal if both options are requested in the commandline or both are requested in the environment. If one option is requested in the command line, it will override the other option in the environment.
* `slurmrestd` - `openapi/dbv0.0.37` and `openapi/v0.0.37` plugins have been removed.
* `slurmrestd` - `openapi/dbv0.0.38` and `openapi/v0.0.38` plugins have been tagged as deprecated.
* `slurmrestd` - added auto population of `info/version` field.
* `sdiag` - add `--yaml` and `--json` arg support to specify data_parser plugin.
* `sacct` - add `--yaml` and `--json` arg support to specify `data_parser` plugin.
* `scontrol` - add `--yaml` and `--json` arg support to specify `data_parser` plugin.
* `sinfo` - add `--yaml` and `--json` arg support to specify `data_parser` plugin.
* `squeue` - add `--yaml` and `--json` arg support to specify `data_parser` plugin.
* Changed the default `SelectType` to `select/cons_tres` (from `select/linear`).
* Allow `SlurmUser`/`root` to use reservations without specific permissions.
* Fix sending step signals to nodes not allocated by the step.
* Remove `CgroupAutomount=` option from `cgroup.conf`.
* Add `TopologyRoute=RoutePart` to route communications based on partition node lists.
* Added ability for configless to push Prolog and Epilog scripts to `slurmd`s.
* Prolog and Epilog do not have to be fully qualified pathnames.
* Changed default value of `PriorityType` from `priority/basic` to `priority/multifactor`.
* `torque/mpiexec` - Propogate exit code from `launched` process.
* `slurmrestd` - Add new rlimits fields for job submission.
* Define SPANK options environment variables when `--export=[NIL|NONE]` is specified.
* `slurmrestd` - Numeric input fields provided with a null formatted value will now convert to zero (0) where it can be a valid value. This is expected to be only be notable with job submission against v0.0.38 versioned endpoints with job requests with fields provided with null values. These fields were already rejected by v0.0.39+ endpoints, unless `+complex` parser value is provided to v0.0.40+ endpoints.
* `slurmrestd` - Improve parsing of integers and floating point numbers when handling incoming user provided numeric fields. Fields that would have not rejected a number for a numeric field followed by other non-numeric characters will now get rejected. This is expected to be only be notable with job submission against v0.0.38 versioned endpoints with malformed job requests.
* Reject reservation update if it will result in previously submitted jobs losing access to the reservation.
* `data_parser/v0.0.40` - output partition state when dumping partitions.
* Allow for a shared suffix to be used with the hostlist format. E.g., `node[0001-0010]-int`.
* Replace `SRUN_CPUS_PER_TASK` with `SLURM_CPUS_PER_TASK` and get back the previous behavior before Slurm 22.05 since now we have the new external launcher step.
* `job_container/tmpfs` - Add `BasePath=none` option to disable plugin on node subsets when there is a global setting.
* Add QOS flag `Relative`. If set the QOS limits will be treated as percentages of a cluster/partition instead of absolutes.
* Remove `FIRST_CORES` flag from reservations.
* Add cloud instance id and instance type to node records. Can be viewed/updated with `scontrol`.
* `slurmd` - add `instance-id`, `instance-type`, and `extra` options to allow them to be set on startup.
* Add cloud instance accounting to database that can be viewed with `sacctmgr show instance`.
* `select/linear` - fix task launch failure that sometimes occurred when requesting `
*threads-per-core` or `--hint=nomultithread`. This also fixes memory calculation with one of these options and `
*mem-per-cpu`: Previously, memory = mem-per-cpu
* all cpus including unusable threads. Now, memory = mem-per-cpu
* only usuable threads. This behavior matches the documentation and select/cons_tres.
* `gpu/nvml` - Reduce chances of `NVML_ERROR_INSUFFICIENT_SIZE` error when getting gpu memory information.
* `slurmrestd` - Convert to generating `OperationIDs` based on path for all v0.0.40 tagged paths.
* `slurmrestd` - Reduce memory used while dumping a job\'s stdio paths.
* `slurmrestd` - Jobs queried from `data_parser/v0.0.40` from `slurmdb` will have `step/id` field given as a string to match CLI formatting instead of an object.
* `sacct` - Output in JSON or YAML output will will have the `step/id` field given as a string instead of an object.
* `scontrol`/`squeue` - Step output in JSON or YAML output will will have the `id` field given as a string instead of an object.
* `slurmrestd` - For `GET /slurmdb/v0.0.40/jobs` mimick default behavior for handling of job start and end times as `sacct` when one or both fields are not provided as a query parameter.
* `openapi/slurmctld` - Add `GET /slurm/v0.0.40/shares` endpoint to dump same output as `sshare`.
* `sshare` - add JSON/YAML support.
* `data_parser/v0.0.40` - Remove `required/memory` output in json. It is replaced by `required/memory_per_cpu` and `required/memory_per_node`.
* `slurmrestd` - Add numeric id to all association identifiers to allow unique identification where association has been deleted but is still referenced by accounting record.
* `slurmrestd` - Add accounting, id, and comment fields to association dumps.
* Use `memory.current` in cgroup/v2 instead of manually calculating RSS. This makes accounting consistent with OOM Killer.
* `sreport` - cluster Utilization `PlannedDown` field now includes the time that all nodes were in the `POWERED_DOWN` state instead of just cloud nodes.
* `scontrol` update partition now allows `Nodes+=` and `Nodes-=` to add/delete nodes from the existing partition node list. `Nodes=+host1,-host2` is also allowed.
* `sacctmgr` - add `--yaml` and `--json` arg support to specify `data_parser` plugin.
* `sacctmgr` can now modify QOS\'s RawUsage to zero or a positive value.
* `sdiag` - Added statistics on why the main and backfill schedulers have stopped evaluation on each scheduling cycle. the number of `RPC limit exceeded...` messages that are logged.
* Rename `sbcast --fanout` to `--treewidth`.
* Remove `SLURM_NODE_ALIASES` env variable.
* Enable fanout for dynamic and unaddresable cloud nodes.
* Fix how steps are dealloced in an allocation if the last step of an srun never completes due to a node failure.
* Remove redundant database indexes.
* Add database index to suspend table to speed up archive/purges.
* When requesting `--tres-per-task` alter incorrect request for TRES, it should be `TRESType/TRESName` not `TRESType:TRESName`.
* Make it so reservations can reserve GRES.
* `sbcast` - use the specified `--fanout` value on all hops in message forwarding; previously the specified fanout was only used on the first hop, and additional hops used `TreeWidth` in `slurm.conf`.
* `slurmrestd`- remove logger prefix from `-s/-a list` options outputs.
* `switch/hpe_slingshot` - Add support for collectives.
* Nodes with suspended jobs can now be displayed as `MIXED`.
* Fix inconsistent handling of using cli and/or environment options for `tres_per_task=cpu:#` and `cpus_per_gpu`.
* Requesting `--cpus-per-task` will now set `SLURM_TRES_PER_TASK=cpu:#` in the environment.
* For some tres related environment variables such as `SLURM_TRES_PER_TASK`, when `srun` requests a different value for that option, set these environment variables to the value requested by `srun`. Previously these environment variables were unchanged from the job allocation. This bug only affected the output environment variables, not the actual step resource allocation.
* `RoutePlugin=route/topology` has been replaced with `TopologyParam=RouteTree`.
* If `ThreadsPerCore` in `slurm.conf` is configured with less than the number of hardware threads, fix a bug where the task plugins used fewer cores instead of using fewer threads per core.
* Fix arbitrary distribution allowing it to be used with `salloc` and `sbatch` and fix how cpus are allocated to nodes.
* Allow nodes to reboot while node is drained or in a maintenance state.
* Allow `scontrol` reboot to use nodesets to filter nodes to reboot.
* Fix how the topology of typed gres gets updated.
* Changes to the Type option in gres.conf now can be applied with `scontrol` reconfig.
* Allow for jobs that request a newly configured gres type to be queued even when the needed `slurmd`s have not yet registered.
* Kill recovered jobs that require unconfigured gres types.
* If keepalives are configured, enable them on all persistent connections.
* Configless - Also send Includes from configuration files not parsed by the controller (i.e. from `plugstack.conf`).
* Add `gpu/nrt` plugin for nodes using Trainium/Inferentia devices.
* `data_parser/v0.0.40` - Add `START_RECEIVED` to job flags in dumped output.
* SPANK - Failures from most spank functions (not epilog or exit) will now cause the step to be marked as failed and the command (`srun`, `salloc`, `sbatch
*wait`) to return 1.
* Wed Jan 03 2024 Egbert Eich - Update to 23.02.6 to fix (CVE-2023-49933 - bsc#1218046, CVE-2023-49935 - bsc#1218049, CVE-2023-49936 - bsc#1218050, CVE-2023-49937 - bsc#1218051, CVE-2023-49938 - bsc#1218053)
* Security Fixes: + Add `JobAcctGatherParams=DisableGPUAcct` to disable gpu accounting. + `acct_gather_energy/ipmi` - Improve logging of DCMI issues. + `gpu/oneapi` - Add support for new env vars `ZE_FLAT_DEVICE_HIERARCHY` and `ZE_ENABLE_PCI_ID_DEVICE_ORDER`. + `data_parser/v0.0.39` - skip empty string when parsing QOS ids. + Remove error message from `assoc_mgr_update_assocs` when purposefully resetting the default QOS.
* Bug Fixes: + `libslurm_nss` - Avoid causing glibc to assert due to an unexpected return from slurm_nss due to an error during lookup. + Fix job requests with `--tres-per-task` sometimes resulting in bad allocations that cannot run subsequent job steps. + Fix issue with `slurmd` where `srun` fails to be warned when a node prolog script runs beyond `MsgTimeout` set in `slurm.conf`. + `gres/shard` - Fix plugin functions to have matching parameter orders. + `gpu/nvml` - Fix issue that resulted in the wrong MIG devices being constrained to a job + `gpu/nvml` - Fix linking issue with MIGs that prevented multiple MIGs being used in a single job for certain MIG configurations + Fix file descriptor leak in slurmd when using `acct_gather_energy/ipmi` with DCMI devices. + `sview` - avoid crash when job has a node list string > 49 characters. + Prevent `slurmctld` crash during reconfigure when packing job start messages. + Preserve reason uid on reconfig. + Update node reason with updated `INVAL` state reason if different from last registration. + `conmgr` - Avoid NULL dereference when using `auth/none`. + `data_parser/v0.0.39` - Fixed how deleted QOS and associations for jobs are dumped. + `burst_buffer/lua` - fix stage in counter not decrementing when a job is cancelled during stage in. This counter is used to enforce the limit of 128 scripts per stage. + `data_parser/v0.0.39` - Fix how the `INVALID` nodes state is dumped. + `data_parser/v0.0.39` - Fix parsing of flag arrays to allow muliple flags to be set. + Avoid leaking sockets when an x11 application is closed in an allocation. + Fix missing mutex unlock in group cache code which could cause slurmctld to freeze. + Fix scrontab monthly jobs possibly skipping a month if added near the end of the month. + Fix loading of the gpu account gather energy plugin. + Fix `slurmctld` segfault when reconfiguring after a job resize. + Fix crash in slurmstepd that can occur when launching tasks via mpi using the `pmi2` plugin and using the `route/topology` plugin. + Fix `qos doesn\'t exist` error message in `assoc_mgr_update_assocs` to print the attempted new default qos, rather than the current default qos. + `data_parser/v0.0.39` - Fix segfault when POSTing data with association usage.
* Other Changes and Improvements: + Prevent message extension attacks that could bypass the message hash. CVE-2023-49933. + Prevent message hash bypass in slurmd which can allow an attacker to reuse root-level MUNGE tokens and escalate permissions. CVE-2023-49935. + Prevent NULL pointer dereference on `size_valp` overflow. CVE-2023-49936. + Prevent double-xfree() on error in `_unpack_node_reg_resp()`. CVE-2023-49937. + Prevent modified `sbcast` RPCs from opening a file with the wrong group permissions. CVE-2023-49938.- Fix %do_obsoletes macro expansion to work with SLE-12.
* Thu Nov 30 2023 Egbert Eich - Add missing service file for slurmrestd (boo#1217711).
* Tue Nov 28 2023 Egbert Eich - Explicitly create an Obsoletes: entry for each package version that is obsoleted by the present version. These are all published versions of the last two major releases as well as all minor versions of the present release lower than the current one (bsc#1216869 2nd part). This prevents the current version to upgrade a old Slurm version for which no upgrade path exists.
* Mon Nov 20 2023 Egbert Eich - On SLE-12 exclude build for s390x.
* Mon Nov 06 2023 Egbert Eich - Add missing dependencies to slurm-config to plugins package. These should help to tie down the slurm version and help to avoid a package mix (bsc#1216869).
* Thu Oct 12 2023 Christian Goll - update to 23.02.6 to fix (CVE-2023-41914, bsc#1216207)
* Removed Fix-test-32.8.patch as fixed upstream
* Bug Fixes: + Fix `CpusPerTres=` not upgreadable with scontrol update + Fix unintentional gres removal when validating the gres job state. + Fix `--without-hpe-slingshot` configure option. + Fix cgroup v2 memory calculations when transparent huge pages are used. + Fix parsing of `sgather --timeout` option. + Fix regression from 22.05.0 that caused `srun --cpu-bind \"=verbose\"` and `\"=v\"` options give different CPU bind masks. + Fix \"_find_node_record: lookup failure for node\" error message appearing for all dynamic nodes during reconfigure. + Avoid segfault if loading serializer plugin fails. + `slurmrestd` - Correct OpenAPI format for `GET /slurm/v0.0.39/licenses`. + `slurmrestd` - Correct OpenAPI format for `GET /slurm/v0.0.39/job/{job_id}`. + `slurmrestd` - Change format to multiple fields in `GET /slurmdb/v0.0.39/assocations` and `GET /slurmdb/v0.0.39/qos` to handle infinite and unset states. + When a node fails in a job with `--no-kill`, preserve the extern step on the remaining nodes to avoid breaking features that rely on the extern step such as `pam_slurm_adopt`, `x11`, and `job_container/tmpfs`. + `auth/jwt` - Ignore `x5c` field in JWKS files. + `auth/jwt` - Treat \'alg\' field as optional in JWKS files. + Allow job_desc.selinux_context to be read from the job_submit.lua script. + Skip check in slurmstepd that causes a large number of errors in the munge log: \"Unauthorized credential for client UID=0 GID=0\". This error will still appear on `slurmd`/`slurmctld`/`slurmdbd` start up and is not a cause for concern. + `slurmctld` - Allow startup with zero partitions. + Fix some mig profile names in slurm not matching nvidia mig profiles. + Prevent `slurmscriptd` processing delays from blocking other threads in `slurmctld` while trying to launch `{Prolog|Epilog}Slurmctld`. + Fix sacct printing ReqMem field when memory doesn\'t exist in requested TRES. + Fix how heterogenous steps in an allocation with `CR_PACK_NODE` or `-mpack` are created. + Fix `slurmctld` crash from race condition within `job_submit_throttle` plugin. + Fix `--with-systemdsystemunitdir` when requesting a default location. + Fix not being able to cancel an array task by the jobid (i.e. not `_`) through scancel, job launch failure or prolog failure. + Fix cancelling the whole array job when the array task is the meta job and it fails job or prolog launch and is not requeable. Cancel only the specific task instead. + Fix regression in 21.08.2 where MailProg did not run for `mail-type=end` for jobs with non+zero exit codes. + Fix incorrect setting of memory.swap.max in cgroup/v2. + Fix `jobacctgather/cgroup` collection of disk/io, gpumem, gpuutil TRES values. + Fix -d singleton for heterogeneous jobs. + Downgrade info logs about a job meeting a \"maximum node limit\" in the select plugin to `DebugFlags=SelectType`. These info logs could spam the slurmctld log file under certain circumstances. + `prep/script` - Fix `[Srun|Task]` missing `SLURM_JOB_NODELIST`. + gres - Rebuild GRES core bitmap for nodes at startup. This fixes error: \"Core bitmaps size mismatch on node [HOSTNAME]\", which causes jobs to enter state \"Requested node configuration is not available\". + `slurmctd` - Allow startup with zero nodes. + Fix filesystem handling race conditions that could lead to an attacker taking control of an arbitrary file, or removing entire directories\' contents. CVE-2023-41914.
* Mon Sep 18 2023 Egbert Eich - Updated to version 23.02.5 with the following changes:
* Bug Fixes: + Revert a change in 23.02 where `SLURM_NTASKS` was no longer set in the job\'s environment when `--ntasks-per-node` was requested. The method that is is being set, however, is different and should be more accurate in more situations. + Change pmi2 plugin to honor the `SrunPortRange` option. This matches the new behavior of the pmix plugin in 23.02.0. Note that neither of these plugins makes use of the `MpiParams=ports=` option, and previously were only limited by the systems ephemeral port range. + Fix regression in 23.02.2 that caused slurmctld -R to crash on startup if a node features plugin is configured. + Fix and prevent reoccurring reservations from overlapping. + `job_container/tmpfs` - Avoid attempts to share BasePath between nodes. + With `CR_Cpu_Memory`, fix node selection for jobs that request gres and `--mem-per-cpu`. + Fix a regression from 22.05.7 in which some jobs were allocated too few nodes, thus overcommitting cpus to some tasks. + Fix a job being stuck in the completing state if the job ends while the primary controller is down or unresponsive and the backup controller has not yet taken over. + Fix `slurmctld` segfault when a node registers with a configured `CpuSpecList` while `slurmctld` configuration has the node without `CpuSpecList`. + Fix cloud nodes getting stuck in `POWERED_DOWN+NO_RESPOND` state after not registering by `ResumeTimeout`. + `slurmstepd` - Avoid cleanup of `config.json-less` containers spooldir getting skipped. + Fix scontrol segfault when \'completing\' command requested repeatedly in interactive mode. + Properly handle a race condition between `bind()` and `listen()` calls in the network stack when running with SrunPortRange set. + Federation - Fix revoked jobs being returned regardless of the `-a`/`--all` option for privileged users. + Federation - Fix canceling pending federated jobs from non-origin clusters which could leave federated jobs orphaned from the origin cluster. + Fix sinfo segfault when printing multiple clusters with `--noheader` option. + Federation - fix clusters not syncing if clusters are added to a federation before they have registered with the dbd. + `node_features/helpers` - Fix node selection for jobs requesting changeable. features with the `|` operator, which could prevent jobs from running on some valid nodes. + `node_features/helpers` - Fix inconsistent handling of `&` and `|`, where an AND\'d feature was sometimes AND\'d to all sets of features instead of just the current set. E.g. `foo|bar&baz` was interpreted as `{foo,baz}` or `{bar,baz}` instead of how it is documented: `{foo} or {bar,baz}`. + Fix job accounting so that when a job is requeued its allocated node count is cleared. After the requeue, sacct will correctly show that the job has 0 `AllocNodes` while it is pending or if it is canceled before restarting. + `sacct` - `AllocCPUS` now correctly shows 0 if a job has not yet received an allocation or if the job was canceled before getting one. + Fix intel OneAPI autodetect: detect the `/dev/dri/renderD[0-9]+` GPUs, and do not detect `/dev/dri/card[0-9]+`. + Fix node selection for jobs that request `--gpus` and a number of tasks fewer than GPUs, which resulted in incorrectly rejecting these jobs. + Remove `MYSQL_OPT_RECONNECT` completely. + Fix cloud nodes in `POWERING_UP` state disappearing (getting set to `FUTURE`) when an `scontrol reconfigure` happens. + `openapi/dbv0.0.39` - Avoid assert / segfault on missing coordinators list. + `slurmrestd` - Correct memory leak while parsing OpenAPI specification templates with server overrides. + Fix overwriting user node reason with system message. + Prevent deadlock when `rpc_queue` is enabled. + `slurmrestd` - Correct OpenAPI specification generation bug where fields with overlapping parent paths would not get generated. + Fix memory leak as a result of a partition info query. + Fix memory leak as a result of a job info query. + For step allocations, fix `--gres=none` sometimes not ignoring gres from the job. + Fix `--exclusive` jobs incorrectly gang-scheduling where they shouldn\'t. + Fix allocations with `CR_SOCKET`, gres not assigned to a specific socket, and block core distribion potentially allocating more sockets than required. + Revert a change in 23.02.3 where Slurm would kill a script\'s process group as soon as the script ended instead of waiting as long as any process in that process group held the stdout/stderr file descriptors open. That change broke some scripts that relied on the previous behavior. Setting time limits for scripts (such as `PrologEpilogTimeout`) is strongly encouraged to avoid Slurm waiting indefinitely for scripts to finish. + Fix `slurmdbd -R` not returning an error under certain conditions. + `slurmdbd` - Avoid potential NULL pointer dereference in the mysql plugin. + Fix regression in 23.02.3 which broken X11 forwarding for hosts when MUNGE sends a localhost address in the encode host field. This is caused when the node hostname is mapped to 127.0.0.1 (or similar) in `/etc/hosts`. + `openapi/[db]v0.0.39` - fix memory leak on parsing error. + `data_parser/v0.0.39` - fix updating qos for associations. + `openapi/dbv0.0.39` - fix updating values for associations with null users. + Fix minor memory leak with `--tres-per-task` and licenses. + Fix cyclic socket cpu distribution for tasks in a step where `--cpus-per-task` < usable threads per core. + `slurmrestd` - For `GET /slurm/v0.0.39/node[s]`, change format of node\'s energy field `current_watts` to a dictionary to account for unset value instead of dumping 4294967294. + `slurmrestd` - For `GET /slurm/v0.0.39/qos`, change format of QOS\'s field \"priority\" to a dictionary to account for unset value instead of dumping 4294967294. + slurmrestd - For `GET /slurm/v0.0.39/job[s]`, the \'return code\' code field in `v0.0.39_job_exit`_code will be set to -127 instead of being left unset where job does not have a relevant return code.
* Other Changes: + Remove --uid / --gid options from salloc and srun commands. These options did not work correctly since the CVE-2022-29500 fix in combination with some changes made in 23.02.0. + Add the `JobId` to `debug()` messages indicating when `cpus_per_task/mem_per_cpu` or `pn_min_cpus` are being automatically adjusted. + Change the log message warning for rate limited users from verbose to info. + `slurmstepd` - Cleanup per task generated environment for containers in spooldir. + Format batch, extern, interactive, and pending step ids into strings that are human readable. + `slurmrestd` - Reduce memory usage when printing out job CPU frequency. + `data_parser/v0.0.39` - Add `required/memory_per_cpu` and `required/memory_per_node` to `sacct --json` and `sacct --yaml` and `GET /slurmdb/v0.0.39/jobs` from slurmrestd. + `gpu/oneapi` - Store cores correctly so CPU affinity is tracked. + Allow `slurmdbd -R` to work if the root assoc id is not 1. + Limit periodic node registrations to 50 instead of the full `TreeWidth`. Since unresolvable `cloud/dynamic` nodes must disable fanout by setting `TreeWidth` to a large number, this would cause all nodes to register at once.
* Mon Aug 21 2023 Christian Goll - Updated to 23.02.4 with the following changes:
* Bug Fixes: + Fix main scheduler loop not starting after a failover to backup controller. Avoid slurmctld segfault when specifying `AccountingStorageExternalHost` (bsc#1214983). + Fix sbatch return code when `--wait` is requested on a job array. + Fix collected `GPUUtilization` values for `acct_gather_profile` plugins. + Fix `slurmrestd` handling of job hold/release operations. + Fix step running indefinitely when slurmctld takes more than `MessageTimeout` to respond. Now, `slurmctld` will cancel the step when detected, preventing following steps from getting stuck waiting for resources to be released. + Fix regression to make `job_desc.min_cpus` accurate again in `job_submit` when requesting a job with `--ntasks-per-node`. + Fix handling of `ArrayTaskThrottle` in backfill. + Fix regression in 23.02.2 when checking gres state on `slurmctld` startup or reconfigure. Gres changes in the configuration were not updated on slurmctld startup. On startup or reconfigure, these messages were present in the log: `error: Attempt to change gres/gpu Count`. + Fix potential double count of gres when dealing with limits. + Fix `slurmstepd` segfault when `ContainerPath` is not set in `oci.conf` + Fixed an issue where jobs requesting licenses were incorrectly rejected. + `scrontab` - Fix cutting off the final character of quoted variables. + `smail` - Fix issues where e-mails at job completion were not being sent. + `scontrol/slurmctld` - fix comma parsing when updating a reservation\'s nodes. + Fix `--gpu-bind=single binding` tasks to wrong gpus, leading to some gpus having more tasks than they should and other gpus being unused. + Fix regression in 23.02 that causes slurmstepd to crash when `srun` requests more than `TreeWidth` nodes in a step and uses the pmi2 or pmix plugin. + `job_container/tmpfs` - Fix `%h` and `%n` substitution in `BasePath` where `%h` was substituted as the NodeName instead of the hostname, and %n was substituted as an empty string. + Fix regression where `--cpu-bind=verbose` would override `TaskPluginParam`. + `scancel` - Fix `--clusters/-M` for federations. Only filtered jobs (e.g. `-A`, `-u`, `-p`, etc.) from the specified clusters will be canceled, rather than all jobs in the federation. Specific jobids will still be routed to the origin cluster for cancellation.
* Other changes: + Make spank `S_JOB_ARGV` item value hold the requested command `argv` instead of the `srun --bcast` value when `--bcast` requested (only in local context). + `scontrol` - Permit changes to StdErr and StdIn for pending jobs. + `scontrol` - Reset `std`{`err`,`in`,`out`} when set to empty string. + `slurmrestd` - mark environment as a required field for job submission descriptions. + `slurmrestd` - avoid dumping null in OpenAPI schema required fields. + `data_parser/v0.0.39` - avoid rejecting valid `memory_per_node` formatted as dictionary provided with a job description. + `data_parser/v0.0.39` - avoid rejecting valid `memory_per_cpu` formatted as dictionary provided with a job description. + `slurmrestd` - Return HTTP error code 404 when job query fails. + `slurmrestd` - Add return schema to error response to job and license query. + Change the log message warning for rate limited users from debug to verbose. + `cgroup/v2` - Avoid capturing log output for ebpf when constraining devices, as this can lead to inadvertent failure if the log buffer is too small. + Added error message when attempting to use sattach on batch or extern steps. + Reject job `ArrayTaskThrottle` update requests from unprivileged users. + `data_parser/v0.0.39` - populate description fields of property objects in generated OpenAPI specifications where defined. + `slurmstepd` - Avoid segfault caused by `ContainerPath` not being terminated by `/` in `oci.conf`. + `data_parser/v0.0.39` - Change `v0.0.39_job_info` response to tag `exit_code` field as being complex instead of only an unsigned integer.- Updated to 23.02.3 with the following changes:
* Bug Fixes: + `slurmctld` - Fix backup slurmctld crash when it takes control multiple times. + Fix regression in 23.02.2 that ignored the partition `DefCpuPerGPU` setting on the first pass of scheduling a job requesting `--gpus --ntasks`. + `srun` - fix issue creating regular and interactive steps because environment variables were incorrectly set on non-HetSteps. + Fix dynamic nodes getting stuck in allocated states when reconfiguring. + Fix regression in 23.02.2 that set the `SLURM_NTASKS` environment variable in sbatch jobs from `--ntasks-per-node` when `--ntasks` was not requested. + Fix regression in 23.02 that caused sbatch jobs to set the wrong number of tasks when requesting `--ntasks-per-node` without `--ntasks`, and also requesting one of the following options: `--sockets-per-node`, `--cores-per-socket`, `--threads-per-core` (or `--hint=nomultithread`), or `-B,--extra-node-info`. + Fix double counting suspended job counts on nodes when reconfiguring, which prevented nodes with suspended jobs from being powered down or rebooted once the jobs completed. + Fix backfill not scheduling jobs submitted with `--prefer` and `--constraint` properly. + mpi/pmix - fix regression introduced in 23.02.2 which caused PMIx shmem backed files permissions to be incorrect. + api/submit - fix memory leaks when submission of batch regular jobs or batch HetJobs fails (response data is a return code). + Fix regression in 23.02 leading to error() messages being sent at `INFO` instead of `ERR` in syslog. + Fix `TresUsageIn[Tot|Ave]` calculation for `gres/gpumem` and `gres/gpuutil`. + Fix issue in the gpu plugins where gpu frequencies would only be set if both gpu memory and gpu frequencies were set, while one or the other suffices. + Fix reservations group ACL\'s not working with the root group. + Fix updating a job with a ReqNodeList greater than the job\'s node count. + Fix inadvertent permission denied error for `--task-prolog` and `--task-epilog` with filesystems mounted with `root_squash`. + Fix missing detailed cpu and gres information in json/yaml output from `scontrol`, `squeue` and `sinfo`. + Fix regression in 23.02 that causes a failure to allocate job steps that request `--cpus-per-gpu` and gpus with types. + Fix potentially waiting indefinitely for a defunct process to finish, which affects various scripts including `Prolog` and `Epilog`. This could have various symptoms, such as jobs getting stuck in a completing state. + Fix losing list of reservations on job when updating job with list of reservations and restarting the controller. + Fix nodes resuming after down and drain state update requests from clients older than 23.02. + Fix advanced reservation creation/update when an association that should have access to it is composed with partition(s). + Fix job layout calculations with `--ntasks-per-gpu`, especially when `--nodes` has not been explicitly provided. + Fix X11 forwarding for jobs submitted from the slurmctld host. + When a job requests `--no-kill` and one or more nodes fail during the job, fix subsequent job steps unable to use some of the remaining resources allocated to the job. + Fix shared gres allocation when using `--tres-per-task` with tasks that span multiple sockets. + `auth/jwt` - Fix memory leak.
* Other changes: + `openapi/dbv0.0.39/users` - If a default account update failed, resulting in a no-op, the query returned success without any warning. Now a warning is sent back to the client that the default account wasn\'t modified. + Avoid job write lock when nodes are dynamically added/removed. + `burst_buffer/lua` - allow jobs to get scheduled sooner after `slurm_bb_data_in` completes. + `openapi/v0.0.39` - fix memory leak in `_job_post_het_submit()`. + Avoid possible `slurmctld` segfault caused by race condition with already completed `slurmdbd_conn` connections. + `Slurmdbd.conf` checks included conf files for 0600 permissions + `slurmrestd` - fix regression \"oversubscribe\" fields were removed from job descriptions and submissions from v0.0.39 end points. + `accounting_storage/mysql` - Query for indiviual QOS correctly when you have more than 10. + Add warning message about ignoring `--tres-per-tasks=license` when used on a step. + `sshare` - Fix command to work when using `priority/basic`. + Avoid loading `cli_filter` plugins outside of `salloc`/`sbatch`/`scron`/ `srun`. This fixes a number of missing symbol problems that can manifest for executables linked against libslurm (and not `libslurmfull`). + Allow cloud_reg_addrs to update dynamically registered node\'s addrs on subsequent registrations. + Revert a change in 22.05.5 that prevented tasks from sharing a core if `--cpus-per-task` > threads per core, but caused incorrect accounting and cpu binding. Instead, `--ntasks-per-core=1` may be requested to prevent tasks from sharing a core. + Correctly send `assoc_mgr` lock to mcs plugin. + Avoid unnecessary `gres/gpumem` and `gres/gpuutil` `TRES` position lookups. + `sacct` - when printing `PLANNED` time, use end time instead of start time for jobs cancelled before they started. + Hold the job with `(Reservation ... invalid)` state reason if the reservation is not usable by the job. + `sbatch` - Added new `--export=NIL` option.- Removed:
* Fix-test-3.13.patch
* Fix-test-38.11.patch as both tests changed upstream
* Tue Jul 04 2023 Egbert Eich - Create a macro for upgrade dependency to ensure uniform handling.
* Tue May 09 2023 Christian Goll - updated to 23.02.02 which includes a number of fixes to Slurm stability
* Includes a fix for a regression in 23.02 that caused openmpi mpirun to fail to launch tasks.
* It also includes two functional changes: Don\'t update the cron job tasks if the whole crontab file is left untouched after opening it with scrontab -e
* Sort dynamic nodes and include them in topology after scontrol reconfigure or a slurmctld restart.
* Mon Apr 17 2023 Egbert Eich - Web-configurator: changed presets to SUSE defaults.- If %_restart_on_update is no longer defined replace by own macro.- Marked slurm-openlava, slurm-seff and slurm-sjstat noarch.- rpmlint:
* dropped some rpmlint filters which are no longer relevant.
* added/refreshed filters. For Details, see rpmlintrc.- Remove workaround to fix the restart issue in an Slurm package described in bsc#1088693. The Slurm version in this package as 16.05. Any attempt to directly migrate to the current version is bound to fail anyway.- Now require slurm-munge if munge authentication is installed.
* Fri Mar 31 2023 Christian Goll - updated to 23.02.1 with the following changes:
* job_container/tmpfs - cleanup job container even if namespace mount is already unmounted.
* openapi/dbv0.0.38 - Fix not displaying an error when updating QOS or associations fails.
* Fix nodes remaining as PLANNED after slurmctld save state recovery.
* Add cgroup.conf EnableControllers option for cgroup/v2.
* Get correct cgroup root to allow slurmd to run in containers like Docker.
* slurmctld - add missing PrivateData=jobs check to step ContainerID lookup requests originated from \'scontrol show step container-id=\' or certain scrun operations when container state can\'t be directly queried.
* Fix nodes un-draining after being drained due to unkillable step.
* Fix remote licenses allowed percentages reset to 0 during upgrade.
* sacct - Avoid truncating time strings when using SLURM_TIME_FORMAT with the --parsable option.
* Fix regression in 22.05.0rc1 that broke Nodes=ALL in a NodeSet.
* openapi/v0.0.39 - fix jobs submitted via slurmrestd being allocated fewer CPUs than tasks when requesting multiple tasks.
* Fix job not being scheduled on valid nodes and potentially being rejected when using parentheses at the beginning of square brackets in a feature request, for example: \"feat1&[(feat2|feat3)]\".
* Fix regression in 23.02.0rc1 which made --gres-flags=enforce-binding no longer enforce optimal core-gpu job placement.
* mpi/pmix - Fix v5 to load correctly when libpmix.so isn\'t in the normal lib path.
* data_parser/v0.0.39 - fix regression where \"memory_per_node\" would be rejected for job submission.
* data_parser/v0.0.39 - fix regression where \"memory_per_cpu\" would be rejected for job submission.
* slurmctld - add an assert to check for magic number presence before deleting a partition record and clear the magic afterwards to better diagnose potential memory problems.
* Clean up OCI containers task directories correctly.
* scrun - Run under existing job when SLURM_JOB_ID is present.
* Prevent a slurmstepd crash when the I/O subsystem has hung.
* common/conmgr - fix memory leak of complete connection list.
* job_container/tmpfs - avoid printing extraneous error messages when running a spank plugin that implements slurm_spank_job_prolog() or slurm_spank_job_epilog().
* Fix srun < 23.02 always getting an \"exact\" core allocation.
* Prevent scontrol < 23.02 from setting MaxCPUsPerSocket to 0.
* Add ScronParameters=explicit_scancel and corresponding scancel --cron option.- removed right-pmix-path.patch as fixed upstream
* Thu Mar 16 2023 Christian Goll - use libpmix.so.2 instead of libpmix.so to fix (bsc#1209260) this removes the need of pmix-pluginlib added: right-pmix-path.patch
* Wed Mar 15 2023 Christian Goll - slurm-plugins need to require pmix-pluginlib (bsc#1209260)
* Tue Mar 07 2023 Egbert Eich - Stop pulling firewall rules from github. There is no benefit to host these separately.- Remove pre-sle12 pieces.
* Wed Mar 01 2023 Egbert Eich - Add missing Provides:, Conflicts: and Obsoletes: to slurm-cray, slurm-hdf5 and slurm-testsuite to avoid package conflicts.- Unify Obsoletes:.- Consolidate spec files between different Slurm releases in Leap/SLE maintenance.- Add dependency for the general plugin package to the AcctGatherProfile HDF5 plugin.- Adjust node RealMemory in slurm.conf of test suite for 8G test nodes.
* Mon Feb 20 2023 Egbert Eich - updated to 23.02.0
* Highlights + slurmctld - Add new RPC rate limiting feature. This is enabled through SlurmctldParameters=rl_enable, otherwise disabled by default. + Make scontrol reconfigure and sending a SIGHUP to the slurmctld behave the same. If you were using SIGHUP as a \'lighter\' scontrol reconfigure to rotate logs please update your scripts to use SIGUSR2 instead. + Change cloud nodes to show by default. PrivateData=cloud is no longer needed. + sreport - Count planned (FKA reserved) time for jobs running in IGNORE_JOBS reservations. Previously was lumped into IDLE time. + job_container/tmpfs - Support running with an arbitrary list of private mount points (/tmp and /dev/shm are the default, but not required). + job_container/tmpfs - Set more environment variables in InitScript. + Make all cgroup directories created by Slurm owned by root. This was the behavior in cgroup/v2 but not in cgroup/v1 where by default the step directories ownership were set to the user and group of the job. + accounting_storage/mysql - change purge/archive to calculate record ages based on end time, rather than start or submission times. + job_submit/lua - add support for log_user() from slurm_job_modify(). + Run the following scripts in slurmscriptd instead of slurmctld: ResumeProgram, ResumeFailProgram, SuspendProgram, ResvProlog, ResvEpilog, and RebootProgram (only with SlurmctldParameters=reboot_from_controller). + Only permit changing log levels with \'srun --slurmd-debug\' by root or SlurmUser. + slurmctld will fatal() when reconfiguring the job_submit plugin fails. + Add PowerDownOnIdle partition option to power down nodes after nodes become idle. + Add \"[jobid.stepid]\" prefix from slurmstepd and \"slurmscriptd\" prefix from slurmcriptd to Syslog logging. Previously was only happening when logging to a file. + Add purge and archive functionality for job environment and job batch script records. + Extend support for Include files to all \"configless\" client commands. + Make node weight usable for powered down and rebooting nodes. + Removed \'launch\' plugin. + Add \"Extra\" field to job to store extra information other than a comment. + Add usage gathering for AMD (requires ROCM 5.5+) and NVIDIA gpus. + Add job\'s allocated nodes, features, oversubscribe, partition, and reservation to SLURM_RESUME_FILE output for power saving. + Automatically create directories for stdout/stderr output files. Paths may use %j and related substitution characters as well. + Add --tres-per-task to salloc/sbatch/srun. + Allow nodefeatures plugin features to work with cloud nodes. e.g. - Powered down nodes have no active changeable features. - Nodes can\'t be changed to other active features until powered down. - Active changeable features are reset/cleared on power down. + Make slurmstepd cgroups constrained by total configured memory from slurm.conf (NodeName=<> RealMemory=#) instead of total physical memory. + node_features/helpers - add support for the OR and parentheses operators in a --constraint expression. + slurmctld will fatal() when [Prolog|Epilog]Slurmctld are defined but are not executable. + Validate node registered active features are a super set of node\'s currently active changeable features. + On clusters without any PrologFlags options, batch jobs with failed prologs no longer generate an output file. + Add SLURM_JOB_START_TIME and SLURM_JOB_END_TIME environment variables. + Add SuspendExcStates option to slurm.conf to avoid suspending/powering down specific node states. + Add support for DCMI power readings in IPMI plugin. + slurmrestd served /slurm/v0.0.39 and /slurmdb/v0.0.39 endpoints had major changes from prior versions. Almost all schemas have been renamed and modified. Sites using OpenAPI Generator clients are highly suggested to upgrade to to using atleast version 6.x due to limitations with prior versions. + Allow for --nodelist to contain more nodes than required by --nodes. + Rename \"nodes\" to \"nodes_resume\" in SLURM_RESUME_FILE job output. + Rename \"all_nodes\" to \"all_nodes_resume\" in SLURM_RESUME_FILE output. + Add jobcomp/kafka plugin. + Add new PreemptParameters=reclaim_licenses option which will allow higher priority jobs to preempt jobs to free up used licenses. (This is only enabled for with PreemptModes of CANCEL and REQUEUE, as Slurm cannot guarantee suspended jobs will release licenses correctly.) + hpe/slingshot - add support for the instant-on feature. + Add ability to update SuspendExc
* parameters with scontrol. + Add ability to restore SuspendExc
* parameters on restart with slurmctld - R option. + Add ability to clear a GRES specification by setting it to \"0\" via \'scontrol update job\'. + Add SLURM_JOB_OVERSUBSCRIBE environment variable for Epilog, Prolog, EpilogSlurmctld, PrologSlurmctld, and mail ouput. + System node down reasons are appended to existing reasons, separated by \':\'. + New command scrun has been added. scrun acts as an Open Container Initiative (OCI) runtime proxy to run containers seamlessly via Slurm. + Fixed GpuFreqDef option. When set in slurm.conf, it will be used if - -gpu-freq was not explicitly set by the job step.
* Configuration Changes + job_container.conf - Added \"Dirs\" option to list desired private mount points. + node_features plugins - invalid users specified for AllowUserBoot will now result in fatal() rather than just an error. + Deprecate AllowedKmemSpace, ConstrainKmemSpace, MaxKmemPercent, and MinKmemSpace. + Allow jobs to queue even if the user is not in AllowGroups when EnforcePartLimits=no is set. This ensures consistency for all the Partition access controls, and matches the documented behavior for EnforcePartLimits. + Add InfluxDBTimeout parameter to acct_gather.conf. + job_container/tmpfs - add support for expanding %h and %n in BasePath. + slurm.conf - Removed SlurmctldPlugstack option. + Add new SlurmctldParameters=validate_nodeaddr_threads= option to allow concurrent hostname resolution at slurmctld startup. + Add new AccountingStoreFlags=job_extra option to store a job\'s extra field in the database. + Add new \"defer_batch\" option to SchedulerParameters to only defer scheduling for batch jobs. + Add new DebugFlags option \'JobComp\' to replace \'Elasticsearch\'. + Add configurable job requeue limit parameter - MaxBatchRequeue - in slurm.conf to permit changes from the old hard-coded value of 5. + helpers.conf - Allow specification of node specific features. + helpers.conf - Allow many features to one helper script. + job_container/tmpfs - Add \"Shared\" option to support shared namespaces. This allows autofs to work with the job_container/tmpfs plugin when enabled. + acct_gather.conf - Added EnergyIPMIPowerSensors=Node=DCMI and Node=DCMI_ENHANCED. + Add new \"getnameinfo_cache_timeout=\" option to CommunicationParameters to adjust or disable caching the results of getnameinfo(). + Add new PrologFlags=ForceRequeueOnFail option to automatically requeue batch jobs on Prolog failures regardless of the job --requeue setting. + Add HealthCheckNodeState=NONDRAINED_IDLE option. + Add \'explicit\' to Flags in gres.conf. This makes it so the gres is not automatically added to a job\'s allocation when --exclusive is used. Note that this is a per-node flag. + Moved the \"preempt_\" options from SchedulerParameters to PreemptParameters, and dropped the prefix from the option names. (The old options will still be parsed for backwards compatibility, but are now undocumented.) + Add LaunchParameters=ulimit_pam_adopt, which enables setting RLIMIT_RSS in adopted processes. + Update SwitchParameters=job_vni to enable/disable creating job VNIs for all jobs, or when a user requests them. + Update SwitchParameters=single_node_vni to enable/disable creating single node vnis for all jobs, or when a user requests them. + Add ability to preserve SuspendExc
* parameters on reconfig with ReconfigFlags=KeepPowerSaveSettings. + slurmdbd.conf - Add new AllResourcesAbsolute to force all new resources to be created with the Absolute flag. + topology/tree - Add new TopologyParam=SwitchAsNodeRank option to reorder nodes based on switch layout. This can be useful if the naming convention for the nodes does not natually map to the network topology. + Removed the default setting for GpuFreqDef. If unset, no attempt to change the GPU frequency will be made if --gpu-freq is not set for the step.
* Command Changes + sacctmgr - no longer force updates to the AdminComment, Comment, or SystemComment to lower-case. + sinfo - Add -F/--future option to sinfo to display future nodes. + sacct - Rename \'Reserved\' field to \'Planned\' to match sreport and the nomenclature of the \'Planned\' node. + scontrol - advanced reservation flag MAINT will no longer replace nodes, similar to STATIC_ALLOC + sbatch - add parsing for #PBS -d and #PBS -w. + scontrol show assoc_mgr will show username(uid) instead of uid in QoS section. + Add strigger --draining and -R/--resume options. + Change --oversubscribe and --exclusive to be mutually exclusive for job submission. Job submission commands will now fatal if both are set. Previously, these options would override each other, with the last one in the job submission command taking effect. + scontrol - Requested TRES and allocated TRES will now always be printed when showing jobs, instead of one TRES output that was either the requested or allocated. + srun --ntasks-per-core now applies to job and step allocations. Now, use of --ntasks-per-core=1 implies --cpu-bind=cores and - -ntasks-per-core>1 implies --cpu-bind=threads. + salloc/sbatch/srun - Check and abort if ntasks-per-core > threads-per-core. + scontrol - Add ResumeAfter= option to \"scontrol update nodename=\". + Add a new \"nodes=\" argument to scontrol setdebug to allow the debug level on the slurmd processes to be temporarily altered. + Add a new \"nodes=\" argument to \"scontrol setdebugflags\" as well. + Make it so scrontab prints client-side the job_submit() err_msg (which can be set i.e. by using the log_user() function for the lua plugin). + scontrol - Reservations will not be allowed to have STATIC_ALLOC or MAINT flags and REPLACE[_DOWN] flags simultaneously. + scontrol - Reservations will only accept one reoccurring flag when being created or updated. + scontrol - A reservation cannot be updated to be reoccurring if it is already a floating reservation. + squeue - removed unused \'%s\' and \'SelectJobInfo\' formats. + squeue - align print format for exit and derived codes with that of other components (:). + sacct - Add --array option to expand job arrays and display array tasks on separate lines. + Partial support for \'--json\' and \'--yaml\' formated outputs have been implemented for sacctmgr, sdiag, sinfo, squeue, and scontrol. The resultant data ouput will be filtered by normal command arguments. Formatting arguments will continue to be ignored. + salloc/sbatch/srun - extended the --nodes syntax to allow for a list of valid node counts to be allocated to the job. This also supports a \"step count\" value (e.g., --nodes=20-100:20 is equivalent to - -nodes=20,40,60,80,100) which can simplify the syntax when the job needs to scale by a certain \"chunk\" size. + srun - add user requestible vnis with \'--network=job_vni\' option. + srun - add user requestible single node vnis with the \'--network=single_node_vni\' option.
* API Changes + job_container plugins - container_p_stepd_create() function signature replaced uint32_t uid with stepd_step_rec_t
* step. + gres plugins - gres_g_get_devices() function signature replaced pid_t pid with stepd_step_rec_t
* step. + cgroup plugins - task_cgroup_devices_constrain() function signature removed pid_t pid. + task plugins - replace task_p_pre_set_affinity(), task_p_set_affinity(), and task_p_post_set_affinity() with task_p_pre_launch_priv() like it was back in slurm 20.11. + Allow for concurrent processing of job_submit_g_submit() and job_submit_g_modify() calls. If your plugin is not capable of concurrent operation you must add additional locking within your plugin. + Removed return value from slurm_list_append(). + The List and ListIterator types have been removed in favor of list_t and list_itr_t respectively. + burst buffer plugins - add bb_g_build_het_job_script(). bb_g_get_status() - added authenticated UID and GID. bb_g_run_script() - added job_info argument. + burst_buffer.lua - Pass UID and GID to most hooks. Pass job_info (detailed job information) to many hooks. See etc/burst_buffer.lua.example for a complete list of changes. WARNING: Backwards compatibility is broken for slurm_bb_get_status: UID and GID are passed before the variadic arguments. If UID and GID are not explicitly listed as arguments to slurm_bb_get_status(), then they will be included in the variadic arguments. Backwards compatibility is maintained for all other hooks because the new arguments are passed after the existing arguments. + node_features plugins - node_features_p_reboot_weight() function removed. node_features_p_job_valid() - added parameter feature_list. node_features_p_job_xlate() - added parameters feature_list and job_node_bitmap. + New data_parser interface with v0.0.39 plugin.
* Added: Fix-test-1.99.patch
* Reworked: Fix-test-38.11.patch pam_slurm-Initialize-arrays-and-pass-sizes.patch
* Thu Feb 09 2023 Egbert Eich - testsuite: on later SUSE versions claim ownership of directory /etc/security/limits.d.
* Fri Dec 02 2022 Egbert Eich - Move the ext_sensors/rrd plugin to a separate package: this plugin requires librrd which in turn requires huge parts of the client side X Window System stack. There is probably no use in cluttering up a system for a plugin that probably only used by a few.
* Fri Oct 21 2022 Egbert Eich - Test Suite fixes:
* Update README_Testsuite.md.
* Clean up left over files when de-installing test suite.
* Adjustment to test suite package: for SLE mark the openmpi4 devel package and slurm-hdf5 optional.
* Add -ffat-lto-objects to the build flags when LTO is set to make sure the object files we ship with the test suite still work correctly (boo#1204697).
* Improve setup-testsuite.sh: copy ssh fingerprints from all nodes.
* Fri Oct 14 2022 Christian Goll - updated to 22.05.5- NOTE: Slurm validates that libraries are of the same version. Unfortunately, due to an oversight, we failed to notice that the slurmstepd loads the hash_k12 library only after a job has completed. This means that if the hash_k12 library is upgraded before a job finishes, the slurmstepd will load the new library when the job finishes, and will fail due to a mismatch of versions. This results in nodes with slurmstepd processes stuck indefinitely. These processes require manual intervention to clean up. There is no clean way to resolve these hung slurmstepd processes. The only recommended way to upgrade between minor versions of 22.05 with RPM’s or upgrades that replace current binaries and libraries is to drain the nodes of running jobs first.- Fixes a number of moderate severity issues, noteable are:
* Load hash plugin at slurmstepd launch time to prevent issues loading the plugin at step completion if the Slurm installation is upgraded.
* Update nvml plugin to match the unique id format for MIG devices in new Nvidia drivers.
* Fix multi-node step launch failure when nodes in the controller aren\'t in natural order. This can happen with inconsistent node naming (such as node15 and node052) or with dynamic nodes which can register in any order.
* job_container/tmpfs - cleanup containers even when the .ns file isn\'t mounted anymore.
* Wait up to PrologEpilogTimeout before shutting down slurmd to allow prolog and epilog scripts to complete or timeout. Previously, slurmd waited 120 seconds before timing out and killing prolog and epilog scripts.
* Sat Sep 24 2022 Egbert Eich - Do not deduplicate files of testsuite Slurm configuration. This directory is supposed to be mounted over /etc/slurm therefore it must not contain softlinks to the files in this directory.- Improve .a and .o file collection for test suite: find these files even if there are multiple ones in a single line.
* Tue Sep 20 2022 Egbert Eich - Fix build for older product version.
* Tue Aug 02 2022 Egbert Eich - Fix a potential security vulnerability in the test package (bsc#1201674, CVE-2022-31251).
* Thu Jul 21 2022 Bernhard Wiedemann - make slurmtest.tar reproducible
* Thu Jul 14 2022 Egbert Eich - Improve check for mpicc in testsuite package: if binary isn\'t found, don\'t crash.- Patch NOFILE Limit in the slurmd.service copy for the testsuite.
* Mon Jun 20 2022 Christian Goll - update to 22.05.2 with following fixes:
* Fix regression which allowed the oversubscription of licenses.
* Fix a segfault in slurmctld when requesting gres in job arrays.
* Wed Jun 08 2022 Egbert Eich - Package the Slurm testsuite for QA purposes. NOTE: This package is not meant to be used for testing by the user but rather for testing by the maintainers to ensure the package is working properly. DO NOT report test suite failures unless you are able to confirm that the failure is really a bug.
* Fixes for test suite: Keep-logs-of-skipped-test-when-running-test-cases-sequentially.patch Fix-test-21.41.patch Fix-test-38.11.patch Fix-test-32.8.patch Fix-test-3.13.patch Fix-test7.2-to-find-libpmix-under-lib64-as-well.patch
* Add documentation: README_Testsuite.md- Allow log in as user \'slurm\'. This allows admins to run certain priviledged commands more easily without becoming root.
* Tue May 31 2022 Christian Goll - update to 22.05.0 with following changes:- Support for dynamic node addition and removal- Support for native Linux cgroup v2 operation- Newly added plugins to support HPE Slingshot 11 networks (switch/hpe_slingshot), and Intel Xe GPUs (gpu/oneapi)- Added new acct_gather_interconnect/sysfs plugin to collect statistics from arbitrary network interfaces.- Expanded and synced set of environment variables available in the Prolog/Epilog/PrologSlurmctld/EpilogSlurmctld scripts.- New \"--prefer\" option to job submissions to allow for a \"soft constraint\" request to influence node selection.- Optional support for license planning in the backfill scheduler with \"bf_licenses\" option in SchedulerParameters.- removed file slurm-2.4.4-init.patch as sysvinit is now realy deprecated- removed file load-pmix-major-version.patch as fixed upstream- set environment variable SUSE_ZNOW to 0 in %build to avoid module load failures due to unresolved symbols as module take advantage of lazy bindings (bsc#1200030).
* Tue May 10 2022 Egbert Eich - Add a comment about the CommunicationParameters=block_null_hash option warning users who migrate - just in case.
* Fri May 06 2022 Christian Goll - Update to 21.08.8 which fixes CVE-2022-29500 (bsc#1199278), CVE-2022-29501 (bsc#1199279), and CVE-2022-29502 (bsc#1199281).- Added \'CommunicationParameters=block_null_hash\' to slurm.conf, please add this parameter to existing configurations.
* Mon May 02 2022 Christian Goll - Update to 21.08.7 with following changes:
* openapi/v0.0.37 - correct calculation for bf_queue_len_mean in /diag.
* Avoid shrinking a reservation when overlapping with downed nodes.
* Only check TRES limits against current usage for TRES requested by the job.
* Do not allocate shared gres (MPS) in whole-node allocations
* Constrain slurmstepd to job/step cgroup like in previous versions of Slurm.
* Fix warnings on 32-bit compilers related to printf() formats.
* Fix reconfigure issues after disabling/reenabling the GANG PreemptMode.
* Fix race condition where a cgroup was being deleted while another step was creating it.
* Set the slurmd port correctly if multi-slurmd
* Fix FAIL mail not being sent if a job was cancelled due to preemption.
* slurmrestd - move debug logs for HTTP handling to be gated by debugflag NETWORK to avoid unnecessary logging of communication contents.
* Fix issue with bad memory access when shrinking running steps.
* Fix various issues with internal job accounting with GRES when jobs are shrunk.
* Fix ipmi polling on slurmd reconfig or restart.
* Fix srun crash when reserved ports are being used and het step fails to launch.
* openapi/dbv0.0.37 - fix DELETE execution path on /user/{user_name}.
* slurmctld - Properly requeue all components of a het job if PrologSlurmctld fails.
* rlimits - remove final calls to limit nofiles to 4096 but to instead use the max possible nofiles in slurmd and slurmdbd.
* Allow the DBD agent to load large messages (up to MAX_BUF_SIZE) from state.
* Fix potential deadlock during slurmctld restart when there is a completing job.
* slurmstepd - reduce user requested soft rlimits when they are above max hard rlimits to avoid rlimit request being completely ignored and processes using default limits.
* Fix Slurm user commands displaying available features as active features when no features were active.
* Don\'t power down nodes that are rebooting.
* Clear pending node reboot on power down request.
* Ignore node registrations while node is powering down.
* Don\'t reboot any node that is power down.
* Don\'t allow a node to reboot if it\'s marked for power down.
* Fix issuing reboot and downing when rebooting a powering up node.
* Clear DRAIN on node after failing to resume before ResumeTimeout.
* Prevent repeating power down if node fails to resume before ResumeTimeout.
* Fix federated cloud node communication with srun and cloud_dns.
* Fix jobs being scheduled on nodes marked to be powered_down when idle.
* Fix problem where a privileged user could not view array tasks specified by _ when PrivateData had the jobs value set. - Changes in Slurm 21.08.6
* Fix plugin_name definitions in a number of plugins to improve logging.
* Close sbcast file transfers when job is cancelled.
* scrontab - fix handling of --gpus and --ntasks-per-gpu options.
* sched/backfill - fix job_queue_rec_t memory leak.
* Fix magnetic reservation logic in both main and backfill schedulers.
* job_container/tmpfs - fix memory leak when using InitScript.
* slurmrestd / openapi - fix memory leaks.
* Fix slurmctld segfault due to job array resv_list double free.
* Fix multi-reservation job testing logic.
* Fix slurmctld segfault due to insufficient job reservation parse validation.
* Fix main and backfill schedulers handling for already rejected job array.
* sched/backfill - restore resv_ptr after yielding locks.
* acct_gather_energy/xcc - appropriately close and destroy the IPMI context.
* Protect slurmstepd from making multiple calls to the cleanup logic.
* Prevent slurmstepd segfault at cleanup time in mpi_fini().
* Fix slurmctld sometimes hanging if shutdown while PrologSlurmctld or EpilogSlurmctld were running and PrologEpilogTimeout is set in slurm.conf.
* Fix affinity of the batch step if batch host is different than the first node in the allocation.
* slurmdbd - fix segfault after multiple failover/failback operations.
* Fix jobcomp filetxt job selection condition.
* Fix -f flag of sacct not being used.
* Select cores for job steps according to the socket distribution. Previously, sockets were always filled before selecting cores from the next socket.
* Keep node in Future state if epilog completes while in Future state.
* Fix erroneous --constraint behavior by preventing multiple sets of brackets.
* Make ResetAccrueTime update the job\'s accrue_time to now.
* Fix sattach initialization with configless mode.
* Revert packing limit checks affecting pmi2.
* sacct - fixed assertion failure when using -c option and a federation display
* Fix issue that allowed steps to overallocate the job\'s memory.
* Fix the sanity check mode of AutoDetect so that it actually works.
* Fix deallocated nodes that didn\'t actually launch a job from waiting for Epilogslurmctld to complete before clearing completing node\'s state.
* Job should be in a completing state if EpilogSlurmctld when being requeued.
* Fix job not being requeued properly if all node epilog\'s completed before EpilogSlurmctld finished.
* Keep job completing until EpilogSlurmctld is completed even when \"downing\" a node.
* Fix handling reboot with multiple job features.
* Fix nodes getting powered down when creating new partitions.
* Fix bad bit_realloc which potentially could lead to bad memory access.
* slurmctld - remove limit on the number of open files.
* Fix bug where job_state file of size above 2GB wasn\'t saved without any error message.
* Fix various issues with no_consume gres.
* Fix regression in 21.08.0rc1 where job steps failed to launch on systems that reserved a CPU in a cgroup outside of Slurm (for example, on systems with WekaIO).
* Fix OverTimeLimit not being reset on scontrol reconfigure when it is removed from slurm.conf.
* serializer/yaml - use dynamic buffer to allow creation of YAML outputs larger than 1MiB.
* Fix minor memory leak affecting openapi users at process termination.
* Fix batch jobs not resolving the username when nss_slurm is enabled.
* slurmrestd - Avoid slurmrestd ignoring invalid HTTP method if the response serialized without error.
* openapi/dbv0.0.37 - Correct conditional that caused the diag output to give an internal server error status on success.
* Make --mem-bind=sort work with task_affinity
* Fix sacctmgr to set MaxJobsAccruePer{User|Account} and MinPrioThres in sacctmgr add qos, modify already worked correctly.
* job_container/tmpfs - avoid printing extraneous error messages in Prolog and Epilog, and when the job completes.
* Fix step CPU memory allocation with --threads-per-core without --exact.
* Remove implicit --exact when --threads-per-core or --hint=nomultithread is used.
* Do not allow a step to request more threads per core than the allocation did.
* Remove implicit --exact when --cpus-per-task is used.
* Wed Dec 22 2021 Christian Goll - update to 21.08.5 with following changes:
* Fix issue where typeless GRES node updates were not immediately reflected.
* Fix setting the default scrontab job working directory so that it\'s the home of the different user (
*u ) and not that of root or SlurmUser editor.
* Fix stepd not respecting SlurmdSyslogDebug.
* Fix concurrency issue with squeue.
* Fix job start time not being reset after launch when job is packed onto already booting node.
* Fix updating SLURM_NODE_ALIASES for jobs packed onto powering up nodes.
* Cray - Fix issues with starting hetjobs.
* auth/jwks - Print fatal() message when jwks is configured but file could not be opened.
* If sacctmgr has an association with an unknown qos as the default qos print \'UNKN
*###\' instead of leaving a blank name.
* Correctly determine task count when giving --cpus-per-gpu, --gpus and
* -ntasks-per-node without task count.
* slurmctld - Fix places where the global last_job_update was not being set to the time of update when a job\'s reason and description were updated.
* slurmctld - Fix case where a job submitted with more than one partition would not have its reason updated while waiting to start.
* Fix memory leak in node feature rebooting.
* Fix time limit permanetly set to 1 minute by backfill for job array tasks higher than the first with QOS NoReserve flag and PreemptMode configured.
* Fix sacct -N to show jobs that started in the current second
* Fix issue on running steps where both SLURM_NTASKS_PER_TRES and SLURM_NTASKS_PER_GPU are set.
* Handle oversubscription request correctly when also requesting
* -ntasks-per-tres.
* Correctly detect when a step requests bad gres inside an allocation.
* slurmstepd - Correct possible deadlock when UnkillableStepTimeout triggers.
* srun - use maximum number of open files while handling job I/O.
* Fix writing to Xauthority files on root_squash NFS exports, which was preventing X11 forwarding from completing setup.
* Fix regression in 21.08.0rc1 that broke --gres=none.
* Fix srun --cpus-per-task and --threads-per-core not implicitly setting
* -exact. It was meant to work this way in 21.08.
* Fix regression in 21.08.0 that broke dynamic future nodes.
* Fix dynamic future nodes remembering active state on restart.
* Fix powered down nodes getting stuck in COMPLETING+POWERED_DOWN when job is cancelled before nodes are powering up.
* Wed Nov 17 2021 Christian Goll - updated to 21.08.4 which fixes (CVE-2021-43337) which is only present in 21.08 tree.
* CVE-2021-43337: For sites using the new AccountingStoreFlags=job_script and/or job_env options, an issue was reported with the access control rules in SlurmDBD that will permit users to request job scripts and environment files that they should not have access to. (Scripts/environments are meant to only be accessible by user accounts with administrator privileges, by account coordinators for jobs submitted under their account, and by the user themselves.)- changes from 21.08.3:
* This includes a number of fixes since the last release a month ago, including one critical fix to prevent a communication issue between slurmctld and slurmdbd for sites that have started using the new AccountingStoreFlags=job_script functionality.
* Fri Oct 29 2021 Egbert Eich - Utilize sysuser infrastructure to set user/group slurm. For munge authentication slurm should have a fixed UID across all nodes including the management server. Set it to 120- Limit firewalld service definitions to SUSE versions >= 15.
* Mon Oct 18 2021 Christian Goll - added service definitions for firewalld (JSC#SLE-22741)
* Wed Oct 06 2021 Christian Goll - update to 21.08.2- major change:
* removed of support of the TaskAffinity=yes option in cgroup.conf. Please consider using \"TaskPlugins=cgroup,affinity\" in slurm.conf as an option.- minor changes and bugfixes:
* slurmctld - fix how the max number of cores on a node in a partition are calculated when the partition contains multi
*socket nodes. This in turn corrects certain jobs node count estimations displayed client
*side.
* job_submit/cray_aries - fix \"craynetwork\" GRES specification after changes introduced in 21.08.0rc1 that made TRES always have a type prefix.
* Ignore nonsensical check in the slurmd for [Pro|Epi]logSlurmctld.
* Fix writing to stderr/syslog when systemd runs slurmctld in the foreground.
* Fix issue with updating job started with node range.
* Fix issue with nodes not clearing state in the database when the slurmctld is started with clean
*start.
* Fix hetjob components > 1 timing out due to InactiveLimit.
* Fix sprio printing -nan for normalized association priority if PriorityWeightAssoc was not defined.
* Disallow FirstJobId=0.
* Preserve job start info in the database for a requeued job that hadn\'t registered the first time in the database yet.
* Only send one message on prolog failure from the slurmd.
* Remove support for TaskAffinity=yes in cgroup.conf.
* accounting_storage/mysql - fix issue where querying jobs via sacct
* -whole-hetjob=yes or slurmrestd (which automatically includes this flag) could in some cases return more records than expected.
* Fix issue for preemption of job array task that makes afterok dependency fail. Additionally, send emails when requeueing happens due to preemption.
* Fix sending requeue mail type.
* Properly resize a job\'s GRES bitmaps and counts when resizing the job.
* Fix node being able to transition to CLOUD state from non-cloud state.
* Fix regression introduced in 21.08.0rc1 which broke a step\'s ability to inherit GRES from the job when the step didn\'t request GRES but the job did.
* Fix errors in logic when picking nodes based on bracketed anded constraints. This also enforces the requirement to have a count when using such constraints.
* Handle job resize better in the database.
* Exclude currently running, resized jobs from the runaway jobs list.
* Make it possible to shrink a job more than once.
* Tue Sep 28 2021 Christian Goll - moved pam module from /lib64 to /usr/lib64 which fixes boo#1191095 via the macro %_pam_moduledir
* Fri Sep 17 2021 Christian Goll - updated to 21.08.1 with following bug fixes:
* Fix potential memory leak if a problem happens while allocating GRES for a job.
* If an overallocation of GRES happens terminate the creation of a job.
* AutoDetect=nvml: Fatal if no devices found in MIG mode.
* Print federation and cluster sacctmgr error messages to stderr.
* Fix off by one error in --gpu-bind=mask_gpu.
* Add --gpu-bind=none to disable gpu binding when using --gpus-per-task.
* Handle the burst buffer state \"alloc-revoke\" which previously would not display in the job correctly.
* Fix issue in the slurmstepd SPANK prolog/epilog handler where configuration values were used before being initialized.
* Restore a step\'s ability to utilize all of an allocations memory if --mem=0.
* Fix --cpu-bind=verbose garbage taskid.
* Fix cgroup task affinity issues from garbage taskid info.
* Make gres_job_state_validate() client logging behavior as before 44466a4641.
* Fix steps with --hint overriding an allocation with --threads-per-core.
* Require requesting a GPU if --mem-per-gpu is requested.
* Return error early if a job is requesting --ntasks-per-gpu and no gpus or task count.
* Properly clear out pending step if unavailable to run with available resources.
* Kill all processes spawned by burst_buffer.lua including decendents.
* openapi/v0.0.{35,36,37} - Avoid setting default values of min_cpus, job name, cwd, mail_type, and contiguous on job update.
* openapi/v0.0.{35,36,37} - Clear user hold on job update if hold=false.
* Prevent CRON_JOB flag from being cleared when loading job state.
* sacctmgr - Fix deleting WCKeys when not specifying a cluster.
* Fix getting memory for a step when the first node in the step isn\'t the first node in the allocation.
* Make SelectTypeParameters=CR_Core_Memory default for cons_tres and cons_res.
* Correctly handle mutex unlocks in the gres code if failures happen.
* Give better error message if -m plane is given with no size.
* Fix --distribution=arbitrary for salloc.
* Fix jobcomp/script regression introduced in 21.08.0rc1 0c75b9ac9d.
* Only send the batch node in the step_hostlist in the job credential.
* When setting affinity for the batch step don\'t assume the batch host is node 0.
* In task/affinity better checking for node existence when laying out affinity.
* slurmrestd - fix job submission with auth/jwt.- removed Fix-statement-condition-in-netloc-autoconf-macro.patch issue was fixed upstream
* Mon Sep 06 2021 Egbert Eich - Fix-statement-condition-in-netloc-autoconf-macro.patch: Fix netloc check, reestablish netloc disable code.- Make configure arg \'--with-pmix\' conditional.- Move openapi plugins to package slurm-restd.
* Thu Sep 02 2021 Christian Goll - updated to 21.08.0, major changes:
* A new \"AccountingStoreFlags=job_script\" option to store the job scripts directly in SlurmDBD.
* Added \"sacct -o SubmitLine\" format option to get the submit line of a job/step.
* Changes to the node state management so that nodes are marked as PLANNED instead of IDLE if the scheduler is still accumulating resources while waiting to launch a job on them.
* RS256 token support in auth/jwt.
* Overhaul of the cgroup subsystems to simplify operation, mitigate a number of inherent race conditions, and prepare for future cgroup v2 support.
* Further improvements to cloud node power state management.
* A new child process of the Slurm controller called \"slurmscriptd\" responsible for executing PrologSlurmctld and EpilogSlurmctld scripts, which significantly reduces performance issues associated with enabling those options.
* A new burst_buffer/lua plugin allowing for site-specific asynchronous job data management.
* Fixes to the job_container/tmpfs plugin to allow the slurmd process to be restarted while the job is running without issue.
* Added json/yaml output to sacct, squeue, and sinfo commands.
* Added a new node_features/helpers plugin to provide a generic way to change settings on a compute node across a reboot.
* Added support for automatically detecting and broadcasting shared libraries for an executable launched with \"srun --bcast\".
* Added initial OCI container execution support with a new --container option to sbatch and srun.
* Improved \"configless\" support by allowing multiple control servers to be specified through the slurmd --conf-server option, and send additional configuration files at startup including cli_filter.lua.- minor changes:
* If an overallocation of GRES happens terminate the creation of a job.
* AutoDetect=nvml: Fatal if no devices found in MIG mode.
* Print federation and cluster sacctmgr error messages to stderr.
* Add --gpu-bind=none to disable gpu binding when using --gpus-per-task.
* Handle the burst buffer state \"alloc-revoke\" which previously would not display in the job correctly.
* Fix issue in the slurmstepd SPANK prolog/epilog handler where configuration values were used before being initialized.
* Restored --gpu-bind=single: to check core affinity like
* -gpu-bind=closest does. This removal of this behavior only was in rc2.
* slurmd - Fix assert failure on initialization due to bad node name.
* Fix error codes in cgroup/v1.
* Don\'t destroy the memory step outside fini, which leads to a double destroy causing an error message.
* Add support for lua 5.4.
* Force cgroup.clone_children to 0 in slurm cgroup directories. This caused issues in task cpuset plugin in systems with it enabled by default.
* Clear GRES HAS_TYPE flag when removing type name.
* Environment flags in gres.conf now override flags set by AutoDetect.
* Environment flags in gres.conf now apply to subsequent gres.conf lines where Environment flags are not set.
* Set missing job_uid and job_gid members when preparing a kill_job_msg_t in abort_job_on_node(), abort_job_on_nodes() and kill_job_on_node().
* Fix swappiness not being set in cgroups.
* Fix coordinators for new subaccounts.
* Fix coordinators when adding existing users with PrivateData=users.
* slurmctld - do not attempt to relinquish control to self.
* openapi/v0.0.37 - Honor kill_on_invalid_dependency as job parameter.
* Check max_gres when doing step allocation, fix for regression in rc2.
* SPANK plugins are now required to match the current Slurm version, and must be recompiled for each new Slurm release.
* node_features/helpers - add ExecTime configuration option.
* srun - Fix force termination with -X.
* On slurmctld restart set node typed GRES counts correctly.
* Fix places where a step wasn\'t allocated in the slurmctld but wasn\'t ever removed from the job.
* Fix step allocation memory when using --threads-per-core.
* Fix step allocations to consume all threads on a core when using threads-per-core.
* Add check to validate cpu request on a step if --threads-per-core is given and it is less than what the core on the node has in the allocation.
* Fix issue where a step could request more gres than the job had and the step would hang forever. This bug was only introduced in 21.08.0rc2.
* Only print \\r\
for logging messages on stderr when --pty has been explicitly requested.
* Relax check on SPANK plugins to only require Slurm major + minor versions to match.
* job_container/tmpfs - delegate handling of /dev/shm to the extern step so new step launches will be attached correctly even after the slurmd process has been restarted.
* Limit the wait time in proctrack_g_wait() to UnkillableStepTimeout instead of a hardcoded value of 256 seconds, and limit the delay between tests to a maximum of 32 seconds.
* fatal() on start if using job_container/tmpfs without PrologFlags=Contian.
* Load bf_when_last_cycle from job state only if protocol version >= 21.08.
* Docs - remove man3 section entirely.
* Set step memory when using MemPerGPU or DefMemPerGPU. Previously a step\'s memory was not set even when it requested
*-mem-per-gpu and at least one GPU.
* Add cli_filter.lua support in configless mode.
* Check that the step requests at least as many gres as nodes.
* Make job\'s SLURM_JOB_GPUS print global GPU IDs instead of MIG unique_ids.
* Fix miscounting of GPU envs in prolog/epilog if MultipleFiles was used.
* Support MIGs in prolog/epilog\'s CUDA_VISIBLE_DEVICES & co.
* Add SLURM_JOB_GPUS back into Prolog; add it to Epilog.
* Fix issue where the original executable, not the bcast\'d version, was executed with \'srun
*-bcast\'.
* sacct - print \'-\' header correctly for fields over 53-characters wide.
* openapi/dbv0.0.37 - replace \"REST\" with \"Slurm OpenAPI\" for plugin_name.
* openapi/v0.0.37 - replace \"REST\" with \"Slurm OpenAPI\" for plugin_name.
* configless - fix segfault on \'scontrol reconfigure\'.
* Use FREE_NULL_LIST instead of list_destroy.
* If we made are running an interactive session we need to force track_steps.
* Disable OPOST flag when using --pty to avoid issues with Emac.
* Fix issue where extra bonus core was allocated in some situations.
* Avoid putting gres with count of 0 on a TRES req/alloc.
* Fix memory in requested TRES when --mem-per-gpu is used.
* Changed ReqMem field in sacct to match memory from ReqTRES.
* Changed --gpu-bind=single: to no longer check core affinity like
* -gpu-bind=closest does. This consequently affects --ntasks-per-gpu.
* slurmrestd - add v0.0.37 OpenAPI plugin.
* slurmrestd/v0.0.37 - rename standard_in -> standard_input.
* slurmrestd/v0.0.37 - rename standard_out -> standard_output.
* Changed the --format handling for negative field widths (left justified) to apply to the column headers as well as the printed fields.
* Add LimitFactor to the QOS. A float that is factored into an associations [Grp|Max]TRES limits. For example, if the LimitFactor is 2, then an association with a GrpTRES of 30 CPUs, would be allowed to allocate 60 CPUs when running under this QOS.
* slurmrestd - Pass SLURM_NO_CHANGE_IN_DATA to client as 403 (Not Modified).
* slurmrestd/v0.0.37 - Add update_time field to Jobs query to allow clients to only get jobs list based on change timestamp.
* Reset job eligible time when job is manually held.
* Add DEBUG_FLAG_JAG to improve logging related to job account gathering.
* Convert logging in account_gather/common to DEBUG_FLAG_JAG.
* Add more logging for jag_common_poll_data() when prec_extra() called.
* slurmrestd/v0.0.37 - add API to fetch reservation(s) info.
* Catch more errors in task/cgroup initalization and cleanup to avoid allowing jobs to start when cgroups failure to configure correctly.
* Fix cgroup ns detection when using containers (e.g. LXC or Docker).
* Reset job\'s next_step_id counter to 0 after being requeued.
* Make scontrol exit with non-zero status after failing to delete a partition or reservation.
* Make NtasksPerTRES optional in slurm_sprint_job_info().
* slurmrestd/v0.0.37 - Add update_time field to nodes query to allow clients to only get nodes list based on change timestamp.
* common/parse_config - catch and propagate return codes when handling a match on a key-value pattern. This implies error codes detected in the handlers are now not ignored and users of _handle_keyvalue_match() can fatal().
* common/hostlist - fix hostlist_delete_nth() xassert() upper bound check.
* API change: Removed slurm_kill_job_msg and modified the function signature for slurm_kill_job2. slurm_kill_job2 should be used instead of slurm_kill_job_msg.
* Fix non-zero exit code for scontrol ping when all controllers are down.
* Enforce a valid configuration for AccountingStorageEnforce in slurm.conf. If the configuration is invalid, then an error message will be printed and the command or daemon (including slurmctld) will not run.
* slurmrestd/v0.0.37 - Add update_time field to partitions/reservations query to allow clients to only get the entities list when something changed.
* slurmdbd.service - add \"After\" relationship to all common names for MariaDB to reduce startup delays.
* slurmrestd/v0.0.37 - Correct displaying node states that are UNKNOWN.
* slurmrestd/v0.0.37 - Add flags to node states.
* Fix first job on fresh cluster not being assigned JobId=1 (or FirstJobId).
* squeue - make it so --nodelist is sensitive to --clusters.
* squeue - do --nodelist node validation in the same order as listing.
* Removed AccountingStoreJobComment option. Please update your config to use AccountingStoreFlags=job_comment instead.
* AccountingStoreFlags=job_script allows you to store the job\'s batch script.
* AccountingStoreFlags=job_env allows you to store the job\'s env vars.
* Add sacct -o SubmitLine to get the submit line of a job/step.
* Removed DefaultStorage{Host,Loc,Pass,Port,Type,User} options.
* Fix NtasksPerTRES delimiter from : to = in scontrol show job output.
* Removed CacheGroups, CheckpointType, JobCheckpointDir, MemLimitEnforce, SchedulerPort, SchedulerRootFilter options.
* Make job accounting queries use consistent timeframes with and w/o jobs.
* --cpus-per-task and --threads-per-core now imply --exact. This fixes issues where steps would be allocated the wrong number of CPUs.
* configure: the --with option handling has been made consistent across the various optional libraries. Specifying
*-with-foo=/path/to/foo will only check that directory for the applicable library (rather than, in some cases, falling back to the default directories), and will always error the build if the library is not found (instead of a mix of error messages and non- fatal warning messages).
* configure: replace --with-rmsi_dir option with proper handling for
* -with-rsmi=dir.
* Pass additional job environment variables to MailProg.
* Add SLURM_JOB_WORK_DIR to Prolog, Epilog.
* Removed sched/hold plugin.
* Fix srun overwriting SLURM_SUBMIT_DIR and SLURM_SUBMIT_HOST when within an existing allocation.
* step_ctx code has been removed from the api.
* cli_filter/lua, jobcomp/lua, job_submit/lua now load their scripts from the same directory as the slurm.conf file (and thus now will respect changes to the SLURM_CONF environment variable).
* SPANK - call slurm_spank_init if defined without slurm_spank_slurmd_exit in slurmd context.
* job_container/tmpfs - Remove need for .active file to allow salloc without an interactive step to work.
* slurmd - Delay background node registration on every failure up to 128s on startup.
* slurmctld - Always notify slurmd that node registration was accepted to avoid slurmd needless attempting to re-register if there is configuration issue.
* Put node into \"INVAL\" state upon registering with an invalid node configuration. Node must register with a valid configuration to continue.
* Make --cpu-bind=threads default for --threads-per-core -- cli and env can override.
* jobcomp/elasticsearch - Use data_t to serialize data. The plugin now has the JSON-C library as a prerequisite.
* scrontab - create the temporary file under the TMPDIR environment variable (if set), otherwise continue to use TmpFS as configured in slurm.conf.
* Add LastBusyTime to \"scontrol show nodes\" and slurmrestd nodes output, which represents the time the node last had jobs on it.
* slurmd - allow multiple comma-separated controllers to be specified in configless mode with
*-conf-server
* sacctmgr - changed column headings to \"ParentID\" and \"ParentName\" instead of \"Par ID\" and \"Par Name\" respectively.
* Perl API - make sure man pages are installed under the --prefix given to configure.
* Manually powering down of nodes with scontrol now ignores SuspendExc.
* SALLOC_THREADS_PER_CORE and SBATCH_THREADS_PER_CORE have been added as input environment variables for salloc and sbatch, respectively. They do the same thing as
*-threads-per-core.
* Distinguish queued reboot requests (REBOOT) from issued reboots (REBOOT^).
* Set the maximum number of open files per process to 4096 to avoid performance issues when closing the entire range with closeall().
* auth/jwt - add support for RS256 tokens.
* Relax reservation purge due to any invalid uid after creation time.
* Reject srun that requests both --exclusive and --overlap.
* service files - change dependency to network-online rather than just network to ensure DNS and other services are available.
* RSMI: Fix incorrect PCI BDF bits.
* plugins/cli_filter - Convert to using data_t to serialize JSON.
* Fix testing array job after regaining locks in backfill.
* Don\'t display node\'s comment with \"scontrol show nodes\" unless set.
* Add \"Extra\" field to node to store extra information other than a comment.
* scrontab - Use /tmp instead of TmpFS if TMPDIR is not set.
* Add ResumeTimeout, SuspendTimeout and SuspendTime to Partitions.
* sreport - change to sorting TopUsage by the --tres option.
* slurmrestd - do not run allow operation as SlurmUser/root by default.
* Allow map_cpu and mask_cpu for non-whole node allocation.
* TaskPluginParam=verbose is now treated as a default. Previously it would be applied regardless of the job specifying a
*-cpu-bind.
* Add \"node_reg_mem_percent\" SlurmctldParameter to define percentage of memory nodes are allowed to register with.
* Show correct number of SocketsPerBoard in slurmd -C with hwloc2.
* Alter sreport\'s cluster utilization report column name from \'Reserved\' to \'Planned\' to match the nomenclature of the \'Planned\' node.
* Add StateComplete format option to sinfo to show base_state+flags.
* \"scontrol show node\" now shows State as base_state+flags instead of shortened state with flags appended. eg. IDLE#
*> IDLE+POWERING_UP. Also \"POWER\" state flag string is \"POWERED_DOWN\".
* slurmd/req - add missing job_env_t\'s het_job_id initialization off the request in _rpc_{abort,terminate}_job(). This caused problems for Native Cray builds when joining a CNCU job_container plugin with Epilog configured.
* Fix joining a CNCU job_container on a Native Cray build before executing the UnkillableStepProgram for a HetJob step.
* slurmrestd/v0.0.35 - Plugin has been tagged as deprecated.
* srun - Job steps requiring more cores than available to be rejected unless \'--overlap\' is specificed.
* Add bf_node_space_size to SchedulerParameters.
* Add scontrol update node state=POWER_DOWN_FORCE and POWER_DOWN_ASAP as new ways to power off and reset especially CLOUD nodes.
* Define and separate node power state transitions. Previously a powering down node was in both states, POWERING_OFF and POWERED_OFF. These are now separated.
* Create a new process called slurmscriptd which runs PrologSlurmctld and EpilogSlurmctld. This avoids fork() calls from slurmctld, and can avoid performance issues if the slurmctld has a large memory footprint.
* Added new Script option to DebugFlags for debugging slurmscriptd.
* scrontab - add ability to update crontab from a file or standard input.
* scrontab - add ability to set and expand variables.
* Pass JSON of job to node mappings to ResumeProgram.
* If running steps in an allocation with CR_PACK_NODE or -mpack the srun will only attempt to allocate as much as needed from the allocation instead of always trying to allocate every node in the allocation.
* Jobs that request the whole node now check to see if any gres are allocated.
* Rename SbcastParameters to BcastParameters.
* Make srun sensitive to BcastParameters.
* RSMI: Add gres_links_create_empty() and preserve RSMI enumeration order.
* GPUs: Use index instead of dev_num for CUDA_VISIBLE_DEVICES
* Don\'t run epilog on nodes if job never launched.
* QOS accrue limits only apply to the job QOS, not partition QOS.
* Add --gpu-bind=per_task: option, --gpus-per-task will now set this option by default.
* Treat any return code from SPANK plugin that is not SLURM_SUCCESS to be an error or rejection.
* Print the statistics for extern step adopted processes in sstat.
* Fix SLURM_NODE_ALIASES to work for ipv6 node addrs.
* Add support for automatically detecting and broadcasting executable shared object dependencies for sbcast and srun
*-bcast.
* Fri Jul 02 2021 Christian Goll - Updated to 20.11.8:
* slurmctld - fix erroneous \"StepId=CORRUPT\" messages in error logs.
* Correct the error given when auth plugin fails to pack a credential.
* Fix unused-variable compiler warning on FreeBSD in fd_resolve_path().
* acct_gather_filesystem/lustre - only emit collection error once per step.
* Add GRES environment variables (e.g., CUDA_VISIBLE_DEVICES) into the interactive step, the same as is done for the batch step.
* Fix various potential deadlocks when altering objects in the database dealing with every cluster in the database.
* slurmrestd: - handle slurmdbd connection failures without segfaulting. - fix segfault for searches in slurmdb/v0.0.36/jobs. - remove (non-functioning) users query parameter for slurmdb/v0.0.36/jobs from openapi.json - fix segfault in slurmrestd db/jobs with numeric queries - add argv handling for job/submit endpoint. - add description for slurmdb/job endpoint.
* slurmrestd/dbv0.0.36: - Fix values dumped in job state/current and job step state. - Correct description for previous state property.
* srun: - fix broken node step allocation in a heterogeneous allocation. - leave SLURM_DIST_UNKNOWN as default for --interactive.
* Fail step creation if -n is not multiple of --ntasks-per-gpu.
* job_container/tmpfs - Fix slowdown on teardown.
* Fix problem with SlurmctldProlog where requeued jobs would never launch.
* job_container/tmpfs - Fix issue when restarting slurmd where the namespace mount points could disappear.
* sacct: - avoid truncating JobId at 34 characters. - fix segfault when printing StepId (or when using --long).
* scancel - fix segfault when --wckey filtering option is used.
* select/cons_tres - Fix memory leak.
* Prevent file descriptor leak in job_container/tmpfs on slurmd restart.
* perlapi/libslurmdb - expose tres_req_str to job hash.
* scrontab - close and reopen temporary crontab file to deal with editors that do not change the original file, but instead write out then rename a new file.
* sstat - fix linking so that it will work when --without-shared-libslurm was used to build Slurm.
* Clear allocated cpus for running steps in a job before handling requested nodes on new step.
* Don\'t reject a step if not enough nodes are available. Instead, defer the step until enough nodes are available to satisfy the request.
* Don\'t reject a step if it requests at least one specific node that is already allocated to another step. Instead, defer the step until the requested node(s) become available.
* Better handling of --mem=0.
* Ignore DefCpuPerGpu when --cpus-per-task given.
* Fri May 14 2021 Christian Goll - Updated to 20.11.7 which fixes CVE-2021-31215 (bsc#1186024)- New features in 20.11.7:
* slurmd - handle configless failures gracefully instead of hanging indefinitely.
* select/cons_tres - fix Dragonfly topology not selecting nodes in the same leaf switch when it should as well as requests with
*-switches option.
* Fix issue where certain step requests wouldn\'t run if the first node in the job allocation was full and there were idle resources on other nodes in the job allocation.
* Fix deadlock issue with Slurmctld.
* torque/qstat - fix printf error message in output.
* When adding associations or wckeys avoid checking multiple times a user or cluster name.
* Fix wrong jobacctgather information on a step on multiple nodes due to timeouts sending its the information gathered on its node.
* Fix missing xstrdup which could result in slurmctld segfault on array jobs.
* Fix security issue in PrologSlurmctld and EpilogSlurmctld by always prepending SPANK_ to all user-set environment variables. CVE-2021-31215.- New features in 20.11.6:
* Fix sacct assert with the --qos option.
* Use pkg-config --atleast-version instead of --modversion for systemd.
* common/fd - fix getsockopt() call in fd_get_socket_error().
* Properly handle the return from fd_get_socket_error() in _conn_readable().
* cons_res - Fix issue where running jobs were not taken into consideration when creating a reservation.
* Avoid a deadlock between job_list for_each and assoc QOS_LOCK.
* Fix TRESRunMins usage for partition qos on restart/reconfig.
* Fix printing of number of tasks on a completed job that didn\'t request tasks.
* Fix updating GrpTRESRunMins when decrementing job time is bigger than it.
* Make it so we handle multithreaded allocations correctly when doing - -exclusive or --core-spec allocations.
* Fix incorrect round-up division in _pick_step_cores
* Use appropriate math to adjust cpu counts when --ntasks-per-core=1.
* cons_tres - Fix consideration of power downed nodes.
* cons_tres - Fix DefCpuPerGPU, increase cpus-per-task to match with gpus-per-task
* cpus-per-gpu.
* Fix under-cpu memory auto-adjustment when MaxMemPerCPU is set.
* Make it possible to override CR_CORE_DEFAULT_DIST_BLOCK.
* Perl API - fix retrieving/storing of slurm_step_id_t in job_step_info_t.
* Recover state of burst buffers when slurmctld is restarted to avoid skipping burst buffer stages.
* Fix race condition in burst buffer plugin which caused a burst buffer in stage-in to not get state saved if slurmctld stopped.
* auth/jwt - print an error if jwt_file= has not been set in slurmdbd.
* Fix RESV_DEL_HOLD not being a valid state when using squeue --states.
* Add missing squeue selectable states in valid states error message.
* Fix scheduling last array task multiple times on error, causing segfault.
* Fix issue where a step could be allocated more memory than the job when dealing with --mem-per-cpu and --threads-per-core.
* Fix removing qos from assoc with -= can lead to assoc with no qos
* auth/jwt - fix segfault on invalid credential in slurmdbd due to missing validate_slurm_user() function in context.
* Fix single Port= not being applied to range of nodes in slurm.conf
* Fix Jobs not requesting a tres are not starting because of that tres limit.
* acct_gather_energy/rapl - fix AveWatts calculation.
* job_container/tmpfs - Fix issues with cleanup and slurmd restarting on running jobs.
* Mon May 03 2021 Egbert Eich - Ship REST API version and auth plugins with slurmrestd.- Add YAML support for REST API to build (bsc#1185603).
* Wed Mar 17 2021 Christian Goll - Udpate to 20.11.5:- New features:
* New job_container/tmpfs plugin developed by NERSC that can be used to create per-job filesystem namespaces. Documentaiion and configuration can be found in the respecting man page.- Bug fixes:
* Fix main scheduler bug where bf_hetjob_prio truncates SchedulerParameters.
* Fix sacct not displaying UserCPU, SystemCPU and TotalCPU for large times.
* scrontab - fix to return the correct index for a bad #SCRON option.
* scrontab - fix memory leak when invalid option found in #SCRON line.
* Add errno for when a user requests multiple partitions and they are using partition based associations.
* Fix issue where a job could run in a wrong partition when using EnforcePartLimits=any and partition based associations.
* Remove possible deadlock when adding associations/wckeys in multiple threads.
* When using PrologFlags=alloc make sure the correct Slurm version is set in the credential.
* When sending a job a warning signal make sure we always send SIGCONT beforehand.
* Fix issue where a batch job would continue running if a prolog failed on a node that wasn\'t the batch host and requeuing was disabled.
* Fix issue where sometimes salloc/srun wouldn\'t get a message about a prolog failure in the job\'s stdout.
* Requeue or kill job on a prolog failure when PrologFlags is not set.
* Fix race condition causing node reboots to get requeued before ResumeTimeout expires.
* Preserve node boot_req_time on reconfigure.
* Preserve node power_save_req_time on reconfigure.
* Fix node reboots being queued and issued multiple times and preventing the reboot to time out.
* Fix run_command to exit correctly if track_script kills the calling thread.
* Only requeue a job when the PrologSlurmctld returns nonzero.
* When a job is signaled with SIGKILL make sure we flush all prologs/setup scripts.
* Handle burst buffer scripts if the job is canceled while stage_in is happening.
* When shutting down the slurmctld make note to ignore error message when we have to kill a prolog/setup script we are tracking.
* scrontab - add support for the --open-mode option.
* acct_gather_profile/influxdb - avoid segfault on plugin shutdown if setup has not completed successfully.
* Reduce delay in starting salloc allocations when running with prologs.
* Alter AllocNodes check to work if the allocating node\'s domain doesn\'t match the slurmctld\'s. This restores the pre
*20.11 behavior.
* Fix slurmctld segfault if jobs from a prior version had the now-removed INVALID_DEPEND state flag set and were allowed to run in 20.11.
* Add job_container/tmpfs plugin to give a method to provide a private /tmp per job.
* Set the correct core affinity when using AutoDetect.
* slurmrestd - mark \"environment\" as required for job submissions in schema.
* Tue Feb 23 2021 Christian Goll - Udpate to 20.11.04
* Fix node selection for advanced reservations with features.
* mpi/pmix: Handle pipe failure better when using ucx.
* mpi/pmix: include PMIX_NODEID for each process entry.
* Fix job getting rejected after being requeued on same node that died.
* job_submit/lua - add \"network\" field.
* Fix situations when a reoccuring reservation could erroneously skip a period.
* Ensure that a reservations [pro|epi]log are ran on reoccuring reservations.
* Fix threads-per-core memory allocation issue when using CR_CPU_MEMORY.
* Fix scheduling issue with --gpus.
* Fix gpu allocations that request --cpus-per-task.
* mpi/pmix: fixed print messages for all PMIXP_
* macros
* Add mapping for XCPU to --signal option.
* Fix regression in 20.11 that prevented a full pass of the main scheduler from ever executing.
* Work around a glibc bug in which \"0\" is incorrectly printed as \"nan\" which will result in corrupted association state on restart.
* Fix regression in 20.11 which made slurmd incorrectly attempt to find the parent slurmd address when not applicable and send incorrect reverse
*tree info to the slurmstepd.
* Fix cgroup ns detection when using containers (e.g. LXC or Docker).
* scrontab - change temporary file handling to work with emacs.- Removed check-for-lipmix.so.MAJOR.patch- Added: load-pmix-major-version.patch
* Wed Jan 20 2021 Ana Guerrero Lopez - Update to 20.11.03- This release includes a major functional change to how job step launch is handled compared to the previous 20.11 releases. This affects srun as well as MPI stacks - such as Open MPI - which may use srun internally as part of the process launch. One of the changes made in the Slurm 20.11 release was to the semantics for job steps launched through the \'srun\' command. This also inadvertently impacts many MPI releases that use srun underneath their own mpiexec/mpirun command. For 20.11.{0,1,2} releases, the default behavior for srun was changed such that each step was allocated exactly what was requested by the options given to srun, and did not have access to all resources assigned to the job on the node by default. This change was equivalent to Slurm setting the --exclusive option by default on all job steps. Job steps desiring all resources on the node needed to explicitly request them through the new \'--whole\' option. In the 20.11.3 release, we have reverted to the 20.02 and older behavior of assigning all resources on a node to the job step by default. This reversion is a major behavioral change which we would not generally do on a maintenance release, but is being done in the interest of restoring compatibility with the large number of existing Open MPI (and other MPI flavors) and job scripts that exist in production, and to remove what has proven to be a significant hurdle in moving to the new release. Please note that one change to step launch remains - by default, in 20.11 steps are no longer permitted to overlap on the resources they have been assigned. If that behavior is desired, all steps must explicitly opt-in through the newly added \'--overlap\' option. Further details and a full explanation of the issue can be found at: https://bugs.schedmd.com/show_bug.cgi?id=10383#c63- Other changes from 20.11.03
* Fix segfault when parsing bad \"#SBATCH hetjob\" directive.
* Allow countless gpu:* PMIx - Don\'t set UCX_MEM_MMAP_RELOC for older version of UCX (pre 1.5).
* Don\'t green-light any GPU validation when core conversion fails.
* Allow updates to a reservation in the database that starts in the future.
* Better check/handling of primary key collision in reservation table.
* Improve reported error and logging in _build_node_list().
* Fix uninitialized variable in _rpc_file_bcast() which could lead to an incorrect error return from sbcast / srun --bcast.
* mpi/cray_shasta - fix use-after-free on error in _multi_prog_parse().
* Cray - Handle setting correct prefix for cpuset cgroup with respects to expected_usage_in_bytes. This fixes Cray\'s OOM killer.
* mpi/pmix: Fix PMIx_Abort support.
* Don\'t reject jobs allocating more cores than tasks with MaxMemPerCPU.
* Fix false error message complaining about oversubscribe in cons_tres.
* scrontab - fix parsing of empty lines.
* Fix regression causing spank_process_option errors to be ignored.
* Avoid making multiple interactive steps.
* Fix corner case issues where step creation should fail.
* Fix job rejection when --gres is less than --gpus.
* Fix regression causing spank prolog/epilog not to be called unless the spank plugin was loaded in slurmd context.
* Fix regression preventing SLURM_HINT=nomultithread from being used to set defaults for salloc->srun, sbatch->srun sequence.
* Reject job credential if non-superuser sets the LAUNCH_NO_ALLOC flag.
* Make it so srun --no-allocate works again.
* jobacct_gather/linux - Don\'t count memory on tasks that have already finished.
* Fix 19.05/20.02 batch steps talking with a 20.11 slurmctld.
* jobacct_gather/common - Do not process jobacct\'s with same taskid when calling prec_extra.
* Cleanup all tracked jobacct tasks when extern step child process finishes.
* slurmrestd/dbv0.0.36 - Correct structure of dbv0.0.36_tres_list.
* Fix regression causing task/affinity and task/cgroup to be out of sync when configured ThreadsPerCore is different than the physical threads per core.
* Fix situation when --gpus is given but not max nodes (-N1-1) in a job allocation.
* Interactive step - ignore cpu bind and mem bind options, and do not set the associated environment variables which lead to unexpected behavior from srun commands launched within the interactive step.
* Handle exit code from pipe when using UCX with PMIx.
* Fri Jan 08 2021 Egbert Eich - Fix fallout introduced by: \"Replace \'%service_del_postun -n\' with \'%service_del_postun_without_restart\'\" for older Leap/SLE versions.
* Fri Jan 08 2021 Egbert Eich - Fix Provides:/Conflicts: for libnss_slurm (bsc#1180700).
* Tue Jan 05 2021 Ana Guerrero Lopez - Add support for configuration files from external plugins. While built-in plugins have their configuration added in slurm.conf, external SPANK plugins add their configuration to plugstack.conf To allow packaging easily spank plugins, their configuration files should be added independently at /etc/spack/plugstack.conf.d and plugstack.conf should be left with an oneliner including all the files under /etc/spack/plugstack.conf.d