Accounting

Presently job completion is not supported with the SlurmDBD, but can be written directly to a database, script or flat file. If you are running with the accounting storage, you may not need to run this since it contains much of the same information. If you would like to configure this, some of the more important parameters include:

• JobCompHost: Only needed if using a database. The name or address of the host where the database server executes.
• JobCompPass: Only needed if using a database. Password for the user connecting to the database. Since the password can not be security maintained, storing the information directly in a database is not recommended.
• JobCompPort: Only needed if using a database. The network port that the database accepts communication on.
• JobCompType: Type of jobcomp plugin set to "jobcomp/mysql", "jobcomp/pgsql", or "jobcomp/filetxt".
• JobCompUser: Only needed if using a database. User name to connect to the database with.

SLURM Accounting Configuration Before Build

While the SlurmDBD will work with a flat text file for recording job completions and such this configuration will not allow "associations" between a user and account. A database allows such a configuration.

MySQL is the preferred database, PostgreSQL is supported for job and step accounting only. The infrastructure for PostgresSQL for use with associations is not yet supported, meaning sacctmgr will not work correcting. If interested in adding this capability for PostgresSQL, please contact us at slurm-dev@lists.llnl.gov.

To enable this database support one only needs to have the development package for the database they wish to use on the system. The slurm configure script uses mysql_config and pg-config to find out the information it needs about installed libraries and headers. You can specify where your mysql_config script is with the --with-mysql_conf=/path/to/mysql_config option when configuring your slurm build. A similar option is also available for PostgreSQL. On a successful configure, output is something like this:

checking for mysql_config... /usr/bin/mysql_config
MySQL test program built properly.

SLURM Accounting Configuration After Build

For simplicity sake we are going to reference everything as if you are running with the SlurmDBD. You can communicate with a storage plugin directly, but that offers minimal security.

Several SLURM configuration parameters must be set to support archiving information in SlurmDBD. SlurmDBD has a separate configuration file which is documented in a separate section. Note that you can write accounting information to SlurmDBD while job completion records are written to a text file or not maintained at all. If you don't set the configuration parameters that begin with "AccountingStorage" then accounting information will not be referenced or recorded.

• AccountingStorageEnforce: This option contains a comma separated list of options you may want to enforce. The valid options are any comma separated combination of
  • associations - This will prevent users from running jobs if their association is not in the database. This option will prevent users from accessing invalid accounts.
  • limits - This will enforce limits set to associations. By setting this option, the 'associations' option is also set.
  • qos - This will require all jobs to specify (either overtly or by default) a valid qos (Quality of Service). QOS values are defined for each association in the database. By setting this option, the 'associations' option is also set.
  • wckeys - This will prevent users from running jobs under a wckey that they don't have access to. By using this option, the 'associations' option is also set. The 'TrackWCKey' option is also set to true.
  (NOTE: The association is a combination of cluster, account, user names and optional partition name.)
  Without AccountingStorageEnforce being set (the default behavior) jobs will be executed based upon policies configured in SLURM on each cluster.
  It is advisable to run without the option 'limits' set when running a scheduler on top of SLURM, like Moab, that does not update in real time their limits per association.
• AccountingStorageHost: The name or address of the host where SlurmDBD executes
• AccountingStoragePass: If using SlurmDBD with a second Munge daemon, store the pathname of the named socket used by Munge to provide enterprise-wide. Otherwise the default Munge daemon will be used.
• AccountingStoragePort: The network port that SlurmDBD accepts communication on.
• AccountingStorageType: Set to "accounting_storage/slurmdbd".
• ClusterName: Set to a unique name for each Slurm-managed cluster so that accounting records from each can be identified.
• TrackWCKey: Boolean. If you want to track wckeys (Workload Characterization Key) of users. A Wckey is an orthogonal way to do accounting against maybe a group of unrelated accounts. WCKeys can be defined using sacctmgr add wckey 'name'. When a job is run use srun --wckey and time will be summed up for this wckey.

SlurmDBD Configuration

SlurmDBD requires its own configuration file called "slurmdbd.conf". This file should be only on the computer where SlurmDBD executes and should only be readable by the user which executes SlurmDBD (e.g. "slurm"). This file should be protected from unauthorized access since it contains a database login name and password. See "man slurmdbd.conf" for a more complete description of the configuration parameters. Some of the more important parameters include:

• AuthInfo: If using SlurmDBD with a second Munge daemon, store the pathname of the named socket used by Munge to provide enterprise-wide. Otherwise the default Munge daemon will be used.
• AuthType: Define the authentication method for communications between SLURM components. A value of "auth/munge" is recommended.
• DbdHost: The name of the machine where the Slurm Database Daemon is executed. This should be a node name without the full domain name (e.g. "lx0001"). This defaults to localhost but should be supplied to avoid a warning message.
• DbdPort: The port number that the Slurm Database Daemon (slurmdbd) listens to for work. The default value is SLURMDBD_PORT as established at system build time. If none is explicitly specified, it will be set to 6819. This value must be equal to the AccountingStoragePort parameter in the slurm.conf file.
• LogFile: Fully qualified pathname of a file into which the Slurm Database Daemon's logs are written. The default value is none (performs logging via syslog).
• PluginDir: Identifies the places in which to look for SLURM plugins. This is a colon-separated list of directories, like the PATH environment variable. The default value is the prefix given at configure time + "/lib/slurm".
• SlurmUser: The name of the user that the slurmctld daemon executes as. This user must exist on the machine executing the Slurm Database Daemon and have the same user ID as the hosts on which slurmctld execute. For security purposes, a user other than "root" is recommended. The default value is "root". This name should also be the same SlurmUser on all clusters reporting to the SlurmDBD.
• StorageHost: Define the name of the host the database is running where we are going to store the data. Ideally this should be the host on which SlurmDBD executes. But could be a different machine.
• StorageLoc: Specifies the name of the database where accounting records are written, for databases the default database is slurm_acct_db. Note the name can not have a '/' in it or the default will be used.
• StoragePass: Define the password used to gain access to the database to store the job accounting data.
• StoragePort: Define the port on which the database is listening.
• StorageType: Define the accounting storage mechanism type. Acceptable values at present include "accounting_storage/mysql" and "accounting_storage/pgsql". The value "accounting_storage/mysql" indicates that accounting records should be written to a MySQL database specified by the StorageLoc parameter. The value "accounting_storage/pgsql" indicates that accounting records should be written to a PostgreSQL database specified by the StorageLoc parameter. This value must be specified.
• StorageUser: Define the name of the user we are going to connect to the database with to store the job accounting data.

MySQL Configuration

While SLURM will create the database automatically you will need to make sure the StorageUser is given permissions in MySQL to do so. As the mysql user grant privileges to that user using a command such as:

GRANT ALL ON StorageLoc.* TO 'StorageUser'@'StorageHost'; (The ticks are needed)

(You need to be root to do this. Also in the info for password usage there is a line that starts with '->'. This a continuation prompt since the previous mysql statement did not end with a ';'. It assumes that you wish to input more info.)

mysql@snowflake:~$ mysql
Welcome to the MySQL monitor.Commands end with ; or \g.
Your MySQL connection id is 538
Server version: 5.0.51a-3ubuntu5.1 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> grant all on slurm_acct_db.* TO 'slurm'@'localhost';
Query OK, 0 rows affected (0.00 sec)

or with a password...

mysql> grant all on slurm_acct_db.* TO 'slurm'@'localhost'
    -> identified by 'some_pass' with grant option;
Query OK, 0 rows affected (0.00 sec)

This will grant user 'slurm' access to do what it needs to do on the local host. This should be done before the SlurmDBD will work properly. After you grant permission to the Slurm user in mysql then you can start SlurmDBD and Slurm. You start SlurmDBD by typing 'slurmdbd'. You can verify that SlurmDBD is running by typing 'ps aux | grep slurmdbd'. After SlurmDBD and the slurmctld start you can verify that the database was created by using the mysql command 'show databases;'. You can display the tables that slurm created in the database by using the mysql command 'use slurm4_db;' and then 'show tables;'.

Use the mysql 'show databases;' command

mysql> show databases;

+--------------------+
| Database           |
+--------------------+
| information_schema |
| slurm4_db          |
| test               |
+--------------------+

3 rows in set (0.00 sec)

Select the database that you created.

mysql> use slurm4_db;

Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

Now do a mysql 'show tables;' command.

mysql> show tables;

+---------------------------+
| Tables_in_slurm4_db       |
+---------------------------+
| acct_coord_table          |
| acct_table                |
| assoc_day_usage_table     |
| assoc_hour_usage_table    |
| assoc_month_usage_table   |
| assoc_table               |
| cluster_day_usage_table   |
| cluster_event_table       |
| cluster_hour_usage_table  |
| cluster_month_usage_table |
| cluster_table             |
| job_table                 |
| last_ran_table            |
| qos_table                 |
| resv_table                |
| step_table                |
| suspend_table             |
| table_defs_table          |
| txn_table                 |
| user_table                |
| wckey_day_usage_table     |
| wckey_hour_usage_table    |
| wckey_month_usage_table   |
| wckey_table               |
+---------------------------+

24 rows in set (0.02 sec)

mysql> quit

If the database is not created or SlurmDBD is not running you can use the -v option when you start SlurmDBD to get more detailed information.

Tools

There are a few tools available to work with accounting data, sacct, sacctmgr, and sreport. These tools all get or set data through the SlurmDBD daemon.

• sacct is used to generate accounting report for both running and completed jobs.
• sacctmgr is used to manage associations in the database: add or remove clusters, add or remove users, etc.
• sreport is used to generate various reports on usage collected over a given time period.

See the man pages for each command for more information.

Web interfaces with graphical output is currently under development and should be available in the Fall of 2009. A tool to report node state information is also under development.

Database Configuration

Accounting records are maintained based upon what we refer to as an Association, which consists of four elements: cluster, account, user names and an optional partition name. Use the sacctmgr command to create and manage these records. There is an order to set up accounting associations. You must define clusters before you add accounts and you must add accounts before you can add users.

For example, to add a cluster named "snowflake" to the database execute this line:

sacctmgr add cluster snowflake

Add accounts "none" and "test" to cluster "snowflake" with an execute line of this sort:

sacctmgr add account none,test Cluster=snowflake \
  Description="none" Organization="none"

If you have more clusters you want to add these accounts, to you can either not specify a cluster, which will add the accounts to all clusters in the system, or comma separate the cluster names you want to add to in the cluster option. Note that multiple accounts can be added at the same time by comma separating the names. Some description of the account and the organization which it belongs must be specified. These terms can be used later to generated accounting reports. Accounts may be arranged in a hierarchical fashion, for example accounts chemistry and physics may be children of the account science. The hierarchy may have an arbitrary depth. Just specify the parent='' option in the add account line to construct the hierarchy. For the example above execute

sacctmgr add account science \
 Description="science accounts" Organization=science
sacctmgr add account chemistry,physics parent=science \
 Description="physical sciences" Organization=science

Add users to accounts using similar syntax. For example, to permit user da to execute jobs on all clusters with a default account of test execute:

sacctmgr add user da DefaultAccount=test

If AccountingStorageEnforce=associations is configured in the slurm.conf of the cluster snowflake then user da would be allowed to run in account test and any other accounts added in the future. Any attempt to use other accounts will result in the job being aborted. Account test will be the default if he doesn't specify one in the job submission command.

Partition names can also be added to an "add user" command with the Partition='partitionname' option to specify an association specific to a slurm partition.

Cluster Options

When either adding or modifying a cluster, these are the options available with sacctmgr:

• Name= Cluster name

Account Options

When either adding or modifying an account, the following sacctmgr options are available:

• Cluster= Only add this account to these clusters. The account is added to all defined clusters by default.
• Description= Description of the account. (Default is account name)
• Name= Name of account
• Organization=Organization of the account. (Default is parent account unless parent account is root then organization is set to the account name.)
• Parent= Make this account a child of this other account (already added).

User Options

When either adding or modifying a user, the following sacctmgr options are available:

• Account= Account(s) to add user to
• AdminLevel= This field is used to allow a user to add accounting privileges to this user. Valid options are
  • None
  • Operator: can add, modify,and remove users, and add other operators)
  • Admin: In addition to operator privileges these users can add, modify, and remove accounts and clusters
• Cluster= Only add to accounts on these clusters (default is all clusters)
• DefaultAccount= Default account for the user, used when no account is specified when a job is submitted. (Required on creation)
• DefaultWCKey= Default wckey for the user, used when no wckey is specified when a job is submitted. (Only used when tracking wckeys.)
• Name= User name
• Partition= Name of SLURM partition this association applies to

Limit enforcement

When limits are developed they will work in this order. If a user has a limit set SLURM will read in those, if not we will refer to the account associated with the job. If the account doesn't have the limit set we will refer to the cluster's limits. If the cluster doesn't have the limit set no limit will be enforced.

All of the above entities can include limits as described below...

• Fairshare= Used for determining priority. Essentially this is the amount of claim this association and it's children have to the above system.
• GrpJobs= The total number of jobs able to run at any given time from this association and its children. If this limit is reached new jobs will be queued but only allowed to run after previous jobs complete from this group.
• GrpNodes= The total count of nodes able to be used at any given time from jobs running from this association and its children. If this limit is reached new jobs will be queued but only allowed to run after resources have been relinquished from this group.
• GrpSubmitJobs= The total number of jobs able to be submitted to the system at any given time from this association and its children. If this limit is reached new submission requests will be denied until previous jobs complete from this group.
• GrpWall= The maximum wall clock time any job submitted to this group can run for. If this limit is reached submission requests will be denied.
• MaxJobs= The total number of jobs able to run at any given time from this association. If this limit is reached new jobs will be queued but only allowed to run after previous jobs complete from this association.
• MaxNodesPerJob= The maximum size in nodes any given job can have from this association. If this limit is reached the job will be denied at submission.
• MaxSubmitJobs= The maximum number of jobs able to be submitted to the system at any given time from this association. If this limit is reached new submission requests will be denied until previous jobs complete from this association.
• MaxWallDurationPerJob= The maximum wall clock time any job submitted to this association can run for. If this limit is reached the job will be denied at submission.
• QOS= comma separated list of QOS's this association is able to run.

Modifying Entities

When modifying entities, you can specify many different options in SQL-like fashion, using key words like where and set. A typical execute line has the following form:

sacctmgr modify <entity> set <options> where <options>

For example:

sacctmgr modify user set default=none where default=test

will change all users with a default account of "test" to account "none". Once an entity has been added, modified or removed, the change is sent to the appropriate SLURM daemons and will be available for use instantly.

Removing Entities

Removing entities using an execute line similar to the modify example above, but without the set options. For example, remove all users with a default account "test" using the following execute line:

sacctmgr remove user where default=test

Note: In most cases, removed entities are preserved, but flagged as deleted. If an entity has existed for less than 1 day, the entity will be removed completely. This is meant to clean up after typographic errors.

Last modified 25 January 2010