Carnegie Mellon University School of Computer Science

SCS Dragon Linux Local Host Administration

There are more than a thousand Linux workstations throughout the School of Computer Science. In order to provide functional and scalable management, SCS Computing Facilities has implemented a customized environment as well as deployed Linux distributions with customizations specific to our environment.

This page describes the major changes that we have made to OS-level software, and how to make local modifications to this software on supported Linux workstations. It also describes some of our policies for local system administration. If you make configuration changes to your host, or install network-aware software, please be aware of SCS network use policies

Important: Proceed with caution or reach out to us for assistance!

Below you will find helpful procedures for local Linux system administration. These procedures assume a degree of familiarity with Linux system administration as they may involve elevated access (sudoer or root) when modifying configuration files. Please reach out to us if you are unfamiliar with these procedures or general local system administration. Making incorrect changes can affect system availability ot completely disable the system.

If you do not have root access to your machine, please submit an access request.

We provide root access by adding a username/root@CS.CMU.EDU entry of the the file.k5login in root's home directory. This entry will allow you to su (super user/elevation) using your username/root Kerberos instance password, and will also allow you to login remotely (via ssh) as root if you have username/root credentials on the client system.

If you do not have a /root Kerberos instance, you can create one yourself using the ;Instance Manager. If you forget your /root instance password, you can submit a request for assistance.

Because root access depends on Kerberos, and a network connection, you may want to have a local root password on your machine. You can set a local root password by becoming root and running the passwd root command (please pick a secure password).

Please do not remove SCS Computing Facilities staff members from root's .k5login file or change the root's home directory, The .k5login file must remain in the root home directory in order to enable remote access.

On machines where you use sudo to perform administrator actions, we will also add you to the appropriate system groups that allow you to invoke administrator commands by providing your normal SCS Kerberos password to sudo.

If you have a Linux host, you can boot it into single user mode by appending the word Single to the line in the GRUB bootloader menu. You may have to type e to modify the GRUB commands. If your machine asks for a local root password during boot and you don't know (or have not set) a local root password, you may be able to boot directly to a shell by appending init=/bin/sh to the GRUB command line.

Submit a ticket if you would like us to create accounts on your SCSCF-supported machine.

We attempt to keep usernames and user IDs (ie the number that corresponds to a username) synchronized across machines. We do not do any synchronization of group IDs. If you are creating a local account for somebody who already has an SCS identity, you can use the following command to add their account: /usr/cs/bin/scsuseradd <username>. This will create a local account for the user, with their Unix UID, full name, default shell, etc, as they appear in LDAP.

By default, the scsuseradd program will create accounts that use the user's AFS home directory. On hosts that are to be used as desktop workstations, it is recommended that accounts be created with a local home directory instead, as GNOME and some desktop software do not deal well with keeping their dotfiles in a shared filesystem.

To create an account with a local homedir, run the following command: /usr/cs/bin/scsuseradd -L <username>. The local homedir will also be populated with the standard SCS shell profiles.

The scsuseradd program can take additional flags and arguments. Usage information can be seen by running the following command: /usr/cs/bin/scsuseradd –h 

View extended documentation by running the following command:  perldoc /usr/cs/bin/scsuseradd. Note that this command cannot be used to create local accounts for users who do not have SCS identities.

We have set aside local UID's for the ability to create true local (non-SCS) accounts. We recommend confirming the local username being created does not match an existing SCS username as this will create login or even mail delivery problems.  

A word about local accounts: Please use caution when creating a local account for a non-SCS person. Please be aware you are responsible when providing access to our computers or network. We are not able to provide support to non-SCS, local users on our systems. 

Generally, we recommend using the packages supplied by the vendor and available via the local SCS mirror of the official Canonical repositories. Dragon, (the SCS custom managed-computing environment), is designed to ensure that vendor packages an end user chooses to install will not interfere or collide with that installation. By design, our Dragon environment will NOT alter or modify a vendor installation. The following are some recommendations and suggestions to consider when installing certain packages on a Dragon (SCS built) based machine.

Important note: Backups

By default, the only partitions backed up on a Dragon system which is subscribed to the Facilities backup service are /etc and /usrN, for any integer value of N. Many vendor packages may default to storing data under /var (as in the specific examples of Apache and MySQL below, but this caveat may apply to any vendor package in general), which is not backed up by default, as it also contains temporary data, runtime- specific data, and often-voluminous logfiles. You machine's /var partition may also not be large enough to store large quantities of data alongside everything else that lives there. You may wish to consider moving your app's data store to a /usrN partition.

Important Note: AppArmor

Some Dragon platforms (e.g. Ubuntu) employ AppArmor by default, which is a kernel-based security module that can be used to restrict access by various daemons and programs to a limited list of directory paths in the filesystem. If you do decide to move an application's datastore from /var to a larger or backed-up partition like /usr0, you may need to adjust the application's AppArmor profile under /etc/apparmor.d.

Apache 2 Web Server

Please refer to the following URL (https://help.ubuntu.com/16.04/serverguide/httpd.html) when installing the Apache 2 Web Server package for details on how to install and configure this application. 

If it is expected that this application will involve a significant utilization of disk space, SCS recommends moving the document root directory to another location with more disk space. The vendor installs the document root directory in /var/www. On Dragon machines the /var partition may not be big enough to support a large scale web site. SCS Facilities recommends creating the document root directory under /usr0 (e.g. /usr0/www) and then symlinking /var/www to this newly created directory. 

Alternatively, you may change the DocumentRoot directive in /etc/apache2/sites-available/default (or whichever config stub file is being used for a site in question) to point to the alternative location.

MySQL Database

Please refer to the following URL (https://help.ubuntu.com/16.04/serverguide/mysql.html) when installing the MySQL database package for details on how to install and configure this application. 

If it is expected that this application will involve a significant utilization of disk space, SCS recommends moving the data directory to another location with more disk space. The vendor installs the database in /var/lib/mysql . On Dragon machines the /var partition may not be big enough to support large databases. As with Apache 2 above, SCS Facilities recommends relocating the MySQL data directory to a larger partition such as /usr0 if it is expected the database will house large amounts of data. 

The default data directory location can either be moved and a symlink provided to the new location, or moved and the DATADIR directive updated in /etc/mysql/my.cnf to point to the new location. 

Note that MySQL heavily employs AppArmor to restrict access via the mysqld binary. If you relocate MySQL's datastore from /var/lib/mysql to, for example, /usr0/mysql, you will need to add the following to /etc/apparmor.d/local/usr.sbin.mysqld:

  /usr0/mysql/ r,  /usr0/mysql/** rwk, 

Note that the above lines must end in commas, as this file is included in the middle of a list of allowed paths by /etc/apparmor.d/usr.sbin.mysqld.

 

You should assume that our network is hostile, and that any traffic on it may be monitored by potential intruders. For this reason, you should use ssh when connecting to machines. You should assume that if somebody can log in to a machine that they can become root on it by exploiting some vulnerability (we install patches for all known remote exploits, but we may not install patches for all local-only exploits, and patch availability may lag behind the most recent known exploits). For this reason, you should be aware that there is a risk in typing passwords at any machine on which other people have accounts. If you are an administrator for a group of machines, it is suggested that you give yourself root access on those machines by adding your root instance to root's .k5login file. By su-ing on your local machine, and typing:

ssh remote-host 

you will be able to ssh in as root to machines you administer without typing a password on them.

You can also use the program iptables to set up a firewall on your machine. Note that the default behavior when setting up iptables is to deny access to all machines, so the default configuration must be adjusted to allow at a minimum SCS hosts and users access to your machine. See the iptables(8) man page for details.

Please do not configure iptables, tcpwrappers, or any other security service in such a way that denies access to the machine by our staff or services. We cannot provide assistance to any machine to which access has been blocked.

If you suspect that your machine has been broken into, contact us at once, so that our security staff can handle the situation. If you are looking for signs of a break-in yourself, be aware that it is common for intruders to replace system binaries such as ls, ps, netstat, etc, so you should not trust the output of such programs.

 

We have made a few modifications to the standard set of services that system vendors ship by default:

  • All machines run sshd. The ssh client and server are the vendor versions but may have some non-default configuration to support various features in the SCS environment.
  • Dragonfly MTA is the provided mail transport agent. See the section below on mail for more details.
  • terad is the backup system. Please do not remove this.

Dragonfly MTA is the supported mail transport agent on current SCS Dragon machines.

  • Dragonfly does not spool mail locally. It relays all locally-sourced mail into the central SCS mail system.
  • Dragonfly MTA does not accept mail from remote machines. If you need (or think you need) to accept mail, please contact Facilities for instruction on how to change your system's MTA.

SCS Computing Facilities-written system management software on SCS computers is typically automatically updated by a program called depot. Global system configuration information is distributed by a system called SUP. These programs are run nightly by /usr/cs/bin/dosupdepot. Occasionally, a machine will fail to depot/SUP because of some problem (such as a full disk, or AFS problems). If you suspect that your machine is not getting software updates, you can look at /usr/fac/log/depot.log to see when it last successfully ran dosupdepot.

You can run dosupdepot by manually, for example, force an update after you've subscribed a machine to a different software releaselevel (see below). Automatic dosupdepot upgrades can be disabled entirely by creating a file called /etc/disableupgrade. (Note: don't do this— you will not get security fixes and other important software upgrades if you do). You can force a depot run even if /etc/disableupgrade exists by running dosupdepot with the -force option.

Every machine has a particular release level of software that it is subscribed to by default (individual software collection release levels may be overridden by entries in /usr/cs/depot/depot.pref.local. The machine-wide software release level is controlled by the file /etc/releaselevel. By default (if that file does not exist), the release level is omega. You can subscribe a machine to alpha or beta release levels of software by putting a single line reading alpha or beta in /etc/releaselevel.

Depot controls only the contents of /usr/cs/, and does not modify any files outside of that directory.

Important note: We recommend not refusing upgrades unless there is an overwhelming reason to do so. Facilities staff will not debug problems caused by refusing some portion of (or all) Facilities software updates.

The information above applies to machines running the SCS Dragon computing environment. SCS Facilities can only provide best-effort support for machines that do not run our environment. However, we can provide a set of Service Configuration Add-Ons for certain Unix and Linux platforms, to be used on machines for which the SCS Dragon computing environment is inappropriate (for example, grant-funded projects with access restrictions that would be violated by allowing Facilities staff members access to your systems) or for machines you would prefer to administer yourself.

These configuration add-ons are provided in a vendor-native format and provide service configuration information only, to facilitate interoperation with core SCS services, such as AFS, Kerberos authentication, printing to SCS printers, and system backups.

If you or your project have such a machine, then you or your project is responsible for taking care of it. In particular, you are responsible for providing security patches and upgrades, and ensuring that it does not become a problem (eg runs a password sniffer or is used for denial of service attacks) for the rest of the facility.