Technical Recommendations and Best Practices for Centron’s Tutorials
Introduction
This guide is an effort to summarize established best practices and strong recommendations for authors of Centron tutorials. It is intended to provide a foundation for consistency, technical correctness, and ease of use in Centron’s instructional material.
This guide is by nature both a work in progress and an opinionated document, based on the growing experience of in-house technical writers and editors, community managers, and engineers. Its recommendations are subject to change, and are written specifically for educational content with a broad range of readers and end-users in mind.
Software Sources and Installation
Many tutorials will rely on existing tutorials as prerequisites. Put all prerequisites for the article (including any nested prerequisites for the prerequisites) into the article, rather than having deeply nested prereq lists.
Preferred Sources
In rough, descending order of preference, use the following installation mechanisms:
- The project recommended method, when evaluated to be best. Many projects change quickly and recommend going beyond the official repositories, but some installations (like curl | bash patterns) require a judgement call on whether or not to use them.
- The official package repositories for the current distribution and release.
- Language-specific official packages (NPM, CPAN, PIP, RubyGems, Composer, etc.)
- Project-specific package repositories (e.g. Nginx provides its own repos for up-to-date versions) or, on Ubuntu, a trusted PPA. Make sure these are from a well-trusted source, like the project’s developers or the Debian/Ubuntu package maintainers.
- Binaries from the project’s GitHub releases page or a similar official web source.
- wget or curl install scripts piped to the shell, with an appropriate warning about inspecting scripts.
Preferred Installation Locations
In general, avoid unnecessary complication. For unpackaged software installed from source or binaries, you should generally accept the default installation prefix unless it’s very unusual or introduces conflicts.
An init script, conforming to official recommendations for the distribution, should be given for service-oriented software, if not provided by the package or other installation method.
On Linux systems, put self-contained binaries or directories in /opt and standalone scripts in /usr/local/bin.
Software and System Maintenance
Ubuntu and Debian systems should have unattended-upgrades with at least security updates installed and configured. We recommend no auto-reboot or auto-update all, given context.
We generally recommend using the apt command for Ubuntu 18.04 and newer. Change apt-get and apt-cache to use apt. When updating apt commands, do not use the -y flag; readers should be guided through any necessary inputs and prompts.
For CentOS and Rocky Linux tutorials, we now recommend using dnf, which has superseded yum and provides better performance.
If a tutorial relies on the latest updates, call update and upgrade during Step 1 setup or as needed. Call update first so that your server pulls the latest versions of packages. When including upgrade, which downloads and installs new versions for every package, please be aware that some users may choose to keep certain packages at a lower version.
Service Management
Make sure to use native init system commands, even when legacy compatibility commands are available. For instance, use sudo systemctl start [service_name] even though sudo service [service_name] start will work.
Provide information about how to enable or disable the service from starting at boot. Indicate how to inspect outcome of service-related commands when not clearly indicated by the interface (journalctl -u, systemctl status, etc).
Prefer restarts over reloads for services as a rule of thumb. In most cases, it’s more important to ensure a known state than avoid a split-second service interruption, and restarts are also more useful in the case of a complete service failure.
Bootstrapping systems
Unless it’s part of a config management workflow, prefer user-data scripts, and prefer cloudinit scripts to bash scripts in user-data in most cases.
Logging and Troubleshooting
Explain where and how to access logs for installed services. Where relevant, explain systemctl and journalctl commands for checking service status and log output. Where possible, offer concise suggestions for diagnosing common failure cases.
Make sure to handle log rotation for any cases where it’s not handled by packages or other installation mechanisms.
For following plaintext log files, use tail -F, not tail -f, as the latter will not track a file across renames and might cause confusion if logs are rotated while a user is watching them.
User and Group Management
Create sudo users instead of using root directly. Reference the appropriate initial server setup guides which explain this task as a prerequisite.
On Debian-based distributions, add and remove users with adduser sammy and deluser –remove-home sammy respectively; on RHEL-based distributions, use adduser sammy (set a password with passwd sammy if necessary) and userdel -r sammy.
Grant sudo privileges with usermod -aG sudo sammy on Ubuntu. CentOS is a little more complicated. Modern versions use usermod -aG wheel sammy, but some versions require visudo to uncomment wheel group permissions first. Specifically, on CentOS 5, sudo needs to be installed and the wheel group needs to be uncommented with visudo; on CentOS 6, sudo is already installed, but wheel needs to be uncommented; CentOS 7 has sudo and the wheel group is already set up.
When using privilege escalated commands, make sure to test them as written. To pass environment variables through sudo, use sudo -E command_to_run (if trusted with entire environment) or sudo FOO=BAR command_to_run. For instances that require a root shell, use sudo -i. For instances that require redirection, use tee -a to append to rather than replace the destination file: [sudo] command_to_run | sudo tee [-a] file_to_change.
Preferred Tools
For interactive shells, assume Bash on GNU/Linux systems, mentioned explicitly when relevant. On FreeBSD, use tcsh, as it’s available out of the box and has useful features.
For text editors, we include the copy “use [preferred] or your favorite text editor”, and include the following beginner-friendly editors in commands for those copy and pasting. On Linux, we default to nano; on FreeBSD, we default to ee. vi(m) is permissible, but avoid it in introductory topics where it might present a stumbling block for beginners.
For file transfer, we generally recommend sftp in most cases for its interactive and scp-alike uses, though it lacks push functionality, so scp is acceptable as well. rsync is useful for backups and large transfers (or many small files). Do not use FTP under any circumstances. We also make an effort to standardize on curl over wget because of its robustness. wget’s advantage is mostly recursive download (i.e. a special use case which is not common for our kind of content).
On machines that ship with iproute2 utilities, we prefer them to the net-tools suite, which is considered obsolete. In general, iproute2 utilities like ss will have better support for multiple interfaces, IPv6, new kernel functionality, etc. So likewise, we should use ip route over route, ip addr show over ifconfig, etc. Sometimes the older utilities output is a bit cleaner by default, but the output itself is a bit less trustworthy since they don’t handle edge cases as well. When possible, we will control the more verbose output using available flags.
Scripting
Within the context of systems administration tutorials, generally avoid lengthy custom scripts and long shell scripts.
Author-written scripts (and possibly other resources) should live in a per-article repository in the do-community GitHub account with a link back to the published tutorial. Follow good scripting practices in general. For example, put any variables the user will have to fill in at the top of the script, preferably in a well marked section. Provide in-line comments where needed to provide human-readable scripts. Ensure that your descriptions of the code are not exclusively provided in comments, but that you also provide prose descriptions with further explanation than appears in comments.
Prefer /bin/sh to bash and avoid Bash-specific features when portability or cross-platform reuse are a concern. Use the shell and coreutils/standard Unix tools for small tasks; avoid introducing new dependencies purely for glue-language tasks unless the benefits are substantial. Prefer #!/usr/bin/env interpreter to #!/path/to/interpreter.
In general, use cron for scheduling recurring tasks, but systemd timers are also acceptable.
Filesystem Locations
Use your_server_ip and your_domain when possible instead of documentation IP ranges or example.com.
When downloading scripts or data, ensure that the user is in a writeable directory or paths are explicitly specified. For files which should be available for reference or reuse, use the user’s home directory, unless they belong in some standard well-defined path elsewhere on the filesystem (such as /opt or /etc). For throwaway files, use /tmp.
When a tutorial relies on a specific directory, be sure to provide the cd commands that route the reader to that directory before they run commands.
Web Servers
We recommend the Debian-style configuration directories for distributions that don’t structure it that way by default. Always test configuration changes (Apache uses sudo apachectl configtest, and Nginx uses sudo nginx -t). /var/www/html should be used as the document root for all web servers. Nginx’s /usr/share/nginx/html default should be changed because that directory is owned by and can potentially be modified by package updates. This is no longer a problem in Ubuntu 16.04, but will remain relevant for previous releases.
For Apache or Nginx tutorials, use dedicated virtual host blocks instead of editing the default config files. This route can avoid common mistakes and maintain the default files as the fallback configuration as intended.
Security
Encrypt and authenticate all connections between systems. Do not encourage (explicitly or implicitly) users to send credentials or transmit non-public data in the clear.
Specifically, passwords and key material must not be transmitted over unsecured connections. Database connections, logging, cluster management, and other services should ideally be encrypted at all times.
Web-based control panels must be served over HTTPS connections, and TLS/SSL should be used for services where it’s supported. All web servers should be HTTPS-enabled (or capable, at least). Use a certbot prerequisite to provide SSL certification. Public-facing services like plain HTTP are permissible, as users may still want or need to offer them, but should be strongly discouraged in the general case, especially for dynamic content. For articles that provide a plain HTTP connection, add a note or warning label to discourage plain HTTP and encourage HTTPS.
Avoid practices which constitute low- benefit security through obscurity or theatrics, like changing the default SSH port. Do configure a firewall. Our distro-specific recommendations are ufw for Ubuntu, iptables for Debian, and firewalld for CentOS. However, iptables is most consistent across platforms, and has many tools that hook into it.
SSH
Maintain the default SSH port as a norm. Changing the port should only be done in specific situations where that is a primary concern.
Disable password authentication and use key-only authentication for root or, alternatively, disable root login completely. Use strong SSH keys: at least 2048-bit RSA but recommended 4096; ECDSA is no longer recommended for technical reasons; and Ed25519 and elliptic curve algorithms are not widely supported enough.
Use passphrases for any interactive keys, but not for non-interactive processes. Set up or copy and change ownership on SSH keys from the root account to the user home directory. Install fail2ban where it’s practical.
Note that while SSH Agent Forwarding is necessary for normal workflows on platforms like CoreOS, it comes with some security concerns. Essentially, anyone with permissions on your host will be able to use the forwarding socket to connect to your local ssh-agent.
SSL/TLS
We strongly encourage the use of Let’s Encrypt for ease of use, and recommend TLS. Do use strong SSL security; look at https://cipherli.st/ (both modern and legacy recommendations).
For hosts without a domain name, we suggest a self-signed certificate of adequate strength. Again, look at https://cipherli.st/ plus the self-signed cert creation used in guides like this Self-Signed Certification on Nginx guide. Set up a strong Diffie-Hellman key when enabling encryption, as in Self-Signed SSL Certifications on Apache and Nginx.
VPN
We recommend VPNs as a solution for general encrypted communication between servers. VPNs become increasingly valuable when multiple services need to be protected between servers; instead of encrypting each service individually, all internal communication can be piped to the VPN. This approach is particularly useful if the services in question don’t support native encryption.
Conclusion
This is inherently an opinionated, living document, and will be updated often. Tech changes quickly and we at Centron do our best to keep up, but if you notice any errors or have feedback, we’ll be monitoring the comments below.
