This domain name is tebibyte.io. You have to admit it’s pretty cool.
Tebibyte was originally only a storage server intended for my family and close friends. I was the one with more IT knowledge and free time, so I set up the server with a 2010 Pentium and 8 TB of storage. Those times were simply quite pleasant. In the following months I kept adding more features to it, such as single sign-on, email and other services. As we used it more, I started to appreciate the efficiency of such practice, in terms of cost, human labor, and privacy. When we upgraded our server hardware, we did not buy a prebuilt server largely because of the cost. Instead, we used another existing computer, pooled from our users. The additional cost was low.
And I only used very little time managing the server that benefits a number of users. The key to reducing cost and labor is good IT management. Even for such a scale as 20 users, I can imagine a huge difference between good management and a bad one. But this is also behind a great knowledge barrier. For example, when I did not know how to write systemd unit files, I could only use screen to keep a service running. However, the knowledge itself is less than key, but rather how it is systematically obtained and organized. This affects both high-level concepts and very specific software projects.
Thus, to greatly organize research, I have compiled this document as a learning catalog. Use this memo as a compass if you are quite new to server administration or if you are not systematic with it.
This document is always updated.
General Good Practice
Keep in mind that implementation-specific configurations are only the final boring step of server administration. Understand each step you do. Make decisions in an engineering sense.
The Architecture of Things
Web Services
There are a two major designs:
- A directory with resource files and PHP or other scripts that are called by the webserver when a path is visited.
- If the scripts are left out, the site just becomes purely static.
- These PHP scripts are stateless.
- Any scheduled jobs can only be triggered explicitly.
- A long-running program that binds to a port ready to directly accept requests.
- This port is usually proxied.
- Any scheduled jobs are run automatically.
Web services typically need you to configure a database and file storage location. All data consistency is enforced by the database and filesystem.
Identity, Authentication, and Authorization
Identity, authentication, and authorization are distinct yet closely interworking systems. In the past, some protocols transgress into the other two of these three problems. Today, it is clear that every system should treat identity, authentication, and authorization as different problems, each addresses by a dedicated subsystem.
Identity (Directory): Who are there? Does this identifier exist? Should only be accessible internally.
Authentication (OpenID Connect): Given an identifier, make sure only the correct agent can act as it Open to the Internet.
Authorization (OAuth): Given an authenticated user, and any resource, make allow/reject/etc. decisions, using information from the directory and database. Handled by the resource owner, open to the Internet.
Before talking about this I need to make the distinction between internal services (usually operated by your own organization) and external services (usually not operated by your own organization).
As a general rule, in internal services, all users have a presence by default. In external services, the presence of a user is opt-in. Therefore, it is important to note that they require different ways of managing the full list of users (directory; IdM). Internal services have prescribed access to the directory of your domain. It is usually wrong to configure internal services without direct access to the full directory. (Can you imagine having to open Google Drive once before someone can share with your new Google Drive account?) External services add users to their directories as users log in for the first time, and it is the responsibility of the said service to maintain a subdirectory.
Identity
There is only one directory standard to choose from: LDAP.
Authentication
There are a large number of IdPs available, many of which do not need adapter code written.
How we did it: Tebibyte uses OpenID Connect, Keycloak implementation.
Recent candidates are: Ory, Kanidm, Authentik, Authelia.
Ideally, all authentication should be managed by a central service. In that case, when the user logs out from the IdP, all web apps subsequently recognize the user as logged out. The IdP is also able to keep log of all authentication events. Note that in such an architecture, if the IdP is slow to respond, the web service also slows down.
If you provide binary client apps, they need to support OpenID Connect separately from the web apps.
Authorization
Authorization is only about making a decision based on authenticated information. At this point, it is already certain that the logged-in user is the actual user.
If you think about it, authorization is handled by the owner of a resource or on behalf of it. The owner usually means the app server.
Checking for Authorization
The Authorization: Bearer token is sent to the app server. The app server queries a central identity server to find out (introspect) if it is valid and whose it is, then make authorization decisions.
Other Protocols
The authentication mode varies from program to program. Bearer authentication has become the industry standard.
How we did it: Our SMTP and IMAP ports use bearer authentication (both OAUTHBEARER and XOAUTH2) thanks to Postfix and Dovecot.
System Accounts
UNIX system accounts can use an IdP backend with a PAM module.
How we did it: We are only using this after migrating all system users to another machine. It can be problematic to let a service manage the system that supports itself.
System Organization
Learn the File Hierarchy Standard (FHS) or whatever standard for your system. Take note of any locations that may be incompliant or difficult to remember.
Use you package manager. Use other package managers (such as Python’s PIP) carefully and not for the whole system.
Create the service users and set permissions for their directories. Use your init manager (systemd or otherwise) to manage startup dependencies.
Storage
User data storage can either be a mounted system volume on the same server as the service software, or a separate server typically through some object storage interface (which forms three-sided connections: web service, storage, user). If scaling is expected, it can be very suitable to first deploy a dedicated storage server on the same machine to make any future migrations easier. If so, assign whole disks to a storage server, not directories.
If redundancy is a concern, use a disk array. But a disk array is not a backup. You should ideally keep a backup somewhere easily accessible, and another backup at a different location.
Database
If you are using a UNIX-like server and the database is on the same machine as the web app, consider using a UNIX socket. This not only handles connection authentication on some systems but also achieves zero copy.
Make sure to put your database on an SSD, and do not forget to back it up.
Network
If you are using a single machine for everything, try to imagine that each service is actually on a separate machine. In your configuration files, before cramming everything onto a single apex domain (or even worse, localhost (or even worse, 127.0.0.1)), assign them wisely and write the correct subdomains for each component in your very first configuration file. (You get to decide what is considered a component.) Use CNAME in DNS. This way, when you only migrate certain services, nothing will break.
TLS
Use TLS for all possible protocols and preferably disable insecure versions. Your Let’s Encrypt certificate is applicable to not only the HTTPS port but all TLS ports. Make this a natural practice. Security works better when everyone is used to it.
Some ports use “StartTLS”, a command the client sends to establish TLS connection on an existing plaintext connection.
When you update the TLS certificates, reload all services that use the certificates, like the web and mail servers.
Use TLS outgoing requests even for same-machine connections across components, such as OpenID Connect login requests from the web service to the IdP. This way, when you only migrate certain services, nothing will break.
HTTP Proxy
Pay attention to proxy-specific headers. Also keep track of port numbers. Disable external access of naked ports.
Firewall
Use them whenever you need.
The modern email security framework consists of DKIM, SPF, and DMARC. Other online spam detection services can be an addition.
Web Server
With a correct web server configuration, the served directory structure can be very different from the structure stored on the server. An example is this website, where there are actually no /~user/ directories stored on the server.
In addition to the main configuration files, there are per-directory .ht* files for fine control. Use this to avoid a thousand location cases in your main configuration files just for access control.