GwynSoft

{jcomments off}

Part 2: Prepare Ubuntu LTS To Accept VirtualMin

Introduction

In this article I set up an AWS EC2 Instance from scratch to run VirtualMin, an open-source virtual server management tool. While Virtumin can be set up on an in-use server, it requires a lot of reconfiguration so it is simpler to set up a new server and proceed from there. At the end of this article we shall have a clean install of VirtualMin ready to accept new websites. This is part 2 where we prepare our Ubuntu system to accept VirtualMin.

The article is broken into 4 main steps:

1. Set up an AWS instance with sufficient EBS storage space
2. Prepare Ubuntu LTS to accept VirtualMin This page
3. Download and install VirtualMin
4. Prepare VirtualMin to run websites

The article is based on various pages by numerous authors who I thank for their invaluable insights, as well as my own experiences. Any errors are most likely mine. I give no guarantee that what you find in these pages will work on your setup - as usual YMMV.

In this series of articles I shall be using an FQDN of myserver.example.com and a user called myuser. The server will reside at the fictitious public ip 12.34.56.78. Make sure you substitute your own FQDN, user and ip where necessary - the example values given will not work.

---

Set Up some swap space

Introduction

One of the easiest way of increasing the responsiveness of your server and guarding against out-of-memory errors in applications is to add some swap space. In this guide, we will cover how to add a swap file to an Ubuntu 17.04 server.

What is Swap?

Swap is an area on a hard drive that has been designated as a place where the operating system can temporarily store data that it can no longer hold in RAM. Basically, this gives you the ability to increase the amount of information that your server can keep in its working "memory", with some caveats. The swap space on the hard drive will be used mainly when there is no longer sufficient space in RAM to hold in-use application data.

The information written to disk will be significantly slower than information kept in RAM, but the operating system will prefer to keep running application data in memory and use swap for the older data. Overall, having swap space as a fall back for when your system's RAM is depleted can be a good safety net against out-of-memory exceptions on systems with non-SSD storage available.

Check the System for Swap Information

Before we begin, we can check if the system already has some swap space available. It is possible to have multiple swap files or swap partitions, but generally one should be enough.

We can see if the system has any configured swap by typing:

• sudo swapon --show

If you don't get back any output, this means your system does not have swap space available currently.

You can verify that there is no active swap using the free utility:

• free -h

Output

• total used free shared buff/cache available
• Mem: 488M 36M 104M 652K 348M 426M
• Swap: 0B 0B 0B

As you can see in the "Swap" row of the output, no swap is active on the system.

Check Available Space on the Hard Drive Partition

The most common way of allocating space for swap is to use a separate partition devoted to the task. However, altering the partitioning scheme is not always possible. We can just as easily create a swap file that resides on an existing partition.

Before we do this, we should check the current disk usage by typing:

• df -h

Output

• Filesystem Size Used Avail Use% Mounted on
• udev 992M 0 992M 0% /dev
• tmpfs 200M 3.1M 197M 2% /run
• /dev/xvda1 25G 1.3G 23G 6% /
• tmpfs 1000M 0 1000M 0% /dev/shm
• tmpfs 5.0M 0 5.0M 0% /run/lock
• tmpfs 1000M 0 1000M 0% /sys/fs/cgroup
• tmpfs 200M 0 200M 0% /run/user/1000

The device under /dev is our disk in this case. We have plenty of space available in this example (only 1.3G used). Your usage will probably be different.

Although there are many opinions about the appropriate size of a swap space, it really depends on your personal preferences and your application requirements. Generally, an amount equal to or double the amount of RAM on your system is a good starting point. Another good rule of thumb is that anything over 4G of swap is probably unnecessary if you are just using it as a RAM fallback.

Create a Swap File

Now that we know our available hard drive space, we can go about creating a swap file within our filesystem. We will create a file of the swap size that we want called swapfile in our root (/) directory.

The best way of creating a swap file is with the fallocate program. This command creates a file of a preallocated size instantly.

Since the server in our example has 2GB of RAM, we will create a 4 Gigabyte file in this guide. Adjust this to meet the needs of your own server:

• sudo fallocate -l 4G /swapfile

We can verify that the correct amount of space was reserved by typing:

• ls -lh /swapfile
• -rw-r--r-- 1 root root 4.0G Aug 25 12:59 /swapfile

Our file has been created with the correct amount of space set aside.

Enabling the Swap File

Now that we have a file of the correct size available, we need to actually turn this into swap space.

First, we need to lock down the permissions of the file so that only the users with root privileges can read the contents. This prevents normal users from being able to access the file, which would have significant security implications.
Make the file only accessible to root by typing:

• sudo chmod 600 /swapfile

Verify the permissions change by typing:

• ls -lh /swapfile

Output

• -rw------- 1 root root 4.0G Aug 25 12:59 /swapfile

As you can see, only the root user has the read and write flags enabled.

We can now mark the file as swap space by typing:

• sudo mkswap /swapfile

Output

• Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
• no label, UUID=6f7f4bcc-1942-42b0-9efc-e29ddd7e9eb9

After marking the file, we can enable the swap file, allowing our system to start utilizing it:

• sudo swapon /swapfile

We can verify that the swap is available by typing:

• sudo swapon --show

Output

• NAME TYPE SIZE USED PRIO
• /swapfile file 4G 0B -1

We can check the output of the free utility again to corroborate our findings:

• free -h

Output

• total used free shared buff/cache available
• Mem: 2.0G 43M 1.7G 3.1M 181M 1.8G
• Swap: 4.0G 0B 4.0G

Our swap has been set up successfully and our operating system will begin to use it as necessary.

Make the Swap File Permanent

Our recent changes have enabled the swap file for the current session. However, if we reboot, the server will not retain the swap settings automatically. We can change this by adding the swap file to our /etc/fstab file.

Back up the /etc/fstab file in case anything goes wrong:

• sudo cp /etc/fstab /etc/fstab.bak

You can add the swap file information to the end of your /etc/fstab file by typing:

• echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Tweak your Swap Settings

There are a few options that you can configure that will have an impact on your system's performance when dealing with swap.

Adjusting the Swappiness Property

The swappiness parameter configures how often your system swaps data out of RAM to the swap space. This is a value between 0 and 100 that represents a percentage.

With values close to zero, the kernel will not swap data to the disk unless absolutely necessary. Remember, interactions with the swap file are "expensive" in that they take a lot longer than interactions with RAM and they can cause a significant reduction in performance. Telling the system not to rely on the swap much will generally make your system faster.

Values that are closer to 100 will try to put more data into swap in an effort to keep more RAM space free. Depending on your applications' memory profile or what you are using your server for, this might be better in some cases.

We can see the current swappiness value by typing:

• cat /proc/sys/vm/swappiness

Output

• 60

For a Desktop, a swappiness setting of 60 is not a bad value. For a server, you might want to move it closer to 0.

We can set the swappiness to a different value by using the sysctl command.

For instance, to set the swappiness to 10, we could type:

• sudo sysctl vm.swappiness=10

Output

• vm.swappiness = 10

This setting will persist until the next reboot. We can set this value automatically at restart by adding the line to our /etc/sysctl.conf file:

• sudo nano /etc/sysctl.conf

At the bottom, you can add:

• ...
• vm.swappiness=10

Save and close the file when you are finished.

Adjusting the Cache Pressure Setting

Another related value that you might want to modify is the vfs_cache_pressure. This setting configures how much the system will choose to cache inode and dentry information over other data.

Basically, this is access data about the filesystem. This is generally very costly to look up and very frequently requested, so it's an excellent thing for your system to cache. You can see the current value by querying the proc filesystem again:

• cat /proc/sys/vm/vfs_cache_pressure

Output

• 100

As it is currently configured, our system removes inode information from the cache too quickly. We can set this to a more conservative setting like 50 by typing:

• sudo sysctl vm.vfs_cache_pressure=50

Output

• vm.vfs_cache_pressure = 50

Again, this is only valid for our current session. We can change that by adding it to our configuration file like we did with our swappiness setting:

• sudo nano /etc/sysctl.conf

At the bottom, add the line that specifies your new value:

• ...
• vm.vfs_cache_pressure=50

Save and close the file when you are finished.

Conclusion

Following the steps in this guide will give you some breathing room in cases that would otherwise lead to out-of-memory exceptions. Swap space can be incredibly useful in avoiding some of these common problems.
If you are running into OOM (out of memory) errors, or if you find that your system is unable to use the applications you need, the best solution is to optimize your application configurations or upgrade your server.

Check your hostname

Are you sure you set a fully qualified domain name for your system? This one is really important. Find out with the following command:

• sudo hostname -f

If it is fully qualified, continue to the next step. If not, read up on fully qualified domain names before proceeding. You'll thank me later.

Fully qualified domain name

If your system does not have a fully qualified domain name (FQDN), the installer will stop and ask you to choose one. This is mandatory because many services rely on having a fully qualified domain name in order to function. Mail, in particular, but some Apache configurations and many of the Virtualmin-created configuration files, also require a valid fully qualified domain name to function correctly. A fully qualified domain name is one of the form "www.virtualmin.com", or simply "virtualmin.com". We recommend you choose a name that is not one for which you will be receiving mail, in order to simplify later configuration. A good choice is to use a name server designator, such as "ns1.virtualmin.com". Some customers also choose something like "host1.virtualmin.com" or "primary.virtualmin.com". Any of these would be valid and would satisfy the install script and the services that rely on this option. The install script will add this name to /etc/hosts, which will satisfy all local services. It is even better if this name resolves correctly when looked up from outside of the system--this requires the name be added to your DNS zone for the second level domain. If the Virtualmin server you are installing will be the authoritative name server for this zone, you can later use Webmin to add a record for this name to the zone.

Add an EBS disk to store user data

We shall keep as much user data as possible on an external device. For the most part the data is all kept in /home, so we should set up an area for this; we also want to keep databases and our local web directory externally, so the same applies to them.

Structure:

/mnt/data/
home
mysql
mysql
mysql-files
mysql-keyring
mysql-upgrade

We will need about 50GB – 45GB for home and 5GB for mysql.

To create an EBS volume using the console

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

From the navigation bar, select the region in which you would like to create your volume. This choice is important because some Amazon EC2 resources can be shared between regions, while others can't. For more information, see Resource Locations.

In the navigation pane, choose ELASTIC BLOCK STORE, Volumes.

Choose Create Volume.

For Volume Type, choose ‘General Purpose SSD (GP2). For more information, see Amazon EBS Volume Types.

For Size (GiB), type the size of the volume (50GB).

For Availability Zone, choose the Availability Zone in which to create the volume (eu-central-1b). EBS volumes can only be attached to EC2 instances within the same Availability Zone.

(Optional) To create an encrypted volume, select the Encrypted box and choose the master key you want to use when encrypting the volume. You can choose the default master key for your account, or you can choose any customer master key (CMK) that you have previously created using the AWS Key Management Service. Available keys are visible in the Master Key menu, or you can paste the full ARN of any key that you have access to. For more information, see the AWS Key Management Service Developer Guide.

Note

Encrypted volumes can only be attached to selected instance types. For more information, see Supported Instance Types.

(Optional) Choose Create additional tags to add tags to the volume. For each tag, provide a tag key and a tag value.

Choose Create Volume.

You can attach an EBS volume to one of your instances that is in the same Availability Zone as the volume.

Prerequisites

Determine the device names that you'll use. For more information, see Device Naming on Linux Instances.

Determine how many volumes you can attach to your instance. For more information, see Instance Volume Limits.

If a volume is encrypted, it can only be attached to an instance that supports Amazon EBS encryption. For more information, see Supported Instance Types.

If a volume has an AWS Marketplace product code:

The volume can only be attached to a stopped instance.

You must be subscribed to the AWS Marketplace code that is on the volume.

The configuration (instance type, operating system) of the instance must support that specific AWS Marketplace code. For example, you cannot take a volume from a Windows instance and attach it to a Linux instance.

AWS Marketplace product codes are copied from the volume to the instance.

To attach an EBS volume to an instance using the console

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

In the navigation pane, choose Volumes.

Select a volume and choose Actions > Attach Volume.

In the Attach Volume dialog box, start typing the name or ID of the instance to attach the volume to for Instance, and select it from the list of suggestion options (only instances that are in the same Availability Zone as the volume are displayed).

You can keep the suggested device name, or enter a different supported device name.

Important

The block device driver for the instance assigns the actual volume name when mounting the volume, and the name assigned can be different from the name that Amazon EC2 recommends.

Choose Attach.

Connect to your instance and make the volume available. For more information, see Making an Amazon EBS Volume Available for Use.

To make an EBS volume available for use on Linux

Connect to your instance using SSH. For more information, see Step 2: Connect to Your Instance.

Depending on the block device driver of the kernel, the device might be attached with a different name than what you specify. For example, if you specify a device name of /dev/sdh, your device might be renamed /dev/xvdh or /dev/hdh by the kernel; in most cases, the trailing letter remains the same.

Use the lsblk command to view your available disk devices and their mount points (if applicable) to help you determine the correct device name to use.

• lsblk

• NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
• xvda 202:0 0 25G 0 disk
• └─xvda1 202:1 0 25G 0 part /
• xvdf 202:80 0 50G 0 disk

The output of lsblk removes the /dev/ prefix from full device paths. In this example, /dev/xvda1 is mounted as the root device (note that MOUNTPOINT is listed as /, the root of the Linux file system hierarchy), and /dev/xvdf is attached, but it has not been mounted yet.

Determine whether you need to create a file system on the volume. New volumes are raw block devices, and you need to create a file system on them before you can mount and use them. Volumes that have been restored from snapshots likely have a file system on them already; if you create a new file system on top of an existing file system, the operation overwrites your data. Use the sudo file -s device command to list special information, such as file system type.

• sudo file -s /dev/xvdf

/dev/xvdf: data

If the output of the previous command shows simply data for the device, then there is no file system on the device and you need to create one. You can go on to Step 4. If you run this command on a device that contains a file system, then your output will be different.

• sudo file -s /dev/xvdf1
• /dev/xvda1: Linux rev 1.0 ext4 filesystem data, UUID=1701d228-e1bd-4094-a14c-8c64d6819362 (needs journal recovery) (extents) (large files) (huge files)

In the previous example, the device contains Linux rev 1.0 ext4 filesystem data, so this volume does not need a file system created (you can skip Step 4 if your output shows file system data).

(Conditional) Use the following command to create an ext4 file system on the volume. Substitute the device name (such as /dev/xvdf) for device_name. Depending on the requirements of your application or the limitations of your operating system, you can choose a different file system type, such as ext3 or XFS.

Warning

This step assumes that you're mounting an empty volume. If you're mounting a volume that already has data on it (for example, a volume that was restored from a snapshot), don't use mkfs before mounting the volume (skip to the next step instead). Otherwise, you'll format the volume and delete the existing data.

Identify the New Disk on the System

Before we set up the drive, we need to be able to properly identify it on the server.

If this is a completely new drive, the easiest way to find it on your server may be to look for the absence of a partitioning scheme. If we ask parted to list the partition layout of our disks, it will give us an error for any disks that don't have a valid partition scheme. This can be used to help us identify the new disk:

• sudo parted -l | grep Error

You should see an unrecognized disk label error for the new, unpartitioned disk:

Error: /dev/sdf: unrecognised disk label

You can also use the lsblk command and look for a disk of the correct size that has no associated partitions:

lsblk

• NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
• sda 8:0 0 100G 0 disk
• vda 253:0 0 20G 0 disk
• └─vda1 253:1 0 20G 0 part /

Warning

Remember to check lsblk in every session before making changes. The /dev/sd* and /dev/hd* disk identifiers will not necessarily be consistent between boots, which means there is some danger of partitioning or formatting the wrong disk if you do not verify the disk identifier correctly.

Consider using more persistent disk identifiers like /dev/disk/by-uuid, /dev/disk/by-label, or /dev/disk/by-id. See our introduction to storage concepts and terminology in Linux article for more information.
When you know the name the kernel has assigned your disk, you can partition your drive.

Partition the New Drive

As mentioned in the introduction, we'll create two partitions in this guide.

Choose a Partitioning Standard

To do this, we first need to specify the partitioning standard we wish to use. GPT is the more modern partitioning standard, while the MBR standard offers wider support among operating systems. If you do not have any special requirements, it is probably better to use GPT at this point.

To choose the GPT standard, pass in the disk you identified like this:

• sudo parted /dev/xvdf mklabel gpt

Create the New Partition

Once the format is selected, you can create a partition spanning the entire drive by typing:

• sudo parted -a opt /dev/xvdf mkpart primary ext4 0% 90%
• sudo parted -a opt /dev/xvdf mkpart primary ext4 90% 100%

If we check lsblk, we should see the new partition available:

lsblk

• NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
• xvda 202:0 0 25G 0 disk
• └─xvda1 202:1 0 25G 0 part /
• xvdf 202:80 0 50G 0 disk
• ├─xvdf1 202:81 0 45G 0 part
• └─xvdf2 202:82 0 5G 0 part

Create a Filesystem on the New Partition

Now that we have a partition available, we can format it as an Ext4 filesystem. To do this, pass the partition to the mkfs.ext4 utility.

We can add a partition label by passing the -L flag. Select a name that will help you identify this particular drive:

note

Make sure you pass in the partition and not the entire disk. In Linux, disks have names like sda, sdb, hda, etc. The partitions on these disks have a number appended to the end. So we would want to use something like sda1 and not sda.

• sudo mkfs.ext4 -L HOME /dev/xvdf1
• sudo mkfs.ext4 -L MYSQL /dev/xvdf2

If you want to change the partition label at a later date, you can use the e2label command

• sudo e2label /dev/sdf1 newlabel

You can see all of the different ways to identify your partition with lsblk. We want to find the name, label, and UUID of the partition.

Some versions of lsblk will print all of this information if we type:

• sudo lsblk --fs

If your version does not show all of the appropriate fields, you can request them manually:

• sudo lsblk -o NAME,FSTYPE,LABEL,UUID,MOUNTPOINT

You should see something like this. The highlighted output indicate different methods you can use to refer to the new filesystem:

• NAME FSTYPE LABEL UUID MOUNTPOINT
• xvda
• └─xvda1 ext4 cloudimg-rootfs 3e13556e-d28d-407b-bcc6-97160eafebe1 /
• xvdf
  ├─xvdf1 ext4 HOME e346c928-0c1b-4acd-a8e6-8bdddc6f4805
• └─xvdf2 ext4 MYSQL 1e965d41-2099-4828-adbc-4123b629b1dc

Mount the New Filesystem

Now, we can mount the filesystem for use.

The Filesystem Hierarchy Standard recommends using /mnt or a subdirectory under it for temporarily mounted filesystems. It makes no recommendations on where to mount more permanent storage, so you can choose whichever scheme you'd like. For this tutorial, we'll mount the drive under /mnt/data.

Create the directory by typing:

• sudo mkdir /mnt/home /mnt/mysql

Mounting the Filesystem Temporarily

You can mount the filesystem temporarily by typing:

• sudo mount -o defaults /dev/xvdf1 /mnt/home
• sudo mount -o defaults /dev/xvdf2 /mnt/mysql

Move /home to /mnt/home and create mysql directories

• sudo su

• cp -a /home /mnt
• mkdir /mnt/mysql/mysql
• mkdir /mnt/mysql/mysql-files
• mkdir /mnt/mysql/mysql-keyring
• exit

Mounting the Filesystem Automatically at Boot

If you wish to mount the filesystem automatically each time the server boots, adjust the /etc/fstab file:

• sudo nano /etc/fstab

Earlier, we issued a sudo lsblk --fs command to display three filesystem identifiers for our filesystem. We can use any of these in this file. We've used the partition label below, but you can see what the lines would look like using the other two identifiers in the commented out lines:

If you ever intend to boot your instance without this volume attached (for example, so this volume could move back and forth between different instances), you should add the nofail mount option that allows the instance to boot even if there are errors in mounting the volume.

/etc/fstab

• LABEL=cloudimg-rootfs / ext4 defaults,discard 0 0
• UUID=e346c928-0c1b-4acd-a8e6-8bdddc6f4805 /home ext4 defaults,nofail 0 2
• UUID=1e965d41-2099-4828-adbc-4123b629b1dc /mnt/mysql ext4 defaults,nofail 0 2
• /swapfile none swap sw 0 0

Note

You can learn about the various fields in the /etc/fstab file by typing man fstab. For information about the mount options available for a specific filesystem type, check man [filesystem] (like man ext4). For now, the mount lines above should get you started.

For SSDs, the discard option is sometimes appended to enable continuous TRIM. There is debate over the performance and integrity impacts of performing continuous TRIM in this manner, and most distributions include method of performing periodic TRIM as an alternative.

Save and close the file when you are finished.

If you did not mount the filesystem previously, you can now mount it by typing:

• sudo mount -a

(Optional) If you are unsure how to correct /etc/fstab errors, you can always restore your backup /etc/fstab file with the following command.

• sudo mv /etc/fstab.orig /etc/fstab

Review the file permissions of your new volume mount to make sure that your users and applications can write to the volume. For more information about file permissions, see File security at The Linux Documentation Project.

Reboot the system

• sudo reboot

At this point, our Basic server setup is complete

{jcomments on}