A lazy little “swapping” trick to save up on server storage costs.

At work, we have:

• An Nginx server that serves a lot of files from /var/www/{somedirectories}
• A few web services that provide different kinds of metadata on those {somedirectories}

In an ideal world, storage would have been cheap and fast. There would have been no choices to make. But, in this timeline, HDDs are still a lot cheaper than NVMe SSDs. And our cloud provider did not even offer NVMe drives as large as our requirements.

We purchased 2 drives, and mounted them as:

• /mnt/storage - A small NVMe drive that can hold up to 6 months of our data.
• /mmt/archive - A large hard drive that can hold up to 10 years of our data.

As most of our users are only interested in the files from the latest month or two, Overlayfs to the rescue!

Now, we mounted /var/www as /mnt/storage/var/www (upper dir) + /mnt/archive/var/www (lowerdir)

Aand wrote a little python script that regularly moved older files from /mnt/storage to /mnt/archive.

0 0 * * * /opt/services/archiver/archive.py && mount -o remount /var/www

I know. There’s also LVM. And CDNs. But this was the simplest and cheapest trick that fit our needs and took me 15 minutes to set up. With no code changes anywhere else.

Maybe some day I will write a more customizable cachefs, that does all this transparently. But for now, I am happy with this solution.