How to Set Up a Public Mirror for Multiple Projects (Rocky Linux, CentOS, Debian, Ubuntu, and More)

This guide will help you set up a public mirror server that can host multiple open-source project mirrors (e.g., Rocky Linux, CentOS, Debian, Ubuntu). The mirror will support HTTP, HTTPS, and RSYNC with matching URL paths, use a dedicated storage drive, and allow future scalability for additional projects.

Step 1: Prepare the Server

Prerequisites

1. Server Requirements:
   • At least 4TB of storage to handle multiple projects (adjust based on needs).
   • High-bandwidth network connection (1Gbps or more recommended).
2. Software Requirements:
   • Nginx
   • Rsync
   • Certbot (for Let’s Encrypt SSL certificates)
3. Domain Name:
   You need a registered domain like mirror.example.com with DNS pointing to your server.

Step 2: Set Up the Separate Drive

Identify the Drive

Run the following command to locate the new drive:

lsbl

Assume the drive is /dev/sdb.

Partition and Format the Drive

Create a new partition and format it as ext4:

sudo fdisk /dev/sdb
# Follow prompts to create a new partition.

sudo mkfs.ext4 /dev/sdb1

Mount the Drive

Mount the drive at /mnt/mirrors:

sudo mkdir -p /mnt/mirrors
sudo mount /dev/sdb1 /mnt/mirrors

Persist the Mount

To ensure the drive is mounted automatically at boot, edit /etc/fstab:

sudo nano /etc/fstab

Add the following line:

/dev/sdb1  /mnt/mirrors  ext4  defaults  0  2

Set Permissions

Set ownership for the mount point:

sudo chown -R $USER:$USER /mnt/mirrors

Step 3: Create Directory Structure for Multiple Projects

Create directories for each project under /mnt/mirrors:

sudo mkdir -p /mnt/mirrors/{rocky,centos,debian,ubuntu}

This structure ensures easy management:

• /mnt/mirrors/rocky for Rocky Linux
• /mnt/mirrors/centos for CentOS
• /mnt/mirrors/debian for Debian
• /mnt/mirrors/ubuntu for Ubuntu

Step 4: Sync Files with Rsync

Rsync Commands for Each Project

Use the following commands to sync files for each project:

Rocky Linux:

rsync -avzH --delete --progress rsync://mirrors.rockylinux.org/rocky/ /mnt/mirrors/rocky/

CentOS:

rsync -avzH --delete --progress rsync://mirror.centos.org/centos/ /mnt/mirrors/centos/

Debian:

rsync -avzH --delete --progress rsync://ftp.debian.org/debian/ /mnt/mirrors/debian/

Ubuntu:

rsync -avzH --delete --progress rsync://archive.ubuntu.com/ubuntu/ /mnt/mirrors/ubuntu/

Automate Rsync with Cron

To keep the mirror updated, configure a cron job:

crontab -e

Add the following:

0 2 * * * rsync -avzH --delete rsync://mirrors.rockylinux.org/rocky/ /mnt/mirrors/rocky/ >> /var/log/rsync_rocky.log 2>&1
0 3 * * * rsync -avzH --delete rsync://mirror.centos.org/centos/ /mnt/mirrors/centos/ >> /var/log/rsync_centos.log 2>&1
0 4 * * * rsync -avzH --delete rsync://ftp.debian.org/debian/ /mnt/mirrors/debian/ >> /var/log/rsync_debian.log 2>&1
0 5 * * * rsync -avzH --delete rsync://archive.ubuntu.com/ubuntu/ /mnt/mirrors/ubuntu/ >> /var/log/rsync_ubuntu.log 2>&1

Step 5: Configure Nginx

Install Nginx

sudo dnf install -y nginx

Nginx Configuration for Multiple Projects

Create a configuration file for the mirrors:

sudo nano /etc/nginx/conf.d/mirrors.conf

Add the following:

server {
    listen 80;
    server_name mirror.example.com;

    # Redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name mirror.example.com;

    ssl_certificate /etc/letsencrypt/live/mirror.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mirror.example.com/privkey.pem;

    root /mnt/mirrors;
    index index.html;

    location / {
        autoindex on;
        autoindex_exact_size off;
        autoindex_localtime on;
    }

    # Custom locations for each project
    location /rocky {
        alias /mnt/mirrors/rocky;
    }

    location /centos {
        alias /mnt/mirrors/centos;
    }

    location /debian {
        alias /mnt/mirrors/debian;
    }

    location /ubuntu {
        alias /mnt/mirrors/ubuntu;
    }

    access_log /var/log/nginx/mirrors_access.log;
    error_log /var/log/nginx/mirrors_error.log;
}

Test and Reload Nginx

Test the configuration:

sudo nginx -t

Reload the Nginx service:

sudo systemctl restart nginx

Step 6: Enable HTTPS with Let’s Encrypt

Install Certbot

sudo dnf install -y certbot python3-certbot-nginx

Obtain SSL Certificate

Run the following to secure your mirror:

sudo certbot --nginx -d mirror.example.com

Test Certificate Renewal

Certbot should automatically renew certificates, but you can test manually:

sudo certbot renew --dry-run

Step 7: Set Up Rsync for Public Access

Install Rsync

sudo dnf install -y rsync

Configure rsyncd.conf

Edit the Rsync configuration file:

sudo nano /etc/rsyncd.conf

Add:

uid = nobody
gid = nobody
use chroot = yes
max connections = 50
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock

[rocky]
path = /mnt/mirrors/rocky
comment = Rocky Linux Mirror
read only = yes

[centos]
path = /mnt/mirrors/centos
comment = CentOS Mirror
read only = yes

[debian]
path = /mnt/mirrors/debian
comment = Debian Mirror
read only = yes

[ubuntu]
path = /mnt/mirrors/ubuntu
comment = Ubuntu Mirror
read only = yes

Restart Rsync Service

sudo systemctl enable --now rsyncd

Verify Rsync

Test Rsync:

rsync mirror.example.com::

Step 8: Test and Monitor the Mirror

1. Test URLs:
   • HTTP/HTTPS:
     • https://mirror.example.com/rocky
     • https://mirror.example.com/centos
   • Rsync:
     • rsync://mirror.example.com/rocky
2. Monitor Logs:
   • Nginx Logs: tail -f /var/log/nginx/mirrors_access.log
   • Rsync Logs: tail -f /var/log/rsyncd.log
3. Check Disk Space: df -h /mnt/mirrors

Step 9: Add New Projects

To add a new project:

1. Create a directory under /mnt/mirrors.
2. Configure a new Rsync command for the project.
3. Add an Nginx location block for the project.
4. Add a section to rsyncd.conf.

Repeat these steps for any number of projects.

This setup is scalable, robust, and ensures matching paths for HTTP, HTTPS, and RSYNC.