I have multiple VPS servers hosted with different cloud providers, and regularly backing up server data has become a tricky problem. Normally, I would use NFS mounting, using s3fs-fuse to mount the S3 bucket to the local partition, and then write the files that need to be backed up to that partition.
However, the stability of S3 access has been very poor recently, even in AWS's home country, the United States. The partition mounted using s3fs-fuse is not stable for reading and writing, especially when dealing with a large number of files and continuous reading and writing.
Recently, I discovered that using AWS Storage Gateway allows the virtual machine to directly connect to the iscsi target and operate on the corresponding LUN and volume. The iscsi protocol, which uses a lower TCP/IP protocol than HTTP/S, is more stable compared to the s3fs method.
In addition, incoming traffic to AWS Storage Gateway is free, and only storage fees for S3 and EBS are charged.
Server version: Ubuntu 18.04 LTS
Device target: /dev/sdc
- Installing and configuring the iscsi initiator program
- Install open-iscsi and open-iscsi-utils
# Install open-iscsi and open-iscsi-utils
sudo apt-get install open-iscsi open-iscsi-utils
- Discover the iscsi target
# Discover the iscsi target
sudo iscsiadm -m discovery -t sendtargets -p XXX.XXX.XXX.XXX:3260
- Login to the target
# Login to the target
sudo iscsiadm -m node --targetname iqn.1997-05.com.amazon.XXXXXX -p XXX.XXX.XXX.XXX:3260 --login
- Mounting the hard disk
- Discover and partition/format the disk
fdisk -l
- Partition and format the LUN
fdisk /dev/sdc
mkfs.ext4 /dev/sdc1
- Mount to the specified location
sudo mkdir /mnt/backup
sudo mount /dev/sdc1 /mnt/backup
- Unmount the hard disk
sudo umount /mnt/backup
- Automatically connect to the LUN and mount the hard disk at startup
- Automatically login to the target. Modify /etc/iscsi/iscsid.conf to enable automatic login to the Gateway.
vi /etc/iscsi/iscsid.conf
# Enable automatic startup
node.startup = automatic
- Modify /etc/fstab to achieve automatic mounting at startup
# Get the UUID of the file system
tune2fs -l
# Modify /etc/fstab to achieve automatic mounting at startup
sudo vi /etc/fstab
# Add the following line to /etc/fstab
UUID=6dba31ff-xxxx-4430-bbd2-c1a932a53308 /backup ext4 \_netdev 0 0
A few things to note:
- If an unknown interruption occurs during the connection with AWS Storage Gateway, you can try forcing a device refresh.
udevadm test /sys/block/sdc
- If a connection timeout occurs, it is usually related to network quality. AWS recommends modifying iscsid.conf.
vi /etc/iscsi/iscsid.conf
# Modify the connection timeout parameters
node.session.timeo.replacement_timeout = 600
node.conn[0].timeo.noop_out_interval = 60
node.conn[0].timeo.noop_out_timeout = 600
Reference: Customize Linux iSCSI Settings