
Thank you for reading, keep learning and keep growing.
Reactions
No comments yet.
Building backup storage is one of the important things to consider when developing an application, especially an application that is already running in a production environment. Why? The goal is simple: to ensure that data can still be recovered when the primary server experiences failure, disruption, or data loss.
Imagine an application that is already being used by many users, and suddenly the production database experiences a problem or some of its data becomes corrupted. Damn...! The application may start throwing errors, services may become unavailable, and in certain situations, we could even lose data. In addition, the recovery process may take a significant amount of time if we do not have a proper backup mechanism.
Therefore, backup storage becomes an important component of a production application architecture. A backup provides a copy of data that can be used for recovery when something goes wrong with the primary database or storage.
There are various approaches that can be used to store backups. In this article, we will discuss two storage options that can be used in a backup architecture: Network Attached Storage (NAS) and Cloud Object Storage. We will also discuss how the 3-2-1 backup strategy can be applied to reduce the risk of data loss.
NAS (Network Attached Storage) is a storage server or appliance connected to a network. NAS can be used to store:
NAS is generally designed as centralized storage with capacity that can be adjusted based on your needs. It can be accessed by multiple servers or devices over the network according to the permissions granted to them. NAS is suitable when an organization wants to manage its own storage and have greater control over its data and infrastructure.
NFS (Network File System) is a protocol that allows a server to access files or directories on another machine over a network.
NFS is not a storage device. NAS is the storage server, while NFS is one of the protocols that can be used to access files stored on a NAS.
NAS = storage server
NFS = network file access protocol
Mount point = local directory tempat NFS share di-mountNAS = storage server
NFS = network file access protocol
Mount point = local directory tempat NFS share di-mountFor example, a NAS provides an NFS share for storing backups. The application server then mounts the NFS share to a local directory such as /mnt/backup-nas.
From the application server's perspective, this directory can be used like a regular local directory, but the data written to it is actually stored on the NAS over the network.
First, we need to configure the backup tool. A backup tool is software, a script, or a service responsible for creating and managing the backup process.
In this example, the backup tool first creates a database backup on local storage. Once the backup process is complete, the backup file is copied to the NFS mount point, which stores the file on the NAS.
Example directory configuration:
Local backup: /var/lib/backup-tool/backups
NFS mount: /mnt/backup-nas
Destination: /mnt/backup-nas/application/backupsLocal backup: /var/lib/backup-tool/backups
NFS mount: /mnt/backup-nas
Destination: /mnt/backup-nas/application/backupsThe backup flow looks like this:
Production Database
ā
ā¼
Backup Tool
(e.g., backup script)
ā
ā¼
Local Storage
ā
ā Copy / Sync
ā¼
NFS Mount Point
ā
ā NFS
ā¼
NASProduction Database
ā
ā¼
Backup Tool
(e.g., backup script)
ā
ā¼
Local Storage
ā
ā Copy / Sync
ā¼
NFS Mount Point
ā
ā NFS
ā¼
NASFor example, a backup script can execute a utility such as pg_dump for PostgreSQL, mysqldump for MySQL, or another database-specific backup command. The resulting backup file is temporarily stored on local storage before being copied to the NAS.
Each application should have a dedicated backup directory on the NAS with appropriate permissions. Before performing write or delete operations, the backup process should also validate the destination path to reduce the risk of accidentally modifying or deleting data belonging to another application's directory.
Cloud Object Storage is a cloud-based storage service that stores data as objects inside buckets. This type of storage is commonly used to store database backups, files, logs, archives, and other types of unstructured data.
A bucket is the main container used to store objects in object storage.
A simple way to imagine it:
Object Storage = warehouse complex
Bucket = one warehouse
Object = item inside the warehouseObject Storage = warehouse complex
Bucket = one warehouse
Object = item inside the warehouseCloud storage is a general term for data storage services provided by cloud providers. In general, cloud storage can be categorized into Object Storage, File Storage, and Block Storage.
Simply put, the three can be distinguished as follows:
Cloud Storage
ā
āāā Object Storage
ā āāā Bucket + Object
ā Access: API / SDK
ā
āāā File Storage
ā āāā File + Directory
ā Access: NFS / SMB
ā
āāā Block Storage
āāā Virtual Disk / Block Device
Access: attached to serverCloud Storage
ā
āāā Object Storage
ā āāā Bucket + Object
ā Access: API / SDK
ā
āāā File Storage
ā āāā File + Directory
ā Access: NFS / SMB
ā
āāā Block Storage
āāā Virtual Disk / Block Device
Access: attached to serverThe easiest way to remember them:
Object Storage ā "I have a bucket"
File Storage ā "I have a shared folder"
Block Storage ā "I have a disk"Object Storage ā "I have a bucket"
File Storage ā "I have a shared folder"
Block Storage ā "I have a disk"With Cloud Object Storage, backups do not need to be accessed through an NFS mount like in the previous NAS example. A backup tool can upload backup files to a bucket using an API, SDK, or CLI provided by the storage provider.
One example of the backup flow is:
Production Database
ā
ā¼
Backup Tool
(e.g., backup script)
ā
ā¼
Local Storage
ā
ā Upload via API / SDK
ā¼
Cloud Object Storage
ā
ā¼
Bucket
ā
āāā database-backup.sqlProduction Database
ā
ā¼
Backup Tool
(e.g., backup script)
ā
ā¼
Local Storage
ā
ā Upload via API / SDK
ā¼
Cloud Object Storage
ā
ā¼
Bucket
ā
āāā database-backup.sqlFor example, the backup tool generates a backup file:
/var/lib/backup-tool/backups/database-backup.sql/var/lib/backup-tool/backups/database-backup.sqlThe file is then uploaded to a bucket:
production-backups/
āāā database/
āāā database-backup.sqlproduction-backups/
āāā database/
āāā database-backup.sqlThe main difference compared to the previous NAS example is how the storage is accessed.
NAS
Backup ā Local Storage ā NFS Mount ā NAS
Cloud Object Storage
Backup ā Local Storage ā API / SDK ā BucketNAS
Backup ā Local Storage ā NFS Mount ā NAS
Cloud Object Storage
Backup ā Local Storage ā API / SDK ā BucketWith NAS, the server can access remote storage as a filesystem through a protocol such as NFS. Meanwhile, with Cloud Object Storage, an application or backup tool generally interacts with the storage through an API or SDK to perform operations such as uploading, downloading, listing, and deleting objects.
The 3-2-1 backup strategy is an approach used to reduce the risk of data loss by keeping multiple copies of data across different storage systems and locations.
The principle is quite simple:
3 ā Keep at least three copies of the data
2 ā Store the copies on two different media or storage systems
1 ā Keep at least one backup in an offsite location3 ā Keep at least three copies of the data
2 ā Store the copies on two different media or storage systems
1 ā Keep at least one backup in an offsite locationWith this strategy, a failure of one server or storage system does not immediately cause all copies of the data to be lost.
The first principle is to have at least three copies of the data, consisting of one primary copy and at least two backup copies.
Simply put:
1 Production Copy + 2 Backup Copies = 3 Copies1 Production Copy + 2 Backup Copies = 3 CopiesFor example:
1. Production Database
2. Backup Copy #1
3. Backup Copy #21. Production Database
2. Backup Copy #1
3. Backup Copy #2Backup copies should not all depend on the same disk, server, or storage system.
Several possible storage combinations include:
Option A
Production Database
āāā Local Backup
āāā NAS
Option B
Production Database
āāā Local Backup
āāā Cloud Object Storage
Option C
Production Database
āāā NAS
āāā Cloud Object StorageOption A
Production Database
āāā Local Backup
āāā NAS
Option B
Production Database
āāā Local Backup
āāā Cloud Object Storage
Option C
Production Database
āāā NAS
āāā Cloud Object StorageThe main idea is to reduce the risk of a single point of failure. If one storage system fails, another backup copy is still available on a different storage system.
The final principle is to keep at least one backup in an offsite location.
Offsite means that the backup is stored in a location or failure domain that is separate from the primary infrastructure. This helps prevent a failure at the primary location from affecting all backup copies at the same time.
Both NAS and Cloud Object Storage can be used as offsite backup options:
NAS
āāā Offsite if located at another site / data center
Cloud Object Storage
āāā Can be used as offsite storage
if separated from the primary failure domainNAS
āāā Offsite if located at another site / data center
Cloud Object Storage
āāā Can be used as offsite storage
if separated from the primary failure domainBackup is not only about creating copies of data. It is also about how and where those copies are stored so they remain available when the primary system fails.
NAS can be a good option when centralized storage needs to be self-managed and accessed over a network, for example through NFS. Meanwhile, Cloud Object Storage provides an object-and-bucket-based approach that is generally accessed through APIs or SDKs and can be used as an option for storing backups outside the primary infrastructure.
In practice, NAS and Cloud Object Storage do not have to be used together. They are storage options that can be selected or combined depending on the organization's needs, infrastructure, costs, and policies.
More importantly, we need to think about how the backup architecture itself is designed. The 3-2-1 backup strategy can be used as a baseline: keep at least three copies of the data, store them across two different media or storage systems, and keep at least one backup in an offsite location.