TechBask

Learn | Explore | Share | Challenge Yourself | Enjoy | Make This World Beautiful Place To Live For Everyone|

Archive for the category “Linux Learning”

Automate Backup of MongoDB.

I work with MongoDB  in day today basis and one of the challenge is taking backup of different DB’s deployed. Taking backup is important for recovery in case of  mishap. Mostly there are enormous write operations and database size keeps growing so the chances of failure. Initially we started to do this manually which was okay till we had limited traffic but as  registration and engagement increased so we started looking for automating this process. Cron job scheduler in Linux and Amazon S3 came to the rescue. 

Pre-requisites:

  • MongoDB 2.33 or higher deployed on Linux Environment.
  • Amazon S3 account to store all backup data.

 

Tools Required(or their alternatives):

  • S3cmd – S3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3.
  • Linux Cron utility
  • Mongodump utility provided by MongoDB
  • Tar archiving utility– GNU version.
  • Command Line Interpretor(CLI).

 

Installation and Configuration of S3cmd:

  1. Run to install in CLI or Shell
    sudo su
    sudo yum –enablerepo epel install s3cmd
  2. configure
    s3cmd –configure
  3. Display all buckets in your Amazon S3 account
    s3cmd ls

 

Writing a Shell Script to Backup MongoDB:

  1. Force file syncronization and lock writes on MongoDB using fsync
    mongo admin –eval “printjson(db.fsyncLock())”
  2. Declare some variables
    MONGODUMP_PATH=”path-to-mongodb-utility”
    MONGO_HOST=”server-ip”
    MONGO_PORT=”port-here”
    USER_NAME=”username-used-for-mongodb-authentication”
    PASSWORD=”password-used-for-mongodb-authentication”
    MONGO_DATABASE=”DB-Name”
    TIMESTAMP=`date +%F-%H%M` // use as it is. S3_BUCKET_NAME=”name-of-s3-bucket”
    S3_BUCKET_PATH=”mongodb-backups”
  3. Take mongoDB dump using mongodump utility
    $MONGODUMP_PATH -h $MONGO_HOST:$MONGO_PORT -d $MONGO_DATABASE -u $USER_NAME -p $PASSWORD
  4. Unlock database writes
    mongo admin –eval “printjson(db.fsyncUnlock())”
  5. Add timestamp to backup folder name
    mv dump mongodb-$HOSTNAME-$TIMESTAMP
  6. Compress the folder
    tar cf mongodb-$HOSTNAME-$TIMESTAMP.tar mongodb-$HOSTNAME-$TIMESTAMP
  7. Upload to S3
    /usr/local/bin/s3cmd put mongodb-$HOSTNAME-$TIMESTAMP.tar s3://$S3_BUCKET_NAME/$S3_BUCKET_PATH/mongodb-$HOSTNAME-$TIMESTAMP.tar
  8. Delete local dump folder
    /bin/rm -r mongodb-$HOSTNAME-* // to delete all use *

Find full bash script here

 

Run this script at a particular time using crontab

Quick Linux..

-linux is kernel.

-Kernel–lower abstraction layer of OS.

-linux distributions —ubantu , fedora ,opensuSE.

-OS Family—-Unix-like.

-Users operate a Linux-based system through a CLI, a GUI, or through controls attached to the associated hardware, which is common for embedded systems.

-CLI(Command Line Interface)–method of interaction with computer based on text commands -entered at the prompt.

-GUI-graphical user interface.

-Linux kernel is maintained by FLOSS(free/libre/open source  software).

-POSIX—portable operating system interface.

-Fact—Ubantu is an ancient african word, meaning ‘humaniy to others’

-Similar to explorer in Windows, there is nautilus in Ubantu.

for GUI(Gnome,KDE,XFCE) in Linux:-

-The syntax for a lookup in the manual is: man [-] [-k keywords] topic.

-Windows and Mac OS X have something in common: one desktop -GUI. Mac OS X doesnot even have multiple themes

-Linux  on the other hand, has 3 major desktop managers and more than 10 window managers.

-The GNOME project provides two things: The GNOME desktop environment, an intuitive andattractive desktop for users, and the GNOME development platform, an extensive framework forbuilding applications that integrate into the rest of the desktop.

Commands:::

-date—>shows current date.

-uptime—->shows time since last login.

-ls—->lists all the files.

-pwd—->shows your current location

-sudo—->makes you enter command as different user.

-cd dir_name—->navigates you to that directory.

-cd ..—->navigates you to home directory.

-whoami—->shows information about the current user.

-wget {path to file}–download the file from the internet.

-sudo apt-get install{packagename}–dowloads and install the package.

-uname -a: shows info about the kernel.

-df—shows info about disk usage.

-mkdir dir_namee—–creates new diretory.

-rm -r::::removes directory or file.

-cp -r ::: copy diectory or file.

-locate -r file*.txt—searches the indexes for all entries matching  file*.txt.

-ping{host}—sends ping to address u provide.

-whois{domain}—-shows whois info about the domain.

-iwlist scan—scans for available wireless network.

-lpq—  displays current print queue(A reserved area of memory or disk that holds output designated for the printer until the printer can receive it).

-smbtree—-shows available windows network share.

-ifconfig—configure TCP/IP network interfaces.

-apropose {subject}–list manual pages for subject.

Post Navigation