MAT – Matillion : Automation – Restart matillion Services Weekly

 Matillion : Automate – Restart matillion Services Weekly 

Non-PRD  Matillion Servers are heavily used by
developers daily and there will be many process / connections opened on server
level, due to this it will not allow other users to login into system or server
performance will be degraded. To resolve this issue admin team has developed an
automated script which will recycle Matillion non-prod services without any
manual intervention and sends email alerts on success / failure of service. On
recycle of services all hung process or unused process will be killed. This
script execution flushes out temporary Memory on Linux servers used.

1. The script gets executed every time
   from a external scheduler or cron tab.
2. The script initially checks the
   tomcat service run state.
3. If the tomcat services not running
   by default, then restart activity is not performed and a failure mail is sent.
4. If the tomcat service runs then the
   script executes stop command and stops the tomcat service.
5. After a minute the script checks
   whether the tomcat service got stopped.
6. If the tomcat services are stopped
   then the script executes start command. It starts the tomcat service and
   confirms the tomcat service start, then sends a success notification email to
   recipients added.
7. If the tomcat services not stopped
   then the script executes stop command. It tries to stop once again and proceed
   on step 4.
8. If the tomcat services are not
   stopping after 3 consecutive attempt then a failure notification mail is sent.
9. All the activities are captured in
   Restart_log.txt file.

Benefits:

v 
Kills unwanted process and refresh server CPU
more  to Matillion.

v 
Avoids Jobs does not get into hung state.

v 
Keeps web services always alive

v 
Kills the existing long running jobs

v 
No manual intervention

v 
Alerts the admin team by sending notifications
on success/failures

 Process Highlight:

v  If
the webserver is in stop state as per any adhoc request, then the script never
executes stop command but sends an alert. A failure mail notification. 

v  The
script tries three attempts to stop the webserver and sends 3 failure attempt
mails.

v  The
script writes its actions as a log for easy analysis.

Log Files: Every action executed
in script is written in restart_log.txt log file. Timestamp is written along
with log for tracing. The time            format used in log is UST. The file
gets updated during every execution of the script.

Email
Notifications: The script sends 3 types of mails as per the scenario

•       Restart
Success Mail

•       Restart
Failure Mail

•       Already
in stop state Mail

Script
Details

·      
Script name: restart_sh.sh

·      
Script Path: /home/centos

·      
Script Owner: Root

·      
Script Permission: Read, Write, Execute

·      
Supported files: The script generates
below files.

o  
Status.txt – This file is used to check the
status of tomcat service within the script. This file gets newly created during
every execution in the same path.

Restart_log.txt
– It is a log file, It contains the actions performed by script as text with
date and time. The time format used in log is UST. The file gets updated during
every execution of the script

Script:

#!/bin/bash
recipients=”ambarish@abcd.com”
cd /home/admin_usr1/
systemctl status tomcat8.service > /home/admin_usr1/status.txt
run_flag=`grep ‘active (running)’ /home/admin_usr1/status.txt|wc -l`

if [ $run_flag -eq 1 ]
then
echo “`date “+%D %T”` :webserver is active”>> restart_log.txt
else
echo “`date “+%D %T”` :webserver is inactive”>> restart_log.txt
fi

attempt=1
while [ $attempt -le 3 ]
do
echo “`date “+%D %T”` :stop webserver attempt: $attempt”>> restart_log.txt
if [ $run_flag -eq 1 ]
then
sudo systemctl stop tomcat8.service
echo “`date “+%D %T”` :stop command executed”
echo “`date “+%D %T”` :stop webserver cmd executed”>> restart_log.txt
echo “`date “+%D %T”` :waiting for webserver to stop”>> restart_log.txt
#echo “`date “+%D %T”` :waiting for webserver to stop”
sleep 50s
systemctl status tomcat8.service > /home/admin_usr1/status.txt
run_flag=`grep ‘active (running)’ /home/admin_usr1/status.txt|wc -l`

if [ $run_flag -eq 0 ]
then
sudo systemctl start tomcat8.service
echo “`date “+%D %T”` :start command executed”
echo “`date “+%D %T”` :start tomcat cmd executed, attempt $attempt successful”>> restart_log.txt

sleep 50s
systemctl status tomcat8.service > /home/admin_usr1/status.txt
run_flag=`grep ‘active (running)’ /home/admin_usr1/status.txt|wc -l`

if [ $run_flag -eq 1 ]
then
#echo “`date “+%D %T”` :start tomcat cmd executed, attempt $attempt successful”
echo “`date “+%D %T”` :service is in running state”>> restart_log.txt
echo “subject: QA Matillion Services Restarted Successfully”| /usr/sbin/sendmail -f ambarish@abcd.com -t “$recipients”
echo “`date “+%D %T”` :success mail sent”>> restart_log.txt
fi

break
else
echo “`date “+%D %T”` :websever not responding, attempt $attempt failed”>> restart_log.txt
echo “subject: QA Matillion Services Restart Attemp $attempt failed”| /usr/sbin/sendmail -f ambarish@abcd.com -t “$recipients”
echo “`date “+%D %T”` :failure mail sent”>> restart_log.txt