Matillion: Automation – Catalina file Archival
For every event happening in
Matillion application corresponding logs gets generated in Catalina.out file.
As the file size grows more than a permissible limit the performance of the
server goes down and may even hang the Matillion process. So to avoid this,
Matillion has a built in auto archival process that archives the Catalina.out
file. It is scheduled every Sunday by default.
The
file size can be changed from 50MB to desired file size in the /etc/logrotate.d/tomcat8
file. Still if the file size goes beyond desired file size within a week, this
will not work. Also there is a high possibility of server getting hanged.
https://documentation.matillion.com/docs/en/6052919
/var/log/tomcat8/catalina.out
{copytruncate
daily
rotate
7
compress
missingok
size
50M
}
Process Highlight:
To avoid this, a script has been
written to check the Catalina.out file size every three hours, if the size goes
than the desired file size, the script will automatically archives the file
else will wait for the next schedule.
The script also cleans the archival folder by deleting the history files
older than the given days.
Benefits:
v
Avoids Jobs does not get into hung state.
v
Keeps web services always alive
v
No manual intervention
Flow chart:
Script Details
·
Script name: autolog_archive.sh
·
Script Path: /home/centos
·
Script Owner: Root
·
Script Permission: Read, Write, Execute
·
Script Parameters:
o
file_size_limit_mb : the desired file size to get archived
o
history_days_limit= the desired number of days the
archived files to be kept in server.
·
Supported files: The script generates
below files.
o
autolog_archivelog.txt – It is a log file in
same home directory, It contains the action logs performed by script as text
with date and time. The time format used in log is UST. The file gets updated
during every execution of the script.
Script:
#!/bin/bash
file_size_limit_mb=500
history_days_limit=10
log_file_size=`ls -lh /var/log/tomcat8/catalina.out|cut -d’ ‘ -f5|grep -o -E “[0-9]+”`
echo $log_file_size >> /home/centos/autolog_archivelog.txt
if [ $log_file_size -gt $file_size_limit_mb ]
then
filename=catalina`date +%Y%m%d%H%M%S`.out
cp /var/log/tomcat8/catalina.out /var/log/tomcat8/archival/catalina`date +%Y%m%d%H%M%S`.out
gzip /var/log/tomcat8/archival/$filename
sudo logrotate -f /etc/logrotate.d/tomcatrotate
echo ‘log clear cmd executed’ >> /home/centos/autolog_archivelog.txt
find /var/log/tomcat8/archival -name “*.gz” -type f -mtime $history_days_limit -delete
echo ‘deleted files older than given days’ >> /home/centos/autolog_archivelog.txt
echo `date` >> /home/centos/autolog_archivelog.txt
echo ——————————- >> /home/centos/autolog_archivelog.txt
else
echo ‘not archived’ >> /home/centos/autolog_archivelog.txt
echo `date` >> /home/centos/autolog_archivelog.txt
echo ——————————- >> /home/centos/autolog_archivelog.txt
fi
#crontab entries are in below file
#/etc/crontab