Matillion: Automation - Catalina File Archival

 Matillion: Automation - Catalina file Archival

For every event happening in Matillion application corresponding logs gets generated in Catalina.out file. As the file size grows more than a permissible limit the performance of the server goes down and may even hang the Matillion process. So to avoid this, Matillion has a built in auto archival process that archives the Catalina.out file. It is scheduled every Sunday by default.

The file size can be changed from 50MB to desired file size in the /etc/logrotate.d/tomcat8 file. Still if the file size goes beyond desired file size within a week, this will not work. Also there is a high possibility of server getting hanged.

https://documentation.matillion.com/docs/en/6052919

/var/log/tomcat8/catalina.out {copytruncate

daily

rotate 7

compress

missingok

size 50M

}

Process Highlight:

To avoid this, a script has been written to check the Catalina.out file size every three hours, if the size goes than the desired file size, the script will automatically archives the file else will wait for the next schedule.  The script also cleans the archival folder by deleting the history files older than the given days.

Benefits:

v  Avoids Jobs does not get into hung state.

v  Keeps web services always alive

v  No manual intervention

Flow chart:

Script Details

·       Script name: autolog_archive.sh

·       Script Path: /home/centos

·       Script Owner: Root

·       Script Permission: Read, Write, Execute

·       Script Parameters:

o   file_size_limit_mb :  the desired file size to get archived

o   history_days_limit= the desired number of days the archived files to be kept in server.

·       Supported files: The script generates below files.

o   autolog_archivelog.txt – It is a log file in same home directory, It contains the action logs performed by script as text with date and time. The time format used in log is UST. The file gets updated during every execution of the script.

Script:

#!/bin/bash
file_size_limit_mb=500
history_days_limit=10

log_file_size=`ls -lh /var/log/tomcat8/catalina.out|cut -d' ' -f5|grep -o -E "[0-9]+"`
echo $log_file_size >> /home/centos/autolog_archivelog.txt

if [ $log_file_size -gt $file_size_limit_mb ]
then
filename=catalina`date +%Y%m%d%H%M%S`.out
cp /var/log/tomcat8/catalina.out /var/log/tomcat8/archival/catalina`date +%Y%m%d%H%M%S`.out
gzip /var/log/tomcat8/archival/$filename

sudo logrotate -f /etc/logrotate.d/tomcatrotate
echo 'log clear cmd executed' >> /home/centos/autolog_archivelog.txt
find /var/log/tomcat8/archival -name "*.gz" -type f -mtime $history_days_limit -delete
echo 'deleted files older than given days' >> /home/centos/autolog_archivelog.txt
echo `date` >> /home/centos/autolog_archivelog.txt
echo ------------------------------- >> /home/centos/autolog_archivelog.txt

else

echo 'not archived' >> /home/centos/autolog_archivelog.txt
echo `date` >> /home/centos/autolog_archivelog.txt
echo ------------------------------- >> /home/centos/autolog_archivelog.txt
fi

#crontab entries are in below file
#/etc/crontab