Keeping your JIRA dev and production instances in check


Posted by
Abdulrazaq OMAR

August 2, 2016

In this article, we’ll be discussing and going through some tips for JIRA sysadmins that can help keep your Atlassian applications healthy, clean and in sync. This involves looking after logs which can grow substantially, keeping dev, test and production environments and moving data and configuration between them in an organised manner with failover. So let’s get started, shall we?

Log rotation

JIRA writes logs into two locations: one is the JIRA home directory, which is rotated at 20MB in the same location, and the second is tomcat log files, which are in JIRA’s installation directory and are not rotated automatically. In large JIRA instances, these files can grow in size to a level where they halt performance of the server due to disk space issues. It’s a pain to keep cleaning these log files and to look after what’s eating up space on your servers, so before you stress out, let’s look at how we can limit those panic attacks by rotating our logs. We can do this by configuring what JIRA logs and by setting up Logrotate,  the default application used to rotate log files by Linux.

Method 1: limiting the console logging and Logrotate:

Since JIRA writes its logs to the logs in the home directory, we can stop JIRA from writing console logs to atlassian-jira.log by removing the log-specific information from there, and then configuring logrotate on catalina.out.

  1. Navigate to log4j.properties in <JIRA-Installation Directory>/atlassian-jira/WEB-INF/classes/log4j.properties
  2. Remove any console entries from there:
sed -i 's/, console,/,/g' log4j.properties

3. This will get rid of the majority of console logging into atlassian-jira.log, which will significantly reduce the amount of logging and log file size.

4. Now we can configure log rotation to rotate the Catalina.out (the tomcat log in the installation directory) so that JIRA does not rotate- log file when it reaches a certain size (or after a certain number of days). Create a file in/etc/logrotate.d/ directory, where the custom logrotate files are stored, and give it any name (for example jira-logrotate.conf), before pasting the below contents into it. Referring to the logrotate main page, you can set, for example:

/usr/local/atlassian/jira/logs/* {
    copytruncate
    dateext
    weekly
    rotate 5
    compress
    delaycompress
    missingok
}

The above configuration will rotate all the logs in the /logs directory with the following settings:

  • copytruncate: instructs logrotate to create the copy of the original file (i.e rotate the original log file) and truncate the original file to zero byte size
  • dateext: archive old versions of log files adding a daily extension like YYYYMMDD
  • weekly: rotate every week (this can be changed to daily, or as desired)
  • rotate 5: limit the number of log file rotations, so this setting will limit the number of created files to five
  • compress: old log files are compressed
  • delaycompress: postpone compression of the previous log file to the next rotation cycle
  • missingok: if the log file is missing, go on to the next one without issuing an error message

Method 2: Removing files older than a certain number of days

You can also chose to write a cron job to delete any log files that older than a certain amount of days, for example:

30 2 * * * find /opt/atlassian/jira/logs/*.* -mtime +10 | xargs rm

This removes any files older than 10 days, and it will run every day at 2:30 AM. You can adjust this to suit your preference.

You might want to be cautious with this method, however, because you will lose your log file history. So unless you are sure you want to delete them, do not perform this!

Keeping dev and production instances in sync

In any application, especially in enterprise environments, it’s crucial to have a development instance where all tests and development work happens before everything is transferred to the production instance. But in some circumstances your dev or test instance might be corrupt with too many tests and playing around, and you might need to replicate your production instances again – or maybe set a new dev instance if you didn’t have one before.

Let’s see how we can manage that:

Creating a full production backup and restoring it to your dev instance

  • Back up your production home directory. This is where your data from attachments, logos, etc. is stored. To locate them, check out this guide for JIRA and this guide for Confluence.
  • Back up your Confluence/JIRA installation directories.
  • Back up your databases, using the native database tools for the database type you are using.
  • Restore all the files to your dev server. You can follow the detailed steps in Atlassian documentation here.

Using rsync to synchronise directories and default files

Now that you have a copy of your production server, you might want to automate these steps of copying files over from one server to another. Linux provides a great way to sync directories using the command rsync. The reason that rsync is more suitable in this situation compared to for example scp or rcp, is that rsync will do incremental copy and has very minimal impact on system performance.You can install it using the command:

apt-get install rsync

or for RedHat, CentOs systems:

yum install rsync

Once rsync is installed, you can configure some rsync commands into a simple bash script to run them. Here’s a guide on how rsync command works, as well as an example for copying files from a remote server (production JIRA), to a local system (dev server):

rsync -v -e ssh user@server.com:~/file /tmp

The above command will use SSH to secure the rsync connection, and will connect to the server to copy the “file” and copy it to the temp directory of the server that this command is run from (the target). In order to synchronise a local directory with a remote directory, we will add few other commands:

rsync -r -a -v -e "ssh -l Username" --delete /local/Directory remote.server:/remoteDirectory
  • delete : delete files that don’t exist on sender (system). A bit of details on the command options below:
  • v : verbose (try -vv for more detailed information)
  • e “ssh options”: specify the ssh as remote shell
  • a : archive mode
  • r : recurse into directories
  • z: compress file data

You can then put those rsync commands into a simple bash script:

#!/bin/sh
# Backup of production files to dev server
rsync -r -a -v -e "ssh -l Username" --delete /local/Directory remote.server:/remoteDirectory

Important note: Once your script is saved, make it executable by running the command chmod +x filename. Now you simply need to run your script each time you need to synchronise your home directory from your production server.

Obviously you don’t want to sync the files with specific information, like the config file that contains the database URL. This is because your dev and production JIRA instances are on different databases, and nobody likes to mess with their production database (smile). So bear in mind that you should exclude some very specific files from the rsync, such as:

  • dbconfig.xml file at the root of your JIRA Home Directory
  • jira-application.properties file located within the <jira-application-dir>/WEB-INF/classes
  • Any other customised files in your environment

I hope you find these tips for JIRA sysadmins useful. Have more of your own? Leave a comment below!

  • Corentin Mehat

    You might want to use the rsync exclude options directly in your script either by specifying a file with all the exclusions to perform, or by specifying them in the command line directly :

    –exclude-from
    –exclude

    For example :
    rsync -r -a -v -e “ssh -l Username” –exclude ‘dbconfig.xml’ –excude ‘WEB-INF/classes/’ –delete /local/Directory remote.server:/remoteDirectory

    One important thing to keep in mind when excluding a directory is that rsync will always consider the path to be relative to the source directory.

    I personally prefer to use the –exclude-from option to keep the rsync command more readable.