How to maintain custom fields in a Jira instance migration


Posted by
Remy Laborieux

April 12, 2018

You have two production Jira instances that need to be merged into one to optimize your company’s license costs. Your company has reorganized, or it has been acquired, and you have a number of Jira instances that duplicate tasks. There is a project in a test environment that is ready to be rolled into production.

These are just some examples where you need to migrate Jira projects. Yet merging instances and migrating data is, at best, an unwieldy and complex process that has to be done with minimal disruption to applications and end users. But even a simple aspect like maintaining custom fields during a Jira instance migration can be a major headache because it’s still largely a manual task.

Indeed, custom fields have an id that may change when migrated over to a new instance during a Jira migration. Adding to the complexity, custom field ids can be used in many different ways (groovy scripts, apps configurations, workflows, etc).

If you are dealing with a small data migration that only includes a few projects, you can use a Jira native project import to move data from one instance to another. It’s a whole other ballgame if you are dealing with larger Jira instances, where you need to move hundreds of custom fields, screens and permissions. To help you with this process, you can use a Jira app, like Project Configurator for Jira, to migrate the configuration.

Marketplace apps are a great option but they are not bullet-proof, particularly in the case of Project Configurator. For example, this app does not embed all the Jira Service Desk project-specific settings (SLAs, reports, calendars, etc.) during the migration process and it also does not automatically create custom fields that are locked, such as Jira Software specific fields like Epic link. They must be created manually before the import. Project Configurator also can’t manage two fields with the same name and type. Finally, the backup file cannot easily be restored to a different Jira instance. In this case, you will have to adjust/update the ids manually beforehand during a Jira migration.

Despite the challenges, solutions are evolving to make these migrations less time-consuming. Here are some semi-scripted solutions to help you reduce the workload and minimize errors in a two-Jira instance migration scenario.

Migrating custom field ids

We need to update custom field ids (‘customfield_<id>’) wherever they are used in the backup files. The backup files could be native XML backups from Jira or any custom backup procedure available in some Jira apps, such as nFeed. To do this, you could use manual search/replace features from any text editor. But if you have 100 custom fields to rename, a manual process is not reliable.

Now let’s create a search and replace script that will ease the burden of this Jira migration task.

Step 1: Get all source custom field ids and transform the JSON output to CSV files

First, collect the list of all custom fields from the source and target instances into readable CSV files.

# Get the list of all custom fields from the source instance
curl --silent -H -k -u user:password -H 'Content-Type: application/json' -o 'source_cf.json' -X GET https://Jirasource.domain/rest/api/2/field
 
# Parse the results to transform into a readable CSV file
xidel --silent --xquery='for $_s in $json() where $_s/custom eq true return fn:string-join(($_s / (id, name, custom, schema/type, schema/custom, schema/customId)),";")' source_cf.json
customfield_10001;Epic Link;true;array;com.pyxis.greenhopper.Jira:gh-epic-link;10001
....
customfield_10006;Flagged;true;array;com.atlassian.Jira.plugin.system.customfieldtypes:multicheckboxes;10006
customfield_10005;Rank;true;array;com.pyxis.greenhopper.Jira:gh-lexo-rank;10005
 
# Add a header to the source_cf.csv file
# "id";"name";"custom";"schema.type";"schema.custom";"schema.customId"
 
# Here's the final result
$ cat source_cf.csv
"id";"name";"custom";"schema.type";"schema.custom";"schema.customId"
"customfield_10001";"Epic Link";"true";"array";"com.pyxis.greenhopper.Jira:gh-epic-link";"10001"
....
"customfield_10006";"Flagged";"true";"array";"com.atlassian.Jira.plugin.system.customfieldtypes:multicheckboxes";"10006"
"customfield_10005";"Rank";"true";"array";"com.pyxis.greenhopper.Jira:gh-lexo-rank";"10005"
Create the source Jira CSV file

Apply the exact same thing for the the target Jira instance after the project import on the target instance.

Step 2: Import CSV files in a SQLite database in order to generate the sed script file

The aim here is to use SQL queries to read the CSV files in order to generate a set of “sed” command lines that will help us search and replace custom field ids.

$ sqlite3
SQLite version 3.21.0 2017-10-24 18:55:49
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .separator ';'
sqlite> .import source_cf.csv SOURCE_CF
sqlite> .import target_cf.csv TARGET_CF
sqlite> .separator '/'
sqlite> .once replace.sed
sqlite> SELECT "s", SOURCE_CF.id, upper(TARGET_CF.id), "g" from SOURCE_CF, TARGET_CF WHERE (SOURCE_CF.name = TARGET_CF.name) AND (SOURCE_CF."schema.type" = TARGET_CF."schema.type") AND (SOURCE_CF."schema.custom" = TARGET_CF."schema.custom");
sqlite> .quit
 
# This SQLite sequence above produce the script file replace.sed
$ cat replace.sed
s/customfield_10001/CUSTOMFIELD_10301/g
s/customfield_10002/CUSTOMFIELD_10302/g
....
s/customfield_10006/CUSTOMFIELD_10306/g
s/customfield_10005/CUSTOMFIELD_10305/g
# In order to replace customfield_nnnnn only, the trick is to use upper case 'CUSTOMFIELD' as a temporary tag for replacement
# The line 's/CUSTOMFIELD_/customfield_/g' must be added to the script to remove the temporary tag.
$ cat replace.sed
s/customfield_10001/CUSTOMFIELD_10301/g
s/customfield_10002/CUSTOMFIELD_10302/g
....
s/customfield_10006/CUSTOMFIELD_10306/g
s/customfield_10005/CUSTOMFIELD_10305/g
s/CUSTOMFIELD_/customfield_/g
Create the replace.sed script

Step 3: Use the replace script to update the backup file

In this case, our backup file is a nFeed configuration export (XML format). This configuration file contains custom field ids (dependencies between fields mostly).

Now we just have to run the “sed” script on the backup file, which will update all our custom field ids.

## copy <export file> to <import file>
## and then apply sed -i -f replace.sed <import file>
 
cp nFeed-Configuration-Backup-201801_source.xml nFeed-Configuration-Backup-201801_target.xml
sed -i -f replace.sed  nFeed-Configuration-Backup-201801_target.xml
nFeed example

We can now re-import the backup file.

These basic scripted sequences help ensure a secure migration and can be reused for any type of global scale search and replace needed during the Jira migration merging process.

Getting migrations right is a complicated process. If you have other questions about how to migrate your Jira instances, get in touch with one of our Atlassian certified consultants below.