General Import Concepts

To import data into Insight you need to configure an instance of one of the import types that are available in your JIRA environment. The core implementation of the Import functionality is the same for all import types to make the import scalable, robust and below you can read about common configuration concepts.

Import Type

The type of import to be used in Insight. The number of import types can vary depending on installed import modules. Below you will find the configuration that you can specify for the import type. Some of these configuration items can be override by the Object Type Mapping configuration.

There are 4 parts that needs to be performed when creating an import.



  1. First you need to select the import type to use for the import. 
  2. "General Fields" defines general configuration and are common for all import types
  3. "Module Fields" are the import type depended fields (different for each import type)
  4. "Scheduling" fields are common to all import types and is where you specify synchronization details.


So here are the details of each step:

General Fields

NameDescription
NameThe name of the import
DescriptionThe description for your convenience.
Concatenator

You can specify a default concatenator. When joining multiple data locators into one Insight attribute, this will be the default concatenator. One example could be to join two columns like "First name" and "Last name" into one attribute. So "Mathias" (first name) and "Edblom" (last name) will be concatenated as "Mathias Edblom" if using \s as concatenator.

Enter \s for space-concatenated. To include a concatenate character, place the value between double quotes (i.e "\s").

Empty ValuesDefines what should happen when a Data Locator is empty. Should the import remove the attribute value or just ignores it and leave the current value as is.
Defines what should happen if a Data Locator is unknown to Insight. This could happen with attribute types like "Status" and "Select". The value can be added as an option or just ignore the value.

Format for date fields in import source to convert dates into Insight. If no one provided, Insight will automatically try to find correct format. 
The format should be specified according to the Java SimpleDateFormat guidelines.

Format for date/time fields in import source to convert dates into Insight. If no one provided, Insight will automatically try to find correct format. 
The format should be specified according to the Java SimpleDateFormat guidelines.


Module Fields

The import type are responsible of defining and validating the module field configuration. However, it's optional for the Import Type module to implement any specific module fields. If no one is implemented, this section will be ignored and the import will move to the "Scheduling Fields". A list of built in import types and the description of the module field configuration can be found here: Available Import Types

Scheduling Fields

NameDescription
Synchronizing AccountThe JIRA user to use when synchronize data into Insight
Automatically SynchronizeIf the import should be scheduled for automatic synchronization.
Cron ExpressionThe interval for the automatic synchronization.


This is an example of an created CSV import. To see the information on the import, click on the header of the import to expand more information. This is also where you can create Object Type Mappings. Not that it's marked as invalid due to that no Object Type Mappings have been added.

Import Object Type Mapping

You can select what part of the data source that should be imported in an object type. You can configure as many object type mappings as you like. You can not specify more the one object type mapping to the same object type within the same import instance. However, if you create more then one import, you can let the imports updated the same objects from different sources.

ConceptDescription
Object TypeThe Insight object type to import the data into. This value can not be changed, you need to create a new object type mapping to configure another object type.
SelectorThis is the selector of the import type that specifies what data from the data source that should be imported. This will be different to different import types. Read more on each import type documentation
IQL

You can filter the data to be imported by IQL before it actually is imported into Insight

This is a video demonstrating how to split your import source into different object types using IQL

Missing Objects

There are three options when an object is missing in the data source.

  • Update

Update the object with a predefined value to a specified attribute. The Threshold configuration will apply.

  • Delete

The object will be deleted in Insight. The Threshold configuration will apply.

  • Ignore

The import will just ignores it

Threshold TypeWhen we have missing objects in the data source you can specify a threshold for the "Update" and "Delete" configuration. This means that even if an object is missing from the source, you may wait until the import actually will consider doing any action upon it. You can specify if the type should be "Days" or "Synchronizations".
Threshold NumberThe number of (days or synchronizations) before acting on the missing objects
Empty ValuesThis will override the general field configuration of the import type, for this specific object type mapping. What should happen if an Data Locator is empty? Should the import remove the attribute value or just ignore it and leave the current value as is?
This will override the general field configuration of the import type, for this specific object type mapping. Defines what should happen if a Data Locator is unknown to Insight. This could happen with attribute types like "Status" and "Select". The value can be added as an option or just ignore the value.

At least one object type mapping needs to be enabled in order to enable the import.

Import Attribute

Each Object Type Mapping needs to have at least one attribute configured.

ConceptDescription
Identifier

If a specific attribute should be consider as a part of the object identifier. If multiple attributes is used as identifiers the combined value will be used to identify the object and identify duplicates.

The identifiers are used to construct an internal IQL responsible for identifying the object.

To exemplify: If two import attributes are configured for Insight attribute First Name and Last Name and specified to be identifier the following IQL will be used to identify object(s) in Insight

"First Name" == <first name value from source> AND "Last Name" == <last name value from source>
Data LocatorEach import type has the responsibility to provide the available data locators for the import type. An example the CSV import type, where all "column" in the CSV will be the Data Locators.
Insight AttributeThe Insight attribute for the specified Object Type to import the Data Locator values
Object Mapping (IQL)

When importing data, and using Insight attributes of type "Object" (reference) you need to provide an IQL to fetch the objects to set, depending on the data provided by the Data Locator. In the examples that follows, Name is a reference to the value in Insight that the referenced object has, in the attribute Name.

You can use place holders in the IQL like:

  • For single attributes (max cardinality equals 1) the IQL with the placeholder could look like this:

    Name = ${DATA LOCATOR}
  • For multiple attributes (max cardinality larger then 1) the IQL with the placeholder could look like this:

    Name IN (${DATA LOCATOR${0}})


    The "0" (zero) stands for all values, and 1 for the first value, 2 for the second value and so on.

    There is a video available showing an example of how to set this up:

    Read more about object mapping

DescriptionYou can specify a description for the attribute mapping, for you convenience.
ConcatenatorWhen joining multiple Data Locators into one Insight attribute, this will be the default concatenator.

You can use regular expressions to edit the value from the import source.

For the regular expression we use Java syntax https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html. If the regular expression does not find any match the attribute is left empty. It is possible to extract from the value given by the Data Locator by making what you wish to extract the first group in the regular expression, for example using the regular expression "(\w*)\W" (quotation marks excluded) on the value "Blue Whale" yields "Blue" to the corresponding attribute. If there is no group present and the regex finds a match the whole value is given.

Some examples:

PurposePatternExample
Validate that input contains at least one digit\d"foo 123 bar" is returned as is but "foo bar" gives empty attribute value
Extract first digit(\d)"foo 123 bar" gives 1 but "foobar" gives empty attribute value
Extract only first word of text(\w*)\W

"Blue whale" becomes "Blue"

Shorten number to show only two decimals(\d*\.\d\d)12.3456 becomes 12.34


Predefined Structure

Each import module can (but is not required to) specify a predefined structure. The concept is basically that the import module will create an Insight object type structure together with attributes based on the external data source. It is a default Insight structure that can be used to represent the external data. This is just a suggestion and attributes, object types and the order of the same can be configured to your liking after creation. 

The option to create Predefined Structure (and Predefined Configuration) is only available if no object type configuration(s) exists for the current import configuration 

You can use a custom created Insight structure or reuse the one you already have. 

Predefined Configuration

The predefined configuration is often used in conjunction with the Predefined Structure. The Predefined Configuration is a suggested configuration for the external data to your Insight structure. It will create the suggested object type configuration part(s) to your Insight structure. 

The creation is done by using information provided by the import module how object types and attributes should be mapped with the data locators, the IQL references and all above mentioned configuration. Based on the name of the object type as well as zero or more attribute names supplied from the import module Insight will determine which object types(s) that should be added to the configuration.

It is possible to use the Predefined Configuration without first using the Predefined Structure.


Import progress 

To better understand what is happening the following simplified flow illustrates how each import is handled. 

Import steps

Before using the importers

MEMORY

You should configure the JVM memory to at least 4Gb maximum memory, and at least 2Gb in "normal" environments. But in very large instances you need to increase this even more, sometimes up to 32Gb maximum.

CPU

The more CPU cores the importers can use, the faster the import will complete. We recommend at least 8 cores.