- First you need to select the import type to use for the import.
- "General Fields" defines general configuration and are common for all import types.
- "Module Fields" are the import type depended fields (different for each import type).
- "Scheduling" fields are common to all import types and is where you specify synchronization details.
Object Mapping (IQL)
So here are the details of each step:
Note that for Data Center - the Import must be saved on the Scheduling Node, which should be configured in Insight's General configuration
|Name||The name of the import.|
|Description||The description for your convenience.|
You can specify a default concatenator. When joining multiple data locators into one Insight attribute, this will be the default concatenator. One example could be to join two columns like "First name" and "Last name" into one attribute. So "Mathias" (first name) and "Edblom" (last name) will be concatenated as "Mathias Edblom" if using \s as concatenator.
Enter \s for space-concatenated. To include a concatenate character, place the value between double quotes (i.e "\s").
|Empty Values||Defines what should happen when a Data Locator is empty. Should the import remove the attribute value or just ignores it and leave the current value as is.|
|Defines what should happen if a Data Locator is unknown to Insight. This could happen with attribute types like "Status" and "Select". The value can be added as an option or just ignore the value.|
Format for date fields in import source to convert dates into Insight. If left empty, Insight will automatically try to find correct format.
|Format for date/time fields in import source to convert dates into Insight. If left empty, Insight will automatically try to find correct format. |
The format should be specified according to the Java SimpleDateFormat guidelines.
The Import Type is responsible of defining and validating the module field configuration. However, it's optional for the Import Type module to implement any specific module fields. If no one is implemented, this section will be ignored and the import will move to the "Scheduling Fields". A list of built in import types and the description of the module field configuration can be found here: Available Import Types
|Synchronizing Account||The Jira user to use when synchronize data into Insight|
|Automatically Synchronize||If the import should be scheduled for automatic synchronization.|
|Cron Expression||The interval for the automatic synchronization.|
This is an example of an created CSV import. To see the information on the import, click on the header of the import to expand more information. This is also where you can create Object Type Mappings. Not that it's marked as invalid due to that no Object Type Mappings have been added.
Import Object Type Mapping
You can select what part of the data source that should be imported in an object type. You can configure as many object type mappings as you like. You can not specify more the one object type mapping to the same object type within the same import instance. However, if you create more then one import, you can let the imports updated the same objects from different sources.
|Object Type||The Insight object type to import the data into. This value can not be changed, you need to create a new object type mapping to configure another object type.|
|Selector||This is the selector of the import type that specifies what data from the data source that should be imported. This will be different to different import types. Read more on each import type documentation|
|Filter data source by IQL|
You can filter the data to be imported by IQL before it actually is imported into Insight
Filter on object references
It's not possible to filter your IQL based on object references, as references to other objects are handled separately after the object is created or updated. For more information, take a look at the screenshot at the bottom of this page.
This is a video demonstrating how to split your import source into different object types using IQL
There are three options when an object is missing in the data source.
Update the object with a predefined value to a specified attribute. The Threshold configuration will apply.
The object will be deleted in Insight. The Threshold configuration will apply.
The import will just ignores it
|Missing objects outbound references|
The setting configures how Insight should behave when a referenced object is missing in the import. For example if an import is configured to import Host and Application object types and in one run the Host is missing but the Application is included in the import. How should the reference from Application to Host be treated.
There are two options available
The outbound reference will be removed if the referenced object is missing in the current import
The outbound reference will be left as is
|Threshold Type||When we have missing objects in the data source you can specify a threshold for the "Update" and "Delete" configuration. This means that even if an object is missing from the source, you may wait until the import actually will consider doing any action upon it. You can specify if the type should be "Days" or "Synchronizations".|
|Threshold Number||The number of (days or synchronizations) before acting on the missing objects|
|Empty Values||This will override the general field configuration of the import type, for this specific object type mapping. What should happen if an Data Locator is empty? Should the import remove the attribute value or just ignore it and leave the current value as is?|
|This will override the general field configuration of the import type, for this specific object type mapping. Defines what should happen if a Data Locator is unknown to Insight. This could happen with attribute types like "Status" and "Select". The value can be added as an option or just ignore the value.|
Case sensitive - Identifier IQL will be built up with STRICT_EQUAL (==) which means that for text attributes, identifiers will be compared respecting case, 'Data Value' will NOT be considered the same as 'data value'.
At least one object type mapping needs to be enabled in order to enable the import.
Inheritance support for object type mappings
The import module engine supports configuration of inheritance structure. This means that you can point different parts of the source/data to different child object types by configuring an IQL.
Let's imagine that you have a CSV file with a lot of servers and one column in the CSV file is "Vendor". You can then create an inherit structure in Insight with "Server" as parent, and different child object types for each vendor like (Apple, Microsoft, Dell, etc). And by IQL point the source to the different object types by IQL:
Config for objectType "Apple"
Config for objectType "Microsoft"
All data that's not configured for an child object type, will be added to the parent, but you have the possibility to configure a child object type with an empty IQL that will be the default child.
Each Object Type Mapping needs to have at least one attribute configured.
If a specific attribute should be consider as a part of the object identifier. If multiple attributes is used as identifiers the combined value will be used to identify the object and identify duplicates.
The identifiers are used to construct an internal IQL responsible for identifying the object.
To exemplify: If two import attributes are configured for Insight attribute First Name and Last Name and specified to be identifier the following IQL will be used to identify object(s) in Insight
All attribute types are supported to be identifier. But from a performance perspective, try to use "Default" attribute types as identifiers as much as you can. For example, if using "User" attribute as identifier, we will have to query the Jira API to find users from the data source for the configured data locator. This means that Jira might be a bottle neck in this case if you have a large User directory. Try in this case to add another attribute for the user key (Default / Text) and the import will be much faster.
|Data Locator||Each import type has the responsibility to provide the available data locators for the import type. An example the CSV import type, where all "column" in the CSV will be the Data Locators.|
|Insight Attribute||The Insight attribute for the specified Object Type to import the Data Locator values|
Object Mapping (IQL)
When importing data, and using Insight attributes of type "Object" (reference) you need to provide an IQL to fetch the objects to set, depending on the data provided by the Data Locator. In the examples that follows, Name is a reference to the value in Insight that the referenced object has, in the attribute Name.
You can use place holders in the IQL like:
|Description||You can specify a description for the attribute mapping, for you convenience.|
|Concatenator||When joining multiple Data Locators into one Insight attribute, this will be the default concatenator.|
You can use regular expressions to edit the value from the import source.
For the regular expression we use Java syntax https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html. If the regular expression does not find any match the attribute is left empty. It is possible to extract from the value given by the Data Locator by making what you wish to extract the first group in the regular expression, for example using the regular expression "(\w*)\W" (quotation marks excluded) on the value "Blue Whale" yields "Blue" to the corresponding attribute. If there is no group present and the regex finds a match the whole value is given.
Each import module can (but is not required to) specify a predefined structure. The concept is basically that the import module will create an Insight object type structure together with attributes based on the external data source. It is a default Insight structure that can be used to represent the external data. This is just a suggestion and attributes, object types and the order of the same can be configured to your liking after creation.
You can use a custom created Insight structure or reuse the one you already have.
The predefined configuration is often used in conjunction with the Predefined Structure. The Predefined Configuration is a suggested configuration for the external data to your Insight structure. It will create the suggested object type configuration part(s) to your Insight structure.
The creation is done by using information provided by the import module how object types and attributes should be mapped with the data locators, the IQL references and all above mentioned configuration. Based on the name of the object type as well as zero or more attribute names supplied from the import module Insight will determine which object types(s) that should be added to the configuration.
To better understand what is happening the following simplified flow illustrates how each import is handled.
Before using the importers
You should configure the JVM memory to at least 4Gb maximum memory, and at least 2Gb in "normal" environments. But in very large instances you need to increase this even more, sometimes up to 32Gb maximum.
The more CPU cores the importers can use, the faster the import will complete. We recommend at least 8 cores.