Insight is an addon to Jira so you need to make sure Jira is configured to handle the amount of data you plan to import into Insight. We recommend you to read this article from Atlassian about how to setup Jira and recommended hardware requirements:
If you plan to use a lot of scheduling tasks and import a large amount of data we recommend to follow below approach. Remember to always test the memory consumption in an test environment if you plan to manage a lot of Insight data, because it's not always because of number of objects, also about the content of them.
|Objects in Insight||JVM Memory|
On large installations of Insight there are some options that can be tuned to increase performance.
To decrease the pause time we recommend that you switch the GC algoritm to G1 from the default. Configure it by adding the following JVM argument to your Jira setenv script.
Based on your environment other options to tune the G1 may be required. Check out the information from Oracle (http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html) and tune your JVM based on your requirements.
Insight executes tasks in parallel (e.g. reindex, imports). On a large instance it is possible to increase the amount of threads that performs the task. The configuration is located in the Insight general configuration. By default the parallelism is configured to be equal to the amount of cores available to the JVM.
If the parallelism has been increased it is recommended to check the database connection pool monitor and determine if an increased connection pool is required.
On Jira shutdown Insight persist the index on disk for faster startup. In large instances the index file fails to be saved on disk in time before the force shutdown command is sent. This will cause Insight to do a database reindex on each startup. To prevent this change the timeout on force shutdown in the stop-jira script (note that the timeout is located in 2 places)
Change the value 20 (it is seconds) to a value that is more reasonable based on the amount of data in your installation. Below is a snippet from the standard Jira stop-jira.sh and it is the value 20 that should be changed on 2 places.
If you have configured your heap with the CATALINA_OPTS in setenv. Make sure that the JVM configuration (JVM_MAXIMUM_MEMORY) is not the same value as the max heap configured in CATALINA_OPTS.
Using Insight Imports with a large number of objects
If you are using Insight with up to millions of objects and plan to use importers heavily, for instance with the Insight Discovery product. Then we recommend to move to Data Center if not already using it. This will offer you the possibility to isolate the importers to only one node and with the result of less user interaction impact. It will also be more robust since any failing import, any over consumption of memory will only affect the import and not any of your user nodes.
It's impossible for us to set any exact object size, import size recommendation since it's about the data stored on the objects, as well as how ofter the imports should occur. But in general, when reaching millions of objects, or memory usage over 32 Gb, or if you don't ever want to impact any user interaction, then you should consider using a Data Center environment.
During imports and re-index with Data Center we need to send messages to other nodes to update the index. This is done through provided Atlassian functionality and the clustermessage table is the database table where all nodes are pushing & pulling. The problem we have seen is that Atlassian have retention of 30 days for this table. This is way to much according to us, and what you should do is to remove all Insight related data rows that is older then 24 hours with your own retention scripts.