Archi Forum

Archi Plug-ins => Other Plug-ins => Topic started by: Manj75 on October 01, 2019, 15:54:28 PM

Title: ACLI to import from ServiceNow
Post by: Manj75 on October 01, 2019, 15:54:28 PM
Hi Herve et al,

Having made a lot of progress with the ServiceNow plugin I wanted the ability to trigger an import from ServiceNow and had assumed this was possible using the ACLI, however running the --help does not show any option to do this.

Can anyone confirm if this is possible in the ACLI and if not can this be a simple change to add it in.  It would provide great immediate value, especially to import large sets in batches that would have caused an issue in one request rendering it impossible to do.  Running in batch using the service now limit/offset I can create N variations of the ini file to cover the different batches.  I can then run a batch script to initiate the import N times using the appropriate ini file.  Running each batch of 100K typically takes 5hrs to a complete run would take 15hrs and I want to automate this as an overnight run.

Herve - if this is not part the current plugin in how difficult would it be to add and release it?

Typical CLI

> Archi -application com.archimatetool.commandline.app -consoleLog -nosplash --servicenow.import <path of ini file>

Typical files would be like:
- sn_import_0_100K.ini (limit=100K/offset=0)
- sn_import_100K_200K.ini (limit=100K/offset=100K)
- sn_import_200K_300K.ini (limit=100K/offset=200K)

Thanks,
Manjit

Title: Re: ACLI to import from ServiceNow
Post by: Hervé on October 01, 2019, 19:54:35 PM
Hi Manjit,

Unfortunately, My ServiceNow plugin cannot be batched at the moment as the import method directly calls the file dialog that the user chooses the INI file.

Please open a case on GitHub and I'll change that.

In the meantime, I'll have a look on how to run it from CLI.

Best regards
Hervé
Title: Re: ACLI to import from ServiceNow
Post by: Manj75 on October 02, 2019, 08:01:50 AM
Hi Herve,

I understand that the SN plugin does not support batches, but my thinking was to import using separate batch runs of archi command line, i.e.

in a PoweShell script execute sequentially:

Archi -application com.archimatetool.commandline.app -consoleLog -nosplash --servicenow.import c:\servicenow_import_batch_1.ini
Archi -application com.archimatetool.commandline.app -consoleLog -nosplash --servicenow.import c:\servicenow_import_batch_2.ini
Archi -application com.archimatetool.commandline.app -consoleLog -nosplash --servicenow.import c:\servicenow_import_batch_3.ini


The all ini files would have the settings:

servicenow_import_batch_1.ini: servicenow.sysparm_limit = 100000

archi.relations.*.id                                                                                       = sys_id
archi.relations.*.name                                                                                  = type#name
archi.relations.*.source                                                                                = child
archi.relations.*.target                                                                                 = parent
archi.relations.*.folder                                                                                  = ${class}
archi.relations.*.property.strength                                                                 = connection_strength
archi.relations.*.property.source_action                                                         = type#child_descriptor
archi.relations.*.property.target_action                                                          = type#parent_descriptor
archi.relations.*.filter                                                                                 = &sysparm_offset=0

# "Contains::Contained by"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.archi_class                      = "CompositionRelationship"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.folder                             = "Enterprise ServiceNow ITSM"/"Composition Relationship"/"Contains::Contained by"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.name                              = "Contains"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.source                            = parent
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.target                             = child
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.type_name         = type#name
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.source_action     = type#parent_descriptor
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.target_action      = type#child_descriptor

For each separate batch ini file the offset will be different, i.e.

servicenow_import_batch_1.ini: archi.relations.*.filter = &sysparm_offset=0
servicenow_import_batch_1.ini: archi.relations.*.filter = &sysparm_offset=100000
servicenow_import_batch_1.ini: archi.relations.*.filter = &sysparm_offset=200000

Have the ini files and am able to run SN import manually in Archi each ini file which results in importing all 300K relationships, which previously if setup in a single request would cause an exception to do with Jackson JSON parser, if recall.  Now, I just want to automate these manual steps and it can be schedule to run overnight without any manual intervention.

Hopefully this makes it clearer and why it would be good to have SN import supported in the ACLI, similar to CSV and XML Open Exchange for consistency.

I'll raise an issue as an feature request on GitHub.




There is an alternative solution that would not need the ACLI support to get around the issue of large data set run in batches, and this is to support the iterative runs of import for the same sys_id if defined in single ini file, then I could specify the 3 batched offsets in one ini file as follows:

servicenow_import_all.ini

servicenow_import_batch_1.ini: servicenow.sysparm_limit = 100000

archi.relations.*.id                                                                                       = sys_id
archi.relations.*.name                                                                                  = type#name
archi.relations.*.source                                                                                = child
archi.relations.*.target                                                                                 = parent
archi.relations.*.folder                                                                                  = ${class}
archi.relations.*.property.strength                                                                 = connection_strength
archi.relations.*.property.source_action                                                         = type#child_descriptor
archi.relations.*.property.target_action                                                          = type#parent_descriptor
archi.relations.*.filter                                                                                 = &sysparm_offset=0

# "Contains::Contained by" range is 0 to 100K
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.archi_class                      = "CompositionRelationship"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.folder                             = "Enterprise ServiceNow ITSM"/"Composition Relationship"/"Contains::Contained by"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.name                              = "Contains"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.source                            = parent
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.target                             = child
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.type_name         = type#name
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.source_action     = type#parent_descriptor
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.target_action      = type#child_descriptor
archi.relations.*.filter                                                                                 = &sysparm_offset=0

# "Contains::Contained by" range is 100K to 200K
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.archi_class                      = "CompositionRelationship"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.folder                             = "Enterprise ServiceNow ITSM"/"Composition Relationship"/"Contains::Contained by"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.name                              = "Contains"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.source                            = parent
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.target                             = child
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.type_name         = type#name
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.source_action     = type#parent_descriptor
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.target_action      = type#child_descriptor
archi.relations.*.filter                                                                                 = &sysparm_offset=100000

# "Contains::Contained by" range is 200K to 300K
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.archi_class                      = "CompositionRelationship"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.folder                             = "Enterprise ServiceNow ITSM"/"Composition Relationship"/"Contains::Contained by"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.name                              = "Contains"
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.source                            = parent
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.target                             = child
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.type_name         = type#name
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.source_action     = type#parent_descriptor
archi.relations.55c95bf6c0a8010e0118ec7056ebc54d.property.target_action      = type#child_descriptor
archi.relations.*.filter                                                                                 = &sysparm_offset=200000

Sorry for the long post  ;)

Thanks,
Manjit
Title: Re: ACLI to import from ServiceNow
Post by: Hervé on October 02, 2019, 08:36:22 AM
Hi Manjit,

I fully appreciate, that's why I wrote that I'll do it ...

I just explained that unfortunately this is not doable with the current version of the plugin as the first thing the import method does is to open a graphical popup (file dialog) to ask the user for the INI file.

Best regards
Hervé
Title: Re: ACLI to import from ServiceNow
Post by: Manj75 on October 02, 2019, 09:36:41 AM
Thanks - plugin opening a graphical popup file dialog provides clarity as to why it was not doable.

I appreciate you looking into this one.

Kind Regards,
Manjit