New database plugin version 2.1

Started by Hervé, September 25, 2018, 16:49:30 PM

Previous topic - Next topic

Hervé

Dear all,

I'm happy to announce that the version 2.1 of my database plugin has just been released : https://github.com/archi-contribs/database-plugin/tree/master/v2

As usual, you may install it manually by copying the jar file in Archi plugins folder, or by clicking on the "Check for update" button on the perference page if an older version of the plugin is already installed.

There a a huge number of changes in this new version, the biggest one being the collaborative mode (in blue below):
Import components:
  * Added documentation column in the table to help distinguish components having the same name
  * Added tootip with properties to help distinguish components having the same name
  * Add message during the import as it may take some time
  * Imports can now be undone / redone
  * A label now explains that the icons can be selected on the import element window
  * Classes to import are pre-selectionned depending on the folder where the components will be imported to
  * The categories can now be clicked to select/un-select the whole category
  * It is now possible to import (merge) models
  * Introduce new "template" import mode that mixes copy and shared mode

Import model:
  * A context menu entry allowing to import a model has been added when no model is selected
  * Automatically open the default view (if any) at the end of a successful import
  * Fix number of images to import

Export model:
  * Complete rewrite of the comparison management (use timestamps in addition of version number as it is possible to switch from a database to another)
  * In case of exception, the database lock is released before the error message is displayed
  * An option has been added to show or not show the model's comparison to the database before the export (showing comparison gives more information to the user but it takes some time on big models)
  * For relational databases:
    * The export is now in "collaborative mode", which replaces the export to thedatabase by a synchronization :
      * It can be compared to a pull+push to GitHub:
         * Components created or updated in the model are exported to the database
         * Components created or updated in the database by another person are imported into the model
         * Components removed from the model in the database by another person are removed from the model in Archi
         * Components updated in both the model and the database generate conflicts that need to be manually solved
      * It is slower than the previous mode but allows several people to work on the same model at the same time
    * Allow to specify border width and scale factor for views screenshots
    * To simplify the code, it is no more possible to choose between whole export or elements and relationships only
  * For Neo4J databases:
    * Create two export modes: native and extended:
      * Native mode means that Archi relationships are exported as Neo4J relationships (but this mode does not allow to export relationships on relationships)
      * Extended mode means that Archi relationships are exported as Neo4J nodes (this mode makes Neo4J diagrams more complex but allow relationships on relationships)
    * New option to empty the database before the export
    * New option to specialize relationships

Get history from database:
  * It is now possible to get history for all the model components (including folders, diagrams, canvas and sketches)
  * It is now possible to import/export an individual component from the history window

Other:
bug fixes:
    * Fix export blocks and images objects that have got no image set
    * Fix plugin initialization failure that occurred some times
    * Fix progress bar during download new version of the plugin from GitHub
    * Fix centering of GUI windows, especially on hiDPI displays
    * Fix calculation of numbers of images to import
    * Fix properties updates when several of them have got the same name during sync of existing components
    * Fix centering of GUI windows especially on HiDPI displays
Improvements:
    * Fill in the inline help pages
    * Rewrite debug and trace messages to be more useful
    * Increase compiler severity to maximum and resolve all the warnings to improve code resiliency
    * Reduce memory leak
    * Better management of the cancel button during the import and export process
    * Add the ability to import an image from the database (on the Image and Block objects in Canvas)
    * Remove the name, the documentation and the properties from view objects and connections checksum as they are not related to the view objects and connections themselves, but to the linked element or relationship
    * Some annoying popups have been replaced by messages directly in the import/export window
    * Add procedures that can be called by the script plugin
    * The inline help can be accessed using the interrogation mark on every plugin window.
    * Export and import back the model's "metadata" (may be used by other external tools)
    * Do not calculate checksum on images anymore as the path is already a kind of checksum
    * A new "show debug information" window has been created
    * Add "import model from database" menu entry on right click when no model has been loaded yet
    * Manage the objects transparency (introduced in Archi 4.3)
    * Check for max memory available at startup and suggest to increase it (Xmx parameter) if less than 1 GB
Update JDBC drivers
      * Neo4J to 3.1.0
      * SQLite to 3.21.0
      * PostGreSQL to 42.2.1

Please do not hesitate to drop ma a line should you require more information.

Best regards
Hervé

newbie1234

Hi Hervé

Thanks for working on this plugin!

Have a couple of questions for clarification.

Wondering if you would be able to elaborate on what the "template import mode" does?

The GitHub functionality is so when a new version of the plugin is released, that it is recognized, correct?

Again, thanks!

Hervé

Of course ...

First, please note that the plugin includes help pages that can be accessed through the interrogation mark present on all the plugin windows, or using the help menu of Archi. If those pages are not well explained, please do not hesitate to tell me which one, and I will rewrite it.

This said, Here is an extract from the integrated help page:
QuoteThe import mode allows to specify how the components will be imported. Three modes are available:
Force copy mode: in this mode, all the components will have a new ID that will be distinct from the one in the database. The imported components and the components in the database will therefore be independent: updating the ones will not impact the others.
Force shared mode: in this mode, all the components will be imported with their existing ID from the database. All the components will therefore be shared across the models: the updates done on those components in one model will be seen across all the models where they are present.
Template mode: in this mode, which allows to mix copy and shared mode, the import mode is distinct for each component and depends on their properties:
   - If the component has got a property called "template" with value "copy": then the component is imported in copy mode,
   - If the component doesn't have a property called "template", or if its value is not "copy": then the component is imported in shared mode.

Even if available when importing a single element or when merging models, this template mode is mainly useful when importing a view. Let's take a very simple example ...

Let's consider a reference model in which I created a view representing a Solution Building Block having a virtual machine that is part of VSphere cluster itself part of a datacenter (I said it was a very simple example  ;D).

The template mode allows me to instantiate this SBB in a project model by importing this view:
   --> setting the "template property" of the virtual machine to "copy" on the SBB virtual machine, the project's virtual machine will be distinct from the SBB's virtual machine,
   --> not setting the "template property" of the SBB VSphere cluster and the datacenter, they will be shared between my project and my reference models

This way, when I update the VSphere cluster or the datacenter from my reference model, all my projects models are updated accordingly.

Hope this helps.

Best regards
Hervé

newbie1234

Thanks Hervé.

Many thanks for the explanation and makes much more sense.

Macs must be sorta cursed in general perhaps :)  I can go to the main help in the "Help" menu while in the app and get to the section about the database plugin, although the explanation you reference in your reply is not there for me. 

Also, when I click on the "?" button in the preferences screen for contextual help it does not pop up any help screen.  I just upgraded to Mac OS Mohave (10.14) today as well, so who knows what that issue might be.

Regardless, many thanks for your patience and explanation!

Hervé

you're very welcome.

Best regards
Hervé

newbie1234

Hi Hervé.

Hopefully an easy question. 

Is it possible to key off a database export of a model somehow so that a script can auto-generate reports from command line tools in Archi?

Are folks using some type of trigger on a particular table in the database to see when an export is making changes to a model, or is there a simpler way of getting from here to there?

Many thanks again, in advance for your help.

Hervé

Hi,

At work, I'm still using Archi 4.2, and I automated HTML reports generation using my script plugin (https://github.com/archi-contribs/script-plugin). To be honest, I haven't tested if it continues to work with Archi 4.3 (as 4.3 implements a new HTML report mechanism).

This said, my script plugin has been created before ACLI (Archi Command Line Interface) and jArchi were developed and these tools are much more powerful (but more complex as well) than my script plugin. So I need to work with JB and Phil to interface my plugins with them and when this is done, it will be much more easy to create Archi batches using models in a database.

Hope this helps

Best regards
Hervé

newbie1234

Hi Hervé.

Many thanks for your reply. 

For time being, having folks make sure they also save their model changes also to a GitHub repo will get us by I think (this can generate a webhook from the repo, that can be caught and acted upon to re-generate new version of Archi html / Jasper html / archimate files from the command line tools, so the output can be put where folks can use them.

After thinking about my question, it would seem problematic in that if the database is re-sync back and forth between Archi and db (I think your latest version does this?) it probably isn't easy to trap that type of thing to ensure that just the model you are working with is triggered to create reports, but also to include that views or other things from other models in a shared database are brought along as well, where appropriate to do so.

Understand the interesting issues in doing this and thanks again for your reply and thoughts!

-John

Hervé

Hi,

I'm not sure I understand correctly ...

It is always possible to create triggers in your database to run a process that will generate html report or archimate files somewhere.

The difficulty is to determine on which table to create triggers.

On Github, you've got on model per repository, thus it is easy to know that any change on any element, relationship or view is a change in the model.

But in the database: elements, relationships and views can be shared across several models. It is even possible to export a single element or relationship without creating a new version of a model. It is even possible to create or update elements, relationships or views using SQL requests without running Archi and in this case, this can be done outside any model.

That why I implemented an asynchronous process with a daily batch that generate my html reports.

If you need reports that are in sync with your database, then, it may be worth doing SQL requests from your reporting system rather than generating HTML files each time an element or relationship is updated.

Using my database plugin, it is possible to have screenshots of your views in the database in order to show them in your reports.


Regarding your question, it is possible to sync back and forth your model from any archimate file, or even across several databases. Before exporting your model to a database, the plugin calculates a CRC code of every single component to determine their status in the database (present and up to date, present but different, not present, ...). In fact, this ability to mix archimate files and several databases, in addition to the ability to share components between several models, are the core features of my plugin  ;D

Best regards
Hervé

newbie1234

Hi Hervé.

Agree with your points.  Will experiment a bit on the aspects you outline to see what may work for us.

Thanks again for your insight!

-John

Hervé

You're welcome.

Please do not hesitate to ask should you have any question ...

Best regards
Hervé

newbie1234

Hi Hervé.

One more question regarding the database component to ensure I am trying to import views correctly from another model in a database.

I have tried the following with the database set up in both "Template mode" and "Forced Share" modes.

I "right-clicked" on the Views in the Archi pan and selected "Import components from database"

I selected a view to import that was available.

The dialogue at the top of the import screen shows to wait.  Waited until this message was extinguished.

In past with older versions of the database this would bring that view into my view list and I could pull these into the model where they were needed.

With the 2.1.4 plugin, it doesn't appear for me in the view list in Archi.

I enabled the expert logging and it wasn't very large so I have included here in text.  It shows that the import started and ended (I think?)  Should I be waiting for a message or looking somewhere to find the view that was imported or am I too impatient and not waiting long enough (the button to close the dialogue box allows me to close at that time)?

Since I am a mac, I am also going to try this on a p.c. that has the plugin installed today to see if the behavior is different.  If it is, I will let you know.

Many thanks in advance again for your gracious help!

------------------
Database Plugin Log:

2018-11-02 08:53:21 INFO   537:DBPreferencePage                         - Logger initialised.
2018-11-02 08:53:29 INFO    37:DBImporter                               - Importing model.
2018-11-02 08:53:29 INFO  1179:DBGui                                    - Connecting to the database ...
2018-11-02 08:53:29 INFO  1179:DBGui                                    - Checking the database structure...
2018-11-02 08:53:33 INFO   616:DBGuiImportModel                         - Importing model "Training_Hub"
2018-11-02 08:53:33 INFO   719:DBGuiImportModel                         - Importing folders ...
2018-11-02 08:53:33 INFO   726:DBGuiImportModel                         - Importing elements ...
2018-11-02 08:53:34 INFO   733:DBGuiImportModel                         - Importing relationships ...
2018-11-02 08:53:34 INFO   554:DBArchimateModel                         - Resolving source relationships.
2018-11-02 08:53:34 INFO   570:DBArchimateModel                         - Resolving target relationships.
2018-11-02 08:53:34 INFO   741:DBGuiImportModel                         - Importing views ...
2018-11-02 08:53:34 INFO   748:DBGuiImportModel                         - Importing view objects ...
2018-11-02 08:53:34 INFO   758:DBGuiImportModel                         - Importing view connections ...
2018-11-02 08:53:34 INFO   605:DBArchimateModel                         - Resolving source connections.
2018-11-02 08:53:34 INFO   621:DBArchimateModel                         - Resolving target connections.
2018-11-02 08:53:34 INFO   771:DBGuiImportModel                         - Importing images ...
2018-11-02 08:53:34 INFO  1179:DBGui                                    - *** Import successful ***
2018-11-02 08:53:46 INFO  1179:DBGui                                    - Counting model's components
2018-11-02 08:53:46 INFO  1179:DBGui                                    - Connecting to the database ...
2018-11-02 08:53:46 INFO  1179:DBGui                                    - Checking the database structure...
2018-11-02 08:53:46 INFO  1179:DBGui                                    - Getting list of views from database ...
2018-11-02 08:53:51 INFO  1179:DBGui                                    - (1/1) Importing view "F5".


Hervé

Hi,

You did right ... You should definitively see the imported view in the views folder.

Could you please open an issue on Github with more details (version of Archi and database brand). I managed to have a MAC to conduct some testing. And of course, please let me know if you experience the same behaviour on a PC.

For an unknown reason, I had to compile 2 versions of my specialization plugin, one for PC and the other for MAC. Even with the same code, the JAR file have some differences and behave differently. I may need to do the same for the database plugin...

By the way, you configured the logger in "info" level. If you need more details about what the plugin does, you may change the log details to "debug" or even "trace" on the preference page.

Best regards
Hervé

newbie1234

Thanks Hervé.

Will open up issue in Github as you suggested.  Thanks again for your help.


Hervé

Dear all,

I'm please to inform you that the version 2.1.7 of my database plugin has been released on Github (https://github.com/archi-contribs/database-plugin/tree/master/v2).

If the plugin is not yet installed, you may download the JAR file and copy it into Archi's plugin foler.
If the plugin is already installed, you may either replace the JAR file with the new one, or click on the "check for update" button on the preference page.

Please remember to configure your proxy in the archi.ini file if you're behind a corporte proxy:
-Dhttp.proxyHost=<name or ip of the proxy>
-Dhttp.proxyPort=<port>
-Dhttp.proxyUser=<username>
-Dhttp.proxyPassword=<password>
-Dhttps.proxyHost=<name or ip of the proxy>
-Dhttps.proxyPort=<port>
-Dhttps.proxyUser=<username>
-Dhttps.proxyPassword=<password>
-Djdk.http.auth.tunneling.disabledSchemes=
-Djdk.http.auth.proxying.disabledSchemes=


This new version brings few enhancements and fixes:

  • Import components from database:

         
    • updating a view from the database now updates the elements and relationships referenced in the view
  • Fixes:

         
    • Fix version calculation during export
    • Fix SQL requests when the database plugin is called by the script plugin
    • Fix label position in debug window

Best regards
Hervé