Database export/import plugin

Started by Hervé, April 15, 2016, 14:37:31 PM

Previous topic - Next topic

Hervé

Hi Peter,

I just sent you me gmail address in a private message.

Thank you very much for the model.

Best regards
Hervé

Hervé

Hi Peter,

I downloaded your file.

Thank you very much.

It takes me 9 minutes and 20 seconds to open your .archimate file in Archi on my desktop ... I'm impressed. It is really huge !

So my objective for the next version is to manage your model. My priorities are therefore :

  • Manage all Archi components (including sketches and images) which means using blobs ... what I was postponing as long as I could as I'm using a PostGreSQL database and there are some limitations regarding blobs in the PostGreSQL JDBC driver. But time is come to have a look at it ...
  • Show up a monitoring of the import/export process with a time remaining estimate (that becomes mandatory for such a duration)
  • Import a model with all its dependencies, with a depth limited by the slider that I already prepared
  • and of course, correct bugs and improve performances

This one may take a bit more time than the former ones ;)

And of course, the datamodel will change, once again but I hope this will be the last time ...

Best regards
Hervé

Hervé

Dear all,

Please find below some information about the plugin development ...

  • All the Archi components are now exported correctly (including sketches and images that were missing in the previous versions)
  • Nearly all the components are imported back (I still need to work on images and metadata). I manages to solve the import order without using EMF proxies as Jean-Baptiste suggested because I failed using them so I developed my own proxies
  • I need to compress the images as they are taking a lot of space in the database.
  • I may also change the way I manage them at the moment ... as if several models are using the same image or the same set of images, then it will be more efficient to store them once
  • I also still need to add a progress bar but it means I need to use threads and therefore, my code needs to be thread safe

So, as you see, it will take a bit longer than I expected. But be patient, i'll do my best to release something next weekend  :D

Best regards
Hervé


ps: Thanks again Peter, your model is very helpful  ;) By the way, how did you manage to get a DELETED folder after the views ? It's not a standard Archi folder, is it ?

prgee

Hello Hervé,

Glad to hear the model is useful for testing. It grew way to large to manage so I ended up breaking it into smaller models - one which contained only the current-state model tree and a few basic views (e.g. organization, capabilities, principles, etc.) and several others related to various project-related and future-state views. However, in doing so it became more important to be able to manage inter-dependencies from several related model files. It seems that you are addressing this requirement in your database repository approach.

As for the DELETED folder, that seems to be something that was created automatically in earlier versions of Archi but was later dropped. Perhaps it was unnecessary because Archi provides "undo" and "redo" functionality. Maybe Phil could verify this.

Thanks and Regards,
Peter

Hervé

Hi Peter,

Yes, I confirm, allowing dependencies between models is one of my objectives. It is achieved by importing different models into separate Archi folders of the same model and therefore you will be able to move a component between models by moving them between folders, using components of one model in a view of another model, and even create relations between components that belongs to different models. Then, the export process split back the "big model" into separate models which is quite easy as each of them are inside separate folders.

I need this functionality at work so it has to work !

By the way, I optimised my code, and my plugin is now about 3 times quicker to import your model from a SQLite database comparing to open the archimate file (3 minutes 19 seconds versus 10ish minutes). Unfortunately, showing up a progress bar and refreshing it may slow it a bit but I hope the import will stay below the 4 minutes ...

I hope I shall be able to deliver a version at the end of the weekend.

By the way, I had a look to your Internet site and to the PDF that is generated using Archi. I also wish to be able to generate documents as you do. Do you know a good site that describes how to start using Jasper Reports (like Jasper Reports for dummies) ? I begin to have a look and it seems very complicated to me ...

Best regards
Hervé

cadriano.goncalves

Hi Hervé,

After applying the version 0.7 of the plugin, the following error appears:

column "archimateelementid" specified more than once

This error comes up with postgresql and with mysql (with a new and clean database).

Best Regards,
Carlos

Hervé

Hi Carlos,

Thanks for your feedback. I do not remember having this issue but I will release the version 0.8 very soon (tonight in fact if everything goes well).

Best regards
Hervé

Hervé

Dear all,

My apologies but I still have got some issues to manage images. I found a way that works in my dev environment (Eclipse) but doesn't when the JAR is created and copied to the plugins folder.

I won't be able to release the version 0.8 before few days.

Best regards
Hervé

Hervé

    Dear all,

    After being helped by Phil Beauvoir, I'm happy to announce that the version 0.8 of my plugin is available on GitHub (
https://github.com/archi-contribs/database-plugin).

Installation instructions and an embryonic wiki is available on GitHub.

What works :

  • The plugins is able to export to import from 4 different relational database brands (tested on MySQL, PostGreSQL and SQLite, not tested on Oracle but the driver is included),
  • It is now able to handle all the Archi components including images
  • It allows to import models either in standalone mode (i.e. as a standard Archi model) either in shared mode (i.e. in a set of folders of a unique Archi model)
  • It allows to version the models, hence to keep an history on what has been done
  • It allows to search for a model containing a specific element or property (using filter on the import window)
  • It allows to follow the import and export processes through a progress bar
  • It shows up statistics about the number of components imported and exported
  • It compares the number of elements really imported or exported to the number it should have imported o exported, and highlights all errors found
  • And few other minor ones ...

There is one known bug (but as I did not have much time to test it thoroughly, I count on you to find more) :

  • In shared mode, when adding models, the images that have been previously imported are unnecessarily re-imported. This has no impact on Archi behaviour but is not efficient and leads to an unnecessary error report at the end of the import process.

Pre-requisites :

  • The plugin has been compiled and tested against the version 3.3.1 of Archi but should also work on version 3.3.2.

My next priority (except to correct bugs that will necessarily arise) is to allow to use elements from one model into another one, or create relations between elements of distinct models. Basically, this is nothing more that enabling to load and save part of a more global model. The point is that now that models can be imported as folders of a single model, it is already possible to do such things but the import and export procedures need to manage them correctly.[/li]
[/list]

After that, I will study the possibility for several people to work on the same model at the same time. I just discovered that it is possible to subscribe to events that are raised each time an Archi component is updated. Therefore, it will be possible to share those updates with the other people working on the same model (technically easy, but do not know the performance impact). Of course, the tricky part will be, as usual, to manage concurrency (when 2 people are updating the same component at the same time). But imaging having a view opened and see in real time the components added/modified/moved, removed by other people. I am thinking about the ability to set a lock on components when someone whishes to be the only one allowed to update a component (kind of check-in/check-out).

But it's a bit too early to have a clear picture of what I wish to achieve here. I would be interested to have your opinion on this subject ...

Please do not hesitate to drop me a line should you have any question or comment, found a bug or require a new functionality that is not already in my roadmap.

Best regards
Hervé

theunsfvo

Great work Harve... this is what makes open source projects like this so awesome and successful.

I will see if I can help testing and start using it from my side...

Excellent contribution...  ;D

prgee

Good Day Hervé,

Thanks for the update - I'm starting to do some testing on your ver 0.8. So far, I have discovered one small issue. Phil has added a test for invalid characters to the CSV import code so it will no longer process ID's which contain ".". This interferes with your default version numbering scheme which uses a "0.01" type format. I've been able to work around that by entering my own version numbers when I export to the DB but it's probably simple to change your numbering format.

Thanks Again!!
Peter


Phil Beauvoir

Peter, it turns out that "." characters are OK for identifiers. I've just committed a change to GitHub that allows them to be used in IDs.

Phil
If you value and use Archi, please consider making a donation!
Ask your ArchiMate related questions to the ArchiMate Community's Discussion Board.

prgee

Thanks for the speedy response Phil. I'm smiling again ;D

Hervé, please consider this issue resolved.

Thanks and Regards,
Peter

Hervé

I like these kind of issues that solve themselves  ;D

More seriously, I knew that I may one day be confronted to this kind of issue because I am not supposed to modify the ID of objects, especially as I only develop an export/import plugin that is supposed to be non-intrusive ...  but I did not find any other way.

Thanks Phil for your update.

And thanks theunsfvo for your encouragement :D

Phil Beauvoir

I can change the code if it makes sense, and doesn't impact other things.

The only issue is when a new build comes out. I'm trying to solve that problem too with automated builds. So far I have a local Maven and and Ant build process. I'm trying to deploy that to a Jenkins CI process.
If you value and use Archi, please consider making a donation!
Ask your ArchiMate related questions to the ArchiMate Community's Discussion Board.