Database export/import plugin

Started by Hervé, April 15, 2016, 13:37:31 PM

Previous topic - Next topic

prgee

Hi Hervé,

I just downloaded your 0.3 version and have been doing some tests (I'm on Windows 7 using PostgreSQL) and it's looking very good. I have a couple of large Archi models so I'm interested in seeing how the export/import performance is.

Great piece of work!!

Thanks and Regards,
Peter

Hervé

Hi Peter,

Please remember that the plugin is not able to handle all the Archi components at this stage (if some of them are not exported, it will tell you).

I'm interested to have your feedback about performances and please do not hesitate to drop me a line if you discover a bug (I'm sure thay are plenty of them) or require some functionality.

Best regards
Hervé

bdendulk

Hi Herve,

I've looked into my MariaDB server settings. It is using a charset which is case insensitive. I have no clue whether it is possible to make the NAS as a whole case insensitive. I will start using a local MySQL stack om my PC.

Kind regards,

Bart

Hervé

Hi Bart,

Please don't ... You may instead download the plugin version 0.4 that I compiled for you : I updated my code to use lowercase tablenames everywhere ...

Please let me know if this doesn't solve your issue ...

Best regards
Hervé

bdendulk

Hi Herve,

No luck. I still get the error message after I get the screen where I can specify which model I want to export. I reckon it has something to do with MariaDB.

Kind regards,

Bart

cadriano.goncalves

Hi all,

I'm testing archi, and in my point of view, one aspect that is missing is a plugin like this one.

I'm trying with ubuntu 14.04 and MySQL 5.5.47 and i had the same issue related with "Model" and "model"... it seems that is related with DBExporter.java:

...
// we remove the old components (if any) from the database
         for(String table: asList("Model", "archimatediagrammodel", "archimateelement", "diagrammodelarchimateconnection", "diagrammodelarchimateobject", "relationship", "point", "property"))
            DBPlugin.sql(db, "DELETE FROM "+table+" WHERE model = ? AND version = ?", modelSelected.get("id"), modelSelected.get("version"));
...

Regards,
Carlos Gonçalves

Jean-Baptiste Sarrodie

Hi,

Quote from: cadriano.goncalves on May 11, 2016, 18:41:38 PM
I'm testing archi, and in my point of view, one aspect that is missing is a plugin like this one.

Please don't get me wrong, no criticisme intended, but I have to make some comments:

  • This plugin with its current feature set is a good start but doesn't solve at all the real need that most users have which is to do collaborative work. In its current state (and I know Herve's goal is to do more) it only loads and saves to a DB but this is done at model level. So if 2 users open the same model and after some work decide to save it, only changes done by the one who is the latest to save will be kept (the first changes will be lost)
  • This plugin rely eavily on low-level details inside Archi. This means that when model structure will change (and it will in order to be compatible with upcoming version of ArchiMate) big changes will have to be done too on this plugin.
  • As most know, evil is in the details. In this context, this means that trying to implement collaborative features based on a central repository won't work as a simple import/export plugin because Archi UI have to be change in a way that permit to lock/unlock or checkin/checkout small subset of model (element or forlder for example).

These are the reasons why I created the Grafico plugin: the way the model is "exploded" in a bunch of XML files is done using some Eclipse features that are not linked to way Archi work with the model and thus changes at model (Ecore) level have only few impacts on the plugin. And the goal being to use Git, this means that there is no need to manage locks or checkin/out features making it useable as a simple import/export plugin.

So my conclusion could be summarized:

  • If one wants a collaborative mode with locks it has big UI impacts and cannot be done as a plugin like this one.
  • If one wants a collaborative mode that can be used in a distributed environment (like Git), an import/export plugin can be used and Grafico could be a good start.
  • If one wants something that will be supported other time when changes will occure and that supports all Archi feature, the best would be to simply pay Phil for that work (though crowdfunding for example)

And the conclusion of the conclusion is that it would be better if all people that want/are able to contribute on this topic work together on the same direction in a collaborative way. A first step would be to discuss what we want to do with such feature, what we really call "collaborative work"...

Regards,

JB
If you value and use Archi please consider making a donation! https://www.archimatetool.com/donate

Hervé

May 11, 2016, 19:57:13 PM #22 Last Edit: May 11, 2016, 20:03:16 PM by Hervé
Dear Bart, dear Carlos,

I effectively had forgotten to lowercase the tablename in one location  :-[ I've uploaded a new JAR (still version 0.4) that has been corrected. I installed MariaDB on my Synology and succedded in exporting and importing a model  :D

Dear Jean-Baptiste,

Yes you are right. At the moment, the plugin is only able to store models into a database but it already solves few points (at least for my needs) :

  • The models are centralized in a single location
  • The lodels are versionned so if two people save the same model at the same time, it is not overwritten (hum ... at least there is a popup telling one of them that he is going to overwrite an existing model and he can choose to do it)
  • It is possible to search models by element, relation or property

The other points are in my roadmap.

I'm not naive, I know that my plugin is only an export/import plugin  ::). Besides, I'm not a Java developer (everybody who read my code knows that  ;D) and I'm learning Java at the same time I develop the plugin (Thanks Google by the way). My aim is only to extend Archi to my own needs and bypass the key limitation : it works with individual XML files.

I tried Grafico. I created a GitHub server on a virtual server and installed a gitHub client on my desktop. But each time a conflict arised on an XML file, Java style comments were automatically added into the XML file with the data from both XML files (the one from my desktop and the one from the repository). Of course, the XML structure was not kept and the whole model was corrupted. I spent quite a lot of time searching how to disable this behavior but I failed. I know it may be a competence issue coming from me, but as I'm not a developer I do not master developer tools. That's why I decided to create my own plugin.

Once again, my aim was to fulfill my own needs ... but I'm fully available to discuss about features ...

Best regards
Hervé

Jean-Baptiste Sarrodie

Hi Hervé,

Some comments:

"The models are centralized in a single location" : that can be achieved with a simple shared folder
"The lodels are versionned" : can be done with sharepoint, Git, SVN...
"It is possible to search models by element, relation or property" : that's a really nice feature!

"Besides, I'm not a Java developer (everybody who read my code knows that  ;D) and I'm learning Java at the same time I develop the plugin (Thanks Google by the way). My aim is only to extend Archi to my own needs and bypass the key limitation : it works with individual XML files."

I'm not myself a developer so I can really understand you. The point is that because we are not developers we have to minimize the effort to achieve our goal. IMHO using a DB instead of a simple XML file doesn't solve anything at all.

"I tried Grafico. [...] But each time a conflict arised on an XML file [...]"

If conflict arise, that means that two users tried to both edit the same model "at the same time". Storing the model in a DB doesn't address this point: at best you'll end up with two versions that aren't merged. In addition, I think we should discuss (maybe private email discussion in french) why you had conflicts and what tools you used, because first I almost never faced conflicts (separation of work in my team has always prevented such issues) and Git is able to merge conflicts in more than 50% of the cases.

"That's why I decided to create my own plugin"

I understand and really respect this, But (unfortunately there's a but) you're the 4th or 5th user that has worked on this topic in the past 5 years. All previous attempts started with the easy part (storing into a DB instead of a file) but failed short when trying to address the hard part (the merge of changes). That's why about 3 years ago we had some discussion with other users to understand how to manage it and the best solution we came up with was to explode the model and use a source code management system. The first attempt (done by Michael Tapp) exploded the model in a bunch of YAML files but did so in a way that was too tighly linked to the way Archi store informations and thus was limited to only a subset of Archi's features and broke when some changes occured. I myself tried 2 implementation before Grafico but it took me almost one year to find what I think is the best approach (taking current constraints into account): rely on Eclipse EMF serialization feature to create small XML files and let Git manage the hard work.

Of course Grafico is currently in beta and only manages the import/export feature, not the merge itself, but IMHO it would be easier to contribute some UI (like you did for your plugin) and add the Git layer through JGit (pure java git implementation) than to work on a DB based implementation with its own merge feature. I truely believe that merge feature is something hard and that we should not reinvent the wheel.

I think you've done a really great work and I'm sure you've spent several nights and week-end working on it. So If there is a chance that we (and other) can work together to achieve our goal I'd like us to at least try.

Regarding the goal itself, I think we should at first define the features and not the technical solution we can imagine. So IMHO a good collaborative solution for Archi would be a solution that allows me:


  • to query a repository to find model (based on several criterias like name, elements, authors...)

  • to open a model that is stored into a repository

  • to save changes back into the repository in a way that saves my changes without interfering with changes done by other users on the same model in the meantime (but with some kind of alert in case of conflicts)

  • to open a model from a repository, work on it remotely (that is without access to the "official" repository) and (maybe some days or even weeks after) then save it back into the repo. This is important for those who works for several customers but not always from their network. This is the main reason why git is so popular now and SVN almost no more used.

  • to keep track of each versions and to name (tag) them if needed

  • to create branches (for example to model architectural changes needed for a potential project)

Of course, the technical solution should be as simple as possible and not (too) dependant of the way Archi store information (or even which informations Archi stores).

Regards,

JB

PS: On the other hand, that's the typical thing that Phil could develop if he was paid for...
If you value and use Archi please consider making a donation! https://www.archimatetool.com/donate

Hervé

Wow ...

When I started the plugin, I didn't expect to generate such a polemic ...

Hervé

Phil Beauvoir

Well, it's great to see such a level of interest in this initiative. :-)
If you value and use Archi please consider making a donation! https://www.archimatetool.com/donate

Hervé

For information, the functionality I'm currently working on is the ability to import several models and allow relations between them. Technically, this is achieved by allowing to import models in separate folders of a unique Archi model and exporting a folder as a separate model.

That is the simple part. The tricky point I'm working on is that I would like that:
   --> People importing a model should be "aware" of other models update that have an important impact on their model
   --> People exporting a model should be "aware" of the important impact their new version will have on other models

but I begin to have some ideas on how I will manage this  ;D

Best regards
Hervé

Jean-Baptiste Sarrodie

Hi,

Quote from: Hervé on May 11, 2016, 21:30:52 PM
When I started the plugin, I didn't expect to generate such a polemic ...

It wasn't meant to be a polemic, sorry if it seems so.

Quote from: Hervé on May 11, 2016, 22:42:09 PM
For information, the functionality I'm currently working on is the ability to import several models and allow relations between them. Technically, this is achieved by allowing to import models in separate folders of a unique Archi model and exporting a folder as a separate model.

That is the simple part.

Well... This is for sure a very interresting idea. In some aspects this looks like an UML Package.

I'm not so sure it is the simple part though :-) The complexity I can see is that you first have to define what folder you are talking about as there are no links between folders that group model elements from different layers: you have no way to be sure that the model is well splitted at all layers and that relationships are coherent. If you consider diagram folder, then there can obviously be some common elements between views included in different folders. So it can be simple to split the model (using folders or other criteria) but merging them back can be an issue.

Another potential issue if "just" merging model snippets into another model is that you have to make sure that element ids are uniq (should be ok most of the time but could fail if a sub-model has been created through a CSV import that defines Ids).

Maybe you don't know, but this is in fact a feature included into EMF. An EMF model can reference elements through proxies, a proxy being defined by the "path" to another model serialization file and the Id of the referred element. Archi has not been created at the begining with such feature in mind (and thus can't use proxies asof now) but this could be done with some developments.

Quote from: Hervé on May 11, 2016, 22:42:09 PM
The tricky point I'm working on is that I would like that:
   --> People importing a model should be "aware" of other models update that have an important impact on their model
   --> People exporting a model should be "aware" of the important impact their new version will have on other models

I think I got you but could you provide some examples you have in mind?

It seems that it is time to create a new page on the wiki to capture all these really good ideas :-)

Regards

JB
If you value and use Archi please consider making a donation! https://www.archimatetool.com/donate

Hervé

Hi Jean-Baptiste,

In fact, I'm implementing two import modes :

  • The first one that I called standalone is the standard one : one model in the database is imported in its own Archi model.
  • The second one that I called shared is the tricky one : I import several models from the database inside separate folders of a single Archi model. Everything being in a single Archi model, I'm able to include components from one model to another one and create relations between them.

I solved the ID conflict possibility by changing the ID : all the components imported from the database to Archi have got an ID compounded of 3 fields : its own ID and the ID of the model it belongs to (which is compound of its own ID and a version number).

I've seen how Grafico manages proxies and I got around by importing components in the correct order. I cannot guarantee that it will work forever, but at the moment I do not need proxies for the twenty-ish models I've created so far at work. Should I experience issues when importing components, then I will learn about those proxies.

All the other cases might not be well managed, or even not managed at all ... Nevertheless, the plugin will ease my life on a daily basis  8)

Best regards
Hervé


Jean-Baptiste Sarrodie

Hi,

Quote from: Hervé on May 12, 2016, 15:33:03 PM
All the other cases might not be well managed, or even not managed at all ... Nevertheless, the plugin will ease my life on a daily basis  8)

That's the main point and goal :-)

Re proxy: in fact Grafico doesn't manage them, or should I say I added code to managed them because I didn't want to manage it the EMF way (and also because at first I used a YAML serializer that just doesn't support proxies). Import order work if you don't manage view references: in such simple case, importing elements, then relationships then views works, but this is no more enough for view references because you can cannot predict the right order (and in worse case you can have circular references). As Grafico manages all elements type (archimate but also sketches notes and images) I had no other solutions.

Do you think it is possible to use a SQLite JDBC driver instead or MySQL or PostgreSQL ? This could help testing your plugin without having the need for a real DB server...

JB
If you value and use Archi please consider making a donation! https://www.archimatetool.com/donate