Database export/import plugin

Started by Hervé, April 15, 2016, 14:37:31 PM

Previous topic - Next topic

Hervé

Hi,

As long as I do not have to rewrite all my SQL request, it should be easy to add the SQLite drive. Else, I will learn SQLite synax  ;D

I'll try and let you know.

Best regards
Hervé

bdendulk

Hi Herve,

I'm running into a new problem. It is related to the use of tags for custom Jasper reports. I use them quite extensively to make reports based on a single view instead of reporting the whole model (see the forum for discussions on this subject). Apparantly the export plugin stumbles on these.

My model is on GITHUB, hence you can have look and/or test the pluging with my model. The content is in dutch but that shouldn't be a problem since the content is not an issue here ::)

See attached screenshot for error message (the number at the end is the version of the export). '_Project' is the property name that is used to filter a certain view for reporting. I have more than 20 different properties in use with a value for each view that is used in different reports.

Kind regards,

Bart den Dulk

Hervé

Hi Bart,

Thanks for your feedback. I'll  investigate on your error and correct it. Properties are supposed to work but as I do not use them a lot, I may have a bug ...

Could you please provide me with the download link of your model ?

Best regqrds
Hervé

bdendulk

Hi Herve,

Sorry, I meant to include it in my last post but I apparantly got distracted.

https://github.com/bdendulk/NVWA_CA_BLIK

Kind regards,

Bart

Hervé

Hi Bart,

Thanks for the link. I downloaded it and reproduced on my desktop. I shall be able to provide a working version of the plugin for your model within the next few days  :D

Best regards
Hervé

Hervé

Dear all,

The plugin version 0.5 has just been released on GitHub : https://github.com/archi-contribs/database-plugin

You've got some screenshots on the Wiki : https://github.com/archi-contribs/database-plugin/wiki

The key updates are:

  • The SQLite driver has been included and tested
  • The folders and canvas are now managed
  • The plugin now implements two modes : The standalone mode is the standard one where each model is imported into its own container. The shared mode allows to import several models into a single container, each of them inside its own folder.
The shared mode is not finished. Even if it allows to import several models into separate folders, it is not yet possible to include elements from a model to another one ... but it eventually will (because this is my main objective  ;D).

Of course, my code is not optimized at all (I prefer working on adding functionalities), so import and export takes quite a long time at the moment.

Please note that the datamodel has to be upgraded so either you scrach you database and create a new one using the scripts on GitHub, either you need to update the tables manually. I apologize but this is still an alpha release after all. I believe there will be other database upgrades in the coming weeks (at tleast to optimize).

Please feel free to drop me a line with your feedback  ;D


@bdendulk, I tested the plugin against your Grafico model and it seems to work. Could you please check it on your side ? You need to upgrade the datamodel I'm afraid.

Best regards
Hervé

bdendulk

YES!!!

(see sccreenshot for evidence ;-))

Hervé

Hi Bart,

Thanks for your feedback ...

But please do not declare victory too soon, I haven't had enough time double check that nothing has been forgotten  ;) I need to compare your model imported through Grafico and through my plugin ...

Best regards
Hervé

prgee

Hello Hervé,

I just wanted to give you some results on my test of ver 0.5 of your plugin. I have a very large model that I have been working with that contains almost every possible Archi feature. It took about 1.5 minutes to export to an "out-of-the-box" PostgreSQL 1.16.1 installation on Windows 7. However, some things did not export as noted in the attached screen shot. I was not able to re-import the model due to a large number of missing connections. I do have a couple of suggestions:

- display some sort of progress message to indicate how far along the export/import is
- if a problem is encountered with the export/import then offer the user the option to "cancel" or "continue"

That's all for now.

Regards,
Peter

Hervé

Hi Peter,

Thanks a lot for your feedback and your suggestions. I do not have such complex models and the duration you've got tells me to give more priority to optimization.

The missing connexions should disapear in a next release as I will use "proxies" as Jean-Baptiste suggested and showing a progress indicator is definitively a good idea.

Best regards
Hervé

Hervé

Dear all,

The version 0.6 of the plugin is available on GitHub : https://github.com/archi-contribs/database-plugin

You've got some screenshots on the Wiki : https://github.com/archi-contribs/database-plugin/wiki

The key updates are :

  • bug corrections (mainly about missing folders that caused several messages like "cannot import as we do not know its parent"
  • few optimizations (but a lot needs to be done on this part)
  • the plugin beguins to manage models relations : in shared mode, elements that are included in a view of another model are now imported and exported ... even if at the moment the models should be imported in the correct order.

Beware, the datamodel changed from previous version.

In the next release, it will be possible to import several models at the same time, and eventually recursively import model dependencies (a slider is already in place in the import popup to limit the recurse depth). I will also replace the statistics popups by a new one that will provide more "real time" information about the import/export process.

I tested the plugin against my own models and against Bart's but I know that does not cover all Archi features. therefore I'm interested if some people can provide me more test models.

@Peter, As your model includes all Archi features and is quite big, I would be interested to have your model, if it does not contain confidential information of course. It would definitely help me  :)

Best regards
Hervé

prgee

Good Day Hervé,

I tried exporting the same "test" model that I did previously but got the attached error message. As for your request for a copy of the file, it does contain confidential data but I will create a version of it with masked values so you can use for testing. I'll try to get that for you in the next day or two.

Thanks and Regards,
Peter

Hervé

Dear Peter,

I've got the same error message. I've been too much focused on performances issues yesterday that I messed up a bit. I just uploaded a new JAR on GitHub (version 0.6c) that should solve this issue.

At the moment, it takes me 2.3s to export Bart's model (which is the biggest model I've got so far) when it takes me more than 12s to import it back. This means that I've got an improvement ratio of 5:1. If you needed 1.5 minutes to export your model, then it means that it would take 10ish minutes to import should no error popup shows up ... Optimization is now my number one priority  :D

Using your huge model, it will be much easier to calculate where I'm spending time in my code and therefore, where to concentrate my effort. So thank you very much in advance for your effort in providing me an anonymized version of it ...

Best regards
Hervé

Hervé

Dear all,

I uploaded the version 0.7 of the plugin on GitHub (https://github.com/archi-contribs/database-plugin).

This version has no new functionality and works with the same datamodel as version 0.6 ... I only concentrated on Java code optimization.

The export method is still the same speed (arround 2.3s to export Bart's model) but the import method duration has been divided by 5 and is now the same speed than the export method (around 2.3s instead of 12s before).

I believe there still is a lot to do regarding optimization and look forward to receiving Peter's model to work on a huge model.

Best regards
Hervé

prgee

Hello Hervé,

I have a version of that large Archi model ready - we just need to figure out how to get it to you. I have placed it on my Google Drive so if you have a gmail or other Google account I will grant you access. I also have a GitHub account if that would be easier to use.

Regards,
Peter