database plugin vs collaborative plugin : your opinion ?

Started by Hervé, December 07, 2016, 13:23:56 PM

Previous topic - Next topic

Would such a collaborative plugin usefull for you ?

Yes, definitely
4 (80%)
Yes for a collaborative plugin, but not as you describe it
1 (20%)
Not realy, I will stick to the database plugin
0 (0%)
no opinion
0 (0%)

Total Members Voted: 5

Hervé

Dear all,

As you may know, I'm currently working on a database export/import plugin (https://github.com/archi-contribs/database-plugin).

It does what I expect from a database export and import plugin : it allows to export models to a database (and of course, import them back).

Nevertheless, it has got an intrinsec limitation : it considers models as a whole. One can import a whole model or export a whole model. Thus it is not possible for several people to work on the same components and adapting my database plugin would require a huge code rewrite and will impact (again) the datamodel.


So rewrite for rewrite, and after talking with some of you guys, I was thinking of releasing the version 1.0 of my database plugin (removing the "shared mode" that does not work as expected) ...

... and start a brand new collaborative plugin.


The key concept being separating the components that would become common across all the models and the models that would only exist to store the components' graphical representation (views, colors, fonts, position, line thickness, ...).

It will of course still be possible to isolate a model (i.e. dedicating components to a very specific model) as it is done with archimate files by dedicating a database or a database schema to this model.

But storing several models in the same database or database schema will allow these models to share all their components.




To provide more details, I was thinking about (helter-skelter) :

On the components (elements and relationships) level :

  • The Archi components whould all be stored in a common store (without any notion of model nor folder at this stage)
  • The components would all be versionned individually (version number, creation/update data, username, purpose, ...)
  • The components' owner would be able to set access rights (at least read/write or read/only at the beguinning)
  • One could update a component (i.e. adding a new version of a component in the database) if and only if it posseses the last version of the component. If this is not the case, a kind of gitHub comparison mecanism will allow to choose what to do (replace our component by the last one in the database, create a new version of the component in the database, merge inforamtion from the two versions, ...)
  • It would not be possible de delete a component, but just to flag it as deleted. This way, it would be possible to keep the information that the component existed at one stage

On the models level :

  • A model would just be a collection of graphical objects, able to represent any component from the common store
  • It would be possible to use any component in any model, thus a component could be present in several models, each using a specific graphical representation and storing them in their specific folder.
  • This mean that it would be possible to import a single component or a group of components from the common store or even all the components from another model
  • When syncing a model with the database, the components that have been created or modified would be stored as a new version (keeping an history on all the modifications), and the components that have not be modified would be left untouched in the database
  • A model could reference a specific component version (i.e. not beeing impacted when a new version of the component is created) or reference a component without the version reference (i.e. at all time the latest version of the component. if a new version of the component is created, even by another person, even through another model, then your model is updated)
  • An option will allow to choose when this update is done : when importing the model from the database or each time the model is exported (in that case, it would be a synchronisation process).


In a more or less far future, I will probably evolve to a real-time collaborative plugin (i.e. when a user updates a compoment that is prensent in your model, then your model is updated "real-time". But at the moment, the database sync would be done through selecting a menu option :-)


Ok, so your turn now ... Please tell me what you think about this ...
     Would it be something usefull to you ?
     What would be your expectations if I develop such plugin ?
     etc ...

Hervé

brain_work

Hello Hervé,

That's a lot to think about...

The easy part: I agree with releasing the current database plugin without the shared mode things :)

But I find it hard to say something about the collaborative plugin because I'm not sure if I understand the way of working (and therefore the consequences) correctly. You say "A model would just be a collection of graphical objects, able to represent any component from the common store". Does that mean that a model becomes just the set of specific views and that all modelling elements are stored in the common area?

How would that work if I would want to make a new model/view. How can I select existing elements from the common area?

Regards,
Peter











Hervé

Hi Peter,

Yes, something like that.

While you are working in a model, you will of course be able to create new elements and relationships, exactly the same way as you do now.

But you would also be able to import elements and/or relationships using a search mechanism :

  • by name
  • by creation or update date
  • by owner
  • by type
  • by properties
  • by its usage in another existing model
  • etc ...
[li]
[/li][/list]

You will then be able to narrow your choice by viewing all the components details. The point is that you would be able to import one individual component, or several of them.

On a practical point of view, this search mecanism could be accessible by a new menu entry or a right click in a view.



The key point is that rather than creating relationships between components of several models (the way I imagined my database plugin), the new way would be to physically reference the same element or the same relationship inside several models. This is now possible as the new beta version of Archi 4 generates real unique components IDs.



As I will keep an history of all updates on all the elements and relationships, it will be possible to reference a component "as such" or in a point in time :
     --> "as such" meaning that when the element is updated by anyone in any other model, then the update is repercuted into your model
     --> "point in time" meaning that your model is using the component as it was at a specific date and therefore will not change when a new version is created



So the plugin behaviour would be :

  • when you work on your model

  • If you try to modify a component that is tagged as being a specific version, then your modification is cancelled and a popup window would explain why
  • A new menu (or right click sub-menu) would allow you to import existing components from the common store


  • when you export your model in the database

  • The new components you created are also created in the common store
  • The components you modified are saved in the common store as new versions of the existing components
  • If a conflict is detected while crating this new version (i.e. someone created a new version of some components before you), then you will be able to compare your updates vs the other person updates and choose which one to keep (like GitHub does)
  • all the components that you did not update are left untouched in the database


  • when you import a model in the database

  • For the components tagged as "as such" : the latest version is imported and modifications are allowed
  • For the components tagged as "point in time" : then the specific version is imported and modifications are forbidden



I hope I answered your questions. If not, please do not hesitate to tell me.

Best regards
Hervé



brain_work

Hello Hervé,

Thanks for the additional info. I think it sounds very good so you got my vote :-)

Regards,
Peter

Hervé


Jean-Baptiste Sarrodie

Hi,

My comment in its short version: we have to meet and discuss about it, that will be much more efficient that write about it...

My comment in a longer version....

I have the deep feeling that you are here describing a very complex mecanism because you don't know EMF and haven't really played with my grafico plugin. Archi model is based on EMF (Eclipse Modelling Framework) but uses maybe 10 or 20% of its features for several historical reasons. For example, sharing elements or relationships across models is supported in a very simple way, it is also fairly easy (as EMF can be of course) to extend the save/load mecanism to use whatever technology you want (file, database...). There is even something called CDO which provides Database support, versionning and branching out of the box.

The only issue with all these "advanced" features is the time to implement them in Archi and support them in the longer term. But for anyone wanted to implement such feature set, that's the only real way to go because that's designed from the ground up for EMF. IMHO any other approach involving a similar amount of work but without those technologies would be a waste of time.

This lead us to an alternative, really light, way of providing very similar features: Grafico and Git...

Grafico is designed to make the Archi model usable in Git, this means that as soon as you export/import using the Grafico format, you can use all git features: versionning, branching, merging. I use this solution with several colleagues and even some customers now. The great advantage is that being based on Git you can imagine new kind of workflows and also can do things close to what you describe. For example, if you have you main model (A) in Git and want to merge it with another one (B), you just have to create a branch off the first/root commit (i.e. empty model), commit B on this new branch and do a cherry pick of this commit applied to the current master branch. And voilà you have merge A and B. If B is a kind of reference architecture that could evolve over time, then you keep the associated branch and each time B is changed, you do the same thing: commit the new version of B in its branch and cherry pick the commit to apply it to the master branch.

As of today the only real limitations with grafico are: imagination and Git knowledge. That's why we published our ideas about ArchOps: we want to keep it simple but powerful and open. So we'd like to develop a solution that would hide Git as much as possible, and we could do it this way:

  • The first step could be to implement a plugin for a GitHub like solution. This plugin would add a download/upload feature, server-side. A download would (server-side) load the grafico model of the selected branch and convert it to a .archimate file (with original branch name and commit id saved on model metadata). An upload would take a .archimate file as input, load it, convert it in grafico, commit it on a temporary branch based on the right commit and do a merge. This would then allow people to just connect to this server, download a model, work on it offline and later upload it with an automatic merge.
  • The next step would be to create an Archi plugin that would use webservices to authenticate to this GitHub like server and embed a UI inside Archi to browse the list of models and branches.

I even have a good candidate for such GitHub like solution GitBlit. GitBlit is an open source, java based solution that is easily extendable through (java) plugins. This makes it easy to include the code needed to work on the model and my grafico code.

For the moment the only thing that is missing is companies willing to co-fund this (open source) project. Of course funding here is to have Phil work on it and make it a well integrated solution.

Last (but not least), being based on Git would allow (power) users to automate things, for example re-generate an HTML export after each commit, sync a CMDB...

Again, I think we have to discuss about it as I'm sure we could and should join our efforts.

Regards,

JB

PS: I really don't want to be harsh here because I really like your contributions. The issue is that in the past 5 years I've seen several people trying to achieve this goal but wasted their (precious) time. All these tries (I known about at least 4) stopped when people realized that they either need to really master EMF or have try to do too many things. My UNIX background makes me adhere to the KISS principle and that's why I think that instead of reinventing the wheel we should use solutions that already exists, like CDO or Git, Git being IMHO the simplier one.
If you value and use Archi please consider making a donation! https://www.archimatetool.com/donate

Hervé

I understand and fully agree with you : I do it my way because I've got no knowledge whatsoever about EMF.

The EMF framework may be the best framework in the world and may be able to do what I wish to do out of the box ... I haven't got a clue on how to use it and I do not understand the documentation as they take for granted a lot of concepts that I never heard about.

Please remember : I am not a developer and I wrote my first Java code 2 years ago.

My will is not to spend hundreds of hours to learn the EMF framework and create a beautiful and optimised code. My will is to ease my day to day work using Archi and to achieve this I need to bypass its key limitation : the dot Archimate file.

  • It may be possible to store models in a central database using EMF out of the box but nobody did it so far. Today my database plugin does it.
  • It may be possible to have collaborative work using EMF out of the box but nobody did it so far. So I begin a plugin that will do it.

Graphico is a great plugin, and Git is very powerful but I (and everyone in my team at work) failed to resolve the conflicts because our Git clients (we tried 3 of them) were adding Java comments inside the XML files and therefore we could not import them back into Archi without editing all of them one by one !!!

I know you will argue that we were not using the right software or that we did not configure it correctly. Once again, you will be right. But the point is that we did not find a way to have something working smoothly, without a huge amount of time of manual work. Git, and Graphico by extension, are tools made by developers, for developers and we failed using them because we are not developers !



Please do not misunderstand me : I'm not using the EMF framework because I do not want to; I'm not using the EMF framework because it is too complex for me (at least for now).


You're not harsh and I'm not upset. What you describe is the reality. But I've got three choices here :

  • do my best to write the plugins I need with my (very) little programming skills and share them with the community (without any waranty that I will provide any long term support, once again you're right)
  • do my best to write the plugins I need with my (very) little programming skills but keep them for my own selfish usage
  • wait until someone more qualified than me uses the EMF framework and release their own plugin.

You already know my choice  ;D



This said, I've always been and will always be opened to all criticism (constructive of course, as you did today). I would be very happy to talk with you about this and gather any advice you may provide me.

I'm looking forward to seeing you.

Best regards
Hervé

Jean-Baptiste Sarrodie

Hi,

QuotePlease remember : I am not a developer and I wrote my first Java code 2 years ago.

I'm neither a developper and started working on Archi 4 years ago without a clue on what the hell eclipse was :-) Hopefuly, Phil helped me a lot (and still does it almost every day).

QuoteMy will is not to spend hundreds of hours to learn the EMF framework and create a beautiful and optimised code. My will is to ease my day to day work using Archi and to achieve this I need to bypass its key limitation : the dot Archimate file

That was exactly the same for me, and also like you I didn't wanted to interfer with Phil's work and Archi's main code, so as CDO is an intrusive thing that was not my first choice (but someone did it 4 years ago but considered it a hack and unfortunately never published his work). That's the reason why I developed Grafico which only relies on EMF's default XML mecanism to load/save elements (meaning that I really don't care about EMF metamodel changes, i.e. I had nothing to change to use it with Archi 4).

QuoteGit, and Graphico by extension, are tools made by developers, for developers and we failed using them because we are not developers !

Grafico hasn't been made by developers because I'm not one. The focus has been on simplicty (Grafico is less than 500 lines of code splitted in only 2 files). Of course, Grafico alone provides no value, so of course Git (or any other simplier merging solution, like winmerge) is needed. This was a choice based on the fact that I wanted a simple solution for me and my team (and we were able to use git). But instead of not using it and reinventing the wheel, it would be better to keep Grafico and Git but make the whole solution more user friendly.

As I said, Grafico is in no way a finished product, that's only one piece in a system that has to be extended. That's why instead of trying to develop another solution I'd really like you to:

  • Retry Grafico with me in support to understand what issues you faced (TBH I never ever faced the issue you speak about, and I used more than 4 git clients).
  • Work with me on a plugin (or any other solution) that would hide Git to the user.

Let's join our efforts.

Regards,

JB
If you value and use Archi please consider making a donation! https://www.archimatetool.com/donate

Hervé

Hi JB,

I sent you my mobile phone number in a private message. Using a Git repository wouldeffectively be the best solution, but I won't be able to do it on my own.

I fully agree : let's join our efforts  ;D

Best regards
Hervé

brain_work

So it will be wise for me to learn more about GIT and the GRAFICO plugin?

Then I have the following questions:

  • Is the GRAFICO plugin so designed/written that it works with all (near-future) versions of Archi?
  • If not, are there any known issues with the GRAFICO plugin in combination with Archi 4 Beta?

Regards,
Peter

Jean-Baptiste Sarrodie

Hi Peter,

Quote from: brain_work on December 15, 2016, 08:16:13 AM
So it will be wise for me to learn more about GIT and the GRAFICO plugin?

Then I have the following questions:

  • Is the GRAFICO plugin so designed/written that it works with all (near-future) versions of Archi?
  • If not, are there any known issues with the GRAFICO plugin in combination with Archi 4 Beta?

As I both work on Archi and maintain the Grafico plugin, the short answer is that it will support all upcoming versions. And the latest version (beta 0.4) already supports Archi 4 beta.

As said, for the moment you need some git knowledge so you'd better read some documentation. Of course it is simplier to use a GUI frontend than the command line (in my case I worked with: SmartGit, SourceTree, TortoiseGit).

I'll start soon a separated topic to discuss about Grafico and its future, but the key thing is that I expect people to test it and provide as much feedback as possible so that we can alltogether define the best (but simpliest) way of wollaborating on an Archi model.

Regards,

JB
If you value and use Archi please consider making a donation! https://www.archimatetool.com/donate

brain_work

Thanks, I've found the free Pro GIT book (luckiliy for me also in Dutch) on the internet and I've installed GIT locally so I can start learning GRAFICO :-)

Regards,
Peter

brain_work

I'm missing the pictures on the archi-contribs/archi-grafico-plugin wiki so I've a hard time to understand how I need to use GIT in combination with the GRAFICO plugin. Is it possible to restore the missing images or is there another description available for GRAFICO newbies like me?

Regards,
Peter

Jean-Baptiste Sarrodie

Hi,

Ouch, I forgot that images were hosted on Google Drive but Google remove the direct file serving options some month ago. I'll work on it, but in the meantime you can find all images here

JB
If you value and use Archi please consider making a donation! https://www.archimatetool.com/donate

brain_work

Thanks for the images:-)

It took me a while to get a bit comfartable with Git (Ry's Git Tutorial was very very helpful, it's a free for kindle, see https://www.amazon.com/Rys-Git-Tutorial-Ryan-Hodson-ebook/dp/B00QFIA5OC) but I think I've got a working workflow now with a central bare repository on SharePoint and local working repositories on desktops.
BTW I must work with Git for Windows Portable because our laptops are locked down so we can't install software.

Regards,
Peter