Archi 4.0.3 Beta 1 - faster load times!

[Phil Beauvoir]

Hi,

some of you may have noticed that large Archi models can be quite slow to load. I had one example model that took around 45 seconds to load. I may have found the cure. With a simple change, this test model now loads in under 1 second. 

However, there may be side-effects that I am not aware of. If you'd like to test the beta version please download it here:

https://www.archimatetool.com/dev/beta

Please only test on copies of models, or have backups. There may be side-effects!

Phil

[xbrossard]

Hi,

I hope that next Archi version include your improvment; actually my 6,5 Mo model took about 3 minutes to load at startup...
when I watch source code, I've see that the problem is apparently when creating views, when searching the corresponding model in the tree model object. Is that the optimization you have done?

[Phil Beauvoir]

Best thing is to try it and see. If you still have long load times then try closing some Views.

[Hervé]

Hi Phil,

I just came back from vacation and hadn't time do do a lot of tests, but on one of my test archimate file (16MB size) that took 25 minutes to load with previous version, the new 4.0.3 beta 1 takes only few seconds.

Very impressive

Well done mate 

[Phil Beauvoir]

Thanks, Hervé.

It's an interesting story...

I'd always assumed that the slow load speed was due to EMF taking time to load a monolithic XML file and had resigned myself to the idea that this doesn't scale on large models. I was then planning on making deep changes to the persistence format to save and load multiple files and loading them on demand.

Then when testing the new repository plugin, using JB's "grafico" format we noticed that a large model loads very quickly. The grafico format uses multiple XML files for each object. I started to think...

So I Googled "EMF slow load" and found this:

https://www.eclipse.org/forums/index.php/t/796300/

Ding!

So I added:

resource.getDefaultLoadOptions().put(XMLResource.OPTION_DEFER_IDREF_RESOLUTION, Boolean.TRUE);
resource.setIntrinsicIDToEObjectMap(new HashMap());

and made some adjustments elsewhere and bingo!

Now, the thing is, I want to know that if storing the IDs in a HashMap has any side effects. So far, I don't think it has because the Resource is discarded once the model has loaded.

Phil

[Hervé]

I must admit that I do not know the Eclipse framework so I do not understand the impact of your two lines but I use several hashmaps in my database plugin without any side effect so far.

So I do not expect to discover side effects with your hashmap ... but I will do some more testing ...

Best regards
Hervé

[Phil Beauvoir]

Well, it's not that it's a Hashmap, it's that IDs are now being stored there. The danger is when changing an ID and not updating the Hashmap.

[ejess]

tested with following mode l: 7,5Mo ; 241 View(s) ; 3600 Element(s) ; 6473 Relationship(s)

from 17 minutes to 13 seconds

Bravo!

I'll continue testing and look at side effects.

[Phil Beauvoir]

Thanks for testing. 13 seconds is still quite a long time.... ;-)

[Hervé]

Hi Phil,

Just to give you more precise time on my side:
     load duration : 30 seconds
     model size : 16.5 MB
     # elements : 9.375
     # relationships : 11.509
     # views : 266
     # images : 25

Best regards
Hervé

[Phil Beauvoir]

Thanks Hervé,

that's quite a large model, so I don't think I can improve the load time on that one.

Phil

[Hervé]

Hi Phil,

Going from more than 25 minutes to 30 seconds is very impressive indeed. From my point of view, improving more is not really necessary 

Best regards
Hervé