- Welcome to Archi Forum.
Archi - ArchiMate Modelling
Recent posts
#1
General Archi Discussion / How I cope with model of 400 e...
Last post by maksim aniskov - Today at 06:49:04 AMHi all!
I want to share my story about how I deal with a model representing a real complex data processing setup that constantly mutates. I have been on a challenge of building the model and automation around in course of past two months as a part-time assignment. Now I'm having solid proofs that my setup flies flawlessly and is saving me and the team a ton of time and effort each day. I'm very much proud of it!
My situation is that there is a team of approximately ten data engineers and devops who are full-time busy on improving or modifying the setup. A whole bunch of Apache NiFi dataflows running on several AWS EC2 instances comprise the setup. They pull data from several dozens of external APIs, then fan out processed data to downstream consumers, as well as data storage services. The whole setup is undergoing migrating from one AWS region to another. Those APIs live their own life, sometimes changing their endpoint location, or getting decommissioned. Or a new API appears.
How do I make sure that my architecture diagrams always correctly represent all that pretty complex "reality"? And how do I sleep well spending just 10 minutes a day fixing drift, when it gets detected?
The keyword here is "drift detection". Let me describe my toolchain.
Data harvesting. I use two sources to fetch all necessary data about "the real world": AWS CLI output and NiFi flow definition JSON files.
Data preparation. Before feeding the data to model validation scripts, I prefer to transform it to comma- or tab-separated text files. Making AWS CLI to produce such output is not a problem at all. Retrieving necessary data from NiFi flow definitions is the trickiest part of the challenge. Those JSON files that the data harvesting script retrieves from the servers, are around two million lines in total, if pretty-printed. I chose jq to do all the heavy-lifting, looping through objects, looking up for NiFi processor group and processor descriptions, and matching parameter substitutions in expressions against values in context definitions. What jq passes output are two comma-separated files. One maps NiFi root process group names to ids of AWS EC2 instances the groups run on. The second file lists all data sources and data sinks the process groups have in configuration of their data processors. What I'm interested in first of all is to know what kind of database, data streaming service or API is the NiFi processor's peer, and what is its URL, address or identification. I want to know all the ins and outs of data processing setup under analysis.
Validating model correctness. For marching my ArchiMate model against "the real world" data, I employ archi-powertools-verifier. The tool leverages Docker Compose to orchestrate Archi, Neo4j and sqllite containers.
The basic idea is pretty straightforward. A model validation batch does the following.
So, once a day my automation runs data harvesting, data preparation and model validation scripts. If my SQL and CQL scripts detect any discrepancies between the architecture model and AWS infrastructure or NiFi flow definitions, I get Drift detected! lines in the output accompanied with necessary problem descriptions and resource names or ids. To bring the architecture model back to state when it correctly represents the NiFi flows, I edit the model manually in Archi. Before committing my changes to git, I validate the model one more time. When there are no more drift warnings in the output, I'm ready to push the updated model to the repository to make it available to other collaborators. Doing this process again and again routinely, I have already collected statistics that our NiFi setup gets 2 to 4 changes requiring mirroring in the model each week. When drift is detected, it normally takes 10 or 15 minutes to edit, re-validate, commit and push changes. And I'd say that those several minutes spent is nothing if we consider that that is all the cost of making it guaranteed that the system architecture model is fully correct 100% of time.
I want to share my story about how I deal with a model representing a real complex data processing setup that constantly mutates. I have been on a challenge of building the model and automation around in course of past two months as a part-time assignment. Now I'm having solid proofs that my setup flies flawlessly and is saving me and the team a ton of time and effort each day. I'm very much proud of it!
My situation is that there is a team of approximately ten data engineers and devops who are full-time busy on improving or modifying the setup. A whole bunch of Apache NiFi dataflows running on several AWS EC2 instances comprise the setup. They pull data from several dozens of external APIs, then fan out processed data to downstream consumers, as well as data storage services. The whole setup is undergoing migrating from one AWS region to another. Those APIs live their own life, sometimes changing their endpoint location, or getting decommissioned. Or a new API appears.
How do I make sure that my architecture diagrams always correctly represent all that pretty complex "reality"? And how do I sleep well spending just 10 minutes a day fixing drift, when it gets detected?
The keyword here is "drift detection". Let me describe my toolchain.
Data harvesting. I use two sources to fetch all necessary data about "the real world": AWS CLI output and NiFi flow definition JSON files.
Data preparation. Before feeding the data to model validation scripts, I prefer to transform it to comma- or tab-separated text files. Making AWS CLI to produce such output is not a problem at all. Retrieving necessary data from NiFi flow definitions is the trickiest part of the challenge. Those JSON files that the data harvesting script retrieves from the servers, are around two million lines in total, if pretty-printed. I chose jq to do all the heavy-lifting, looping through objects, looking up for NiFi processor group and processor descriptions, and matching parameter substitutions in expressions against values in context definitions. What jq passes output are two comma-separated files. One maps NiFi root process group names to ids of AWS EC2 instances the groups run on. The second file lists all data sources and data sinks the process groups have in configuration of their data processors. What I'm interested in first of all is to know what kind of database, data streaming service or API is the NiFi processor's peer, and what is its URL, address or identification. I want to know all the ins and outs of data processing setup under analysis.
Validating model correctness. For marching my ArchiMate model against "the real world" data, I employ archi-powertools-verifier. The tool leverages Docker Compose to orchestrate Archi, Neo4j and sqllite containers.
The basic idea is pretty straightforward. A model validation batch does the following.
- Convert content of ArchiMate model to set of comma-separated files;
- Run SQL database engine (sqlite) or graph database engine (Neo4j);
- Make it import model files and any additional data we have as result of data preparation we did before;
- Run the set of SQL or CQL (Neo4j) queries I wrote. The queries compare what is in the tables (or elements and relationships, if Neo4j) which contain the model against what is those which contain the "real world" data. Read this for mode details and examples.
So, once a day my automation runs data harvesting, data preparation and model validation scripts. If my SQL and CQL scripts detect any discrepancies between the architecture model and AWS infrastructure or NiFi flow definitions, I get Drift detected! lines in the output accompanied with necessary problem descriptions and resource names or ids. To bring the architecture model back to state when it correctly represents the NiFi flows, I edit the model manually in Archi. Before committing my changes to git, I validate the model one more time. When there are no more drift warnings in the output, I'm ready to push the updated model to the repository to make it available to other collaborators. Doing this process again and again routinely, I have already collected statistics that our NiFi setup gets 2 to 4 changes requiring mirroring in the model each week. When drift is detected, it normally takes 10 or 15 minutes to edit, re-validate, commit and push changes. And I'd say that those several minutes spent is nothing if we consider that that is all the cost of making it guaranteed that the system architecture model is fully correct 100% of time.
#2
jArchi / Re: Hide all relationship line...
Last post by Mate - May 17, 2024, 20:10:51 PMThat works perfectly. Thank you. 
#3
jArchi / Re: Hide all relationship line...
Last post by Phil Beauvoir - May 17, 2024, 08:43:15 AMQuote from: Mate on May 16, 2024, 22:38:54 PMUnder properties.appearance there is a checkbox option called "Show Label" that does what I want, so was wondering if that can be done programmatically.
Yes, I forgot about that option, sorry. Ignore what I said earlier. See https://github.com/archimatetool/archi-scripting-plugin/wiki/Visual-Connections#labelvisible
Code Select
connection.labelVisible = false;
#4
jArchi / Re: Hide all relationship line...
Last post by Mate - May 16, 2024, 22:38:54 PM@Phil Beauvoir: Thank you, Sir! This gets me successfully moving along on my learning journey. Appreciate the detailed response and your patience.
"When you say you want to remove the connection labels, a connection label will show if you have given the relationship a name. So you'd have to set the relation's name to the empty string:" --> you are correct, it's the name that displays as a label. I don't want to delete the name from the model, just it's visibility in the view. Under properties.appearance there is a checkbox option called "Show Label" that does what I want, so was wondering if that can be done programmatically.
I'll check out the option (ie bogus label) you suggest it seems like a good workaround.
"When you say you want to remove the connection labels, a connection label will show if you have given the relationship a name. So you'd have to set the relation's name to the empty string:" --> you are correct, it's the name that displays as a label. I don't want to delete the name from the model, just it's visibility in the view. Under properties.appearance there is a checkbox option called "Show Label" that does what I want, so was wondering if that can be done programmatically.
I'll check out the option (ie bogus label) you suggest it seems like a good workaround.
#5
General Archi Discussion / Re: List of organizations know...
Last post by AdGerrits - May 16, 2024, 12:53:04 PMThe VNG, an association that represents all Dutch municipalities, makes intensive use of Archi. In particular (but not only) for modeling the reference architecture for Dutch municipalities ('GEMMA'). The GEMMA is published via the website gemmaonline.nl. Views created with Archi are published there, among other things.
Archi repositories used within GEMMA are published at https://github.com/VNG-Realisatie/GEMMA-Archi-repository. They are used, among others, by municipalities and by software suppliers (who often, but not necessarily, also use Archi). Parts of the Archi models are also used within the software catalog (https://www.softwarecatalogue.nl) in which suppliers describe which software they supply and municipalities which software they use. The catalog makes visible, among other things, which software packages realize which application components and which standards they support.
Archi repositories used within GEMMA are published at https://github.com/VNG-Realisatie/GEMMA-Archi-repository. They are used, among others, by municipalities and by software suppliers (who often, but not necessarily, also use Archi). Parts of the Archi models are also used within the software catalog (https://www.softwarecatalogue.nl) in which suppliers describe which software they supply and municipalities which software they use. The catalog makes visible, among other things, which software packages realize which application components and which standards they support.
#6
Share your Models, Templates and Reports / Re: Archi Collaboration (GIT f...
Last post by AdGerrits - May 16, 2024, 12:31:07 PMI don't look at the forum that often, but I thought this was a nice topic to model with some elements that I don't use that often. In a completely different way than the first model. That's why I created a small new model, which can be found at https://github.com/adgerrits/coArchi-model.
#7
jArchi / Re: Hide all relationship line...
Last post by Phil Beauvoir - May 16, 2024, 09:17:11 AMHi,
You can delete ArchiMate connections from a View without deleting the model relationships with something like the following:
When you say you want to remove the connection labels, a connection label will show if you have given the relationship a name. So you'd have to set the relation's name to the empty string:
If you want to keep the relation's name but hide it you could assign a bogus label expression that resolves to an empty string:
Edit - ignore this part, see answer below!
Hope that helps, and thanks for your support on Patreon!
Phil
You can delete ArchiMate connections from a View without deleting the model relationships with something like the following:
Code Select
selectedView = ... // Get the View you want to work with
// Find each ArchiMate relation in the View and delete the visual connection
$(selectedView).find('relation').each(connection => {
connection.delete();
});
When you say you want to remove the connection labels, a connection label will show if you have given the relationship a name. So you'd have to set the relation's name to the empty string:
Code Select
selectedView = ... // Get the View you want to work with
// Find each ArchiMate relation in the View and reset name
$(selectedView).find('relation').each(connection => {
connection.name = "";
});
Edit - ignore this part, see answer below!
Code Select
selectedView = ... // Get the View you want to work with
// Find each ArchiMate relation in the View and assign label expression
$(selectedView).find('relation').each(connection => {
connection.labelExpression = "${property:empty}";
});
Hope that helps, and thanks for your support on Patreon!
Phil
#8
jArchi / Hide all relationship lines or...
Last post by Mate - May 15, 2024, 18:01:17 PMHi jArchi wizards...
I am using jArchi 1.6.1 in Archi 5.3, I have been a Patron for a few years, but only now starting to do more complex modeling, size-wise, with Archi
Confess I am a noob in Javascript and jArchi, but eager to learn. Have searched the forum and couldn't find a noob-level answer
I have a view with about 50 elements in it and their respective relationship connections. The relationship lines and labels make the visual very busy and they do not add any meaningful value to my Stakeholder Viewpoint and conversation. So I like to remove them from the view. In some views just the labels, while in others both labels and lines.
I can iterate through all the relationships in each selected view but don't know how to hide the labels (or labels and lines), via a jArchi script, without deleting the relationship from the model. Like I can do from the UI.
Any guidance is much appreciated. Thanks in advance.
I am using jArchi 1.6.1 in Archi 5.3, I have been a Patron for a few years, but only now starting to do more complex modeling, size-wise, with Archi
Confess I am a noob in Javascript and jArchi, but eager to learn. Have searched the forum and couldn't find a noob-level answer
I have a view with about 50 elements in it and their respective relationship connections. The relationship lines and labels make the visual very busy and they do not add any meaningful value to my Stakeholder Viewpoint and conversation. So I like to remove them from the view. In some views just the labels, while in others both labels and lines.
I can iterate through all the relationships in each selected view but don't know how to hide the labels (or labels and lines), via a jArchi script, without deleting the relationship from the model. Like I can do from the UI.
Any guidance is much appreciated. Thanks in advance.
#9
jArchi / Re: Passing jArchi out to Node...
Last post by Phil Beauvoir - May 14, 2024, 09:59:17 AMHi, jArchi does have some support for CommonJS modules, see:
https://github.com/archimatetool/archi-scripting-plugin/wiki/Using-Node.js-modules
But maybe you're trying to do something more complex?
https://github.com/archimatetool/archi-scripting-plugin/wiki/Using-Node.js-modules
But maybe you're trying to do something more complex?
#10
jArchi / Passing jArchi out to Node usi...
Last post by 5-tom - May 13, 2024, 17:57:32 PMI'm using a bit of a hack to get jArchi into an environment where I can use Node's builtins and ES modules, and was wondering if anyone has a better solution?
I'm piping the stdout of an .ajs script to a socket like so:
My Node code:
I'm piping the stdout of an .ajs script to a socket like so:
Code Select
<run .ajs script here> | socat - /tmp/mysock.sockMy Node code:
Code Select
import { Server } from "net";
const server = Server({});
server.on("connection", (client) => {
client.on("data", (data) => {
// use data here
}
});
server.listen("/tmp/mysock.sock");