Tool or plug-in for importing AWS/Azure/GCP resources

Started by maksim aniskov, February 04, 2022, 13:52:17 PM

Previous topic - Next topic

maksim aniskov

Hi everyone,

I'm working on an enterprise model and I want to make it incorporate some technology elements reflecting real resources we have in the cloud. As I work with hundreds of cloud resources, I'm looking for how to automate resource importing and reconciling with real current status of the things in the cloud. I need to make sure that my model is always up to real situation in the cloud.

I've done some basic research on what means Archi gives us. Though, I've never done any Archi development before, so I'm in a sort of doubt.

I see three options how I could automate the process.
  • External script which takes .archimate file and connects to AWS/Azure/GCP; it analyzes and reconciles the model vs resources in the cloud; then it generates archimate XML containing all new or updated cloud resources. Then it will require get imported into the existing model by hand.
  • jArchi script
  • A plugin

Please help me to decide on how to approach the challenge. Do you think there is a better solution?

Thank you and take care!

Jean-Baptiste Sarrodie

Hi,

Quote from: maksim aniskov on February 04, 2022, 13:52:17 PMPlease help me to decide on how to approach the challenge. Do you think there is a better solution?

Well, Avoid 1 in all cases (raises lots of issues and make you rely on an internal format that change from time to time and will most certainly impact your work if any), try 2 because that's the easiest thing to work on and there's already a lot we can do with jArchi, but if it really doesn't work, then try 3.

Regards,

JB

If you value and use Archi, please consider making a donation!
Ask your ArchiMate related questions to the ArchiMate Community's Discussion Board.

Alberto

I haven't dealt with cloud yet as all my assets are on premise, but the problem is fairly similar... lots of assets, need to inventory them all, find the relationships, create views, keep the element in those views up-to-date. I usually leverage CSV imports only because I don't have the bandwidth to do jArchi or a plugin.

What I've done so far:
1) Create a reference model, keep track of how you define your real world assets/resources to archimate elements/specializations.
2) Create properties to track data source, first created, last updated, etc.  Leverage jArchi scripts to highlight in your views when elements are outdated or real world resource no longer exists. 
3) If you use multiple sources for your Archi model, define a strategy for element IDs and IDs for relationships that doesn't depend on ID is data sources. (e.g. ID for Node = device_OS_name+device_environment+"_node" guarantees me a unique ID that I can easily reuse when creating a relationship between nodes element and say communication_network or application_service element that are sourced from other places)
4) Use jArchi script to remove all elements that don't exist on a view. This will keep the model lean and reduce the time of commits (if you are using a repo).
5) Use CLI to get latest commit from Git, publish HTML report to a web server.

It's a lot of work and I'd love to streamline it, but it's working for me so far.

maksim aniskov

QuoteWell, Avoid 1 in all cases (raises lots of issues and make you rely on an internal format that change from time to time and will most certainly impact your work if any), try 2 because that's the easiest thing to work on and there's already a lot we can do with jArchi, but if it really doesn't work, then try 3.
Thanks a lot. Very helpful!
I hoped having it as a plugin would allow the best user experience.

maksim aniskov

Quote from: Alberto on February 04, 2022, 19:38:01 PMI haven't dealt with cloud yet as all my assets are on premise, but the problem is fairly similar... lots of assets, need to inventory them all, find the relationships, create views, keep the element in those views up-to-date. I usually leverage CSV imports only because I don't have the bandwidth to do jArchi or a plugin.

What I've done so far:
1) Create a reference model, keep track of how you define your real world assets/resources to archimate elements/specializations.
2) Create properties to track data source, first created, last updated, etc.  Leverage jArchi scripts to highlight in your views when elements are outdated or real world resource no longer exists. 

Hi Alberto,
Sounds like exactly what I'm looking for. Could we team up to make it support cloud provider. I'm going to begin with AWS. I have a sort of proof of concept script demonstrating that I'm able to cope :) with jArchi, reading/modifying the model, and accessing AWS from the script.

Quote3) If you use multiple sources for your Archi model, define a strategy for element IDs and IDs for relationships that doesn't depend on ID is data sources. (e.g. ID for Node = device_OS_name+device_environment+"_node" guarantees me a unique ID that I can easily reuse when creating a relationship between nodes element and say communication_network or application_service element that are sourced from other places)
AWS resources have unique ids.

Quote4) Use jArchi script to remove all elements that don't exist on a view. This will keep the model lean and reduce the time of commits (if you are using a repo).
My initial though was to implement Detect and import new resources feature which would bring the elements to some temporal view e.g. New resources for your review. Of course, the feature should allow filtering by resource type and tag, at minimum, to not not suck in thousands of unwanted garbage.

maksim aniskov

Pulled together what I have been working on in the background and published the script on GitHub.
https://github.com/MaksimAniskov/archi-aws