AWS Cloud Scanners
— Find my “active” cloud resources
With each day the cloud is getting bigger and bigger. The cloud resources used either for PoC, development, testing or in production are increasing and with that the universal problem of resource “sprawling” is faced once again. We saw it earlier with VMs and now its making headway in cloud as well.
Yesterday I happened to stumble upon this reddit post https://www.reddit.com/r/aws/comments/jlhvch/best_way_to_find_all_active_resources/ where u/Delta4o asked about the tools that can be used to find all “active” or “live” cloud resources. The responses are fun and insightful to read. I definitely recommend you to go through them once.
While everyone assumes the Infrastructure as Code, should take care of the problem i.e. the problem should not occur in itself, reality is far different. We have good tools like Terraform, CloudFormation etc, but still these tools sometimes lag behind the AWS feature release cycle. I think CloudFormation particularly. You can always have something getting created, updated or destroyed in cloud out of realm of IaC. The inherent nature of cloud itself is “programmable” — with plenty of tools, to make it happen. Be it AWS Console itself, through AWS SDKs or even some third party tools that sometimes require “read/write” permissions or have such accesses.
So what else can help our friend here? Of course, Tagging. The basic rule in cloud is to tag. For me, To Tag is to Cloud. But again, that needs enforcement and lot of decipline. This can be enforced through SCPs, but require our DevOps/CloudOps folks to be well versed with them, and of cousrse time to spend on it.
So it seems to me, no matter what, we will always have this problem, that feeling that something, somewhere is lurking in my cloud which may cause problem for me. Its like a ticking bomb!
So I though why not just try to compile a list of Open Source and thrid party tools and techniques that we can use to help us know our cloud better.
- Use AWS billing console to identify the AWS services that you are using. Chances is that you would see services you are “not” supposed to be in use. Of course, I think it would require lot of efforts to identify the specific instances of cloud resources which are not in active use this way. E.g. A Security group/VPC lying idle in a corner waiting to be exploited.
- AWS Tag Editor — Tag everything that is needed and use tag editor to search for all the resources that does not have this tag. I would say, first thing you wouldn’t end up looking for tool here if your were already disciplined. In my opinion you will always have something untagged or uncontrolled. That nagging feeling in your head that something is amiss!
- AWS Trusted Advisor — (I have not used it personally)
- AWS Config — I’m bit skeptical about this service, specially from pricing perspective. Also it seems the coverage is bit limited, approx 90 Cloud resources across 32 AWS service.
- https://github.com/jckuester/awsls — this is certainly a piece of gem. An open source tool, written in Golang that you can use in your account. It is an user friendly tool with coverage of over 200 cloud resources (217 to be precise) across 77 AWS services. Quite decent coverage I would say. Moreover your could pipe the output for further filtering. Multiple account support too, awesome!!
- https://github.com/nccgroup/aws-inventory — I see some issues with this tool though. First it is in Python 2.7 and second but more important — it seems to be abandoned. Last checkin shows Aug, 2018. Gosh, we are almost crossing 2020. I would skip it :)
- https://github.com/nccgroup/ScoutSuite — now this comes from creators of aws-inventory, and is in active development. This Open source tool is meant for Continuous Security Posture Management (CSPM). Wow quite a term! Now this tool is written in Python 3.0, which is a relief. A look at the tool indicate that it supports about 25 AWS Services. That could a minus point though, in my opinion. This tool does support multi-vendor clouds, Azure, Google, Alibaba, Oracle which may be a plus for some.
- https://aws.amazon.com/solutions/implementations/aws-perspective/ — AWS released this solution just recently. I have not used it, but seems decent. It uses AWS Config in the backend. It is more like a gobbling pieces together. You can as well export Cloud infrastructure diagrams from this. Morever you can combine and associate cost of each cloud resource. Please keep an eye on cost though.
- https://github.com/rebuy-de/aws-nuke — One word, Stay away!
- https://www.cloudyali.io — It has a broad coverage of the cloud resources (~250 cloud resources). You may view and search for cloud resources spanned across multiple AWS accounts, multiple regions in a single place. One distinguishing feature is, it also maintains the lifetime of the resources and allows to view even the resources which are currently non-existent. Also, it helps to view the cloud resource property history in a single place. This comes handy when you are doing troubleshooting.
- https://vantage.sh — a new paid tool which would help to view cloud resources per your requirements (based on tags, or some other conditions )— I don’t know what kind of conditions does it support or how many resources/AWS Services? No details on their page. I shall update once I try them.
- https://github.com/lyft/cartography — This open source tool can help you to visualize your cloud as graph and help you understand the hidden relationships. Novel idea! What lacks is broad coverage. At this moment it supports — approx 13 AWS services. But I must say, looking at cloud through this tool simplifies a lot. And fun too!
- https://github.com/duo-labs/cloudmapper — perhaps most talked about open source tool in this segment. Though started as visualization tool for AWS, it now has functionality for auditing as well. It also has a ‘find_unused’ feature, that we are looking for.
- https://www.fugue.co — an absolutely brillient service when it comes for visualization. In one screen you get to see everything needed. I have tried the trial and I’m overwhelmed. But again what falls short is AWS Services and resource type coverage. Some of you may have already used their open source tool https://github.com/fugue/regula for terraform checks.
- https://github.com/cloud-custodian/cloud-custodian — definitely the pioneer tool in this segment. Broadest AWS service coverage (~80 AWS Services).
Now that we have quite a comprehensive list of tools, here is what I think is missing.
- Broad coverage of AWS services and resources types. — Surprisingly lot of tools have a very limited coverage, mostly around core AWS services and resource types.
- Resource type relationships — AWS Config, Cartography, AWS-perspective and CloudMapper are few of the tools which care for resource relationships. Without identifying relationships between the resources it is difficult to identify unused or orphaned resources, or resources which may lead to a security threat. But again the coverage is very short compared to the range of AWS services. I think it is understandable, as the it almost impossible to match AWS feature release speed.
- Visualization — As cloud becomes more complex, it becomes difficult to manage from the console or AWS CLI alone. Very difficult to comprehend the complexity this way. Now I see visualization tools emerging but very few have really nailed it. Again for me fugue.co has absolutely done it brilliently.
- Querability — what is use of the inventory tool if you can’t query it? IMO, Slicing dicing in the tool itself it absolute must. I hate to generate csv files from these tools, export to excelsheet or some other tools and use for analysis — what a waste of time.
Other tools (not necessasirily for inventory) — That I have not tried yet.
- AWSets
- https://jupiterone.com/
- https://www.cloudaware.com/
- https://abridge.io
- https://bridgecrew.io/
- https://kumolus.com/
- https://turbot.com/
- https://www.nops.io/
- https://github.com/Stuxend/dagobah
- https://github.com/toniblyx/prowler
- https://github.com/tensult/cloud-reports
- https://github.com/disruptops/ElectricEye
- https://github.com/RiotGames/cloud-inquisitor
- https://github.com/salesforce/policy_sentry
- https://github.com/cyberark/SkyArk/tree/master/AWStealth
- https://cloudsploit.com
- https://komiser.io
- https://www.cloudpassage.com/cloud-computing-security/
- https://www.qualys.com/public-cloud/#aws
Please feel free to add Open source tools or third party services which can be used for AWS Cloud Inventory, Visualization or Query/Analysis purpose.
