delicate yarrow
This is in reference to an issue I opened (https://github.com/projectdiscovery/nuclei-templates/issues/8699)
TLDR, template tagging is super inconsistent which makes technology detection -> targeted scanning increasingly difficult. I was hoping to consult the PD team on how to standardize the process, update the current tags and make improvements for the future. Issue post has the majority of my thoughts
trim cipher
Thanks for the issue and posting here as well!
delicate yarrow
Thanks! Main goal is to have a discussion about tagging templates since there are a number of places that aren't ideal. I'll copy/paste my original report over and see if anyone has thoughts
TLDR: I've noticed that a considerable amount of templates are tagged inconsistently or in an non-specific way. A few examples can be seen below:
We can take a few Oracle related templates for example:
./http/technologies/weblogic-detect.yaml Searches Shodan for the string Oracle Weblogic, but the tags does not contain oracle./http/default-logins/weblogic/weblogic-weak-login.yaml checks for default Oracle Weblogic creds, but does not contain the oracle tag./http/exposed-panels/weblogic-login.yaml also locates Oracle Weblogic login panels, but DOES have the oracle tag./http/exposures/logs/oracle-ebs-sqllog-disclosure.yaml Has some Oracle EBS log leak files, so the tags are oracle and ebs./http/vulnerabilities/oracle/oracle-ebs-xss.yaml is targeting EBS, which has oracle and ebs tags./http/exposures/configs/oracle-ebs-credentials.yaml Is also targeting Oracle EBS config file leak, but the tags are oracle but not ebs./http/vulnerabilities/oracle/oracle-ebs-bispgraph-file-access.yaml also targets EBS, but does not have the ebs tagd-link tag, and a dlink tag, both are the same thing but running specifically would miss one or the other without prior knowledge. This also means that the d-link tags will never be run by -as because it would break it into d and link without a manual wappalyzer mappingSimply put, there are considerable inconsistencies between the tagging that is integral to testing. In my opinion there are a few things that may need to be done to remedy this, but frankly it may be best for a wider discussion over it.
I think generally there are two smaller things that can be done for the current templates, but I have some thoughts on future improvements below
Even though the Wappalyzer mappings file is specifically only for the automatic scan, it may be beneficial to expand that use-case to the -tags input as well, so custom user defined technology mappings could be created (wordpress == wordpress, wp, etc (also lol what is the difference between wp and wordpress tags??)) rather than only being able to use this with -as. Since the tech-detect, fingerprinthub and favicon detection templates oftentimes returns tags that don't correspond to any templates (outlook-web-application, f5-big-ip, etc) that correlation is needed. Similarly spring-boot is returned, but all SpringBoot related templates are tagged springboot, and there are many other examples like that. There isn't the standardization, nor the way to map those tags manually
And lastly (a future goal), it may be better to format tags better, especially for Companies (like Oracle) that have many products. Instead of tagging things oracle,access,manager a more specific tag would be oracle-access-manager. As an example, Wappalyzer output breaks up oracle-application-server into oracle,application,server which will run far more undesired templates against the unrelated and generic application and server tags. There are ~5 (including what appears to be duped in http/technologies/oracle-httpserver-12c.yaml and http/technologies/oracle/oracle-http-server-12c.yaml) templates that are specifically related to the Oracle Application/HTTP Server, but running the oracle, application, server tags it will run 70 templates, most of which are completely unrelated
For one off products, a simple tag like wordpress makes sense, but for companies (Oracle, VMWare, etc) that have multiple products, it would make sense to standardize tagging to be {company}-{product} as well as just {company} so tagging is both generic at an overall company level and highly specific at a specific service
Like I said earlier, I don't know what's best and maybe these are just the ponderings of a senseless man, but I've had issues where undesired templates are run, or applicable templates aren't run because they're tagged incorrectly (or the technology detection templates return different tags)
Probably best for a wider discussion, can be had in the thread. I'm happy to help of course and I've already done a bit of work re-tagging things, but as for the direction of the overall effort I feel like it would need to be decided by the maintainers and I don't want to do a bunch of work re-tagging things that would need to be re-tagged again if the standard is different from what I envisioned
And another question for the PD Team, as I start re-tagging and de-duping templates, would you prefer one large PR with everything or individual ones for greater control over the content and to not get overwhelmed
magic remnant
Hey @knave,
Thanks for the heads-up on the tagging issues! You're right, we've got some inconsistencies. The good news is, we're already on it. We're rolling out an automatic tagging enhancement soon. Once that's in place, feel free to pitch in with a PR if you spot anything amiss.
The plan is to update tags based on vendor, product, framework, directory, protocol, vulnerability type, and info block content. This should make things way more streamlined. We're also focusing on the correlation between tech and templates. Once we enhance the tags, we’ll work on adding this correlation into the repo.
About the {company}-{product} tags, we like the idea but worry it might be too long. We're leaning towards a {company},{product} format for simplicity.
We've kicked off a PR for some basic tag improvements, especially with CVE templates. Other templates might need more manual review. Your input and any further ideas are super welcome.
Really appreciate your deep dive into this. We're on track to sorting out the tagging chaos. Thank you
delicate yarrow
PR looks awesome @magic remnant, I'll follow along and open PRs where needed!
What are your thoughts on making the wappalyzer-mapping file apply to the -tags flag as well? IMO that would address at least some of the issues, and then even if {company}-{product} isn't the way template tagging goes, you can still make explicit mappings for the same impact
And really I suppose the only difficulty with not doing something like {company}-{product}, is that generic names may overlap, especially like oracle-http-server, where http and server are so generic that they'll match many other unrelated technologies. I feel like the logic makes sense for something like Oracle Weblogic, where weblogic is a defined product and the name is unique, but not for others. Maybe just for those edge cases, a unique name would need to be derived
delicate yarrow
Added a small PR to address some tagging inconsistencies in the SSL and Network categories. More to come, but HTTP will be a slow and rough process haha.
https://github.com/projectdiscovery/nuclei-templates/pull/8784
delicate yarrow
Another stray note of an example where tags aren't super specific, Microsoft Exchange/OWA vs Alienvault Open Threat Exchange. The tags for each respectively are microsoft,exchange and alienvault,exchange which means that the exchange tag corresponds to both. IMO this isn't a great practice, which is why I advocated for the {company}-{product} tagging which would eliminate broad tags that apply to multiple things
perhaps if that format isn't desired, we modify one to msexchange or avexchange?