-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Shared: Improve sensitive data heuristics #20024
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
99e62d6
Rust: Add sensitive data patterns.
geoffw0 8f95e26
Rust: Combine regexs where possible (likely better performance).
geoffw0 a6b4a18
Rust: Add negative patterns.
geoffw0 123458f
Sync identical files.
geoffw0 8f6f9f4
Add change notes.
geoffw0 4778ef6
Rust: Add a test case for password_confirmation.
geoffw0 9f59a35
Rust: Revert ipaddr and fingerprint terms (too many FPs).
geoffw0 e121579
Rust: Adjust the test labels slightly.
geoffw0 30c6082
Sync identical files.
geoffw0 da0742f
Rust: Update path resolution consistency .expected.
geoffw0 918700f
Merge branch 'main' into moresensitive2
geoffw0 4f6b698
Merge branch 'main' into moresensitive2
geoffw0 68f0dfe
Shared: Fix after merge.
geoffw0 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
4 changes: 4 additions & 0 deletions
4
javascript/ql/lib/change-notes/2025-07-11-sensitive-data-heuristics.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
--- | ||
category: minorAnalysis | ||
--- | ||
* The regular expressions in `SensitiveDataHeuristics.qll` have been extended to find more instances of sensitive data such as secrets used in authentication, finance and health information, and device data. The heuristics have also been refined to find fewer false positive matches. This will improve results for queries related to sensitive information. |
4 changes: 4 additions & 0 deletions
4
python/ql/lib/change-notes/2025-07-11-sensitive-data-heuristics.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
--- | ||
category: minorAnalysis | ||
--- | ||
* The regular expressions in `SensitiveDataHeuristics.qll` have been extended to find more instances of sensitive data such as secrets used in authentication, finance and health information, and device data. The heuristics have also been refined to find fewer false positive matches. This will improve results for queries related to sensitive information. |
4 changes: 4 additions & 0 deletions
4
ruby/ql/lib/change-notes/2025-07-11-sensitive-data-heuristics.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
--- | ||
category: minorAnalysis | ||
--- | ||
* The regular expressions in `SensitiveDataHeuristics.qll` have been extended to find more instances of sensitive data such as secrets used in authentication, finance and health information, and device data. The heuristics have also been refined to find fewer false positive matches. This will improve results for queries related to sensitive information. |
4 changes: 4 additions & 0 deletions
4
rust/ql/lib/change-notes/2025-07-11-sensitive-data-heuristics.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
--- | ||
category: minorAnalysis | ||
--- | ||
* The regular expressions in `SensitiveDataHeuristics.qll` have been extended to find more instances of sensitive data such as secrets used in authentication, finance and health information, and device data. The heuristics have also been refined to find fewer false positive matches. This will improve results for queries related to sensitive information. |
56 changes: 28 additions & 28 deletions
56
rust/ql/test/library-tests/sensitivedata/CONSISTENCY/PathResolutionConsistency.expected
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,29 +1,29 @@ | ||
multipleCallTargets | ||
| test.rs:55:7:55:26 | ... .as_str() | | ||
| test.rs:56:7:56:21 | ... .as_str() | | ||
| test.rs:72:7:72:26 | ... .as_str() | | ||
| test.rs:73:7:73:36 | ... .as_str() | | ||
| test.rs:74:7:74:34 | ... .as_str() | | ||
| test.rs:75:7:75:27 | ... .as_str() | | ||
| test.rs:258:7:258:36 | ... .as_str() | | ||
| test.rs:260:7:260:33 | ... .as_str() | | ||
| test.rs:261:7:261:36 | ... .as_str() | | ||
| test.rs:262:7:262:26 | ... .as_str() | | ||
| test.rs:266:7:266:28 | ... .as_str() | | ||
| test.rs:267:7:267:37 | ... .as_str() | | ||
| test.rs:268:7:268:36 | ... .as_str() | | ||
| test.rs:271:7:271:32 | ... .as_str() | | ||
| test.rs:281:7:281:34 | ... .as_str() | | ||
| test.rs:284:7:284:36 | ... .as_str() | | ||
| test.rs:288:7:288:39 | ... .as_str() | | ||
| test.rs:295:7:295:53 | ... .as_str() | | ||
| test.rs:296:7:296:45 | ... .as_str() | | ||
| test.rs:298:7:298:39 | ... .as_str() | | ||
| test.rs:299:7:299:34 | ... .as_str() | | ||
| test.rs:300:7:300:42 | ... .as_str() | | ||
| test.rs:302:7:302:48 | ... .as_str() | | ||
| test.rs:303:7:303:35 | ... .as_str() | | ||
| test.rs:304:7:304:35 | ... .as_str() | | ||
| test.rs:313:8:313:19 | num.as_str() | | ||
| test.rs:324:8:324:19 | num.as_str() | | ||
| test.rs:343:7:343:39 | ... .as_str() | | ||
| test.rs:56:7:56:26 | ... .as_str() | | ||
| test.rs:57:7:57:21 | ... .as_str() | | ||
| test.rs:73:7:73:26 | ... .as_str() | | ||
| test.rs:74:7:74:36 | ... .as_str() | | ||
| test.rs:75:7:75:34 | ... .as_str() | | ||
| test.rs:76:7:76:27 | ... .as_str() | | ||
| test.rs:262:7:262:36 | ... .as_str() | | ||
| test.rs:264:7:264:33 | ... .as_str() | | ||
| test.rs:265:7:265:36 | ... .as_str() | | ||
| test.rs:266:7:266:26 | ... .as_str() | | ||
| test.rs:270:7:270:28 | ... .as_str() | | ||
| test.rs:271:7:271:37 | ... .as_str() | | ||
| test.rs:272:7:272:36 | ... .as_str() | | ||
| test.rs:275:7:275:32 | ... .as_str() | | ||
| test.rs:285:7:285:34 | ... .as_str() | | ||
| test.rs:288:7:288:36 | ... .as_str() | | ||
| test.rs:292:7:292:39 | ... .as_str() | | ||
| test.rs:299:7:299:53 | ... .as_str() | | ||
| test.rs:300:7:300:45 | ... .as_str() | | ||
| test.rs:302:7:302:39 | ... .as_str() | | ||
| test.rs:303:7:303:34 | ... .as_str() | | ||
| test.rs:304:7:304:42 | ... .as_str() | | ||
| test.rs:306:7:306:48 | ... .as_str() | | ||
| test.rs:307:7:307:35 | ... .as_str() | | ||
| test.rs:308:7:308:35 | ... .as_str() | | ||
| test.rs:317:8:317:19 | num.as_str() | | ||
| test.rs:328:8:328:19 | num.as_str() | | ||
| test.rs:347:7:347:39 | ... .as_str() | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 4 additions & 0 deletions
4
swift/ql/lib/change-notes/2025-07-11-sensitive-data-heuristics.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
--- | ||
category: minorAnalysis | ||
--- | ||
* The regular expressions in `SensitiveDataHeuristics.qll` have been extended to find more instances of sensitive data such as secrets used in authentication, finance and health information, and device data. The heuristics have also been refined to find fewer false positive matches. This will improve results for queries related to sensitive information. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the reason for adding
(?!_iter)
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In Rust there are library methods called
from_trusted_iterator
, where "trusted" means it's trusted not to overrun the container (or something along those lines), not anything to do with secret or confidential data. There's actually quite a lot of flow from those calls to sinks, i.e. it's causing a good number of false positive results, not to mention wobble in nightly DCA runs. So the exclusion is a bit specific, but I'm very keen to add it.There is a test case
sink(MyArray::from_trusted_iterator(iter));
that resembles (simplified) what we see in the wild.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah makes sense, thanks for clarifying.