diff --git a/.gitignore b/.gitignore index be0721be..b09e8255 100644 --- a/.gitignore +++ b/.gitignore @@ -4,4 +4,5 @@ .DS_Store *.pem -/.idea \ No newline at end of file +/.idea +/.vscode \ No newline at end of file diff --git a/Instructions_on_Adding_New_Renders.md b/Instructions_on_Adding_New_Renders.md new file mode 100644 index 00000000..8180addf --- /dev/null +++ b/Instructions_on_Adding_New_Renders.md @@ -0,0 +1,103 @@ +# Instructions on Adding New Renders +## General +The translation engine of Uncoder IO is conceptually composed of renders, parsers, and mapping files for each supported language. +- A render is a component that generates the target query/rule from a source language (Sigma, Roota). +- A parser is a component that parses the source rule into objects that consequently are used to generate the destination query/rule by the render. +- A mapping file is a dictionary that maps the source fields to target fields as well as source log sources to target log sources. + +Uncoder IO supports the following parsers: +- Roota +- Sigma +- IOCs + +We encourage you to contribute renders that enable translating from Roota, Sigma, and IOCs into your SIEM, EDR/XDR, or Data Lake. + +You can find the list of supported target platforms in the [platforms](https://github.com/UncoderIO/Uncoder_IO/tree/main/translator/app/translator/platforms) directory. + +## How to Add a New Render + +All code related to translation has to be in the directory with the corresponding platform name in `uncoder-core/app/translator/platforms`. + +- `uncoder-core/app/translator/platforms//renders` – a directory that contains platform renders for different content types (such as rules and queries translated from a source language or queries generated based on parsed IOCs). +- `const.py` – a Python file that contains metainformation about the platform. +- `escape_manager.py` – a Python file that contains classes describing the rules of escaping special characters. +- `mapping.py` – a Python file that contains classes that describe working with mappings. + +To add a new render: + +1. Create a directory with the platform name in `app/translator/platforms/`. +2. Describe the metainformation about the platform in the `const.py` file. +3. Create a class that processes mappings in the `mapping.py` file. +4. Create a class that processes special characters in the `escaping_manager.py` file. +5. Create the `renders` directory in `uncoder-core/app/translator/platforms//`. +6. Create a file with the name that matches the name of the platform. +7. The render is composed of two classes: + a. `BaseQueryRender` – the class that describes the general mechanism of rendering a query from the tokens parsed from the input query. + b. `BaseQueryFieldValue` – the class that describes the mechanism of creating the `Field-Value` component of the query. + + +## Render Classes +These classes should be described in the `uncoder-core/app/translator/platforms//renders/.py` file. + +### BaseQueryRender Class + +The class has the following attributes: +- `mappings` – a class responsinble for working with mapping of fields and tables/indexes +- `details` – the metainformation about the platform (described in the `const.py` file) +- `is_strict_mapping` – a boolean flag that defines if the render's mapping is strict. When set to `True`, the render returns an error if a field has no mapping +- `platform_functions` – a class responsible for parsing and rendering functions +- `or_token/and_token/not_token` – corresponding platform operators that should be used when generating the target query +- `field_value_map` – a class that creates `Field-Value` +- `query_pattern` – the template of the source query +- `comment_symbol` – escaping character (metainformation provided for a better context is passed to the output together with the query so it should be commented). If this character allows commenting multiple lines, set the flag `is_multi_line_comment == True` + +The class has the following methods: +- The entry point into the process of query rendering is the `generate` method. Query generation process is started for all mappings that match the input query: + - `generate_prefix` method – a table or an index is generated (depending on the mapping) + - `generate_query` method – the query's body is generated. The `generate_query` method goes through each token parsed from the source query, and using the `BaseQueryFieldValue` class transforms the `FieldValue` token into a string value that conforms to the rules and standards of the target platform + - `finalize_query` method – the output query is generated based on `query_pattern` with the metainformation added +- The process ends with the `finalize` method validating the translation output and joining the queries that are identical (that is they match multiple log sources) + + +### BaseQueryFieldValue Class + +The class has the following attributes: +- `details` – the metainformation about the platform (described in the `const.py` file) +- `escape_manager`– a class responsible for processing and escaping special characters + +The class has the following methods: +- `__init__` creates a dictionary (map) named `field_value` where a processing method is connected that depends on the operator that was between the field and its value + +## Mapping Classes +These classes should be described in the `uncoder-core/app/translator/platforms//mapping.py` file. + +To describe mappings, you need two classes: +- A class that inherits the `BasePlatformMappings` class – responsible for choosing mapping +- A class that inherits the `LogSourceSignature` class – describes the mapping structure. +It's also important to invoke and initialize a class created by inheritance from `BasePlatformMappings` because later it should be connected to the render. + +### LogSourceSignature +A class that describes the mapping structure. + +The `__init__` method describes tabels/indexes that can be applied for a log source. Also there's an important but optional field `_default_source`, a dictionary that contains the table/index values that can be used for a certain log source at the time of rendering. + +The `is_suitable` method is required. It's used to determine the mapping. + +### BasePlatformMappings +This class has one required attribute – the name of the directory from which mappings should be taken (all mappings are in `uncoder-core/app/translator/mappings/`). Only the directory name should be indicated. + +This class contains two required methods: +- `prepare_log_source_signature` – a method that transforms mappings obtained from the YAML file into objects +- `get_suitable_source_mappings` – a method that contains the conditions for checking for a suitable mapping depending on fields and tables/indexes. + +## Escape Manager Class +This class inherits the basic class `EscapeManager`. It contains a required attribute `escape_map`. Depending on the `Value` type (the values searched for in the field) you need to define special characters to be escaped. `Value` types are defined in `uncoder-core/app/translator/core/custom_types/values.py`. + +## const.py +The file where the metainformation about the platform and the rule templates (if any) are stored. + +## Metainformation +`platform_id` – unique platform identifier +`group_name` – platform name to be displayed in the platform selection dropdown in the UI +`platform_name` – the name of the content type to be displayed on the tab (as well as in the sub-menu of the platform) +`group_id` – the unique identifier of all content types for a platform diff --git a/README.md b/README.md index f13ce8a3..dcc2be8a 100644 --- a/README.md +++ b/README.md @@ -3,10 +3,12 @@

# What is Uncoder IO (Public Beta) +:earth_americas: [English](README.md) [Українська](README_Ukrainian.md) + Uncoder IO is an open-source version of it's SaaS counterpart https://uncoder.io and its AI co-pilot version Uncoder AI. Since 2018, Uncoder IO has been a fast, private, and easy-to-use online translator for Sigma Rules, maintaining 100% privacy of its users. An open-source Uncoder IO expands use cases into the following: - Translation from Sigma Rules, a generic rule format for SIEM systems, to specific SIEM, EDR, and Data Lake languages - IOC packaging from any non-binary format such as PDF, text, STIX, or OpenIOC to specific SIEM, EDR, and Data Lake languages -- Translation from RootA Rules, the newly released language for collective cyber defense, to specific SIEM, EDR, and Data Lake languages. +- Translation from [Roota](https://github.com/UncoderIO/RootA/blob/main/README.md) Rules, the newly released language for collective cyber defense, to specific SIEM, EDR, and Data Lake languages. Uncoder is developed by a team of Detection Engineers, Threat Hunters, and CTI Analysts from Ukraine, Europe, USA, Argentina, and Australia to perform their daily job and nightly cyber defense hobbies faster & better, making their outcomes easier to share for the collective good. @@ -27,27 +29,27 @@ Uncoder is developed by a team of Detection Engineers, Threat Hunters, and CTI A # :heart_eyes_cat: Why Uncoder IO -## :pretzel: RootA & Sigma Translation Engine +## :pretzel: Roota & Sigma Translation Engine -Uncoder IO supports automated translation of RootA and Sigma rules into multiple SIEM, EDR, XDR, and Data Lake formats. +Uncoder IO supports automated translation of [Roota](https://github.com/UncoderIO/RootA/blob/main/README.md) and Sigma rules into multiple SIEM, EDR, XDR, and Data Lake formats. - **Sigma** is a generic and open signature format that allows you to describe relevant log events in a straightforward manner, which received industry adoption across 155 countries by over 8000 organizations according to SOC Prime's download and translation statistics. -- **RootA** is an open-source language that supports query definition directly in specific SIEM languages, vendor-agnostic correlation syntax, MITRE ATT&CK 14.0 for code autocompletion, and log source taxonomy autocomplete function based on Amazon's OCSF or Sigma. RootA+Uncoder serve as the first bridge towards full cyber security languages compatibility, where one day, knowing one specific language (say SPL or KQL) or generic language (say RootA or Sigma) would mean that you have master expertise in them all. This way, your complex detection logic can be rendered in other languages in an automated fashion. In case a native rule or query contains functions unsupported by RootA or target technology, those functions won’t be translated, with a corresponding note appended to the code translation. This is done so that experts can either manually complete translations if they know both source and destination languages, or use Uncoder AI to manually take care of such scenarios. If sharing with Sigma was easy, sharing with RootA is natural and future-proof. +- **[Roota](https://github.com/UncoderIO/RootA/blob/main/README.md)** is an open-source language that supports query definition directly in specific SIEM languages, vendor-agnostic correlation syntax, MITRE ATT&CK 14.0 for code autocompletion, and log source taxonomy autocomplete function based on Amazon's OCSF or Sigma. Roota+Uncoder serve as the first bridge towards full cyber security languages compatibility, where one day, knowing one specific language (say SPL or KQL) or generic language (say Roota or Sigma) would mean that you have master expertise in them all. This way, your complex detection logic can be rendered in other languages in an automated fashion. In case a native rule or query contains functions unsupported by Roota or target technology, those functions won’t be translated, with a corresponding note appended to the code translation. This is done so that experts can either manually complete translations if they know both source and destination languages, or use Uncoder AI to manually take care of such scenarios. If sharing with Sigma was easy, sharing with Roota is natural and future-proof. -## :pizza: RootA & Sigma Rule Editor +## :pizza: Roota & Sigma Rule Editor -Uncoder IO supports a built-in Sigma and RootA rules autocompletion wizard suggesting code enhancements with latest MITRE ATT&CK and log source dictionaries to streamline the rule creation process. AI or not, Uncoder is here to make it easier to code. +Uncoder IO supports a built-in Sigma and [Roota](https://github.com/UncoderIO/RootA/blob/main/README.md) rules autocompletion wizard suggesting code enhancements with latest MITRE ATT&CK and log source dictionaries to streamline the rule creation process. AI or not, Uncoder is here to make it easier to code. ## :popcorn: IOC Query Generator -Uncoder IO acts as an open-source IOC packager helping CTI and SOC analysts as well as Threat Hunters to quickly parse any number of IOCs directly from any digital non-binary format (a simple copy-paste of a web page, CSV, OpenIOC, PDF, STIX etc.) and convert them into performance-optimized IOC queries ready to run in a chosen security analytics platform. As Indicators of Compromise sharing is regulated by TLP, it is not advised to share them in Sigma or RootA rules, as the latter are not part of threat intelligence and thus are shared easily without borders. Yet, we need IOC matching just as we need Threat Behavior detections, so Uncoder IO is made to help solve both tasks in an easy-to-use and intuitive manner. +Uncoder IO acts as an open-source IOC packager helping CTI and SOC analysts as well as Threat Hunters to quickly parse any number of IOCs directly from any digital non-binary format (a simple copy-paste of a web page, CSV, OpenIOC, PDF, STIX etc.) and convert them into performance-optimized IOC queries ready to run in a chosen security analytics platform. As Indicators of Compromise sharing is regulated by TLP, it is not advised to share them in Sigma or Roota rules, as the latter are not part of threat intelligence and thus are shared easily without borders. Yet, we need IOC matching just as we need Threat Behavior detections, so Uncoder IO is made to help solve both tasks in an easy-to-use and intuitive manner. ## :smile_cat: Full Privacy Uncoder IO can be run on-prem without a need for an internet connection, thus supporting air-gapped network operation. We do however suggest checking for updates and deploying them regularly. Meanwhile, a SaaS version still ensures 100% privacy with no cookie tracking, no data or code logging, or sharing with third parties. Even with options for Uncoder AI functions, you are always in control of your code and data. # :dna: Supported Language Formats -RootA and Sigma Rules can be translated into the following formats: +[Roota](https://github.com/UncoderIO/RootA/blob/main/README.md) and Sigma Rules can be translated into the following formats: - AWS OpenSearch Query - `opensearch-lucene-query` - AWS Athena Query (Security Lake) - `athena-sql-query` - Falcon LogScale Query - `logscale-lql-query` @@ -61,22 +63,38 @@ RootA and Sigma Rules can be translated into the following formats: - CrowdStrike Query - `crowdstrike-spl-query` - Elasticsearch Query - `elastic-lucene-query` - Elasticsearch Rule - `elastic-lucene-rule` +- ElastAlert Rule - `elastalert-lucene-rule` - Sigma Rule - `sigma-yml-rule` - Chronicle Security Query - `chronicle-yaral-query` - Chronicle Security Rule - `chronicle-yaral-rule` +- Graylog Query - `graylog-lucene-query` +- FortiSIEM Rule - `fortisiem-rule` +- LogRhythm Axon Rule - `axon-ads-rule` +- LogRhythm Axon Query - `axon-ads-query` IOC-based queries can be generated in the following formats: - Microsoft Sentinel Query - `sentinel-kql-query` - Microsoft Defender for Endpoint Query - `mde-kql-query` - Splunk Query - `splunk-spl-query` -- CrowdStrike Query - `crowdstrike-spl-query` -- Elasticsearch Query - `elastic-lucene-query` +- CrowdStrike Endpoint Security Query - `crowdstrike-spl-query` +- Elastic Stack Query - `elastic-lucene-query` - AWS OpenSearch Query - `opensearch-lucene-query` - Falcon LogScale Query - `logscale-lql-query` - IBM QRadar Query - `qradar-aql-query` - AWS Athena Query (Security Lake) - `athena-sql-query` - Chronicle Security Query - `chronicle-yaral-query` +- ArcSight Query - `arcsight` +- FireEye Query - `fireeye_helix` +- Graylog Query - `graylog-lucene-query` +- Logpoint Query - `logpoint` +- Qualys IOC Query - `qualys` +- RSA NetWitness Query - `rsa_netwitness` +- Securonix Query - `securonix` +- SentinelOne Query (Events) - `s1-events` +- Snowflake Query - `snowflake` +- Sumo Logic Query - `sumologic` +- VMware Carbon Black Query (Cloud) - `carbonblack` The following types of IOCs are supported: - Hash @@ -85,7 +103,8 @@ The following types of IOCs are supported: - IP TODO list of languages we will support shortly: -- Graylog +- ~LogRhythm Axon~ :white_check_mark: +- ~Graylog~ :white_check_mark: - Devo - LimaCharlie - Sumo Logic @@ -93,10 +112,10 @@ TODO list of languages we will support shortly: - ArcSight - Databricks - Cribl -- FortiSIEM +- ~FortiSIEM~ :white_check_mark: - Exabeam - Palo Alto Cortex XSOAR -- ElastAlert +- ~ElastAlert~ :white_check_mark: - FireEye OpenIOC - SentinelOne - Datadog @@ -150,13 +169,13 @@ docker-compose up -d ## :rocket: Translation 1. Select input type: - - RootA rule + - Roota rule - Sigma rule 2. Paste or upload a rule in the selected language into the input panel. 3. Select the output (language, content type, and data schema) 4. Click Translate. -If the input rule cannot be translated, you'll see an error message. When translating a RootA rule, any functions that are not supported in the target language or are not yet supported by Uncoder IO will be listed in the output as a comment. +If the input rule cannot be translated, you'll see an error message. When translating a Roota rule, any functions that are not supported in the target language or are not yet supported by Uncoder IO will be listed in the output as a comment. ## :flashlight: IOC-based Query Generation 1. Select IoCs as the input type. @@ -174,15 +193,17 @@ If the input rule cannot be translated, you'll see an error message. When transl - URL - IP - Set the number of IOCs per query to take into account the performance of your platform - - Define exceptions: specify hashes, domains, IPs, emails, files, or URLs (in full or only partially) you want to exclude from your queries + - Define exceptions: specify hashes, domains, IPs, or URLs (in full or only partially) you want to exclude from your queries 6. Click Translate. ## :coffee: Writing rules -Write a RootA or Sigma rule in the input panel. Benefit from code templates, syntax highlighting, autocomplete suggester with MITRE ATT&CK, and other nice little features that improve coding experience. +Write a [Roota](https://github.com/UncoderIO/RootA/blob/main/README.md) or Sigma rule in the input panel. Benefit from code templates, syntax highlighting, autocomplete suggester with MITRE ATT&CK, and other nice little features that improve coding experience. # :bulb: How to Contribute Thank you for your interest in the Uncoder IO open-source project! Your contribution really matters in evolving the project and helping us make Uncoder IO even more useful for the global cyber defender community. +We encourage you to commit renders into new platforms. Start with reading these [Instructions on Adding New Renders](Instructions_on_Adding_New_Renders.md). + To submit your pull request with your ideas or suggestions for changes, take the following steps: 1. Fork the [Uncoder repository](https://github.com/UncoderIO/UncoderIO) and clone your fork to your local environment. @@ -212,8 +233,8 @@ We are genuinely grateful to security professionals who contribute their time, e Uncoder IO Comunity Edition is licensed under Apache 2.0. Commercial Edition features that are released as open-source can be used non-commercially if you do not have a paid SOC Prime subscription. Commercial use rights are complimentary with SOC Prime SaaS license. Please see [LICENSE](https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE/) for details on the Uncoder IO licensing. # :book: Resources & Useful Links -[Uncoder IO](https://uncoder.io/) - free online translation engine for RootA, Sigma, and IOC-based queries +[Uncoder IO](https://uncoder.io/) - free online translation engine for Roota, Sigma, and IOC-based queries [Uncoder AI](https://tdm.socprime.com/uncoder-ai) - SaaS version of Uncoder acting as advanced IDE for detection engineering -[RootA.IO](https://roota.io/) - the main website page of the single language for threat detection & response +[Roota.IO](https://roota.io/) - the main website page of the single language for threat detection & response [SOC Prime Platform](https://tdm.socprime.com/login) - the industry-first platform for collective cyber defense [About SOC Prime](https://socprime.com/) - learn more about SOC Prime and its mission diff --git a/README_Ukrainian.md b/README_Ukrainian.md new file mode 100644 index 00000000..249685ef --- /dev/null +++ b/README_Ukrainian.md @@ -0,0 +1,240 @@ +

+ +

+ +# Що таке Uncoder IO (публічна бета-версія) + +:earth_americas: [English](README.md) [Українська](README_Ukrainian.md) + +Uncoder IO є опенсорсною версією SaaS-інструменту https://uncoder.io та Uncoder AI, другого пілоту для фахівців з кібербезпеки. Uncoder IO – це швидкий та простий у використанні онлайн-перекладач для Sigma-правил, який працює з 2018 року, забезпечуючи користувачам 100% конфіденційності. Опенсорсний Uncoder IO має такі сценарії використання: +- Переклад правил, написаних на Sigma, стандартизованому форматі для SIEM-систем, на нативні мови різних SIEM, EDR чи Data Lake. +- Пакування індикаторів компрометації з будь-якого небінарного формату, як-то PDF, текстового, STIX чи OpenIOC, у запити нативними мовами різних SIEM, EDR чи Data Lake. +- Переклад правил, написаних на Roota, новій вільно розповсюджуваній мові для колективного кіберзахисту, на нативні мови різних SIEM, EDR чи Data Lake. + +Uncoder розроблено командою спеціалістів з Detection Engineering, спеціалістів з Threat Hunting та CTI-аналітиків з України, Європи, США, Аргентини та Австралії. Цей інструмент покликаний допомогти фахівцям з кібербезпеки працювати краще та швидше на роботі та в хобі-проєктах, а також легко ділитися результатами для покращення колективного кіберзахисту. + + ![Огляд Uncoder IO](images/uncoder_io_overview.png) + +**Зміст:** +- [Ключові переваги Uncoder IO](#heart_eyes_cat-ключові-переваги-uncoder-io) +- [Підтримка мовних форматів](#dna-підтримка-мовних-форматів) +- [Встановлення](#computer-встановлення) +- [Як користуватися](#mortar_board-як-користуватися) +- [Як долучитися до проєкту](#bulb-як-долучитися-до-проєкту) +- [Питання та відгуки](#mailbox_with_no_mail-питання-та-відгуки) +- [Хто веде проєет](#wrench-хто-веде-проєкт) +- [Учасники](#kissing_heart-учасники) +- [Ліцензії](#briefcase-ліцензії) +- [Ресурси та корисні посилання](#book-ресурси-та-корисні-посилання) + +# :heart_eyes_cat: Ключові переваги Uncoder IO + +## :pretzel: Перекладач правил на Roota й Sigma + +Uncoder IO підтримує автоматичний переклад правил, написаних на Roota й Sigma, на нативні мови різних систем SIEM, EDR, XDR та Data Lake. + - **Sigma** – це відкритий універсальний формат сігнатур, який дає змогу просто та зрозуміло описувати потрібні події в логах. Згідно зі статистикою SOC Prime з перекладу та завантаження контенту, мовою Sigmа користуються експерти у галузі кібербезпеки з більше ніж 8000 організацій та 155 країн. + +- **Roota** – це мова з відкритим кодом, що підтримує запити нативними мовами SIEM-систем, синтаксис кореляції, який не залежить від вендора, а також автодоповнення коду за фреймворком MITRE ATT&CK 14.0 та автодоповнення джерел логів на основі формату OCSF від Amazon чи мови Sigma. Поєднання Roota та Uncoder є першим кроком до повної сумісності різних мов кібербезпеки, щоб одного дня фахівці, які знають певну нативну мову запитів (як-то SPL чи KQL) або універсальну мову (Roota чи Sigma), могли однаково добре користуватися всіма мовами кібербезпеки. Таким чином, складну логіку детектування загроз можна автоматично відтворити іншими мовами. У разі якщо нативне правило або запит містять функції, що не підтримуються мовою Roota або цільовою технологією, ці функції не будуть перекладені, а код перекладу буде доповнено відповідним коментарем. Це дозволяє експертам або самостійно вручну закінчити переклад, якщо вони знають мови перекладу та оригіналу, або застосувати Uncoder AI для ручної обробки подібних сценаріїв. Обмінюватися знаннями за допомогою Sigma було легко, Roota ж робить цей процес абсолютно природним – як зараз, так і в майбутньому. + +## :pizza: Редактор коду для Rootа та Sigma + +Uncoder IO має вбудований функціонал автодоповнення коду для правил Rootа та Sigma на основі останньої версії MITRE ATT&CK та словників джерел логів, що спрощує та пришвидшує процес написання правил. Будь-яка з трьох версій Uncoder полегшує написання коду. + +## :popcorn: Генератор запитів на основі індикаторів компрометації + +Uncoder IO працює як пакувальник індикаторів компрометації. За його допомогою CTI- та SOC-аналітики, а також фахівці з Threat Hunting можуть швидко розпарсити будь-яку кількість індикаторів компрометації з будь-якого цифрового небінарного формату (простого скопіювавши та вставився вебсторінку, завантаживши CSV, OpenIOC, PDF, STIX, тощо) та конвертувати їх в запити, оптимізовані для підвищення продуктивності, які можна зразу запустити у відповідній системі. Оскільки поширення індикаторів компрометації регулюється протоколом TLP, не рекомендується поширювати їх у виді правил Sigma чи Roota, оскільки останні не відносяться до аналітичної інформації про загрози та, відповідно, не мають обмежень на поширення. Однак, для галузі кібербезпеки потрібні як поведінкові сигнатури, так і індикатори компрометації, тож Uncoder IO створений, щоб допомагати у легкому вирішенні обох задач. + +## :smile_cat: Повна конфіденційність + +Uncoder IO може працювати локально без потреби в інтернет-з'єднанні, що дозволяє використовувати його у фізично ізольованих мережах. Тим не менш, ми рекомендуємо регулярно перевіряти наявність апдейтів та встановлювати їх. SaaS-версія продукту забезпечує 100% конфіденційності: в ній не використовуються cookies, не відстежуються дані або код і нічого не передається третім сторонам. Навіть використовуючи функціоність Uncoder AI, ви зберігаєте повний контроль над даними та кодом. + +# :dna: Підтримка мовних форматів +Правила на Roota та Sigma можна перекласти в такі формати: +- AWS OpenSearch Query - `opensearch-lucene-query` +- AWS Athena Query (Security Lake) - `athena-sql-query` +- Falcon LogScale Query - `logscale-lql-query` +- Falcon LogScale Rule - `logscale-lql-rule` +- Splunk Query - `splunk-spl-query` +- Splunk Alert - `splunk-spl-rule` +- Microsoft Sentinel Query - `sentinel-kql-query` +- Microsoft Sentinel Rule - `sentinel-kql-rule` +- Microsoft Defender for Endpoint Query - `mde-kql-query` +- IBM QRadar Query - `qradar-aql-query` +- CrowdStrike Query - `crowdstrike-spl-query` +- Elasticsearch Query - `elastic-lucene-query` +- Elasticsearch Rule - `elastic-lucene-rule` +- ElastAlert Rule - `elastalert-lucene-rule` +- Sigma Rule - `sigma-yml-rule` +- Chronicle Security Query - `chronicle-yaral-query` +- Chronicle Security Rule - `chronicle-yaral-rule` +- Graylog Query - `graylog-lucene-query` +- FortiSIEM Rule - `fortisiem-rule` +- LogRhythm Axon Rule - `axon-ads-rule` +- LogRhythm Axon Query - `axon-ads-query` + +Запити на основі індикаторів компрометації можна генерувати в таких форматах: +- Microsoft Sentinel Query - `sentinel-kql-query` +- Microsoft Defender for Endpoint Query - `mde-kql-query` +- Splunk Query - `splunk-spl-query` +- CrowdStrike Endpoint Security Query - `crowdstrike-spl-query` +- Elastic Stack Query - `elastic-lucene-query` +- AWS OpenSearch Query - `opensearch-lucene-query` +- Falcon LogScale Query - `logscale-lql-query` +- IBM QRadar Query - `qradar-aql-query` +- AWS Athena Query (Security Lake) - `athena-sql-query` +- Chronicle Security Query - `chronicle-yaral-query` +- ArcSight Query - `arcsight` +- FireEye Query - `fireeye_helix` +- Graylog Query - `graylog-lucene-query` +- Logpoint Query - `logpoint` +- Qualys IOC Query - `qualys` +- RSA NetWitness Query - `rsa_netwitness` +- Securonix Query - `securonix` +- SentinelOne Query (Events) - `s1-events` +- Snowflake Query - `snowflake` +- Sumo Logic Query - `sumologic` +- VMware Carbon Black Query (Cloud) - `carbonblack` + +Підтримуються такі типи індикаторів компрометації: +- Хеші +- Домени +- URL-адреси +- IP-адреси + +Найближчим часом планується підтримка таких форматів: +- ~LogRhythm Axon~ :white_check_mark: +- ~~Graylog~~ :white_check_mark: +- Devo +- LimaCharlie +- Sumo Logic +- Sumo Logic CSE +- ArcSight +- Databricks +- Cribl +- ~FortiSIEM~ :white_check_mark: +- Exabeam +- Palo Alto Cortex XSOAR +- ~~ElastAlert~~ :white_check_mark: +- FireEye OpenIOC +- SentinelOne +- Datadog +- FireEye Helix +- Logpoint +- RSA NetWitness +- PowerShell +- Snowflake +- SQL +- VMware Carbon Black +- Apache Kafka ksqlDB +- HawkSearch +- Regex Grep +- Logiq +- Qualys +- Securonix +- STIX +- StreamAlert +- Sysmon +- UberAgent ESA + +# :computer: Встановлення +Uncoder IO можна встановити в один із таких способів: + 1. Docker-контейнер з вебсервером та UI-інтерфейсом [Інструкція із запуску Docker-контейнеру](#launch-instructions-for-docker-container) + 2. Docker-контейнер з API та CLI-інтерфейсом (у процесі розробки) + 3. Збірка безпосередньо з вихідного коду (для досвідчених користувачів; виконується на основі інструкцій з файлів Docker) + 4. Готова до використання SaaS-версія із забезпеченням конфіденційності, без cookie та без реєстрації: [https://uncoder.io/](https://uncoder.io/) + 5. Готова до використання версія на SaaS-платформі SOC Prime з конфіденційним збагаченням даними з використанням AI, сертифікацією SOC 2 Type II, Умовами використання, що регулюють конфіденційність даних, дотриманням GDPR, тощо: [https://tdm.socprime.com/uncoder-ai](https://tdm.socprime.com/uncoder-ai) + +Нижче викладено вимоги та інструкції із запуску Docker-контейнера з вебсервером та UI-інтерфейсом. + +## Вимоги до Docker-контейнера +* Хост із Windows, Linux або іншою операційною системою, яку підтримує Docker. +* На хості мають бути встановлені такі пакети: + * Docker v23.0.1 чи новішої версії + * Docker Compose v2.21.0 чи новішої версії. + +## Інструкція із запуску Docker-контейнера +1. Завантажте й розпакуйте архів `UncoderIO-main`. +2. Використовуючи консоль (командний рядок), перейдіть до папки з розпакованими файлами: +``` +cd UncoderIO-main/ +``` +3. Виконайте цю команду для запуску Docker-контейнера: +``` +docker-compose up -d +``` +4. Відкрийте `http://localhost:4010/` у браузері для початку роботи. + +# :mortar_board: Як користуватися + +## :rocket: Переклад +1. Виберіть формат оригіналу: + - Roota + - Sigma +2. Вставте або завантажте правило у вибраному форматі до панелі введення зліва. +3. Виберіть формат перекладу (мову, тип контенту та схему даних). +4. Натисніть кнопку Translate. + +Якщо оригінал правила неможливо перекласти, з’явиться повідомлення про помилку. У разі перекладу правил Roota будь-які функції, що не підтримує цільова мова або Uncoder IO, будуть перелічені в коментарі на панелі з результатами перекладу. + +## :flashlight: Генерація запитів на основі індикаторів компрометації +1. Виберіть IoCs (індикатори компрометації) як формат оригіналу. +2. Вставте або завантажте текст з індикаторами компрометації до панелі введення зліва. +3. Виберіть налаштування парсингу: + - **Select all:** – застосувати всіх перераховані опції + - **Replace (.) [.] {.} with dot** – замінити (.) [.] {.} на крапку + - **Replace hxxp with http** – замінити hxxp на http. Зверність увагу, що ця функціональність не чутлива до регістру, тому hXXp, HXXP, HXXp та hXXP будуть також замінені + - **Exclude Private & Reserved Networks:** – ігнорувати під час визначення індикаторів компрометації приватні та зарезервовані IP-адреси на кшталт 224.0.0.0/4 чи 127.0.0.0/8 +4. Оберіть мову, якою потрібно згенерувати запити. +5. Встановіть налаштування для генерації запиту: + - Виберіть, які типи індикаторів компрометації включати в запити: + - Хеші + - Домени + - URL-адреси + - IP-адреси + - Встановіть кількість індикаторів компрометації на запит з урахуванням продуктивності вашої платформи. + - Зазначте виключення: хеші, домени, IP-адреси чи URL-адреси (повністю або частково), які ви хочете не враховувати у генерації запиту. +6. Натисніть кнопку Translate. + +## :coffee: Написання правил +Пишіть правила Roota чи Sigma прямо у панелі введення зліва. Використовуйте шаблони коду, підсвічування синтаксису, функцію автодоповнення із підказками на основі MITRE ATT&CK та інші приємні переваги Uncoder IO, щоб полегшити та пришвидшити роботу з кодом. + +# :bulb: Як долучитися до проєкту +Дякуємо за інтерес до відкритого проєкту Uncoder IO! Ми вдячні кожному, хто допомагає розвивати цей проєкт і робити Uncoder IO ще більш корисним для глобальної спільноти спеціалістів з кіберзахисту. + +Ми будемо раді вашим комітам з рендерами для нових платформ перекладу. Для початку прочитайте цю [Інструкцію з додавання нових рендерів](Instructions_on_Adding_New_Renders.md). + +Щоб зробити пул-реквест з ідеями або пропозиціями, виконайте такі дії: + +1. Зробіть форк [репозиторія Uncoder](https://github.com/UncoderIO/UncoderIO) і створіть його локальну копію. +2. Створіть гілку, в якій ви вноситимете зміни. +3. Зробіть коміт зі внесеними змінами в створену вами гілку. +4. Відправте зміни у свій форк. +5. Створіть новий пул-реквест. + a. Натисніть кнопку New Pull Request (Новий пул-реквест). + b. Виберіть свій форк з гілкою, яка містить зміни. + c. Зазначте назву та опис змін. Вони мають бути чіткими й інформативними. + d. Відправте пул-реквест і очікуйте на його схвалення. + +Дякуємо за ваш внесок в проєкт Uncoder IO! + +# :mailbox_with_no_mail: Питання та відгуки +Відправляйте ваші технічні відгуки чи пропозиції на адресу support@socprime.com або у спеціальному каналі **Uncoder** у нашій спільноті на [Discord](https://discord.gg/socprime). Також ви можете переглянути [інструкцію, як долучитися до проєкту](#how-to-contribute) для підтримки Uncoder IO або просто [повідомити про баги](https://github.com/UncoderIO/UncoderIO/issues). + +# :wrench: Хто веде проєкт +Команда SOC Prime розвиває проєкт Uncoder починаючи з 2018 року. Першими кроками були підтримка правил Sigma та проєкт Uncoder IO, інтегроване середовище розробки (IDE) для Detection Engineering, доступне онлайн з повним збереженням конфіденційності. Тепер команда SOC Prime виклала Uncoder IO як проєкт з відкритим вихідним кодом. + +Проєкт Uncoder IO підтримується командою SOC Prime. Ми вдячні за будь-які пропозиції та повідомлення про баги, але наголошуємо, що фінальне рішення щодо схвалення або відхилення будь-якого пул-реквесту залишається за командою розробників SOC Prime. + +# :kissing_heart: Учасники + +Ми щиро вдячні всім спеціалістам з кіберзахисту, які застосовують свої знання, проявляють кмітливість і докладають час для розвитку відкритого проєкту Uncoder IO. + +# :briefcase: Ліцензії +Безкоштовна версія Uncoder IO (Uncoder IO Community Edition) випущена під ліцензією Apache 2.0. Функції платної версії Uncoder IO (Commercial Edition), за умови, що вони були випущені у відкритому доступі, можуть бути доступними не на комерційній основі для користувачів, що не мають преміальної підписки SOC Prime. Права на комерційне використання доповнюють ліцензію, що діє для SOC Prime SaaS-платформи. Дивіться файл [ЛІЦЕНЗІЯ](https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE/) для отримання більш детальної інформації щодо ліцензування Uncoder IO. + +# :book: Ресурси та корисні посилання +[Uncoder IO](https://uncoder.io/) – безкоштовний онлайн-перекладач правил Roota та Sigma, а також генератор запитів на основі індикаторів компрометації (IOC) +[Uncoder AI](https://tdm.socprime.com/uncoder-ai) – SaaS-версія Uncoder, що функціонує як інтегроване середовище розробки (IDE) для Detection Engineering +[Roota.IO](https://roota.io/) – основна вебсторінка проєкту Roota, універсальної мови для детектування загроз та реагування +[SOC Prime Platform](https://tdm.socprime.com/login) – перша в індустрії платформа для побудови колективного кіберзахисту +[About SOC Prime](https://socprime.com/) – більше інформації про SOC Prime та місію компанії diff --git a/docker-compose.yml b/docker-compose.yml index 713c61cc..01877c90 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -9,10 +9,10 @@ services: - HOST=0.0.0.0 ports: - '4010:4010' - siem-converter: + translator: build: - context: './siem-converter/' - container_name: siem-converter + context: './uncoder-core/' + container_name: uncoder-core restart: always environment: - HOST=0.0.0.0 diff --git a/siem-converter/app/converter/const.py b/siem-converter/app/converter/const.py deleted file mode 100644 index 03d94491..00000000 --- a/siem-converter/app/converter/const.py +++ /dev/null @@ -1,5 +0,0 @@ -from os.path import abspath, dirname - -APP_PATH = dirname(abspath(__file__)) - -CTI_MIN_LIMIT_QUERY = 10000 diff --git a/siem-converter/app/converter/converter.py b/siem-converter/app/converter/converter.py deleted file mode 100644 index 87652d46..00000000 --- a/siem-converter/app/converter/converter.py +++ /dev/null @@ -1,68 +0,0 @@ -import logging - -from app.converter.platforms.roota.parsers.roota import RootAParser -from app.converter.core.exceptions.core import UnsupportedPlatform -from app.converter.core.operator_types.output import SiemContainer -from app.converter.managers import RenderManager, ParserManager, render_manager, parser_manager -from app.converter.tools.decorators import handle_translation_exceptions - - -class SiemConverter: - renders: RenderManager = render_manager - parsers: ParserManager = parser_manager - - def __init__(self): - self.logger = logging.getLogger("siem_converter") - - @handle_translation_exceptions - def __parse_incoming_data(self, text: str, source: str) -> SiemContainer: - parser = RootAParser() if source == "roota" else self.parsers.get(source) - if not parser: - raise UnsupportedPlatform(platform=source, is_parser=True) - return parser.parse(text=text) - - @handle_translation_exceptions - def __render_translation(self, parsed_data: SiemContainer, target: str) -> str: - render = self.renders.get(target) - if not render: - raise UnsupportedPlatform(platform=target) - return render.generate( - query=parsed_data.query, - meta_info=parsed_data.meta_info, - functions=parsed_data.functions - ) - - def __generate_one_translation(self, text: str, source: str, target: str) -> (bool, str): - status, parsed_data = self.__parse_incoming_data(text=text, source=source) - if status: - return self.__render_translation(parsed_data=parsed_data, target=target) - return status, parsed_data - - def __generate_all(self, text: str, source: str) -> list[dict]: - status, parsed_data = self.__parse_incoming_data(text=text, source=source) - if status: - result = [] - for target in self.renders.all_platforms(): - if target == source: - continue - translation = {"siem_type": target} - render_status, data = self.__render_translation(parsed_data=parsed_data, target=target) - translation.update({"status": render_status, "result": data}) - result.append(translation) - return result - return [{"status": status, "result": parsed_data, "siem_type": source}] - - def generate_translation(self, text: str, source: str, target: str) -> (bool, str): - return self.__generate_one_translation(text=text, source=source, target=target) - - def generate_all_translation(self, text: str, source: str) -> list[dict]: - return self.__generate_all(text=text, source=source) - - def get_all_platforms(self) -> tuple: - return self.get_renders(), self.get_parsers() - - def get_parsers(self) -> list: - return self.parsers.get_platforms_details - - def get_renders(self) -> list: - return self.renders.get_platforms_details diff --git a/siem-converter/app/converter/core/compiler.py b/siem-converter/app/converter/core/compiler.py deleted file mode 100644 index e7522b92..00000000 --- a/siem-converter/app/converter/core/compiler.py +++ /dev/null @@ -1,59 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from app.converter.core.models.field import Field, Keyword -from app.converter.core.models.group import Group -from app.converter.core.models.identifier import Identifier -from app.converter.core.models.operator import Operator, NOT -from app.converter.core.operator_types.tokens import LogicalOperatorType, GroupType - - -class DataStructureCompiler: - # pylint: disable=inconsistent-return-statements - def generate(self, tokens: list, group: Group = None): - if not tokens: - group.finalize() - return group - group = group if group else Group() - token = tokens[0] - if isinstance(token, (Field, Keyword)): - group += token - return self.generate(tokens=tokens[1::], group=group) - elif token.token_type == LogicalOperatorType.OR or token.token_type == LogicalOperatorType.AND: - group.items = Operator(operator_type=token.token_type) - return self.generate(tokens=tokens[1::], group=group) - elif token.token_type == LogicalOperatorType.NOT: - if isinstance(tokens[1], (Field, Keyword)): - tokens.insert(2, Identifier(token_type=GroupType.R_PAREN)) - tokens.insert(1, Identifier(token_type=GroupType.L_PAREN)) - sub_group = Group() - sub_group.items = NOT() - not_sub_group, new_tokens = self.generate(tokens=tokens[1::], group=sub_group) - group += not_sub_group - return self.generate(new_tokens, group) - elif token.token_type == GroupType.L_PAREN: - sub_group, new_tokens = self.generate(tokens=tokens[1::]) - if not sub_group.is_null: - group += sub_group - if isinstance(group.items, NOT): - group.finalize() - return group, new_tokens - return self.generate(new_tokens, group) - elif token.token_type == GroupType.R_PAREN: - group.finalize() - return group, tokens[1::] diff --git a/siem-converter/app/converter/core/exceptions/core.py b/siem-converter/app/converter/core/exceptions/core.py deleted file mode 100644 index 1cb9e5e3..00000000 --- a/siem-converter/app/converter/core/exceptions/core.py +++ /dev/null @@ -1,79 +0,0 @@ -from typing import List - - -class NotImplementedException(BaseException): - ... - - -class BasePlatformException(BaseException): - ... - - -class StrictPlatformException(BasePlatformException): - - def __init__(self, platform_name: str, field_name: str): - message = f"Platform {platform_name} has strict mapping. Source field {field_name} has no mapping." - super().__init__(message) - - -class StrictPlatformFieldException(BasePlatformException): - - def __init__(self, platform_name: str, field_name: str): - message = f"Source field `{field_name}` has no mapping for platform {platform_name}." - super().__init__(message) - - -class UnsupportedPlatform(BasePlatformException): - - def __init__(self, platform: str, is_parser=False): - converter_direction = "input" if is_parser else "output" - if platform: - message = f"The selected {converter_direction} language `{platform}` is not supported. Please, select an option in the dropdown." - else: - message = f"Please, select an {converter_direction} language." - super().__init__(message) - - -class UnsupportedRootAParser(BasePlatformException): - def __init__(self, parser: str): - message = f"The platform `{parser}` specified in the body is not supported. Please, use platform names listed in the readme.md" - super().__init__(message) - - -class RootAParserException(BasePlatformException): - def __init__(self, platform_name: str, error: str): - message = f"The body part of the detection section has invalid {platform_name} syntax. Error: {error}" - super().__init__(message) - - -class YamlRuleValidationException(BasePlatformException): - rule = None - - def __init__(self, missing_fields: List[str]): - prepared_missing_fields = "\n".join(f"- {field}" for field in missing_fields) - message = f"Invalid {self.rule} rule structure. Some required fields are missing: {prepared_missing_fields}." - super().__init__(message) - - -class RootARuleValidationException(YamlRuleValidationException): - rule = "RootA" - - -class SigmaRuleValidationException(YamlRuleValidationException): - rule = "Sigma" - - -class InvalidRuleStructure(BasePlatformException): - rule_type = None - - def __init__(self, error: str): - message = f"Invalid {self.rule_type} structure. Issue: {error}." - super().__init__(message) - - -class InvalidYamlStructure(InvalidRuleStructure): - rule_type: str = "YAML" - - -class InvalidJSONStructure(InvalidRuleStructure): - rule_type: str = "JSON" diff --git a/siem-converter/app/converter/core/exceptions/parser.py b/siem-converter/app/converter/core/exceptions/parser.py deleted file mode 100644 index c39edf2b..00000000 --- a/siem-converter/app/converter/core/exceptions/parser.py +++ /dev/null @@ -1,35 +0,0 @@ - -class BaseParserException(BaseException): - ... - - -class TokenizerGeneralException(BaseParserException): - - def __init__(self, error: str): - message = f"It looks like there's been an issue during translation. {error}. Resolve the issue and try again. If you think there's no issue, please contact us. We are improving the translation capabilities for this language, and some operators may be not supported yet." - super().__init__(message) - - -class UnsupportedOperatorException(BaseParserException): - - def __init__(self, operator: str): - message = f"Cannot translate. Operator {operator} is not supported." - super().__init__(message) - - -class QueryParenthesesException(BaseParserException): - def __init__(self): - message = f"The query logic is broken. In the input, the numbers of opening and closing parentheses do not match. If you think there's no error, please contact us via GitHub." - super().__init__(message) - - -class FieldMappingException(BaseParserException): - def __init__(self, field_name: str): - message = f"Cannot translate. The field {field_name} does not exist in the output language or its translation is not supported yet. If you want to contribute to the open-source translation capabilities into this language, do it via our UncoderIO GitHub project." - super().__init__(message) - - -class FunctionParseException(BaseParserException): - ... - # def __init__(self, message: str): - # self.message = message \ No newline at end of file diff --git a/siem-converter/app/converter/core/exceptions/render.py b/siem-converter/app/converter/core/exceptions/render.py deleted file mode 100644 index a6e20fe6..00000000 --- a/siem-converter/app/converter/core/exceptions/render.py +++ /dev/null @@ -1,20 +0,0 @@ - -class BaseRenderException(BaseException): - ... - - -class UnexpectedLogsourceException(BaseRenderException): - def __init__(self, platform_name: str, log_source: str): - message = f"{platform_name} can't convert query with logsources: {log_source}" - super().__init__(message) - - -class FunctionRenderException(BaseRenderException): - ... - - -class UnsupportedRenderMethod(BaseRenderException): - - def __init__(self, platform_name: str, method: str): - message = f"Cannot translate. {platform_name} backend does not support {method}." - super().__init__(message) diff --git a/siem-converter/app/converter/core/functions.py b/siem-converter/app/converter/core/functions.py deleted file mode 100644 index 1240fa76..00000000 --- a/siem-converter/app/converter/core/functions.py +++ /dev/null @@ -1,24 +0,0 @@ -from app.converter.core.models.functions.types import ParsedFunctions, NotSupportedFunction - - -class Functions: - siem_type: str - parser_functions_map = {} - function_delimiter = "|" - - def parse(self, query: str) -> ParsedFunctions: - result = [] - not_supported = [] - functions = query.split(self.function_delimiter) - for function in functions: - function_name = function.split(' ')[0] - if function_name in self.parser_functions_map: - pass - else: - not_supported.append(NotSupportedFunction(name=function_name, query=function)) - return ParsedFunctions(not_supported=self.prepare_not_supported(not_supported), functions=result) - - def prepare_not_supported(self, not_supported): - for n in range(len(not_supported)): - not_supported[n] = f'| {not_supported[n].query}' - return not_supported diff --git a/siem-converter/app/converter/core/mapping.py b/siem-converter/app/converter/core/mapping.py deleted file mode 100644 index f9d281a2..00000000 --- a/siem-converter/app/converter/core/mapping.py +++ /dev/null @@ -1,124 +0,0 @@ -from __future__ import annotations - -from abc import ABC, abstractmethod -from typing import List, Dict, Optional, TypeVar - -from app.converter.mappings.utils.load_from_files import LoaderFileMappings - - -DEFAULT_MAPPING_NAME = "default" - - -class LogSourceSignature(ABC): - _default_source: dict - - @abstractmethod - def is_suitable(self, *args, **kwargs) -> bool: - raise NotImplementedError("Abstract method") - - @abstractmethod - def __str__(self) -> str: - raise NotImplementedError("Abstract method") - - -class FieldMapping: - def __init__(self, generic_field_name: str, platform_field_name: str): - self.generic_field_name = generic_field_name - self.platform_field_name = platform_field_name - - -class FieldsMapping: - - def __init__(self, fields_mapping: list): - self.__parser_mapping = self.__build_parser_mapping(fields_mapping) - self.__render_mapping = self.__build_render_mapping(fields_mapping) - - @staticmethod - def __build_parser_mapping(fields: List[FieldMapping]) -> Dict[str, FieldMapping]: - result = {} - for field in fields: - if isinstance(field.platform_field_name, list): - for f in field.platform_field_name: - result.update({f: field}) - else: - result.update({field.platform_field_name: field}) - return result - - @staticmethod - def __build_render_mapping(fields: List[FieldMapping]) -> Dict[str, FieldMapping]: - return {field.generic_field_name: field for field in fields} - - def get_generic_field_name(self, platform_field_name: str) -> Optional[str]: - field = self.__parser_mapping.get(platform_field_name) - return field and field.generic_field_name - - def get_platform_field_name(self, generic_field_name: str) -> Optional[str]: - field = self.__render_mapping.get(generic_field_name) - return field and field.platform_field_name - - def update(self, fields_mapping: FieldsMapping) -> None: - self.__parser_mapping.update(fields_mapping.__parser_mapping) - self.__render_mapping.update(fields_mapping.__render_mapping) - - def is_suitable(self, field_names: List[str]) -> bool: - return set(field_names).issubset(set(self.__parser_mapping.keys())) - - -_LogSourceSignatureType = TypeVar('_LogSourceSignatureType', bound=LogSourceSignature) - - -class SourceMapping: - def __init__(self, - source_id: str, - log_source_signature: _LogSourceSignatureType = None, - fields_mapping: FieldsMapping = FieldsMapping([])): - self.source_id = source_id - self.log_source_signature = log_source_signature - self.fields_mapping = fields_mapping - - -class BasePlatformMappings: - - def __init__(self, platform_dir: str): - self.__loader = LoaderFileMappings() - self.__platform_dir = platform_dir - self._source_mappings = self.prepare_mapping() - - def prepare_mapping(self) -> Dict[str, SourceMapping]: - source_mappings = {} - default_mapping = SourceMapping(source_id=DEFAULT_MAPPING_NAME) - for mapping_dict in self.__loader.load_siem_mappings(self.__platform_dir): - if (source_id := mapping_dict["source"]) == DEFAULT_MAPPING_NAME: - default_mapping.log_source_signature = self.prepare_log_source_signature(mapping=mapping_dict) - continue - - fields_mapping = self.prepare_fields_mapping(field_mapping=mapping_dict.get("field_mapping", {})) - default_mapping.fields_mapping.update(fields_mapping) - log_source_signature = self.prepare_log_source_signature(mapping=mapping_dict) - source_mappings[source_id] = SourceMapping( - source_id=source_id, - log_source_signature=log_source_signature, - fields_mapping=fields_mapping - ) - - source_mappings[DEFAULT_MAPPING_NAME] = default_mapping - - return source_mappings - - @staticmethod - def prepare_fields_mapping(field_mapping: dict) -> FieldsMapping: - fields = [] - for generic_field_name, platform_field_name in field_mapping.items(): - fields.append(FieldMapping(generic_field_name=generic_field_name, platform_field_name=platform_field_name)) - return FieldsMapping(fields_mapping=fields) - - @abstractmethod - def prepare_log_source_signature(self, mapping: dict) -> LogSourceSignature: - raise NotImplementedError("Abstract method") - - @abstractmethod - def get_suitable_source_mappings(self, *args, **kwargs) -> List[SourceMapping]: - raise NotImplementedError("Abstract method") - - def get_source_mapping(self, source_id: str) -> Optional[SourceMapping]: - return self._source_mappings.get(source_id) diff --git a/siem-converter/app/converter/core/mixins/operator.py b/siem-converter/app/converter/core/mixins/operator.py deleted file mode 100644 index 39a3e27f..00000000 --- a/siem-converter/app/converter/core/mixins/operator.py +++ /dev/null @@ -1,65 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from typing import Union, List, Tuple - -from app.converter.core.models.identifier import Identifier -from app.converter.core.operator_types.tokens import OperatorType - - -class WildCardMixin: - - @staticmethod - def _clean_value(value: str, wildcard_symbol: str) -> str: - return value.strip(wildcard_symbol) - - def get_field_value_operator(self, - value: Union[str, List[str]], - operator: str = None, - wildcard_symbol: str = None) -> Tuple[Union[List[str], str], Identifier]: - if wildcard_symbol: - value = self._clean_value(value, wildcard_symbol) - return value, Identifier(token_type=operator) - - -class OperatorBasedMixin(WildCardMixin): - - def get_field_value_operator(self, - value: Union[str, List[str]], - operator: str = None, - wildcard_symbol: str = None) -> Tuple[Union[List[str], str], Identifier]: - value, operator = self.__get_value_operator(operator=operator, value=value) - if wildcard_symbol: - value = self._clean_value(value, wildcard_symbol) - return value, operator - - @staticmethod - def __get_value_operator(operator: str, value: Union[str, List[str]]) -> Tuple[Union[List[str], str], Identifier]: - if operator == OperatorType.CONTAINS: - operator = Identifier(token_type=OperatorType.CONTAINS) - elif operator == OperatorType.ENDSWITH: - operator = Identifier(token_type=OperatorType.ENDSWITH) - elif operator == OperatorType.STARTSWITH: - operator = Identifier(token_type=OperatorType.STARTSWITH) - elif operator == OperatorType.REGEX: - operator = Identifier(token_type=OperatorType.REGEX) - elif operator == OperatorType.EQ: - operator = Identifier(token_type=OperatorType.EQ) - else: - raise Exception("Unknown operator") - return value, operator diff --git a/siem-converter/app/converter/core/mixins/rule.py b/siem-converter/app/converter/core/mixins/rule.py deleted file mode 100644 index c6d011be..00000000 --- a/siem-converter/app/converter/core/mixins/rule.py +++ /dev/null @@ -1,23 +0,0 @@ -import json - -import yaml - -from app.converter.core.exceptions.core import InvalidYamlStructure, InvalidJSONStructure - - -class JsonRuleMixin: - - def load_rule(self, text): - try: - return json.loads(text) - except json.JSONDecodeError as err: - raise InvalidJSONStructure(error=str(err)) - - -class YamlRuleMixin: - - def load_rule(self, text): - try: - return yaml.safe_load(text) - except yaml.YAMLError as err: - raise InvalidYamlStructure(error=str(err)) diff --git a/siem-converter/app/converter/core/models/field.py b/siem-converter/app/converter/core/models/field.py deleted file mode 100644 index e881cf3b..00000000 --- a/siem-converter/app/converter/core/models/field.py +++ /dev/null @@ -1,85 +0,0 @@ -from typing import Union, List - -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.identifier import Identifier -from app.converter.core.operator_types.tokens import OperatorType - - -class Field: - def __init__(self, operator: Identifier, source_name: str, value: Union[int, str, list, tuple]): - self.operator = operator - self.values = [] - self.__add_value(value) - self.source_name = source_name # input translation field name - self.generic_names_map = {} - - @property - def value(self): - if isinstance(self.values, list) and len(self.values) == 1: - return self.values[0] - return self.values - - def __add_value(self, value: Union[int, str, list, tuple]): - if value and isinstance(value, (list, tuple)): - self.values.extend(value) - elif value and isinstance(value, str) and value.isnumeric(): - self.values.append(int(value)) - elif value is not None and isinstance(value, (int, str)): - self.values.append(value) - - def __add__(self, other): - self.values.append(other) - - def __repr__(self): - return f"{self.source_name} {self.operator.token_type} {self.values}" - - def __eq__(self, other): - if isinstance(other, Field): - return self._hash == other._hash - """For OR operator check""" - if self.source_name == other.source_name and self.operator == other.operator: - return True - return False - - def __neq__(self, other): - """For AND operator check""" - if self.source_name != other.source_name: - return True - return False - - @property - def _hash(self): - return hash(str(self)) - - def __hash__(self): - return hash(str(self)) - - -class Keyword: - def __init__(self, value): - self.operator: Identifier = Identifier(token_type=OperatorType.KEYWORD) - self.name = "keyword" - self.values: [str] = [] - self.__add_value(value=value) - - @property - def value(self): - if isinstance(self.values, list) and len(self.values) == 1: - return self.values[0] - return self.values - - def __add_value(self, value): - if value and isinstance(value, (list, tuple)): - self.values.extend(value) - elif value and isinstance(value, str): - self.values.append(value) - - def __add__(self, other): - if other and isinstance(other, (list, tuple)): - self.values.extend(other) - elif other and isinstance(other, str): - self.values.append(other) - return self - - def __repr__(self): - return f"{self.name} {self.operator.token_type} {self.values}" diff --git a/siem-converter/app/converter/core/models/functions/aggregation.py b/siem-converter/app/converter/core/models/functions/aggregation.py deleted file mode 100644 index 1f9acc8d..00000000 --- a/siem-converter/app/converter/core/models/functions/aggregation.py +++ /dev/null @@ -1,25 +0,0 @@ -from app.converter.core.models.functions.types import AggregationType - - -class AggregationField: - def __init__( - self, - fieldname: str, - operation_type: AggregationType, - render_as: str = None - ): - self.fieldname = fieldname - self.operation_type = operation_type - self.render_as = render_as - - def __repr__(self): - base = f'{self.operation_type}:{self.fieldname}' - if self.render_as: - base += f' as {self.render_as}' - return base - - -class AggregationExpression: - def __init__(self): - self.fields = [] - self.group_by = [] diff --git a/siem-converter/app/converter/core/models/functions/search.py b/siem-converter/app/converter/core/models/functions/search.py deleted file mode 100644 index 9ec484b9..00000000 --- a/siem-converter/app/converter/core/models/functions/search.py +++ /dev/null @@ -1,32 +0,0 @@ -from enum import Enum, auto - -from app.converter.core.models.functions.types import ComparsionType, ModifierType, OperatorType - - -class SearchValueType(Enum): - ANY = auto() - - -class SearchField: - def __init__( - self, - fieldname: str, - value: [str, SearchValueType], - operator: ComparsionType, - modifier: [None, ModifierType] = None, - case_sensitive: bool = False, - full_match: bool = False - ): - self.operator = operator - self.fieldname = fieldname - self.value = value - self.modifier = modifier - self.case_sensitive = case_sensitive - self.full_match = full_match - - -class SearchExpression: - def __init__(self, operator: OperatorType, fields: list, case_sensitive: bool = False): - self.operator = operator - self.fields = fields - self.case_sensitive = case_sensitive diff --git a/siem-converter/app/converter/core/models/functions/sort.py b/siem-converter/app/converter/core/models/functions/sort.py deleted file mode 100644 index 4af2551f..00000000 --- a/siem-converter/app/converter/core/models/functions/sort.py +++ /dev/null @@ -1,33 +0,0 @@ -from enum import Enum, auto - - -class SortTransformType(Enum): - INT = auto() - STRING = auto() - - -class SortOrderType(Enum): - ASC = auto() - DESC = auto() - - -class SortField: - def __init__(self, fieldname: str, order: SortOrderType = SortOrderType.ASC, transform_into: SortTransformType = None): - self.fieldname = fieldname - self.order = order - self.transform_into = transform_into - - -class SortExpression: - def __init__(self, count: int = 10_000, order: str = SortOrderType.DESC): - self.count = count - self.order = order - self.fields = [] - self.is_count_default = True - - def set_count(self, value): - self.is_count_default = False - self.count = value - - def is_default_count(self): - return self.is_count_default diff --git a/siem-converter/app/converter/core/models/functions/table.py b/siem-converter/app/converter/core/models/functions/table.py deleted file mode 100644 index c8127df3..00000000 --- a/siem-converter/app/converter/core/models/functions/table.py +++ /dev/null @@ -1,13 +0,0 @@ -from app.converter.core.operator_types.tokens import OperatorType - - -class TableField: - def __init__(self, fieldname: str, raw_fieldname: str, modifier: OperatorType = OperatorType.EQ): - self.fieldname = fieldname - self.raw_fieldname = raw_fieldname - self.modifier = modifier - - -class TableExpression: - def __init__(self): - self.fields = [] diff --git a/siem-converter/app/converter/core/models/functions/types.py b/siem-converter/app/converter/core/models/functions/types.py deleted file mode 100644 index d9c82214..00000000 --- a/siem-converter/app/converter/core/models/functions/types.py +++ /dev/null @@ -1,57 +0,0 @@ -from enum import Enum, auto -from dataclasses import dataclass, field - - -class ComparsionType(Enum): - EQUAL = auto() - ILIKE = auto() - NOT_EQUAL = auto() - GTE = auto() - LTE = auto() - GT = auto() - LT = auto() - - -class ModifierType(Enum): - CONTAINS = auto() - STARTSWITH = auto() - ENDSWITH = auto() - REGEX = auto() - EQUAL = auto() - - -class OperatorType(Enum): - AND = auto() - OR = auto() - NOT = auto() - XOR = auto() - - -class AggregationType(Enum): - SUM = auto() - COUNT = auto() - AVG = auto() - MIN = auto() - MAX = auto() - MEDIAN = auto() - DISTINCT_COUNT = auto() - RANGE = auto() - - -class NotSupportedFunction: - def __init__(self, name: str, query: str): - self.name = name - self.query = query - - -class SubFunc: - def __init__(self, func_name: str, fieldname: str, values: list): - self.func_name = func_name - self.fieldname = fieldname - self.values = values - - -@dataclass -class ParsedFunctions: - functions: list = field(default_factory=list) - not_supported: list = field(default_factory=list) diff --git a/siem-converter/app/converter/core/models/functions/union.py b/siem-converter/app/converter/core/models/functions/union.py deleted file mode 100644 index 8dd029e1..00000000 --- a/siem-converter/app/converter/core/models/functions/union.py +++ /dev/null @@ -1,4 +0,0 @@ -class UnionExpression: - def __init__(self, fieldname: str, fields: list = []): - self.fieldname = fieldname - self.fields = fields \ No newline at end of file diff --git a/siem-converter/app/converter/core/models/functions/where.py b/siem-converter/app/converter/core/models/functions/where.py deleted file mode 100644 index b69deaa2..00000000 --- a/siem-converter/app/converter/core/models/functions/where.py +++ /dev/null @@ -1,5 +0,0 @@ -from app.converter.core.models.functions.search import SearchExpression - - -class WhereExpression(SearchExpression): - ... diff --git a/siem-converter/app/converter/core/models/identifier.py b/siem-converter/app/converter/core/models/identifier.py deleted file mode 100644 index c62c6bdb..00000000 --- a/siem-converter/app/converter/core/models/identifier.py +++ /dev/null @@ -1,15 +0,0 @@ -from dataclasses import dataclass - -from app.converter.core.operator_types.tokens import ValidTokens - - -@dataclass -class Identifier: - def __init__(self, *, token_type: str) -> None: - if token_type not in ValidTokens: - raise Exception(f"Unexpected token type: {token_type}") - - self.token_type = token_type - - def __repr__(self): - return f"{self.token_type}" diff --git a/siem-converter/app/converter/core/models/operator.py b/siem-converter/app/converter/core/models/operator.py deleted file mode 100644 index c59c24f2..00000000 --- a/siem-converter/app/converter/core/models/operator.py +++ /dev/null @@ -1,50 +0,0 @@ -from abc import ABC - -from app.converter.core.operator_types.tokens import LogicalOperatorType - - -class BaseOperator(ABC): - token_type: str = None - - def __init__(self, items=None): - self.items = [] if items is None else items - - def __add__(self, other): - if isinstance(other, list): - self.items.extend(other) - else: - self.items.append(other) - return self - - def __repr__(self): - return f"{self.token_type}({self.items})" - - def remove_last_item(self): - self.items = self.items[:-1] - - -class AND(BaseOperator): - token_type = LogicalOperatorType.AND - - -class OR(BaseOperator): - token_type = LogicalOperatorType.OR - - -class NOT(BaseOperator): - token_type = LogicalOperatorType.NOT - - def __add__(self, other): - self.items = other - return self - - -class Operator: - def __new__(cls, operator_type, items=None): - if operator_type == LogicalOperatorType.AND: - return AND() - elif operator_type == LogicalOperatorType.OR: - return OR() - elif operator_type == LogicalOperatorType.NOT: - return NOT(items) - return None diff --git a/siem-converter/app/converter/core/operator_types/output.py b/siem-converter/app/converter/core/operator_types/output.py deleted file mode 100644 index cf3d3ca8..00000000 --- a/siem-converter/app/converter/core/operator_types/output.py +++ /dev/null @@ -1,45 +0,0 @@ -import uuid -from dataclasses import dataclass, field -from datetime import datetime -from typing import List - -from app.converter.core.mapping import DEFAULT_MAPPING_NAME -from app.converter.core.models.functions.types import ParsedFunctions - - -class MetaInfoContainer: - def __init__(self, *, - id_: str = None, - title: str = None, - description: str = None, - author: str = None, - date: str = None, - license_: str = None, - severity: str = None, - references: List[str] = None, - tags: List[str] = None, - mitre_attack: List[str] = None, - status: str = None, - false_positives: List[str] = None, - source_mapping_ids: List[str] = None - ) -> None: - self.id = id_ or str(uuid.uuid4()) - self.title = title or "" - self.description = description or "" - self.author = author or "" - self.date = date or datetime.now().date().strftime("%Y-%m-%d") - self.license = license_ or "DRL 1.1" - self.severity = severity or "low" - self.references = references or [] - self.tags = tags or [] - self.mitre_attack = mitre_attack or [] - self.status = status or "stable" - self.false_positives = false_positives or [] - self.source_mapping_ids = source_mapping_ids or [DEFAULT_MAPPING_NAME] - - -@dataclass -class SiemContainer: - query: list - meta_info: MetaInfoContainer - functions: ParsedFunctions = field(default_factory=ParsedFunctions) diff --git a/siem-converter/app/converter/core/operator_types/tokens.py b/siem-converter/app/converter/core/operator_types/tokens.py deleted file mode 100644 index c491fe1a..00000000 --- a/siem-converter/app/converter/core/operator_types/tokens.py +++ /dev/null @@ -1,32 +0,0 @@ -from app.converter.tools.custom_enum import CustomEnum - - -class LogicalOperatorType(CustomEnum): - AND = "and" - OR = "or" - NOT = "not" - - -class OperatorType(CustomEnum): - LT = "<" - LTE = "<=" - GT = ">" - GTE = ">=" - EQ = "=" - NEQ = "!=" - COLON = ":" - CONTAINS = "contains" - STARTSWITH = "startswith" - ENDSWITH = "endswith" - REGEX = "re" - KEYWORD = "keyword" - - -class GroupType(CustomEnum): - L_PAREN = "(" - R_PAREN = ")" - GROUP = "group" - - -class ValidTokens(LogicalOperatorType, OperatorType, GroupType): - pass diff --git a/siem-converter/app/converter/core/parser.py b/siem-converter/app/converter/core/parser.py deleted file mode 100644 index 778cda88..00000000 --- a/siem-converter/app/converter/core/parser.py +++ /dev/null @@ -1,52 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from abc import ABC, abstractmethod -from typing import Dict, List, Tuple - -from app.converter.core.mapping import BasePlatformMappings, SourceMapping -from app.converter.core.models.field import Field -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer -from app.converter.core.tokenizer import QueryTokenizer, TOKEN_TYPE - - -class Parser(ABC): - mappings: BasePlatformMappings = None - tokenizer: QueryTokenizer = None - details: PlatformDetails = None - - @abstractmethod - def _get_meta_info(self, *args, **kwargs) -> MetaInfoContainer: - raise NotImplementedError("Abstract method") - - @abstractmethod - def parse(self, text: str) -> SiemContainer: - raise NotImplementedError("Abstract method") - - def get_tokens_and_source_mappings(self, - query: str, - log_sources: Dict[str, List[str]] - ) -> Tuple[List[TOKEN_TYPE], List[SourceMapping]]: - tokens = self.tokenizer.tokenize(query=query) - field_tokens = self.tokenizer.filter_tokens(tokens, Field) - field_names = [field.source_name for field in field_tokens] - suitable_source_mappings = self.mappings.get_suitable_source_mappings(field_names=field_names, **log_sources) - self.tokenizer.set_field_generic_names_map(field_tokens, suitable_source_mappings, self.mappings) - - return tokens, suitable_source_mappings diff --git a/siem-converter/app/converter/core/parser_cti.py b/siem-converter/app/converter/core/parser_cti.py deleted file mode 100644 index 4d33fe31..00000000 --- a/siem-converter/app/converter/core/parser_cti.py +++ /dev/null @@ -1,134 +0,0 @@ -import re -import ipaddress - -from typing import Optional, List -from pydantic import BaseModel - -from app.converter.core.exceptions.iocs import IocsLimitExceededException, EmptyIOCSException -from app.converter.tools.const import IP_IOC_REGEXP_PATTERN, DOMAIN_IOC_REGEXP_PATTERN, URL_IOC_REGEXP_PATTERN, \ - hash_regexes, IOCType, HashType, IocParsingRule, HASH_MAP - - -class Iocs(BaseModel): - ip: list[str] = [] - url: list[str] = [] - domain: list[str] = [] - hash_dict: dict = {} - - def get_total_count(self) -> int: - hash_len = 0 - for value in self.hash_dict.values(): - hash_len += len(value) - return len(self.ip) + len(self.url) + len(self.domain) + hash_len - - def return_iocs(self) -> dict: - if all(not value for value in [self.ip, self.url, self.domain, self.hash_dict]): - raise EmptyIOCSException() - result = {"DestinationIP": self.ip, "URL": self.url, "Domain": self.domain} - for key, value in self.hash_dict.items(): - result[HASH_MAP[key]] = value - return result - - -class CTIParser: - - def get_iocs_from_string( - self, - string: str, - include_ioc_types: Optional[List[IOCType]] = None, - include_hash_types: Optional[List[HashType]] = None, - exceptions: Optional[List[str]] = None, - ioc_parsing_rules: Optional[List[IocParsingRule]] = None, - limit: Optional[int] = None - ) -> Iocs: - iocs = Iocs() - string = self.replace_dots_hxxp(string, ioc_parsing_rules) - if not include_ioc_types or "ip" in include_ioc_types: - iocs.ip.extend(self._find_all_str_by_regex(string, IP_IOC_REGEXP_PATTERN)) - if not include_ioc_types or "domain" in include_ioc_types: - iocs.domain.extend(self._find_all_str_by_regex(string, DOMAIN_IOC_REGEXP_PATTERN)) - if not include_ioc_types or "url" in include_ioc_types: - iocs.url.extend(self._find_all_str_by_regex(string, URL_IOC_REGEXP_PATTERN)) - if not include_ioc_types or "hash" in include_ioc_types: - if not include_hash_types: - include_hash_types = list(hash_regexes.keys()) - for hash_type in include_hash_types: - iocs.hash_dict[hash_type] = self._find_all_str_by_regex(string, hash_regexes[hash_type]) - iocs = self.remove_duplicates(iocs) - iocs = self.remove_exceptions(iocs, exceptions) - if ioc_parsing_rules is None or "remove_private_and_reserved_ips" in ioc_parsing_rules: - self._exclude_private_and_reserved_ips(iocs) - if limit is not None: - total_count = iocs.get_total_count() - if total_count > limit: - raise IocsLimitExceededException(f"IOCs count {total_count} exceeds limit {limit}.") - return iocs.return_iocs() - - def replace_dots_hxxp(self, string, ioc_parsing_rules): - if ioc_parsing_rules is None or "replace_dots" in ioc_parsing_rules: - string = self._replace_dots(string) - if ioc_parsing_rules is None or "replace_hxxp" in ioc_parsing_rules: - string = self._replace_hxxp(string) - return string - - def remove_duplicates(self, iocs): - iocs.ip = self._remove_duplicates_from_list(iocs.ip) - iocs.domain = self._remove_duplicates_from_list(iocs.domain) - iocs.url = self._remove_duplicates_from_list(iocs.url) - for key, value in iocs.hash_dict.items(): - iocs.hash_dict[key] = self._remove_duplicates_from_list(value) - return iocs - - def remove_exceptions(self, iocs, exceptions=None): - iocs.ip = self._remove_exceptions(iocs.ip, exceptions) - iocs.domain = self._remove_exceptions(iocs.domain, exceptions) - iocs.url = self._remove_exceptions(iocs.url, exceptions) - for key, value in iocs.hash_dict.items(): - iocs.hash_dict[key] = self._remove_exceptions(value, exceptions) - return iocs - - @staticmethod - def _find_all_str_by_regex(s: str, pattern: str) -> list[str]: - return re.findall(pattern, s, re.MULTILINE) - - @staticmethod - def _remove_duplicates_from_list(ls: list) -> list: - return list(dict.fromkeys(ls)) - - @classmethod - def _remove_exceptions( - cls, values: list[str], exceptions: Optional[List[str]] - ) -> list[str]: - if not exceptions: - return values - return [v for v in values if not cls._str_contain_one_of_exceptions(v, exceptions)] - - @staticmethod - def _str_contain_one_of_exceptions(s: str, exceptions: list[str]) -> bool: - for e in exceptions: - if e in s: - return True - return False - - @staticmethod - def _replace_dots(s: str) -> str: - s = s.replace("(.)", ".") - s = s.replace("[.]", ".") - s = s.replace("{.}", ".") - return s - - @staticmethod - def _replace_hxxp(s: str) -> str: - return s.replace("hxxp", "http") - - @classmethod - def _exclude_private_and_reserved_ips(cls, iocs: Iocs) -> None: - iocs.ip = [ip for ip in iocs.ip if not cls._is_private_or_reserved(ip)] - - @staticmethod - def _is_private_or_reserved(ip: str) -> bool: - try: - addr = ipaddress.ip_address(ip) - return addr.is_private or addr.is_reserved - except Exception: - return False diff --git a/siem-converter/app/converter/core/render.py b/siem-converter/app/converter/core/render.py deleted file mode 100644 index cf2c97ba..00000000 --- a/siem-converter/app/converter/core/render.py +++ /dev/null @@ -1,204 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from abc import ABC -from typing import Union, List, Dict - -from app.converter.core.exceptions.core import NotImplementedException, StrictPlatformException -from app.converter.core.exceptions.parser import UnsupportedOperatorException -from app.converter.core.mapping import BasePlatformMappings, SourceMapping, LogSourceSignature, DEFAULT_MAPPING_NAME -from app.converter.core.models.field import Field, Keyword -from app.converter.core.models.functions.types import ParsedFunctions -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer -from app.converter.core.operator_types.tokens import LogicalOperatorType, OperatorType, GroupType - - -class BaseQueryFieldValue(ABC): - details: PlatformDetails = None - - def __init__(self, or_token): - self.field_value = { - OperatorType.EQ: self.equal_modifier, - OperatorType.CONTAINS: self.contains_modifier, - OperatorType.ENDSWITH: self.endswith_modifier, - OperatorType.STARTSWITH: self.startswith_modifier, - OperatorType.REGEX: self.regex_modifier, - OperatorType.KEYWORD: self.keywords - } - self.or_token = f" {or_token} " - - def equal_modifier(self, field, value): - raise NotImplementedException - - def contains_modifier(self, field, value): - raise NotImplementedException - - def endswith_modifier(self, field, value): - raise NotImplementedException - - def startswith_modifier(self, field, value): - raise NotImplementedException - - def regex_modifier(self, field, value): - raise NotImplementedException - - def keywords(self, field, value): - raise NotImplementedException - - def apply_field_value(self, field, operator, value): - if modifier_function := self.field_value.get(operator.token_type): - return modifier_function(field, value) - raise UnsupportedOperatorException(operator.token_type) - - -class BaseQueryRender: - mappings: BasePlatformMappings = None - details: PlatformDetails = None - is_strict_mapping = False - - or_token = "or" - and_token = "and" - not_token = "not" - - group_token = "(%s)" - - field_value_map = BaseQueryFieldValue(or_token=or_token) - - query_pattern = '{table} {query} {functions}' - - comment_symbol: str = None - is_multi_line_comment: bool = False - unsupported_functions_text = 'Unsupported functions were excluded from the result query:' - - def __init__(self): - self.operator_map = { - LogicalOperatorType.AND: f" {self.and_token} ", - LogicalOperatorType.OR: f" {self.or_token} ", - LogicalOperatorType.NOT: f" {self.not_token} ", - } - - def generate_prefix(self, log_source_signature: LogSourceSignature) -> str: - if str(log_source_signature): - return f"{str(log_source_signature)} {self.and_token}" - return "" - - def generate_functions(self, functions: list) -> str: - return "" - - def map_field(self, field: Field, source_mapping: SourceMapping) -> List[str]: - generic_field_name = field.generic_names_map[source_mapping.source_id] - # field can be mapped to corresponding platform field name or list of platform field names - mapped_field = source_mapping.fields_mapping.get_platform_field_name(generic_field_name=generic_field_name) - if not mapped_field and self.is_strict_mapping: - raise StrictPlatformException(field_name=field.source_name, platform_name=self.details.name) - - if isinstance(mapped_field, str): - mapped_field = [mapped_field] - - return mapped_field if mapped_field else [generic_field_name] if generic_field_name else [field.source_name] - - def apply_token(self, - token: Union[Field, Keyword, LogicalOperatorType, GroupType], - source_mapping: SourceMapping) -> str: - if isinstance(token, (Field, Keyword)): - mapped_fields = self.map_field(token, source_mapping) if isinstance(token, Field) else [None] - if len(mapped_fields) > 1: - return self.group_token % self.operator_map[LogicalOperatorType.OR].join([ - self.field_value_map.apply_field_value(field=field, operator=token.operator, value=token.value) - for field in mapped_fields - ]) - - return self.field_value_map.apply_field_value(field=mapped_fields[0], - operator=token.operator, - value=token.value) - elif token.token_type in LogicalOperatorType: - return self.operator_map.get(token.token_type) - return token.token_type - - def generate_query(self, - query: List[Union[Field, Keyword, LogicalOperatorType, GroupType]], - source_mapping: SourceMapping) -> str: - result_values = [] - for token in query: - result_values.append(self.apply_token(token=token, source_mapping=source_mapping)) - return "".join(result_values) - - def finalize_query(self, - prefix: str, - query: str, - functions: str, - meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, - not_supported_functions: list = None) -> str: - query = self.query_pattern.format(prefix=prefix, query=query, functions=functions).strip() - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return query + rendered_not_supported - return query - - def render_not_supported_functions(self, not_supported_functions: list) -> str: - line_template = f"{self.comment_symbol} " if self.comment_symbol and self.is_multi_line_comment else "" - not_supported_functions_str = "\n".join(line_template + func for func in not_supported_functions) - return "\n\n" + self.wrap_with_comment(f"{self.unsupported_functions_text}\n{not_supported_functions_str}") - - def wrap_with_comment(self, value: str) -> str: - return f"{self.comment_symbol} {value}" - - def finalize(self, queries_map: Dict[str, str]) -> str: - unique_queries = set(queries_map.values()) - if len(set(queries_map.values())) == 1: - return next(iter(unique_queries)) - - result = "" - for source_id, query in queries_map.items(): - result = result + self.wrap_with_comment(source_id) + f"\n{query}\n\n" - - return result - - def __get_source_mappings(self, source_mapping_ids: List[str]) -> List[SourceMapping]: - source_mappings = [] - for source_mapping_id in source_mapping_ids: - if source_mapping := self.mappings.get_source_mapping(source_mapping_id): - source_mappings.append(source_mapping) - - if not source_mappings: - source_mappings = [self.mappings.get_source_mapping(DEFAULT_MAPPING_NAME)] - - return source_mappings - - def generate(self, query: list, meta_info: MetaInfoContainer, functions: ParsedFunctions) -> str: - queries_map = {} - source_mappings = self.__get_source_mappings(meta_info.source_mapping_ids) - - for source_mapping in source_mappings: - prefix = self.generate_prefix(source_mapping.log_source_signature) - result = self.generate_query(query=query, source_mapping=source_mapping) - - finalized_query = self.finalize_query( - prefix=prefix, - query=result, - functions=self.generate_functions(functions.functions), - not_supported_functions=functions.not_supported, - meta_info=meta_info, - source_mapping=source_mapping - ) - queries_map[source_mapping.source_id] = finalized_query - - return self.finalize(queries_map) diff --git a/siem-converter/app/converter/core/render_cti.py b/siem-converter/app/converter/core/render_cti.py deleted file mode 100644 index bf8d2c68..00000000 --- a/siem-converter/app/converter/core/render_cti.py +++ /dev/null @@ -1,66 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from typing import List - -from app.converter.core.models.iocs import IocsChunkValue - - -class RenderCTI: - - # EXAMPLE OF A BACKEND ATTRIBUTES - data_map: str = "{key} =~ '{value}'" - or_operator: str = " or " - group_or_operator: str = " or " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "union * | where ({result})\n" - final_result_for_one: str = "union * | where {result}\n" - default_mapping = None - - def create_field_value(self, field: str, value: str, generic_field: str): - return self.data_map.format(key=field, value=value) - - def render(self, data: List[List[IocsChunkValue]]) -> list[str]: - final_result = [] - for iocs_chunk in data: - data_values = self.collect_data_values(iocs_chunk) - if len(data_values) > 1: - final_result.append(self.final_result_for_many.format(result=self.group_or_operator.join(data_values))) - else: - final_result.append(self.final_result_for_one.format(result=data_values[0])) - return final_result - - def collect_data_values(self, chunk): - data_values = [] - key_chunk = [] - processing_key = chunk[0].platform_field - for value in chunk: - if processing_key != value.platform_field: - data_values.append(self.or_group.format(or_group=self.or_operator.join(key_chunk), - processing_key=processing_key)) - key_chunk = [] - processing_key = value.platform_field - key_chunk.append(self.create_field_value(field=value.platform_field, - value=value.value, - generic_field=value.generic_field)) - if key_chunk: - data_values.append( - self.or_group.format(or_group=self.or_operator.join(key_chunk), processing_key=processing_key)) - return data_values diff --git a/siem-converter/app/converter/core/tokenizer.py b/siem-converter/app/converter/core/tokenizer.py deleted file mode 100644 index 6dab969c..00000000 --- a/siem-converter/app/converter/core/tokenizer.py +++ /dev/null @@ -1,238 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Tuple, Union, List, Any, Optional, Type - -from app.converter.core.exceptions.parser import ( - UnsupportedOperatorException, - TokenizerGeneralException, - QueryParenthesesException -) -from app.converter.core.mapping import SourceMapping, DEFAULT_MAPPING_NAME, BasePlatformMappings -from app.converter.core.models.field import Field, Keyword -from app.converter.core.models.identifier import Identifier -from app.converter.core.models.group import GroupType -from app.converter.core.operator_types.tokens import OperatorType -from app.converter.tools.utils import get_match_group - -TOKEN_TYPE = Union[Field, Keyword, Identifier] - - -class QueryTokenizer: - field_pattern = r"(?P[a-zA-Z\._\-]+)" - operator_pattern = r"\s?(?Pand|or|not|AND|OR|NOT)\s?" - field_value_pattern = r"""^___field___\s*___match_operator___\s*___value___""" - match_operator_pattern = r"""(?:___field___\s?(?Pilike|contains|endswith|startswith|in|==|=|=~|!=|:|\:))\s?""" - base_value_pattern = r"(?:___value_pattern___)" - _value_pattern = r"""(?:\"|\')*(?P[:a-zA-Z\*0-9=+%#\-_\/\\'\,.&^@!\(\s]*)(?:\*|\'|\"|\s|\$)*""" - value_pattern = base_value_pattern.replace('___value_pattern___', _value_pattern) - multi_value_pattern = r"""\((?P[:a-zA-Z\"\*0-9=+%#\-_\/\\'\,.&^@!\(\s]*)\)""" - keyword_pattern = None # do not modify, use subclasses to define this attribute - - multi_value_operators = tuple() - multi_value_delimiter = "," - wildcard_symbol = None - - operators_map = { - "=": OperatorType.EQ, - "in": OperatorType.EQ, - "contains": OperatorType.CONTAINS, - "startswith": OperatorType.STARTSWITH, - "endswith": OperatorType.ENDSWITH - } - - def __init_subclass__(cls, **kwargs): - cls.value_pattern = cls.base_value_pattern.replace('___value_pattern___', cls._value_pattern) - - def map_operator(self, operator: str) -> str: - try: - return self.operators_map[operator.lower()] - except KeyError as e: - raise UnsupportedOperatorException(operator) - - def search_field(self, query): - field_search = re.search(self.field_pattern, query) - if field_search is None: - raise TokenizerGeneralException(error=f"Field couldn't be found in query part: {query}") - field = field_search.group("field_name") - return field - - def escape_field_name(self, field_name): - return field_name.replace(".", r"\.") - - def search_match_operator(self, query, field_name) -> str: - field_name = self.escape_field_name(field_name) - match_operator_pattern = self.match_operator_pattern.replace("___field___", field_name) - match_operator_regex = re.compile(match_operator_pattern, re.IGNORECASE) - match_operator_search = re.search(match_operator_regex, query) - if match_operator_search is None: - raise TokenizerGeneralException(error=f"Operator couldn't be found in query part: {query}") - match_operator = match_operator_search.group("match_operator") - match_operator = match_operator.strip(" ") - return match_operator - - def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> Tuple[str, Any]: - return operator, get_match_group(match, group_name='value') - - @staticmethod - def clean_multi_value(value: Union[int, str]) -> Union[int, str]: - if isinstance(value, str): - value = value.strip(" ") - if value.startswith("'") and value.endswith("'"): - value = value[1:-1] - elif value.startswith('"') and value.endswith('"'): - value = value[1:-1] - - return value - - def search_value(self, query: str, operator: str, field_name: str) -> Tuple[str, str, Any]: - field_value_pattern = self.get_field_value_pattern(operator, field_name) - value_pattern = self.value_pattern - is_multi = False - if operator.lower() in self.multi_value_operators: - value_pattern = self.multi_value_pattern - is_multi = True - - field_value_pattern = field_value_pattern.replace("___value___", value_pattern) - field_value_regex = re.compile(field_value_pattern, re.IGNORECASE) - field_value_search = re.match(field_value_regex, query) - if field_value_search is None: - raise TokenizerGeneralException(error=f"Value couldn't be found in query part: {query}") - - operator, value = self.get_operator_and_value(field_value_search, self.map_operator(operator)) - value = [self.clean_multi_value(v) for v in value.split(",")] if is_multi else value - pos = field_value_search.end() - return query[pos:], operator, value - - def search_keyword(self, query: str) -> Tuple[Keyword, str]: - keyword_search = re.search(self.keyword_pattern, query) - _, value = self.get_operator_and_value(keyword_search) - keyword = Keyword(value=value) - pos = keyword_search.end() - return keyword, query[pos:] - - def get_field_value_pattern(self, operator, field_name): - field_value_pattern = self.field_value_pattern.replace("___field___", self.escape_field_name(field_name)) - return field_value_pattern.replace("___match_operator___", operator) - - @staticmethod - def _clean_value(value: str, wildcard_symbol: str) -> str: - return value.strip(wildcard_symbol) if wildcard_symbol else value - - @staticmethod - def __get_operator_token(value: str, operator: str, wildcard_symbol: str) -> Identifier: - if not wildcard_symbol: - return Identifier(token_type=operator) - - if operator == OperatorType.REGEX: - if not (value.startswith(wildcard_symbol) and value.endswith(wildcard_symbol)): - return Identifier(token_type=OperatorType.REGEX) - - if value.startswith(wildcard_symbol) and value.endswith(wildcard_symbol): - return Identifier(token_type=OperatorType.CONTAINS) - elif value.startswith(wildcard_symbol): - return Identifier(token_type=OperatorType.ENDSWITH) - elif value.endswith(wildcard_symbol): - return Identifier(token_type=OperatorType.STARTSWITH) - else: - return Identifier(token_type=operator) - - def process_value_wildcard_symbols(self, - value: Union[List[str], str], - operator: str, - wildcard_symbol: Optional[str]) -> Tuple[Union[List[str], str], Identifier]: - if isinstance(value, list): - op = self.__get_operator_token(value=value[0], operator=operator, wildcard_symbol=wildcard_symbol) - return [self._clean_value(value=v, wildcard_symbol=wildcard_symbol) for v in value], op - - op = self.__get_operator_token(value=value, operator=operator, wildcard_symbol=wildcard_symbol) - return self._clean_value(value, wildcard_symbol), op - - @staticmethod - def create_field(field_name: str, operator: Identifier, value: Union[str, List]) -> Field: - return Field(operator=operator, value=value, source_name=field_name) - - def search_field_value(self, query): - field_name = self.search_field(query) - operator = self.search_match_operator(query, field_name) - query, operator, value = self.search_value(query=query, operator=operator, field_name=field_name) - value, operator_token = self.process_value_wildcard_symbols(value=value, - operator=operator, - wildcard_symbol=self.wildcard_symbol) - field = self.create_field(field_name=field_name, operator=operator_token, value=value) - return field, query - - def __get_identifier(self, query: str) -> (list, str): - query = query.strip("\n").strip(" ").strip("\n") - if query.startswith(GroupType.L_PAREN): - return Identifier(token_type=GroupType.L_PAREN), query[1:] - elif query.startswith(GroupType.R_PAREN): - return Identifier(token_type=GroupType.R_PAREN), query[1:] - elif operator_search := re.match(self.operator_pattern, query): - operator = operator_search.group("operator") - pos = operator_search.end() - return Identifier(token_type=operator.lower()), query[pos:] - elif self.keyword_pattern and re.match(self.keyword_pattern, query): - return self.search_keyword(query) - else: - return self.search_field_value(query) - - @staticmethod - def _validate_parentheses(tokens): - parentheses = [] - for token in tokens: - if isinstance(token, Identifier) and token.token_type in (GroupType.L_PAREN, GroupType.R_PAREN): - if token.token_type == GroupType.L_PAREN: - parentheses.append(token) - elif not parentheses or parentheses[-1].token_type == GroupType.R_PAREN: - raise QueryParenthesesException() - else: - parentheses.pop() - if parentheses: - raise QueryParenthesesException() - return True - - def tokenize(self, query: str) -> List[Union[Field, Keyword, Identifier]]: - tokenized = [] - while query: - identifier, query = self.__get_identifier(query=query) - tokenized.append(identifier) - self._validate_parentheses(tokenized) - return tokenized - - @staticmethod - def filter_tokens(tokens: List[TOKEN_TYPE], - token_type: Union[Type[Field], Type[Keyword], Type[Identifier]]) -> List[TOKEN_TYPE]: - return [token for token in tokens if isinstance(token, token_type)] - - @staticmethod - def set_field_generic_names_map(tokens: List[Field], - source_mappings: List[SourceMapping], - platform_mappings: BasePlatformMappings) -> None: - for token in tokens: - generic_names_map = { - source_mapping.source_id: source_mapping.fields_mapping.get_generic_field_name(token.source_name) - for source_mapping in source_mappings - } - if DEFAULT_MAPPING_NAME not in generic_names_map: - default_source_mapping = platform_mappings.get_source_mapping(DEFAULT_MAPPING_NAME) - fields_mapping = default_source_mapping.fields_mapping - generic_names_map[DEFAULT_MAPPING_NAME] = fields_mapping.get_generic_field_name(token.source_name) - - token.generic_names_map = generic_names_map diff --git a/siem-converter/app/converter/cti_converter.py b/siem-converter/app/converter/cti_converter.py deleted file mode 100644 index a0070b1f..00000000 --- a/siem-converter/app/converter/cti_converter.py +++ /dev/null @@ -1,82 +0,0 @@ -import logging -from typing import Dict, List - -from app.converter.const import CTI_MIN_LIMIT_QUERY -from app.converter.core.models.iocs import IocsChunkValue -from app.converter.core.parser_cti import CTIParser, Iocs -from app.converter.core.render_cti import RenderCTI -from app.converter.managers import RenderCTIManager, render_cti_manager -from app.converter.tools.decorators import handle_translation_exceptions -from app.models.ioc_translation import CTIPlatform - - -class CTIConverter: - renders: RenderCTIManager = render_cti_manager - - def __init__(self): - self.logger = logging.getLogger("cti_converter") - self.parser = CTIParser() - - def _get_render_mapping(self, platform: CTIPlatform, include_source_ip: bool = False) -> Dict[str, str]: - return self.renders.get(platform.name).default_mapping - - @handle_translation_exceptions - def __parse_iocs_from_string(self, text: str, include_ioc_types: list = None, include_hash_types: list = None, - exceptions: list = None, ioc_parsing_rules: list = None) -> Iocs: - return self.parser.get_iocs_from_string(string=text, - include_ioc_types=include_ioc_types, - include_hash_types=include_hash_types, - exceptions=exceptions, - ioc_parsing_rules=ioc_parsing_rules, - limit=CTI_MIN_LIMIT_QUERY) - - @handle_translation_exceptions - def __render_translation(self, parsed_data: dict, platform_data: CTIPlatform, iocs_per_query: int, - include_source_ip: bool = False) -> List[str]: - mapping = self._get_render_mapping(platform=platform_data, include_source_ip=include_source_ip) - platform = self.renders.get(platform_data.name) - platform_generation = self.generate(data=parsed_data, platform=platform, iocs_per_query=iocs_per_query, - mapping=mapping) - return platform_generation - - def convert(self, text: str, - platform_data: CTIPlatform, - iocs_per_query: int = 25, - include_ioc_types: list = None, - include_hash_types: list = None, - exceptions: list = None, - ioc_parsing_rules: list = None, - include_source_ip: bool = False) -> (bool, List[str]): - status, parsed_data = self.__parse_iocs_from_string(text=text, - include_ioc_types=include_ioc_types, - include_hash_types=include_hash_types, - exceptions=exceptions, - ioc_parsing_rules=ioc_parsing_rules) - if status: - return self.__render_translation(parsed_data=parsed_data, - include_source_ip=include_source_ip, - platform_data=platform_data, - iocs_per_query=iocs_per_query - ) - return status, parsed_data - - @staticmethod - def _get_iocs_chunk(chunks_size: int, data: Dict[str, List[str]], - mapping: Dict[str, str]) -> List[List[IocsChunkValue]]: - result = [] - for generic_field, iocs_list in data.items(): - for ioc in iocs_list: - if mapping.get(generic_field): - result.append(IocsChunkValue(generic_field=generic_field, - platform_field=mapping[generic_field], - value=ioc)) - return [result[i:i + chunks_size] for i in range(0, len(result), chunks_size)] - - def generate(self, platform: RenderCTI, iocs_per_query, data: Dict[str, List[str]], - mapping: Dict[str, str]) -> List[str]: - chunked_iocs = self._get_iocs_chunk(chunks_size=iocs_per_query, data=data, mapping=mapping) - return platform.render(chunked_iocs) - - @classmethod - def get_renders(cls) -> list: - return cls.renders.get_platforms_details diff --git a/siem-converter/app/converter/managers.py b/siem-converter/app/converter/managers.py deleted file mode 100644 index 62bdd435..00000000 --- a/siem-converter/app/converter/managers.py +++ /dev/null @@ -1,58 +0,0 @@ -from abc import ABC - -from app.converter.platforms import __ALL_PARSERS as PARSERS -from app.converter.platforms import __ALL_RENDERS as RENDERS -from app.converter.platforms import __ALL_RENDERS_CTI as RENDERS_CTI -from app.converter.core.exceptions.core import UnsupportedRootAParser -from app.models.translation import ConvertorPlatform - - -class Manager(ABC): - platforms_class = tuple() - - @property - def platforms(self) -> dict: - return {platform.details.siem_type: platform for platform in self.platforms_class} - - def get(self, siem): - if platform := self.platforms.get(siem): - return platform - raise UnsupportedRootAParser(parser=siem) - - def all_platforms(self): - return list(self.platforms) - - @property - def get_platforms_details(self): - platforms = [ - ConvertorPlatform( - id=platform.details.siem_type, - name=platform.details.name, - code=platform.details.siem_type, - group_name=platform.details.group_name, - group_id=platform.details.group_id, - platform_name=platform.details.platform_name, - platform_id=platform.details.siem_type, - alt_platform_name=platform.details.alt_platform_name, - alt_platform=platform.details.alt_platform, - first_choice=platform.details.first_choice, - ) for platform in self.platforms_class - ] - return sorted(platforms, key=lambda platform: platform.group_name) - - -class RenderManager(Manager): - platforms_class = RENDERS - - -class ParserManager(Manager): - platforms_class = PARSERS - - -class RenderCTIManager(Manager): - platforms_class = RENDERS_CTI - - -parser_manager = ParserManager() -render_manager = RenderManager() -render_cti_manager = RenderCTIManager() diff --git a/siem-converter/app/converter/mappings/platforms/athena/aws_cloudtrail.yml b/siem-converter/app/converter/mappings/platforms/athena/aws_cloudtrail.yml deleted file mode 100644 index 2b9e924b..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/aws_cloudtrail.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Athena -source: aws_cloudtrail -description: Text that describe current mapping - -log_source: - table: - - cloudtrail_logs_* - -default_log_source: - table: cloudtrail_logs_* - -field_mapping: - eventSource: api.service.name - eventName: api.operation - AdditionalEventData: additionalEventData - additionalEventData.MFAUsed: JSON_VALUE(unmapped['additionalEventData'], 'strict$.MFAUsed') - requestParameters.attribute: unmapped['eventType'] - responseElements.pendingModifiedValues.masterUserPassword: unmapped['responseElements'] - responseElements.publiclyAccessible: JSON_VALUE(unmapped['responseElements'], 'strict$.ConsoleLogin') - userIdentity.principalId: http_request.user_agent - userIdentity.sessionContext.sessionIssuer.type: unmapped['userIdentity_sessionContext_sessionIssuer_arn'] - userIdentity.type: unmapped['userIdentity_sessionContext_sessionIssuer_principalId'] - userIdentity.userName: unmapped['userIdentity_sessionContext_sessionIssuer_type'] diff --git a/siem-converter/app/converter/mappings/platforms/athena/default.yml b/siem-converter/app/converter/mappings/platforms/athena/default.yml deleted file mode 100644 index 2ba77248..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/default.yml +++ /dev/null @@ -1,7 +0,0 @@ -platform: Athena -source: default -description: Text that describe current mapping - - -default_log_source: - table: eventlog \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/athena/linux_file_event.yml b/siem-converter/app/converter/mappings/platforms/athena/linux_file_event.yml deleted file mode 100644 index 715ec496..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/linux_file_event.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: Athena -source: linux_file_event -description: Text that describe current mapping - -default_log_source: - table: eventlog - -field_mapping: - Image: actor.process.file.path - TargetFilename : file.name \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/athena/linux_process_creation.yml b/siem-converter/app/converter/mappings/platforms/athena/linux_process_creation.yml deleted file mode 100644 index e0c63cd4..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/linux_process_creation.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Athena -source: linux_process_creation -description: Text that describe current mapping - - -default_log_source: - table: eventlog - - -field_mapping: - CommandLine: process.cmd_line - Image: process.file.path - ParentCommandLine: - - process.parent_process.cmd_line - - actor.process.cmd_line - ParentImage: - - process.parent_process.file.path - - actor.process.file.path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/athena/macos_file_event.yml b/siem-converter/app/converter/mappings/platforms/athena/macos_file_event.yml deleted file mode 100644 index b97baec9..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/macos_file_event.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: Athena -source: macos_file_event -description: Text that describe current mapping - -default_log_source: - table: eventlog - -field_mapping: - Image: actor.process.file.path - TargetFilename : file.name \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/athena/macos_process_creation.yml b/siem-converter/app/converter/mappings/platforms/athena/macos_process_creation.yml deleted file mode 100644 index 0c08242f..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/macos_process_creation.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Athena -source: macos_process_creation -description: Text that describe current mapping - - -default_log_source: - table: eventlog - -field_mapping: - CommandLine: process.cmd_line - Image: process.file.path - ParentCommandLine: process.parent_process.cmd_line;actor.process.cmd_line - ParentImage: process.parent_process.file.path;actor.process.file.path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/athena/windows_file_event.yml b/siem-converter/app/converter/mappings/platforms/athena/windows_file_event.yml deleted file mode 100644 index ad4f7674..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/windows_file_event.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Microsoft Sentinel -source: windows_file_event -description: Text that describe current mapping - -default_log_source: - table: eventlog - -field_mapping: - CreationUtcTime: file.created_time - FileName: file.name - Image: actor.process.file.path - OriginalFilename: file.name - TargetFilename: file.name - User: - - actor.user.name - - actor.process.user.name \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/athena/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/athena/windows_image_load.yml deleted file mode 100644 index bc6c62fb..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/windows_image_load.yml +++ /dev/null @@ -1,11 +0,0 @@ -platform: Athena -source: windows_image_load -description: Text that describe current mapping - -default_log_source: - table: eventlog - -field_mapping: - Image: actor.process.file.path - ImageLoaded: module.file.path - OriginalFileName: module.file.path diff --git a/siem-converter/app/converter/mappings/platforms/athena/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/athena/windows_process_creation.yml deleted file mode 100644 index c4af2a94..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/windows_process_creation.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Athena -source: windows_process_creation -description: Text that describe current mapping - -default_log_source: - table: eventlog - -field_mapping: - CommandLine: process.cmd_line - Hashes: - - proces.file.fingerprints.value - - actor.proces.file.fingerprints.value - Image: process.file.path - IntegrityLevel: process.integrity - ParentCommandLine: - - process.parent_process.cmd_line - - actor.process.cmd_line - ParentImage: - - process.parent_process.file.path - - actor.process.file.path - ParentUser: process.parent_process.user - Product: - - process.file.product.name - - process.file.product.vendor_name - User: process.user.name \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/athena/windows_registry_event.yml b/siem-converter/app/converter/mappings/platforms/athena/windows_registry_event.yml deleted file mode 100644 index 0843fcc3..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/windows_registry_event.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: Athena -source: windows_registry_event -description: Text that describe current mapping - -default_log_source: - table: eventlog - -field_mapping: - TargetObject: - - reg_key.path - - reg_value.name - - reg_value.path - Image: actor.process.file.path - User: actor.process.user.name \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/athena/windows_security.yml b/siem-converter/app/converter/mappings/platforms/athena/windows_security.yml deleted file mode 100644 index 37c81abc..00000000 --- a/siem-converter/app/converter/mappings/platforms/athena/windows_security.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Athena -source: windows_security -description: Text that describe current mapping - -default_log_source: - table: eventlog - -field_mapping: - AccessMask: unmapped.Access Mask - ObjectName: unmapped.win_resource.name - ObjectType: unmapped.win_resource.type - CommandLine: process.cmd_line - PrivilegeList: unmapped.Service Request Information.Privileges - ProcessName: actor.process.file.path - SubjectUserName: actor.user.name - Service: unmapped.win_resource.name \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/default.yml b/siem-converter/app/converter/mappings/platforms/chronicle/default.yml deleted file mode 100644 index 87878bf6..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/default.yml +++ /dev/null @@ -1,3 +0,0 @@ -platform: Chronicle -source: default -description: Text that describe current mapping \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_create_remote_thread.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_create_remote_thread.yml deleted file mode 100644 index 2cf5546a..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_create_remote_thread.yml +++ /dev/null @@ -1,11 +0,0 @@ -platform: Chronicle -source: windows_create_remote_thread -description: Text that describe current mapping - - -field_mapping: - SourceImage: principal.process.file.full_path - TargetImage: target.process.file.full_path - StartModule: target.resource.name - StartAddress: target.resource.name - StartFunction: target.resource.name \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_dns_query.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_dns_query.yml deleted file mode 100644 index ec1fc703..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_dns_query.yml +++ /dev/null @@ -1,9 +0,0 @@ -platform: Chronicle -source: windows_dns_query -description: Text that describe current mapping - - -field_mapping: - Image: principal.process.file.full_path - QueryName: network.dns.questions - QueryResults: network.dns.answers.data \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_file_event.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_file_event.yml deleted file mode 100644 index 1d441da9..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_file_event.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: Chronicle -source: windows_file_event -description: Text that describe current mapping - - -field_mapping: - CreationUtcTime: target.resource.name - Image: principal.process.file.full_path - PreviousCreationUtcTime: target.resource.attribute.labels.value - TargetFilename: target.file.full_path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_image_load.yml deleted file mode 100644 index 464a8115..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_image_load.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Chronicle -source: windows_image_load -description: Text that describe current mapping - - -field_mapping: - Image: principal.process.file.full_path - ImageLoaded: - - target.process.file.full_path - - principal.process.file.full_path - SignatureStatus: target.resource.attribute.labels[2].value - Signed: target.resource.attribute.labels[0].value - Hashes: - - target.process.file.md5 - - target.process.file.sha256;target.process.file.sha1 \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_network_connection.yml deleted file mode 100644 index 6ed52541..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_network_connection.yml +++ /dev/null @@ -1,12 +0,0 @@ -platform: Chronicle -source: windows_network_connection -description: Text that describe current mapping - - -field_mapping: - Image: principal.process.file.full_path - DestinationHostname: target.hostname - DestinationIp: target.ip - DestinationPort: target.port - SourceIp: principal.ip - SourcePort: principal.port diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_pipe_created.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_pipe_created.yml deleted file mode 100644 index ddab5077..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_pipe_created.yml +++ /dev/null @@ -1,7 +0,0 @@ -platform: Chronicle -source: windows_pipe_created -description: Text that describe current mapping - -field_mapping: - PipeName: target.resource.name - Image: target.process.file.full_path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_process_access.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_process_access.yml deleted file mode 100644 index 87c4993f..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_process_access.yml +++ /dev/null @@ -1,8 +0,0 @@ -platform: Chronicle -source: windows_process_access -description: Text that describe current mapping - -field_mapping: - TargetImage: target.process.file.full_path - SourceImage: principal.process.file.full_path - GrantedAccess: target.resource.name \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_process_creation.yml deleted file mode 100644 index f9fb10fd..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_process_creation.yml +++ /dev/null @@ -1,12 +0,0 @@ -platform: Chronicle -source: windows_process_creation -description: Text that describe current mapping - - -field_mapping: - CommandLine: target.process.command_line - CurrentDirectory: src.file.full_path - Hashes: target.process.file.md5 - Image: target.process.file.full_path - ParentCommandLine: principal.process.command_line - ParentImage: principal.process.file.full_path diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_registry_event.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_registry_event.yml deleted file mode 100644 index 16952fd3..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_registry_event.yml +++ /dev/null @@ -1,9 +0,0 @@ -platform: Chronicle -source: windows_registry_event -description: Text that describe current mapping - - -field_mapping: - TargetObject: target.registry.registry_key - Image: principal.process.file.full_path - Details: target.registry.registry_value_data \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_security.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_security.yml deleted file mode 100644 index 63063339..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_security.yml +++ /dev/null @@ -1,12 +0,0 @@ -platform: Chronicle -source: windows_security -description: Text that describe current mapping - - -field_mapping: - EventID: metadata.product_event_type - AccessMask: security_result.about.resource.name - AccountName: principal.user.role_name - AuthenticationPackageName: security_result.about.resource.name - ComputerName: principal.hostname - GrantedAccess: resource.name \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/chronicle/windows_sysmon.yml b/siem-converter/app/converter/mappings/platforms/chronicle/windows_sysmon.yml deleted file mode 100644 index 4c2fb75e..00000000 --- a/siem-converter/app/converter/mappings/platforms/chronicle/windows_sysmon.yml +++ /dev/null @@ -1,33 +0,0 @@ -platform: Chronicle -source: windows_sysmon -description: Text that describe current mapping - - -field_mapping: - CommandLine: target.process.command_line - Image: target.process.file.full_path - ParentImage: principal.process.file.full_path - EventID: metadata.product_event_type - CallTrace: principal.process.file.full_path - Description: metadata.description - DestinationHostname: target.hostname - DestinationIp: target.ip - DestinationPort: target.port - ParentCommandLine: principal.process.command_line - Protocol: network.ip_protocol - RuleName: security_result.rule_name - SourceHostname: principal.hostname - SourceIp: principal.ip - SourcePort: principal.port - TargetFilename: target.file.full_path - TargetObject: target.registry.registry_key - Details: target.registry.registry_value_data - QueryName: network.dns.questions - QueryResults: network.dns.answers - PipeName: target.resource.name - ImageLoaded: principal.process.file.full_path - Imphash: target.process.file.file_metadata.pe.import_hash - SourceImage: principal.process.file.full_path - StartModule: target.resource.name - TargetImage: target.process.file.full_path - StartFunction: ScriptBlockText diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/default.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/default.yml deleted file mode 100644 index c9ae3511..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/default.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: Crowdstrike -source: default -description: Text that describe current mapping - -log_source: - event_simpleName: - - ProcessRollup2 - -default_log_source: - event_simpleName: ProcessRollup2 diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/linux_dns_query.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/linux_dns_query.yml deleted file mode 100644 index 2a7e16e1..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/linux_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Crowdstrike -source: linux_dns_query -description: Text that describe current mapping - -log_source: - event_simpleName: [DnsRequest] - -default_log_source: - event_simpleName: DnsRequest - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/linux_network_connection.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/linux_network_connection.yml deleted file mode 100644 index 9a37c725..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/linux_network_connection.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: Crowdstrike -source: linux_network_connection -description: Text that describe current mapping - -log_source: - event_simpleName: [NetworkConnectIP4] - -default_log_source: - event_simpleName: NetworkConnectIP4 - -field_mapping: - CommandLine: CommandLine - Image: Image - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationPort: DestinationPort - SourceIp: SourceIp - SourcePort: SourcePort - ParentImage: ParentImage \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/linux_process_creation.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/linux_process_creation.yml deleted file mode 100644 index 3e679dbf..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/linux_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Crowdstrike -source: linux_process_creation -description: Text that describe current mapping - -log_source: - event_simpleName: [ProcessRollup2] - -default_log_source: - event_simpleName: ProcessRollup2 - -field_mapping: - CommandLine: CommandLine - Image: Image - ParentCommandLine: ParentCommandLine - ParentImage: ParentImage diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/macos_dns_query.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/macos_dns_query.yml deleted file mode 100644 index 3420d9ec..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/macos_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Crowdstrike -source: macos_dns_query -description: Text that describe current mapping - -log_source: - event_simpleName: [DnsRequest] - -default_log_source: - event_simpleName: DnsRequest - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/macos_network_connection.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/macos_network_connection.yml deleted file mode 100644 index 4677c38b..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/macos_network_connection.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: Crowdstrike -source: macos_network_connection -description: Text that describe current mapping - -log_source: - event_simpleName: [NetworkConnectIP4] - -default_log_source: - event_simpleName: NetworkConnectIP4 - -field_mapping: - CommandLine: CommandLine - Image: Image - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationPort: DestinationPort - SourceIp: SourceIp - SourcePort: SourcePort - ParentImage: ParentImage \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/macos_process_creation.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/macos_process_creation.yml deleted file mode 100644 index fe8163bc..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/macos_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Crowdstrike -source: macos_process_creation -description: Text that describe current mapping - -log_source: - event_simpleName: [ProcessRollup2] - -default_log_source: - event_simpleName: ProcessRollup2 - -field_mapping: - CommandLine: CommandLine - Image: Image - ParentCommandLine: ParentCommandLine - ParentImage: ParentImage diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_dns_query.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_dns_query.yml deleted file mode 100644 index 6ad67d95..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Crowdstrike -source: windows_dns_query -description: Text that describe current mapping - -log_source: - event_simpleName: [DnsRequest] - -default_log_source: - event_simpleName: DnsRequest - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_driver_load.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_driver_load.yml deleted file mode 100644 index 0e381939..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_driver_load.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: Crowdstrike -source: windows_driver_load -description: Text that describe current mapping - -log_source: - event_simpleName: [DriverLoad] - -default_log_source: - event_simpleName: DriverLoad - -field_mapping: - ImageLoaded: ImageLoaded - ImagePath: ImagePath - Hashes: Hashes \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_image_load.yml deleted file mode 100644 index 276844f3..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_image_load.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Crowdstrike -source: windows_image_load -description: Text that describe current mapping - -log_source: - event_simpleName: [LoadImage] - -default_log_source: - event_simpleName: LoadImage - -field_mapping: - Image: Image - ImageLoaded: ImageLoaded - SignatureStatus: SignatureStatus - OriginalFileName: OriginalFileName - Signed: Signed \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_network_connection.yml deleted file mode 100644 index cc5cc3ce..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_network_connection.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Crowdstrike -source: windows_network_connection -description: Text that describe current mapping - -log_source: - event_simpleName: [NetworkConnectIP4] - -default_log_source: - event_simpleName: NetworkConnectIP4 - -field_mapping: - Image: Image - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationPort: DestinationPort - SourceIp: SourceIp - SourcePort: SourcePort - Initiated: Initiated \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_process_creation.yml deleted file mode 100644 index 4484830a..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_process_creation.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: Crowdstrike -source: windows_process_creation -description: Text that describe current mapping - -log_source: - event_simpleName: [ProcessRollup2] - -default_log_source: - event_simpleName: ProcessRollup2 - -field_mapping: - CommandLine: CommandLine - CurrentDirectory: CurrentDirectory - Hashes: Hashes - Image: Image - IntegrityLevel: IntegrityLevel - ParentCommandLine: ParentCommandLine - ParentImage: ParentImage - ParentUser: ParentUser - Product: Product - User: User diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_registry_event.yml b/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_registry_event.yml deleted file mode 100644 index 16d57e28..00000000 --- a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_registry_event.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Crowdstrike -source: windows_registry_event -description: Text that describe current mapping - -log_source: - event_simpleName: [RegistryOperationDetectInfo] - -default_log_source: - event_simpleName: RegistryOperationDetectInfo - -field_mapping: - TargetObject: TargetObject - Image: Image - Details: Details - EventType: EventType - User: User diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/aws_cloudtrail.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/aws_cloudtrail.yml deleted file mode 100644 index 3520a860..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/aws_cloudtrail.yml +++ /dev/null @@ -1,35 +0,0 @@ -platform: ElasticSearch -source: aws_cloudtrail -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - eventSource: event.provider - eventName: event.action - AdditionalEventData: AdditionalEventData - additionalEventData.MFAUsed: additionalEventData.MFAUsed - errorCode: errorCode - errorMessage: errorMessage - eventType: eventType - requestParameters: requestParameters - requestParameters.attribute: requestParameters.attribute - requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP - requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort - requestParameters.userData: requestParameters.userData - responseElements: responseElements - responseElements.ConsoleLogin: responseElements.ConsoleLogin - responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword - responseElements.publiclyAccessible: responseElements.publiclyAccessible - status: status - terminatingRuleId: terminatingRuleId - userAgent: userAgent - userIdentity.arn: userIdentity.arn - userIdentity.principalId: userIdentity.principalId - userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type - userIdentity.type: userIdentity.type - userIdentity.userName: userIdentity.userName diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/aws_eks.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/aws_eks.yml deleted file mode 100644 index cbddc9c1..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/aws_eks.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: ElasticSearch -source: aws_eks -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision - annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy - aws_node_type: aws_node_type - objectRef.namespace: objectRef.namespace - objectRef.resource: objectRef.resource - objectRef.subresource: objectRef.subresource - requestObject.rules.resources: requestObject.rules.resources - requestObject.rules.verbs: requestObject.rules.verbs - requestObject.spec.containers.image: requestObject.spec.containers.image - requestURI: requestURI - stage: stage - user.groups: user.groups - user.username: user.username - verb: verb diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_AzureDiagnostics.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_AzureDiagnostics.yml deleted file mode 100644 index 01805a1b..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_AzureDiagnostics.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: ElasticSearch -source: azure_AzureDiagnostics -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ResultDescription: ResultDescription - Category: Category \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_BehaviorAnalytics.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_BehaviorAnalytics.yml deleted file mode 100644 index 93b4ccc5..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_BehaviorAnalytics.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: ElasticSearch -source: azure_BehaviorAnalytics -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ActionType: ActionType - ActivityInsights: ActivityInsights - ActivityType: ActivityType - EventSource: EventSource - DevicesInsights: DevicesInsights - RiskDetail: RiskDetail - UsersInsights: UsersInsights - UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount - UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_aadnoninteractiveusersigninlogs.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_aadnoninteractiveusersigninlogs.yml deleted file mode 100644 index b729dd11..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_aadnoninteractiveusersigninlogs.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: ElasticSearch -source: azure_aadnoninteractiveusersigninlogs -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - UserAgent: UserAgent - Type: Type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_azureactivity.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_azureactivity.yml deleted file mode 100644 index 2e3320dc..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_azureactivity.yml +++ /dev/null @@ -1,22 +0,0 @@ -platform: ElasticSearch -source: azure_azureactivity -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ActivityStatus: ActivityStatus - ActivityStatusValue: ActivityStatusValue - ActivitySubstatusValue: ActivitySubstatusValue - Authorization: Authorization - Category: Category - CategoryValue: CategoryValue - OperationName: OperationName - OperationNameValue: OperationNameValue - ResourceId: ResourceId - ResourceProviderValue: ResourceProviderValue - Type: Type diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_azuread.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_azuread.yml deleted file mode 100644 index 1987e2b6..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_azuread.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: ElasticSearch -source: azure_azuread -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ActivityDisplayName: event.action - Category: azure.auditlogs.properties.category - LoggedByService: azure.auditlogs.properties.logged_by_service - Result: event.outcome - OperationName: OperationName - TargetResources: TargetResources - AADOperationType: AADOperationType - InitiatedBy: InitiatedBy - ResultReason: ResultReason - Status: Status - Status.errorCode: Status.errorCode - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_m365.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_m365.yml deleted file mode 100644 index dff402d0..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_m365.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: ElasticSearch -source: azure_m365 -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ClientInfoString: ClientInfoString - LogonError: LogonError - ModifiedProperties: ModifiedProperties - OfficeObjectId: OfficeObjectId - OfficeWorkload: OfficeWorkload - Operation: Operation - Parameters: Parameters - RecordType: RecordType - ResultStatus: ResultStatus - SourceFileExtension: SourceFileExtension - SourceFileName: SourceFileName - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_signinlogs.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_signinlogs.yml deleted file mode 100644 index babfa1ad..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/azure_signinlogs.yml +++ /dev/null @@ -1,29 +0,0 @@ -platform: ElasticSearch -source: azure_signinlogs -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - AppDisplayName: AppDisplayName - AppId: AppId - AuthenticationRequirement: AuthenticationRequirement - Category: Category - ConditionalAccessStatus: ConditionalAccessStatus - DeviceDetail: DeviceDetail - IsInteractive: IsInteractive - NetworkLocationDetails: NetworkLocationDetails - ResourceDisplayName: ResourceDisplayName - ResourceIdentity: ResourceIdentity - ResultDescription: ResultDescription - ResultType: ResultType - Status.errorCode: Status.errorCode - Status: Status - Status.failureReason: Status.failureReason - TokenIssuerType: TokenIssuerType - UserAgent: UserAgent - UserPrincipalName: UserPrincipalName diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/default.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/default.yml deleted file mode 100644 index 204bf2c7..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/default.yml +++ /dev/null @@ -1,6 +0,0 @@ -platform: ElasticSearch -source: default -description: Text that describe current mapping - -default_log_source: - index: "*" \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/dns.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/dns.yml deleted file mode 100644 index 28d3844d..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/dns.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: ElasticSearch -source: dns -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - dns-query: dns.question.name - parent-domain: - - url.registered_domain - - destination.registered_domain - dns-answer: dns.answers.name - dns-record: dns.question.type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/firewall.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/firewall.yml deleted file mode 100644 index 6c1c2b6d..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/firewall.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: ElasticSearch -source: firewall -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - src-ip: - - source.address - - source.ip - src-port: source.port - dst-ip: - - destination.address - - destination.ip - dst-port: destination.port \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/gcp_gcp.audit.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/gcp_gcp.audit.yml deleted file mode 100644 index d015752e..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/gcp_gcp.audit.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: ElasticSearch -source: gcp_gcp.audit -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - backupConfiguration.enabled: backupConfiguration.enabled - gcp.audit.method_name: gcp.audit.method_name - jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction - jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue - message: message - methodName: methodName - protoPayload.methodName: protoPayload.methodName - protoPayload.request.canIpForward: protoPayload.request.canIpForward - protoPayload.serviceName: protoPayload.serviceName - serviceName: serviceName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/gcp_pubsub.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/gcp_pubsub.yml deleted file mode 100644 index ebe10578..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/gcp_pubsub.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: ElasticSearch -source: gcp_pubsub -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - data.protoPayload.methodName: data.protoPayload.methodName - data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member - data.resource.type: data.resource.type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/linux_auditd.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/linux_auditd.yml deleted file mode 100644 index 3bb79b1c..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/linux_auditd.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: ElasticSearch -source: linux_auditd -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - a0: process.args - a1: process.args - a2: process.args - a3: process.args - exe: process.executable diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/linux_dns_query.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/linux_dns_query.yml deleted file mode 100644 index c5abaca1..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/linux_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: ElasticSearch -source: linux_dns_query -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - Image: process.executable - User: user.name - QueryName: dns.question.name - QueryResults: dns.answers \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/linux_process_creation.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/linux_process_creation.yml deleted file mode 100644 index 0380fc2a..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/linux_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: ElasticSearch -source: linux_process_creation -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - CommandLine: process.command_line - Image: process.executable - ParentCommandLine: process.parent.command_line - ParentImage: process.parent.executable diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/macos_dns_query.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/macos_dns_query.yml deleted file mode 100644 index 8ba450ba..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/macos_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: ElasticSearch -source: macos_dns_query -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - Image: process.executable - User: user.name - QueryName: dns.question.name - QueryResults: dns.answers \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/macos_network_connection.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/macos_network_connection.yml deleted file mode 100644 index 03898bfa..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/macos_network_connection.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: ElasticSearch -source: macos_network_connection -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - Image: process.executable - DestinationHostname: destination.domain - DestinationIp: destination.ip - DestinationPort: destination.port - SourceIp: source.ip - SourcePort: source.port - ParentImage: process.parent.executable diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/macos_process_creation.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/macos_process_creation.yml deleted file mode 100644 index 0fdb2e8e..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/macos_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: ElasticSearch -source: macos_process_creation -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - CommandLine: process.command_line - Image: process.executable - ParentCommandLine: process.parent.command_line - ParentImage: process.parent.executable diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/okta_okta.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/okta_okta.yml deleted file mode 100644 index 26dce70c..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/okta_okta.yml +++ /dev/null @@ -1,33 +0,0 @@ -platform: ElasticSearch -source: okta_okta -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - client.user.id: okta.actor.id - source.user.id: okta.actor.id - User: okta.actor.type - alternateId: okta.actor.alternate_id - client.user.full_name: okta.actor.display_name - source.user.full_name: okta.actor.display_name - related.user: okta.actor.display_name - client.ip: okta.client.ip - source.ip: okta.client.ip - user_agent.original: okta.client.user_agent.raw_user_agent - userAgent.os: okta.client.user_agent.os - userAgent.browser: okta.client.user_agent.browser - client.zone: okta.client.zone - client.device: okta.client.device - client.id: okta.client.id - event.action: okta.event_type - outcome.reason: okta.outcome.reason - event.outcome: okta.event.outcome - client.as.number: okta.security_context.as.number - client.as.organization.name: okta.security_context.as.organization.name - client.domain: okta.security_context.isp - source.domain: okta.security_context.domain diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/proxy.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/proxy.yml deleted file mode 100644 index ca7c701e..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/proxy.yml +++ /dev/null @@ -1,30 +0,0 @@ -platform: ElasticSearch -source: proxy -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - c-uri: url.original - c-useragent: user_agent.original - cs-method: http.request.method - cs-bytes: http.request.body.bytes - cs-cookie-vars: http.cookie_vars - c-uri-extension: url.extension - c-uri-query: url.query - cs-cookie: http.cookie_vars - cs-host: - - url.domain - - destination.domain - - cs-referrer: http.request.referrer - cs-version: http.version - r-dns: - - destination.domain - - url.domain - sc-status: http.response.status_code - post-body: http.post_body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/webserver.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/webserver.yml deleted file mode 100644 index 37644316..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/webserver.yml +++ /dev/null @@ -1,29 +0,0 @@ -platform: ElasticSearch -source: webserver -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - c-uri: url.original - c-useragent: user_agent.original - cs-method: http.request.method - cs-bytes: http.request.body.bytes - cs-cookie-vars: http.cookie_vars - c-uri-extension: url.extension - c-uri-query: url.query - cs-cookie: http.cookie_vars - cs-host: - - url.domain - - destination.domain - cs-referrer: http.request.referrer - cs-version: http.version - r-dns: - - destination.domain - - url.domain - sc-status: http.response.status_code - post-body: http.post_body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_bits_client.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_bits_client.yml deleted file mode 100644 index e5041805..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_bits_client.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: ElasticSearch -source: windows_bits_client -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - LocalName: winlog.event_data.LocalName - EventID: - - event.code - - winlog.event_id - RemoteName: winlog.event_data.RemoteName - processPath: winlog.event_data.processPath diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_dns_query.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_dns_query.yml deleted file mode 100644 index 99c24361..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_dns_query.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: ElasticSearch -source: macos_dns_query -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - Image: process.executable - User: - - user.name - - winlog.user.domain - - winlog.user.name - QueryName: dns.question.name - QueryResults: dns.answers \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_driver_load.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_driver_load.yml deleted file mode 100644 index 74560799..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_driver_load.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: ElasticSearch -source: windows_driver_load -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - ImageLoaded: file.path - ImagePath: file.path - Hashes: winlog.event_data.Hashes \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_image_load.yml deleted file mode 100644 index 4fdf83f9..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_image_load.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: ElasticSearch -source: windows_image_load -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - Image: process.executable - ImageLoaded: file.path - SignatureStatus: SignatureStatus - OriginalFileName: OriginalFileName - Signed: Signed \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_ldap_debug.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_ldap_debug.yml deleted file mode 100644 index 275d6ba8..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_ldap_debug.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: ElasticSearch -source: windows_ldap_debug -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - EventID: - - event.code - - winlog.event_id - SearchFilter: winlog.event_data.SearchFilter \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_network_connection.yml deleted file mode 100644 index 98f2a4e4..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_network_connection.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: ElasticSearch -source: windows_network_connection -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - Image: process.executable - DestinationHostname: destination.domain - DestinationIp: destination.ip - DestinationPort: destination.port - SourceIp: source.ip - SourcePort: source.port diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_ntlm.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_ntlm.yml deleted file mode 100644 index 49a1408d..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_ntlm.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: ElasticSearch -source: windows_ntlm -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - ProcessName: ProcessName - WorkstationName: winlog.event_data.WorkstationName - EventID: - - event.code - - winlog.event_id - TargetName: winlog.event_data.TargetName diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_powershell.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_powershell.yml deleted file mode 100644 index 16a39e83..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_powershell.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: ElasticSearch -source: windows_powershell -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - EventID: - - event.code - - winlog.event_id - CommandLine: powershell.command.value - ScriptBlockText: powershell.file.script_block_text - Payload: - - powershell.command.invocation_details - - winlog.event_data.Payload - HostApplication: process.command_line - ContextInfo: winlog.event_data.ContextInfo - HostName: process.title - EngineVersion: - - powershell.engine.version - - powershell.process.executable_version \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_process_creation.yml deleted file mode 100644 index 13f2da64..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_process_creation.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: ElasticSearch -source: windows_process_creation -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - CommandLine: process.command_line - CurrentDirectory: CurrentDirectory - Hashes: Hashes - Image: process.executable - IntegrityLevel: IntegrityLevel - ParentCommandLine: process.parent.command_line - ParentImage: process.parent.executable - ParentUser: ParentUser - Product: Product - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_security.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_security.yml deleted file mode 100644 index 4662cb3e..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_security.yml +++ /dev/null @@ -1,149 +0,0 @@ -platform: ElasticSearch -source: windows_security -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - EventID: - - event.code - - winlog.event_id - ParentImage: process.parent.executable - AccessMask: winlog.event_data.AccessMask - AccountName: winlog.event_data.AccountName - AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo - AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName - AuditPolicyChanges: winlog.event_data.AuditPolicyChanges - AuthenticationPackageName: winlog.event_data.AuthenticationPackageName - CallingProcessName: winlog.event_data.CallingProcessName - Channel: winlog.channel - ComputerName: winlog.ComputerName - EventType: winlog.event_data.EventType - FailureReason: winlog.event_data.FailureCode - FileName: file.path - GrantedAccess: winlog.event_data.GrantedAccess - Hashes: winlog.event_data.Hashes - HiveName: winlog.event_data.HiveName - IpAddress: source.ip - IpPort: source.port - KeyLength: winlog.event_data.KeyLength - LogonProcessName: winlog.event_data.LogonProcessName - LogonType: winlog.event_data.LogonType - LinkName: LinkName - MemberName: winlog.event_data.MemberName - MemberSid: winlog.event_data.MemberSid - NewProcessName: winlog.event_data.NewProcessName - ObjectClass: winlog.event_data.ObjectClass - ObjectName: winlog.event_data.ObjectName - ObjectType: winlog.event_data.ObjectType - ObjectValueName: winlog.event_data.ObjectValueName - Path: winlog.event_data.Path - CommandLine: winlog.event_data.ProcessCommandLine - OldUacValue: winlog.event_data.OldUacValue - CertIssuerName: winlog.event_data.CertIssuerName - SubStatus: winlog.event_data.SubStatus - DisplayName: winlog.event_data.DisplayName - TaskContent: winlog.event_data.TaskContent - ServiceSid: winlog.event_data.ServiceSid - CertThumbprint: winlog.event_data.CertThumbprint - ClassName: winlog.event_data.ClassName - NotificationPackageName: winlog.event_data.NotificationPackageName - NewSd: winlog.event_data.NewSd - TestSigning: winlog.event_data.TestSigning - TargetInfo: winlog.event_data.TargetInfo - ClientProcessId: winlog.event_data.ClientProcessId - ParentProcessId: winlog.event_data.ParentProcessId - AccessList: winlog.event_data.AccessList - GroupMembership: winlog.event_data.GroupMembership - FilterName: winlog.event_data.FilterName - ChangeType: winlog.event_data.ChangeType - LayerName: winlog.event_data.LayerName - ServiceAccount: winlog.event_data.ServiceAccount - AttributeValue: winlog.event_data.AttributeValue - SessionName: winlog.event_data.SessionName - TaskName: winlog.event_data.TaskName - ObjectDN: winlog.event_data.ObjectDN - TemplateContent: winlog.event_data.TemplateContent - NewTemplateContent: winlog.event_data.NewTemplateContent - SourcePort: winlog.event_data.SourcePort - PasswordLastSet: winlog.event_data.PasswordLastSet - PrivilegeList: winlog.event_data.PrivilegeList - DeviceDescription: winlog.event_data.DeviceDescription - TargetServerName: winlog.event_data.TargetServerName - NewTargetUserName: winlog.event_data.NewTargetUserName - OperationType: winlog.event_data.OperationType - DestPort: winlog.event_data.DestPort - ServiceStartType: winlog.event_data.ServiceStartType - OldTargetUserName: winlog.event_data.OldTargetUserName - UserPrincipalName: winlog.event_data.UserPrincipalName - Accesses: winlog.event_data.Accesses - DnsHostName: winlog.event_data.DnsHostName - DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks - AuditSourceName: winlog.event_data.AuditSourceName - Workstation: winlog.event_data.Workstation - DestAddress: winlog.event_data.DestAddress - PreAuthType: winlog.event_data.PreAuthType - SecurityPackageName: winlog.event_data.SecurityPackageName - SubjectLogonId: winlog.event_data.SubjectLogonId - NewUacValue: winlog.event_data.NewUacValue - EnabledPrivilegeList: winlog.event_data.EnabledPrivilegeList - RelativeTargetName: winlog.event_data.RelativeTargetName - CertSerialNumber: winlog.event_data.CertSerialNumber - SidHistory: winlog.event_data.SidHistory - TargetLogonId: winlog.event_data.TargetLogonId - KernelDebug: winlog.event_data.KernelDebug - CallerProcessName: winlog.event_data.CallerProcessName - ProcessName: process.executable - Properties: winlog.event_data.Properties - UserAccountControl: winlog.event_data.UserAccountControl - RegistryValue: winlog.event_data.RegistryValue - SecurityID: winlog.event_data.SecurityID - ServiceFileName: winlog.event_data.ServiceFileName - SecurityDescriptor: - - winlog.event_data.SecurityDescriptor - - winlog.event_data.TaskContentNew - ServiceName: winlog.event_data.ServiceName - ShareName: winlog.event_data.ShareName - NewValue: winlog.event_data.NewValue - Source: winlog.event_data.Source - Status: winlog.event_data.Status - SubjectDomainName: winlog.event_data.SubjectDomainName - SubjectUserName: winlog.event_data.SubjectUserName - SubjectUserSid: winlog.event_data.SubjectUserSid - SourceAddr: SourceAddr - SourceAddress: SourceAddress - TargetName: TargetName - ServicePrincipalNames: winlog.event_data.ServicePrincipalNames - TargetDomainName: winlog.event_data.TargetDomainName - TargetSid: winlog.event_data.TargetSid - TargetUserName: winlog.event_data.TargetUserName - ObjectServer: - TargetUserSid: winlog.event_data.TargetUserSid - TicketEncryptionType: winlog.event_data.TicketEncryptionType - TicketOptions: winlog.event_data.TicketOptions - WorkstationName: source.domain - TransmittedServices: winlog.event_data.TransmittedServices - AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm - LayerRTID: LayerRTID - BSSID: winlog.event_data.BSSID - BSSType: winlog.event_data.BSSType - CipherAlgorithm: winlog.event_data.CipherAlgorithm - ConnectionId: winlog.event_data.ConnectionId - ConnectionMode: winlog.event_data.ConnectionMode - InterfaceDescription: winlog.event_data.InterfaceDescription - InterfaceGuid: winlog.event_data.InterfaceGuid - OnexEnabled: winlog.event_data.OnexEnabled - PHYType: winlog.event_data.PHYType - ProfileName: winlog.event_data.ProfileName - SSID: winlog.event_data.SSID - Domain: winlog.event_data.Domain - ServiceType: winlog.event_data.ServiceType - SourceName: winlog.event_data.SourceName - StartType: winlog.event_data.StartType - UserID: winlog.event_data.UserID - ParentProcessName: process.parent.name - Service: Service diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_sysmon.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_sysmon.yml deleted file mode 100644 index d862c88e..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_sysmon.yml +++ /dev/null @@ -1,67 +0,0 @@ -platform: ElasticSearch -source: windows_sysmon -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - - -field_mapping: - CommandLine: process.command_line - Image: process.executable - ParentImage: process.parent.executable - EventID: - - event.code - - winlog.event_id - CallTrace: winlog.event_data.CallTrace - Company: winlog.event_data.Company - CurrentDirectory: process.working_directory - Description: winlog.event_data.Description - DestinationHostname: destination.domain - DestinationIp: destination.ip - DestinationIsIpv6: winlog.event_data.DestinationIsIpv6 - DestinationPort: destination.port - DestinationPortName: network.protocol - Hashes: winlog.event_data.Hashes - Initiated: winlog.event_data.Initiated - IntegrityLevel: winlog.event_data.IntegrityLevel - ParentCommandLine: process.parent.command_line - Product: winlog.event_data.Product - Protocol: winlog.event_data.Protocol - RuleName: winlog.event_data.RuleName - SourceHostname: source.domain - SourceIp: source.ip - SourceIsIpv6: winlog.event_data.SourceIsIpv6 - SourcePort: source.port - SourcePortName: winlog.event_data.SourcePortName - TargetFilename: file.path - User: - - user.name - - winlog.user.domain - - winlog.user.name - OriginalFileName: winlog.event_data.OriginalFileName - Signed: winlog.event_data.Signed - Signature: winlog.event_data.Signature - SignatureStatus: winlog.event_data.SignatureStatus - TargetObject: winlog.event_data.TargetObject - Details: winlog.event_data.Details - QueryName: dns.question.name - QueryResults: dns.answers - QueryStatus: winlog.event_data.QueryStatus - IsExecutable: winlog.event_data.IsExecutable - PipeName: file.name - ImageLoaded: file.path - ImagePath: winlog.event_data.ImagePath - Imphash: winlog.event_data.Imphash - SourceImage: process.executable - StartModule: winlog.event_data.StartModule - TargetImage: winlog.event_data.TargetImage - Device: Device - ProcessID: ProcessID - FileVersion: FileVersion - StartAddress: StartAddress - StartFunction: StartFunction - EventType: winlog.event_data.EventType \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_wmi_event.yml b/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_wmi_event.yml deleted file mode 100644 index 200eae2d..00000000 --- a/siem-converter/app/converter/mappings/platforms/elasticsearch/windows_wmi_event.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: ElasticSearch -source: windows_wmi_event -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - Destination: Destination - EventID: - - event.code - - winlog.event_id diff --git a/siem-converter/app/converter/mappings/platforms/logscale/aws_cloudtrail.yml b/siem-converter/app/converter/mappings/platforms/logscale/aws_cloudtrail.yml deleted file mode 100644 index 27b43dbe..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/aws_cloudtrail.yml +++ /dev/null @@ -1,30 +0,0 @@ -platform: LogScale -source: aws_cloudtrail -description: Text that describe current mapping - - -field_mapping: - eventSource: event.provider - eventName: event.action - AdditionalEventData: AdditionalEventData - additionalEventData.MFAUsed: additionalEventData.MFAUsed - errorCode: errorCode - errorMessage: errorMessage - eventType: eventType - requestParameters: requestParameters - requestParameters.attribute: requestParameters.attribute - requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP - requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort - requestParameters.userData: requestParameters.userData - responseElements: responseElements - responseElements.ConsoleLogin: responseElements.ConsoleLogin - responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword - responseElements.publiclyAccessible: responseElements.publiclyAccessible - status: status - terminatingRuleId: terminatingRuleId - userAgent: userAgent - userIdentity.arn: userIdentity.arn - userIdentity.principalId: userIdentity.principalId - userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type - userIdentity.type: userIdentity.type - userIdentity.userName: userIdentity.userName diff --git a/siem-converter/app/converter/mappings/platforms/logscale/aws_eks.yml b/siem-converter/app/converter/mappings/platforms/logscale/aws_eks.yml deleted file mode 100644 index 4b5366a6..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/aws_eks.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: LogScale -source: aws_eks -description: Text that describe current mapping - - -field_mapping: - annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision - annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy - aws_node_type: aws_node_type - objectRef.namespace: objectRef.namespace - objectRef.resource: objectRef.resource - objectRef.subresource: objectRef.subresource - requestObject.rules.resources: requestObject.rules.resources - requestObject.rules.verbs: requestObject.rules.verbs - requestObject.spec.containers.image: requestObject.spec.containers.image - requestURI: requestURI - stage: stage - user.groups: user.groups - user.username: user.username - verb: verb diff --git a/siem-converter/app/converter/mappings/platforms/logscale/azure_AzureDiagnostics.yml b/siem-converter/app/converter/mappings/platforms/logscale/azure_AzureDiagnostics.yml deleted file mode 100644 index ae9d2fc4..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/azure_AzureDiagnostics.yml +++ /dev/null @@ -1,8 +0,0 @@ -platform: LogScale -source: azure_AzureDiagnostics -description: Text that describe current mapping - - -field_mapping: - ResultDescription: ResultDescription - Category: Category \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/azure_BehaviorAnalytics.yml b/siem-converter/app/converter/mappings/platforms/logscale/azure_BehaviorAnalytics.yml deleted file mode 100644 index 8a80e33f..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/azure_BehaviorAnalytics.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: LogScale -source: azure_BehaviorAnalytics -description: Text that describe current mapping - - -field_mapping: - ActionType: ActionType - ActivityInsights: ActivityInsights - ActivityType: ActivityType - EventSource: EventSource - DevicesInsights: DevicesInsights - RiskDetail: RiskDetail - UsersInsights: UsersInsights - UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount - UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/siem-converter/app/converter/mappings/platforms/logscale/azure_aadnoninteractiveusersigninlogs.yml b/siem-converter/app/converter/mappings/platforms/logscale/azure_aadnoninteractiveusersigninlogs.yml deleted file mode 100644 index 38767435..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/azure_aadnoninteractiveusersigninlogs.yml +++ /dev/null @@ -1,8 +0,0 @@ -platform: LogScale -source: azure_aadnoninteractiveusersigninlogs -description: Text that describe current mapping - - -field_mapping: - UserAgent: UserAgent - Type: Type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/azure_azureactivity.yml b/siem-converter/app/converter/mappings/platforms/logscale/azure_azureactivity.yml deleted file mode 100644 index ba8d789e..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/azure_azureactivity.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: LogScale -source: azure_azureactivity -description: Text that describe current mapping - - -field_mapping: - ActivityStatus: ActivityStatus - ActivityStatusValue: ActivityStatusValue - ActivitySubstatusValue: ActivitySubstatusValue - Authorization: Authorization - Category: Category - CategoryValue: CategoryValue - OperationName: OperationName - OperationNameValue: OperationNameValue - ResourceId: ResourceId - ResourceProviderValue: ResourceProviderValue - Type: Type diff --git a/siem-converter/app/converter/mappings/platforms/logscale/azure_azuread.yml b/siem-converter/app/converter/mappings/platforms/logscale/azure_azuread.yml deleted file mode 100644 index f479eac5..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/azure_azuread.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: LogScale -source: azure_azuread -description: Text that describe current mapping - - -field_mapping: - ActivityDisplayName: event.action - Category: azure.auditlogs.properties.category - LoggedByService: azure.auditlogs.properties.logged_by_service - Result: event.outcome - OperationName: OperationName - TargetResources: TargetResources - AADOperationType: AADOperationType - InitiatedBy: InitiatedBy - ResultReason: ResultReason - Status: Status - Status.errorCode: Status.errorCode - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/logscale/azure_m365.yml b/siem-converter/app/converter/mappings/platforms/logscale/azure_m365.yml deleted file mode 100644 index 96c3bd3a..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/azure_m365.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: LogScale -source: azure_m365 -description: Text that describe current mapping - - -field_mapping: - ClientInfoString: ClientInfoString - LogonError: LogonError - ModifiedProperties: ModifiedProperties - OfficeObjectId: OfficeObjectId - OfficeWorkload: OfficeWorkload - Operation: Operation - Parameters: Parameters - RecordType: RecordType - ResultStatus: ResultStatus - SourceFileExtension: SourceFileExtension - SourceFileName: SourceFileName - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/logscale/azure_signinlogs.yml b/siem-converter/app/converter/mappings/platforms/logscale/azure_signinlogs.yml deleted file mode 100644 index 785d6fe9..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/azure_signinlogs.yml +++ /dev/null @@ -1,24 +0,0 @@ -platform: LogScale -source: azure_signinlogs -description: Text that describe current mapping - - -field_mapping: - AppDisplayName: AppDisplayName - AppId: AppId - AuthenticationRequirement: AuthenticationRequirement - Category: Category - ConditionalAccessStatus: ConditionalAccessStatus - DeviceDetail: DeviceDetail - IsInteractive: IsInteractive - NetworkLocationDetails: NetworkLocationDetails - ResourceDisplayName: ResourceDisplayName - ResourceIdentity: ResourceIdentity - ResultDescription: ResultDescription - ResultType: ResultType - Status.errorCode: Status.errorCode - Status: Status - Status.failureReason: Status.failureReason - TokenIssuerType: TokenIssuerType - UserAgent: UserAgent - UserPrincipalName: UserPrincipalName diff --git a/siem-converter/app/converter/mappings/platforms/logscale/default.yml b/siem-converter/app/converter/mappings/platforms/logscale/default.yml deleted file mode 100644 index c048b941..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/default.yml +++ /dev/null @@ -1,6 +0,0 @@ -platform: LogScale -source: default -description: Text that describe current mapping - -default_log_source: - index: "" \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/dns.yml b/siem-converter/app/converter/mappings/platforms/logscale/dns.yml deleted file mode 100644 index b0215ec5..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/dns.yml +++ /dev/null @@ -1,12 +0,0 @@ -platform: LogScale -source: dns -description: Text that describe current mapping - - -field_mapping: - dns-query: dns.question.name - parent-domain: - - url.registered_domain - - destination.registered_domain - dns-answer: dns.answers.name - dns-record: dns.question.type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/firewall.yml b/siem-converter/app/converter/mappings/platforms/logscale/firewall.yml deleted file mode 100644 index 21a35a46..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/firewall.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: LogScale -source: firewall -description: Text that describe current mapping - - -field_mapping: - src-ip: - - source.address - - source.ip - src-port: source.port - dst-ip: - - destination.address - - destination.ip - dst-port: destination.port \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/gcp_gcp.audit.yml b/siem-converter/app/converter/mappings/platforms/logscale/gcp_gcp.audit.yml deleted file mode 100644 index 2fadd360..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/gcp_gcp.audit.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: LogScale -source: gcp_gcp.audit -description: Text that describe current mapping - - -field_mapping: - backupConfiguration.enabled: backupConfiguration.enabled - gcp.audit.method_name: gcp.audit.method_name - jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction - jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue - message: message - methodName: methodName - protoPayload.methodName: protoPayload.methodName - protoPayload.request.canIpForward: protoPayload.request.canIpForward - protoPayload.serviceName: protoPayload.serviceName - serviceName: serviceName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/gcp_pubsub.yml b/siem-converter/app/converter/mappings/platforms/logscale/gcp_pubsub.yml deleted file mode 100644 index 56845e08..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/gcp_pubsub.yml +++ /dev/null @@ -1,11 +0,0 @@ -platform: LogScale -source: gcp_pubsub -description: Text that describe current mapping - - -field_mapping: - data.protoPayload.methodName: data.protoPayload.methodName - data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member - data.resource.type: data.resource.type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/linux_auditd.yml b/siem-converter/app/converter/mappings/platforms/logscale/linux_auditd.yml deleted file mode 100644 index 67784ccb..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/linux_auditd.yml +++ /dev/null @@ -1,11 +0,0 @@ -platform: LogScale -source: linux_auditd -description: Text that describe current mapping - - -field_mapping: - a0: process.args - a1: process.args - a2: process.args - a3: process.args - exe: process.executable diff --git a/siem-converter/app/converter/mappings/platforms/logscale/linux_dns_query.yml b/siem-converter/app/converter/mappings/platforms/logscale/linux_dns_query.yml deleted file mode 100644 index bacbbdd7..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/linux_dns_query.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: LogScale -source: linux_dns_query -description: Text that describe current mapping - - -field_mapping: - Image: process.executable - User: user.name - QueryName: dns.question.name - QueryResults: dns.answers \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/linux_process_creation.yml b/siem-converter/app/converter/mappings/platforms/logscale/linux_process_creation.yml deleted file mode 100644 index 3c7c55cd..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/linux_process_creation.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: LogScale -source: linux_process_creation -description: Text that describe current mapping - - -field_mapping: - CommandLine: process.command_line - Image: process.executable - ParentCommandLine: process.parent.command_line - ParentImage: process.parent.executable diff --git a/siem-converter/app/converter/mappings/platforms/logscale/macos_dns_query.yml b/siem-converter/app/converter/mappings/platforms/logscale/macos_dns_query.yml deleted file mode 100644 index 71f76701..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/macos_dns_query.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: LogScale -source: macos_dns_query -description: Text that describe current mapping - - -field_mapping: - Image: process.executable - User: user.name - QueryName: dns.question.name - QueryResults: dns.answers \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/macos_network_connection.yml b/siem-converter/app/converter/mappings/platforms/logscale/macos_network_connection.yml deleted file mode 100644 index 5688ef34..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/macos_network_connection.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: LogScale -source: macos_network_connection -description: Text that describe current mapping - - -field_mapping: - Image: process.executable - DestinationHostname: destination.domain - DestinationIp: destination.ip - DestinationPort: destination.port - SourceIp: source.ip - SourcePort: source.port - ParentImage: process.parent.executable diff --git a/siem-converter/app/converter/mappings/platforms/logscale/macos_process_creation.yml b/siem-converter/app/converter/mappings/platforms/logscale/macos_process_creation.yml deleted file mode 100644 index bc0e0d22..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/macos_process_creation.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: LogScale -source: macos_process_creation -description: Text that describe current mapping - - -field_mapping: - CommandLine: process.command_line - Image: process.executable - ParentCommandLine: process.parent.command_line - ParentImage: process.parent.executable diff --git a/siem-converter/app/converter/mappings/platforms/logscale/okta_okta.yml b/siem-converter/app/converter/mappings/platforms/logscale/okta_okta.yml deleted file mode 100644 index 2d6abed4..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/okta_okta.yml +++ /dev/null @@ -1,28 +0,0 @@ -platform: LogScale -source: okta_okta -description: Text that describe current mapping - - -field_mapping: - client.user.id: okta.actor.id - source.user.id: okta.actor.id - User: okta.actor.type - alternateId: okta.actor.alternate_id - client.user.full_name: okta.actor.display_name - source.user.full_name: okta.actor.display_name - related.user: okta.actor.display_name - client.ip: okta.client.ip - source.ip: okta.client.ip - user_agent.original: okta.client.user_agent.raw_user_agent - userAgent.os: okta.client.user_agent.os - userAgent.browser: okta.client.user_agent.browser - client.zone: okta.client.zone - client.device: okta.client.device - client.id: okta.client.id - event.action: okta.event_type - outcome.reason: okta.outcome.reason - event.outcome: okta.event.outcome - client.as.number: okta.security_context.as.number - client.as.organization.name: okta.security_context.as.organization.name - client.domain: okta.security_context.isp - source.domain: okta.security_context.domain diff --git a/siem-converter/app/converter/mappings/platforms/logscale/proxy.yml b/siem-converter/app/converter/mappings/platforms/logscale/proxy.yml deleted file mode 100644 index df116261..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/proxy.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: LogScale -source: proxy -description: Text that describe current mapping - - -field_mapping: - c-uri: url.original - c-useragent: user_agent.original - cs-method: http.request.method - cs-bytes: http.request.body.bytes - cs-cookie-vars: http.cookie_vars - c-uri-extension: url.extension - c-uri-query: url.query - cs-cookie: http.cookie_vars - cs-host: - - url.domain - - destination.domain - - cs-referrer: http.request.referrer - cs-version: http.version - r-dns: - - destination.domain - - url.domain - sc-status: http.response.status_code - post-body: http.post_body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/webserver.yml b/siem-converter/app/converter/mappings/platforms/logscale/webserver.yml deleted file mode 100644 index d3363c71..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/webserver.yml +++ /dev/null @@ -1,24 +0,0 @@ -platform: LogScale -source: webserver -description: Text that describe current mapping - - -field_mapping: - c-uri: url.original - c-useragent: user_agent.original - cs-method: http.request.method - cs-bytes: http.request.body.bytes - cs-cookie-vars: http.cookie_vars - c-uri-extension: url.extension - c-uri-query: url.query - cs-cookie: http.cookie_vars - cs-host: - - url.domain - - destination.domain - cs-referrer: http.request.referrer - cs-version: http.version - r-dns: - - destination.domain - - url.domain - sc-status: http.response.status_code - post-body: http.post_body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_application.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_application.yml deleted file mode 100644 index 4c479f1e..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_application.yml +++ /dev/null @@ -1,11 +0,0 @@ -platform: LogScale -source: windows_application -description: Text that describe current mapping - - -field_mapping: - Provider_Name: winlog.provider_name - EventID: winlog.event_id - Data: winlog.event_data.Data - Message: winlog.event_data.Message - Level: winlog.opcode \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_bits_client.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_bits_client.yml deleted file mode 100644 index 8906eaaa..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_bits_client.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: LogScale -source: windows_bits_client -description: Text that describe current mapping - - -field_mapping: - LocalName: winlog.event_data.LocalName - EventID: winlog.event_id - RemoteName: winlog.event_data.RemoteName - processPath: winlog.event_data.processPath diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_create_remote_thread.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_create_remote_thread.yml deleted file mode 100644 index fa2f53a9..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_create_remote_thread.yml +++ /dev/null @@ -1,11 +0,0 @@ -platform: LogScale -source: windows_create_remote_thread -description: Text that describe current mapping - - -field_mapping: - SourceImage: winlog.event_data.SourceImage - TargetImage: winlog.event_data.TargetImage - StartModule: winlog.event_data.StartModule - StartAddress: winlog.event_data.StartAddress - StartFunction: winlog.event_data.StartFunction \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_create_stream_hash.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_create_stream_hash.yml deleted file mode 100644 index fa6d9f8e..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_create_stream_hash.yml +++ /dev/null @@ -1,8 +0,0 @@ -platform: LogScale -source: windows_create_stream_hash -description: Text that describe current mapping - - -field_mapping: - Image: winlog.event_data.Image - Hashes: winlog.event_data.Hashes \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_dns_query.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_dns_query.yml deleted file mode 100644 index 24cc8453..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_dns_query.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: LogScale -source: windows_dns_query -description: Text that describe current mapping - - -field_mapping: - Image: winlog.event_data.Image - User: winlog.event_data.User - QueryName: winlog.event_data.QueryName - QueryResults: winlog.event_data.QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_driver_load.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_driver_load.yml deleted file mode 100644 index f80f7d17..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_driver_load.yml +++ /dev/null @@ -1,9 +0,0 @@ -platform: LogScale -source: windows_driver_load -description: Text that describe current mapping - - -field_mapping: - ImageLoaded: winlog.event_data.ImageLoaded - ImagePath: winlog.event_data.ImagePath - Hashes: winlog.event_data.Hashes \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_file_event.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_file_event.yml deleted file mode 100644 index 9fc0679e..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_file_event.yml +++ /dev/null @@ -1,11 +0,0 @@ -platform: LogScale -source: windows_file_event -description: Text that describe current mapping - - -field_mapping: - CreationUtcTime: winlog.event_data.CreationUtcTime - Image: winlog.event_data.Image - PreviousCreationUtcTime: winlog.event_data.PreviousCreationUtcTime - TargetFilename: winlog.event_data.TargetFilename - User: winlog.event_data.User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_image_load.yml deleted file mode 100644 index e3c550f7..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_image_load.yml +++ /dev/null @@ -1,11 +0,0 @@ -platform: LogScale -source: windows_image_load -description: Text that describe current mapping - - -field_mapping: - Image: winlog.event_data.Image - ImageLoaded: winlog.event_data.ImageLoaded - SignatureStatus: winlog.event_data.SignatureStatus - OriginalFileName: winlog.event_data.OriginalFileName - Signed: winlog.event_data.Signed diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_ldap_debug.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_ldap_debug.yml deleted file mode 100644 index 3334719e..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_ldap_debug.yml +++ /dev/null @@ -1,8 +0,0 @@ -platform: LogScale -source: windows_ldap_debug -description: Text that describe current mapping - - -field_mapping: - EventID: winlog.event_id - SearchFilter: winlog.event_data.SearchFilter \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_network_connection.yml deleted file mode 100644 index 7e88b61c..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_network_connection.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: LogScale -source: windows_network_connection -description: Text that describe current mapping - - -field_mapping: - Image: winlog.event_data.Image - DestinationHostname: winlog.event_data.DestinationHostname - DestinationIp: winlog.event_data.DestinationIp - DestinationPort: winlog.event_data.DestinationPort - SourceIp: winlog.event_data.SourceIp - SourcePort: winlog.event_data.SourcePort - Initiated: winlog.event_data.Initiated \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_ntlm.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_ntlm.yml deleted file mode 100644 index f73c1608..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_ntlm.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: LogScale -source: windows_ntlm -description: Text that describe current mapping - - -field_mapping: - WorkstationName: winlog.event_data.WorkstationName - ProcessName: winlog.event_data.ProcessName - EventID: winlog.event_id - TargetName: winlog.event_data.TargetName diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_pipe_created.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_pipe_created.yml deleted file mode 100644 index b24a1a17..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_pipe_created.yml +++ /dev/null @@ -1,8 +0,0 @@ -platform: LogScale -source: windows_pipe_created -description: Text that describe current mapping - - -field_mapping: - PipeName: winlog.event_data.PipeName - Image: winlog.event_data.Image \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_powershell.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_powershell.yml deleted file mode 100644 index d9615c01..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_powershell.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: LogScale -source: windows_powershell -description: Text that describe current mapping - - -field_mapping: - EventID: winlog.event_id - ScriptBlockText: winlog.event_data.ScriptBlockText - Payload: winlog.event_data.Payload - HostApplication: winlog.event_data.HostApplication - ContextInfo: winlog.event_data.ContextInfo - HostName: winlog.event_data.HostName - EngineVersion: winlog.event_data.EngineVersion - HostVersion: winlog.event_data.HostVersion \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_process_access.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_process_access.yml deleted file mode 100644 index 1ccaf4b0..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_process_access.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: LogScale -source: windows_process_access -description: Text that describe current mapping - - -field_mapping: - TargetImage: winlog.event_data.TargetImage - SourceImage: winlog.event_data.SourceImage - CallTrace: winlog.event_data.CallTrace - GrantedAccess: winlog.event_data.GrantedAccess \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_process_creation.yml deleted file mode 100644 index 679cd123..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_process_creation.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: LogScale -source: windows_process_creation -description: Text that describe current mapping - - -field_mapping: - CommandLine: winlog.event_data.CommandLine - CurrentDirectory: winlog.event_data.CurrentDirectory - Hashes: winlog.event_data.Hashes - Image: winlog.event_data.Image - IntegrityLevel: winlog.event_data.IntegrityLevel - ParentCommandLine: winlog.event_data.ParentCommandLine - ParentImage: winlog.event_data.ParentImage - ParentUser: winlog.event_data.ParentUser - Product: winlog.event_data.Product - User: winlog.event_data.User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_raw_access_thread.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_raw_access_thread.yml deleted file mode 100644 index 19ba93c7..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_raw_access_thread.yml +++ /dev/null @@ -1,8 +0,0 @@ -platform: LogScale -source: windows_raw_access_thread -description: Text that describe current mapping - - -field_mapping: - Image: winlog.event_data.Image - Device: winlog.event_data.Device \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_registry_event.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_registry_event.yml deleted file mode 100644 index a80e4c57..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_registry_event.yml +++ /dev/null @@ -1,11 +0,0 @@ -platform: LogScale -source: windows_registry_event -description: Text that describe current mapping - - -field_mapping: - TargetObject: winlog.event_data.TargetObject - Image: winlog.event_data.Image - Details: winlog.event_data.Details - EventType: winlog.event_data.EventType - User: winlog.event_data.User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_security.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_security.yml deleted file mode 100644 index 34479b0d..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_security.yml +++ /dev/null @@ -1,140 +0,0 @@ -platform: LogScale -source: windows_security -description: Text that describe current mapping - - -field_mapping: - EventID: winlog.event_id - ParentImage: winlog.event_data.ParentImage - AccessMask: winlog.event_data.AccessMask - AccountName: winlog.event_data.AccountName - AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo - AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName - AuditPolicyChanges: winlog.event_data.AuditPolicyChanges - AuthenticationPackageName: winlog.event_data.AuthenticationPackageName - CallingProcessName: winlog.event_data.CallingProcessName - Channel: winlog.channel - ComputerName: winlog.computer_name - EventType: winlog.event_data.EventType - FailureReason: winlog.event_data.FailureCode - FileName: winlog.event_data.FileName - GrantedAccess: winlog.event_data.GrantedAccess - Hashes: winlog.event_data.Hashes - HiveName: winlog.event_data.HiveName - IpAddress: winlog.event_data.IpAddress - IpPort: winlog.event_data.IpPort - KeyLength: winlog.event_data.KeyLength - LogonProcessName: winlog.event_data.LogonProcessName - LogonType: winlog.event_data.LogonType - LinkName: winlog.event_data.LinkName - MemberName: winlog.event_data.MemberName - MemberSid: winlog.event_data.MemberSid - NewProcessName: winlog.event_data.NewProcessName - ObjectClass: winlog.event_data.ObjectClass - ObjectName: winlog.event_data.ObjectName - ObjectType: winlog.event_data.ObjectType - ObjectValueName: winlog.event_data.ObjectValueName - Path: winlog.event_data.Path - CommandLine: winlog.event_data.ProcessCommandLine - OldUacValue: winlog.event_data.OldUacValue - CertIssuerName: winlog.event_data.CertIssuerName - SubStatus: winlog.event_data.SubStatus - DisplayName: winlog.event_data.DisplayName - TaskContent: winlog.event_data.TaskContent - ServiceSid: winlog.event_data.ServiceSid - CertThumbprint: winlog.event_data.CertThumbprint - ClassName: winlog.event_data.ClassName - NotificationPackageName: winlog.event_data.NotificationPackageName - NewSd: winlog.event_data.NewSd - TestSigning: winlog.event_data.TestSigning - TargetInfo: winlog.event_data.TargetInfo - ClientProcessId: winlog.event_data.ClientProcessId - ParentProcessId: winlog.event_data.ParentProcessId - AccessList: winlog.event_data.AccessList - GroupMembership: winlog.event_data.GroupMembership - FilterName: winlog.event_data.FilterName - ChangeType: winlog.event_data.ChangeType - LayerName: winlog.event_data.LayerName - ServiceAccount: winlog.event_data.ServiceAccount - AttributeValue: winlog.event_data.AttributeValue - SessionName: winlog.event_data.SessionName - TaskName: winlog.event_data.TaskName - ObjectDN: winlog.event_data.ObjectDN - TemplateContent: winlog.event_data.TemplateContent - NewTemplateContent: winlog.event_data.NewTemplateContent - SourcePort: winlog.event_data.SourcePort - PasswordLastSet: winlog.event_data.PasswordLastSet - PrivilegeList: winlog.event_data.PrivilegeList - DeviceDescription: winlog.event_data.DeviceDescription - TargetServerName: winlog.event_data.TargetServerName - NewTargetUserName: winlog.event_data.NewTargetUserName - OperationType: winlog.event_data.OperationType - DestPort: winlog.event_data.DestPort - ServiceStartType: winlog.event_data.ServiceStartType - OldTargetUserName: winlog.event_data.OldTargetUserName - UserPrincipalName: winlog.event_data.UserPrincipalName - Accesses: winlog.event_data.Accesses - DnsHostName: winlog.event_data.DnsHostName - DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks - AuditSourceName: winlog.event_data.AuditSourceName - Workstation: winlog.event_data.Workstation - DestAddress: winlog.event_data.DestAddress - PreAuthType: winlog.event_data.PreAuthType - SecurityPackageName: winlog.event_data.SecurityPackageName - SubjectLogonId: winlog.event_data.SubjectLogonId - NewUacValue: winlog.event_data.NewUacValue - EnabledPrivilegeList: winlog.event_data.EnabledPrivilegeList - RelativeTargetName: winlog.event_data.RelativeTargetName - CertSerialNumber: winlog.event_data.CertSerialNumber - SidHistory: winlog.event_data.SidHistory - TargetLogonId: winlog.event_data.TargetLogonId - KernelDebug: winlog.event_data.KernelDebug - CallerProcessName: winlog.event_data.CallerProcessName - ProcessName: winlog.event_data.ProcessName - Properties: winlog.event_data.Properties - UserAccountControl: winlog.event_data.UserAccountControl - RegistryValue: winlog.event_data.RegistryValue - SecurityID: winlog.event_data.SecurityID - ServiceFileName: winlog.event_data.ServiceFileName - SecurityDescriptor: winlog.event_data.SecurityDescriptor - ServiceName: winlog.event_data.ServiceName - ShareName: winlog.event_data.ShareName - NewValue: winlog.event_data.NewValue - Source: winlog.event_data.Source - Status: winlog.event_data.Status - SubjectDomainName: winlog.event_data.SubjectDomainName - SubjectUserName: winlog.event_data.SubjectUserName - SubjectUserSid: winlog.event_data.SubjectUserSid - SourceAddr: winlog.event_data.SourceAddr - SourceAddress: winlog.event_data.SourceAddress - TargetName: winlog.event_data.TargetName - ServicePrincipalNames: winlog.event_data.ServicePrincipalNames - TargetDomainName: winlog.event_data.TargetDomainName - TargetSid: winlog.event_data.TargetSid - TargetUserName: winlog.event_data.TargetUserName - ObjectServer: winlog.event_data.ObjectServer - TargetUserSid: winlog.event_data.TargetUserSid - TicketEncryptionType: winlog.event_data.TicketEncryptionType - TicketOptions: winlog.event_data.TicketOptions - WorkstationName: winlog.event_data.WorkstationName - TransmittedServices: winlog.event_data.TransmittedServices - AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm - LayerRTID: winlog.event_data.LayerRTID - BSSID: winlog.event_data.BSSID - BSSType: winlog.event_data.BSSType - CipherAlgorithm: winlog.event_data.CipherAlgorithm - ConnectionId: winlog.event_data.ConnectionId - ConnectionMode: winlog.event_data.ConnectionMode - InterfaceDescription: winlog.event_data.InterfaceDescription - InterfaceGuid: winlog.event_data.InterfaceGuid - OnexEnabled: winlog.event_data.OnexEnabled - PHYType: winlog.event_data.PHYType - ProfileName: winlog.event_data.ProfileName - SSID: winlog.event_data.SSID - Domain: winlog.event_data.Domain - ServiceType: winlog.event_data.ServiceType - SourceName: winlog.event_data.SourceName - StartType: winlog.event_data.StartType - UserID: winlog.event_data.UserID - ParentProcessName: winlog.event_data.ParentProcessName - Service: winlog.event_data.Service \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_sysmon.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_sysmon.yml deleted file mode 100644 index fdd88078..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_sysmon.yml +++ /dev/null @@ -1,57 +0,0 @@ -platform: LogScale -source: windows_sysmon -description: Text that describe current mapping - - - -field_mapping: - CommandLine: winlog.event_data.CommandLine - Image: winlog.event_data.Image - ParentImage: winlog.event_data.ParentImage - EventID: winlog.event_id - CallTrace: winlog.event_data.CallTrace - Company: winlog.event_data.Company - CurrentDirectory: winlog.event_data.CurrentDirectory - Description: winlog.event_data.Description - DestinationHostname: winlog.event_data.DestinationHostname - DestinationIp: winlog.event_data.DestinationIp - DestinationIsIpv6: winlog.event_data.DestinationIsIpv6 - DestinationPort: winlog.event_data.DestinationPort - DestinationPortName: winlog.event_data.DestinationPortName - Hashes: winlog.event_data.Hashes - Initiated: winlog.event_data.Initiated - IntegrityLevel: winlog.event_data.IntegrityLevel - ParentCommandLine: winlog.event_data.ParentCommandLine - Product: winlog.event_data.Product - Protocol: winlog.event_data.Protocol - RuleName: winlog.event_data.RuleName - SourceHostname: winlog.event_data.SourceHostname - SourceIp: winlog.event_data.SourceIp - SourceIsIpv6: winlog.event_data.SourceIsIpv6 - SourcePort: winlog.event_data.SourcePort - SourcePortName: winlog.event_data.SourcePortName - TargetFilename: winlog.event_data.TargetFilename - User: winlog.event_data.User - OriginalFileName: winlog.event_data.OriginalFileName - Signed: winlog.event_data.Signed - Signature: winlog.event_data.Signature - SignatureStatus: winlog.event_data.SignatureStatus - TargetObject: winlog.event_data.TargetObject - Details: winlog.event_data.Details - QueryName: winlog.event_data.QueryName - QueryResults: winlog.event_data.QueryResults - QueryStatus: winlog.event_data.QueryStatus - IsExecutable: winlog.event_data.IsExecutable - PipeName: winlog.event_data.PipeName - ImageLoaded: winlog.event_data.ImageLoaded - ImagePath: winlog.event_data.ImagePath - Imphash: winlog.event_data.Imphash - SourceImage: winlog.event_data.SourceImage - StartModule: winlog.event_data.StartModule - TargetImage: winlog.event_data.TargetImage - Device: winlog.event_data.Device - ProcessID: winlog.event_data.ProcessID - FileVersion: winlog.event_data.FileVersion - StartAddress: winlog.event_data.StartAddress - StartFunction: winlog.event_data.StartFunction - EventType: winlog.event_data.EventType \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_system.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_system.yml deleted file mode 100644 index 21da9e90..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_system.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: LogScale -source: windows_system -description: Text that describe current mapping - - -field_mapping: - EventID: winlog.event_id - AccountName: winlog.event_data.AccountName - ImagePath: winlog.event_data.ImagePath - ServiceName: winlog.event_data.ServiceName - ServiceType: winlog.event_data.ServiceType - StartType: winlog.event_data.StartType - Provider_Name: winlog.provider_name - Origin: winlog.event_data.Origin - HiveName: winlog.event_data.HiveName - Caption: winlog.event_data.Caption - param1: winlog.event_data.param1 - param2: winlog.event_data.param2 - Channel: winlog.channel - DeviceName: winlog.event_data.DeviceName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/logscale/windows_wmi_event.yml b/siem-converter/app/converter/mappings/platforms/logscale/windows_wmi_event.yml deleted file mode 100644 index b500b3e2..00000000 --- a/siem-converter/app/converter/mappings/platforms/logscale/windows_wmi_event.yml +++ /dev/null @@ -1,8 +0,0 @@ -platform: LogScale -source: windows_wmi_event -description: Text that describe current mapping - - -field_mapping: - Destination: winlog.event_data.Destination - EventID: winlog.event_id diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/default.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/default.yml deleted file mode 100644 index cb1f1dfb..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/default.yml +++ /dev/null @@ -1,9 +0,0 @@ -platform: Microsoft Defender -source: default -description: Text that describe current mapping - -log_source: - table: [union *] - -default_log_source: - table: union * \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/linux_file_event.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/linux_file_event.yml deleted file mode 100644 index ec63729c..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/linux_file_event.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Defender -source: linux_file_event -description: Text that describe current mapping - -log_source: - table: [DeviceFileEvents] - -default_log_source: - table: DeviceFileEvents - -field_mapping: - Image: InitiatingProcessFolderPath - TargetFilename : FolderPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/linux_network_connection.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/linux_network_connection.yml deleted file mode 100644 index 7d160d48..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/linux_network_connection.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: Microsoft Defender -source: linux_network_connection -description: Text that describe current mapping - -log_source: - table: [DeviceNetworkEvents] - -default_log_source: - table: DeviceNetworkEvents - -field_mapping: - CommandLine: ProcessCommandLine - Image: FolderPath - ParentImage: InitiatingProcessFolderPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/macos_file_event.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/macos_file_event.yml deleted file mode 100644 index f98460a0..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/macos_file_event.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Defender -source: macos_file_event -description: Text that describe current mapping - -log_source: - table: [DeviceFileEvents] - -default_log_source: - table: DeviceFileEvents - -field_mapping: - Image: InitiatingProcessFolderPath - TargetFilename : FolderPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/macos_network_connection.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/macos_network_connection.yml deleted file mode 100644 index 24e498e8..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/macos_network_connection.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: Microsoft Defender -source: macos_network_connection -description: Text that describe current mapping - -log_source: - table: [DeviceNetworkEvents] - -default_log_source: - table: DeviceNetworkEvents - -field_mapping: - CommandLine: ProcessCommandLine - Image: FolderPath - ParentImage: InitiatingProcessFolderPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/macos_process_creation.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/macos_process_creation.yml deleted file mode 100644 index 6eb9dae8..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/macos_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Microsoft Defender -source: macos_process_creation -description: Text that describe current mapping - -log_source: - table: [DeviceProcessEvents] - -default_log_source: - table: DeviceProcessEvents - -field_mapping: - CommandLine: ProcessCommandLine - Image: FolderPath - ParentCommandLine: InitiatingProcessCommandLine - ParentImage: InitiatingProcessFolderPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_file_event.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_file_event.yml deleted file mode 100644 index 34ddff9c..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_file_event.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Defender -source: windows_file_event -description: Text that describe current mapping - -log_source: - table: [DeviceFileEvents] - -default_log_source: - table: DeviceFileEvents - -field_mapping: - Image: InitiatingProcessFolderPath - TargetFilename: FolderPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_image_load.yml deleted file mode 100644 index 3c50a560..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_image_load.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Defender -source: windows_image_load -description: Text that describe current mapping - -log_source: - table: [DeviceImageLoadEvents] - -default_log_source: - table: DeviceImageLoadEvents - -field_mapping: - Image: InitiatingProcessFolderPath - ImageLoaded: FolderPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_network_connection.yml deleted file mode 100644 index 0a675234..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_network_connection.yml +++ /dev/null @@ -1,12 +0,0 @@ -platform: Microsoft Defender -source: windows_network_connection -description: Text that describe current mapping - -log_source: - table: [DeviceNetworkEvents] - -default_log_source: - table: DeviceNetworkEvents - -field_mapping: - Image: FolderPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_process_creation.yml deleted file mode 100644 index 66e4c827..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Microsoft Defender -source: windows_process_creation -description: Text that describe current mapping - -log_source: - table: [DeviceProcessEvents] - -default_log_source: - table: DeviceProcessEvents - -field_mapping: - CommandLine: ProcessCommandLine - Image: FolderPath - ParentCommandLine: InitiatingProcessCommandLine - ParentImage: InitiatingProcessFolderPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_registry_event.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_registry_event.yml deleted file mode 100644 index 02b16f31..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_registry_event.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Microsoft Defender -source: windows_registry_event -description: Text that describe current mapping - -log_source: - table: [DeviceRegistryEvents] - -default_log_source: - table: DeviceRegistryEvents - -field_mapping: - TargetObject: RegistryKey - Image: InitiatingProcessFolderPath - Details: - - RegistryValueData - - RegistryValueType \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_sysmon.yml b/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_sysmon.yml deleted file mode 100644 index f9c6b2c0..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_defender/windows_sysmon.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Microsoft Defender -source: windows_sysmon -description: Text that describe current mapping - -log_source: - table: [DeviceProcessEvents] - -default_log_source: - table: DeviceProcessEvents - - -field_mapping: - CommandLine: ProcessCommandLine - Image: FolderPath - ParentImage: InitiatingProcessFolderPath diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/aws_cloudtrail.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/aws_cloudtrail.yml deleted file mode 100644 index 18cc9279..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/aws_cloudtrail.yml +++ /dev/null @@ -1,35 +0,0 @@ -platform: Microsoft Sentinel -source: aws_cloudtrail -description: Text that describe current mapping - -log_source: - table: [AWSCloudTrail] - -default_log_source: - table: AWSCloudTrail - -field_mapping: - eventSource: EventSource - eventName: EventName - AdditionalEventData: AdditionalEventData - additionalEventData.MFAUsed: additionalEventData.MFAUsed - errorCode: ErrorCode - errorMessage: ErrorMessage - eventType: EventTypeName - requestParameters: RequestParameters - requestParameters.attribute: requestParameters.attribute - requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP - requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort - requestParameters.userData: requestParameters.userData - responseElements: ResponseElements - responseElements.ConsoleLogin: responseElements.ConsoleLogin - responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword - responseElements.publiclyAccessible: responseElements.publiclyAccessible - status: status - terminatingRuleId: terminatingRuleId - userAgent: UserAgent - userIdentity.arn: UserIdentityArn - userIdentity.principalId: UserIdentityPrincipalid - userIdentity.sessionContext.sessionIssuer.type: SessionIssuerType - userIdentity.type: UserIdentityType - userIdentity.userName: UserIdentityUserName diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/aws_eks.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/aws_eks.yml deleted file mode 100644 index 70da2d04..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/aws_eks.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Microsoft Sentinel -source: aws_eks -description: Text that describe current mapping - -log_source: - table: [AWSCloudTrail] - -default_log_source: - table: AWSCloudTrail - -field_mapping: - annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision - annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy - aws_node_type: aws_node_type - objectRef.namespace: objectRef.namespace - objectRef.resource: objectRef.resource - objectRef.subresource: objectRef.subresource - requestObject.rules.resources: requestObject.rules.resources - requestObject.rules.verbs: requestObject.rules.verbs - requestObject.spec.containers.image: requestObject.spec.containers.image - requestURI: requestURI - stage: stage - user.groups: user.groups - user.username: user.username - verb: verb diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_AzureDiagnostics.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_AzureDiagnostics.yml deleted file mode 100644 index ef714957..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_AzureDiagnostics.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Sentinel -source: azure_AzureDiagnostics -description: Text that describe current mapping - -log_source: - table: [AzureDiagnostics] - -default_log_source: - table: AzureDiagnostics - -field_mapping: - ResultDescription: ResultDescription - Category: Category diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_BehaviorAnalytics.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_BehaviorAnalytics.yml deleted file mode 100644 index 25fc73bf..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_BehaviorAnalytics.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Microsoft Sentinel -source: azure_BehaviorAnalytics -description: Text that describe current mapping - -log_source: - table: [BehaviorAnalytics] - -default_log_source: - table: BehaviorAnalytics - -field_mapping: - ActionType: ActionType - ActivityInsights: ActivityInsights - ActivityType: ActivityType - EventSource: EventSource - DevicesInsights: DevicesInsights - RiskDetail: RiskDetail - UsersInsights: UsersInsights - UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount - UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_aadnoninteractiveusersigninlogs.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_aadnoninteractiveusersigninlogs.yml deleted file mode 100644 index 6b42921d..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_aadnoninteractiveusersigninlogs.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Sentinel -source: azure_aadnoninteractiveusersigninlogs -description: Text that describe current mapping - -log_source: - table: [AADNonInteractiveUserSignInLogs] - -default_log_source: - table: AADNonInteractiveUserSignInLogs - -field_mapping: - UserAgent: UserAgent - Type: Type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_azureactivity.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_azureactivity.yml deleted file mode 100644 index 9124ab2f..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_azureactivity.yml +++ /dev/null @@ -1,22 +0,0 @@ -platform: Microsoft Sentinel -source: azure_azureactivity -description: Text that describe current mapping - -log_source: - table: [AzureActivity] - -default_log_source: - table: AzureActivity - -field_mapping: - ActivityStatus: ActivityStatus - ActivityStatusValue: ActivityStatusValue - ActivitySubstatusValue: ActivitySubstatusValue - Authorization: Authorization - Category: Category - CategoryValue: CategoryValue - OperationName: OperationName - OperationNameValue: OperationNameValue - ResourceId: ResourceId - ResourceProviderValue: ResourceProviderValue - Type: Type diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_azuread.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_azuread.yml deleted file mode 100644 index 127a9f98..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_azuread.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Microsoft Sentinel -source: azure_azuread -description: Text that describe current mapping - -log_source: - table: [AuditLogs] - -default_log_source: - table: AuditLogs - -field_mapping: - ActivityDisplayName: ActivityDisplayName - Category: Category - LoggedByService: LoggedByService - Result: Result - OperationName: OperationName - TargetResources: TargetResources - AADOperationType: AADOperationType - InitiatedBy: InitiatedBy - ResultReason: ResultReason - Status: Status - Status.errorCode: Status.errorCode - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_m365.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_m365.yml deleted file mode 100644 index 62098275..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_m365.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Microsoft Sentinel -source: azure_m365 -description: Text that describe current mapping - -log_source: - table: [OfficeActivity] - -default_log_source: - table: OfficeActivity - -field_mapping: - ClientInfoString: ClientInfoString - LogonError: LogonError - ModifiedProperties: ModifiedProperties - OfficeObjectId: OfficeObjectId - OfficeWorkload: OfficeWorkload - Operation: Operation - Parameters: Parameters - RecordType: RecordType - ResultStatus: ResultStatus - SourceFileExtension: SourceFileExtension - SourceFileName: SourceFileName - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_office365.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_office365.yml deleted file mode 100644 index 85cc213c..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_office365.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Microsoft Sentinel -source: azure_office365 -description: Text that describe current mapping - -log_source: - table: [OfficeActivity] - -default_log_source: - table: OfficeActivity - -field_mapping: - ClientInfoString: ClientInfoString - LogonError: LogonError - ModifiedProperties: ModifiedProperties - OfficeObjectId: OfficeObjectId - OfficeWorkload: OfficeWorkload - Operation: Operation - Parameters: Parameters - RecordType: RecordType - ResultStatus: ResultStatus - SourceFileExtension: SourceFileExtension - SourceFileName: SourceFileName - UserAgent: UserAgent \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/default.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/default.yml deleted file mode 100644 index 1723b330..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/default.yml +++ /dev/null @@ -1,7 +0,0 @@ -platform: Microsoft Sentinel -source: default -description: Text that describe current mapping - - -default_log_source: - table: union * \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_dns_query.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_dns_query.yml deleted file mode 100644 index 1864fa63..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Microsoft Sentinel -source: linux_dns_query -description: Text that describe current mapping - -log_source: - table: [SysmonEvent] - -default_log_source: - table: SysmonEvent - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_file_event.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_file_event.yml deleted file mode 100644 index 566eae30..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_file_event.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Sentinel -source: linux_file_event -description: Text that describe current mapping - -log_source: - table: [SysmonEvent, DeviceFileEvents] - -default_log_source: - table: SysmonEvent - -field_mapping: - Image: Image - TargetFilename : TargetFilename \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_network_connection.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_network_connection.yml deleted file mode 100644 index 575054b2..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_network_connection.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: Microsoft Sentinel -source: linux_network_connection -description: Text that describe current mapping - -log_source: - table: [SysmonEvent, DeviceNetworkEvents] - -default_log_source: - table: SysmonEvent - -field_mapping: - CommandLine: CommandLine - Image : Image - DestinationHostname: DestinationHostname - DestinationIp : DestinationIp - DestinationPort : DestinationPort - SourceIp : SourceIp - SourcePort : SourcePort - ParentImage : ParentImage \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_process_creation.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_process_creation.yml deleted file mode 100644 index 3cd89818..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Microsoft Sentinel -source: linux_process_creation -description: Text that describe current mapping - -log_source: - table: [SysmonEvent] - -default_log_source: - table: SysmonEvent - -field_mapping: - CommandLine: CommandLine - Image: Image - ParentCommandLine: ParentCommandLine - ParentImage: ParentImage \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/macos_file_event.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/macos_file_event.yml deleted file mode 100644 index bcbf1779..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/macos_file_event.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Sentinel -source: macos_file_event -description: Text that describe current mapping - -log_source: - table: [DeviceFileEvents] - -default_log_source: - table: DeviceFileEvents - -field_mapping: - Image: Image - TargetFilename : TargetFilename \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/macos_network_connection.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/macos_network_connection.yml deleted file mode 100644 index d8108596..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/macos_network_connection.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: Microsoft Sentinel -source: macos_network_connection -description: Text that describe current mapping - -log_source: - table: [DeviceNetworkEvents] - -default_log_source: - table: DeviceNetworkEvents - -field_mapping: - CommandLine: CommandLine - Image : Image - DestinationHostname: DestinationHostname - DestinationIp : DestinationIp - DestinationPort : DestinationPort - SourceIp : SourceIp - SourcePort : SourcePort - ParentImage : ParentImage diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/macos_process_creation.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/macos_process_creation.yml deleted file mode 100644 index f493bdab..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/macos_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Microsoft Sentinel -source: macos_process_creation -description: Text that describe current mapping - -log_source: - table: [DeviceProcessEvents] - -default_log_source: - table: DeviceProcessEvents - -field_mapping: - Image: FolderPath - ParentCommandLine: InitiatingProcessCommandLine - ParentImage: InitiatingProcessFolderPath - CommandLine: ProcessCommandLine \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/okta_okta.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/okta_okta.yml deleted file mode 100644 index 44e1fdf8..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/okta_okta.yml +++ /dev/null @@ -1,33 +0,0 @@ -platform: Microsoft Sentinel -source: okta_okta -description: Text that describe current mapping - -log_source: - table: [Okta_CL] - -default_log_source: - table: Okta_CL - -field_mapping: - client.user.id: client.user.id - source.user.id: source.user.id - User: User - alternateId: alternateId - client.user.full_name: client.user.full_name - source.user.full_name: source.user.full_name - related.user: related.user - client.ip: client.ip - source.ip: source.ip - user_agent.original: user_agent.original - userAgent.os: userAgent.os - userAgent.browser: userAgent.browser - client.zone: client.zone - client.device: client.device - client.id: client.id - event.action: event.action - outcome.reason: outcome.reason - event.outcome: event.outcome - client.as.number: client.as.number - client.as.organization.name: client.as.organization.name - client.domain: client.domain - source.domain: source.domain diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_bits_client.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_bits_client.yml deleted file mode 100644 index fb61a948..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_bits_client.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Microsoft Sentinel -source: windows_bits_client -description: Text that describe current mapping - -log_source: - table: [Event] - -default_log_source: - table: Event - -field_mapping: - LocalName: LocalName - EventID: EventID - RemoteName: RemoteName - processPath: processPath diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_dns_query.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_dns_query.yml deleted file mode 100644 index a326503d..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Microsoft Sentinel -source: windows_dns_load -description: Text that describe current mapping - -log_source: - table: [SysmonEvent, DeviceNetworkEvents] - -default_log_source: - table: SysmonEvent - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_driver_load.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_driver_load.yml deleted file mode 100644 index 68815eeb..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_driver_load.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: Microsoft Sentinel -source: windows_driver_load -description: Text that describe current mapping - -log_source: - table: [SysmonEvent] - -default_log_source: - table: SysmonEvent - -field_mapping: - ImageLoaded: ImageLoaded - ImagePath: ImagePath - Hashes: Hashes diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_file_event.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_file_event.yml deleted file mode 100644 index 7e1456dd..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_file_event.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Microsoft Sentinel -source: windows_file_event -description: Text that describe current mapping - -log_source: - table: [SysmonEvent, DeviceFileEvents] - -default_log_source: - table: SysmonEvent - -field_mapping: - CreationUtcTime: CreationUtcTime -# FileName - Image: Image -# OriginalFilename: PreviousCreationUtcTime - PreviousCreationUtcTime: PreviousCreationUtcTime - TargetFilename: TargetFilename - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_image_load.yml deleted file mode 100644 index dab65aa6..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_image_load.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Microsoft Sentinel -source: windows_image_load -description: Text that describe current mapping - -log_source: - table: [SysmonEvent, DeviceImageLoadEvents] - -default_log_source: - table: SysmonEvent - -field_mapping: - Image: Image - ImageLoaded: ImageLoaded - SignatureStatus: SignatureStatus - OriginalFileName: OriginalFileName - Signed: Signed \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_ldap_debug.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_ldap_debug.yml deleted file mode 100644 index f9143955..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_ldap_debug.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Sentinel -source: windows_ldap_debug -description: Text that describe current mapping - -log_source: - table: [Event] - -default_log_source: - table: Event | where EventLog in~ ('Microsoft-Windows-LDAP-Client/Debug') - -field_mapping: - EventID: EventID - SearchFilter: SearchFilter diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_network_connection.yml deleted file mode 100644 index 8325ebb2..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_network_connection.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Microsoft Sentinel -source: windows_network_connection -description: Text that describe current mapping - -log_source: - table: [SysmonEvent, DeviceNetworkEvents] - -default_log_source: - table: SysmonEvent - -field_mapping: - Image: Image - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationPort: DestinationPort - SourceIp: SourceIp - SourcePort: SourcePort - Initiated: Initiated \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_ntlm.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_ntlm.yml deleted file mode 100644 index 4e0b1083..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_ntlm.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Microsoft Sentinel -source: windows_ntlm -description: Text that describe current mapping - -log_source: - table: [Event] - -default_log_source: - table: Event - -field_mapping: - WorkstationName: WorkstationName - ProcessName: ProcessName - EventID: EventID - TargetName: TargetName diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_powershell.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_powershell.yml deleted file mode 100644 index ab4eb377..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_powershell.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: Microsoft Sentinel -source: windows_powershell -description: Text that describe current mapping - -log_source: - table: [Event] - -default_log_source: - table: "Event | where EventLog in~ ('Microsoft-Windows-PowerShell/Operational', 'Windows PowerShell')" - -field_mapping: - EventID: EventID - CommandLine: CommandLine - ScriptBlockText: EventData - Payload: Payload - HostApplication: HostApplication - ContextInfo: ContextInfo - HostName: HostName - EngineVersion: EngineVersion \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_process_creation.yml deleted file mode 100644 index 47ba144d..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_process_creation.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: Microsoft Sentinel -source: windows_process_creation -description: Text that describe current mapping - -log_source: - table: [SysmonEvent, SecurityEvent, WindowsEvent, DeviceProcessEvents] - -default_log_source: - table: SecurityEvent - -field_mapping: - CommandLine: CommandLine - CurrentDirectory: CurrentDirectory - Hashes: Hashes - Image: NewProcessName - IntegrityLevel: IntegrityLevel - ParentCommandLine: ParentCommandLine - ParentImage: ParentProcessName - ParentUser: ParentUser - Product: Product - User: Account \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_registry_event.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_registry_event.yml deleted file mode 100644 index 84b94b76..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_registry_event.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Microsoft Sentinel -source: windows_registry_event -description: Text that describe current mapping - -log_source: - table: [SysmonEvent, DeviceRegistryEvents] - -default_log_source: - table: SysmonEvent - -field_mapping: - TargetObject: TargetObject - Image: Image - Details: Details - EventType: EventType - User: User diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_security.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_security.yml deleted file mode 100644 index 5795179a..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_security.yml +++ /dev/null @@ -1,145 +0,0 @@ -platform: Microsoft Sentinel -source: windows_security -description: Text that describe current mapping - -log_source: - table: [SecurityEvent, WindowsEvent] - -default_log_source: - table: SecurityEvent - -field_mapping: - EventID: EventID - ParentImage: ParentProcessName - AccessMask: AccessMask - AccountName: AccountName - AllowedToDelegateTo: AllowedToDelegateTo - AttributeLDAPDisplayName: - AuditPolicyChanges: AuditPolicyChanges - AuthenticationPackageName: AuthenticationPackageName - CallingProcessName: CallingProcessName - Channel: Channel - ComputerName: Computer - EventType: EventType - FailureReason: FailureReason - FileName: FilePath - GrantedAccess: - Hashes: FileHash - HiveName: HiveName - IpAddress: IpAddress - IpPort: IpPort - KeyLength: KeyLength - LogonProcessName: LogonProcessName - LogonType: LogonType - LinkName: LinkName - MemberName: MemberName - MemberSid: MemberSid - NewProcessName: NewProcessName - ObjectClass: ObjectClass - ObjectName: ObjectName - ObjectType: ObjectType - ObjectValueName: ObjectValueName - Path: Path - CommandLine: CommandLine - OldUacValue: OldUacValue - CertIssuerName: CertIssuerName - SubStatus: SubStatus - DisplayName: DisplayName - TaskContent: TaskContent - ServiceSid: ServiceSid - CertThumbprint: CertThumbprint - ClassName: duplicate - NotificationPackageName: ClassName - NewSd: NewSd - TestSigning: TestSigning - TargetInfo: TargetInfo - ClientProcessId: TargetInfo - ParentProcessId: ParentProcessId - AccessList: AccessList - GroupMembership: GroupMembership - FilterName: FilterName - ChangeType: ChangeType - LayerName: LayerName - ServiceAccount: ServiceAccount - AttributeValue: ServiceAccount - SessionName: SessionName - TaskName: TaskName - ObjectDN: SessionName - TemplateContent: TemplateContent - NewTemplateContent: NewTemplateContent - SourcePort: TemplateContent - PasswordLastSet: PasswordLastSet - PrivilegeList: PrivilegeList - DeviceDescription: PasswordLastSet - TargetServerName: PrivilegeList - NewTargetUserName: DeviceDescription - OperationType: TargetServerName - DestPort: DestPort - ServiceStartType: OperationType - OldTargetUserName: OldTargetUserName - UserPrincipalName: ServiceStartType - Accesses: Accesses - DnsHostName: UserPrincipalName - DisableIntegrityChecks: AccessList - AuditSourceName: AuditSourceName - Workstation: Workstation - DestAddress: DestAddress - PreAuthType: Workstation - SecurityPackageName: SecurityPackageName - SubjectLogonId: SubjectLogonId - NewUacValue: NewUacValue - EnabledPrivilegeList: SubjectLogonId - RelativeTargetName: NewUacValue - CertSerialNumber: CertSerialNumber - SidHistory: RelativeTargetName - TargetLogonId: TargetLogonId - KernelDebug: SidHistory - CallerProcessName: TargetLogonId - ProcessName: ProcessName - Properties: CallerProcessName - UserAccountControl: ProcessName - RegistryValue: Properties - SecurityID: UserAccountControl - ServiceFileName: ServiceFileName - SecurityDescriptor: SecurityID - ServiceName: ServiceFileName - ShareName: SecurityDescriptor - NewValue: ServiceName - Source: ShareName - Status: NewValue - SubjectDomainName: SubjectDomainName - SubjectUserName: Status - SubjectUserSid: SubjectDomainName - SourceAddr: SubjectUserName - SourceAddress: SubjectUserSid - TargetName: TargetName - ServicePrincipalNames: ServicePrincipalNames - TargetDomainName: TargetName - TargetSid: TargetSid - TargetUserName: TargetDomainName - ObjectServer: TargetSid - TargetUserSid: TargetUserName - TicketEncryptionType: ObjectServer - TicketOptions: TargetUserSid - WorkstationName: WorkstationName - TransmittedServices: TransmittedServices - AuthenticationAlgorithm: WorkstationName - LayerRTID: TransmittedServices - BSSID: BSSID - BSSType: BSSType - CipherAlgorithm: CipherAlgorithm - ConnectionId: ConnectionId - ConnectionMode: ConnectionMode - InterfaceDescription: InterfaceDescription - InterfaceGuid: InterfaceGuid - OnexEnabled: OnexEnabled - PHYType: PHYType - ProfileName: ProfileName - SSID: SSID - Domain: Domain - ServiceType: ServiceType - SourceName: SourceName - StartType: ServiceType - UserID: UserID - ParentProcessName: ParentProcessName - Service: Service diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_sysmon.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_sysmon.yml deleted file mode 100644 index 3551f8d8..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_sysmon.yml +++ /dev/null @@ -1,61 +0,0 @@ -platform: Microsoft Sentinel -source: windows_sysmon -description: Text that describe current mapping - -log_source: - table: [SysmonEvent] - -default_log_source: - table: SysmonEvent - -field_mapping: - CommandLine: CommandLine - Image: Image - ParentImage: ParentImage - EventID: EventID - CallTrace: CallTrace - Company: Company - CurrentDirectory: CurrentDirectory - Description: Description - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationIsIpv6: DestinationIsIpv6 - DestinationPort: DestinationPort - DestinationPortName: DestinationPortName - Hashes: Hashes - Initiated: Initiated - IntegrityLevel: IntegrityLevel - ParentCommandLine: ParentCommandLine - Product: Product - Protocol: Protocol - RuleName: RuleName - SourceHostname: SourceHostname - SourceIp: SourceIp - SourceIsIpv6: SourceIsIpv6 - SourcePort: SourcePort - SourcePortName: SourcePortName - TargetFilename: TargetFilename - User: User - OriginalFileName: OriginalFileName - Signed: Signed - Signature: Signature - SignatureStatus: SignatureStatus - TargetObject: TargetObject - Details: Details - QueryName: QueryName - QueryResults: QueryResults - QueryStatus: QueryStatus - IsExecutable: IsExecutable - PipeName: PipeName - ImageLoaded: ImageLoaded - ImagePath: ImagePath - Imphash: Imphash - SourceImage: SourceImage - StartModule: StartModule - TargetImage: TargetImage - Device: Device - ProcessID: ProcessID - FileVersion: FileVersion - StartAddress: StartAddress - StartFunction: StartFunction - EventType: EventType \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_wmi_event.yml b/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_wmi_event.yml deleted file mode 100644 index c2f0dd01..00000000 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/windows_wmi_event.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Microsoft Sentinel -source: windows_wmi_event -description: Text that describe current mapping - -log_source: - table: [Event] - -default_log_source: - table: Event - -field_mapping: - Destination: Destination - EventID: EventID \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/aws_cloudtrail.yml b/siem-converter/app/converter/mappings/platforms/opensearch/aws_cloudtrail.yml deleted file mode 100644 index 73e0c6fe..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/aws_cloudtrail.yml +++ /dev/null @@ -1,35 +0,0 @@ -platform: OpenSearch -source: aws_cloudtrail -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - eventSource: event.provider - eventName: event.action - AdditionalEventData: AdditionalEventData - additionalEventData.MFAUsed: additionalEventData.MFAUsed - errorCode: errorCode - errorMessage: errorMessage - eventType: eventType - requestParameters: requestParameters - requestParameters.attribute: requestParameters.attribute - requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP - requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort - requestParameters.userData: requestParameters.userData - responseElements: responseElements - responseElements.ConsoleLogin: responseElements.ConsoleLogin - responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword - responseElements.publiclyAccessible: responseElements.publiclyAccessible - status: status - terminatingRuleId: terminatingRuleId - userAgent: userAgent - userIdentity.arn: userIdentity.arn - userIdentity.principalId: userIdentity.principalId - userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type - userIdentity.type: userIdentity.type - userIdentity.userName: userIdentity.userName diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/aws_eks.yml b/siem-converter/app/converter/mappings/platforms/opensearch/aws_eks.yml deleted file mode 100644 index c40bc05c..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/aws_eks.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: OpenSearch -source: aws_eks -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision - annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy - aws_node_type: aws_node_type - objectRef.namespace: objectRef.namespace - objectRef.resource: objectRef.resource - objectRef.subresource: objectRef.subresource - requestObject.rules.resources: requestObject.rules.resources - requestObject.rules.verbs: requestObject.rules.verbs - requestObject.spec.containers.image: requestObject.spec.containers.image - requestURI: requestURI - stage: stage - user.groups: user.groups - user.username: user.username - verb: verb diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/azure_AzureDiagnostics.yml b/siem-converter/app/converter/mappings/platforms/opensearch/azure_AzureDiagnostics.yml deleted file mode 100644 index d2136688..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/azure_AzureDiagnostics.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: OpenSearch -source: azure_AzureDiagnostics -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ResultDescription: ResultDescription - Category: Category \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/azure_BehaviorAnalytics.yml b/siem-converter/app/converter/mappings/platforms/opensearch/azure_BehaviorAnalytics.yml deleted file mode 100644 index 333964f1..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/azure_BehaviorAnalytics.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: OpenSearch -source: azure_BehaviorAnalytics -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ActionType: ActionType - ActivityInsights: ActivityInsights - ActivityType: ActivityType - EventSource: EventSource - DevicesInsights: DevicesInsights - RiskDetail: RiskDetail - UsersInsights: UsersInsights - UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount - UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/azure_aadnoninteractiveusersigninlogs.yml b/siem-converter/app/converter/mappings/platforms/opensearch/azure_aadnoninteractiveusersigninlogs.yml deleted file mode 100644 index 29fa38bb..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/azure_aadnoninteractiveusersigninlogs.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: OpenSearch -source: azure_aadnoninteractiveusersigninlogs -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - UserAgent: UserAgent - Type: Type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/azure_azureactivity.yml b/siem-converter/app/converter/mappings/platforms/opensearch/azure_azureactivity.yml deleted file mode 100644 index c9b4e4fa..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/azure_azureactivity.yml +++ /dev/null @@ -1,22 +0,0 @@ -platform: OpenSearch -source: azure_azureactivity -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ActivityStatus: ActivityStatus - ActivityStatusValue: ActivityStatusValue - ActivitySubstatusValue: ActivitySubstatusValue - Authorization: Authorization - Category: Category - CategoryValue: CategoryValue - OperationName: OperationName - OperationNameValue: OperationNameValue - ResourceId: ResourceId - ResourceProviderValue: ResourceProviderValue - Type: Type diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/azure_azuread.yml b/siem-converter/app/converter/mappings/platforms/opensearch/azure_azuread.yml deleted file mode 100644 index ee7f960f..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/azure_azuread.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: OpenSearch -source: azure_azuread -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ActivityDisplayName: event.action - Category: azure.auditlogs.properties.category - LoggedByService: azure.auditlogs.properties.logged_by_service - Result: event.outcome - OperationName: OperationName - TargetResources: TargetResources - AADOperationType: AADOperationType - InitiatedBy: InitiatedBy - ResultReason: ResultReason - Status: Status - Status.errorCode: Status.errorCode - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/azure_m365.yml b/siem-converter/app/converter/mappings/platforms/opensearch/azure_m365.yml deleted file mode 100644 index 3e79d528..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/azure_m365.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: OpenSearch -source: azure_m365 -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - ClientInfoString: ClientInfoString - LogonError: LogonError - ModifiedProperties: ModifiedProperties - OfficeObjectId: OfficeObjectId - OfficeWorkload: OfficeWorkload - Operation: Operation - Parameters: Parameters - RecordType: RecordType - ResultStatus: ResultStatus - SourceFileExtension: SourceFileExtension - SourceFileName: SourceFileName - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/azure_signinlogs.yml b/siem-converter/app/converter/mappings/platforms/opensearch/azure_signinlogs.yml deleted file mode 100644 index f77a7cbe..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/azure_signinlogs.yml +++ /dev/null @@ -1,29 +0,0 @@ -platform: OpenSearch -source: azure_signinlogs -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - AppDisplayName: AppDisplayName - AppId: AppId - AuthenticationRequirement: AuthenticationRequirement - Category: Category - ConditionalAccessStatus: ConditionalAccessStatus - DeviceDetail: DeviceDetail - IsInteractive: IsInteractive - NetworkLocationDetails: NetworkLocationDetails - ResourceDisplayName: ResourceDisplayName - ResourceIdentity: ResourceIdentity - ResultDescription: ResultDescription - ResultType: ResultType - Status.errorCode: Status.errorCode - Status: Status - Status.failureReason: Status.failureReason - TokenIssuerType: TokenIssuerType - UserAgent: UserAgent - UserPrincipalName: UserPrincipalName diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/default.yml b/siem-converter/app/converter/mappings/platforms/opensearch/default.yml deleted file mode 100644 index 0110799d..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/default.yml +++ /dev/null @@ -1,6 +0,0 @@ -platform: OpenSearch -source: default -description: Text that describe current mapping - -default_log_source: - index: "*" \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/dns.yml b/siem-converter/app/converter/mappings/platforms/opensearch/dns.yml deleted file mode 100644 index 8bf6f107..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/dns.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: OpenSearch -source: dns -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - dns-query: dns.question.name - parent-domain: - - url.registered_domain - - destination.registered_domain - dns-answer: dns.answers.name - dns-record: dns.question.type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/firewall.yml b/siem-converter/app/converter/mappings/platforms/opensearch/firewall.yml deleted file mode 100644 index 87fbc72c..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/firewall.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: OpenSearch -source: firewall -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - src-ip: - - source.address - - source.ip - src-port: source.port - dst-ip: - - destination.address - - destination.ip - dst-port: destination.port \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/gcp_gcp.audit.yml b/siem-converter/app/converter/mappings/platforms/opensearch/gcp_gcp.audit.yml deleted file mode 100644 index f027df1e..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/gcp_gcp.audit.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: OpenSearch -source: gcp_gcp.audit -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - backupConfiguration.enabled: backupConfiguration.enabled - gcp.audit.method_name: gcp.audit.method_name - jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction - jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue - message: message - methodName: methodName - protoPayload.methodName: protoPayload.methodName - protoPayload.request.canIpForward: protoPayload.request.canIpForward - protoPayload.serviceName: protoPayload.serviceName - serviceName: serviceName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/gcp_pubsub.yml b/siem-converter/app/converter/mappings/platforms/opensearch/gcp_pubsub.yml deleted file mode 100644 index c57c099a..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/gcp_pubsub.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: OpenSearch -source: gcp_pubsub -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - data.protoPayload.methodName: data.protoPayload.methodName - data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member - data.resource.type: data.resource.type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/linux_auditd.yml b/siem-converter/app/converter/mappings/platforms/opensearch/linux_auditd.yml deleted file mode 100644 index bdb34b90..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/linux_auditd.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: OpenSearch -source: linux_auditd -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - a0: process.args - a1: process.args - a2: process.args - a3: process.args - exe: process.executable diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/linux_dns_query.yml b/siem-converter/app/converter/mappings/platforms/opensearch/linux_dns_query.yml deleted file mode 100644 index 152c9654..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/linux_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: OpenSearch -source: linux_dns_query -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - Image: process.executable - User: user.name - QueryName: dns.question.name - QueryResults: dns.answers \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/linux_process_creation.yml b/siem-converter/app/converter/mappings/platforms/opensearch/linux_process_creation.yml deleted file mode 100644 index fc9d2846..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/linux_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: OpenSearch -source: linux_process_creation -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - CommandLine: process.command_line - Image: process.executable - ParentCommandLine: process.parent.command_line - ParentImage: process.parent.executable diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/macos_dns_query.yml b/siem-converter/app/converter/mappings/platforms/opensearch/macos_dns_query.yml deleted file mode 100644 index 237cb23c..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/macos_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: OpenSearch -source: macos_dns_query -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - Image: process.executable - User: user.name - QueryName: dns.question.name - QueryResults: dns.answers \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/macos_network_connection.yml b/siem-converter/app/converter/mappings/platforms/opensearch/macos_network_connection.yml deleted file mode 100644 index f26025f0..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/macos_network_connection.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: OpenSearch -source: macos_network_connection -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - Image: process.executable - DestinationHostname: destination.domain - DestinationIp: destination.ip - DestinationPort: destination.port - SourceIp: source.ip - SourcePort: source.port - ParentImage: process.parent.executable diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/macos_process_creation.yml b/siem-converter/app/converter/mappings/platforms/opensearch/macos_process_creation.yml deleted file mode 100644 index e4cedd81..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/macos_process_creation.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: OpenSearch -source: macos_process_creation -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - CommandLine: process.command_line - Image: process.executable - ParentCommandLine: process.parent.command_line - ParentImage: process.parent.executable diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/okta_okta.yml b/siem-converter/app/converter/mappings/platforms/opensearch/okta_okta.yml deleted file mode 100644 index 2d5e7523..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/okta_okta.yml +++ /dev/null @@ -1,33 +0,0 @@ -platform: OpenSearch -source: okta_okta -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - client.user.id: okta.actor.id - source.user.id: okta.actor.id - User: okta.actor.type - alternateId: okta.actor.alternate_id - client.user.full_name: okta.actor.display_name - source.user.full_name: okta.actor.display_name - related.user: okta.actor.display_name - client.ip: okta.client.ip - source.ip: okta.client.ip - user_agent.original: okta.client.user_agent.raw_user_agent - userAgent.os: okta.client.user_agent.os - userAgent.browser: okta.client.user_agent.browser - client.zone: okta.client.zone - client.device: okta.client.device - client.id: okta.client.id - event.action: okta.event_type - outcome.reason: okta.outcome.reason - event.outcome: okta.event.outcome - client.as.number: okta.security_context.as.number - client.as.organization.name: okta.security_context.as.organization.name - client.domain: okta.security_context.isp - source.domain: okta.security_context.domain diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/proxy.yml b/siem-converter/app/converter/mappings/platforms/opensearch/proxy.yml deleted file mode 100644 index 5a621952..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/proxy.yml +++ /dev/null @@ -1,30 +0,0 @@ -platform: OpenSearch -source: proxy -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - c-uri: url.original - c-useragent: user_agent.original - cs-method: http.request.method - cs-bytes: http.request.body.bytes - cs-cookie-vars: http.cookie_vars - c-uri-extension: url.extension - c-uri-query: url.query - cs-cookie: http.cookie_vars - cs-host: - - url.domain - - destination.domain - - cs-referrer: http.request.referrer - cs-version: http.version - r-dns: - - destination.domain - - url.domain - sc-status: http.response.status_code - post-body: http.post_body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/webserver.yml b/siem-converter/app/converter/mappings/platforms/opensearch/webserver.yml deleted file mode 100644 index 5e87ab77..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/webserver.yml +++ /dev/null @@ -1,29 +0,0 @@ -platform: OpenSearch -source: webserver -description: Text that describe current mapping - -log_source: - index: [logs-*] - -default_log_source: - index: logs-* - -field_mapping: - c-uri: url.original - c-useragent: user_agent.original - cs-method: http.request.method - cs-bytes: http.request.body.bytes - cs-cookie-vars: http.cookie_vars - c-uri-extension: url.extension - c-uri-query: url.query - cs-cookie: http.cookie_vars - cs-host: - - url.domain - - destination.domain - cs-referrer: http.request.referrer - cs-version: http.version - r-dns: - - destination.domain - - url.domain - sc-status: http.response.status_code - post-body: http.post_body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_bits_client.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_bits_client.yml deleted file mode 100644 index 270299fa..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_bits_client.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: OpenSearch -source: windows_bits_client -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - LocalName: winlog.event_data.LocalName - EventID: - - event.code - - winlog.event_id - RemoteName: winlog.event_data.RemoteName - processPath: winlog.event_data.processPath diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_dns_query.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_dns_query.yml deleted file mode 100644 index 999af290..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_dns_query.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: OpenSearch -source: macos_dns_query -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - Image: process.executable - User: - - user.name - - winlog.user.domain - - winlog.user.name - QueryName: dns.question.name - QueryResults: dns.answers \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_driver_load.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_driver_load.yml deleted file mode 100644 index bdaf6ad7..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_driver_load.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: OpenSearch -source: windows_driver_load -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - ImageLoaded: file.path - ImagePath: file.path - Hashes: winlog.event_data.Hashes \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_image_load.yml deleted file mode 100644 index db5648b6..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_image_load.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: OpenSearch -source: windows_image_load -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - Image: process.executable - ImageLoaded: file.path - SignatureStatus: SignatureStatus - OriginalFileName: OriginalFileName - Signed: Signed \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_ldap_debug.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_ldap_debug.yml deleted file mode 100644 index 39c86fcc..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_ldap_debug.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: OpenSearch -source: windows_ldap_debug -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - EventID: - - event.code - - winlog.event_id - SearchFilter: winlog.event_data.SearchFilter \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_network_connection.yml deleted file mode 100644 index 0db4aba0..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_network_connection.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: OpenSearch -source: windows_network_connection -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - Image: process.executable - DestinationHostname: destination.domain - DestinationIp: destination.ip - DestinationPort: destination.port - SourceIp: source.ip - SourcePort: source.port diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_ntlm.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_ntlm.yml deleted file mode 100644 index dd051ca2..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_ntlm.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: OpenSearch -source: windows_ntlm -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - ProcessName: ProcessName - WorkstationName: winlog.event_data.WorkstationName - EventID: - - event.code - - winlog.event_id - TargetName: winlog.event_data.TargetName diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_powershell.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_powershell.yml deleted file mode 100644 index 74ad5991..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_powershell.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: OpenSearch -source: windows_powershell -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - EventID: - - event.code - - winlog.event_id - CommandLine: powershell.command.value - ScriptBlockText: powershell.file.script_block_text - Payload: - - powershell.command.invocation_details - - winlog.event_data.Payload - HostApplication: process.command_line - ContextInfo: winlog.event_data.ContextInfo - HostName: process.title - EngineVersion: - - powershell.engine.version - - powershell.process.executable_version \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_process_creation.yml deleted file mode 100644 index 3111099d..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_process_creation.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: OpenSearch -source: windows_process_creation -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - CommandLine: process.command_line - CurrentDirectory: CurrentDirectory - Hashes: Hashes - Image: process.executable - IntegrityLevel: IntegrityLevel - ParentCommandLine: process.parent.command_line - ParentImage: process.parent.executable - ParentUser: ParentUser - Product: Product - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_security.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_security.yml deleted file mode 100644 index 158bfffa..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_security.yml +++ /dev/null @@ -1,149 +0,0 @@ -platform: OpenSearch -source: windows_security -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - EventID: - - event.code - - winlog.event_id - ParentImage: process.parent.executable - AccessMask: winlog.event_data.AccessMask - AccountName: winlog.event_data.AccountName - AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo - AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName - AuditPolicyChanges: winlog.event_data.AuditPolicyChanges - AuthenticationPackageName: winlog.event_data.AuthenticationPackageName - CallingProcessName: winlog.event_data.CallingProcessName - Channel: winlog.channel - ComputerName: winlog.ComputerName - EventType: winlog.event_data.EventType - FailureReason: winlog.event_data.FailureCode - FileName: file.path - GrantedAccess: winlog.event_data.GrantedAccess - Hashes: winlog.event_data.Hashes - HiveName: winlog.event_data.HiveName - IpAddress: source.ip - IpPort: source.port - KeyLength: winlog.event_data.KeyLength - LogonProcessName: winlog.event_data.LogonProcessName - LogonType: winlog.event_data.LogonType - LinkName: LinkName - MemberName: winlog.event_data.MemberName - MemberSid: winlog.event_data.MemberSid - NewProcessName: winlog.event_data.NewProcessName - ObjectClass: winlog.event_data.ObjectClass - ObjectName: winlog.event_data.ObjectName - ObjectType: winlog.event_data.ObjectType - ObjectValueName: winlog.event_data.ObjectValueName - Path: winlog.event_data.Path - CommandLine: winlog.event_data.ProcessCommandLine - OldUacValue: winlog.event_data.OldUacValue - CertIssuerName: winlog.event_data.CertIssuerName - SubStatus: winlog.event_data.SubStatus - DisplayName: winlog.event_data.DisplayName - TaskContent: winlog.event_data.TaskContent - ServiceSid: winlog.event_data.ServiceSid - CertThumbprint: winlog.event_data.CertThumbprint - ClassName: winlog.event_data.ClassName - NotificationPackageName: winlog.event_data.NotificationPackageName - NewSd: winlog.event_data.NewSd - TestSigning: winlog.event_data.TestSigning - TargetInfo: winlog.event_data.TargetInfo - ClientProcessId: winlog.event_data.ClientProcessId - ParentProcessId: winlog.event_data.ParentProcessId - AccessList: winlog.event_data.AccessList - GroupMembership: winlog.event_data.GroupMembership - FilterName: winlog.event_data.FilterName - ChangeType: winlog.event_data.ChangeType - LayerName: winlog.event_data.LayerName - ServiceAccount: winlog.event_data.ServiceAccount - AttributeValue: winlog.event_data.AttributeValue - SessionName: winlog.event_data.SessionName - TaskName: winlog.event_data.TaskName - ObjectDN: winlog.event_data.ObjectDN - TemplateContent: winlog.event_data.TemplateContent - NewTemplateContent: winlog.event_data.NewTemplateContent - SourcePort: winlog.event_data.SourcePort - PasswordLastSet: winlog.event_data.PasswordLastSet - PrivilegeList: winlog.event_data.PrivilegeList - DeviceDescription: winlog.event_data.DeviceDescription - TargetServerName: winlog.event_data.TargetServerName - NewTargetUserName: winlog.event_data.NewTargetUserName - OperationType: winlog.event_data.OperationType - DestPort: winlog.event_data.DestPort - ServiceStartType: winlog.event_data.ServiceStartType - OldTargetUserName: winlog.event_data.OldTargetUserName - UserPrincipalName: winlog.event_data.UserPrincipalName - Accesses: winlog.event_data.Accesses - DnsHostName: winlog.event_data.DnsHostName - DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks - AuditSourceName: winlog.event_data.AuditSourceName - Workstation: winlog.event_data.Workstation - DestAddress: winlog.event_data.DestAddress - PreAuthType: winlog.event_data.PreAuthType - SecurityPackageName: winlog.event_data.SecurityPackageName - SubjectLogonId: winlog.event_data.SubjectLogonId - NewUacValue: winlog.event_data.NewUacValue - EnabledPrivilegeList: winlog.event_data.EnabledPrivilegeList - RelativeTargetName: winlog.event_data.RelativeTargetName - CertSerialNumber: winlog.event_data.CertSerialNumber - SidHistory: winlog.event_data.SidHistory - TargetLogonId: winlog.event_data.TargetLogonId - KernelDebug: winlog.event_data.KernelDebug - CallerProcessName: winlog.event_data.CallerProcessName - ProcessName: process.executable - Properties: winlog.event_data.Properties - UserAccountControl: winlog.event_data.UserAccountControl - RegistryValue: winlog.event_data.RegistryValue - SecurityID: winlog.event_data.SecurityID - ServiceFileName: winlog.event_data.ServiceFileName - SecurityDescriptor: - - winlog.event_data.SecurityDescriptor - - winlog.event_data.TaskContentNew - ServiceName: winlog.event_data.ServiceName - ShareName: winlog.event_data.ShareName - NewValue: winlog.event_data.NewValue - Source: winlog.event_data.Source - Status: winlog.event_data.Status - SubjectDomainName: winlog.event_data.SubjectDomainName - SubjectUserName: winlog.event_data.SubjectUserName - SubjectUserSid: winlog.event_data.SubjectUserSid - SourceAddr: SourceAddr - SourceAddress: SourceAddress - TargetName: TargetName - ServicePrincipalNames: winlog.event_data.ServicePrincipalNames - TargetDomainName: winlog.event_data.TargetDomainName - TargetSid: winlog.event_data.TargetSid - TargetUserName: winlog.event_data.TargetUserName - ObjectServer: - TargetUserSid: winlog.event_data.TargetUserSid - TicketEncryptionType: winlog.event_data.TicketEncryptionType - TicketOptions: winlog.event_data.TicketOptions - WorkstationName: source.domain - TransmittedServices: winlog.event_data.TransmittedServices - AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm - LayerRTID: LayerRTID - BSSID: winlog.event_data.BSSID - BSSType: winlog.event_data.BSSType - CipherAlgorithm: winlog.event_data.CipherAlgorithm - ConnectionId: winlog.event_data.ConnectionId - ConnectionMode: winlog.event_data.ConnectionMode - InterfaceDescription: winlog.event_data.InterfaceDescription - InterfaceGuid: winlog.event_data.InterfaceGuid - OnexEnabled: winlog.event_data.OnexEnabled - PHYType: winlog.event_data.PHYType - ProfileName: winlog.event_data.ProfileName - SSID: winlog.event_data.SSID - Domain: winlog.event_data.Domain - ServiceType: winlog.event_data.ServiceType - SourceName: winlog.event_data.SourceName - StartType: winlog.event_data.StartType - UserID: winlog.event_data.UserID - ParentProcessName: process.parent.name - Service: Service diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_sysmon.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_sysmon.yml deleted file mode 100644 index e9ed8464..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_sysmon.yml +++ /dev/null @@ -1,67 +0,0 @@ -platform: OpenSearch -source: windows_sysmon -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - - -field_mapping: - CommandLine: process.command_line - Image: process.executable - ParentImage: process.parent.executable - EventID: - - event.code - - winlog.event_id - CallTrace: winlog.event_data.CallTrace - Company: winlog.event_data.Company - CurrentDirectory: process.working_directory - Description: winlog.event_data.Description - DestinationHostname: destination.domain - DestinationIp: destination.ip - DestinationIsIpv6: winlog.event_data.DestinationIsIpv6 - DestinationPort: destination.port - DestinationPortName: network.protocol - Hashes: winlog.event_data.Hashes - Initiated: winlog.event_data.Initiated - IntegrityLevel: winlog.event_data.IntegrityLevel - ParentCommandLine: process.parent.command_line - Product: winlog.event_data.Product - Protocol: winlog.event_data.Protocol - RuleName: winlog.event_data.RuleName - SourceHostname: source.domain - SourceIp: source.ip - SourceIsIpv6: winlog.event_data.SourceIsIpv6 - SourcePort: source.port - SourcePortName: winlog.event_data.SourcePortName - TargetFilename: file.path - User: - - user.name - - winlog.user.domain - - winlog.user.name - OriginalFileName: winlog.event_data.OriginalFileName - Signed: winlog.event_data.Signed - Signature: winlog.event_data.Signature - SignatureStatus: winlog.event_data.SignatureStatus - TargetObject: winlog.event_data.TargetObject - Details: winlog.event_data.Details - QueryName: dns.question.name - QueryResults: dns.answers - QueryStatus: winlog.event_data.QueryStatus - IsExecutable: winlog.event_data.IsExecutable - PipeName: file.name - ImageLoaded: file.path - ImagePath: winlog.event_data.ImagePath - Imphash: winlog.event_data.Imphash - SourceImage: process.executable - StartModule: winlog.event_data.StartModule - TargetImage: winlog.event_data.TargetImage - Device: Device - ProcessID: ProcessID - FileVersion: FileVersion - StartAddress: StartAddress - StartFunction: StartFunction - EventType: winlog.event_data.EventType \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/opensearch/windows_wmi_event.yml b/siem-converter/app/converter/mappings/platforms/opensearch/windows_wmi_event.yml deleted file mode 100644 index f4670f3b..00000000 --- a/siem-converter/app/converter/mappings/platforms/opensearch/windows_wmi_event.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: OpenSearch -source: windows_wmi_event -description: Text that describe current mapping - -log_source: - index: [winlogbeat-*, logs-*] - -default_log_source: - index: winlogbeat-* - -field_mapping: - Destination: Destination - EventID: - - event.code - - winlog.event_id diff --git a/siem-converter/app/converter/mappings/platforms/qradar/aws_cloudtrail.yml b/siem-converter/app/converter/mappings/platforms/qradar/aws_cloudtrail.yml deleted file mode 100644 index 193ace66..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/aws_cloudtrail.yml +++ /dev/null @@ -1,35 +0,0 @@ -platform: Qradar -source: aws_cloudtrail -description: Text that describe current mapping - -log_source: - devicetype: [347] - -default_log_source: - devicetype: 347 - -field_mapping: - eventSource: eventSource - eventName: eventName - AdditionalEventData: AdditionalEventData - additionalEventData.MFAUsed: additionalEventData.MFAUsed - errorCode: errorCode - errorMessage: errorMessage - eventType: eventType - requestParameters: requestParameters - requestParameters.attribute: requestParameters.attribute - requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP - requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort - requestParameters.userData: requestParameters.userData - responseElements: responseElements - responseElements.ConsoleLogin: responseElements.ConsoleLogin - responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword - responseElements.publiclyAccessible: responseElements.publiclyAccessible - status: status - terminatingRuleId: terminatingRuleId - userAgent: userAgent - userIdentity.arn: userIdentity.arn - userIdentity.principalId: userIdentity.principalId - userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type - userIdentity.type: userIdentity.type - userIdentity.userName: userIdentity.userName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/aws_eks.yml b/siem-converter/app/converter/mappings/platforms/qradar/aws_eks.yml deleted file mode 100644 index bf7e2d70..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/aws_eks.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Qradar -source: aws_eks -description: Text that describe current mapping - -log_source: - devicetype: [382] - -default_log_source: - devicetype: 382 - -field_mapping: - annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision - annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy - aws_node_type: aws_node_type - objectRef.namespace: objectRef.namespace - objectRef.resource: objectRef.resource - objectRef.subresource: objectRef.subresource - requestObject.rules.resources: requestObject.rules.resources - requestObject.rules.verbs: requestObject.rules.verbs - requestObject.spec.containers.image: requestObject.spec.containers.image - requestURI: requestURI - stage: stage - user.groups: user.groups - user.username: user.username - verb: verb \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/azure_azureactivity.yml b/siem-converter/app/converter/mappings/platforms/qradar/azure_azureactivity.yml deleted file mode 100644 index 900456d4..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/azure_azureactivity.yml +++ /dev/null @@ -1,22 +0,0 @@ -platform: Qradar -source: azure_azureactivity -description: Text that describe current mapping - -log_source: - devicetype: [413] - -default_log_source: - devicetype: 413 - -field_mapping: - ActivityStatus: ActivityStatus - ActivityStatusValue: ActivityStatusValue - ActivitySubstatusValue: ActivitySubstatusValue - Authorization: Authorization - Category: Category - CategoryValue: CategoryValue - OperationName: OperationName - OperationNameValue: OperationNameValue - ResourceId: ResourceId - ResourceProviderValue: ResourceProviderValue - Type: Type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/azure_azuread.yml b/siem-converter/app/converter/mappings/platforms/qradar/azure_azuread.yml deleted file mode 100644 index 9795331a..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/azure_azuread.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Qradar -source: azure_azuread -description: Text that describe current mapping - -log_source: - devicetype: [445] - -default_log_source: - devicetype: 445 - -field_mapping: - ActivityDisplayName: ActivityDisplayName - Category: Category - LoggedByService: LoggedByService - Result: Result - OperationName: OperationName - TargetResources: TargetResources - AADOperationType: AADOperationType - InitiatedBy: InitiatedBy - ResultReason: ResultReason - Status: Status - Status.errorCode: Status.errorCode - UserAgent: UserAgent \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/azure_m365.yml b/siem-converter/app/converter/mappings/platforms/qradar/azure_m365.yml deleted file mode 100644 index f8741fce..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/azure_m365.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Qradar -source: azure_m365 -description: Text that describe current mapping - -log_source: - devicetype: [397] - -default_log_source: - devicetype: 397 - -field_mapping: - ClientInfoString: ClientInfoString - LogonError: LogonError - ModifiedProperties: ModifiedProperties - OfficeObjectId: OfficeObjectId - OfficeWorkload: OfficeWorkload - Operation: Operation - Parameters: - RecordType: RecordType - ResultStatus: ResultStatus - SourceFileExtension: SourceFileExtension - SourceFileName: SourceFileName - UserAgent: UserAgent \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/azure_signinlogs.yml b/siem-converter/app/converter/mappings/platforms/qradar/azure_signinlogs.yml deleted file mode 100644 index 63fcbea0..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/azure_signinlogs.yml +++ /dev/null @@ -1,29 +0,0 @@ -platform: Qradar -source: azure_signinlogs -description: Text that describe current mapping - -log_source: - devicetype: [413] - -default_log_source: - devicetype: 413 - -field_mapping: - AppDisplayName: AppDisplayName - AppId: AppId - AuthenticationRequirement: AuthenticationRequirement - Category: Category - ConditionalAccessStatus: ConditionalAccessStatus - DeviceDetail: DeviceDetail - IsInteractive: IsInteractive - NetworkLocationDetails: NetworkLocationDetails - ResourceDisplayName: ResourceDisplayName - ResourceIdentity: ResourceIdentity - ResultDescription: ResultDescription - ResultType: ResultType - Status.errorCode: Status.errorCode - Status: Status - Status.failureReason: Status.failureReason - TokenIssuerType: TokenIssuerType - UserAgent: UserAgent - UserPrincipalName: UserPrincipalName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/default.yml b/siem-converter/app/converter/mappings/platforms/qradar/default.yml deleted file mode 100644 index 34f8378e..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/default.yml +++ /dev/null @@ -1,10 +0,0 @@ -platform: Qradar -source: default -description: Text that describe current mapping - -log_source: - devicetype: - - 12 - -default_log_source: - devicetype: 12 \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/dns.yml b/siem-converter/app/converter/mappings/platforms/qradar/dns.yml deleted file mode 100644 index 3e3f6aec..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/dns.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Qradar -source: dns -description: Text that describe current mapping - -log_source: - devicetype: [185, 384] - -default_log_source: - devicetype: 185 - -field_mapping: - dns-query: dns-query - parent-domain: parent-domain - dns-answer: dns-answer - dns-record: dns-record \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/firewall.yml b/siem-converter/app/converter/mappings/platforms/qradar/firewall.yml deleted file mode 100644 index 518a229c..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/firewall.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Qradar -source: firewall -description: Text that describe current mapping - -log_source: - devicetype: [4, 5, 6, 41, 49, 73, 82, 104, 120, 194, 206, 278, 296, 307, 331, 340, 455, 456] - -default_log_source: - devicetype: 4 - -field_mapping: - src-ip: sourceip - src-port: sourceport - dst-ip: destinationip - dst-port: sestinationport \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/gcp_gcp.audit.yml b/siem-converter/app/converter/mappings/platforms/qradar/gcp_gcp.audit.yml deleted file mode 100644 index 623e2db6..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/gcp_gcp.audit.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: Qradar -source: gcp_gcp.audit -description: Text that describe current mapping - -log_source: - devicetype: [449] - -default_log_source: - devicetype: 449 - -field_mapping: - backupConfiguration.enabled: backupConfiguration.enabled - gcp.audit.method_name: gcp.audit.method_name - jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction - jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue - message: message - methodName: methodName - protoPayload.methodName: protoPayload.methodName - protoPayload.request.canIpForward: protoPayload.request.canIpForward - protoPayload.serviceName: protoPayload.serviceName - serviceName: serviceName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/linux_auditd.yml b/siem-converter/app/converter/mappings/platforms/qradar/linux_auditd.yml deleted file mode 100644 index 2ef6d9ec..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/linux_auditd.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Qradar -source: linux_auditd -description: Text that describe current mapping - -log_source: - devicetype: [11] - -default_log_source: - devicetype: 11 - -field_mapping: - a0: a0 - a1: a1 - a2: a2 - a3: a3 - exe: exe \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/linux_dns_query.yml b/siem-converter/app/converter/mappings/platforms/qradar/linux_dns_query.yml deleted file mode 100644 index 7b0d6785..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/linux_dns_query.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Qradar -source: linux_dns_query -description: Text that describe current mapping - -log_source: - devicetype: [11] - category: [18081] - -default_log_source: - devicetype: 11 - category: 18081 - -field_mapping: - Image: Process Path - User: username - QueryName: URL - QueryResults: URL \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/linux_file_event.yml b/siem-converter/app/converter/mappings/platforms/qradar/linux_file_event.yml deleted file mode 100644 index 3fb52c56..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/linux_file_event.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Qradar -source: linux_file_event -description: Text that describe current mapping - -log_source: - devicetype: [11] - category: [8028] - -default_log_source: - devicetype: 11 - category: 8028 - -field_mapping: - Image: Process Path - TargetFilename: Filename \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/linux_network_connection.yml b/siem-converter/app/converter/mappings/platforms/qradar/linux_network_connection.yml deleted file mode 100644 index 226f5e15..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/linux_network_connection.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: Qradar -source: linux_network_connection -description: Text that describe current mapping - -log_source: - devicetype: [11] - category: [4012] - -default_log_source: - devicetype: 11 - category: [4012] - -field_mapping: - CommandLine: Command - Image: Process Path - DestinationHostname: Destination Hostname - DestinationIp: destinationip - DestinationPort: destinationport - SourceIp: sourceip - SourcePort: sourceport - ParentImage: Parent Process Path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/linux_process_creation.yml b/siem-converter/app/converter/mappings/platforms/qradar/linux_process_creation.yml deleted file mode 100644 index ec7fa6ea..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/linux_process_creation.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Qradar -source: linux_process_creation -description: Text that describe current mapping - -log_source: - devicetype: [11] - category: [8110] - -default_log_source: - devicetype: 11 - category: 8110 - -field_mapping: - CommandLine: Command - Image: Process Path - ParentCommandLine: Parent Command - ParentImage: Parent Process Path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/macos_dns_query.yml b/siem-converter/app/converter/mappings/platforms/qradar/macos_dns_query.yml deleted file mode 100644 index 1c518abe..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/macos_dns_query.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Qradar -source: macos_dns_query -description: Text that describe current mapping - -log_source: - devicetype: [102] - category: [18081] - -default_log_source: - devicetype: 102 - category: 18081 - -field_mapping: - Image: Process Path - User: username - QueryName: URL - QueryResults: URL \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/macos_file_event.yml b/siem-converter/app/converter/mappings/platforms/qradar/macos_file_event.yml deleted file mode 100644 index f6b53dbd..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/macos_file_event.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Qradar -source: macos_file_event -description: Text that describe current mapping - -log_source: - devicetype: [102] - category: [8028] - -default_log_source: - devicetype: 102 - category: 8028 - -field_mapping: - Image: Process Path - TargetFilename: Filename \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/macos_network_connection.yml b/siem-converter/app/converter/mappings/platforms/qradar/macos_network_connection.yml deleted file mode 100644 index 87923f4d..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/macos_network_connection.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: Qradar -source: macos_network_connection -description: Text that describe current mapping - -log_source: - devicetype: [102] - category: [4012] - -default_log_source: - devicetype: 102 - category: [4012] - -field_mapping: - CommandLine: Command - Image: Process Path - DestinationHostname: Destination Hostname - DestinationIp: destinationip - DestinationPort: destinationport - SourceIp: sourceip - SourcePort: sourceport - ParentImage: Parent Process Path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/macos_process_creation.yml b/siem-converter/app/converter/mappings/platforms/qradar/macos_process_creation.yml deleted file mode 100644 index ff5f2d62..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/macos_process_creation.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Qradar -source: macos_process_creation -description: Text that describe current mapping - -log_source: - devicetype: [102] - category: [8110] - -default_log_source: - devicetype: 102 - category: 8110 - -field_mapping: - CommandLine: Command - Image: Process Path - ParentCommandLine: Parent Command - ParentImage: Parent Process Path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/okta_okta.yml b/siem-converter/app/converter/mappings/platforms/qradar/okta_okta.yml deleted file mode 100644 index 3e52ca49..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/okta_okta.yml +++ /dev/null @@ -1,33 +0,0 @@ -platform: Qradar -source: okta_okta -description: Text that describe current mapping - -log_source: - devicetype: [382] - -default_log_source: - devicetype: 382 - -field_mapping: - client.user.id: client.user.id - source.user.id: source.user.id - User: User - alternateId: alternateId - client.user.full_name: client.user.full_name - source.user.full_name: source.user.full_name - related.user: related.user - client.ip: client.ip - source.ip: source.ip - user_agent.original: user_agent.original - userAgent.os: userAgent.os - userAgent.browser: userAgent.browser - client.zone: client.zone - client.device: client.device - client.id: client.id - event.action: event.action - outcome.reason: outcome.reason - event.outcome: event.outcome - client.as.number: client.as.number - client.as.organization.name: client.as.organization.name - client.domain: client.domain - source.domain: source.domain \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/proxy.yml b/siem-converter/app/converter/mappings/platforms/qradar/proxy.yml deleted file mode 100644 index a8a79b2e..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/proxy.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Qradar -source: proxy -description: Text that describe current mapping - -log_source: - devicetype: [46, 103, 154, 191, 206, 200] - -default_log_source: - devicetype: 46 - -field_mapping: - c-uri: URL - c-useragent: c-useragent - cs-method: cs-method - cs-bytes: Bytes Sent - cs-cookie-vars: cs-cookie-vars - c-uri-extension: URL - c-uri-query: URL - cs-cookie: cs-cookie - cs-host: cs-host - cs-referrer: URL Referrer - cs-version: cs-version - r-dns: r-dns - sc-status: sc-status - post-body: post-body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/webserver.yml b/siem-converter/app/converter/mappings/platforms/qradar/webserver.yml deleted file mode 100644 index a33ce837..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/webserver.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Qradar -source: webserver -description: Text that describe current mapping - -log_source: - devicetype: [10, 13, 239, 439] - -default_log_source: - devicetype: 10 - -field_mapping: - c-uri: URL - c-useragent: c-useragent - cs-method: cs-method - cs-bytes: Bytes Sent - cs-cookie-vars: cs-cookie-vars - c-uri-extension: c-uri-extension - c-uri-query: URL - cs-cookie: cs-cookie - cs-host: cs-host - cs-referrer: URL Referrer - cs-version: cs-version - r-dns: r-dns - sc-status: sc-status - post-body: post-body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_application.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_application.yml deleted file mode 100644 index ab504a51..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_application.yml +++ /dev/null @@ -1,28 +0,0 @@ -platform: Qradar -source: windows_application -description: Text that describe current mapping - -log_source: - devicetype: [12] - qideventcategory: [Application_Information, Application_Warning, Application_Error] - -default_log_source: - devicetype: 12 - qideventcategory: Application_Information - -field_mapping: - EventID: Event ID - Provider_Name: Provider_Name - Level: Level - src_ip: sourceip - source: source - additional_information: additional_information - EventData: EventData - Channel: Channel - statement: statement - Faulting application path: Faulting application path - object_name: Object Name - class_type: class_type - action_id: action_id - Data: Data - Message: Message \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_create_remote_thread.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_create_remote_thread.yml deleted file mode 100644 index afcb2d4c..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_create_remote_thread.yml +++ /dev/null @@ -1,22 +0,0 @@ -platform: Qradar -source: windows_create_remote_thread -description: Text that describe current mapping - -log_source: - devicetype: [12] - qid: [5001845] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - qid: 5001845 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - SourceImage: Source Process Path - TargetImage: Target Process Path - StartModule: Start Module - StartAddress: Start Address - StartFunction: Start Function - Image: Process Path - Hashes: File Hash diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_create_stream_hash.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_create_stream_hash.yml deleted file mode 100644 index 73eb7c89..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_create_stream_hash.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Qradar -source: windows_create_stream_hash -description: Text that describe current mapping - -log_source: - devicetype: [12] - qid: [5001834] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - qid: 5001834 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - Image: Process Path - Hashes: File Hash \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_dns_query.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_dns_query.yml deleted file mode 100644 index 87eacd1d..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_dns_query.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: Qradar -source: windows_dns_query -description: Text that describe current mapping - -log_source: - devicetype: [12] - category: [18081] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - category: 18081 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - Image: Process Path - User: username - QueryName: URL - QueryResults: URL \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_driver_load.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_driver_load.yml deleted file mode 100644 index b9e335f6..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_driver_load.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Qradar -source: windows_driver_load -description: Text that describe current mapping - -log_source: - devicetype: [12] - qid: [5001843] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - qid: 5001843 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - ImageLoaded: Process Path - ImagePath: ImagePath - Hashes: File Hash \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_file_event.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_file_event.yml deleted file mode 100644 index 23c4d5cd..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_file_event.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Qradar -source: windows_file_event -description: Text that describe current mapping - -log_source: - devicetype: [12] - category: [8028] - -default_log_source: - devicetype: 12 - category: 8028 - -field_mapping: - CreationUtcTime: endtime - FileName: Filename - Image: Process Path - OriginalFilename: OriginalFilename - PreviousCreationUtcTime: PreviousCreationUtcTime - TargetFilename: Filename - User: username \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_image_load.yml deleted file mode 100644 index a500ad48..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_image_load.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Qradar -source: windows_image_load -description: Text that describe current mapping - -log_source: - devicetype: [12] - category: [19247] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - category: 19247 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - Image: username - ImageLoaded: Process Path - SignatureStatus: Signature Status - OriginalFileName: OriginalFileName - Signed: Signed \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_ldap_debug.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_ldap_debug.yml deleted file mode 100644 index 5ca2fce8..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_ldap_debug.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Qradar -source: windows_ldap_debug -description: Text that describe current mapping - -log_source: - devicetype: [12] - qideventcategory: [Microsoft-Windows-LDAP-Client/Debug] - -default_log_source: - devicetype: 12 - qideventcategory: Microsoft-Windows-LDAP-Client/Debug - -field_mapping: - EventID: Event ID - SearchFilter: SearchFilter \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_network_connection.yml deleted file mode 100644 index 56211f7f..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_network_connection.yml +++ /dev/null @@ -1,22 +0,0 @@ -platform: Qradar -source: windows_network_connection -description: Text that describe current mapping - -log_source: - devicetype: [12] - category: [4012] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - category: [4012] - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - Image: Process Path - DestinationHostname: Destination Hostname - DestinationIp: destinationip - DestinationPort: destinationport - SourceIp: sourceip - SourcePort: sourceport - Initiated: Initiated \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_ntlm.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_ntlm.yml deleted file mode 100644 index 5c1e0617..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_ntlm.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Qradar -source: windows_ntlm -description: Text that describe current mapping - -log_source: - devicetype: [12] - qideventcategory: [Microsoft-Windows-NTLM/Operational] - -default_log_source: - devicetype: 12 - qideventcategory: Microsoft-Windows-NTLM/Operational - -field_mapping: - WorkstationName: WorkstationName - ProcessName: ProcessName - EventID: Event ID - TargetName: TargetName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_pipe_created.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_pipe_created.yml deleted file mode 100644 index 4554b381..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_pipe_created.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Qradar -source: windows_pipe_created -description: Text that describe current mapping - -log_source: - devicetype: [12] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - PipeName: Pipe Name - Image: Process Path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_powershell.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_powershell.yml deleted file mode 100644 index a731eb43..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_powershell.yml +++ /dev/null @@ -1,22 +0,0 @@ -platform: Qradar -source: windows_powershell -description: Text that describe current mapping - -log_source: - devicetype: [12] - qideventcategory: [Microsoft-Windows-PowerShell/Operational] - -default_log_source: - devicetype: 12 - qideventcategory: Microsoft-Windows-PowerShell/Operational - -field_mapping: - EventID: Event ID - CommandLine: Command - ScriptBlockText: ScriptBlockText - Payload: Payload - HostApplication: HostApplication - ContextInfo: ContextInfo - HostName: HostName - EngineVersion: EngineVersion - HostVersion: HostVersion \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_process_access.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_process_access.yml deleted file mode 100644 index 66ef6aae..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_process_access.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: Qradar -source: windows_process_access -description: Text that describe current mapping - -log_source: - devicetype: [12] - qid: [5001829] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - qid: 5001829 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - TargetImage: Target Process Path - SourceImage: Source Process Path - CallTrace: CallTrace - GrantedAccess: GrantedAccess \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_process_creation.yml deleted file mode 100644 index 95a4d70b..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_process_creation.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Qradar -source: windows_process_creation -description: Text that describe current mapping - -log_source: - devicetype: [12] - category: [8110] - -default_log_source: - devicetype: 12 - category: 8110 - -field_mapping: - CommandLine: Command - CurrentDirectory: CurrentDirectory - Hashes: File Hash - Image: Process Path - IntegrityLevel: IntegrityLevel - ParentCommandLine: Parent Command - ParentImage: Parent Process Path - ParentUser: ParentUser - Product: Product - User: username \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_raw_access_thread.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_raw_access_thread.yml deleted file mode 100644 index ff1de2e4..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_raw_access_thread.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Qradar -source: windows_raw_access_thread -description: Text that describe current mapping - -log_source: - devicetype: [12] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - Device: Device - Image: Process Path \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_registry_event.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_registry_event.yml deleted file mode 100644 index 654f1225..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_registry_event.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Qradar -source: windows_registry_event -description: Text that describe current mapping - -log_source: - devicetype: [12] - category: [8012] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - category: 8012 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - TargetObject: Target Object - Image: Process Path - Details: Target Details - EventType: EventType - User: username \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_security.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_security.yml deleted file mode 100644 index 3bd21429..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_security.yml +++ /dev/null @@ -1,144 +0,0 @@ -platform: Qradar -source: windows_security -description: Text that describe current mapping - -log_source: - devicetype: [12] - -default_log_source: - devicetype: 12 - -field_mapping: - EventID: Event ID - ParentImage: Parent Process Path - AccessMask: AccessMask - AccountName: Account Name - AllowedToDelegateTo: AllowedToDelegateTo - AttributeLDAPDisplayName: AttributeLDAPDisplayName - AuditPolicyChanges: AuditPolicyChanges - AuthenticationPackageName: AuthenticationPackageName - CallingProcessName: CallingProcessName - Channel: Channel - ComputerName: Machine Identifier - EventType: EventType - FailureReason: FailureReason - FileName: Filename - GrantedAccess: GrantedAccess - Hashes: File Hash - HiveName: HiveName - IpAddress: IpAddress - IpPort: IpPort - KeyLength: KeyLength - LogonProcessName: LogonProcessName - LogonType: Logon Type - LinkName: LinkName - MemberName: MemberName - MemberSid: MemberSid - NewProcessName: Process Name - ObjectClass: ObjectClass - ObjectName: Object Name - ObjectType: Object Type - ObjectValueName: ObjectValueName - Path: Path - CommandLine: Command - OldUacValue: OldUacValue - SubStatus: SubStatus - DisplayName: DisplayName - TaskContent: TaskContent - ServiceSid: ServiceSid - CertThumbprint: CertThumbprint - ClassName: ClassName - NotificationPackageName: NotificationPackageName - NewSd: NewSd - TestSigning: TestSigning - TargetInfo: TargetInfo - ClientProcessId: ClientProcessId - ParentProcessId: ParentProcessId - AccessList: AccessList - GroupMembership: GroupMembership - FilterName: FilterName - ChangeType: ChangeType - LayerName: LayerName - ServiceAccount: ServiceAccount - AttributeValue: AttributeValue - SessionName: SessionName - TaskName: TaskName - ObjectDN: ObjectDN - TemplateContent: TemplateContent - NewTemplateContent: NewTemplateContent - SourcePort: sourceport - PasswordLastSet: PasswordLastSet - PrivilegeList: PrivilegeList - DeviceDescription: DeviceDescription - TargetServerName: TargetServerName - NewTargetUserName: NewTargetUserName - OperationType: OperationType - DestPort: destinationport - ServiceStartType: ServiceStartType - OldTargetUserName: OldTargetUserName - UserPrincipalName: UserPrincipalName - Accesses: Accesses - DnsHostName: DnsHostName - DisableIntegrityChecks: DisableIntegrityChecks - AuditSourceName: AuditSourceName - Workstation: Machine Identifier - DestAddress: destinationip - PreAuthType: PreAuthType - SecurityPackageName: SecurityPackageName - SubjectLogonId: SubjectLogonId - NewUacValue: NewUacValue - EnabledPrivilegeList: EnabledPrivilegeList - RelativeTargetName: RelativeTargetName - CertSerialNumber: RelativeTargetName - SidHistory: SidHistory - TargetLogonId: TargetLogonId - KernelDebug: KernelDebug - CallerProcessName: CallerProcessName - ProcessName: Process Name - Properties: Properties - UserAccountControl: UserAccountControl - RegistryValue: Target Object - SecurityID: SecurityID - ServiceFileName: Service Filename - SecurityDescriptor: SecurityDescriptor - ServiceName: Service Name - ShareName: Share Name - NewValue: NewValue - Source: Source - Status: Status - SubjectDomainName: SubjectDomainName - SubjectUserName: Target Username - SubjectUserSid: SubjectUserSid - SourceAddr: sourceip - SourceAddress: sourceip - TargetName: Target Username - ServicePrincipalNames: ServicePrincipalNames - TargetDomainName: TargetDomainName - TargetSid: TargetSid - TargetUserName: Target Username - ObjectServer: ObjectServer - TargetUserSid: TargetUserSid - TicketEncryptionType: TicketEncryptionType - TicketOptions: TicketOptions - WorkstationName: Machine Identifier - TransmittedServices: TransmittedServices - AuthenticationAlgorithm: AuthenticationAlgorithm - LayerRTID: LayerRTID - BSSID: BSSID - BSSType: BSSType - CipherAlgorithm: CipherAlgorithm - ConnectionId: ConnectionId - ConnectionMode: ConnectionMode - InterfaceDescription: InterfaceDescription - InterfaceGuid: InterfaceGuid - OnexEnabled: OnexEnabled - PHYType: PHYType - ProfileName: ProfileName - SSID: SSID - Domain: Domain - ServiceType: ServiceType - SourceName: SourceName - StartType: StartType - UserID: UserID - ParentProcessName: Parent Process Name - Service: Service \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_sysmon.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_sysmon.yml deleted file mode 100644 index 053ca58f..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_sysmon.yml +++ /dev/null @@ -1,63 +0,0 @@ -platform: Qradar -source: windows_sysmon -description: Text that describe current mapping - -log_source: - devicetype: [12] - qideventcategory: [Microsoft-Windows-Sysmon/Operational] - -default_log_source: - devicetype: 12 - qideventcategory: Microsoft-Windows-Sysmon/Operational - -field_mapping: - CommandLine: Command - Image: Process Path - ParentImage: Parent Process Path - EventID: Event ID - DestinationHostname: Destination Host Name - DestinationIp: destinationip - DestinationPort: destinationport - Hashes: File Hash - ParentCommandLine: Parent Command - RuleName: Rule Name - SourceHostname: Source Hostname - SourceIp: sourceip - SourcePort: sourceport - TargetFilename: Filename - User: username - SignatureStatus: Signature Status - TargetObject: Target Object - Details: Target Details - QueryName: URL - PipeName: Pipe Name - ImageLoaded: Process Path - Imphash: IMP Hash - SourceImage: Source Process Path - StartModule: Start Module - TargetImage: Target Process Path - Device: Device Name - ProcessID: Process ID - StartAddress: Start Address - StartFunction: Start Function - CallTrace: CallTrace - Company: Company - CurrentDirectory: CurrentDirectory - Description: Description - DestinationIsIpv6: DestinationIsIpv6 - DestinationPortName: DestinationPortName - Initiated: Initiated - IntegrityLevel: IntegrityLevel - Product: Product - Protocol: Protocol - SourceIsIpv6: SourceIsIpv6 - SourcePortName: SourcePortName - OriginalFileName: OriginalFileName - Signed: Signed - Signature: Signature - QueryResults: QueryResults - QueryStatus: QueryStatus - IsExecutable: IsExecutable - ImagePath: ImagePath - FileVersion: FileVersion - EventType: EventType diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_system.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_system.yml deleted file mode 100644 index 53db0cd1..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_system.yml +++ /dev/null @@ -1,27 +0,0 @@ -platform: Qradar -source: windows_system -description: Text that describe current mapping - -log_source: - devicetype: [12] - qideventcategory: [System_Information, System_Warning, System_Error] - -default_log_source: - devicetype: 12 - qideventcategory: System_Information - -field_mapping: - EventID: Event ID - AccountName: AccountName - ImagePath: ImagePath - ServiceName: ServiceName - ServiceType: ServiceType - StartType: StartType - Provider_Name: Provider_Name - Origin: Origin - HiveName: HiveName - Caption: Caption - param1: param1 - param2: param2 - Channel: Channel - DeviceName: DeviceName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/qradar/windows_wmi_event.yml b/siem-converter/app/converter/mappings/platforms/qradar/windows_wmi_event.yml deleted file mode 100644 index af13bd0d..00000000 --- a/siem-converter/app/converter/mappings/platforms/qradar/windows_wmi_event.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Qradar -source: windows_wmi_event -description: Text that describe current mapping - -log_source: - devicetype: [12] - qideventcategory: [Microsoft-Windows-WMI-Activity/Operational] - -default_log_source: - devicetype: 12 - qideventcategory: Microsoft-Windows-WMI-Activity/Operational - -field_mapping: - EventID: Event ID - Destination: Destination \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/aws_cloudtrail.yml b/siem-converter/app/converter/mappings/platforms/sigma/aws_cloudtrail.yml deleted file mode 100644 index 5b3f16af..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/aws_cloudtrail.yml +++ /dev/null @@ -1,36 +0,0 @@ -platform: Sigma -source: aws_cloudtrail -description: Text that describe current mapping - -log_source: - product: [aws] - service: [cloudtrail] - -default_log_source: - product: aws - service: cloudtrail - -field_mapping: - eventSource: eventSource - eventName: eventName - AdditionalEventData: additionalEventData.MFAUsed - errorCode: errorCode - errorMessage: errorMessage - eventType: eventType - requestParameters: requestParameters - requestParameters.attribute: requestParameters.attribute - requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP - requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort - requestParameters.userData: requestParameters.userData - responseElements: responseElements - responseElements.ConsoleLogin: responseElements.ConsoleLogin - responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword - responseElements.publiclyAccessible: responseElements.publiclyAccessible - status: status - terminatingRuleId: terminatingRuleId - userAgent: userAgent - userIdentity.arn: userIdentity.arn - userIdentity.principalId: userIdentity.principalId - userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type - userIdentity.type: userIdentity.type - userIdentity.userName: userIdentity.userName diff --git a/siem-converter/app/converter/mappings/platforms/sigma/aws_eks.yml b/siem-converter/app/converter/mappings/platforms/sigma/aws_eks.yml deleted file mode 100644 index f9b0215a..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/aws_eks.yml +++ /dev/null @@ -1,27 +0,0 @@ -platform: Sigma -source: aws_eks -description: Text that describe current mapping - -log_source: - product: [aws] - service: [eks] - -default_log_source: - product: aws - service: eks - -field_mapping: - annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision - annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy - aws_node_type: aws_node_type - objectRef.namespace: objectRef.namespace - objectRef.resource: objectRef.resource - objectRef.subresource: objectRef.subresource - requestObject.rules.resources: requestObject.rules.resources - requestObject.rules.verbs: requestObject.rules.verbs - requestObject.spec.containers.image: requestObject.spec.containers.image - requestURI: requestURI - stage: stage - user.groups: user.groups - user.username: user.username - verb: verb diff --git a/siem-converter/app/converter/mappings/platforms/sigma/azure_AzureDiagnostics.yml b/siem-converter/app/converter/mappings/platforms/sigma/azure_AzureDiagnostics.yml deleted file mode 100644 index fe87b8e6..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/azure_AzureDiagnostics.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Sigma -source: azure_AzureDiagnostics -description: Text that describe current mapping - -log_source: - product: [azure] - service: [AzureDiagnostics] - -default_log_source: - product: azure - service: AzureDiagnostics - -field_mapping: - ResultDescription: ResultDescription - Category: Category diff --git a/siem-converter/app/converter/mappings/platforms/sigma/azure_BehaviorAnalytics.yml b/siem-converter/app/converter/mappings/platforms/sigma/azure_BehaviorAnalytics.yml deleted file mode 100644 index 5a2eb466..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/azure_BehaviorAnalytics.yml +++ /dev/null @@ -1,22 +0,0 @@ -platform: Sigma -source: azure_BehaviorAnalytics -description: Text that describe current mapping - -log_source: - product: [azure] - service: [BehaviorAnalytics] - -default_log_source: - product: azure - service: BehaviorAnalytics - -field_mapping: - ActionType: ActionType - ActivityInsights: ActivityInsights - ActivityType: ActivityType - EventSource: EventSource - DevicesInsights: DevicesInsights - RiskDetail: RiskDetail - UsersInsights: UsersInsights - UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount - UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/siem-converter/app/converter/mappings/platforms/sigma/azure_aadnoninteractiveusersigninlogs.yml b/siem-converter/app/converter/mappings/platforms/sigma/azure_aadnoninteractiveusersigninlogs.yml deleted file mode 100644 index fa4ad8a2..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/azure_aadnoninteractiveusersigninlogs.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Sigma -source: azure_aadnoninteractiveusersigninlogs -description: Text that describe current mapping - -log_source: - product: [azure] - service: [aadnoninteractiveusersigninlogs] - -default_log_source: - product: azure - service: aadnoninteractiveusersigninlogs - -field_mapping: - UserAgent: UserAgent - Type: Type diff --git a/siem-converter/app/converter/mappings/platforms/sigma/azure_azureactivity.yml b/siem-converter/app/converter/mappings/platforms/sigma/azure_azureactivity.yml deleted file mode 100644 index fb55d1ae..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/azure_azureactivity.yml +++ /dev/null @@ -1,24 +0,0 @@ -platform: Sigma -source: azure_azureactivity -description: Text that describe current mapping - -log_source: - product: [azure] - service: [azureactivity] - -default_log_source: - product: azure - service: azureactivity - -field_mapping: - ActivityStatus: ActivityStatus - ActivityStatusValue: ActivityStatusValue - ActivitySubstatusValue: ActivitySubstatusValue - Authorization: Authorization - Category: Category - CategoryValue: CategoryValue - OperationName: OperationName - OperationNameValue: OperationNameValue - ResourceId: ResourceId - ResourceProviderValue: ResourceProviderValue - Type: Type diff --git a/siem-converter/app/converter/mappings/platforms/sigma/azure_azuread.yml b/siem-converter/app/converter/mappings/platforms/sigma/azure_azuread.yml deleted file mode 100644 index f257dd4f..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/azure_azuread.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Sigma -source: azure_azuread -description: Text that describe current mapping - -log_source: - product: [azure] - service: [azuread] - -default_log_source: - product: azure - service: azuread - -field_mapping: - ActivityDisplayName: ActivityDisplayName - Category: Category - LoggedByService: LoggedByService - Result: Result - OperationName: OperationName - TargetResources: TargetResources - AADOperationType: AADOperationType - InitiatedBy: InitiatedBy - ResultReason: ResultReason - Status: Status - Status.errorCode: Status.errorCode - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/sigma/azure_m365.yml b/siem-converter/app/converter/mappings/platforms/sigma/azure_m365.yml deleted file mode 100644 index cc6ec909..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/azure_m365.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Sigma -source: azure_m365 -description: Text that describe current mapping - -log_source: - product: [azure] - service: [m365] - -default_log_source: - product: azure - service: m365 - -field_mapping: - ClientInfoString: ClientInfoString - LogonError: LogonError - ModifiedProperties: ModifiedProperties - OfficeObjectId: OfficeObjectId - OfficeWorkload: OfficeWorkload - Operation: Operation - Parameters: Parameters - RecordType: RecordType - ResultStatus: ResultStatus - SourceFileExtension: SourceFileExtension - SourceFileName: SourceFileName - UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/sigma/azure_signinlogs.yml b/siem-converter/app/converter/mappings/platforms/sigma/azure_signinlogs.yml deleted file mode 100644 index 2b600b2d..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/azure_signinlogs.yml +++ /dev/null @@ -1,31 +0,0 @@ -platform: Sigma -source: azure_signinlogs -description: Text that describe current mapping - -log_source: - product: [azure] - service: [signinlogs] - -default_log_source: - product: azure - service: signinlogs - -field_mapping: - AppDisplayName: AppDisplayName - AppId: AppId - AuthenticationRequirement: AuthenticationRequirement - Category: Category - ConditionalAccessStatus: ConditionalAccessStatus - DeviceDetail: DeviceDetail - IsInteractive: IsInteractive - NetworkLocationDetails: NetworkLocationDetails - ResourceDisplayName: ResourceDisplayName - ResourceIdentity: ResourceIdentity - ResultDescription: ResultDescription - ResultType: ResultType - Status.errorCode: Status.errorCode - Status: Status - Status.failureReason: Status.failureReason - TokenIssuerType: TokenIssuerType - UserAgent: UserAgent - UserPrincipalName: UserPrincipalName diff --git a/siem-converter/app/converter/mappings/platforms/sigma/default.yml b/siem-converter/app/converter/mappings/platforms/sigma/default.yml deleted file mode 100644 index 49a0494c..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/default.yml +++ /dev/null @@ -1,9 +0,0 @@ -platform: Sigma -source: default -description: Text that describe current mapping - -log_source: - product: windows - -default_log_source: - product: windows \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/dns.yml b/siem-converter/app/converter/mappings/platforms/sigma/dns.yml deleted file mode 100644 index c4bef3bc..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/dns.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Sigma -source: dns -description: Text that describe current mapping - -log_source: - category: [dns] - -default_log_source: - category: dns - -field_mapping: - dns-query: dns-query - parent-domain: parent-domain - dns-answer: dns-answer - dns-record: dns-record \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/firewall.yml b/siem-converter/app/converter/mappings/platforms/sigma/firewall.yml deleted file mode 100644 index 58f9aeb6..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/firewall.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Sigma -source: firewall -description: Text that describe current mapping - -log_source: - category: [firewall] - -default_log_source: - category: firewall - -field_mapping: - src-ip: src-ip - src-port: src-port - dst-ip: dst-ip - dst-port: dst-port \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/gcp_gcp.audit.yml b/siem-converter/app/converter/mappings/platforms/sigma/gcp_gcp.audit.yml deleted file mode 100644 index 071d3570..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/gcp_gcp.audit.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Sigma -source: gcp_gcp.audit -description: Text that describe current mapping - -log_source: - product: [gcp] - service: [gcp.audit] - -default_log_source: - product: gcp - service: gcp.audit - -field_mapping: - backupConfiguration.enabled: backupConfiguration.enabled - gcp.audit.method_name: gcp.audit.method_name - jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction - jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue - message: message - methodName: methodName - protoPayload.methodName: protoPayload.methodName - protoPayload.request.canIpForward: protoPayload.request.canIpForward - protoPayload.serviceName: protoPayload.serviceName - serviceName: serviceName diff --git a/siem-converter/app/converter/mappings/platforms/sigma/gcp_pubsub.yml b/siem-converter/app/converter/mappings/platforms/sigma/gcp_pubsub.yml deleted file mode 100644 index f45ca2ae..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/gcp_pubsub.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Sigma -source: gcp_pubsub -description: Text that describe current mapping - -log_source: - product: [gcp] - service: [pubsub] - -default_log_source: - product: gcp - service: pubsub - -field_mapping: - data.protoPayload.methodName: data.protoPayload.methodName - data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member - data.resource.type: data.resource.type diff --git a/siem-converter/app/converter/mappings/platforms/sigma/linux_auditd.yml b/siem-converter/app/converter/mappings/platforms/sigma/linux_auditd.yml deleted file mode 100644 index a80fcaf2..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/linux_auditd.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Sigma -source: linux_auditd -description: Text that describe current mapping - -log_source: - product: [linux] - service: [auditd] - -default_log_source: - product: linux - service: auditd - -field_mapping: - a0: a0 - a1: a1 - a2: a2 - a3: a3 - exe: exe \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/linux_dns_query.yml b/siem-converter/app/converter/mappings/platforms/sigma/linux_dns_query.yml deleted file mode 100644 index 523484c2..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/linux_dns_query.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Sigma -source: linux_dns_query -description: Text that describe current mapping - -log_source: - product: [linux] - category: [dns_query] - -default_log_source: - product: linux - category: dns_query - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/linux_network_connection.yml b/siem-converter/app/converter/mappings/platforms/sigma/linux_network_connection.yml deleted file mode 100644 index 46f49db8..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/linux_network_connection.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Sigma -source: linux_network_connection -description: Text that describe current mapping - -log_source: - product: [linux] - category: [network_connection] - -default_log_source: - product: linux - category: network_connection - -field_mapping: - Image: Image - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationPort: DestinationPort - SourceIp: SourceIp - SourcePort: SourcePort - ParentImage: ParentImage \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/linux_process_creation.yml b/siem-converter/app/converter/mappings/platforms/sigma/linux_process_creation.yml deleted file mode 100644 index bfea1785..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/linux_process_creation.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Sigma -source: linux_process_creation -description: Text that describe current mapping - -log_source: - product: [linux] - category: [process_creation] - -default_log_source: - product: linux - category: process_creation - -field_mapping: - CommandLine: CommandLine - Image: Image - ParentCommandLine: ParentCommandLine - ParentImage: ParentImage \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/macos_dns_query.yml b/siem-converter/app/converter/mappings/platforms/sigma/macos_dns_query.yml deleted file mode 100644 index fc0d5432..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/macos_dns_query.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Sigma -source: macos_dns_query -description: Text that describe current mapping - -log_source: - product: [macos] - category: [dns_query] - -default_log_source: - product: macos - category: dns_query - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/macos_network_connection.yml b/siem-converter/app/converter/mappings/platforms/sigma/macos_network_connection.yml deleted file mode 100644 index 070674ca..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/macos_network_connection.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Sigma -source: macos_network_connection -description: Text that describe current mapping - -log_source: - product: [macos] - category: [network_connection] - -default_log_source: - product: macos - category: network_connection - -field_mapping: - Image: Image - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationPort: DestinationPort - SourceIp: SourceIp - SourcePort: SourcePort - ParentImage: ParentImage \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/macos_process_creation.yml b/siem-converter/app/converter/mappings/platforms/sigma/macos_process_creation.yml deleted file mode 100644 index 631bfdde..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/macos_process_creation.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Sigma -source: macos_process_creation -description: Text that describe current mapping - -log_source: - product: [macos] - category: [process_creation] - -default_log_source: - product: macos - category: process_creation - -field_mapping: - CommandLine: CommandLine - Image: Image - ParentCommandLine: ParentCommandLine - ParentImage: ParentImage \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/okta_okta.yml b/siem-converter/app/converter/mappings/platforms/sigma/okta_okta.yml deleted file mode 100644 index 7da07629..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/okta_okta.yml +++ /dev/null @@ -1,35 +0,0 @@ -platform: Sigma -source: okta_okta -description: Text that describe current mapping - -log_source: - product: [okta] - service: [okta] - -default_log_source: - product: okta - service: okta - -field_mapping: - client.user.id: client.user.id - source.user.id: source.user.id - User: User - alternateId: alternateId - client.user.full_name: client.user.full_name - source.user.full_name: source.user.full_name - related.user: related.user - client.ip: client.ip - source.ip: source.ip - user_agent.original: user_agent.original - userAgent.os: userAgent.os - userAgent.browser: userAgent.browser - client.zone: client.zone - client.device: client.device - client.id: client.id - event.action: event.action - outcome.reason: outcome.reason - event.outcome: event.outcome - client.as.number: client.as.number - client.as.organization.name: client.as.organization.name - client.domain: client.domain - source.domain: source.domain diff --git a/siem-converter/app/converter/mappings/platforms/sigma/proxy.yml b/siem-converter/app/converter/mappings/platforms/sigma/proxy.yml deleted file mode 100644 index 47549276..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/proxy.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Sigma -source: proxy -description: Text that describe current mapping - -log_source: - category: [proxy] - -default_log_source: - category: proxy - -field_mapping: - c-uri: c-uri - c-useragent: c-useragent - cs-method: cs-method - cs-bytes: cs-bytes - cs-cookie-vars: cs-cookie-vars - c-uri-extension: c-uri-extension - c-uri-query: c-uri-query - cs-cookie: cs-cookie - cs-host: cs-host - cs-referrer: cs-referrer - cs-version: cs-version - r-dns: r-dns - sc-status: sc-status - post-body: post-body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/webserver.yml b/siem-converter/app/converter/mappings/platforms/sigma/webserver.yml deleted file mode 100644 index cd9a6101..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/webserver.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Sigma -source: webserver -description: Text that describe current mapping - -log_source: - category: [webserver] - -default_log_source: - category: webserver - -field_mapping: - c-uri: c-uri - c-useragent: c-useragent - cs-method: cs-method - cs-bytes: cs-bytes - cs-cookie-vars: cs-cookie-vars - c-uri-extension: c-uri-extension - c-uri-query: c-uri-query - cs-cookie: cs-cookie - cs-host: cs-host - cs-referrer: cs-referrer - cs-version: cs-version - r-dns: r-dns - sc-status: sc-status - post-body: post-body \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_bits_client.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_bits_client.yml deleted file mode 100644 index 05bb1370..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_bits_client.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Sigma -source: windows_bits_client -description: Text that describe current mapping - -log_source: - product: [windows] - category: [bits-client] - -default_log_source: - product: windows - category: bits-client - -field_mapping: - LocalName: LocalName - EventID: EventID - RemoteName: RemoteName - processPath: processPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_dns_query.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_dns_query.yml deleted file mode 100644 index 153e4650..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_dns_query.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Sigma -source: windows_dns_query -description: Text that describe current mapping - -log_source: - product: [windows] - category: [dns_query] - -default_log_source: - product: windows - category: dns_query - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_driver_load.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_driver_load.yml deleted file mode 100644 index 1f65db2a..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_driver_load.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Sigma -source: windows_driver_load -description: Text that describe current mapping - -log_source: - product: [windows] - category: [driver_load] - -default_log_source: - product: macos - category: driver_load - -field_mapping: - ImageLoaded: ImageLoaded - ImagePath: ImagePath - Hashes: Hashes \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_image_load.yml deleted file mode 100644 index 04cdd2ed..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_image_load.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Sigma -source: windows_image_load -description: Text that describe current mapping - -log_source: - product: [windows] - category: [image_load] - -default_log_source: - product: windows - category: image_load - -field_mapping: - Image: Image - ImageLoaded: ImageLoaded - SignatureStatus: SignatureStatus - OriginalFileName: OriginalFileName - Signed: Signed \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_ldap_debug.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_ldap_debug.yml deleted file mode 100644 index 4f0add1a..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_ldap_debug.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Sigma -source: windows_ldap_debug -description: Text that describe current mapping - -log_source: - product: [windows] - service: [ldap_debug] - -default_log_source: - product: windows - service: ldap_debug - -field_mapping: - EventID: EventID - SearchFilter: SearchFilter \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_network_connection.yml deleted file mode 100644 index 613a4984..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_network_connection.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Sigma -source: windows_network_connection -description: Text that describe current mapping - -log_source: - product: [windows] - category: [network_connection] - -default_log_source: - product: windows - category: network_connection - -field_mapping: - Image: Image - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationPort: DestinationPort - SourceIp: SourceIp - SourcePort: SourcePort - Initiated: Initiated \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_ntlm.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_ntlm.yml deleted file mode 100644 index 42347598..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_ntlm.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Sigma -source: windows_ntlm -description: Text that describe current mapping - -log_source: - product: [windows] - service: [ntlm] - -default_log_source: - product: windows - service: ntlm - -field_mapping: - ProcessName: ProcessName - TargetName: TargetName - WorkstationName: WorkstationName - EventID: EventID \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_powershell.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_powershell.yml deleted file mode 100644 index f56e145a..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_powershell.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: Sigma -source: windows_powershell -description: Text that describe current mapping - -log_source: - product: [windows] - service: [powershell] - -default_log_source: - product: windows - service: powershell - -field_mapping: - EventID: EventID - CommandLine: CommandLine - ScriptBlockText: ScriptBlockText - Payload: Payload - HostApplication: HostApplication - ContextInfo: ContextInfo - HostName: HostName - EngineVersion: EngineVersion \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_process_creation.yml deleted file mode 100644 index 17a19ecd..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_process_creation.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Sigma -source: windows_process_creation -description: Text that describe current mapping - -log_source: - product: [windows] - category: [process_creation] - -default_log_source: - product: windows - category: process_creation - -field_mapping: - CommandLine: CommandLine - CurrentDirectory: CurrentDirectory - Hashes: Hashes - Image: Image - IntegrityLevel: IntegrityLevel - ParentCommandLine: ParentCommandLine - ParentImage: ParentImage - ParentUser: ParentUser - Product: Product - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_security.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_security.yml deleted file mode 100644 index b24f7dc7..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_security.yml +++ /dev/null @@ -1,147 +0,0 @@ -platform: Sigma -source: windows_security -description: Text that describe current mapping - -log_source: - product: [windows] - service: [security] - -default_log_source: - product: windows - service: security - -field_mapping: - EventID: EventID - ParentImage: ParentImage - AccessMask: AccessMask - AccountName: AccountName - AllowedToDelegateTo: AllowedToDelegateTo - AttributeLDAPDisplayName: AttributeLDAPDisplayName - AuditPolicyChanges: AuditPolicyChanges - AuthenticationPackageName: AuthenticationPackageName - CallingProcessName: CallingProcessName - Channel: Channel - ComputerName: ComputerName - EventType: EventType - FailureReason: FailureReason - FileName: FileName - GrantedAccess: GrantedAccess - Hashes: Hashes - HiveName: HiveName - IpAddress: IpAddress - IpPort: IpPort - KeyLength: KeyLength - LogonProcessName: LogonProcessName - LogonType: LogonType - LinkName: LinkName - MemberName: MemberName - MemberSid: MemberSid - NewProcessName: NewProcessName - ObjectClass: ObjectClass - ObjectType: ObjectType - ObjectValueName: ObjectValueName - Path: Path - CommandLine: CommandLine - OldUacValue: OldUacValue - CertIssuerName: CertIssuerName - SubStatus: SubStatus - DisplayName: DisplayName - TaskContent: TaskContent - ServiceSid: ServiceSid - CertThumbprint: CertThumbprint - ObjectName: ObjectName - ClassName: ClassName - NotificationPackageName: NotificationPackageName - NewSd: NewSd - TestSigning: TestSigning - TargetInfo: TargetInfo - ParentProcessId: ParentProcessId - AccessList: AccessList - GroupMembership: GroupMembership - FilterName: FilterName - ChangeType: ChangeType - LayerName: LayerName - ServiceAccount: ServiceAccount - ClientProcessId: ClientProcessId - AttributeValue: AttributeValue - SessionName: SessionName - TaskName: TaskName - ObjectDN: ObjectDN - TemplateContent: TemplateContent - NewTemplateContent: NewTemplateContent - SourcePort: SourcePort - PasswordLastSet: PasswordLastSet - PrivilegeList: PrivilegeList - DeviceDescription: DeviceDescription - TargetServerName: TargetServerName - NewTargetUserName: NewTargetUserName - OperationType: OperationType - DestPort: DestPort - ServiceStartType: ServiceStartType - OldTargetUserName: OldTargetUserName - UserPrincipalName: UserPrincipalName - Accesses: Accesses - DnsHostName: DnsHostName - DisableIntegrityChecks: DisableIntegrityChecks - AuditSourceName: AuditSourceName - Workstation: Workstation - DestAddress: DestAddress - PreAuthType: PreAuthType - SecurityPackageName: SecurityPackageName - SubjectLogonId: SubjectLogonId - NewUacValue: NewUacValue - EnabledPrivilegeList: EnabledPrivilegeList - RelativeTargetName: RelativeTargetName - CertSerialNumber: CertSerialNumber - SidHistory: SidHistory - TargetLogonId: TargetLogonId - KernelDebug: KernelDebug - CallerProcessName: CallerProcessName - Properties: Properties - UserAccountControl: UserAccountControl - RegistryValue: RegistryValue - SecurityID: SecurityID - ServiceFileName: ServiceFileName - SecurityDescriptor: SecurityDescriptor - ServiceName: ServiceName - ShareName: ShareName - NewValue: NewValue - Source: Source - Status: Status - SubjectDomainName: SubjectDomainName - SubjectUserName: SubjectUserName - SubjectUserSid: SubjectUserSid - SourceAddr: SourceAddr - SourceAddress: SourceAddress - TargetName: TargetName - ServicePrincipalNames: ServicePrincipalNames - TargetDomainName: TargetDomainName - TargetSid: TargetSid - TargetUserName: TargetUserName - ObjectServer: ObjectServer - TargetUserSid: TargetUserSid - TicketEncryptionType: TicketEncryptionType - TicketOptions: TicketOptions - WorkstationName: WorkstationName - TransmittedServices: TransmittedServices - AuthenticationAlgorithm: AuthenticationAlgorithm - LayerRTID: LayerRTID - BSSID: BSSID - BSSType: BSSType - CipherAlgorithm: CipherAlgorithm - ConnectionId: ConnectionId - ConnectionMode: ConnectionMode - InterfaceDescription: InterfaceDescription - InterfaceGuid: InterfaceGuid - OnexEnabled: OnexEnabled - PHYType: PHYType - ProfileName: ProfileName - SSID: SSID - Domain: Domain - ServiceType: ServiceType - SourceName: SourceName - StartType: StartType - UserID: UserID - ParentProcessName: ParentProcessName - Service: Service - ProcessName: ProcessName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_sysmon.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_sysmon.yml deleted file mode 100644 index f0612d05..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_sysmon.yml +++ /dev/null @@ -1,63 +0,0 @@ -platform: Sigma -source: windows_sysmon -description: Text that describe current mapping - -log_source: - product: [windows] - service: [sysmon] - -default_log_source: - product: windows - service: sysmon - -field_mapping: - CommandLine: CommandLine - Image: Image - ParentImage: ParentImage - EventID: EventID - CallTrace: CallTrace - Company: Company - CurrentDirectory: CurrentDirectory - Description: Description - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationIsIpv6: DestinationIsIpv6 - DestinationPort: DestinationPort - DestinationPortName: DestinationPortName - Hashes: Hashes - Initiated: Initiated - IntegrityLevel: IntegrityLevel - ParentCommandLine: ParentCommandLine - Product: Product - Protocol: Protocol - RuleName: RuleName - SourceHostname: SourceHostname - SourceIp: SourceIp - SourceIsIpv6: SourceIsIpv6 - SourcePort: SourcePort - SourcePortName: SourcePortName - TargetFilename: TargetFilename - User: User - OriginalFileName: OriginalFileName - Signed: Signed - Signature: Signature - SignatureStatus: SignatureStatus - TargetObject: TargetObject - Details: Details - QueryName: QueryName - QueryResults: QueryResults - QueryStatus: QueryStatus - IsExecutable: IsExecutable - PipeName: PipeName - ImageLoaded: ImageLoaded - ImagePath: ImagePath - Imphash: Imphash - SourceImage: SourceImage - StartModule: StartModule - TargetImage: TargetImage - Device: Device - ProcessID: ProcessID - FileVersion: FileVersion - StartAddress: StartAddress - StartFunction: StartFunction - EventType: EventType \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/sigma/windows_wmi_event.yml b/siem-converter/app/converter/mappings/platforms/sigma/windows_wmi_event.yml deleted file mode 100644 index cccd50db..00000000 --- a/siem-converter/app/converter/mappings/platforms/sigma/windows_wmi_event.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Sigma -source: windows_wmi_event -description: Text that describe current mapping - -log_source: - product: [windows] - category: [wmi_event] - -default_log_source: - product: windows - category: wmi_event - -field_mapping: - Destination: Destination - EventID: EventID diff --git a/siem-converter/app/converter/mappings/platforms/splunk/aws_cloudtrail.yml b/siem-converter/app/converter/mappings/platforms/splunk/aws_cloudtrail.yml deleted file mode 100644 index d24d93a0..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/aws_cloudtrail.yml +++ /dev/null @@ -1,35 +0,0 @@ -platform: Splunk -source: aws_cloudtrail -description: Text that describe current mapping - -log_source: - sourceType: [aws:cloudtrail] - -default_log_source: - sourceType: aws:cloudtrail - -field_mapping: - eventSource: eventSource - eventName: eventName - AdditionalEventData: AdditionalEventData - additionalEventData.MFAUsed: additionalEventData.MFAUsed - errorCode: errorCode - errorMessage: errorMessage - eventType: eventType - requestParameters: requestParameters - requestParameters.attribute: requestParameters.attribute - requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP - requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort - requestParameters.userData: requestParameters.userData - responseElements: responseElements - responseElements.ConsoleLogin: responseElements.ConsoleLogin - responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword - responseElements.publiclyAccessible: responseElements.publiclyAccessible - status: status - terminatingRuleId: terminatingRuleId - userAgent: userAgent - userIdentity.arn: userIdentity.arn - userIdentity.principalId: userIdentity.principalId - userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type - userIdentity.type: userIdentity.type - userIdentity.userName: userIdentity.userName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/aws_eks.yml b/siem-converter/app/converter/mappings/platforms/splunk/aws_eks.yml deleted file mode 100644 index df2bbbd9..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/aws_eks.yml +++ /dev/null @@ -1,25 +0,0 @@ -platform: Splunk -source: aws_eks -description: Text that describe current mapping - -log_source: - sourceType: [aws:*] - -default_log_source: - sourceType: aws:* - -field_mapping: - annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision - annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy - aws_node_type: aws_node_type - objectRef.namespace: objectRef.namespace - objectRef.resource: objectRef.resource - objectRef.subresource: objectRef.subresource - requestObject.rules.resources: requestObject.rules.resources - requestObject.rules.verbs: requestObject.rules.verbs - requestObject.spec.containers.image: requestObject.spec.containers.image - requestURI: requestURI - stage: stage - user.groups: user.groups - user.username: user.username - verb: verb \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/azure_AzureDiagnostics.yml b/siem-converter/app/converter/mappings/platforms/splunk/azure_AzureDiagnostics.yml deleted file mode 100644 index 505f60eb..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/azure_AzureDiagnostics.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Splunk -source: azure_AzureDiagnostics -description: Text that describe current mapping - -log_source: - sourceType: [azure:*] - -default_log_source: - sourceType: azure:* - -field_mapping: - ResultDescription: ResultDescription - Category: Category \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/azure_BehaviorAnalytics.yml b/siem-converter/app/converter/mappings/platforms/splunk/azure_BehaviorAnalytics.yml deleted file mode 100644 index cca8980c..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/azure_BehaviorAnalytics.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Splunk -source: azure_BehaviorAnalytics -description: Text that describe current mapping - -log_source: - sourceType: [azure:*] - -default_log_source: - sourceType: azure:* - -field_mapping: - ActionType: ActionType - ActivityInsights: ActivityInsights - ActivityType: ActivityType - EventSource: EventSource - DevicesInsights: DevicesInsights - RiskDetail: RiskDetail - UsersInsights: UsersInsights - UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount - UsersInsights.IsNewAccount: UsersInsights.IsNewAccount \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/azure_aadnoninteractiveusersigninlogs.yml b/siem-converter/app/converter/mappings/platforms/splunk/azure_aadnoninteractiveusersigninlogs.yml deleted file mode 100644 index 2c416638..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/azure_aadnoninteractiveusersigninlogs.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Splunk -source: azure_aadnoninteractiveusersigninlogs -description: Text that describe current mapping - -log_source: - sourceType: [azure:*] - -default_log_source: - sourceType: azure:* - -field_mapping: - UserAgent: UserAgent - Type: Type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/azure_azureactivity.yml b/siem-converter/app/converter/mappings/platforms/splunk/azure_azureactivity.yml deleted file mode 100644 index 8ed0cdb8..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/azure_azureactivity.yml +++ /dev/null @@ -1,22 +0,0 @@ -platform: Splunk -source: azure_azureactivity -description: Text that describe current mapping - -log_source: - sourceType: [mscs:azure:*, azure:*] - -default_log_source: - sourceType: mscs:azure:* - -field_mapping: - ActivityStatus: ActivityStatus - ActivityStatusValue: ActivityStatusValue - ActivitySubstatusValue: ActivitySubstatusValue - Authorization: Authorization - Category: Category - CategoryValue: CategoryValue - OperationName: OperationName - OperationNameValue: OperationNameValue - ResourceId: ResourceId - ResourceProviderValue: ResourceProviderValue - Type: Type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/azure_azuread.yml b/siem-converter/app/converter/mappings/platforms/splunk/azure_azuread.yml deleted file mode 100644 index 7e06ef66..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/azure_azuread.yml +++ /dev/null @@ -1,23 +0,0 @@ -platform: Splunk -source: azure_azuread -description: Text that describe current mapping - -log_source: - sourceType: [azure:aad:*] - -default_log_source: - sourceType: azure:aad:* - -field_mapping: - ActivityDisplayName: ActivityDisplayName - Category: Category - LoggedByService: LoggedByService - Result: Result - OperationName: OperationName - TargetResources: TargetResources - AADOperationType: AADOperationType - InitiatedBy: InitiatedBy - ResultReason: ResultReason - Status: Status - Status.errorCode: Status.errorCode - UserAgent: UserAgent \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/azure_signinlogs.yml b/siem-converter/app/converter/mappings/platforms/splunk/azure_signinlogs.yml deleted file mode 100644 index 86eb795e..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/azure_signinlogs.yml +++ /dev/null @@ -1,29 +0,0 @@ -platform: Splunk -source: azure_signinlogs -description: Text that describe current mapping - -log_source: - sourceType: [azure:aad:*] - -default_log_source: - sourceType: azure:aad:* - -field_mapping: - AppDisplayName: AppDisplayName - AppId: AppId - AuthenticationRequirement: AuthenticationRequirement - Category: Category - ConditionalAccessStatus: ConditionalAccessStatus - DeviceDetail: DeviceDetail - IsInteractive: IsInteractive - NetworkLocationDetails: NetworkLocationDetails - ResourceDisplayName: ResourceDisplayName - ResourceIdentity: ResourceIdentity - ResultDescription: ResultDescription - ResultType: ResultType - Status.errorCode: Status.errorCode - Status: Status - Status.failureReason: Status.failureReason - TokenIssuerType: TokenIssuerType - UserAgent: UserAgent - UserPrincipalName: UserPrincipalName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/default.yml b/siem-converter/app/converter/mappings/platforms/splunk/default.yml deleted file mode 100644 index 4c5e7f39..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/default.yml +++ /dev/null @@ -1,9 +0,0 @@ -platform: Splunk -source: default -description: Text that describe current mapping - -log_source: - source: WinEventLog:* - -default_log_source: - source: WinEventLog:* \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/gcp_gcp.audit.yml b/siem-converter/app/converter/mappings/platforms/splunk/gcp_gcp.audit.yml deleted file mode 100644 index f213fe60..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/gcp_gcp.audit.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: Splunk -source: gcp_gcp.audit -description: Text that describe current mapping - -log_source: - sourceType: [google:gcp:*] - -default_log_source: - index: google:gcp:* - -field_mapping: - backupConfiguration.enabled: backupConfiguration.enabled - gcp.audit.method_name: gcp.audit.method_name - jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction - jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue - message: message - methodName: methodName - protoPayload.methodName: protoPayload.methodName - protoPayload.request.canIpForward: protoPayload.request.canIpForward - protoPayload.serviceName: protoPayload.serviceName - serviceName: serviceName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/gcp_pubsub.yml b/siem-converter/app/converter/mappings/platforms/splunk/gcp_pubsub.yml deleted file mode 100644 index aa72f273..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/gcp_pubsub.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Splunk -source: gcp_pubsub -description: Text that describe current mapping - -log_source: - sourceType: [google:gcp:*] - -default_log_source: - index: google:gcp:* - -field_mapping: - data.protoPayload.methodName: data.protoPayload.methodName - data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action - data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member - data.resource.type: data.resource.type \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_auditd.yml b/siem-converter/app/converter/mappings/platforms/splunk/linux_auditd.yml deleted file mode 100644 index 78a15677..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/linux_auditd.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Splunk -source: linux_auditd -description: Text that describe current mapping - -log_source: - sourceType: [linux:audit] - -default_log_source: - sourceType: linux:audit - -field_mapping: - a0: a0 - a1: a1 - a2: a2 - a3: a3 - exe: exe \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_dns_query.yml b/siem-converter/app/converter/mappings/platforms/splunk/linux_dns_query.yml deleted file mode 100644 index ea6bceb8..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/linux_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Splunk -source: linux_dns_query -description: Text that describe current mapping - -log_source: - index: [linux] - -default_log_source: - index: linux - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_event.yml b/siem-converter/app/converter/mappings/platforms/splunk/linux_file_event.yml deleted file mode 100644 index b75e41cd..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_event.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Splunk -source: linux_file_event -description: Text that describe current mapping - -log_source: - index: [linux] - -default_log_source: - index: linux - -field_mapping: - Image: Image - TargetFilename: TargetFilename diff --git a/siem-converter/app/converter/mappings/platforms/splunk/macos_dns_query.yml b/siem-converter/app/converter/mappings/platforms/splunk/macos_dns_query.yml deleted file mode 100644 index b3f6b659..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/macos_dns_query.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Splunk -source: macos_dns_query -description: Text that describe current mapping - -log_source: - index: [macos] - -default_log_source: - index: macos - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_event.yml b/siem-converter/app/converter/mappings/platforms/splunk/macos_file_event.yml deleted file mode 100644 index 64fb7232..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_event.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Splunk -source: macos_file_event -description: Text that describe current mapping - -log_source: - index: [macos] - -default_log_source: - index: macos - -field_mapping: - Image: Image - TargetFilename: TargetFilename \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/okta_okta.yml b/siem-converter/app/converter/mappings/platforms/splunk/okta_okta.yml deleted file mode 100644 index 8f5bbcdd..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/okta_okta.yml +++ /dev/null @@ -1,33 +0,0 @@ -platform: Splunk -source: okta_okta -description: Text that describe current mapping - -log_source: - sourceType: [OktaIM2:*] - -default_log_source: - sourceType: OktaIM2:* - -field_mapping: - client.user.id: client.user.id - source.user.id: source.user.id - User: User - alternateId: alternateId - client.user.full_name: client.user.full_name - source.user.full_name: source.user.full_name - related.user: related.user - client.ip: client.ip - source.ip: source.ip - user_agent.original: user_agent.original - userAgent.os: userAgent.os - userAgent.browser: userAgent.browser - client.zone: client.zone - client.device: client.device - client.id: client.id - event.action: event.action - outcome.reason: outcome.reason - event.outcome: event.outcome - client.as.number: client.as.number - client.as.organization.name: client.as.organization.name - client.domain: client.domain - source.domain: source.domain diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_bits_client.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_bits_client.yml deleted file mode 100644 index 5a426d10..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_bits_client.yml +++ /dev/null @@ -1,14 +0,0 @@ -platform: Splunk -source: windows_bits_client - -log_source: - sourceType: [XmlWinEventLog:Microsoft-Windows-Bits-Client/Operational] - -default_log_source: - sourceType: XmlWinEventLog:Microsoft-Windows-Bits-Client/Operational - -field_mapping: - LocalName: LocalName - EventID: EventID - RemoteName: RemoteName - processPath: processPath \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_dns_query.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_dns_query.yml deleted file mode 100644 index 2440c1d3..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_dns_query.yml +++ /dev/null @@ -1,17 +0,0 @@ -platform: Splunk -source: windows_dns_query -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - Image: Image - User: User - QueryName: QueryName - QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_driver_load.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_driver_load.yml deleted file mode 100644 index 60423f57..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_driver_load.yml +++ /dev/null @@ -1,16 +0,0 @@ -platform: Splunk -source: windows_driver_load -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - ImageLoaded: ImageLoaded - ImagePath: ImagePath - Hashes: Hashes \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_access.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_file_access.yml deleted file mode 100644 index 87dfe203..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_access.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Splunk -source: windows_file_access -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - CreationUtcTime: CreationUtcTime - FileName: FileName - Image: Image - OriginalFilename: OriginalFilename - PreviousCreationUtcTime: PreviousCreationUtcTime - TargetFilename: TargetFilename - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_change.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_file_change.yml deleted file mode 100644 index ec6e148a..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_change.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Splunk -source: windows_file_change -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - CreationUtcTime: CreationUtcTime - FileName: FileName - Image: Image - OriginalFilename: OriginalFilename - PreviousCreationUtcTime: PreviousCreationUtcTime - TargetFilename: TargetFilename - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_create.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_file_create.yml deleted file mode 100644 index 2d30b2ec..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_create.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Splunk -source: windows_file_create -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - CreationUtcTime: CreationUtcTime - FileName: FileName - Image: Image - OriginalFilename: OriginalFilename - PreviousCreationUtcTime: PreviousCreationUtcTime - TargetFilename: TargetFilename - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_delete.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_file_delete.yml deleted file mode 100644 index 875fefde..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_delete.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Splunk -source: windows_file_delete -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - CreationUtcTime: CreationUtcTime - FileName: FileName - Image: Image - OriginalFilename: OriginalFilename - PreviousCreationUtcTime: PreviousCreationUtcTime - TargetFilename: TargetFilename - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_event.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_file_event.yml deleted file mode 100644 index 850c6c40..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_event.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Splunk -source: windows_file_event -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - CreationUtcTime: CreationUtcTime - FileName: FileName - Image: Image - OriginalFilename: OriginalFilename - PreviousCreationUtcTime: PreviousCreationUtcTime - TargetFilename: TargetFilename - User: User diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_rename.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_file_rename.yml deleted file mode 100644 index a414634f..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_file_rename.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Splunk -source: windows_file_rename -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - CreationUtcTime: CreationUtcTime - FileName: FileName - Image: Image - OriginalFilename: OriginalFilename - PreviousCreationUtcTime: PreviousCreationUtcTime - TargetFilename: TargetFilename - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_image_load.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_image_load.yml deleted file mode 100644 index 18795300..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_image_load.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Splunk -source: windows_image_load -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - Image: Image - ImageLoaded: ImageLoaded - SignatureStatus: SignatureStatus - OriginalFileName: OriginalFileName - Signed: Signed \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_ldap_debug.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_ldap_debug.yml deleted file mode 100644 index e0277d47..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_ldap_debug.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Splunk -source: windows_ldap_debug -description: Text that describe current mapping - -log_source: - sourceType: [XmlWinEventLog:Microsoft-Windows-LDAP-Client/Debug] - -default_log_source: - sourceType: XmlWinEventLog:Microsoft-Windows-LDAP-Client/Debug - -field_mapping: - EventID: EventID - SearchFilter: SearchFilter \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_network_connection.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_network_connection.yml deleted file mode 100644 index 6d10f3fb..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_network_connection.yml +++ /dev/null @@ -1,20 +0,0 @@ -platform: Splunk -source: windows_network_connection -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - Image: Image - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationPort: DestinationPort - SourceIp: SourceIp - SourcePort: SourcePort - Initiated: Initiated \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_ntlm.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_ntlm.yml deleted file mode 100644 index a3a1ce73..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_ntlm.yml +++ /dev/null @@ -1,15 +0,0 @@ -platform: Splunk -source: windows_ntlm -description: Text that describe current mapping - -log_source: - sourceType: [XmlWinEventLog:Microsoft-Windows-NTLM/Operational] - -default_log_source: - sourceType: XmlWinEventLog:Microsoft-Windows-NTLM/Operational - -field_mapping: - WorkstationName: WorkstationName - ProcessName: ProcessName - EventID: EventID - TargetName: TargetName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_powershell.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_powershell.yml deleted file mode 100644 index 8dc67d65..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_powershell.yml +++ /dev/null @@ -1,19 +0,0 @@ -platform: Splunk -source: windows_powershell -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-PowerShell/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-PowerShell/Operational - -field_mapping: - EventID: EventID - CommandLine: CommandLine - ScriptBlockText: ScriptBlockText - Payload: Payload - HostApplication: HostApplication - ContextInfo: ContextInfo - HostName: HostName - EngineVersion: EngineVersion \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_process_creation.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_process_creation.yml deleted file mode 100644 index ae7381b2..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_process_creation.yml +++ /dev/null @@ -1,21 +0,0 @@ -platform: Splunk -source: windows_process_creation -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational, WinEventLog:Security] - -default_log_source: - source: WinEventLog:Security - -field_mapping: - CommandLine: CommandLine - CurrentDirectory: CurrentDirectory - Hashes: Hashes - Image: Image - IntegrityLevel: IntegrityLevel - ParentCommandLine: ParentCommandLine - ParentImage: ParentImage - ParentUser: ParentUser - Product: Product - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_registry_event.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_registry_event.yml deleted file mode 100644 index ef5c702b..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_registry_event.yml +++ /dev/null @@ -1,18 +0,0 @@ -platform: Splunk -source: windows_registry_event -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - TargetObject: TargetObject - Image: Image - Details: Details - EventType: EventType - User: User \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_security.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_security.yml deleted file mode 100644 index bcc2f573..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_security.yml +++ /dev/null @@ -1,148 +0,0 @@ -platform: Splunk -source: windows_security -description: Text that describe current mapping - -log_source: - source: [WinEventLog:Security] - -default_log_source: - source: WinEventLog:Security - -field_mapping: - EventID: EventID - ParentImage: ParentImage - AccessMask: AccessMask - AccountName: AccountName - AllowedToDelegateTo: AllowedToDelegateTo - AttributeLDAPDisplayName: AttributeLDAPDisplayName - AuditPolicyChanges: AuditPolicyChanges - AuthenticationPackageName: AuthenticationPackageName - CallingProcessName: CallingProcessName - Channel: Channel - ComputerName: ComputerName - EventType: EventType - FailureReason: FailureReason - FileName: FileName - GrantedAccess: GrantedAccess - Hashes: Hashes - HiveName: HiveName - IpAddress: IpAddress - IpPort: IpPort - KeyLength: KeyLength - LogonProcessName: LogonProcessName - LogonType: LogonType - LinkName: LinkName - MemberName: MemberName - MemberSid: MemberSid - NewProcessName: NewProcessName - ObjectClass: ObjectClass - ObjectName: ObjectName - ObjectType: ObjectType - ObjectValueName: ObjectValueName - Path: Path - CommandLine: CommandLine - OldUacValue: OldUacValue - CertIssuerName: CertIssuerName - SubStatus: SubStatus - DisplayName: DisplayName - TaskContent: TaskContent - ServiceSid: ServiceSid - CertThumbprint: CertThumbprint - ObjectName: ObjectName - ClassName: ClassName - NotificationPackageName: NotificationPackageName - NewSd: NewSd - TestSigning: TestSigning - TargetInfo: TargetInfo - ClientProcessId: ClientProcessId - ParentProcessId: ParentProcessId - AccessList: AccessList - GroupMembership: GroupMembership - FilterName: FilterName - ChangeType: ChangeType - LayerName: LayerName - ServiceAccount: ServiceAccount - ClientProcessId: ClientProcessId - AttributeValue: AttributeValue - SessionName: SessionName - TaskName: TaskName - ObjectDN: ObjectDN - TemplateContent: TemplateContent - NewTemplateContent: NewTemplateContent - SourcePort: SourcePort - PasswordLastSet: PasswordLastSet - PrivilegeList: PrivilegeList - DeviceDescription: DeviceDescription - TargetServerName: TargetServerName - NewTargetUserName: NewTargetUserName - OperationType: OperationType - DestPort: DestPort - ServiceStartType: ServiceStartType - OldTargetUserName: OldTargetUserName - UserPrincipalName: UserPrincipalName - Accesses: Accesses - DnsHostName: DnsHostName - DisableIntegrityChecks: DisableIntegrityChecks - AuditSourceName: AuditSourceName - Workstation: Workstation - DestAddress: DestAddress - PreAuthType: PreAuthType - SecurityPackageName: SecurityPackageName - SubjectLogonId: SubjectLogonId - NewUacValue: NewUacValue - EnabledPrivilegeList: EnabledPrivilegeList - RelativeTargetName: RelativeTargetName - CertSerialNumber: CertSerialNumber - SidHistory: SidHistory - TargetLogonId: TargetLogonId - KernelDebug: KernelDebug - CallerProcessName: CallerProcessName - ProcessName: ProcessName - Properties: Properties - UserAccountControl: UserAccountControl - RegistryValue: RegistryValue - SecurityID: SecurityID - ServiceFileName: ServiceFileName - SecurityDescriptor: SecurityDescriptor - ServiceName: ServiceName - ShareName: ShareName - NewValue: NewValue - Source: Source - Status: Status - SubjectDomainName: SubjectDomainName - SubjectUserName: SubjectUserName - SubjectUserSid: SubjectUserSid - SourceAddr: SourceAddr - SourceAddress: SourceAddress - TargetName: TargetName - ServicePrincipalNames: ServicePrincipalNames - TargetDomainName: TargetDomainName - TargetSid: TargetSid - TargetUserName: TargetUserName - ObjectServer: ObjectServer - TargetUserSid: TargetUserSid - TicketEncryptionType: TicketEncryptionType - TicketOptions: TicketOptions - WorkstationName: WorkstationName - TransmittedServices: TransmittedServices - AuthenticationAlgorithm: AuthenticationAlgorithm - LayerRTID: LayerRTID - BSSID: BSSID - BSSType: BSSType - CipherAlgorithm: CipherAlgorithm - ConnectionId: ConnectionId - ConnectionMode: ConnectionMode - InterfaceDescription: InterfaceDescription - InterfaceGuid: InterfaceGuid - OnexEnabled: OnexEnabled - PHYType: PHYType - ProfileName: ProfileName - SSID: SSID - Domain: Domain - ServiceType: ServiceType - SourceName: SourceName - StartType: StartType - UserID: UserID - ParentProcessName: ParentProcessName - Service: Service - ProcessName: ProcessName \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_sysmon.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_sysmon.yml deleted file mode 100644 index ce20dac7..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_sysmon.yml +++ /dev/null @@ -1,62 +0,0 @@ -platform: Splunk -source: windows_sysmon - -log_source: - source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] - sourceType: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] - -default_log_source: - source: WinEventLog:Microsoft-Windows-Sysmon/Operational - sourceType: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational - -field_mapping: - CommandLine: CommandLine - Image: Image - ParentImage: ParentImage - EventID: EventCode - CallTrace: CallTrace - Company: Company - CurrentDirectory: CurrentDirectory - Description: Description - DestinationHostname: DestinationHostname - DestinationIp: DestinationIp - DestinationIsIpv6: DestinationIsIpv6 - DestinationPort: DestinationPort - DestinationPortName: DestinationPortName - Hashes: Hashes - Initiated: Initiated - IntegrityLevel: IntegrityLevel - ParentCommandLine: ParentCommandLine - Product: Product - Protocol: Protocol - RuleName: RuleName - SourceHostname: SourceHostname - SourceIp: SourceIp - SourceIsIpv6: SourceIsIpv6 - SourcePort: SourcePort - SourcePortName: SourcePortName - TargetFilename: TargetFilename - User: User - OriginalFileName: OriginalFileName - Signed: Signed - Signature: Signature - SignatureStatus: SignatureStatus - TargetObject: TargetObject - Details: Details - QueryName: QueryName - QueryResults: QueryResults - QueryStatus: QueryStatus - IsExecutable: IsExecutable - PipeName: PipeName - ImageLoaded: ImageLoaded - ImagePath: ImagePath - Imphash: Imphash - SourceImage: SourceImage - StartModule: StartModule - TargetImage: TargetImage - Device: Device - ProcessID: ProcessID - FileVersion: FileVersion - StartAddress: StartAddress - StartFunction: StartFunction - EventType: EventType \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/windows_wmi_event.yml b/siem-converter/app/converter/mappings/platforms/splunk/windows_wmi_event.yml deleted file mode 100644 index 9fa9a11c..00000000 --- a/siem-converter/app/converter/mappings/platforms/splunk/windows_wmi_event.yml +++ /dev/null @@ -1,13 +0,0 @@ -platform: Splunk -source: windows_wmi_event -description: Text that describe current mapping - -log_source: - sourceType: [XmlWinEventLog:Microsoft-Windows-WMI-Activity/Operational] - -default_log_source: - sourceType: XmlWinEventLog:Microsoft-Windows-WMI-Activity/Operational - -field_mapping: - Destination: Destination - EventID: EventID \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/utils/load_from_files.py b/siem-converter/app/converter/mappings/utils/load_from_files.py deleted file mode 100644 index 3a1729f8..00000000 --- a/siem-converter/app/converter/mappings/utils/load_from_files.py +++ /dev/null @@ -1,23 +0,0 @@ -import os - -import yaml - -from app.converter.const import APP_PATH - - -class LoaderFileMappings: - base_mapping_filepath = os.path.join(APP_PATH, "mappings/platforms") - - @staticmethod - def load_mapping(mapping_file_path: str): - with open(mapping_file_path) as yaml_obj: - try: - return yaml.safe_load(yaml_obj) - except yaml.YAMLError as err: - print(err) - return {} - - def load_siem_mappings(self, platform_dir: str): - platform_path = os.path.join(self.base_mapping_filepath, platform_dir) - for mapping_file in os.listdir(platform_path): - yield self.load_mapping(mapping_file_path=os.path.join(platform_path, mapping_file)) diff --git a/siem-converter/app/converter/platforms/__init__.py b/siem-converter/app/converter/platforms/__init__.py deleted file mode 100644 index ea79cde2..00000000 --- a/siem-converter/app/converter/platforms/__init__.py +++ /dev/null @@ -1,122 +0,0 @@ -from app.converter.platforms.athena.parsers.athena import AthenaParser -from app.converter.platforms.athena.renders.athena import AthenaQueryRender -from app.converter.platforms.athena.renders.athena_cti import AthenaCTI -from app.converter.platforms.carbonblack.renders.carbonblack_cti import CarbonBlackCTI -from app.converter.platforms.chronicle.parsers.chronicle import ChronicleParser -from app.converter.platforms.chronicle.parsers.chronicle_rule import ChronicleRuleParser -from app.converter.platforms.chronicle.renders.chronicle import ChronicleQueryRender -from app.converter.platforms.chronicle.renders.chronicle_cti import ChronicleQueryCTI -from app.converter.platforms.chronicle.renders.chronicle_rule import ChronicleSecurityRuleRender -from app.converter.platforms.crowdstrike.parsers.crowdstrike import CrowdStrikeParser -from app.converter.platforms.crowdstrike.renders.crowdstrike import CrowdStrikeQueryRender -from app.converter.platforms.crowdstrike.renders.crowdstrike_cti import CrowdStrikeCTI -from app.converter.platforms.elasticsearch.parsers.detection_rule import ElasticSearchRuleParser -from app.converter.platforms.elasticsearch.parsers.elasticsearch import ElasticSearchParser -from app.converter.platforms.elasticsearch.renders.detection_rule import ElasticSearchRuleRender -from app.converter.platforms.elasticsearch.renders.elast_alert import ElastAlertRuleRender -from app.converter.platforms.elasticsearch.renders.elasticsearch import ElasticSearchQueryRender -from app.converter.platforms.elasticsearch.renders.elasticsearch_cti import ElasticsearchCTI -from app.converter.platforms.elasticsearch.renders.kibana import KibanaRuleRender -from app.converter.platforms.elasticsearch.renders.xpack_watcher import XPackWatcherRuleRender -from app.converter.platforms.fireeye_helix.renders.fireeye_helix_cti import FireeyeHelixCTI -from app.converter.platforms.graylog.renders.graylog_cti import GraylogCTI -from app.converter.platforms.logpoint.renders.logpoint_cti import LogpointCTI -from app.converter.platforms.logscale.parsers.logscale import LogScaleParser -from app.converter.platforms.logscale.parsers.logscale_alert import LogScaleAlertParser -from app.converter.platforms.logscale.renders.logscale_cti import LogScaleCTI -from app.converter.platforms.logscale.renders.logscale import LogScaleQueryRender -from app.converter.platforms.logscale.renders.logscale_alert import LogScaleAlertRender -from app.converter.platforms.microsoft.parsers.microsoft_defender import MicrosoftDefenderQueryParser -from app.converter.platforms.microsoft.parsers.microsoft_sentinel import MicrosoftParser -from app.converter.platforms.microsoft.parsers.microsoft_sentinel_rule import MicrosoftRuleParser -from app.converter.platforms.microsoft.renders.microsoft_defender import MicrosoftDefenderQueryRender -from app.converter.platforms.microsoft.renders.microsoft_defender_cti import MicrosoftDefenderCTI -from app.converter.platforms.microsoft.renders.microsoft_sentinel import MicrosoftSentinelQueryRender -from app.converter.platforms.microsoft.renders.microsoft_sentinel_cti import MicrosoftSentinelCTI -from app.converter.platforms.microsoft.renders.microsoft_sentinel_rule import MicrosoftSentinelRuleRender -from app.converter.platforms.opensearch.parsers.opensearch import OpenSearchParser -from app.converter.platforms.opensearch.renders.opensearch import OpenSearchQueryRender -from app.converter.platforms.opensearch.renders.opensearch_cti import OpenSearchCTI -from app.converter.platforms.opensearch.renders.opensearch_rule import OpenSearchRuleRender -from app.converter.platforms.qradar.parsers.qradar import QradarParser -from app.converter.platforms.qradar.renders.qradar import QradarQueryRender -from app.converter.platforms.qradar.renders.qradar_cti import QRadarCTI -from app.converter.platforms.qualys.renders.qualys_cti import QualysCTI -from app.converter.platforms.rsa_netwitness.renders.rsa_netwitness_cti import RSANetwitnessCTI -from app.converter.platforms.securonix.renders.securonix_cti import SecuronixCTI -from app.converter.platforms.sentinel_one.renders.s1_cti import S1EventsCTI -from app.converter.platforms.sigma.parsers.sigma import SigmaParser -from app.converter.platforms.sigma.renders.sigma import SigmaRender -from app.converter.platforms.snowflake.renders.snowflake_cti import SnowflakeCTI -from app.converter.platforms.splunk.parsers.splunk import SplunkParser -from app.converter.platforms.splunk.parsers.splunk_alert import SplunkAlertParser -from app.converter.platforms.splunk.renders.splunk import SplunkQueryRender -from app.converter.platforms.splunk.renders.splunk_alert import SplunkAlertRender -from app.converter.platforms.splunk.renders.splunk_cti import SplunkCTI -from app.converter.platforms.sumo_logic.renders.sumologic_cti import SumologicCTI - -__ALL_RENDERS = ( - SigmaRender(), - MicrosoftSentinelQueryRender(), - MicrosoftSentinelRuleRender(), - MicrosoftDefenderQueryRender(), - QradarQueryRender(), - CrowdStrikeQueryRender(), - SplunkQueryRender(), - SplunkAlertRender(), - ChronicleQueryRender(), - ChronicleSecurityRuleRender(), - AthenaQueryRender(), - ElasticSearchQueryRender(), - LogScaleQueryRender(), - LogScaleAlertRender(), - ElasticSearchRuleRender(), - ElastAlertRuleRender(), - KibanaRuleRender(), - XPackWatcherRuleRender(), - OpenSearchQueryRender(), - OpenSearchRuleRender() -) - -__ALL_PARSERS = ( - AthenaParser(), - ChronicleParser(), - ChronicleRuleParser(), - SplunkParser(), - SplunkAlertParser(), - SigmaParser(), - QradarParser(), - MicrosoftParser(), - MicrosoftRuleParser(), - MicrosoftDefenderQueryParser(), - CrowdStrikeParser(), - LogScaleParser(), - LogScaleAlertParser(), - ElasticSearchParser(), - ElasticSearchRuleParser(), - OpenSearchParser() -) - - -__ALL_RENDERS_CTI = ( - MicrosoftSentinelCTI(), - MicrosoftDefenderCTI(), - QRadarCTI(), - SplunkCTI(), - ChronicleQueryCTI(), - CrowdStrikeCTI(), - SumologicCTI(), - ElasticsearchCTI(), - LogScaleCTI(), - OpenSearchCTI(), - FireeyeHelixCTI(), - CarbonBlackCTI(), - GraylogCTI(), - LogpointCTI(), - QualysCTI(), - RSANetwitnessCTI(), - S1EventsCTI(), - SecuronixCTI(), - SnowflakeCTI(), - AthenaCTI() -) diff --git a/siem-converter/app/converter/platforms/athena/const.py b/siem-converter/app/converter/platforms/athena/const.py deleted file mode 100644 index 408768d4..00000000 --- a/siem-converter/app/converter/platforms/athena/const.py +++ /dev/null @@ -1,12 +0,0 @@ -from app.converter.core.models.platform_details import PlatformDetails - -ATHENA_QUERY_DETAILS = { - "siem_type": "athena-sql-query", - "name": "AWS Athena Query", - "group_name": "AWS Athena", - "platform_name": "Query", - "group_id": "athena", - "alt_platform_name": "OCSF" -} - -athena_details = PlatformDetails(**ATHENA_QUERY_DETAILS) diff --git a/siem-converter/app/converter/platforms/athena/mapping.py b/siem-converter/app/converter/platforms/athena/mapping.py deleted file mode 100644 index 1a604111..00000000 --- a/siem-converter/app/converter/platforms/athena/mapping.py +++ /dev/null @@ -1,44 +0,0 @@ -from typing import List, Optional - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class AthenaLogSourceSignature(LogSourceSignature): - def __init__(self, tables: Optional[List[str]], default_source: dict): - self.tables = set(tables or []) - self._default_source = default_source or {} - - def is_suitable(self, table: str) -> bool: - return table in self.tables - - def __str__(self) -> str: - return self._default_source.get("table", "") - - -class AthenaMappings(BasePlatformMappings): - def prepare_log_source_signature(self, mapping: dict) -> AthenaLogSourceSignature: - tables = mapping.get("log_source", {}).get("table") - default_log_source = mapping["default_log_source"] - return AthenaLogSourceSignature(tables=tables, default_source=default_log_source) - - def get_suitable_source_mappings(self, field_names: List[str], table: Optional[str]) -> List[SourceMapping]: - suitable_source_mappings = [] - for source_mapping in self._source_mappings.values(): - if source_mapping.source_id == DEFAULT_MAPPING_NAME: - continue - - log_source_signature: AthenaLogSourceSignature = source_mapping.log_source_signature - if table and log_source_signature.is_suitable(table=table): - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - else: - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - - if not suitable_source_mappings: - suitable_source_mappings = [self._source_mappings[DEFAULT_MAPPING_NAME]] - - return suitable_source_mappings - - -athena_mappings = AthenaMappings(platform_dir="athena") diff --git a/siem-converter/app/converter/platforms/athena/mappings/athena_cti.py b/siem-converter/app/converter/platforms/athena/mappings/athena_cti.py deleted file mode 100644 index 5aa4f796..00000000 --- a/siem-converter/app/converter/platforms/athena/mappings/athena_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_ATHENA_MAPPING = { - "SourceIP": "src_endpoint", - "DestinationIP": "dst_endpoint", - "Domain": "dst_endpoint", - "URL": "http_request", - "HashMd5": "unmapped.file.hash.md5", - "HashSha1": "unmapped.file.hash.sha1", - "HashSha256": "unmapped.file.hash.sha256", - "HashSha512": "unmapped.file.hash.sha512", - "Email": "email", - "FileName": "file.name" -} diff --git a/siem-converter/app/converter/platforms/athena/parsers/athena.py b/siem-converter/app/converter/platforms/athena/parsers/athena.py deleted file mode 100644 index f1cc04cf..00000000 --- a/siem-converter/app/converter/platforms/athena/parsers/athena.py +++ /dev/null @@ -1,57 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import List, Tuple, Dict, Optional - -from app.converter.platforms.athena.const import athena_details -from app.converter.platforms.athena.mapping import athena_mappings, AthenaMappings -from app.converter.platforms.athena.tokenizer import AthenaTokenizer -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.parser import Parser -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class AthenaParser(Parser): - details: PlatformDetails = athena_details - mappings: AthenaMappings = athena_mappings - tokenizer = AthenaTokenizer() - query_delimiter_pattern = r"\sFROM\s\S*\sWHERE\s" - table_pattern = r"\sFROM\s(?P[a-zA-Z\.\-\*]+)\sWHERE\s" - - @staticmethod - def _get_meta_info(source_mapping_ids: List[str]) -> MetaInfoContainer: - return MetaInfoContainer(source_mapping_ids=source_mapping_ids) - - def _parse_query(self, text: str) -> Tuple[str, Dict[str, Optional[str]]]: - log_source = {"table": None} - if re.search(self.query_delimiter_pattern, text, flags=re.IGNORECASE): - table_search = re.search(self.table_pattern, text) - table = table_search.group("table") - log_source["table"] = table - return re.split(self.query_delimiter_pattern, text, flags=re.IGNORECASE)[1], log_source - - return text, log_source - - def parse(self, text: str) -> SiemContainer: - query, log_sources = self._parse_query(text) - tokens, source_mappings = self.get_tokens_and_source_mappings(query, log_sources) - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info([source_mapping.source_id for source_mapping in source_mappings]), - ) diff --git a/siem-converter/app/converter/platforms/athena/renders/athena.py b/siem-converter/app/converter/platforms/athena/renders/athena.py deleted file mode 100644 index cf9b3dc3..00000000 --- a/siem-converter/app/converter/platforms/athena/renders/athena.py +++ /dev/null @@ -1,75 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.athena.const import athena_details -from app.converter.platforms.athena.mapping import AthenaMappings, athena_mappings -from app.converter.core.exceptions.render import UnsupportedRenderMethod -from app.converter.core.mapping import LogSourceSignature -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render import BaseQueryRender, BaseQueryFieldValue - - -class AthenaFieldValue(BaseQueryFieldValue): - details: PlatformDetails = athena_details - - def equal_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join([self.equal_modifier(field=field, value=v) for v in value])})" - return f"{field} = '{value}'" - - def contains_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" - return f"{field} ILIKE '%{value}%' ESCAPE '\\'" - - def endswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" - return f"{field} ILIKE '%{value}' ESCAPE '\\'" - - def startswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" - return f"{field} ILIKE '{value}%' ESCAPE '\\'" - - def regex_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" - return f"{field} ILIKE '{value}' ESCAPE '\\'" - - def keywords(self, field, value): - raise UnsupportedRenderMethod(platform_name=self.details.name, method="Keywords") - - -class AthenaQueryRender(BaseQueryRender): - details: PlatformDetails = athena_details - mappings: AthenaMappings = athena_mappings - - or_token = "OR" - and_token = "AND" - not_token = "NOT" - - field_value_map = AthenaFieldValue(or_token=or_token) - query_pattern = "{prefix} WHERE {query} {functions}" - comment_symbol = "--" - is_multi_line_comment = True - - def generate_prefix(self, log_source_signature: LogSourceSignature) -> str: - table = str(log_source_signature) if str(log_source_signature) else "eventlog" - return f"SELECT * FROM {table}" diff --git a/siem-converter/app/converter/platforms/athena/renders/athena_cti.py b/siem-converter/app/converter/platforms/athena/renders/athena_cti.py deleted file mode 100644 index f1da0734..00000000 --- a/siem-converter/app/converter/platforms/athena/renders/athena_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.athena.const import athena_details -from app.converter.platforms.athena.mappings.athena_cti import DEFAULT_ATHENA_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class AthenaCTI(RenderCTI): - details: PlatformDetails = athena_details - - data_map: str = "{key} = '{value}'" - or_operator: str = " OR " - group_or_operator: str = " OR " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "SELECT * from eventlog where {result}\n" - final_result_for_one: str = "SELECT * from eventlog where {result}\n" - default_mapping = DEFAULT_ATHENA_MAPPING diff --git a/siem-converter/app/converter/platforms/athena/tokenizer.py b/siem-converter/app/converter/platforms/athena/tokenizer.py deleted file mode 100644 index 4ce65d6d..00000000 --- a/siem-converter/app/converter/platforms/athena/tokenizer.py +++ /dev/null @@ -1,83 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Tuple, Any - -from app.converter.core.models.identifier import Identifier -from app.converter.core.tokenizer import QueryTokenizer -from app.converter.core.operator_types.tokens import OperatorType -from app.converter.tools.utils import get_match_group - - -class AthenaTokenizer(QueryTokenizer): - field_pattern = r'(?P"[a-zA-Z\._\-\s]+"|[a-zA-Z\._\-]+)' - match_operator_pattern = r"""(?:___field___\s?(?Plike|in|=|>|<|>=|<=|<>|!=))\s?""" - num_value_pattern = r"(?P\d+(?:\.\d+)*)\s*" - bool_value_pattern = r"(?Ptrue|false)\s*" - single_quotes_value_pattern = r"""'(?P(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{\}\s]|'')*)'""" - _value_pattern = fr"{num_value_pattern}|{bool_value_pattern}|{single_quotes_value_pattern}" - multi_value_pattern = r"""\((?P\d+(?:,\s*\d+)*|'(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{\}\s]|'')*'(?:,\s*'(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{\}\s]|'')*')*)\)""" - - multi_value_operators = ("in",) - wildcard_symbol = "%" - operators_map = { - "like": OperatorType.EQ - } - - def __init__(self): - super().__init__() - self.operators_map.update(super().operators_map) - - @staticmethod - def should_process_value_wildcard_symbols(operator: str) -> bool: - return operator.lower() in ("like",) - - def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> Tuple[str, Any]: - if (num_value := get_match_group(match, group_name='num_value')) is not None: - return operator, num_value - - elif (bool_value := get_match_group(match, group_name='bool_value')) is not None: - return operator, bool_value - - elif (s_q_value := get_match_group(match, group_name='s_q_value')) is not None: - return operator, s_q_value - - return super().get_operator_and_value(match, operator) - - def search_field_value(self, query): - field_name = self.search_field(query) - operator = self.search_match_operator(query, field_name) - should_process_value_wildcard_symbols = self.should_process_value_wildcard_symbols(operator) - query, operator, value = self.search_value(query=query, operator=operator, field_name=field_name) - - operator_token = Identifier(token_type=OperatorType.EQ) - if should_process_value_wildcard_symbols: - value, operator_token = self.process_value_wildcard_symbols( - value=value, - operator=operator, - wildcard_symbol=self.wildcard_symbol - ) - - field_name = field_name.strip('"') - field = self.create_field(field_name=field_name, operator=operator_token, value=value) - return field, query - - def tokenize(self, query: str) -> list: - query = re.sub(r"\s*ESCAPE\s*'.'", '', query) # remove `ESCAPE 'escape_char'` in LIKE expr - return super().tokenize(query) diff --git a/siem-converter/app/converter/platforms/base/lucene/mapping.py b/siem-converter/app/converter/platforms/base/lucene/mapping.py deleted file mode 100644 index d44700fa..00000000 --- a/siem-converter/app/converter/platforms/base/lucene/mapping.py +++ /dev/null @@ -1,41 +0,0 @@ -from typing import List, Optional - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class LuceneLogSourceSignature(LogSourceSignature): - def __init__(self, indices: Optional[List[str]], default_source: dict): - self.indices = set(indices or []) - self._default_source = default_source or {} - - def is_suitable(self, index: Optional[List[str]]) -> bool: - return set(index or []).issubset(self.indices) - - def __str__(self) -> str: - return self._default_source.get("index", "") - - -class LuceneMappings(BasePlatformMappings): - def prepare_log_source_signature(self, mapping: dict) -> LuceneLogSourceSignature: - indices = mapping.get("log_source", {}).get("index") - default_log_source = mapping["default_log_source"] - return LuceneLogSourceSignature(indices=indices, default_source=default_log_source) - - def get_suitable_source_mappings(self, field_names: List[str], index: List[str] = None) -> List[SourceMapping]: - suitable_source_mappings = [] - for source_mapping in self._source_mappings.values(): - if source_mapping.source_id == DEFAULT_MAPPING_NAME: - continue - - log_source_signature: LuceneLogSourceSignature = source_mapping.log_source_signature - if index and log_source_signature.is_suitable(index=index): - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - else: - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - - if not suitable_source_mappings: - suitable_source_mappings = [self._source_mappings[DEFAULT_MAPPING_NAME]] - - return suitable_source_mappings diff --git a/siem-converter/app/converter/platforms/base/lucene/parsers/lucene.py b/siem-converter/app/converter/platforms/base/lucene/parsers/lucene.py deleted file mode 100644 index 52ea134e..00000000 --- a/siem-converter/app/converter/platforms/base/lucene/parsers/lucene.py +++ /dev/null @@ -1,60 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import List, Tuple, Dict - -from app.converter.platforms.base.lucene.tokenizer import LuceneTokenizer -from app.converter.core.parser import Parser -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class LuceneParser(Parser): - tokenizer = LuceneTokenizer() - - log_source_pattern = r"___source_type___\s*(?:[:=])\s*(?:\"?(?P[%a-zA-Z_*:0-9\-/]+)\"|(?P[%a-zA-Z_*:0-9\-/]+))(?:\s+(?:and|or)\s+|\s+)?" - log_source_key_types = ("index", "event\.category") - - def _parse_log_sources(self, query: str) -> Tuple[str, Dict[str, List[str]]]: - log_sources = {} - for source_type in self.log_source_key_types: - pattern = self.log_source_pattern.replace('___source_type___', source_type) - while search := re.search(pattern, query, flags=re.IGNORECASE): - group_dict = search.groupdict() - value = group_dict.get("d_q_value") or group_dict.get("value") - log_sources.setdefault(source_type, []).append(value) - pos_start = search.start() - pos_end = search.end() - query = query[:pos_start] + query[pos_end:] - - return query, log_sources - - @staticmethod - def _get_meta_info(source_mapping_ids: List[str], meta_info: dict) -> MetaInfoContainer: - return MetaInfoContainer(source_mapping_ids=source_mapping_ids) - - def _parse_query(self, query: str) -> Tuple[str, Dict[str, List[str]]]: - return self._parse_log_sources(query) - - def parse(self, text: str) -> SiemContainer: - query, log_sources = self._parse_query(text) - tokens, source_mappings = self.get_tokens_and_source_mappings(query, log_sources) - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info([source_mapping.source_id for source_mapping in source_mappings], {}), - ) diff --git a/siem-converter/app/converter/platforms/base/lucene/renders/lucene.py b/siem-converter/app/converter/platforms/base/lucene/renders/lucene.py deleted file mode 100644 index 3f73d180..00000000 --- a/siem-converter/app/converter/platforms/base/lucene/renders/lucene.py +++ /dev/null @@ -1,81 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" -from typing import Union - -from app.converter.core.render import BaseQueryRender -from app.converter.core.render import BaseQueryFieldValue - - -class LuceneFieldValue(BaseQueryFieldValue): - - def apply_value(self, value: Union[str, int]): - return value - - def equal_modifier(self, field, value): - if isinstance(value, list): - values = self.or_token.join(self.apply_value(f'{v}') for v in value) - return f"{field}:({values})" - return f'{field}:{self.apply_value(value)}' - - def contains_modifier(self, field, value): - if isinstance(value, list): - values = self.or_token.join(self.apply_value(f'*{v}*') for v in value) - return f"{field}:({values})" - prepared_value = self.apply_value(f"*{value}*") - return f'{field}:{prepared_value}' - - def endswith_modifier(self, field, value): - if isinstance(value, list): - values = self.or_token.join(self.apply_value(f'*{v}') for v in value) - return f"{field}:({values})" - prepared_value = self.apply_value(f"*{value}") - return f'{field}:{prepared_value}' - - def startswith_modifier(self, field, value): - if isinstance(value, list): - values = self.or_token.join(self.apply_value(f'{v}*') for v in value) - return f"{field}:({values})" - prepared_value = self.apply_value(f"{value}*") - return f'{field}:{prepared_value}' - - def regex_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" - return f'{field}:/{value}/' - - def keywords(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" - return self.apply_value(f"*{value}*") - - -class LuceneQueryRender(BaseQueryRender): - - or_token = "OR" - and_token = "AND" - not_token = "NOT" - - query_pattern = "{query} {functions}" - - comment_symbol = "//" - is_multi_line_comment = True - - def generate_prefix(self, logsource: dict) -> str: - return "" - diff --git a/siem-converter/app/converter/platforms/base/lucene/tokenizer.py b/siem-converter/app/converter/platforms/base/lucene/tokenizer.py deleted file mode 100644 index 3afd2316..00000000 --- a/siem-converter/app/converter/platforms/base/lucene/tokenizer.py +++ /dev/null @@ -1,109 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" -import re - -from typing import Tuple, Union, List, Any - -from app.converter.core.exceptions.parser import TokenizerGeneralException -from app.converter.core.models.field import Keyword, Field -from app.converter.core.models.identifier import Identifier -from app.converter.core.tokenizer import QueryTokenizer -from app.converter.core.operator_types.tokens import OperatorType -from app.converter.tools.utils import get_match_group - - -class LuceneTokenizer(QueryTokenizer): - field_pattern = r"(?P[a-zA-Z\.\-_]+)" - match_operator_pattern = r"(?:___field___\s*(?P:))\s*" - - num_value_pattern = r"(?P\d+(?:\.\d+)*)\s*" - double_quotes_value_pattern = r'"(?P(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{\}\s]|\\\"|\\)*)"\s*' - no_quotes_value_pattern = r"(?P(?:[a-zA-Z\*0-9=%#_/,\'\.$@]|\\\"|\\\\)+)\s*" - re_value_pattern = r"/(?P[:a-zA-Z\*0-9=+%#\\\-_\,\"\'\.$&^@!\(\)\{\}\[\]\s?]+)/\s*" - _value_pattern = fr"{num_value_pattern}|{re_value_pattern}|{no_quotes_value_pattern}|{double_quotes_value_pattern}" - keyword_pattern = r"(?P(?:[a-zA-Z\*0-9=%#_/,\'\.$@]|\\\"|\\\(|\\\)|\\\[|\\\]|\\\{|\\\}|\\\:|\\)+)(?:\s+|\)|$)" - - multi_value_pattern = r"""\((?P[:a-zA-Z\"\*0-9=+%#\-_\/\\'\,.&^@!\(\[\]\s]+)\)""" - multi_value_check_pattern = r"___field___\s*___operator___\s*\(" - - wildcard_symbol = "*" - - operators_map = { - ":": OperatorType.EQ, - } - - def __init__(self): - super().__init__() - self.operators_map.update(super().operators_map) - - @staticmethod - def create_field(field_name: str, operator: Identifier, value: Union[str, List]) -> Field: - field_name = field_name.replace(".text", "") - field_name = field_name.replace(".keyword", "") - return Field(operator=operator, value=value, source_name=field_name) - - @staticmethod - def clean_quotes(value: Union[str, int]): - if isinstance(value, str): - return value.strip('"') if value.startswith('"') and value.endswith('"') else value - return value - - def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> Tuple[str, Any]: - if (num_value := get_match_group(match, group_name='num_value')) is not None: - return operator, num_value - - elif (re_value := get_match_group(match, group_name='re_value')) is not None: - return OperatorType.REGEX, re_value - - elif (n_q_value := get_match_group(match, group_name='n_q_value')) is not None: - return operator, n_q_value - - elif (d_q_value := get_match_group(match, group_name='d_q_value')) is not None: - return operator, d_q_value - - return super().get_operator_and_value(match) - - def search_value(self, query: str, operator: str, field_name: str) -> Tuple[str, str, Union[str, List[str]]]: - check_pattern = self.multi_value_check_pattern - check_regex = check_pattern.replace('___field___', field_name).replace('___operator___', operator) - if re.match(check_regex, query): - value_pattern = self.multi_value_pattern - is_multi = True - else: - value_pattern = self.value_pattern - is_multi = False - - field_value_pattern = self.get_field_value_pattern(operator, field_name) - field_value_pattern = field_value_pattern.replace("___value___", value_pattern) - field_value_regex = re.compile(field_value_pattern, re.IGNORECASE) - field_value_search = re.search(field_value_regex, query) - if field_value_search is None: - raise TokenizerGeneralException(error=f"Value couldn't be found in query part: {query}") - - operator, value = self.get_operator_and_value(field_value_search) - value = [self.clean_quotes(v) for v in re.split(r"\s+OR\s+", value)] if is_multi else value - pos = field_value_search.end() - return query[pos:], operator, value - - def search_keyword(self, query: str) -> Tuple[Keyword, str]: - keyword_search = re.search(self.keyword_pattern, query) - _, value = self.get_operator_and_value(keyword_search) - value = value.strip(self.wildcard_symbol) - keyword = Keyword(value=value) - pos = keyword_search.end() - 1 - return keyword, query[pos:] diff --git a/siem-converter/app/converter/platforms/base/spl/parsers/spl.py b/siem-converter/app/converter/platforms/base/spl/parsers/spl.py deleted file mode 100644 index 6d5afc62..00000000 --- a/siem-converter/app/converter/platforms/base/spl/parsers/spl.py +++ /dev/null @@ -1,81 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Tuple, List, Dict, Optional - -from app.converter.platforms.base.spl.tokenizer import SplTokenizer -from app.converter.core.models.functions.types import ParsedFunctions -from app.converter.core.parser import Parser -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class SplParser(Parser): - siem_functions = None - - log_source_pattern = r"___source_type___\s*=\s*(?:\"(?P[%a-zA-Z_*:0-9\-/]+)\"|(?P[%a-zA-Z_*:0-9\-/]+))(?:\s+(?:and|or)\s+|\s+)?" - log_source_key_types = ("index", "source", "sourcetype", "sourcecategory") - - tokenizer = SplTokenizer() - - def _parse_log_sources(self, query: str) -> Tuple[Dict[str, List[str]], str]: - log_sources = {} - for source_type in self.log_source_key_types: - log_sources.setdefault(source_type, []) - pattern = self.log_source_pattern.replace('___source_type___', source_type) - while search := re.search(pattern, query, flags=re.IGNORECASE): - group_dict = search.groupdict() - value = group_dict.get("d_q_value") or group_dict.get("value") - log_sources.setdefault(source_type, []).append(value) - pos_start = search.start() - pos_end = search.end() - query = query[:pos_start] + query[pos_end:] - - return log_sources, query - - def _parse_functions(self, query: str) -> Tuple[ParsedFunctions, str]: - if search_result := re.search(r"\|(.+?)$", query): - parsed_functions = self.siem_functions.parse(search_result.group().strip(" |")) - pos_start = search_result.start() - pos_end = search_result.end() - query = query[:pos_start] + query[pos_end:] - return parsed_functions, query.strip(" ") - - return ParsedFunctions(), query - - def _parse_query(self, query: str) -> Tuple[Dict[str, List[str]], ParsedFunctions, str]: - query = query.strip() - log_sources, query = self._parse_log_sources(query) - functions, query = self._parse_functions(query) - return log_sources, functions, query - - @staticmethod - def _get_meta_info(source_mapping_ids: List[str], meta_info: Optional[dict]) -> MetaInfoContainer: - return MetaInfoContainer( - source_mapping_ids=source_mapping_ids - ) - - def parse(self, text: str) -> SiemContainer: - log_sources, functions, query = self._parse_query(text) - tokens, source_mappings = self.get_tokens_and_source_mappings(query, log_sources) - - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info([source_mapping.source_id for source_mapping in source_mappings], {}), - functions=functions - ) diff --git a/siem-converter/app/converter/platforms/base/spl/renders/spl.py b/siem-converter/app/converter/platforms/base/spl/renders/spl.py deleted file mode 100644 index b9dafb54..00000000 --- a/siem-converter/app/converter/platforms/base/spl/renders/spl.py +++ /dev/null @@ -1,65 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.core.exceptions.render import UnsupportedRenderMethod -from app.converter.core.render import BaseQueryRender, BaseQueryFieldValue - - -class SplFieldValue(BaseQueryFieldValue): - - def equal_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join([self.equal_modifier(field=field, value=v) for v in value])})" - return f'{field}="{value}"' - - def contains_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join([self.contains_modifier(field=field, value=v) for v in value])})" - return f'{field}="*{value}*"' - - def endswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join([self.endswith_modifier(field=field, value=v) for v in value])})" - return f'{field}="*{value}"' - - def startswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join([self.startswith_modifier(field=field, value=v) for v in value])})" - return f'{field}="{value}*"' - - def keywords(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" - return f'"{value}"' - - def regex_modifier(self, field, value): - raise UnsupportedRenderMethod(platform_name=self.details.name, method="Regex Expression") - - -class SplQueryRender(BaseQueryRender): - - or_token = "OR" - and_token = "AND" - not_token = "NOT" - - query_pattern = "{prefix} {query} {functions}" - comment_symbol = '```' - - def wrap_with_comment(self, value: str) -> str: - return f"{self.comment_symbol} {value} {self.comment_symbol}" diff --git a/siem-converter/app/converter/platforms/base/spl/tokenizer.py b/siem-converter/app/converter/platforms/base/spl/tokenizer.py deleted file mode 100644 index fc920181..00000000 --- a/siem-converter/app/converter/platforms/base/spl/tokenizer.py +++ /dev/null @@ -1,53 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Tuple, Any - -from app.converter.core.tokenizer import QueryTokenizer -from app.converter.core.operator_types.tokens import OperatorType -from app.converter.tools.utils import get_match_group - - -class SplTokenizer(QueryTokenizer): - field_pattern = r"(?P[a-zA-Z\.\-_\{\}]+)" - num_value_pattern = r"(?P\d+(?:\.\d+)*)\s*" - double_quotes_value_pattern = r'"(?P(?:[:a-zA-Z\*0-9=+%#\-_/,;\'\.$&^@!\(\)\{\}\s]|\\\"|\\)*)"\s*' - single_quotes_value_pattern = r"'(?P(?:[:a-zA-Z\*0-9=+%#\-_/,;\"\.$&^@!\(\)\{\}\s]|\\\'|\\)*)'\s*" - no_quotes_value = r"(?P(?:[:a-zA-Z\*0-9=+%#\-_/,\.\\$&^@!])+)\s*" - _value_pattern = fr"{num_value_pattern}|{no_quotes_value}|{double_quotes_value_pattern}|{single_quotes_value_pattern}" - multi_value_pattern = r"""\((?P[:a-zA-Z\"\*0-9=+%#\-_\/\\'\,;.$&^@!\{\}\(\s]+)\)""" - keyword_pattern = double_quotes_value_pattern - - multi_value_operators = ("in",) - wildcard_symbol = "*" - - def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> Tuple[str, Any]: - if (num_value := get_match_group(match, group_name='num_value')) is not None: - return operator, num_value - - elif (no_q_value := get_match_group(match, group_name='no_q_value')) is not None: - return operator, no_q_value - - elif (d_q_value := get_match_group(match, group_name='d_q_value')) is not None: - return operator, d_q_value - - elif (s_q_value := get_match_group(match, group_name='s_q_value')) is not None: - return operator, s_q_value - - return super().get_operator_and_value(match) diff --git a/siem-converter/app/converter/platforms/carbonblack/const.py b/siem-converter/app/converter/platforms/carbonblack/const.py deleted file mode 100644 index a3fbac83..00000000 --- a/siem-converter/app/converter/platforms/carbonblack/const.py +++ /dev/null @@ -1,8 +0,0 @@ - -CARBON_BLACK_QUERY_DETAILS = { - "siem_type": "carbonblack", - "name": "Carbon Black Cloud", - "group_name": "VMware Carbon Black", - "group_id": "carbonblack-pack", - "platform_name": "Query (Cloud)", -} diff --git a/siem-converter/app/converter/platforms/carbonblack/mappings/carbonblack_cti.py b/siem-converter/app/converter/platforms/carbonblack/mappings/carbonblack_cti.py deleted file mode 100644 index 9249d785..00000000 --- a/siem-converter/app/converter/platforms/carbonblack/mappings/carbonblack_cti.py +++ /dev/null @@ -1,10 +0,0 @@ -DEFAULT_CARBONBLACK_MAPPING = { - "SourceIP": "netconn_local_ipv4", - "DestinationIP": "netconn_ipv4", - "Domain": "netconn_domain", - "URL": "netconn_domain", - "HashMd5": "hash", - "HashSha256": "hash", - "Files": "filemod_name", - "Emails": "process_username" -} diff --git a/siem-converter/app/converter/platforms/carbonblack/renders/carbonblack_cti.py b/siem-converter/app/converter/platforms/carbonblack/renders/carbonblack_cti.py deleted file mode 100644 index 97db6d54..00000000 --- a/siem-converter/app/converter/platforms/carbonblack/renders/carbonblack_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.carbonblack.const import CARBON_BLACK_QUERY_DETAILS -from app.converter.platforms.carbonblack.mappings.carbonblack_cti import DEFAULT_CARBONBLACK_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class CarbonBlackCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**CARBON_BLACK_QUERY_DETAILS) - - data_map: str = "{key}:{value}" - or_operator: str = " OR " - group_or_operator: str = " OR " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_CARBONBLACK_MAPPING diff --git a/siem-converter/app/converter/platforms/chronicle/const.py b/siem-converter/app/converter/platforms/chronicle/const.py deleted file mode 100644 index de66c1af..00000000 --- a/siem-converter/app/converter/platforms/chronicle/const.py +++ /dev/null @@ -1,41 +0,0 @@ -from app.converter.core.models.platform_details import PlatformDetails - -DEFAULT_CHRONICLE_SECURITY_RULE = """rule { - meta: - description = "" - license = "" - version = "0.01" - rule_id = "" - status = "" - severity = "" - - events: - - - condition: - $e -}""" - -PLATFORM_DETAILS = { - "group_id": "chronicle-pack", - "group_name": "Chronicle Security", - "alt_platform_name": "UDM" -} - -CHRONICLE_QUERY_DETAILS = { - "siem_type": "chronicle-yaral-query", - "name": "Chronicle Security Query", - "platform_name": "Query (UDM)", - **PLATFORM_DETAILS -} - -CHRONICLE_RULE_DETAILS = { - "siem_type": "chronicle-yaral-rule", - "name": "Chronicle Security Rule", - "platform_name": "Rule (YARA-L)", - "first_choice": 0, - **PLATFORM_DETAILS -} - -chronicle_query_details = PlatformDetails(**CHRONICLE_QUERY_DETAILS) -chronicle_rule_details = PlatformDetails(**CHRONICLE_RULE_DETAILS) diff --git a/siem-converter/app/converter/platforms/chronicle/mapping.py b/siem-converter/app/converter/platforms/chronicle/mapping.py deleted file mode 100644 index 381528a8..00000000 --- a/siem-converter/app/converter/platforms/chronicle/mapping.py +++ /dev/null @@ -1,31 +0,0 @@ -from typing import List - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class ChronicleLogSourceSignature(LogSourceSignature): - - def __str__(self) -> str: - return "" - - -class ChronicleMappings(BasePlatformMappings): - def prepare_log_source_signature(self, mapping: dict) -> ChronicleLogSourceSignature: - return - - def get_suitable_source_mappings(self, field_names: List[str]) -> List[SourceMapping]: - suitable_source_mappings = [] - for source_mapping in self._source_mappings.values(): - if source_mapping.source_id == DEFAULT_MAPPING_NAME: - continue - - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - - if not suitable_source_mappings: - suitable_source_mappings = [self._source_mappings[DEFAULT_MAPPING_NAME]] - - return suitable_source_mappings - - -chronicle_mappings = ChronicleMappings(platform_dir="chronicle") diff --git a/siem-converter/app/converter/platforms/chronicle/mappings/chronicle_cti.py b/siem-converter/app/converter/platforms/chronicle/mappings/chronicle_cti.py deleted file mode 100644 index 042a8a07..00000000 --- a/siem-converter/app/converter/platforms/chronicle/mappings/chronicle_cti.py +++ /dev/null @@ -1,11 +0,0 @@ -DEFAULT_CHRONICLE_MAPPING = { - "DestinationIP": "target.ip", - "SourceIP": "principal.ip", - "HashSha256": "target.file.sha256", - "HashMd5": "target.file.md5", - "Emails": "network.email.from", - "Domain": "target.hostname", - "HashSha1": "target.file.sha1", - "Files": "target.file.full_path", - "URL": "target.url" -} diff --git a/siem-converter/app/converter/platforms/chronicle/parsers/chronicle.py b/siem-converter/app/converter/platforms/chronicle/parsers/chronicle.py deleted file mode 100644 index 97e0183a..00000000 --- a/siem-converter/app/converter/platforms/chronicle/parsers/chronicle.py +++ /dev/null @@ -1,42 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from typing import List - -from app.converter.platforms.chronicle.const import chronicle_query_details -from app.converter.platforms.chronicle.mapping import chronicle_mappings, ChronicleMappings -from app.converter.platforms.chronicle.tokenizer import ChronicleQueryTokenizer -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.parser import Parser -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class ChronicleParser(Parser): - mappings: ChronicleMappings = chronicle_mappings - tokenizer: ChronicleQueryTokenizer = ChronicleQueryTokenizer() - details: PlatformDetails = chronicle_query_details - - def _get_meta_info(self, source_mapping_ids: List[str]) -> MetaInfoContainer: - return MetaInfoContainer(source_mapping_ids=source_mapping_ids) - - def parse(self, text: str) -> SiemContainer: - tokens, source_mappings = self.get_tokens_and_source_mappings(text, {}) - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info([source_mapping.source_id for source_mapping in source_mappings]) - ) diff --git a/siem-converter/app/converter/platforms/chronicle/parsers/chronicle_rule.py b/siem-converter/app/converter/platforms/chronicle/parsers/chronicle_rule.py deleted file mode 100644 index 36c2fd38..00000000 --- a/siem-converter/app/converter/platforms/chronicle/parsers/chronicle_rule.py +++ /dev/null @@ -1,99 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import List, Dict - -from app.converter.platforms.chronicle.const import chronicle_rule_details -from app.converter.platforms.chronicle.mapping import ChronicleMappings, chronicle_mappings -from app.converter.platforms.chronicle.tokenizer import ChronicleRuleTokenizer -from app.converter.core.exceptions.parser import TokenizerGeneralException -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.parser import Parser -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class ChronicleRuleParser(Parser): - details: PlatformDetails = chronicle_rule_details - rule_name_pattern = "rule\s(?P[a-z0-9_]+)\s{" - meta_info_pattern = "meta:\n(?P[a-zA-Z0-9_\\\.*,>–<—~#$’`:;%+^\|?!@\s\"/=\-&'\(\)\[\]]+)\n\s+events:" - rule_pattern = "events:\n\s*(?P[a-zA-Z\w0-9_%{}\|\.,!#^><:~\s\"\/=+?\-–&;$()`\*@\[\]'\\\]+)\n\s+condition:" - event_name_pattern = "condition:\n\s*(?P\$[a-zA-Z_0-9]+)\n" - mappings: ChronicleMappings = chronicle_mappings - tokenizer = ChronicleRuleTokenizer() - - def __parse_rule(self, rule) -> Dict: - rule_name_search = re.search(self.rule_name_pattern, rule) - if rule_name_search is None: - raise TokenizerGeneralException(error=f"Field couldn't be found in rule.") - rule_name = rule_name_search.group("rule_name") - - meta_info_search = re.search(self.meta_info_pattern, rule) - if meta_info_search is None: - raise TokenizerGeneralException(error=f"Rule meta info couldn't be found in rule.") - meta_info = meta_info_search.group("meta_info") - - query_search = re.search(self.rule_pattern, rule) - if query_search is None: - raise TokenizerGeneralException(error=f"Query couldn't be found in rule.") - query = query_search.group("rule") - - event_name_search = re.search(self.event_name_pattern, rule) - if query_search is None: - raise TokenizerGeneralException(error=f"Event name couldn't be found in rule.") - event_name = event_name_search.group("event_name") - - query = query.replace(f"{event_name}.", "") - return {"query": query.strip(" ").strip("\n"), "rule_name": rule_name, "meta_info": meta_info} - - @staticmethod - def __prepare_title(name) -> str: - return " ".join(name.split("_")).title() - - def _get_meta_info(self, rule_name: str, source_mapping_ids: List[str], meta_info: str) -> MetaInfoContainer: - references = tags = [] - description = None - for info in meta_info.strip(" ").strip("\n").split("\n"): - key, value = info.split(" = ") - key = key.strip(" ") - if key == "reference": - references = [value.strip(" ").strip('"')] - elif key == "description": - description = value.strip(" ") - elif key == "tags": - tags = [i.strip(" ").strip('"') for i in value.split(",")] - - return MetaInfoContainer( - title=self.__prepare_title(rule_name), - source_mapping_ids=source_mapping_ids, - references=references, - tags=tags, - description=description - ) - - def parse(self, text: str) -> SiemContainer: - parsed_rule = self.__parse_rule(text) - tokens, source_mappings = self.get_tokens_and_source_mappings(parsed_rule.get("query"), {}) - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info( - rule_name=parsed_rule.get("rule_name"), - meta_info=parsed_rule.get("meta_info"), - source_mapping_ids=[source_mapping.source_id for source_mapping in source_mappings] - ), - ) diff --git a/siem-converter/app/converter/platforms/chronicle/renders/chronicle.py b/siem-converter/app/converter/platforms/chronicle/renders/chronicle.py deleted file mode 100644 index d3c3b9dc..00000000 --- a/siem-converter/app/converter/platforms/chronicle/renders/chronicle.py +++ /dev/null @@ -1,81 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.chronicle.const import chronicle_query_details -from app.converter.platforms.chronicle.mapping import ChronicleMappings, chronicle_mappings -from app.converter.core.exceptions.render import UnsupportedRenderMethod -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render import BaseQueryRender, BaseQueryFieldValue - - -class ChronicleFieldValue(BaseQueryFieldValue): - details: PlatformDetails = chronicle_query_details - - @staticmethod - def apply_field(field): - return field - - def equal_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" - return f'{self.apply_field(field)} = "{value}" nocase' - - def contains_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" - return f'{self.apply_field(field)} = /.*{value}.*/ nocase' - - def endswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" - return f'{self.apply_field(field)} = /.*{value}$/ nocase' - - def startswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" - return f'{self.apply_field(field)} = /^{value}.*/ nocase' - - def regex_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" - return f'{self.apply_field(field)} = /{value}/ nocase' - - def keywords(self, field, value): - raise UnsupportedRenderMethod(platform_name=self.details.name, method="Keywords") - - -class ChronicleQueryRender(BaseQueryRender): - details: PlatformDetails = chronicle_query_details - mappings: ChronicleMappings = chronicle_mappings - - is_strict_mapping = True - - or_token = "or" - and_token = "and" - not_token = "not" - - field_value_map = ChronicleFieldValue(or_token=or_token) - query_pattern = "{query} {functions}" - comment_symbol = r"//" - - def generate_prefix(self, logsource: dict): - return "" - - def generate_functions(self, functions: list): - return "" diff --git a/siem-converter/app/converter/platforms/chronicle/renders/chronicle_cti.py b/siem-converter/app/converter/platforms/chronicle/renders/chronicle_cti.py deleted file mode 100644 index 0a4ada79..00000000 --- a/siem-converter/app/converter/platforms/chronicle/renders/chronicle_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.chronicle.const import chronicle_query_details -from app.converter.platforms.chronicle.mappings.chronicle_cti import DEFAULT_CHRONICLE_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class ChronicleQueryCTI(RenderCTI): - details: PlatformDetails = chronicle_query_details - - data_map: str = '{key} = "{value}"' - or_operator: str = " or " - group_or_operator: str = " or " - or_group: str = "{or_group}" - result_join: str = "" - final_result_for_many: str = "{result}\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_CHRONICLE_MAPPING diff --git a/siem-converter/app/converter/platforms/chronicle/renders/chronicle_rule.py b/siem-converter/app/converter/platforms/chronicle/renders/chronicle_rule.py deleted file mode 100644 index 6a1c27b6..00000000 --- a/siem-converter/app/converter/platforms/chronicle/renders/chronicle_rule.py +++ /dev/null @@ -1,98 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -import re - -from app.converter.platforms.chronicle.renders.chronicle import ChronicleFieldValue, ChronicleQueryRender -from app.converter.platforms.chronicle.const import DEFAULT_CHRONICLE_SECURITY_RULE, chronicle_rule_details -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer -from app.converter.tools.utils import concatenate_str, get_author_str - -_AUTOGENERATED_TITLE = "Autogenerated Chronicle Security rule" -_AUTOGENERATED_DESCRIPTION = "Autogenerated Chronicle Security rule." - - -class ChronicleRuleFieldValue(ChronicleFieldValue): - details: PlatformDetails = chronicle_rule_details - - def equal_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" - return f'{self.apply_field(field)} = "{value}"' - - def contains_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" - return f're.regex({self.apply_field(field)}, `.*{value}.*`)' - - def endswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" - return f're.regex({self.apply_field(field)}, `.*{value}`)' - - def startswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" - return f're.regex({self.apply_field(field)}, `{value}.*`)' - - @staticmethod - def apply_field(field): - return f"$e.{field}" - - def regex_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" - return f're.regex({self.apply_field(field)}, `{value}`)' - - -class ChronicleSecurityRuleRender(ChronicleQueryRender): - details: PlatformDetails = chronicle_rule_details - or_token = "or" - field_value_map = ChronicleRuleFieldValue(or_token=or_token) - - @staticmethod - def prepare_title(title: str) -> str: - if not title: - return title - new_title = re.sub(re.compile('[()*‘’:;+!,\\[\\].?`"-/]'), "", title.lower()) - new_title = re.sub(re.compile("\\s"), "_", new_title.lower()) - index = 0 - for i, title_char in enumerate(new_title): - if not title_char.isdigit(): - index = i - break - new_title = new_title[index:] - new_title = new_title.strip("_") - return new_title - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None) -> str: - query = super().finalize_query(prefix=prefix, query=query, functions=functions, meta_info=meta_info) - rule = DEFAULT_CHRONICLE_SECURITY_RULE.replace("", query) - rule = rule.replace("", self.prepare_title(meta_info.title) or _AUTOGENERATED_TITLE) - description = meta_info.description or _AUTOGENERATED_DESCRIPTION - description = concatenate_str(description, get_author_str(meta_info.author)) - rule = rule.replace("", description) - rule = rule.replace("", meta_info.license) - rule = rule.replace("", meta_info.id) - rule = rule.replace("", meta_info.severity) - rule = rule.replace("", meta_info.status) - return rule diff --git a/siem-converter/app/converter/platforms/chronicle/tokenizer.py b/siem-converter/app/converter/platforms/chronicle/tokenizer.py deleted file mode 100644 index a9427911..00000000 --- a/siem-converter/app/converter/platforms/chronicle/tokenizer.py +++ /dev/null @@ -1,103 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Tuple, Any - -from app.converter.core.exceptions.parser import TokenizerGeneralException -from app.converter.core.tokenizer import QueryTokenizer -from app.converter.core.operator_types.tokens import OperatorType -from app.converter.tools.utils import get_match_group - - -class ChronicleQueryTokenizer(QueryTokenizer): - field_pattern = r"(?P[a-zA-Z0-9\._]+)" - - num_value_pattern = r"(?P\d+(?:\.\d+)*)\s*" - bool_value_pattern = r"(?Ptrue|false)\s*" - double_quotes_value_pattern = r'"(?P(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{\}\s]|\\\"|\\\\)*)"\s*(?:nocase)?' - re_value_pattern = r"/(?P(?:\\\/|[:a-zA-Z\*0-9=+%#\\\-_\,\"\'\.$&^@!\(\)\{\}\s?])+)/\s*(?:nocase)?" - _value_pattern = fr"{num_value_pattern}|{bool_value_pattern}|{double_quotes_value_pattern}|{re_value_pattern}" - - wildcard_symbol = ".*" - - def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> Tuple[str, Any]: - if (num_value := get_match_group(match, group_name='num_value')) is not None: - return operator, num_value - - elif (bool_value := get_match_group(match, group_name='bool_value')) is not None: - return operator, bool_value - - elif (d_q_value := get_match_group(match, group_name='d_q_value')) is not None: - return operator, d_q_value - - elif (re_value := get_match_group(match, group_name='re_value')) is not None: - return OperatorType.REGEX, re_value - - return super().get_operator_and_value(match, operator) - - def escape_field_name(self, field_name): - symbols_to_check = [".", "_", "$"] - for symbol in symbols_to_check: - field_name = field_name.replace(symbol, '\\' + symbol) - return field_name - - -class ChronicleRuleTokenizer(ChronicleQueryTokenizer): - field_pattern = r"(?P[$a-zA-Z0-9\._]+)" - regex_field_regex = r"re\.regex\((?P[$a-zA-Z\._]+)," - - double_quotes_value_pattern = r'"(?P(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{\}\s]|\\\"|\\\\)*)"' - back_quotes_value_pattern = r'`(?P(?:[:a-zA-Z\*0-9=+%#\-_/,\'\"\\\.$&^@!\(\)\{\}\s])*)`' - regex_value_regex = fr"{double_quotes_value_pattern}|{back_quotes_value_pattern}\s*\)\s*(?:nocase)?\s*" - - def search_field_value(self, query): - if query.startswith("re.regex("): - field_search = re.search(self.regex_field_regex, query) - if field_search is None: - raise TokenizerGeneralException(error=f"Field couldn't be found in query part: {query}") - - field = field_search.group("field") - pos = field_search.end() - query = query[pos:] - - value_search = re.search(self.regex_value_regex, query) - if value_search is None: - raise TokenizerGeneralException(error=f"Value couldn't be found in query part: {query}") - - operator = OperatorType.REGEX - operator, value = self.get_operator_and_value(value_search, operator) - value, operator = self.process_value_wildcard_symbols( - value=value, operator=OperatorType.REGEX, wildcard_symbol=self.wildcard_symbol - ) - pos = value_search.end() - query = query[pos:] - - field = self.create_field(field_name=field, operator=operator, value=value) - return field, query - else: - return super().search_field_value(query=query) - - def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> Tuple[str, Any]: - if (d_q_value := get_match_group(match, group_name='d_q_value')) is not None: - return operator, d_q_value - - elif (b_q_value := get_match_group(match, group_name='b_q_value')) is not None: - return operator, b_q_value - - return super().get_operator_and_value(match, operator) diff --git a/siem-converter/app/converter/platforms/crowdstrike/const.py b/siem-converter/app/converter/platforms/crowdstrike/const.py deleted file mode 100644 index 2bf38d1f..00000000 --- a/siem-converter/app/converter/platforms/crowdstrike/const.py +++ /dev/null @@ -1,11 +0,0 @@ -from app.converter.core.models.platform_details import PlatformDetails - -CROWDSTRIKE_QUERY_DETAILS = { - "siem_type": "crowdstrike-spl-query", - "name": "CrowdStrike", - "platform_name": "Query (SPL)", - "group_id": "crowdstrike", - "group_name": "CrowdStrike" -} - -crowdstrike_query_details = PlatformDetails(**CROWDSTRIKE_QUERY_DETAILS) diff --git a/siem-converter/app/converter/platforms/crowdstrike/mapping.py b/siem-converter/app/converter/platforms/crowdstrike/mapping.py deleted file mode 100644 index 9f9c4f53..00000000 --- a/siem-converter/app/converter/platforms/crowdstrike/mapping.py +++ /dev/null @@ -1,41 +0,0 @@ -from typing import List, Optional - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class CrowdStrikeLogSourceSignature(LogSourceSignature): - def __init__(self, event_simple_name: Optional[List[str]], default_source: dict): - self.event_simple_names = set(event_simple_name or []) - self._default_source = default_source or {} - - def is_suitable(self, event_simple_name: List[str]) -> bool: - return set(event_simple_name).issubset(self.event_simple_names) - - def __str__(self) -> str: - return f"event_simpleName={self._default_source['event_simpleName']}" - - -class CrowdstrikeMappings(BasePlatformMappings): - def prepare_log_source_signature(self, mapping: dict) -> CrowdStrikeLogSourceSignature: - log_source = mapping.get("log_source", {}) - default_log_source = mapping["default_log_source"] - return CrowdStrikeLogSourceSignature( - event_simple_name=log_source.get("event_simpleName"), - default_source=default_log_source - ) - - def get_suitable_source_mappings(self, field_names: List[str], event_simpleName: List[str]) -> List[SourceMapping]: - suitable_source_mappings = [] - for source_mapping in self._source_mappings.values(): - if source_mapping.source_id == DEFAULT_MAPPING_NAME: - continue - - source_signature: CrowdStrikeLogSourceSignature = source_mapping.log_source_signature - if source_signature.is_suitable(event_simple_name=event_simpleName): - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - - return suitable_source_mappings or [self._source_mappings[DEFAULT_MAPPING_NAME]] - - -crowdstrike_mappings = CrowdstrikeMappings(platform_dir="crowdstrike") diff --git a/siem-converter/app/converter/platforms/crowdstrike/mappings/crowdstrike_cti.py b/siem-converter/app/converter/platforms/crowdstrike/mappings/crowdstrike_cti.py deleted file mode 100644 index f11c195a..00000000 --- a/siem-converter/app/converter/platforms/crowdstrike/mappings/crowdstrike_cti.py +++ /dev/null @@ -1,11 +0,0 @@ -DEFAULT_CROWDSTRIKE_MAPPING = { - "DestinationIP": "RemoteAddressIP4", - "SourceIP": "LocalAddressIP4", - "HashSha256": "SHA256HashData", - "HashMd5": "MD5HashData", - "Emails": "emails", - "Domain": "DomainName", - "HashSha1": "SHA1HashData", - "Files": "TargetFileName", - "URL": "HttpUrl" -} diff --git a/siem-converter/app/converter/platforms/crowdstrike/parsers/crowdstrike.py b/siem-converter/app/converter/platforms/crowdstrike/parsers/crowdstrike.py deleted file mode 100644 index 0c13b434..00000000 --- a/siem-converter/app/converter/platforms/crowdstrike/parsers/crowdstrike.py +++ /dev/null @@ -1,30 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" -from app.converter.platforms.base.spl.parsers.spl import SplParser -from app.converter.platforms.crowdstrike.const import crowdstrike_query_details -from app.converter.platforms.crowdstrike.mapping import CrowdstrikeMappings, crowdstrike_mappings -from app.converter.core.models.platform_details import PlatformDetails - - -class CrowdStrikeParser(SplParser): - details: PlatformDetails = crowdstrike_query_details - - log_source_pattern = r"___source_type___\s*=\s*(?:\"(?P[%a-zA-Z_*:0-9\-/]+)\"|(?P[%a-zA-Z_*:0-9\-/]+))(?:\s+(?:and|or)\s+|\s+)?" - log_source_key_types = ("event_simpleName",) - - mappings: CrowdstrikeMappings = crowdstrike_mappings diff --git a/siem-converter/app/converter/platforms/crowdstrike/renders/crowdstrike.py b/siem-converter/app/converter/platforms/crowdstrike/renders/crowdstrike.py deleted file mode 100644 index 67a342fe..00000000 --- a/siem-converter/app/converter/platforms/crowdstrike/renders/crowdstrike.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" -from app.converter.platforms.base.spl.renders.spl import SplFieldValue, SplQueryRender -from app.converter.platforms.crowdstrike.const import crowdstrike_query_details -from app.converter.platforms.crowdstrike.mapping import CrowdstrikeMappings, crowdstrike_mappings -from app.converter.core.models.platform_details import PlatformDetails - - -class CrowdStrikeFieldValue(SplFieldValue): - details = crowdstrike_query_details - - -class CrowdStrikeQueryRender(SplQueryRender): - details: PlatformDetails = crowdstrike_query_details - query_pattern = "{prefix} {query} {functions}" - mappings: CrowdstrikeMappings = crowdstrike_mappings - - or_token = "OR" - field_value_map = CrowdStrikeFieldValue(or_token=or_token) - comment_symbol = '`' diff --git a/siem-converter/app/converter/platforms/crowdstrike/renders/crowdstrike_cti.py b/siem-converter/app/converter/platforms/crowdstrike/renders/crowdstrike_cti.py deleted file mode 100644 index dc7f5f8d..00000000 --- a/siem-converter/app/converter/platforms/crowdstrike/renders/crowdstrike_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.crowdstrike.const import crowdstrike_query_details -from app.converter.platforms.crowdstrike.mappings.crowdstrike_cti import DEFAULT_CROWDSTRIKE_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class CrowdStrikeCTI(RenderCTI): - details: PlatformDetails = crowdstrike_query_details - - data_map: str = '{key}="{value}"' - or_operator: str = " OR " - group_or_operator: str = " " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_CROWDSTRIKE_MAPPING diff --git a/siem-converter/app/converter/platforms/elasticsearch/const.py b/siem-converter/app/converter/platforms/elasticsearch/const.py deleted file mode 100644 index a733e8af..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/const.py +++ /dev/null @@ -1,210 +0,0 @@ -from app.converter.core.models.platform_details import PlatformDetails - -PLATFORM_DETAILS = { - "group_id": "elk stack", - "group_name": "Elastic Stack", - "alt_platform_name": "ECS", -} - -ELASTICSEARCH_LUCENE_QUERY_DETAILS = { - "siem_type": "elastic-lucene-query", - "name": "Elasticsearch Query", - "platform_name": "Query (Lucene)", - **PLATFORM_DETAILS -} - -ELASTICSEARCH_RULE_DETAILS = { - "siem_type": "elastic-lucene-rule", - "name": "Elastic Rule", - "platform_name": "Detection Rule (Lucene)", - "first_choice": 0, - **PLATFORM_DETAILS -} - -KIBANA_DETAILS = { - "siem_type": "elastic-kibana-rule", - "name": "Elastic Kibana Saved Search", - "platform_name": "Kibana SavedSearch (JSON)", - "first_choice": 0, - **PLATFORM_DETAILS -} - -ELASTALERT_DETAILS = { - "siem_type": "elastalert-lucene-rule", - "name": "ElastAlert", - "platform_name": "Alert (Lucene)", - "group_name": "ElastAlert", - "group_id": "elastalert" -} - -XPACK_WATCHER_DETAILS = { - "siem_type": "elastic-watcher-rule", - "name": "Elastic Watcher", - "platform_name": "Rule (Watcher)", - "first_choice": 0, - **PLATFORM_DETAILS -} - -elasticsearch_lucene_query_details = PlatformDetails(**ELASTICSEARCH_LUCENE_QUERY_DETAILS) -elasticsearch_rule_details = PlatformDetails(**ELASTICSEARCH_RULE_DETAILS) -elastalert_details = PlatformDetails(**ELASTALERT_DETAILS) -kibana_rule_details = PlatformDetails(**KIBANA_DETAILS) -xpack_watcher_details = PlatformDetails(**XPACK_WATCHER_DETAILS) - -ELASTICSEARCH_DETECTION_RULE = { - "description": "Autogenerated ElasticSearch Detection Rule.", - "author": [], - "enabled": True, - "false_positives": [], - "filters": [], - "from": "now-360s", - "immutable": False, - "index": [], - "interval": "5m", - "rule_id": "", - "language": "lucene", - "output_index": ".siem-signals-default", - "max_signals": 100, - "risk_score": 65, - "name": "", - "query": "", - "meta": { - "from": "1m" - }, - "severity": "high", - "tags": [], - "to": "now", - "type": "query", - "threat": [], - "version": 1, - "references": [], - "license": "" -} - -ELASTICSEARCH_ALERT = """alert: -- debug -description: -filter: -- query_string: - query: -index: winlogbeat-* -name: -priority: -realert: - minutes: 0 -type: any -""" - -KIBANA_RULE = { - "_id": "dcd74b95-3f36-4ed9-9598-0490951643aa-Malicious-PowerView-PowerShell-Commandlets", - "_type": "search", - "_source": { - "title": "Autogenerated Kibana rule", - "description": "Autogenerated Kibana rule.", - "hits": 0, - "columns": [], - "sort": [ - "@timestamp", - "desc" - ], - "version": 1, - "kibanaSavedObjectMeta": { - "searchSourceJSON": "" - } - } -} - - -KIBANA_SEARCH_SOURCE_JSON = { - "index": "winlogbeat-*", - "filter": [], - "highlight": { - "pre_tags": [ - "@kibana-highlighted-field@" - ], - "post_tags": [ - "@/kibana-highlighted-field@" - ], - "fields": { - "*": {} - }, - "require_field_match": False, - "fragment_size": 2147483647 - }, - "query": { - "query_string": { - "query": "", - "analyze_wildcard": True - } - } -} - - -XPACK_WATCHER_RULE = { - "metadata": { - "title": "", - "description": "", - "tags": [], - "query": "" - }, - "trigger": { - "schedule": { - "interval": "30m" - } - }, - "input": { - "search": { - "request": { - "body": { - "size": 0, - "query": { - "bool": { - "must": [ - { - "query_string": { - "query": "", - "analyze_wildcard": True - } - } - ], - "filter": { - "range": { - "@timestamp": { - "gte": "now-30m/m" - } - } - } - } - } - }, - "indices": [] # put indices from mapping - } - } - }, - "condition": { - "compare": { - "ctx.payload.hits.total": { - "not_eq": 0 - } - } - }, - "actions": { - "send_email": { - "throttle_period": "15m", - "email": { - "profile": "standard", - "from": "root@localhost", - "to": "root@localhost", - "subject": "", - "body": "Hits:\n{{#ctx.payload.hits.hits}}{{_source}}\n================================================================================\n{{/ctx.payload.hits.hits}}", - "attachments": { - "data.json": { - "data": { - "format": "json" - } - } - } - } - } - } -} diff --git a/siem-converter/app/converter/platforms/elasticsearch/mapping.py b/siem-converter/app/converter/platforms/elasticsearch/mapping.py deleted file mode 100644 index 4ac3efd6..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/mapping.py +++ /dev/null @@ -1,8 +0,0 @@ -from app.converter.platforms.base.lucene.mapping import LuceneMappings - - -class ElasticSearchMappings(LuceneMappings): - pass - - -elasticsearch_mappings = ElasticSearchMappings(platform_dir="elasticsearch") diff --git a/siem-converter/app/converter/platforms/elasticsearch/mappings/elasticsearch_cti_cti.py b/siem-converter/app/converter/platforms/elasticsearch/mappings/elasticsearch_cti_cti.py deleted file mode 100644 index 921da898..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/mappings/elasticsearch_cti_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_ELASTICSEARCH_MAPPING = { - "DestinationIP": "destination.ip", - "SourceIP": "source.ip", - "HashSha512": "file.hash.sha512", - "HashSha256": "file.hash.sha256", - "HashMd5": "file.hash.md5", - "Emails": "email.from.address", - "Domain": "destination.domain", - "HashSha1": "file.hash.sha1", - "Files": "file.name", - "URL": "url.original" -} diff --git a/siem-converter/app/converter/platforms/elasticsearch/parsers/detection_rule.py b/siem-converter/app/converter/platforms/elasticsearch/parsers/detection_rule.py deleted file mode 100644 index 57fdcb5d..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/parsers/detection_rule.py +++ /dev/null @@ -1,59 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from typing import List, Dict - -from app.converter.platforms.elasticsearch.const import elasticsearch_rule_details -from app.converter.platforms.elasticsearch.parsers.elasticsearch import ElasticSearchParser -from app.converter.core.mixins.rule import JsonRuleMixin -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class ElasticSearchRuleParser(ElasticSearchParser, JsonRuleMixin): - details: PlatformDetails = elasticsearch_rule_details - - def _parse_rule(self, text: str) -> Dict: - rule = self.load_rule(text=text) - query = rule['query'] - description = rule['description'] - name = rule['name'] - return { - 'query': query, - 'title': name, - 'description': description - } - - @staticmethod - def _get_meta_info(source_mapping_ids: List[str], meta_info: dict) -> MetaInfoContainer: - return MetaInfoContainer( - title=meta_info['title'], - description=meta_info['description'], - source_mapping_ids=source_mapping_ids - ) - - def parse(self, text: str) -> SiemContainer: - rule, log_sources = self._parse_rule(text) - tokens, source_mappings = self.get_tokens_and_source_mappings(rule.get("query"), log_sources) - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info( - source_mapping_ids=[source_mapping.source_id for source_mapping in source_mappings], - meta_info=rule - ), - ) diff --git a/siem-converter/app/converter/platforms/elasticsearch/parsers/elasticsearch.py b/siem-converter/app/converter/platforms/elasticsearch/parsers/elasticsearch.py deleted file mode 100644 index 9dfa84f6..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/parsers/elasticsearch.py +++ /dev/null @@ -1,27 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from app.converter.platforms.base.lucene.parsers.lucene import LuceneParser -from app.converter.platforms.elasticsearch.const import elasticsearch_lucene_query_details -from app.converter.platforms.elasticsearch.mapping import ElasticSearchMappings, elasticsearch_mappings -from app.converter.core.models.platform_details import PlatformDetails - - -class ElasticSearchParser(LuceneParser): - details: PlatformDetails = elasticsearch_lucene_query_details - mappings: ElasticSearchMappings = elasticsearch_mappings diff --git a/siem-converter/app/converter/platforms/elasticsearch/renders/detection_rule.py b/siem-converter/app/converter/platforms/elasticsearch/renders/detection_rule.py deleted file mode 100644 index a38c7c44..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/renders/detection_rule.py +++ /dev/null @@ -1,71 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -import copy -import json - -from app.converter.platforms.elasticsearch.const import ELASTICSEARCH_DETECTION_RULE, elasticsearch_rule_details -from app.converter.platforms.elasticsearch.mapping import ElasticSearchMappings, elasticsearch_mappings -from app.converter.platforms.elasticsearch.renders.elasticsearch import ElasticSearchQueryRender, ElasticSearchFieldValue -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer -from app.converter.tools.utils import concatenate_str, get_mitre_attack_str - - -class ElasticSearchRuleFieldValue(ElasticSearchFieldValue): - details: PlatformDetails = elasticsearch_rule_details - - -class ElasticSearchRuleRender(ElasticSearchQueryRender): - details: PlatformDetails = elasticsearch_rule_details - mappings: ElasticSearchMappings = elasticsearch_mappings - - or_token = "OR" - and_token = "AND" - not_token = "NOT" - - field_value_map = ElasticSearchRuleFieldValue(or_token=or_token) - query_pattern = "{prefix} {query} {functions}" - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None): - query = super().finalize_query(prefix=prefix, query=query, functions=functions, meta_info=meta_info) - rule = copy.deepcopy(ELASTICSEARCH_DETECTION_RULE) - description = meta_info.description or rule["description"] - mitre_attack_str = get_mitre_attack_str(meta_info.mitre_attack) - description = concatenate_str(description, mitre_attack_str) - - rule.update({ - "query": query, - "description": description, - "name": meta_info.title, - "rule_id": meta_info.id, - "author": [meta_info.author], - "severity": meta_info.severity, - "references": meta_info.references, - "license": meta_info.license, - "tags": meta_info.mitre_attack, - "false_positives": meta_info.false_positives - }) - rule_str = json.dumps(rule, indent=4, sort_keys=False) - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return rule_str + rendered_not_supported - return rule_str diff --git a/siem-converter/app/converter/platforms/elasticsearch/renders/elast_alert.py b/siem-converter/app/converter/platforms/elasticsearch/renders/elast_alert.py deleted file mode 100644 index 1e02c210..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/renders/elast_alert.py +++ /dev/null @@ -1,61 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.elasticsearch.const import ELASTICSEARCH_ALERT, elastalert_details -from app.converter.platforms.elasticsearch.mapping import ElasticSearchMappings, elasticsearch_mappings -from app.converter.platforms.elasticsearch.renders.elasticsearch import ElasticSearchQueryRender, ElasticSearchFieldValue -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer -from app.converter.tools.utils import get_author_str, concatenate_str, get_mitre_attack_str, get_licence_str - - -SEVERITIES_MAP = {"informational": "5", "low": "4", "medium": "3", "high": "2", "critical": "1"} - - -class ElasticAlertRuleFieldValue(ElasticSearchFieldValue): - details: PlatformDetails = elastalert_details - - -class ElastAlertRuleRender(ElasticSearchQueryRender): - details: PlatformDetails = elastalert_details - mappings: ElasticSearchMappings = elasticsearch_mappings - - or_token = "OR" - and_token = "AND" - not_token = "NOT" - - field_value_map = ElasticAlertRuleFieldValue(or_token=or_token) - query_pattern = "{prefix} {query} {functions}" - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None): - query = super().finalize_query(prefix=prefix, query=query, functions=functions, meta_info=meta_info) - rule = ELASTICSEARCH_ALERT.replace("", query) - description = concatenate_str(meta_info.description, get_author_str(meta_info.author)) - description = concatenate_str(description, get_licence_str(meta_info.license)) - description = concatenate_str(description, get_mitre_attack_str(meta_info.mitre_attack)) - - rule = rule.replace("", description) - rule = rule.replace("", meta_info.title) - rule = rule.replace("", SEVERITIES_MAP[meta_info.severity]) - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return rule + rendered_not_supported - return rule diff --git a/siem-converter/app/converter/platforms/elasticsearch/renders/elasticsearch.py b/siem-converter/app/converter/platforms/elasticsearch/renders/elasticsearch.py deleted file mode 100644 index 0d21cc8c..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/renders/elasticsearch.py +++ /dev/null @@ -1,43 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" -from typing import Union - -from app.converter.platforms.base.lucene.renders.lucene import LuceneQueryRender, LuceneFieldValue -from app.converter.platforms.elasticsearch.const import elasticsearch_lucene_query_details -from app.converter.platforms.elasticsearch.mapping import ElasticSearchMappings, elasticsearch_mappings -from app.converter.core.models.platform_details import PlatformDetails - - -class ElasticSearchFieldValue(LuceneFieldValue): - details: PlatformDetails = elasticsearch_lucene_query_details - - def apply_value(self, value: Union[str, int]): - if isinstance(value, int): - return value - if " " in value: - return f'"{value}"'.replace(" ", r"\ ") - return value - - -class ElasticSearchQueryRender(LuceneQueryRender): - details: PlatformDetails = elasticsearch_lucene_query_details - mappings: ElasticSearchMappings = elasticsearch_mappings - - or_token = "OR" - field_value_map = ElasticSearchFieldValue(or_token=or_token) diff --git a/siem-converter/app/converter/platforms/elasticsearch/renders/elasticsearch_cti.py b/siem-converter/app/converter/platforms/elasticsearch/renders/elasticsearch_cti.py deleted file mode 100644 index ddee131e..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/renders/elasticsearch_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.elasticsearch.const import elasticsearch_lucene_query_details -from app.converter.platforms.elasticsearch.mappings.elasticsearch_cti_cti import DEFAULT_ELASTICSEARCH_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class ElasticsearchCTI(RenderCTI): - details: PlatformDetails = elasticsearch_lucene_query_details - - data_map: str = '"{value}"' - or_operator: str = " OR " - group_or_operator: str = " OR " - or_group: str = "{processing_key}:({or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_ELASTICSEARCH_MAPPING diff --git a/siem-converter/app/converter/platforms/elasticsearch/renders/kibana.py b/siem-converter/app/converter/platforms/elasticsearch/renders/kibana.py deleted file mode 100644 index 2150642b..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/renders/kibana.py +++ /dev/null @@ -1,62 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -import copy -import json - -from app.converter.platforms.elasticsearch.const import KIBANA_SEARCH_SOURCE_JSON, KIBANA_RULE, kibana_rule_details -from app.converter.platforms.elasticsearch.mapping import ElasticSearchMappings, elasticsearch_mappings -from app.converter.platforms.elasticsearch.renders.elasticsearch import ElasticSearchQueryRender, ElasticSearchFieldValue -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer -from app.converter.tools.utils import concatenate_str, get_author_str, get_licence_str, get_mitre_attack_str, \ - get_rule_id_str, get_references_str - - -class KibanaFieldValue(ElasticSearchFieldValue): - details: PlatformDetails = kibana_rule_details - - -class KibanaRuleRender(ElasticSearchQueryRender): - details: PlatformDetails = kibana_rule_details - mappings: ElasticSearchMappings = elasticsearch_mappings - or_token = "OR" - field_value_map = KibanaFieldValue(or_token=or_token) - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None): - query = super().finalize_query(prefix=prefix, query=query, functions=functions, meta_info=meta_info) - search_source = copy.deepcopy(KIBANA_SEARCH_SOURCE_JSON) - search_source["query"]["query_string"]["query"] = query - dumped_rule = json.dumps(search_source, sort_keys=False) - rule = copy.deepcopy(KIBANA_RULE) - rule["_source"]["kibanaSavedObjectMeta"]["searchSourceJSON"] = dumped_rule - rule["_source"]["title"] = meta_info.title - description = meta_info.description or rule["_source"]["description"] - description = concatenate_str(description, get_author_str(meta_info.author)) - description = concatenate_str(description, get_rule_id_str(meta_info.id)) - description = concatenate_str(description, get_licence_str(meta_info.license)) - description = concatenate_str(description, get_references_str(meta_info.references)) - rule["_source"]["description"] = concatenate_str(description, get_mitre_attack_str(meta_info.mitre_attack)) - rule_str = json.dumps(rule, indent=4, sort_keys=False) - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return rule_str + rendered_not_supported - return rule_str diff --git a/siem-converter/app/converter/platforms/elasticsearch/renders/xpack_watcher.py b/siem-converter/app/converter/platforms/elasticsearch/renders/xpack_watcher.py deleted file mode 100644 index 0272f471..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/renders/xpack_watcher.py +++ /dev/null @@ -1,63 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -import copy -import json - -from app.converter.platforms.elasticsearch.mapping import ElasticSearchMappings, elasticsearch_mappings -from app.converter.platforms.elasticsearch.renders.elasticsearch import ElasticSearchQueryRender, ElasticSearchFieldValue -from app.converter.platforms.elasticsearch.const import XPACK_WATCHER_RULE, xpack_watcher_details -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer -from app.converter.tools.utils import concatenate_str, get_author_str, get_licence_str, get_mitre_attack_str - - -class XpackWatcherRuleFieldValue(ElasticSearchFieldValue): - details: PlatformDetails = xpack_watcher_details - - -class XPackWatcherRuleRender(ElasticSearchQueryRender): - details: PlatformDetails = xpack_watcher_details - mappings: ElasticSearchMappings = elasticsearch_mappings - or_token = "OR" - field_value_map = XpackWatcherRuleFieldValue(or_token=or_token) - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None): - query = super().finalize_query(prefix=prefix, query=query, functions=functions, meta_info=meta_info) - rule = copy.deepcopy(XPACK_WATCHER_RULE) - description = concatenate_str(meta_info.description, get_author_str(meta_info.author)) - description = concatenate_str(description, get_licence_str(meta_info.license)) - description = concatenate_str(description, get_mitre_attack_str(meta_info.mitre_attack)) - rule["metadata"].update({ - "query": query, - "title": meta_info.title, - "description": description, - "tags": meta_info.mitre_attack - }) - rule["input"]["search"]["request"]["body"]["query"]["bool"]["must"][0]["query_string"]["query"] = query - indices = source_mapping and [str(source_mapping.log_source_signature)] or [] - rule["input"]["search"]["request"]["indices"] = indices - rule["actions"]["send_email"]["email"]["subject"] = meta_info.title - rule_str = json.dumps(rule, indent=4, sort_keys=False) - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return rule_str + rendered_not_supported - return rule_str diff --git a/siem-converter/app/converter/platforms/elasticsearch/tokenizer.py b/siem-converter/app/converter/platforms/elasticsearch/tokenizer.py deleted file mode 100644 index ca070eb4..00000000 --- a/siem-converter/app/converter/platforms/elasticsearch/tokenizer.py +++ /dev/null @@ -1,22 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" -from app.converter.platforms.base.lucene.tokenizer import LuceneTokenizer - - -class ElasticSearchTokenizer(LuceneTokenizer): - pass diff --git a/siem-converter/app/converter/platforms/fireeye_helix/const.py b/siem-converter/app/converter/platforms/fireeye_helix/const.py deleted file mode 100644 index 2876d208..00000000 --- a/siem-converter/app/converter/platforms/fireeye_helix/const.py +++ /dev/null @@ -1,8 +0,0 @@ - -FIREEYE_HELIX_QUERY_DETAILS = { - "siem_type": "fireeye_helix", - "name": "FireEye Helix", - "group_name": "FireEye", - "group_id": "fireeye", - "platform_name": "Query", -} diff --git a/siem-converter/app/converter/platforms/fireeye_helix/mappings/fireeye_helix.py b/siem-converter/app/converter/platforms/fireeye_helix/mappings/fireeye_helix.py deleted file mode 100644 index 524daf9c..00000000 --- a/siem-converter/app/converter/platforms/fireeye_helix/mappings/fireeye_helix.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_FIREEYE_HELIX_MAPPING = { - "SourceIP": "~srcipv4", - "DestinationIP": "~dstipv4", - "Domain": "domain", - "URL": "url", - "HashMd5": "~hash", - "HashSha1": "~hash", - "HashSha256": "~hash", - "HashSha512": "~hash", - "Emails": "emails", - "Files": "filepath" -} diff --git a/siem-converter/app/converter/platforms/fireeye_helix/renders/fireeye_helix_cti.py b/siem-converter/app/converter/platforms/fireeye_helix/renders/fireeye_helix_cti.py deleted file mode 100644 index 79c1cfb1..00000000 --- a/siem-converter/app/converter/platforms/fireeye_helix/renders/fireeye_helix_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.fireeye_helix.const import FIREEYE_HELIX_QUERY_DETAILS -from app.converter.platforms.fireeye_helix.mappings.fireeye_helix import DEFAULT_FIREEYE_HELIX_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class FireeyeHelixCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**FIREEYE_HELIX_QUERY_DETAILS) - - data_map: str = "`{value}`" - or_operator: str = "," - group_or_operator: str = " " - or_group: str = "{processing_key}:[{or_group}]" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_FIREEYE_HELIX_MAPPING diff --git a/siem-converter/app/converter/platforms/graylog/const.py b/siem-converter/app/converter/platforms/graylog/const.py deleted file mode 100644 index 95066041..00000000 --- a/siem-converter/app/converter/platforms/graylog/const.py +++ /dev/null @@ -1,8 +0,0 @@ - -GRAYLOG_QUERY_DETAILS = { - "siem_type": "graylog", - "name": "Graylog", - "group_name": "Graylog", - "platform_name": "Query", - "group_id": "graylog" -} diff --git a/siem-converter/app/converter/platforms/graylog/mappings/graylog_cti.py b/siem-converter/app/converter/platforms/graylog/mappings/graylog_cti.py deleted file mode 100644 index eb66315b..00000000 --- a/siem-converter/app/converter/platforms/graylog/mappings/graylog_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_GRAYLOG_MAPPING = { - "SourceIP": "source.ip", - "DestinationIP": "destination.ip", - "Domain": "destination.domain", - "URL": "url.original", - "HashMd5": "file.hash.md5", - "HashSha1": "file.hash.sha1", - "HashSha256": "file.hash.sha256", - "HashSha512": "file.hash.sha512", - "Emails": "emails", - "Files": "filePath" -} diff --git a/siem-converter/app/converter/platforms/graylog/renders/graylog_cti.py b/siem-converter/app/converter/platforms/graylog/renders/graylog_cti.py deleted file mode 100644 index 97100e9f..00000000 --- a/siem-converter/app/converter/platforms/graylog/renders/graylog_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.graylog.const import GRAYLOG_QUERY_DETAILS -from app.converter.platforms.graylog.mappings.graylog_cti import DEFAULT_GRAYLOG_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class GraylogCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**GRAYLOG_QUERY_DETAILS) - - data_map: str = '"{value}"' - or_operator: str = " " - group_or_operator: str = " OR " - or_group: str = "{processing_key}:({or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_GRAYLOG_MAPPING diff --git a/siem-converter/app/converter/platforms/logpoint/const.py b/siem-converter/app/converter/platforms/logpoint/const.py deleted file mode 100644 index 1cf02b4c..00000000 --- a/siem-converter/app/converter/platforms/logpoint/const.py +++ /dev/null @@ -1,8 +0,0 @@ - -LOGPOINT_QUERY_DETAILS = { - "siem_type": "logpoint", - "name": "Logpoint", - "group_name": "Logpoint", - "platform_name": "Query", - "group_id": "logpoint" -} diff --git a/siem-converter/app/converter/platforms/logpoint/mappings/logpoint_cti.py b/siem-converter/app/converter/platforms/logpoint/mappings/logpoint_cti.py deleted file mode 100644 index 67ba46bb..00000000 --- a/siem-converter/app/converter/platforms/logpoint/mappings/logpoint_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_LOGPOINT_MAPPING = { - "DestinationIP": "dst_ip", - "SourceIP": "src_ip", - "HashSha512": "hash", - "HashSha256": "hash", - "HashMd5": "hash", - "Emails": "emails", - "Domain": "host", - "HashSha1": "hash", - "Files": "files", - "URL": "url" -} diff --git a/siem-converter/app/converter/platforms/logpoint/renders/logpoint_cti.py b/siem-converter/app/converter/platforms/logpoint/renders/logpoint_cti.py deleted file mode 100644 index 389844fe..00000000 --- a/siem-converter/app/converter/platforms/logpoint/renders/logpoint_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.logpoint.const import LOGPOINT_QUERY_DETAILS -from app.converter.platforms.logpoint.mappings.logpoint_cti import DEFAULT_LOGPOINT_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class LogpointCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**LOGPOINT_QUERY_DETAILS) - - data_map: str = '"{value}"' - or_operator: str = ", " - group_or_operator: str = " " - or_group: str = "{processing_key} IN [{or_group}]" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_LOGPOINT_MAPPING diff --git a/siem-converter/app/converter/platforms/logscale/const.py b/siem-converter/app/converter/platforms/logscale/const.py deleted file mode 100644 index aa064a2f..00000000 --- a/siem-converter/app/converter/platforms/logscale/const.py +++ /dev/null @@ -1,37 +0,0 @@ -from app.converter.core.models.platform_details import PlatformDetails - -DEFAULT_LOGSCALE_ALERT = { - "name": "", - "query": { - "queryString": "", - "isLive": True, - "start": "1h" - }, - "description": "", - "throttleTimeMillis": 60000, - "silenced": False -} - -PLATFORM_DETAILS = { - "group_id": "logscale-pack", - "group_name": "Falcon LogScale" -} - -LOGSCALE_QUERY_DETAILS = { - "siem_type": "logscale-lql-query", - "name": "Falcon LogScale Query", - "platform_name": "Query", - **PLATFORM_DETAILS -} - -LOGSCALE_ALERT_DETAILS = { - "siem_type": "logscale-lql-rule", - "name": "Falcon LogScale Alert", - "platform_name": "Alert", - "first_choice": 0, - **PLATFORM_DETAILS -} - - -logscale_query_details = PlatformDetails(**LOGSCALE_QUERY_DETAILS) -logscale_alert_details = PlatformDetails(**LOGSCALE_ALERT_DETAILS) diff --git a/siem-converter/app/converter/platforms/logscale/mapping.py b/siem-converter/app/converter/platforms/logscale/mapping.py deleted file mode 100644 index 2561614c..00000000 --- a/siem-converter/app/converter/platforms/logscale/mapping.py +++ /dev/null @@ -1,30 +0,0 @@ -from typing import List, Optional - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class LogScaleLogSourceSignature(LogSourceSignature): - def __str__(self) -> str: - return "" - - -class LogScaleMappings(BasePlatformMappings): - def prepare_log_source_signature(self, mapping: dict) -> LogScaleLogSourceSignature: - return - - def get_suitable_source_mappings(self, field_names: List[str]) -> List[SourceMapping]: - suitable_source_mappings = [] - for source_mapping in self._source_mappings.values(): - if source_mapping.source_id == DEFAULT_MAPPING_NAME: - continue - - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - - if not suitable_source_mappings: - suitable_source_mappings = [self._source_mappings[DEFAULT_MAPPING_NAME]] - - return suitable_source_mappings - - -logscale_mappings = LogScaleMappings(platform_dir="logscale") diff --git a/siem-converter/app/converter/platforms/logscale/mappings/logscale_cti.py b/siem-converter/app/converter/platforms/logscale/mappings/logscale_cti.py deleted file mode 100644 index 2844716c..00000000 --- a/siem-converter/app/converter/platforms/logscale/mappings/logscale_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_LOGSCALE_MAPPING = { - "DestinationIP": "dst_ip", - "SourceIP": "src_ip", - "HashSha512": "file.hash.sha512", - "HashSha256": "file.hash.sha256", - "HashMd5": "file.hash.md5", - "Emails": "email", - "Domain": "host", - "HashSha1": "file.hash.sha1", - "Files": "winlog.event_data.TargetFilename", - "URL": "url" -} diff --git a/siem-converter/app/converter/platforms/logscale/parsers/logscale.py b/siem-converter/app/converter/platforms/logscale/parsers/logscale.py deleted file mode 100644 index 13fa877f..00000000 --- a/siem-converter/app/converter/platforms/logscale/parsers/logscale.py +++ /dev/null @@ -1,52 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from typing import Tuple, List - -from app.converter.platforms.logscale.const import logscale_query_details -from app.converter.platforms.logscale.mapping import logscale_mappings, LogScaleMappings -from app.converter.platforms.logscale.siem_functions import LogScaleQueryFunctions -from app.converter.core.models.functions.types import ParsedFunctions -from app.converter.platforms.logscale.tokenizer import LogScaleTokenizer -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.parser import Parser -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class LogScaleParser(Parser): - details: PlatformDetails = logscale_query_details - siem_functions = LogScaleQueryFunctions() - tokenizer = LogScaleTokenizer() - mappings: LogScaleMappings = logscale_mappings - - @staticmethod - def _get_meta_info(source_mapping_ids: List[str], metainfo: dict) -> MetaInfoContainer: - return MetaInfoContainer(source_mapping_ids=source_mapping_ids) - - def _parse_query(self, query: str) -> Tuple[str, ParsedFunctions]: - functions, splited_query = self.siem_functions.parse(query) - return " and ".join(splited_query), functions - - def parse(self, text: str) -> SiemContainer: - query, functions = self._parse_query(query=text) - tokens, source_mappings = self.get_tokens_and_source_mappings(text, {}) - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info([source_mapping.source_id for source_mapping in source_mappings], {}), - functions=functions - ) diff --git a/siem-converter/app/converter/platforms/logscale/parsers/logscale_alert.py b/siem-converter/app/converter/platforms/logscale/parsers/logscale_alert.py deleted file mode 100644 index 7a6484c7..00000000 --- a/siem-converter/app/converter/platforms/logscale/parsers/logscale_alert.py +++ /dev/null @@ -1,61 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from typing import List - -from app.converter.platforms.logscale.const import logscale_alert_details -from app.converter.platforms.logscale.parsers.logscale import LogScaleParser -from app.converter.core.mixins.rule import JsonRuleMixin -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class LogScaleAlertParser(LogScaleParser, JsonRuleMixin): - details: PlatformDetails = logscale_alert_details - - def _parse_rule(self, text: str): - rule = self.load_rule(text=text) - query = rule['query']['queryString'] - description = rule['description'] - name = rule['name'] - return { - 'query': query, - 'name': name, - 'description': description - } - - @staticmethod - def _get_meta_info(source_mapping_ids: List[str], meta_info: dict) -> MetaInfoContainer: - return MetaInfoContainer( - title=meta_info['name'], - description=meta_info['description'], - source_mapping_ids=source_mapping_ids - ) - - def parse(self, text: str) -> SiemContainer: - parsed_rule = self._parse_rule(text) - query, functions = self._parse_query(query=parsed_rule.pop("query")) - tokens, source_mappings = self.get_tokens_and_source_mappings(text, {}) - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info( - meta_info=parsed_rule, - source_mapping_ids=[source_mapping.source_id for source_mapping in source_mappings] - ), - functions=functions - ) diff --git a/siem-converter/app/converter/platforms/logscale/renders/logscale.py b/siem-converter/app/converter/platforms/logscale/renders/logscale.py deleted file mode 100644 index da9dcf4b..00000000 --- a/siem-converter/app/converter/platforms/logscale/renders/logscale.py +++ /dev/null @@ -1,98 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" -from typing import Union - -from app.converter.platforms.logscale.const import logscale_query_details -from app.converter.platforms.logscale.mapping import LogScaleMappings, logscale_mappings -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer -from app.converter.core.render import BaseQueryRender, BaseQueryFieldValue - - -class LogScaleFieldValue(BaseQueryFieldValue): - details: PlatformDetails = logscale_query_details - - def apply_value(self, value: Union[str, int]): - if isinstance(value, str) and '"' in value: - value = value.translate(str.maketrans({'"': r'\"'})) - return value - - def equal_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.equal_modifier(field=field, value=v) for v in value)})" - return f'{field}="{self.apply_value(value)}"' - - def contains_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" - return f'{field}="*{self.apply_value(value)}*"' - - def endswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" - return f'{field}="*{self.apply_value(value)}"' - - def startswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" - return f'{field}="{self.apply_value(value)}*"' - - def regex_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" - return f'{field}="/{self.apply_value(value)}/"' - - def keywords(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" - return f'"{self.apply_value(value)}"' - - -class LogScaleQueryRender(BaseQueryRender): - details: PlatformDetails = logscale_query_details - mappings: LogScaleMappings = logscale_mappings - - or_token = "or" - and_token = "" - not_token = "not" - - field_value_map = LogScaleFieldValue(or_token=or_token) - query_pattern = "{prefix} and {query} {functions}" - - def wrap_with_comment(self, value: str) -> str: - return f"/* {value} */" - - def generate_prefix(self, logsource: dict): - return "" - - def generate_functions(self, functions: list): - if not functions: - return "" - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None) -> str: - if prefix: - query = self.query_pattern.format(prefix=prefix, query=query, functions=functions) - else: - query = f'{query} {functions}' - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return query + rendered_not_supported - return query diff --git a/siem-converter/app/converter/platforms/logscale/renders/logscale_alert.py b/siem-converter/app/converter/platforms/logscale/renders/logscale_alert.py deleted file mode 100644 index 7e87c267..00000000 --- a/siem-converter/app/converter/platforms/logscale/renders/logscale_alert.py +++ /dev/null @@ -1,56 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -import copy -import json - -from app.converter.platforms.logscale.renders.logscale import LogScaleQueryRender, LogScaleFieldValue -from app.converter.platforms.logscale.const import DEFAULT_LOGSCALE_ALERT, logscale_alert_details -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer - - -_AUTOGENERATED_TITLE = "Autogenerated Falcon LogScale Alert" - -class LogScaleAlertFieldValue(LogScaleFieldValue): - details: PlatformDetails = logscale_alert_details - - -class LogScaleAlertRender(LogScaleQueryRender): - details: PlatformDetails = logscale_alert_details - or_token = "or" - field_value_map = LogScaleAlertFieldValue(or_token=or_token) - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None): - query = super().finalize_query(prefix=prefix, query=query, functions=functions, meta_info=meta_info) - rule = copy.deepcopy(DEFAULT_LOGSCALE_ALERT) - rule['query']['queryString'] = query - rule['name'] = meta_info.title or _AUTOGENERATED_TITLE - description = f"{meta_info.description}. Author: {meta_info.author}. License: {meta_info.license}." - if meta_info.mitre_attack: - description += f" MITRE ATT&CK: {', '.join(meta_info.mitre_attack)}" - rule['description'] = description - - json_query = json.dumps(rule, indent=4, sort_keys=False) - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return json_query + rendered_not_supported - return json_query diff --git a/siem-converter/app/converter/platforms/logscale/renders/logscale_cti.py b/siem-converter/app/converter/platforms/logscale/renders/logscale_cti.py deleted file mode 100644 index 50c300aa..00000000 --- a/siem-converter/app/converter/platforms/logscale/renders/logscale_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.logscale.const import logscale_query_details -from app.converter.platforms.logscale.mappings.logscale_cti import DEFAULT_LOGSCALE_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class LogScaleCTI(RenderCTI): - details: PlatformDetails = logscale_query_details - - data_map: str = '{key}="{value}"' - or_operator: str = " or " - group_or_operator: str = " or " - or_group: str = "{or_group}" - result_join: str = "" - final_result_for_many: str = '@stream="http" {result}\n' - final_result_for_one: str = '@stream="http" {result}\n' - default_mapping = DEFAULT_LOGSCALE_MAPPING diff --git a/siem-converter/app/converter/platforms/logscale/siem_functions/__init__.py b/siem-converter/app/converter/platforms/logscale/siem_functions/__init__.py deleted file mode 100644 index 260d5961..00000000 --- a/siem-converter/app/converter/platforms/logscale/siem_functions/__init__.py +++ /dev/null @@ -1,29 +0,0 @@ -from typing import List - -from app.converter.core.functions import Functions, ParsedFunctions, NotSupportedFunction -import re - - -class LogScaleQueryFunctions(Functions): - siem_type = 'humio' - functions_classes = [] - - def prepare_not_supported(self, not_supported: List[NotSupportedFunction]) -> list: - for n in range(len(not_supported)): - not_supported[n] = f'| {not_supported[n].query}' - return not_supported - - def parse(self, query: str) -> ParsedFunctions: - result = [] - not_supported = [] - functions = query.split(self.function_delimiter) - query_result = [] - for i in range(len(functions)): - if func_match := re.search('(\w+)\(([^)]+)\)', functions[i]): - func_name = func_match.group().split('(')[0] - for func in functions[i:]: - not_supported.append(NotSupportedFunction(name=func_name, query=func)) - return ParsedFunctions(not_supported=self.prepare_not_supported(not_supported), functions=result), query_result - else: - query_result.append(functions[i]) - return ParsedFunctions(not_supported=self.prepare_not_supported(not_supported), functions=result), query_result \ No newline at end of file diff --git a/siem-converter/app/converter/platforms/logscale/tokenizer.py b/siem-converter/app/converter/platforms/logscale/tokenizer.py deleted file mode 100644 index 3c02dcd6..00000000 --- a/siem-converter/app/converter/platforms/logscale/tokenizer.py +++ /dev/null @@ -1,81 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Tuple, Any - -from app.converter.core.models.field import Keyword, Field -from app.converter.core.models.identifier import Identifier -from app.converter.core.operator_types.tokens import GroupType, LogicalOperatorType, OperatorType -from app.converter.core.tokenizer import QueryTokenizer -from app.converter.tools.utils import get_match_group - - -class LogScaleTokenizer(QueryTokenizer): - match_operator_pattern = r"""(?:___field___\s?(?P=|!=))\s?""" - num_value_pattern = r"(?P\d+(?:\.\d+)*)\s*" - double_quotes_value_pattern = r'"(?P(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{\}\s]|\\\"|\\)*)"\s*' - re_value_pattern = r"/(?P[:a-zA-Z\*0-9=+%#\\\-_\,\"\'\.$&^@!\(\)\{\}\s?]+)/i?\s*" - _value_pattern = fr"""{num_value_pattern}|{re_value_pattern}|{double_quotes_value_pattern}""" - keyword_pattern = double_quotes_value_pattern - - wildcard_symbol = "*" - - def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> Tuple[str, Any]: - if (num_value := get_match_group(match, group_name='num_value')) is not None: - return operator, num_value - - elif (d_q_value := get_match_group(match, group_name='d_q_value')) is not None: - return operator, d_q_value - - elif (re_value := get_match_group(match, group_name='re_value')) is not None: - return OperatorType.REGEX, re_value - - return super().get_operator_and_value(match, operator) - - def __get_identifier(self, query: str) -> (list, str): - query = query.strip("\n").strip(" ").strip("\n") - if query.startswith(GroupType.L_PAREN): - return Identifier(token_type=GroupType.L_PAREN), query[1:] - elif query.startswith(GroupType.R_PAREN): - return Identifier(token_type=GroupType.R_PAREN), query[1:] - elif query.startswith('!'): - return Identifier(token_type=LogicalOperatorType.NOT), query[1:] - elif operator_search := re.match(self.operator_pattern, query): - operator = operator_search.group("operator") - pos = operator_search.end() - return Identifier(token_type=operator.lower()), query[pos:] - elif self.keyword_pattern and re.match(self.keyword_pattern, query): - return self.search_keyword(query) - else: - return self.search_field_value(query) - - def tokenize(self, query: str) -> list: - tokenized = [] - while query: - identifier, query = self.__get_identifier(query=query) - if tokenized: - if isinstance(identifier, Identifier) and identifier.token_type in (GroupType.L_PAREN, LogicalOperatorType.NOT): - if isinstance(tokenized[-1], (Field, Keyword)) or tokenized[-1].token_type == GroupType.R_PAREN: - tokenized.append(Identifier(token_type=LogicalOperatorType.AND)) - elif isinstance(identifier, (Field, Keyword)): - if isinstance(tokenized[-1], (Field, Keyword)) or tokenized[-1].token_type == GroupType.R_PAREN: - tokenized.append(Identifier(token_type=LogicalOperatorType.AND)) - tokenized.append(identifier) - self._validate_parentheses(tokenized) - return tokenized diff --git a/siem-converter/app/converter/platforms/microsoft/const.py b/siem-converter/app/converter/platforms/microsoft/const.py deleted file mode 100644 index 9e4f226b..00000000 --- a/siem-converter/app/converter/platforms/microsoft/const.py +++ /dev/null @@ -1,50 +0,0 @@ -from app.converter.core.models.platform_details import PlatformDetails - -DEFAULT_MICROSOFT_SENTINEL_RULE = { - "displayName": "Autogenerated Microsoft Sentinel Rule", - "description": "Autogenerated Microsoft Sentinel Rule", - "severity": "medium", - "enabled": True, - "query": "", - "queryFrequency": "PT30M", - "queryPeriod": "PT30M", - "triggerOperator": "GreaterThan", - "triggerThreshold": 0, - "suppressionDuration": "PT2H30M", - "suppressionEnabled": True, - "tactics": [], - "techniques": [] -} - -PLATFORM_DETAILS = { - "group_id": "sentinel", - "group_name": "Microsoft Sentinel" -} - -MICROSOFT_SENTINEL_QUERY_DETAILS = { - "siem_type": "sentinel-kql-query", - "name": "Microsoft Sentinel Query", - "platform_name": "Query (Kusto)", - **PLATFORM_DETAILS -} - -MICROSOFT_SENTINEL_RULE_DETAILS = { - "siem_type": "sentinel-kql-rule", - "name": "Microsoft Sentinel Rule", - "platform_name": "Rule (Kusto)", - "first_choice": 0, - **PLATFORM_DETAILS -} - -MICROSOFT_DEFENDER_DETAILS = { - "siem_type": "mde-kql-query", - "group_name": "Microsoft Defender for Endpoint", - "name": "Microsoft Defender for Endpoint", - "platform_name": "Query (Kusto)", - "group_id": "microsoft-defender" -} - - -microsoft_defender_details = PlatformDetails(**MICROSOFT_DEFENDER_DETAILS) -microsoft_sentinel_query_details = PlatformDetails(**MICROSOFT_SENTINEL_QUERY_DETAILS) -microsoft_sentinel_rule_details = PlatformDetails(**MICROSOFT_SENTINEL_RULE_DETAILS) diff --git a/siem-converter/app/converter/platforms/microsoft/mapping.py b/siem-converter/app/converter/platforms/microsoft/mapping.py deleted file mode 100644 index d3b355ea..00000000 --- a/siem-converter/app/converter/platforms/microsoft/mapping.py +++ /dev/null @@ -1,55 +0,0 @@ -from typing import List, Optional - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class MicrosoftSentinelLogSourceSignature(LogSourceSignature): - def __init__(self, tables: Optional[List[str]], default_source: dict): - self.tables = set(tables or []) - self._default_source = default_source or {} - - def is_suitable(self, table: List[str]) -> bool: - return set(table).issubset(self.tables) - - def __str__(self) -> str: - return self._default_source.get("table", "") - - -class MicrosoftSentinelMappings(BasePlatformMappings): - def prepare_log_source_signature(self, mapping: dict) -> MicrosoftSentinelLogSourceSignature: - tables = mapping.get("log_source", {}).get("table") - default_log_source = mapping["default_log_source"] - return MicrosoftSentinelLogSourceSignature(tables=tables, default_source=default_log_source) - - def get_suitable_source_mappings(self, field_names: List[str], table: List[str]) -> List[SourceMapping]: - suitable_source_mappings = [] - for source_mapping in self._source_mappings.values(): - if source_mapping.source_id == DEFAULT_MAPPING_NAME: - continue - - log_source_signature: MicrosoftSentinelLogSourceSignature = source_mapping.log_source_signature - if log_source_signature.is_suitable(table=table): - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - - if not suitable_source_mappings: - suitable_source_mappings = [self._source_mappings[DEFAULT_MAPPING_NAME]] - - return suitable_source_mappings - - -microsoft_sentinel_mappings = MicrosoftSentinelMappings(platform_dir="microsoft_sentinel") - - -class MicrosoftDefenderLogSourceSignature(MicrosoftSentinelLogSourceSignature): - pass - - -class MicrosoftDefenderMappings(MicrosoftSentinelMappings): - def prepare_log_source_signature(self, mapping: dict) -> MicrosoftDefenderLogSourceSignature: - tables = mapping.get("log_source", {}).get("table") - default_log_source = mapping["default_log_source"] - return MicrosoftDefenderLogSourceSignature(tables=tables, default_source=default_log_source) - - -microsoft_defender_mappings = MicrosoftDefenderMappings(platform_dir="microsoft_defender") diff --git a/siem-converter/app/converter/platforms/microsoft/mappings/mdatp_cti.py b/siem-converter/app/converter/platforms/microsoft/mappings/mdatp_cti.py deleted file mode 100644 index 5f8efb72..00000000 --- a/siem-converter/app/converter/platforms/microsoft/mappings/mdatp_cti.py +++ /dev/null @@ -1,11 +0,0 @@ -DEFAULT_MICROSOFT_DEFENDER_MAPPING = { - "DestinationIP": "RemoteIP", - "SourceIP": "LocalIP", - "HashSha256": "InitiatingProcessSHA256", - "HashMd5": "InitiatingProcessMD5", - "Emails": "SenderFromAddress", - "Domain": "RemoteUrl", - "HashSha1": "InitiatingProcessSHA1", - "Files": "FileName", - "URL": "RemoteUrl" -} diff --git a/siem-converter/app/converter/platforms/microsoft/mappings/microsoft_sentinel_cti.py b/siem-converter/app/converter/platforms/microsoft/mappings/microsoft_sentinel_cti.py deleted file mode 100644 index f2cf3a9d..00000000 --- a/siem-converter/app/converter/platforms/microsoft/mappings/microsoft_sentinel_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_MICROSOFT_SENTINEL_MAPPING = { - "DestinationIP": "DestinationIp", - "SourceIP": "SourceIp", - "HashSha512": "FileHashSha512", - "HashSha256": "FileHashSha256", - "HashMd5": "FileHashMd5", - "Emails": "SenderFromAddress", - "Domain": "DestinationHostname", - "HashSha1": "FileHashSha1", - "Files": "TargetFileName", - "URL": "URL" -} diff --git a/siem-converter/app/converter/platforms/microsoft/parsers/microsoft_defender.py b/siem-converter/app/converter/platforms/microsoft/parsers/microsoft_defender.py deleted file mode 100644 index bdce666d..00000000 --- a/siem-converter/app/converter/platforms/microsoft/parsers/microsoft_defender.py +++ /dev/null @@ -1,27 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from app.converter.platforms.microsoft.const import microsoft_defender_details -from app.converter.platforms.microsoft.parsers.microsoft_sentinel import MicrosoftParser -from app.converter.platforms.microsoft.mapping import microsoft_defender_mappings, MicrosoftDefenderMappings -from app.converter.core.models.platform_details import PlatformDetails - - -class MicrosoftDefenderQueryParser(MicrosoftParser): - mappings: MicrosoftDefenderMappings = microsoft_defender_mappings - details: PlatformDetails = microsoft_defender_details diff --git a/siem-converter/app/converter/platforms/microsoft/parsers/microsoft_sentinel.py b/siem-converter/app/converter/platforms/microsoft/parsers/microsoft_sentinel.py deleted file mode 100644 index 32e210d4..00000000 --- a/siem-converter/app/converter/platforms/microsoft/parsers/microsoft_sentinel.py +++ /dev/null @@ -1,55 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from typing import Tuple, List, Dict - -from app.converter.platforms.microsoft.const import microsoft_sentinel_query_details -from app.converter.platforms.microsoft.siem_functions.base import MicroSoftQueryFunctions -from app.converter.platforms.microsoft.mapping import MicrosoftSentinelMappings, microsoft_sentinel_mappings -from app.converter.platforms.microsoft.tokenizer import MicrosoftSentinelTokenizer -from app.converter.core.models.functions.types import ParsedFunctions -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.parser import Parser -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class MicrosoftParser(Parser): - siem_functions = MicroSoftQueryFunctions() - mappings: MicrosoftSentinelMappings = microsoft_sentinel_mappings - tokenizer = MicrosoftSentinelTokenizer() - details: PlatformDetails = microsoft_sentinel_query_details - - @staticmethod - def _get_meta_info(source_mapping_ids: List[str], meta_info: dict) -> MetaInfoContainer: - return MetaInfoContainer(source_mapping_ids=source_mapping_ids) - - def _parse_query(self, query: str) -> Tuple[str, Dict[str, List[str]], ParsedFunctions]: - functions, split_query = self.siem_functions.parse(query) - table, query = split_query[0], " and ".join(split_query[1:]) - log_sources = dict(table=[split_query[0]]) - return query, log_sources, functions - - def parse(self, text: str) -> SiemContainer: - query, log_sources, functions = self._parse_query(query=text) - tokens, source_mappings = self.get_tokens_and_source_mappings(query, log_sources) - - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info([source_mapping.source_id for source_mapping in source_mappings], {}), - functions=functions - ) diff --git a/siem-converter/app/converter/platforms/microsoft/parsers/microsoft_sentinel_rule.py b/siem-converter/app/converter/platforms/microsoft/parsers/microsoft_sentinel_rule.py deleted file mode 100644 index 156914ac..00000000 --- a/siem-converter/app/converter/platforms/microsoft/parsers/microsoft_sentinel_rule.py +++ /dev/null @@ -1,51 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from typing import List - -from app.converter.platforms.microsoft.const import microsoft_sentinel_rule_details -from app.converter.platforms.microsoft.parsers.microsoft_sentinel import MicrosoftParser -from app.converter.core.mixins.rule import JsonRuleMixin -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class MicrosoftRuleParser(MicrosoftParser, JsonRuleMixin): - details: PlatformDetails = microsoft_sentinel_rule_details - - @staticmethod - def _get_meta_info(source_mapping_ids: List[str], meta_info: dict) -> MetaInfoContainer: - return MetaInfoContainer( - source_mapping_ids=source_mapping_ids, - title=meta_info.get("displayName"), - description=meta_info.get("description"), - ) - - def parse(self, text: str) -> SiemContainer: - rule = self.load_rule(text=text) - query, log_sources, functions = self._parse_query(query=rule.get("query")) - tokens, source_mappings = self.get_tokens_and_source_mappings(query, log_sources) - - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info( - source_mapping_ids=[source_mapping.source_id for source_mapping in source_mappings], - meta_info=rule - ), - functions=functions - ) diff --git a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_defender.py b/siem-converter/app/converter/platforms/microsoft/renders/microsoft_defender.py deleted file mode 100644 index 45030b4b..00000000 --- a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_defender.py +++ /dev/null @@ -1,37 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.microsoft.const import microsoft_defender_details -from app.converter.platforms.microsoft.mapping import MicrosoftDefenderMappings, microsoft_defender_mappings -from app.converter.platforms.microsoft.renders.microsoft_sentinel import MicrosoftSentinelQueryRender, \ - MicrosoftSentinelFieldValue -from app.converter.core.models.platform_details import PlatformDetails - - -class MicrosoftDefenderFieldValue(MicrosoftSentinelFieldValue): - details: PlatformDetails = microsoft_defender_details - - -class MicrosoftDefenderQueryRender(MicrosoftSentinelQueryRender): - mappings: MicrosoftDefenderMappings = microsoft_defender_mappings - details: PlatformDetails = microsoft_defender_details - or_token = "or" - field_value_map = MicrosoftDefenderFieldValue(or_token=or_token) - - is_strict_mapping = True diff --git a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_defender_cti.py b/siem-converter/app/converter/platforms/microsoft/renders/microsoft_defender_cti.py deleted file mode 100644 index 5463b716..00000000 --- a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_defender_cti.py +++ /dev/null @@ -1,41 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.microsoft.const import microsoft_defender_details -from app.converter.platforms.microsoft.mappings.mdatp_cti import DEFAULT_MICROSOFT_DEFENDER_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class MicrosoftDefenderCTI(RenderCTI): - details: PlatformDetails = microsoft_defender_details - - data_map: dict = {"default": '{key} =~ "{value}"', "url": '{key} has "{value}"'} - or_operator: str = " or " - group_or_operator: str = " or " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "union * | where ({result})\n" - final_result_for_one: str = "union * | where {result}\n" - default_mapping = DEFAULT_MICROSOFT_DEFENDER_MAPPING - - def create_field_value(self, field: str, value: str, generic_field: str): - if data_map := self.data_map.get(generic_field): - return data_map.format(key=field, value=value) - return self.data_map.get("default").format(key=field, value=value) diff --git a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_sentinel.py b/siem-converter/app/converter/platforms/microsoft/renders/microsoft_sentinel.py deleted file mode 100644 index 47b4a27e..00000000 --- a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_sentinel.py +++ /dev/null @@ -1,95 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" -from typing import Union - -from app.converter.platforms.microsoft.const import microsoft_sentinel_query_details -from app.converter.platforms.microsoft.mapping import MicrosoftSentinelMappings, microsoft_sentinel_mappings -from app.converter.core.mapping import LogSourceSignature -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render import BaseQueryRender, BaseQueryFieldValue -from app.converter.platforms.microsoft.siem_functions.base import MicroSoftQueryFunctions - - -class MicrosoftSentinelFieldValue(BaseQueryFieldValue): - details: PlatformDetails = microsoft_sentinel_query_details - - @staticmethod - def __escape_value(value: Union[int, str]) -> Union[int, str]: - return value.replace("'", "''") if isinstance(value, str) else value - - def equal_modifier(self, field, value): - if isinstance(value, str): - return f"{field} =~ @'{self.__escape_value(value)}'" - elif isinstance(value, list): - prepared_values = ", ".join(f"@'{self.__escape_value(v)}'" for v in value) - operator = "in~" if all(isinstance(v, str) for v in value) else "in" - return f'{field} {operator} ({prepared_values})' - return f'{field} == {value}' - - def contains_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" - return f"{field} contains @'{self.__escape_value(value)}'" - - def endswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" - return f"{field} endswith @'{self.__escape_value(value)}'" - - def startswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" - return f"{field} startswith @'{self.__escape_value(value)}'" - - def __regex_modifier(self, field, value): - return f"{field} matches regex @'(?i){self.__escape_value(value)}'" - - def regex_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.__regex_modifier(field=field, value=v) for v in value)})" - return f'({self.__regex_modifier(field=field, value=value)})' - - def keywords(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" - return f"* contains @'{self.__escape_value(value)}'" - - -class MicrosoftSentinelQueryRender(BaseQueryRender): - details: PlatformDetails = microsoft_sentinel_query_details - - or_token = "or" - and_token = "and" - not_token = "not" - - field_value_map = MicrosoftSentinelFieldValue(or_token=or_token) - query_pattern = "{prefix} | where {query} {functions}" - - mappings: MicrosoftSentinelMappings = microsoft_sentinel_mappings - comment_symbol = "//" - is_multi_line_comment = True - - def generate_prefix(self, log_source_signature: LogSourceSignature) -> str: - return str(log_source_signature) - - def generate_functions(self, functions: list) -> str: - if not functions: - return "" - result = MicroSoftQueryFunctions().render(functions) - return result diff --git a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_sentinel_cti.py b/siem-converter/app/converter/platforms/microsoft/renders/microsoft_sentinel_cti.py deleted file mode 100644 index d944a7a0..00000000 --- a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_sentinel_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.microsoft.const import microsoft_sentinel_query_details -from app.converter.platforms.microsoft.mappings.microsoft_sentinel_cti import DEFAULT_MICROSOFT_SENTINEL_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class MicrosoftSentinelCTI(RenderCTI): - details: PlatformDetails = microsoft_sentinel_query_details - - data_map: str = '@"{value}"' - or_operator: str = " or " - group_or_operator: str = " or " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "search ({result})\n" - final_result_for_one: str = "search {result}\n" - default_mapping = DEFAULT_MICROSOFT_SENTINEL_MAPPING diff --git a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_sentinel_rule.py b/siem-converter/app/converter/platforms/microsoft/renders/microsoft_sentinel_rule.py deleted file mode 100644 index 78ad74c7..00000000 --- a/siem-converter/app/converter/platforms/microsoft/renders/microsoft_sentinel_rule.py +++ /dev/null @@ -1,60 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -import copy -import json -import re - -from app.converter.platforms.microsoft.renders.microsoft_sentinel import ( - MicrosoftSentinelQueryRender, - MicrosoftSentinelFieldValue -) -from app.converter.platforms.microsoft.const import DEFAULT_MICROSOFT_SENTINEL_RULE, microsoft_sentinel_rule_details -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer -from app.converter.tools.utils import concatenate_str, get_author_str, get_licence_str - - -class MicrosoftSentinelRuleFieldValue(MicrosoftSentinelFieldValue): - details: PlatformDetails = microsoft_sentinel_rule_details - - -class MicrosoftSentinelRuleRender(MicrosoftSentinelQueryRender): - details: PlatformDetails = microsoft_sentinel_rule_details - or_token = "or" - field_value_map = MicrosoftSentinelRuleFieldValue(or_token=or_token) - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None): - query = super().finalize_query(prefix=prefix, query=query, functions=functions, meta_info=meta_info) - rule = copy.deepcopy(DEFAULT_MICROSOFT_SENTINEL_RULE) - rule["query"] = query - rule["displayName"] = meta_info.title - description = meta_info.description or rule["description"] - description = concatenate_str(description, get_author_str(meta_info.author)) - description = concatenate_str(description, get_licence_str(meta_info.license)) - rule["description"] = description - rule["severity"] = meta_info.severity - rule["techniques"] = [el.upper() for el in meta_info.mitre_attack] - json_rule = json.dumps(rule, indent=4, sort_keys=False) - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return json_rule + rendered_not_supported - return json_rule diff --git a/siem-converter/app/converter/platforms/microsoft/siem_functions/base.py b/siem-converter/app/converter/platforms/microsoft/siem_functions/base.py deleted file mode 100644 index 487effbc..00000000 --- a/siem-converter/app/converter/platforms/microsoft/siem_functions/base.py +++ /dev/null @@ -1,20 +0,0 @@ -from app.converter.core.functions import Functions -from app.converter.core.models.functions.types import ParsedFunctions, NotSupportedFunction - - -class MicroSoftQueryFunctions(Functions): - - def parse(self, query: str): - result = [] - functions = query.split(self.function_delimiter) - query_result = [functions.pop(0).strip()] - not_supported = [] - for function in functions: - function_name = function.strip(" ").split(' ')[0] - if function_name.lower() == "where": - query_result.append(f'({function.lstrip("where ")})') - elif function_name in self.parser_functions_map: - pass - else: - not_supported.append(NotSupportedFunction(name=function_name, query=function)) - return ParsedFunctions(not_supported=self.prepare_not_supported(not_supported), functions=result), query_result diff --git a/siem-converter/app/converter/platforms/microsoft/tokenizer.py b/siem-converter/app/converter/platforms/microsoft/tokenizer.py deleted file mode 100644 index 4eafb3f6..00000000 --- a/siem-converter/app/converter/platforms/microsoft/tokenizer.py +++ /dev/null @@ -1,76 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Tuple, Any, Union - -from app.converter.core.mixins.operator import OperatorBasedMixin -from app.converter.core.tokenizer import QueryTokenizer -from app.converter.core.operator_types.tokens import OperatorType -from app.converter.tools.utils import get_match_group - - -class MicrosoftSentinelTokenizer(QueryTokenizer, OperatorBasedMixin): - field_pattern = r"(?P[a-zA-Z\.\-_]+)" - match_operator_pattern = r"""(?:___field___\s?(?Pcontains|endswith|startswith|in~|in|==|=~|!=|=))\s?""" - bool_value_pattern = r"(?Ptrue|false)\s*" - num_value_pattern = r"(?P\d+(?:\.\d+)*)\s*" - double_quotes_value_pattern = r'@?"(?P(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{\}\s]|\\\"|\\\\)*)"\s*' - single_quotes_value_pattern = r"@?'(?P(?:[:a-zA-Z\*0-9=+%#\-_/,\"\.$&^@!\(\)\{\}\s]|\\\'|\\\\)*)'\s*" - str_value_pattern = fr"""{double_quotes_value_pattern}|{single_quotes_value_pattern}""" - _value_pattern = fr"""{bool_value_pattern}|{num_value_pattern}|{str_value_pattern}""" - multi_value_pattern = r"""\((?P[:a-zA-Z\"\*0-9=+%#\-_\/\\'\,.&^@!\(\s]+)\)""" - keyword_pattern = fr"\*\s+contains\s+(?:{str_value_pattern})" - - multi_value_operators = ("in", "in~") - - operators_map = { - "==": OperatorType.EQ, - "in~": OperatorType.EQ, - "=~": OperatorType.EQ - } - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.operators_map.update(super().operators_map) - - def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> Tuple[str, Any]: - if (num_value := get_match_group(match, group_name='num_value')) is not None: - return operator, num_value - - elif (bool_value := get_match_group(match, group_name='bool_value')) is not None: - return operator, bool_value - - elif (d_q_value := get_match_group(match, group_name='d_q_value')) is not None: - return operator, d_q_value - - elif (s_q_value := get_match_group(match, group_name='s_q_value')) is not None: - return operator, s_q_value - - return super().get_operator_and_value(match, operator) - - def clean_multi_value(self, value: Union[int, str]) -> Union[int, str]: - if isinstance(value, str): - value = value.strip(" ") - value = value.lstrip("@") - if value.startswith("'") and value.endswith("'"): - value = value[1:-1] - elif value.startswith('"') and value.endswith('"'): - value = value[1:-1] - - return value diff --git a/siem-converter/app/converter/platforms/opensearch/const.py b/siem-converter/app/converter/platforms/opensearch/const.py deleted file mode 100644 index 9418ea32..00000000 --- a/siem-converter/app/converter/platforms/opensearch/const.py +++ /dev/null @@ -1,89 +0,0 @@ -from app.converter.core.models.platform_details import PlatformDetails - -PLATFORM_DETAILS = { - "group_id": "opensearch", - "group_name": "AWS OpenSearch", - "alt_platform_name": "ECS", -} - -OPENSEARCH_LUCENE_QUERY_DETAILS = { - "siem_type": "opensearch-lucene-query", - "name": "AWS OpenSearch Query", - "platform_name": "Query (Lucene)", - **PLATFORM_DETAILS -} - -OPENSEARCH_RULE_DETAILS = { - "siem_type": "opensearch-lucene-rule", - "name": "AWS OpenSearch Rule", - "platform_name": "Rule (JSON)", - "first_choice": 0, - **PLATFORM_DETAILS -} - -opensearch_query_details = PlatformDetails(**OPENSEARCH_LUCENE_QUERY_DETAILS) -opensearch_rule_details = PlatformDetails(**OPENSEARCH_RULE_DETAILS) - - -OPENSEARCH_RULE = { - "type": "monitor", - "name": "Autogenerated OpenDistro rule", - "enabled": True, - "schedule": { - "period": { - "interval": 1, - "unit": "MINUTES" - } - }, - "inputs": [ - { - "search": { - "indices": [ - "winlogbeat-*" - ], - "query": { - "size": 1000, - "query": { - "bool": { - "must": [ - { - "query_string": { - "query": "", - "analyze_wildcard": True - } - } - ] - } - } - } - } - } - ], - "triggers": [ - { - "name": "Trigger: Malicious PowerView PowerShell Commandlets", - "severity": "4", - "condition": { - "script": { - "source": "ctx.results[0].hits.total.value > 0", - "lang": "painless" - } - }, - "actions": [ - { - "name": "some alert", - "destination_id": "DESTINATION_ID", - "message_template": { - "source": "", - "lang": "mustache" - }, - "throttle_enabled": False, - "subject_template": { - "source": "", - "lang": "mustache" - } - } - ] - } - ] -} \ No newline at end of file diff --git a/siem-converter/app/converter/platforms/opensearch/mapping.py b/siem-converter/app/converter/platforms/opensearch/mapping.py deleted file mode 100644 index 0c6b9e96..00000000 --- a/siem-converter/app/converter/platforms/opensearch/mapping.py +++ /dev/null @@ -1,8 +0,0 @@ -from app.converter.platforms.base.lucene.mapping import LuceneMappings - - -class OpenSearchMappings(LuceneMappings): - pass - - -opensearch_mappings = OpenSearchMappings(platform_dir="opensearch") diff --git a/siem-converter/app/converter/platforms/opensearch/mappings/opensearch_cti.py b/siem-converter/app/converter/platforms/opensearch/mappings/opensearch_cti.py deleted file mode 100644 index 527725fc..00000000 --- a/siem-converter/app/converter/platforms/opensearch/mappings/opensearch_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_OPENSEARCH_MAPPING = { - "DestinationIP": "destination.ip", - "SourceIP": "source.ip", - "HashSha512": "file.hash.sha512", - "HashSha256": "file.hash.sha256", - "HashMd5": "file.hash.md5", - "Emails": "email.from.address", - "Domain": "destination.domain", - "HashSha1": "file.hash.sha1", - "Files": "file.name", - "URL": "url.original" -} diff --git a/siem-converter/app/converter/platforms/opensearch/parsers/opensearch.py b/siem-converter/app/converter/platforms/opensearch/parsers/opensearch.py deleted file mode 100644 index 5114327f..00000000 --- a/siem-converter/app/converter/platforms/opensearch/parsers/opensearch.py +++ /dev/null @@ -1,27 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from app.converter.platforms.base.lucene.parsers.lucene import LuceneParser -from app.converter.platforms.opensearch.const import opensearch_query_details -from app.converter.platforms.opensearch.mapping import OpenSearchMappings, opensearch_mappings -from app.converter.core.models.platform_details import PlatformDetails - - -class OpenSearchParser(LuceneParser): - details: PlatformDetails = opensearch_query_details - mappings: OpenSearchMappings = opensearch_mappings diff --git a/siem-converter/app/converter/platforms/opensearch/renders/opensearch.py b/siem-converter/app/converter/platforms/opensearch/renders/opensearch.py deleted file mode 100644 index 58d9951b..00000000 --- a/siem-converter/app/converter/platforms/opensearch/renders/opensearch.py +++ /dev/null @@ -1,68 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" -from app.converter.platforms.base.lucene.renders.lucene import LuceneQueryRender, LuceneFieldValue -from app.converter.platforms.opensearch.const import opensearch_query_details -from app.converter.platforms.opensearch.mapping import OpenSearchMappings, opensearch_mappings -from app.converter.core.models.platform_details import PlatformDetails - - -class OpenSearchFieldValue(LuceneFieldValue): - details: PlatformDetails = opensearch_query_details - - def equal_modifier(self, field, value): - if isinstance(value, list): - values = self.or_token.join(f'"{v}"' for v in value) - return f"{field}:({values})" - return f'{field}:"{value}"' - - def contains_modifier(self, field, value): - if isinstance(value, list): - values = self.or_token.join(f'"*{v}*"' for v in value) - return f"{field}:({values})" - return f'{field}:"*{value}*"' - - def endswith_modifier(self, field, value): - if isinstance(value, list): - values = self.or_token.join(f'"*{v}"' for v in value) - return f"{field}:({values})" - return f'{field}:"*{value}"' - - def startswith_modifier(self, field, value): - if isinstance(value, list): - values = self.or_token.join(f'"{v}*"' for v in value) - return f"{field}:({values})" - return f'{field}:"{value}*"' - - def regex_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" - return f'{field}:/{value}/"' - - def keywords(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" - return f'"*{value}*"' - - -class OpenSearchQueryRender(LuceneQueryRender): - details: PlatformDetails = opensearch_query_details - mappings: OpenSearchMappings = opensearch_mappings - - or_token = "OR" - field_value_map = OpenSearchFieldValue(or_token=or_token) diff --git a/siem-converter/app/converter/platforms/opensearch/renders/opensearch_cti.py b/siem-converter/app/converter/platforms/opensearch/renders/opensearch_cti.py deleted file mode 100644 index 9780f223..00000000 --- a/siem-converter/app/converter/platforms/opensearch/renders/opensearch_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.opensearch.const import opensearch_query_details -from app.converter.platforms.opensearch.mappings.opensearch_cti import DEFAULT_OPENSEARCH_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class OpenSearchCTI(RenderCTI): - details: PlatformDetails = opensearch_query_details - - data_map: str = '"{value}"' - or_operator: str = " OR " - group_or_operator: str = " OR " - or_group: str = "{processing_key}:({or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_OPENSEARCH_MAPPING diff --git a/siem-converter/app/converter/platforms/opensearch/renders/opensearch_rule.py b/siem-converter/app/converter/platforms/opensearch/renders/opensearch_rule.py deleted file mode 100644 index 7c24d0a8..00000000 --- a/siem-converter/app/converter/platforms/opensearch/renders/opensearch_rule.py +++ /dev/null @@ -1,61 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -import copy -import json - -from app.converter.platforms.opensearch.const import OPENSEARCH_RULE, opensearch_rule_details -from app.converter.platforms.opensearch.mapping import OpenSearchMappings, opensearch_mappings -from app.converter.platforms.opensearch.renders.opensearch import OpenSearchQueryRender, OpenSearchFieldValue -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer - - -SEVERITIES_MAP = {"informational": "5", "low": "4", "medium": "3", "high": "2", "critical": "1"} - - -class OpenSearchRuleFieldValue(OpenSearchFieldValue): - details: PlatformDetails = opensearch_rule_details - - -class OpenSearchRuleRender(OpenSearchQueryRender): - details: PlatformDetails = opensearch_rule_details - mappings: OpenSearchMappings = opensearch_mappings - - or_token = "OR" - and_token = "AND" - not_token = "NOT" - - field_value_map = OpenSearchRuleFieldValue(or_token=or_token) - query_pattern = "{prefix} {query} {functions}" - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None): - query = super().finalize_query(prefix=prefix, query=query, functions=functions, meta_info=meta_info) - rule = copy.deepcopy(OPENSEARCH_RULE) - rule["name"] = meta_info.title - rule["inputs"][0]["search"]["query"]["query"]["bool"]["must"][0]["query_string"]["query"] = query - rule["triggers"][0]["name"] = meta_info.title - rule["triggers"][0]["severity"] = SEVERITIES_MAP[meta_info.severity] - rule_str = json.dumps(rule, indent=4, sort_keys=False) - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return rule_str + rendered_not_supported - return rule_str diff --git a/siem-converter/app/converter/platforms/opensearch/tokenizer.py b/siem-converter/app/converter/platforms/opensearch/tokenizer.py deleted file mode 100644 index ee8d4189..00000000 --- a/siem-converter/app/converter/platforms/opensearch/tokenizer.py +++ /dev/null @@ -1,22 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" -from app.converter.platforms.base.lucene.tokenizer import LuceneTokenizer - - -class OpenSearchTokenizer(LuceneTokenizer): - pass diff --git a/siem-converter/app/converter/platforms/qradar/const.py b/siem-converter/app/converter/platforms/qradar/const.py deleted file mode 100644 index 079a84fa..00000000 --- a/siem-converter/app/converter/platforms/qradar/const.py +++ /dev/null @@ -1,17 +0,0 @@ -from app.converter.core.models.platform_details import PlatformDetails - -UTF8_PAYLOAD_PATTERN = r"UTF8\(payload\)" - -QRADAR_QUERY_DETAILS = { - "siem_type": "qradar-aql-query", - "name": "QRadar Query", - "platform_name": "Query (AQL)", - "group_id": "qradar", - "group_name": "QRadar" -} - -NUM_VALUE_PATTERN = r"(?P\d+(?:\.\d+)*)" -SINGLE_QUOTES_VALUE_PATTERN = r"""'(?P(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{\}\s]|'')*)'""" - - -qradar_query_details = PlatformDetails(**QRADAR_QUERY_DETAILS) diff --git a/siem-converter/app/converter/platforms/qradar/mapping.py b/siem-converter/app/converter/platforms/qradar/mapping.py deleted file mode 100644 index 34c04dd7..00000000 --- a/siem-converter/app/converter/platforms/qradar/mapping.py +++ /dev/null @@ -1,83 +0,0 @@ -from typing import List, Optional - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class QradarLogSourceSignature(LogSourceSignature): - def __init__(self, - tables: Optional[List[str]], - device_types: Optional[List[int]], - categories: Optional[List[int]], - qids: Optional[List[int]], - qid_event_categories: Optional[List[int]], - default_source: dict): - self.tables = set(tables or []) - self.device_types = set(device_types or []) - self.categories = set(categories or []) - self.qids = set(qids or []) - self.qid_event_categories = set(qid_event_categories or []) - self._default_source = default_source or {} - - def is_suitable(self, - table: List[str], - devicetype: Optional[List[int]], - category: Optional[List[int]], - qid: Optional[List[int]], - qideventcategory: Optional[List[int]]) -> bool: - table_match = set(table).issubset(self.tables) - device_type_match = set(devicetype or []).issubset(self.device_types) - category_match = set(category or []).issubset(self.categories) - qid_match = set(qid or []).issubset(self.qids) - qid_event_category_match = set(qideventcategory or []).issubset(self.qid_event_categories) - - return table_match and device_type_match and category_match and qid_match and qid_event_category_match - - def __str__(self) -> str: - return self._default_source.get("table", "events") - - @property - def extra_condition(self) -> str: - default_source = self._default_source - return " AND ".join((f"{key}={value}" for key, value in default_source.items() if key != "table" and value)) - - -class QradarMappings(BasePlatformMappings): - def prepare_log_source_signature(self, mapping: dict) -> QradarLogSourceSignature: - log_source = mapping.get("log_source", {}) - default_log_source = mapping["default_log_source"] - return QradarLogSourceSignature( - tables=log_source.get("table"), - device_types=log_source.get("devicetype"), - categories=log_source.get("category"), - qids=log_source.get("qid"), - qid_event_categories=log_source.get("qideventcategory"), - default_source=default_log_source - ) - - def get_suitable_source_mappings(self, - field_names: List[str], - table: List[str], - devicetype: List[int] = None, - category: List[int] = None, - qid: List[int] = None, - qideventcategory: List[int] = None) -> List[SourceMapping]: - suitable_source_mappings = [] - for source_mapping in self._source_mappings.values(): - if source_mapping.source_id == DEFAULT_MAPPING_NAME: - continue - - log_source_signature: QradarLogSourceSignature = source_mapping.log_source_signature - if table and log_source_signature.is_suitable(table, devicetype, category, qid, qideventcategory): - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - else: - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - - if not suitable_source_mappings: - suitable_source_mappings = [self._source_mappings[DEFAULT_MAPPING_NAME]] - - return suitable_source_mappings - - -qradar_mappings = QradarMappings(platform_dir="qradar") diff --git a/siem-converter/app/converter/platforms/qradar/mappings/qradar_cti.py b/siem-converter/app/converter/platforms/qradar/mappings/qradar_cti.py deleted file mode 100644 index 8896be54..00000000 --- a/siem-converter/app/converter/platforms/qradar/mappings/qradar_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_QRADAR_MAPPING = { - "DestinationIP": "destinationip", - "SourceIP": "sourceip", - "HashSha512": "File Hash", - "HashSha256": "File Hash", - "HashMd5": "File Hash", - "Emails": "emails", - "Domain": "Hostname", - "HashSha1": "File Hash", - "Files": "Filename", - "URL": "URL" -} diff --git a/siem-converter/app/converter/platforms/qradar/parsers/qradar.py b/siem-converter/app/converter/platforms/qradar/parsers/qradar.py deleted file mode 100644 index e9135e61..00000000 --- a/siem-converter/app/converter/platforms/qradar/parsers/qradar.py +++ /dev/null @@ -1,115 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Tuple, List, Union, Dict - -from app.converter.platforms.qradar.const import SINGLE_QUOTES_VALUE_PATTERN, NUM_VALUE_PATTERN, \ - qradar_query_details -from app.converter.platforms.qradar.mapping import QradarMappings, qradar_mappings -from app.converter.platforms.qradar.tokenizer import QradarTokenizer -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.parser import Parser -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer -from app.converter.tools.utils import get_match_group - - -class QradarParser(Parser): - details: PlatformDetails = qradar_query_details - tokenizer = QradarTokenizer() - mappings: QradarMappings = qradar_mappings - - log_source_functions = ("LOGSOURCENAME", "LOGSOURCEGROUPNAME", "LOGSOURCETYPENAME", "CATEGORYNAME") - log_source_function_pattern = r"\(?(?P___func_name___\([a-zA-Z]+\))(?:\s+like\s+|\s+ilike\s+|\s*=\s*)'(?P[%a-zA-Z\s]+)'\s*\)?\s+(?:and|or)?\s" - - log_source_key_types = ("devicetype", "category", "qid", "qideventcategory") - log_source_pattern = fr"___source_type___(?:\s+like\s+|\s+ilike\s+|\s*=\s*)(?:{SINGLE_QUOTES_VALUE_PATTERN}|{NUM_VALUE_PATTERN})(?:\s+(?:and|or)\s+|\s+)?" - num_value_pattern = r"[0-9]+" - multi_num_log_source_pattern = fr"___source_type___\s+in\s+\((?P(?:{num_value_pattern}(?:\s*,\s*)?)+)\)(?:\s+(?:and|or)\s+|\s+)?" - str_value_pattern = r"""(?:')(?P(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{\}\s]|'')+)(?:')""" - multi_str_log_source_pattern = fr"""___source_type___\s+in\s+\((?P(?:{str_value_pattern}(?:\s*,\s*)?)+)\)(?:\s+(?:and|or)\s+|\s+)?""" - - table_pattern = r"\sFROM\s(?P
[a-zA-Z\.\-\*]+)\sWHERE\s" - - def __clean_query(self, query: str) -> str: - for func_name in self.log_source_functions: - pattern = self.log_source_function_pattern.replace('___func_name___', func_name) - while search := re.search(pattern, query, flags=re.IGNORECASE): - pos_start = search.start() - pos_end = search.end() - query = query[:pos_start] + query[pos_end:] - - return query - - @staticmethod - def __parse_multi_value_log_source(match: re.Match, - query: str, - pattern: str) -> Tuple[str, Union[List[str], List[int]]]: - value = match.group("value") - pos_start = match.start() - pos_end = match.end() - query = query[:pos_start] + query[pos_end:] - return query, re.findall(pattern, value) - - def __parse_log_sources(self, query: str) -> Tuple[Dict[str, Union[List[str], List[int]]], str]: - log_sources = {} - - if search := re.search(self.table_pattern, query, flags=re.IGNORECASE): - log_sources["table"] = [search.group("table")] - pos_end = search.end() - query = query[pos_end:] - - for log_source_key in self.log_source_key_types: - pattern = self.log_source_pattern.replace('___source_type___', log_source_key) - while search := re.search(pattern, query, flags=re.IGNORECASE): - num_value = get_match_group(search, group_name='num_value') - str_value = get_match_group(search, group_name='s_q_value') - value = num_value and int(num_value) or str_value - log_sources.setdefault(log_source_key, []).append(value) - pos_start = search.start() - pos_end = search.end() - query = query[:pos_start] + query[pos_end:] - - pattern = self.multi_num_log_source_pattern.replace("___source_type___", log_source_key) - if search := re.search(pattern, query, flags=re.IGNORECASE): - query, values = self.__parse_multi_value_log_source(search, query, self.num_value_pattern) - values = [int(v) for v in values] - log_sources.setdefault(log_source_key, []).extend(values) - - pattern = self.multi_str_log_source_pattern.replace("___source_type___", log_source_key) - if search := re.search(pattern, query, flags=re.IGNORECASE): - query, values = self.__parse_multi_value_log_source(search, query, self.str_value_pattern) - log_sources.setdefault(log_source_key, []).extend(values) - - return log_sources, query - - def _parse_query(self, text: str) -> Tuple[str, Dict[str, Union[List[str], List[int]]]]: - query = self.__clean_query(text) - log_sources, query = self.__parse_log_sources(query) - return query, log_sources - - def _get_meta_info(self, source_mapping_ids: List[str]) -> MetaInfoContainer: - return MetaInfoContainer(source_mapping_ids=source_mapping_ids) - - def parse(self, text: str) -> SiemContainer: - query, log_sources = self._parse_query(text) - tokens, source_mappings = self.get_tokens_and_source_mappings(query, log_sources) - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info([source_mapping.source_id for source_mapping in source_mappings]) - ) diff --git a/siem-converter/app/converter/platforms/qradar/renders/qradar.py b/siem-converter/app/converter/platforms/qradar/renders/qradar.py deleted file mode 100644 index 4cacf1aa..00000000 --- a/siem-converter/app/converter/platforms/qradar/renders/qradar.py +++ /dev/null @@ -1,84 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from typing import Union, List - -from app.converter.platforms.qradar.const import qradar_query_details -from app.converter.platforms.qradar.mapping import QradarLogSourceSignature, QradarMappings, qradar_mappings -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render import BaseQueryRender, BaseQueryFieldValue - - -class QradarFieldValue(BaseQueryFieldValue): - details: PlatformDetails = qradar_query_details - - def equal_modifier(self, field: str, value: Union[Union[int, str, List[int], List[str]]]) -> str: - if isinstance(value, list): - return f"({self.or_token.join([self.equal_modifier(field=field, value=v) for v in value])})" - if field == "UTF8(payload)": - return f"UTF8(payload) ILIKE '{value}'" - if isinstance(value, int): - return f'"{field}"={value}' - - return f'"{field}"=\'{value}\'' - - def contains_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" - return f'"{field}" ILIKE \'%{value}%\'' - - def endswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" - return f'"{field}" ILIKE \'%{value}\'' - - def startswith_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" - return f'"{field}" ILIKE \'{value}%\'' - - def regex_modifier(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" - return f'"{field}" IMATCHES \'{value}\'' - - def keywords(self, field, value): - if isinstance(value, list): - return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" - return f'UTF8(payload) ILIKE "%{value}%"' - - -class QradarQueryRender(BaseQueryRender): - details: PlatformDetails = qradar_query_details - mappings: QradarMappings = qradar_mappings - - or_token = "OR" - and_token = "AND" - not_token = "NOT" - - field_value_map = QradarFieldValue(or_token=or_token) - query_pattern = "{prefix} AND {query} {functions}" - - def generate_prefix(self, log_source_signature: QradarLogSourceSignature) -> str: - table = str(log_source_signature) - extra_condition = log_source_signature.extra_condition - return f"SELECT UTF8(payload) FROM {table} WHERE {extra_condition}" - - def generate_functions(self, functions: list): - return "" diff --git a/siem-converter/app/converter/platforms/qradar/renders/qradar_cti.py b/siem-converter/app/converter/platforms/qradar/renders/qradar_cti.py deleted file mode 100644 index 39ac6e67..00000000 --- a/siem-converter/app/converter/platforms/qradar/renders/qradar_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.qradar.const import qradar_query_details -from app.converter.platforms.qradar.mappings.qradar_cti import DEFAULT_QRADAR_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class QRadarCTI(RenderCTI): - details: PlatformDetails = qradar_query_details - - data_map: str = "\"{key}\" = '{value}'" - or_operator: str = " or " - group_or_operator: str = " or " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "SELECT UTF8(payload) from events where {result}\n" - final_result_for_one: str = "SELECT UTF8(payload) from events where {result}\n" - default_mapping = DEFAULT_QRADAR_MAPPING diff --git a/siem-converter/app/converter/platforms/qradar/tokenizer.py b/siem-converter/app/converter/platforms/qradar/tokenizer.py deleted file mode 100644 index 680d6a0b..00000000 --- a/siem-converter/app/converter/platforms/qradar/tokenizer.py +++ /dev/null @@ -1,94 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Tuple, Any - -from app.converter.platforms.qradar.const import UTF8_PAYLOAD_PATTERN, SINGLE_QUOTES_VALUE_PATTERN, NUM_VALUE_PATTERN -from app.converter.core.models.field import Keyword -from app.converter.core.models.identifier import Identifier -from app.converter.core.tokenizer import QueryTokenizer -from app.converter.core.operator_types.tokens import OperatorType -from app.converter.tools.utils import get_match_group - - -class QradarTokenizer(QueryTokenizer): - field_pattern = r'(?P"[a-zA-Z\._\-\s]+"|[a-zA-Z\._\-]+)' - match_operator_pattern = r"""(?:___field___\s?(?Plike|ilike|matches|imatches|in|=|!=))\s?""" - bool_value_pattern = r"(?Ptrue|false)\s*" - _value_pattern = fr"{NUM_VALUE_PATTERN}|{bool_value_pattern}|{SINGLE_QUOTES_VALUE_PATTERN}" - keyword_pattern = fr"{UTF8_PAYLOAD_PATTERN}\s+(?:like|LIKE|ilike|ILIKE)\s+{SINGLE_QUOTES_VALUE_PATTERN}" - - multi_value_operators = ("in",) - wildcard_symbol = "%" - operators_map = { - "like": OperatorType.EQ, - "ilike": OperatorType.EQ, - "matches": OperatorType.REGEX, - "imatches": OperatorType.REGEX - } - - def __init__(self): - super().__init__() - self.operators_map.update(super().operators_map) - - @staticmethod - def should_process_value_wildcard_symbols(operator: str) -> bool: - return operator.lower() in ("like", "ilike") - - def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> Tuple[str, Any]: - if (num_value := get_match_group(match, group_name='num_value')) is not None: - return operator, num_value - - elif (bool_value := get_match_group(match, group_name='bool_value')) is not None: - return operator, bool_value - - elif (s_q_value := get_match_group(match, group_name='s_q_value')) is not None: - return operator, s_q_value - - return super().get_operator_and_value(match, operator) - - def escape_field_name(self, field_name): - field_name = field_name.replace('"', r'\"') - field_name = field_name.replace(' ', r'\ ') - return field_name - - def search_field_value(self, query): - field_name = self.search_field(query) - operator = self.search_match_operator(query, field_name) - should_process_value_wildcard_symbols = self.should_process_value_wildcard_symbols(operator) - query, operator, value = self.search_value(query=query, operator=operator, field_name=field_name) - - operator_token = Identifier(token_type=OperatorType.EQ) - if should_process_value_wildcard_symbols: - value, operator_token = self.process_value_wildcard_symbols( - value=value, - operator=operator, - wildcard_symbol=self.wildcard_symbol - ) - - field_name = field_name.strip('"') - field = self.create_field(field_name=field_name, operator=operator_token, value=value) - return field, query - - def search_keyword(self, query: str) -> Tuple[Keyword, str]: - keyword_search = re.search(self.keyword_pattern, query) - _, value = self.get_operator_and_value(keyword_search) - keyword = Keyword(value=self._clean_value(value, self.wildcard_symbol)) - pos = keyword_search.end() - return keyword, query[pos:] diff --git a/siem-converter/app/converter/platforms/qualys/const.py b/siem-converter/app/converter/platforms/qualys/const.py deleted file mode 100644 index 7a7868c1..00000000 --- a/siem-converter/app/converter/platforms/qualys/const.py +++ /dev/null @@ -1,8 +0,0 @@ - -QUALYS_QUERY_DETAILS = { - "siem_type": "qualys", - "name": "Qualys", - "platform_name": "IOC Query", - "group_name": "Qualys", - "group_id": "qualys" -} diff --git a/siem-converter/app/converter/platforms/qualys/mappings/qualys_cti.py b/siem-converter/app/converter/platforms/qualys/mappings/qualys_cti.py deleted file mode 100644 index b24135b4..00000000 --- a/siem-converter/app/converter/platforms/qualys/mappings/qualys_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_QUALYS_MAPPING = { - "DestinationIP": "network.remote.address.ip", - "SourceIP": "network.local.address.ip", - "HashSha512": "file.hash.sha512", - "HashSha256": "file.hash.sha256", - "HashMd5": "file.hash.md5", - "Emails": "emails", - "Domain": "domain", - "HashSha1": "file.hash.sha1", - "Files": "file.name", - "URL": "url" -} \ No newline at end of file diff --git a/siem-converter/app/converter/platforms/qualys/renders/qualys_cti.py b/siem-converter/app/converter/platforms/qualys/renders/qualys_cti.py deleted file mode 100644 index aec31146..00000000 --- a/siem-converter/app/converter/platforms/qualys/renders/qualys_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.qualys.const import QUALYS_QUERY_DETAILS -from app.converter.platforms.qualys.mappings.qualys_cti import DEFAULT_QUALYS_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class QualysCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**QUALYS_QUERY_DETAILS) - - data_map: str = "{key}:`{value}`" - or_operator: str = " or " - group_or_operator: str = " or " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_QUALYS_MAPPING diff --git a/siem-converter/app/converter/platforms/roota/mapping.py b/siem-converter/app/converter/platforms/roota/mapping.py deleted file mode 100644 index 5ead94e6..00000000 --- a/siem-converter/app/converter/platforms/roota/mapping.py +++ /dev/null @@ -1,47 +0,0 @@ -from typing import List - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class RootaLogSourceSignature(LogSourceSignature): - def __init__(self, product: List[str] = None, category: List[str] = None, service: List[str] = None, default_source: dict = None): - self.products = set(product or []) - self.categories = set(category or []) - self.services = set(service or []) - self._default_source = default_source or {} - - def is_suitable(self, service: str, product: str, category: str,) -> bool: - product_match = product in self.products - category_match = category in self.categories - service_match = service in self.services - return product_match and category_match and service_match - - @property - def default_source(self) -> str: - return self._default_source - - -class RootaMappings(BasePlatformMappings): - def prepare_source(self, mapping: dict) -> SourceMapping: - product = mapping.get("log_source", {}).get("product") - service = mapping.get("log_source", {}).get("service") - category = mapping.get("log_source", {}).get("category") - default_log_source = mapping["default_log_source"] - return SourceMapping( - source_id=mapping["source"], - source_signature=RootaLogSourceSignature(product=product, - service=service, - category=category, - default_source=default_log_source) - ) - - def get_log_source_id(self, product: str, service: str, category: str) -> str: - for source in self.sources_mapping.values(): - source_signature: RootaLogSourceSignature = source.source_signature - if source_signature.is_suitable(product=product, service=service, category=category): - return source.source_id - - return DEFAULT_MAPPING_NAME - - -roota_mappings = RootaMappings(platform_dir="roota") diff --git a/siem-converter/app/converter/platforms/roota/parsers/roota.py b/siem-converter/app/converter/platforms/roota/parsers/roota.py deleted file mode 100644 index c7690d64..00000000 --- a/siem-converter/app/converter/platforms/roota/parsers/roota.py +++ /dev/null @@ -1,63 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from app.converter.core.exceptions.core import UnsupportedRootAParser, RootARuleValidationException -from app.converter.core.mixins.rule import YamlRuleMixin -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer -from app.converter.core.parser import Parser -from app.converter.managers import parser_manager - - -class RootAParser(YamlRuleMixin): - parsers = parser_manager - mandatory_fields = {"name", "details", "author", "severity", "mitre-attack", "detection", "references", "license"} - - @staticmethod - def __update_meta_info(meta_info: MetaInfoContainer, rule: dict) -> MetaInfoContainer: - mitre_attack = rule.get("mitre-attack") or [] - mitre_attack = [i.strip("") for i in mitre_attack.split(",")] if isinstance(mitre_attack, str) else mitre_attack - meta_info.title = rule.get("name") - meta_info.description = rule.get("details") - meta_info.id = rule.get("uuid", meta_info.id) - meta_info.references = rule.get("references") - meta_info.license = rule.get("license", meta_info.license) - meta_info.tags = rule.get("tags") - meta_info.mitre_attack = mitre_attack - meta_info.date = rule.get("date", meta_info.date) - meta_info.author = rule.get("author", meta_info.author) - meta_info.severity = rule.get("severity", meta_info.severity) - return meta_info - - def _get_parser_class(self, parser: str) -> Parser: - parser_class = self.parsers.get(parser) - if parser_class: - return parser_class - raise UnsupportedRootAParser(parser=parser) - - def __validate_rule(self, rule: dict): - if missing_fields := self.mandatory_fields.difference(set(rule.keys())): - raise RootARuleValidationException(missing_fields=list(missing_fields)) - - def parse(self, text: str) -> SiemContainer: - roota_rule = self.load_rule(text=text) - self.__validate_rule(rule=roota_rule) - detection = roota_rule.get("detection", {}).get("body", "") - parser = self._get_parser_class(roota_rule.get("detection", {}).get("language", "")) - siem_container = parser.parse(text=detection) - siem_container.meta_info = self.__update_meta_info(meta_info=siem_container.meta_info, rule=roota_rule) - return siem_container diff --git a/siem-converter/app/converter/platforms/rsa_netwitness/const.py b/siem-converter/app/converter/platforms/rsa_netwitness/const.py deleted file mode 100644 index 2e89f734..00000000 --- a/siem-converter/app/converter/platforms/rsa_netwitness/const.py +++ /dev/null @@ -1,8 +0,0 @@ - -RSA_NETWITNESS_QUERY_DETAILS = { - "siem_type": "rsa_netwitness", - "name": "RSA NetWitness", - "group_name": "RSA NetWitness", - "platform_name": "Query", - "group_id": "rsa_netwitness" -} diff --git a/siem-converter/app/converter/platforms/rsa_netwitness/mappings/rsa_netwitness_cti.py b/siem-converter/app/converter/platforms/rsa_netwitness/mappings/rsa_netwitness_cti.py deleted file mode 100644 index 56144e91..00000000 --- a/siem-converter/app/converter/platforms/rsa_netwitness/mappings/rsa_netwitness_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_RSA_NETWITNESS_MAPPING = { - "DestinationIP": "ip.dst", - "SourceIP": "ip.src", - "HashSha512": "hash", - "HashSha256": "hash", - "HashMd5": "hash", - "Emails": "emails", - "Domain": "domain", - "HashSha1": "hash", - "Files": "files", - "URL": "web.page" -} diff --git a/siem-converter/app/converter/platforms/rsa_netwitness/renders/rsa_netwitness_cti.py b/siem-converter/app/converter/platforms/rsa_netwitness/renders/rsa_netwitness_cti.py deleted file mode 100644 index da389b09..00000000 --- a/siem-converter/app/converter/platforms/rsa_netwitness/renders/rsa_netwitness_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.rsa_netwitness.const import RSA_NETWITNESS_QUERY_DETAILS -from app.converter.platforms.rsa_netwitness.mappings.rsa_netwitness_cti import DEFAULT_RSA_NETWITNESS_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class RSANetwitnessCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**RSA_NETWITNESS_QUERY_DETAILS) - - data_map: str = "'{value}'" - or_operator: str = ", " - group_or_operator: str = " or " - or_group: str = "({processing_key} = {or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_RSA_NETWITNESS_MAPPING diff --git a/siem-converter/app/converter/platforms/securonix/__init__.py b/siem-converter/app/converter/platforms/securonix/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/securonix/const.py b/siem-converter/app/converter/platforms/securonix/const.py deleted file mode 100644 index 69b9c917..00000000 --- a/siem-converter/app/converter/platforms/securonix/const.py +++ /dev/null @@ -1,8 +0,0 @@ - -SECURONIX_QUERY_DETAILS = { - "siem_type": "securonix", - "name": "Securonix", - "platform_name": "Query", - "group_name": "Securonix", - "group_id": "securonix" -} diff --git a/siem-converter/app/converter/platforms/securonix/mappings/__init__.py b/siem-converter/app/converter/platforms/securonix/mappings/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/securonix/mappings/securonix_cti.py b/siem-converter/app/converter/platforms/securonix/mappings/securonix_cti.py deleted file mode 100644 index fe0e5806..00000000 --- a/siem-converter/app/converter/platforms/securonix/mappings/securonix_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_SECURONIX_MAPPING = { - "DestinationIP": "@destinationaddress", - "SourceIP": "@sourceaddress", - "HashSha512": "@filehash", - "HashSha256": "@filehash", - "HashMd5": "@filehash", - "Emails": "emails", - "Domain": "@destinationhostname", - "HashSha1": "@filehash", - "Files": "@filename", - "URL": "@requesturl" -} diff --git a/siem-converter/app/converter/platforms/securonix/renders/__init__.py b/siem-converter/app/converter/platforms/securonix/renders/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/securonix/renders/securonix_cti.py b/siem-converter/app/converter/platforms/securonix/renders/securonix_cti.py deleted file mode 100644 index ea631912..00000000 --- a/siem-converter/app/converter/platforms/securonix/renders/securonix_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.securonix.const import SECURONIX_QUERY_DETAILS -from app.converter.platforms.securonix.mappings.securonix_cti import DEFAULT_SECURONIX_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class SecuronixCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**SECURONIX_QUERY_DETAILS) - - data_map: str = '{key} CONTAINS "{value}"' - or_operator: str = " OR " - group_or_operator: str = " OR " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "index = archive AND {result}\n" - final_result_for_one: str = "index = archive AND {result}\n" - default_mapping = DEFAULT_SECURONIX_MAPPING diff --git a/siem-converter/app/converter/platforms/sentinel_one/__init__.py b/siem-converter/app/converter/platforms/sentinel_one/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sentinel_one/const.py b/siem-converter/app/converter/platforms/sentinel_one/const.py deleted file mode 100644 index 8604afb0..00000000 --- a/siem-converter/app/converter/platforms/sentinel_one/const.py +++ /dev/null @@ -1,8 +0,0 @@ - -SENTINEL_ONE_EVENTS_QUERY_DETAILS = { - "siem_type": "s1-events", - "name": "SentinelOne Events Query", - "group_name": "SentinelOne", - "group_id": "sentinel-one", - "platform_name": "Query (Events)", -} diff --git a/siem-converter/app/converter/platforms/sentinel_one/mappings/__init__.py b/siem-converter/app/converter/platforms/sentinel_one/mappings/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sentinel_one/mappings/s1_cti.py b/siem-converter/app/converter/platforms/sentinel_one/mappings/s1_cti.py deleted file mode 100644 index 005f5e29..00000000 --- a/siem-converter/app/converter/platforms/sentinel_one/mappings/s1_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_S1EVENTS_MAPPING = { - "SourceIP": "SrcIP", - "DestinationIP": "DstIP", - "Domain": "DNS", - "URL": "Url", - "HashMd5": "Md5", - "HashSha1": "Sha1", - "HashSha256": "Sha256", - "HashSha512": "Sha512", - "Emails": "emails", - "Files": "TgtFilePath" -} diff --git a/siem-converter/app/converter/platforms/sentinel_one/renders/__init__.py b/siem-converter/app/converter/platforms/sentinel_one/renders/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sentinel_one/renders/s1_cti.py b/siem-converter/app/converter/platforms/sentinel_one/renders/s1_cti.py deleted file mode 100644 index 63390496..00000000 --- a/siem-converter/app/converter/platforms/sentinel_one/renders/s1_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.sentinel_one.const import SENTINEL_ONE_EVENTS_QUERY_DETAILS -from app.converter.platforms.sentinel_one.mappings.s1_cti import DEFAULT_S1EVENTS_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class S1EventsCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**SENTINEL_ONE_EVENTS_QUERY_DETAILS) - - data_map: str = '"{value}"' - or_operator: str = ", " - group_or_operator: str = " OR " - or_group: str = "{processing_key} in contains anycase ({or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_S1EVENTS_MAPPING diff --git a/siem-converter/app/converter/platforms/sigma/__init__.py b/siem-converter/app/converter/platforms/sigma/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sigma/const.py b/siem-converter/app/converter/platforms/sigma/const.py deleted file mode 100644 index aa202f65..00000000 --- a/siem-converter/app/converter/platforms/sigma/const.py +++ /dev/null @@ -1,7 +0,0 @@ -SIGMA_RULE_DETAILS = { - "name": "Sigma", - "siem_type": "sigma", - "platform_name": "Sigma", - "group_name": "Sigma", - "group_id": "sigma" -} diff --git a/siem-converter/app/converter/platforms/sigma/mapping.py b/siem-converter/app/converter/platforms/sigma/mapping.py deleted file mode 100644 index 995c8f71..00000000 --- a/siem-converter/app/converter/platforms/sigma/mapping.py +++ /dev/null @@ -1,65 +0,0 @@ -from typing import List, Optional - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class SigmaLogSourceSignature(LogSourceSignature): - def __init__(self, - product: Optional[List[str]], - category: Optional[List[str]], - service: Optional[List[str]], - default_source: dict = None): - self.products = set(product or []) - self.categories = set(category or []) - self.services = set(service or []) - self._default_source = default_source or {} - - def is_suitable(self, - service: Optional[List[str]], - product: Optional[List[str]], - category: Optional[List[str]]) -> bool: - product_match = set(product or []).issubset(self.products) - category_match = set(category or []).issubset(self.categories) - service_match = set(service or []).issubset(self.services) - return product_match and category_match and service_match - - def __str__(self) -> str: - raise NotImplementedError() - - @property - def log_sources(self) -> dict: - return self._default_source - - -class SigmaMappings(BasePlatformMappings): - def prepare_log_source_signature(self, mapping: dict) -> SigmaLogSourceSignature: - product = mapping.get("log_source", {}).get("product") - service = mapping.get("log_source", {}).get("service") - category = mapping.get("log_source", {}).get("category") - default_log_source = mapping["default_log_source"] - return SigmaLogSourceSignature( - product=product, - service=service, - category=category, - default_source=default_log_source - ) - - def get_suitable_source_mappings(self, - field_names: List[str], - product: List[str] = None, - service: List[str] = None, - category: List[str] = None) -> List[SourceMapping]: - suitable_source_mappings = [] - for source_mapping in self._source_mappings.values(): - if source_mapping.source_id == DEFAULT_MAPPING_NAME: - continue - - source_signature: SigmaLogSourceSignature = source_mapping.log_source_signature - if source_signature.is_suitable(product=product, service=service, category=category): - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - - return suitable_source_mappings or [self._source_mappings[DEFAULT_MAPPING_NAME]] - - -sigma_mappings = SigmaMappings(platform_dir="sigma") diff --git a/siem-converter/app/converter/platforms/sigma/models/__init__.py b/siem-converter/app/converter/platforms/sigma/models/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sigma/models/modifiers.py b/siem-converter/app/converter/platforms/sigma/models/modifiers.py deleted file mode 100644 index 847258ce..00000000 --- a/siem-converter/app/converter/platforms/sigma/models/modifiers.py +++ /dev/null @@ -1,83 +0,0 @@ -from typing import Union, List - -from app.converter.core.models.field import Field -from app.converter.core.models.identifier import Identifier -from app.converter.core.operator_types.tokens import LogicalOperatorType, OperatorType, GroupType - - -class ModifierManager: - - and_token = Identifier(token_type=LogicalOperatorType.AND) - or_token = Identifier(token_type=LogicalOperatorType.OR) - - modifier_map = { - "re": OperatorType.REGEX - } - allowed_modifiers = [*modifier_map.keys(), "all", "windash"] - - def __validate_modifiers(self, modifiers: Union[str, list]): - if isinstance(modifiers, list) and len(modifiers) > 2: - raise - elif isinstance(modifiers, list): - return all(self.__validate_modifiers(modifier) for modifier in modifiers) - return True - - def map_modifier(self, modifier: str) -> Identifier: - return Identifier(token_type=self.modifier_map.get(modifier, modifier)) - - def modifier_all(self, field_name: str, modifier: str, - values: Union[str, List[str]]) -> Union[tuple, list]: - if (isinstance(values, list) and len(values) == 1) or isinstance(values, str): - operator = self.map_modifier(modifier=modifier) - return (Field(source_name=field_name, operator=operator, value=values), ) - else: - tokens = [] - for value in values: - tokens.extend(self.modifier_all(field_name=field_name, modifier=modifier, values=value)) - tokens.append(self.and_token) - return [Identifier(token_type=GroupType.L_PAREN), *tokens[:-1], Identifier(token_type=GroupType.R_PAREN)] - - def __prepare_windash_value(self, value: str): - if value.startswith(r"/"): - return [value, value.replace(r"/", "-", 1)] - elif value.startswith(r"-"): - return [value, value.replace("-", r"/", 1)] - return value - - def modifier_windash(self, field_name: str, modifier: Union[str, list], - values: Union[str, List[str]]) -> Union[tuple, list]: - if isinstance(values, list): - tokens = [] - for value in values: - tokens.extend(self.modifier_windash(field_name=field_name, modifier=modifier, values=value)) - tokens.append(self.or_token) - return [Identifier(token_type=GroupType.L_PAREN), *tokens[:-1], Identifier(token_type=GroupType.R_PAREN)] - operator = self.map_modifier(modifier=modifier) - field = Field(source_name=field_name, operator=operator, value=self.__prepare_windash_value(value=values)) - return (field,) - - def apply_multi_modifier(self, field_name: str, modifier: list, - values: Union[str, List[str]]) -> Union[tuple, list]: - if modifier[-1] == "all": - return self.modifier_all(field_name=field_name, modifier=modifier[0], values=values) - elif modifier[-1] == "windash": - return self.modifier_windash(field_name=field_name, modifier=modifier[0], values=values) - - def apply_modifier(self, field_name: str, modifier: list, values: Union[str, List[str]]) -> tuple: - modifier = modifier[0] - if modifier == "windash": - modifier = OperatorType.EQ - return self.modifier_windash(field_name=field_name, modifier=modifier, values=values) - operator = self.map_modifier(modifier=modifier) - return (Field(source_name=field_name, operator=operator, value=values), ) - - def create_token(self, field_name: str, modifier: list, - value: Union[str, List[str], int]) -> Union[tuple, list]: - if len(modifier) == 2: - return self.apply_multi_modifier(field_name=field_name, modifier=modifier, values=value) - return self.apply_modifier(field_name=field_name, modifier=modifier, values=value) - - def generate(self, field_name: str, modifier: list, - value: Union[str, List[str], int]) -> Union[tuple, list]: - if self.__validate_modifiers(modifiers=modifier): - return self.create_token(field_name=field_name, modifier=modifier, value=value) diff --git a/siem-converter/app/converter/platforms/sigma/parsers/__init__.py b/siem-converter/app/converter/platforms/sigma/parsers/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sigma/parsers/sigma.py b/siem-converter/app/converter/platforms/sigma/parsers/sigma.py deleted file mode 100644 index a9635a38..00000000 --- a/siem-converter/app/converter/platforms/sigma/parsers/sigma.py +++ /dev/null @@ -1,84 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - - -import re -from typing import List - -from app.converter.platforms.sigma.const import SIGMA_RULE_DETAILS -from app.converter.platforms.sigma.mapping import SigmaMappings, sigma_mappings -from app.converter.platforms.sigma.tokenizer import SigmaTokenizer, SigmaConditionTokenizer -from app.converter.core.exceptions.core import SigmaRuleValidationException -from app.converter.core.mixins.rule import YamlRuleMixin -from app.converter.core.models.field import Field -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class SigmaParser(YamlRuleMixin): - details: PlatformDetails = PlatformDetails(**SIGMA_RULE_DETAILS) - condition_tokenizer = SigmaConditionTokenizer() - tokenizer = SigmaTokenizer() - mappings: SigmaMappings = sigma_mappings - mandatory_fields = {"title", "description", "references", "logsource", "detection"} - - @staticmethod - def __parse_mitre_attack(tags: List[str]) -> List[str]: - result = [] - for tag in tags: - if search := re.search(r"[tT]\d{4}(?:\.\d{3})?", tag): - result.append(search.group()) - - return result - - def _get_meta_info(self, rule: dict, source_mapping_ids: List[str]) -> MetaInfoContainer: - return MetaInfoContainer( - title=rule.get("title"), - description=rule.get("description"), - author=rule.get("author"), - date=rule.get("date"), - references=rule.get("references", []), - license_=rule.get("license"), - mitre_attack=self.__parse_mitre_attack(rule.get("tags", [])), - severity=rule.get("level"), - status=rule.get("status"), - false_positives=rule.get("falsepositives"), - source_mapping_ids=source_mapping_ids - ) - - def __validate_rule(self, rule: dict): - if missing_fields := self.mandatory_fields.difference(set(rule.keys())): - raise SigmaRuleValidationException(missing_fields=list(missing_fields)) - - def parse(self, text: str) -> SiemContainer: - sigma_rule = self.load_rule(text=text) - self.__validate_rule(rule=sigma_rule) - log_sources = {key: [value] for key, value in (sigma_rule.get("logsource", {})).items()} - tokens = self.tokenizer.tokenize(detection=sigma_rule.get("detection")) - field_tokens = self.tokenizer.filter_tokens(tokens, Field) - field_names = [field.source_name for field in field_tokens] - suitable_source_mappings = self.mappings.get_suitable_source_mappings(field_names=field_names, **log_sources) - self.tokenizer.set_field_generic_names_map(field_tokens, suitable_source_mappings, self.mappings) - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info( - rule=sigma_rule, - source_mapping_ids=[source_mapping.source_id for source_mapping in suitable_source_mappings] - ), - ) diff --git a/siem-converter/app/converter/platforms/sigma/renders/__init__.py b/siem-converter/app/converter/platforms/sigma/renders/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sigma/renders/sigma.py b/siem-converter/app/converter/platforms/sigma/renders/sigma.py deleted file mode 100644 index 2ade383d..00000000 --- a/siem-converter/app/converter/platforms/sigma/renders/sigma.py +++ /dev/null @@ -1,262 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import copy -from typing import Any, List - -import yaml - -from app.converter.platforms.sigma.const import SIGMA_RULE_DETAILS -from app.converter.platforms.sigma.mapping import SigmaMappings, sigma_mappings, SigmaLogSourceSignature -from app.converter.core.compiler import DataStructureCompiler -from app.converter.core.exceptions.core import StrictPlatformFieldException -from app.converter.core.mapping import SourceMapping, DEFAULT_MAPPING_NAME -from app.converter.core.models.field import Field, Keyword -from app.converter.core.models.functions.types import ParsedFunctions -from app.converter.core.models.group import Group -from app.converter.core.models.operator import OR, AND, NOT -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer -from app.converter.core.operator_types.tokens import OperatorType - - -class SigmaRender: - selection_name = "selection" - selection_num = 0 - keyword_name = "keyword" - keyword_num = 0 - - mappings: SigmaMappings = sigma_mappings - details: PlatformDetails = PlatformDetails(**SIGMA_RULE_DETAILS) - - @property - def selection(self): - return f"{self.selection_name}{self.selection_num}" - - @property - def keyword(self): - return f"{self.selection_name}{self.keyword_num}" - - def increase_selection(self) -> None: - self.selection_num += 1 - - def increase_keyword(self) -> None: - self.keyword_num += 1 - - def reset_counters(self) -> None: - self.keyword_num = 0 - self.selection_num = 0 - - def generate_data_structure(self, data: Any, source_mapping: SourceMapping): - if isinstance(data, OR): - return self.generate_or(data, source_mapping) - elif isinstance(data, AND): - return self.generate_and(data, source_mapping) - elif isinstance(data, NOT): - return self.generate_not(data, source_mapping) - elif isinstance(data, Field): - return self.generate_field(data, source_mapping) - elif isinstance(data, Keyword): - return self.generate_keyword(data) - elif isinstance(data, Group): - return self.generate_group(data, source_mapping) - return None - - def generate_group(self, data: Any, source_mapping: SourceMapping): - sub_group = self.generate_data_structure(data.items, source_mapping) - if isinstance(sub_group, list): - return sub_group - if condition := sub_group.get("condition"): - sub_group["condition"] = condition if condition.count(self.selection_name) == 1 or condition.count(self.keyword_name) == 1 else f"({condition})" - return sub_group - - def generate_or(self, data: Any, source_mapping: SourceMapping): - result = {} - condition = "" - for i, item in enumerate(data.items): - updated_node = self.generate_data_structure(item, source_mapping) - if isinstance(updated_node, list): - if result and isinstance(result.get(self.keyword), list): - result[self.keyword].extend(updated_node) - else: - if self.keyword_num == self.selection_num: - self.increase_keyword() - self.increase_selection() - condition += f"{' or ' if condition else ''}{self.selection}" - result[self.selection] = updated_node - elif ( - result - and len(set(result.get(self.selection, [])).intersection(set(updated_node))) != 0 - and isinstance(data.items[i - 1], Field) - and len(updated_node) == 1 - and self.selection not in updated_node - ): - field_name = list(updated_node.keys())[0] - if isinstance(result[self.selection][field_name], list): - result[self.selection][field_name].append(updated_node[field_name]) - else: - result[self.selection][field_name] = [result[self.selection][field_name], updated_node[field_name]] - elif not result and self.selection in updated_node: - result = updated_node - t_c = result.get("condition") - condition = t_c if not condition else f"{condition} or {t_c if t_c.count(self.selection_name) == 1 else f'({t_c})'}" - elif result and self.selection in updated_node: - result.update(updated_node) - t_c = result.get("condition") - condition += f"{' or ' if condition else ''}{t_c if t_c.count(self.selection_name) == 1 else f'({t_c})'}" - else: - self.increase_selection() - result[self.selection] = updated_node - condition += f"{' or ' if condition else ''}{self.selection}" - result.update({"condition": condition}) - self.increase_keyword() - return result - - def generate_and(self, data: Any, source_mapping: SourceMapping): - result = {} - condition = "" - for item in data.items: - updated_node = self.generate_data_structure(item, source_mapping) - if isinstance(updated_node, list): - self.increase_keyword() - self.increase_selection() - condition += f"{' and ' if condition else ''}{self.selection}" - result[self.selection] = updated_node - elif (result and isinstance(updated_node, dict) and isinstance(result.get(self.selection, []), dict) and - len(set(result.get(self.selection, [])).intersection(set(updated_node))) == 0 and - self.selection not in updated_node): - if isinstance(result[self.selection], list): - result[self.selection].append(updated_node) - else: - result[self.selection].update(updated_node) - elif not result and self.selection in updated_node: - result = updated_node - t_c = result.get("condition") - condition = t_c if not condition else f"{condition} and {t_c}" - elif result and self.selection in updated_node: - result.update(updated_node) - condition += f"{' and ' if condition else ''}{result.get('condition')}" - else: - self.increase_selection() - result[self.selection] = updated_node - condition += f"{' and ' if condition else ''}{self.selection}" - result.update({"condition": condition}) - self.increase_keyword() - return result - - def generate_not(self, data: Any, source_mapping: SourceMapping): - not_node = self.generate_data_structure(data.items, source_mapping) - if isinstance(not_node, list): - self.increase_selection() - not_node = {self.selection: not_node, "condition": f"not {self.selection}"} - elif not not_node.get("condition"): - self.increase_selection() - if list(not_node.keys())[0] == self.keyword_name: - not_node = list(not_node.values()) - not_node = {self.selection: not_node, "condition": f"not {self.selection}"} - elif condition := not_node.get("condition"): - not_node["condition"] = f"not {condition}" - return not_node - - @staticmethod - def generate_field(data: Field, source_mapping: SourceMapping): - source_id = source_mapping.source_id - generic_field_name = data.generic_names_map[source_id] - if not generic_field_name: - raise StrictPlatformFieldException(field_name=data.source_name, platform_name="Sigma") - field_name = source_mapping.fields_mapping.get_platform_field_name(generic_field_name) - if data.operator.token_type != OperatorType.EQ: - field_name = f"{field_name}|{data.operator.token_type}" - if isinstance(data.values, list) and len(data.values) == 1 or isinstance(data.values, (str, int)): - return {field_name: data.values[0]} - elif isinstance(data.values, list) and len(data.values) == 0: - return {field_name: ""} - return {field_name: data.values} - - @staticmethod - def generate_keyword(data): - if isinstance(data.values, list) and len(data.values) == 1 or isinstance(data.values, (str, int)): - return [data.values[0]] - return data.values - - def __base_detection(self, data: dict): - self.increase_selection() - return {self.selection: data, "condition": self.selection} - - @staticmethod - def generate_aggregation(detection: dict, agg_data): - if timeframe := agg_data.get("timeframe"): - detection.update({"timeframe": timeframe}) - if agg_func := agg_data.get("agg_func"): - agg_condition = f" | {agg_func}" - if agg_field := agg_data.get("agg_field"): - agg_condition += f"({agg_field})" - else: - agg_condition += "()" - - if group_by := agg_data.get("group_by"): - agg_condition += f" by {', '.join(group_by)}" - - if agg_value := agg_data.get("agg_value"): - agg_condition += f" {agg_value}" - - detection["condition"] += agg_condition - return detection - - def generate_detection(self, data: Any, source_mapping: SourceMapping) -> dict: - detection = self.generate_data_structure(data, source_mapping) - if self.selection not in detection: - detection = self.__base_detection(detection) - condition = detection.pop("condition") - if condition.startswith("(") and condition.endswith(")") and "(" not in condition[1:-1]: - condition = condition.rstrip(")").lstrip("(") - detection.update({"condition": condition}) - self.reset_counters() - - return detection - - def __get_source_mapping(self, source_mapping_ids: List[str]) -> SourceMapping: - for source_mapping_id in source_mapping_ids: - if source_mapping := self.mappings.get_source_mapping(source_mapping_id): - return source_mapping - - return self.mappings.get_source_mapping(DEFAULT_MAPPING_NAME) - - def generate(self, query, meta_info: MetaInfoContainer, functions: ParsedFunctions): - self.reset_counters() - - source_mapping = self.__get_source_mapping(meta_info.source_mapping_ids) - log_source_signature: SigmaLogSourceSignature = source_mapping.log_source_signature - sigma_condition = copy.deepcopy(query) - prepared_data_structure = DataStructureCompiler().generate(tokens=sigma_condition) - - rule = { - "title": meta_info.title or "Autogenerated Sigma Rule", - "id": meta_info.id, - "description": meta_info.description, - "status": "experimental", - "author": meta_info.author, - "references": meta_info.references, - "tags": meta_info.tags, - "logsource": log_source_signature.log_sources, - "fields": [], - "detection": self.generate_detection(prepared_data_structure, source_mapping=source_mapping), - "level": meta_info.severity or "low", - "falsepositives": "", - } - return yaml.dump(rule, default_flow_style=False, sort_keys=False) diff --git a/siem-converter/app/converter/platforms/sigma/tokenizer.py b/siem-converter/app/converter/platforms/sigma/tokenizer.py deleted file mode 100644 index 9d16da6c..00000000 --- a/siem-converter/app/converter/platforms/sigma/tokenizer.py +++ /dev/null @@ -1,163 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import Union, List - -from app.converter.platforms.sigma.models.modifiers import ModifierManager -from app.converter.core.exceptions.parser import TokenizerGeneralException -from app.converter.core.models.field import Field, Keyword -from app.converter.core.models.identifier import Identifier -from app.converter.core.operator_types.tokens import GroupType, LogicalOperatorType -from app.converter.core.tokenizer import QueryTokenizer - - -class Selection: - - def __init__(self, name): - self.name = name - - def __repr__(self): - return f"Selection({self.name})" - - -class SigmaTokenizer(QueryTokenizer): - modifier_manager = ModifierManager() - - def __init__(self): - self.supported_selection_types = { - dict: self.__parse_and_selection, - list: self.__parse_or_selection - } - - def __parse_field(self, field_name: str, values: Union[int, str, List[str]]) -> Union[List, Field]: - field_name, *modifier = field_name.split("|") if "|" in field_name else (field_name, "=") - return self.modifier_manager.generate(field_name=field_name, modifier=modifier, value=values) - - @staticmethod - def __parse_keyword(values): - return (Keyword(value=values), ) - - def __parse_and_selection(self, selection: dict): - result = [] - for field, values in selection.items(): - token = self.__parse_field(field_name=field, values=values) - result.extend((*token, Identifier(token_type=LogicalOperatorType.AND))) - return [Identifier(token_type=GroupType.L_PAREN), *result[:-1], Identifier(token_type=GroupType.R_PAREN)] - - def __parse_or_selection(self, selection: List): - result = [] - for field_values in selection: - field, values = next(iter(field_values.items())) - token = self.__parse_field(field_name=field, values=values) - result.extend((*token, Identifier(token_type=LogicalOperatorType.OR))) - return [Identifier(token_type=GroupType.L_PAREN), *result[:-1], Identifier(token_type=GroupType.R_PAREN)] - - def __get_one_field_tokenized(self, selection): - if isinstance(selection, dict): - field, value = next(iter(selection.items())) - return self.__parse_field(field_name=field, values=value) - elif isinstance(selection, list) and all(isinstance(i, dict) for i in selection): - field, value = next(iter(selection[0].items())) - return self.__parse_field(field_name=field, values=value) - else: - return self.__parse_keyword(values=selection) - - def __parse_selection(self, selection): - if len(selection) == 1 and type(selection) in self.supported_selection_types: - return self.__get_one_field_tokenized(selection=selection) - elif isinstance(selection, list) and all(isinstance(i, str) for i in selection): - return self.__parse_keyword(selection) - elif selection_method := self.supported_selection_types.get(type(selection)): - return selection_method(selection=selection) - raise Exception("Unsupported sigma type") - - def tokenize(self, detection: dict) -> list: - condition = detection.pop("condition") - condition_tokens = SigmaConditionTokenizer().tokenize(condition=condition, detection=detection) - tokenized = [] - for token in condition_tokens: - if isinstance(token, Selection): - selection_values = detection.get(token.name) - selection = self.__parse_selection(selection=selection_values) - tokenized.extend(selection) - else: - tokenized.append(token) - return tokenized - - -class SigmaConditionTokenizer: - operator_pattern = r"\s?(?Pand|or|not)\s?" - selection_pattern = r"(?P[$a-zA-Z\._\-0-9]+)" - multi_selection_pattern = r"(?P[$a-zA-Z\._\-0-9]+)(?:\*){1}" - one_of_selection = "1 of " - all_of_selection = "all of " - detection = {} - - def __get_selection(self, condition): - selection_search = re.search(self.selection_pattern, condition) - if selection_search is None: - raise TokenizerGeneralException(error=f"Selection couldn't be found in condition: {condition}") - selection = selection_search.group("selection_name") - pos = selection_search.end() - return Selection(name=selection), condition[pos:] - - def __get_group(self, condition: str, operator: Identifier) -> (list, str): - selection_pattern_search = re.search(self.multi_selection_pattern, condition) - if selection_pattern_search is None: - raise TokenizerGeneralException(error=f"Selection couldn't be found in condition: {condition}") - selection_pattern = selection_pattern_search.group("selection_pattern") - pos = selection_pattern_search.end() - tokens = [] - for selection in self.detection: - if selection.startswith(selection_pattern): - tokens.append(Selection(name=selection)) - tokens.append(operator) - return [ - Identifier(token_type=GroupType.L_PAREN), *tokens[:-1], Identifier(token_type=GroupType.R_PAREN) - ], condition[pos:] - - def __get_identifier(self, condition): - condition = condition.strip("\n").strip(" ").strip("\n") - if condition.startswith(GroupType.L_PAREN): - return Identifier(token_type=GroupType.L_PAREN), condition[1:] - elif condition.startswith(GroupType.R_PAREN): - return Identifier(token_type=GroupType.R_PAREN), condition[1:] - elif operator_search := re.match(self.operator_pattern, condition, re.IGNORECASE): - operator = operator_search.group("operator") - pos = operator_search.end() - return Identifier(token_type=operator.lower()), condition[pos:] - elif condition.startswith(self.one_of_selection): - condition = re.sub(self.one_of_selection, "", condition, 1) - return self.__get_group(condition=condition, operator=Identifier(token_type=LogicalOperatorType.OR)) - elif condition.startswith(self.all_of_selection): - condition = re.sub(self.all_of_selection, "", condition, 1) - return self.__get_group(condition=condition, operator=Identifier(token_type=LogicalOperatorType.AND)) - else: - return self.__get_selection(condition) - - def tokenize(self, condition: str, detection: dict) -> list: - self.detection = detection - tokenized = [] - while condition: - identifier, condition = self.__get_identifier(condition=condition) - if isinstance(identifier, list): - tokenized.extend(identifier) - else: - tokenized.append(identifier) - return tokenized diff --git a/siem-converter/app/converter/platforms/snowflake/__init__.py b/siem-converter/app/converter/platforms/snowflake/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/snowflake/const.py b/siem-converter/app/converter/platforms/snowflake/const.py deleted file mode 100644 index dd6ed585..00000000 --- a/siem-converter/app/converter/platforms/snowflake/const.py +++ /dev/null @@ -1,8 +0,0 @@ - -SNOWFLAKE_QUERY_DETAILS = { - "siem_type": "snowflake", - "name": "Snowflake Query", - "group_name": "Snowflake", - "group_id": "snowflake-pack", - "platform_name": "Query (SQL)", -} diff --git a/siem-converter/app/converter/platforms/snowflake/mappings/__init__.py b/siem-converter/app/converter/platforms/snowflake/mappings/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/snowflake/mappings/snowflake_cti.py b/siem-converter/app/converter/platforms/snowflake/mappings/snowflake_cti.py deleted file mode 100644 index 65fcf89f..00000000 --- a/siem-converter/app/converter/platforms/snowflake/mappings/snowflake_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_SNOWFLAKE_MAPPING = { - "SourceIP": "source.ip", - "DestinationIP": "destination.ip", - "Domain": "destination.domain", - "URL": "url.original", - "HashMd5": "file.hash.md5", - "HashSha1": "file.hash.sha1", - "HashSha256": "file.hash.sha256", - "HashSha512": "file.hash.sha512", - "Files": "file.path", - "Emails": "user.name" -} diff --git a/siem-converter/app/converter/platforms/snowflake/renders/__init__.py b/siem-converter/app/converter/platforms/snowflake/renders/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/snowflake/renders/snowflake_cti.py b/siem-converter/app/converter/platforms/snowflake/renders/snowflake_cti.py deleted file mode 100644 index 666a034c..00000000 --- a/siem-converter/app/converter/platforms/snowflake/renders/snowflake_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.snowflake.const import SNOWFLAKE_QUERY_DETAILS -from app.converter.platforms.snowflake.mappings.snowflake_cti import DEFAULT_SNOWFLAKE_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class SnowflakeCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**SNOWFLAKE_QUERY_DETAILS) - - data_map: str = "{key} = '{value}'" - or_operator: str = " OR " - group_or_operator: str = " OR " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "SELECT * FROM table WHERE {result}\n" - final_result_for_one: str = "SELECT * FROM table WHERE {result}\n" - default_mapping = DEFAULT_SNOWFLAKE_MAPPING diff --git a/siem-converter/app/converter/platforms/splunk/__init__.py b/siem-converter/app/converter/platforms/splunk/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/splunk/const.py b/siem-converter/app/converter/platforms/splunk/const.py deleted file mode 100644 index 8986b8fa..00000000 --- a/siem-converter/app/converter/platforms/splunk/const.py +++ /dev/null @@ -1,48 +0,0 @@ -from app.converter.core.models.platform_details import PlatformDetails - -DEFAULT_SPLUNK_ALERT = """ -[] -alert.severity = -description = -cron_schedule = 0 * * * * -disabled = 1 -is_scheduled = 1 -is_visible = 1 -dispatch.earliest_time = -60m@m -dispatch.latest_time = now -search = -alert.suppress = 0 -alert.track = 1 -actions = risk,notable -action.risk = 1 -action.risk.param._risk_object_type = user -action.risk.param._risk_score = 75 -action.correlationsearch = 0 -action.correlationsearch.enabled = 1 -action.notable.param.rule_title = -action.notable.param.rule_description = -action.correlationsearch.label = -""" - -PLATFORM_DETAILS = { - "group_id": "splunk-pack", - "group_name": "Splunk" -} - -SPLUNK_QUERY_DETAILS = { - "siem_type": "splunk-spl-query", - "name": "Splunk Query", - "platform_name": "Query (SPL)", - **PLATFORM_DETAILS -} - -SPLUNK_ALERT_DETAILS = { - "siem_type": "splunk-spl-rule", - "name": "Splunk Alert", - "platform_name": "Alert (SPL)", - "first_choice": 0, - **PLATFORM_DETAILS -} - -splunk_query_details = PlatformDetails(**SPLUNK_QUERY_DETAILS) -splunk_alert_details = PlatformDetails(**SPLUNK_ALERT_DETAILS) diff --git a/siem-converter/app/converter/platforms/splunk/mapping.py b/siem-converter/app/converter/platforms/splunk/mapping.py deleted file mode 100644 index 2adf419b..00000000 --- a/siem-converter/app/converter/platforms/splunk/mapping.py +++ /dev/null @@ -1,66 +0,0 @@ -from typing import List, Optional - -from app.converter.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping, DEFAULT_MAPPING_NAME - - -class SplunkLogSourceSignature(LogSourceSignature): - def __init__(self, - sources: Optional[List[str]], - source_types: Optional[List[str]], - source_categories: Optional[List[str]], - indices: Optional[List[str]], - default_source: dict = None): - self.sources = set(sources or []) - self.source_types = set(source_types or []) - self.source_categories = set(source_categories or []) - self.indices = set(indices or []) - self._default_source = default_source or {} - - def is_suitable(self, - source: Optional[List[str]], - source_type: Optional[List[str]], - source_category: Optional[List[str]], - index: Optional[List[str]]) -> bool: - source_match = set(source or []).issubset(self.sources) - source_type_match = set(source_type or []).issubset(self.source_types) - source_category_match = set(source_category or []).issubset(self.source_categories) - index_match = set(index or []).issubset(self.indices) - - return source_match and source_type_match and source_category_match and index_match - - def __str__(self) -> str: - return " AND ".join((f"{key}={value}" for key, value in self._default_source.items() if value)) - - -class SplunkMappings(BasePlatformMappings): - def prepare_log_source_signature(self, mapping: dict) -> SplunkLogSourceSignature: - log_source = mapping.get("log_source", {}) - default_log_source = mapping["default_log_source"] - return SplunkLogSourceSignature( - sources=log_source.get("source"), - source_types=log_source.get("source_type"), - source_categories=log_source.get("source_category"), - indices=log_source.get("index"), - default_source=default_log_source - ) - - def get_suitable_source_mappings(self, - field_names: List[str], - source: List[str] = None, - sourcetype: List[str] = None, - sourcecategory: List[str] = None, - index: List[str] = None) -> List[SourceMapping]: - suitable_source_mappings = [] - for source_mapping in self._source_mappings.values(): - if source_mapping.source_id == DEFAULT_MAPPING_NAME: - continue - - source_signature: SplunkLogSourceSignature = source_mapping.log_source_signature - if source_signature.is_suitable(source, sourcetype, sourcecategory, index): - if source_mapping.fields_mapping.is_suitable(field_names): - suitable_source_mappings.append(source_mapping) - - return suitable_source_mappings or [self._source_mappings[DEFAULT_MAPPING_NAME]] - - -splunk_mappings = SplunkMappings(platform_dir="splunk") diff --git a/siem-converter/app/converter/platforms/splunk/mappings/__init__.py b/siem-converter/app/converter/platforms/splunk/mappings/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/splunk/mappings/splunk_cti.py b/siem-converter/app/converter/platforms/splunk/mappings/splunk_cti.py deleted file mode 100644 index 1afcb05c..00000000 --- a/siem-converter/app/converter/platforms/splunk/mappings/splunk_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_SPLUNK_MAPPING = { - "DestinationIP": "dest_ip", - "SourceIP": "src_ip", - "HashSha512": "file_hash", - "HashSha256": "file_hash", - "HashMd5": "file_hash", - "Emails": "All_Email.src_user", - "Domain": "dest_host", - "HashSha1": "file_hash", - "Files": "file_path", - "URL": "url" -} diff --git a/siem-converter/app/converter/platforms/splunk/parsers/__init__.py b/siem-converter/app/converter/platforms/splunk/parsers/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/splunk/parsers/splunk.py b/siem-converter/app/converter/platforms/splunk/parsers/splunk.py deleted file mode 100644 index 79a24f58..00000000 --- a/siem-converter/app/converter/platforms/splunk/parsers/splunk.py +++ /dev/null @@ -1,33 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -from app.converter.platforms.base.spl.parsers.spl import SplParser -from app.converter.platforms.splunk.const import splunk_query_details -from app.converter.platforms.splunk.mapping import SplunkMappings, splunk_mappings -from app.converter.platforms.splunk.siem_functions import SplunkFunctions -from app.converter.core.models.platform_details import PlatformDetails - - -class SplunkParser(SplParser): - details: PlatformDetails = splunk_query_details - siem_functions = SplunkFunctions() - - log_source_pattern = r"___source_type___\s*=\s*(?:\"(?P[%a-zA-Z_*:0-9\-/]+)\"|(?P[%a-zA-Z_*:0-9\-/]+))(?:\s+(?:and|or)\s+|\s+)?" - log_source_key_types = ("index", "source", "sourcetype", "sourcecategory") - - mappings: SplunkMappings = splunk_mappings diff --git a/siem-converter/app/converter/platforms/splunk/parsers/splunk_alert.py b/siem-converter/app/converter/platforms/splunk/parsers/splunk_alert.py deleted file mode 100644 index 3763720a..00000000 --- a/siem-converter/app/converter/platforms/splunk/parsers/splunk_alert.py +++ /dev/null @@ -1,48 +0,0 @@ -""" -Uncoder IO Commercial Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -This file is part of the Uncoder IO Commercial Edition ("CE") and is -licensed under the Uncoder IO Non-Commercial License (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ------------------------------------------------------------------ -""" - -import re -from typing import List, Optional - -from app.converter.platforms.splunk.const import splunk_alert_details -from app.converter.platforms.splunk.parsers.splunk import SplunkParser -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import SiemContainer, MetaInfoContainer - - -class SplunkAlertParser(SplunkParser): - details: PlatformDetails = splunk_alert_details - - @staticmethod - def _get_meta_info(source_mapping_ids: List[str], meta_info: Optional[str]) -> MetaInfoContainer: - description = re.search(r"description\s*=\s*(?P.+)", meta_info).group("description") - return MetaInfoContainer( - source_mapping_ids=source_mapping_ids, - description=description - ) - - def parse(self, text: str) -> SiemContainer: - query = re.search(r"search\s*=\s*(?P.+)", text).group("query") - log_sources, functions, query = self._parse_query(query) - tokens, source_mappings = self.get_tokens_and_source_mappings(query, log_sources) - - return SiemContainer( - query=tokens, - meta_info=self._get_meta_info([source_mapping.source_id for source_mapping in source_mappings]), - functions=functions - ) diff --git a/siem-converter/app/converter/platforms/splunk/renders/__init__.py b/siem-converter/app/converter/platforms/splunk/renders/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/splunk/renders/splunk.py b/siem-converter/app/converter/platforms/splunk/renders/splunk.py deleted file mode 100644 index ebca22c0..00000000 --- a/siem-converter/app/converter/platforms/splunk/renders/splunk.py +++ /dev/null @@ -1,35 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" -from app.converter.platforms.base.spl.renders.spl import SplFieldValue, SplQueryRender -from app.converter.platforms.splunk.const import splunk_query_details -from app.converter.platforms.splunk.mapping import SplunkMappings, splunk_mappings -from app.converter.core.models.platform_details import PlatformDetails - - -class SplunkFieldValue(SplFieldValue): - details: PlatformDetails = splunk_query_details - - -class SplunkQueryRender(SplQueryRender): - details: PlatformDetails = splunk_query_details - - or_token = "OR" - - field_value_map = SplunkFieldValue(or_token=or_token) - mappings: SplunkMappings = splunk_mappings diff --git a/siem-converter/app/converter/platforms/splunk/renders/splunk_alert.py b/siem-converter/app/converter/platforms/splunk/renders/splunk_alert.py deleted file mode 100644 index f16b9efd..00000000 --- a/siem-converter/app/converter/platforms/splunk/renders/splunk_alert.py +++ /dev/null @@ -1,54 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.splunk.renders.splunk import SplunkQueryRender, SplunkFieldValue -from app.converter.platforms.splunk.const import DEFAULT_SPLUNK_ALERT, splunk_alert_details -from app.converter.core.mapping import SourceMapping -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.operator_types.output import MetaInfoContainer - - -_AUTOGENERATED_TITLE = "Autogenerated Splunk Alert" - - -class SplunkAlertFieldValue(SplunkFieldValue): - details: PlatformDetails = splunk_alert_details - - -class SplunkAlertRender(SplunkQueryRender): - details: PlatformDetails = splunk_alert_details - or_token = "OR" - field_value_map = SplunkAlertFieldValue(or_token=or_token) - - def finalize_query(self, prefix: str, query: str, functions: str, meta_info: MetaInfoContainer, - source_mapping: SourceMapping = None, not_supported_functions: list = None): - query = super().finalize_query(prefix=prefix, query=query, functions=functions, meta_info=meta_info) - severity_map = {"critical": "4", "high": "3", "medium": "2", "low": "1"} - rule = DEFAULT_SPLUNK_ALERT.replace("", query) - rule = rule.replace("", meta_info.title or _AUTOGENERATED_TITLE) - rule = rule.replace("", severity_map.get(meta_info.severity, "1")) - - description = f"{meta_info.description or 'Autogenerated Splunk Alert'}. License: {meta_info.license}." - if meta_info.mitre_attack: - description += f" MITRE ATT&CK: {', '.join(meta_info.mitre_attack)}" - rule = rule.replace("", description) - if not_supported_functions: - rendered_not_supported = self.render_not_supported_functions(not_supported_functions) - return rule + rendered_not_supported - return rule diff --git a/siem-converter/app/converter/platforms/splunk/renders/splunk_cti.py b/siem-converter/app/converter/platforms/splunk/renders/splunk_cti.py deleted file mode 100644 index 1211e706..00000000 --- a/siem-converter/app/converter/platforms/splunk/renders/splunk_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.splunk.const import splunk_query_details -from app.converter.platforms.splunk.mappings.splunk_cti import DEFAULT_SPLUNK_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class SplunkCTI(RenderCTI): - details: PlatformDetails = splunk_query_details - - data_map: str = '{key}="{value}"' - or_operator: str = " OR " - group_or_operator: str = " OR " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_SPLUNK_MAPPING diff --git a/siem-converter/app/converter/platforms/splunk/siem_functions/__init__.py b/siem-converter/app/converter/platforms/splunk/siem_functions/__init__.py deleted file mode 100644 index 9c2a73c7..00000000 --- a/siem-converter/app/converter/platforms/splunk/siem_functions/__init__.py +++ /dev/null @@ -1,7 +0,0 @@ -from app.converter.core.functions import Functions - - -class SplunkFunctions(Functions): - siem_type = 'splunk' - functions_classes = [] - diff --git a/siem-converter/app/converter/platforms/sumo_logic/__init__.py b/siem-converter/app/converter/platforms/sumo_logic/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sumo_logic/const.py b/siem-converter/app/converter/platforms/sumo_logic/const.py deleted file mode 100644 index 0f1ac19e..00000000 --- a/siem-converter/app/converter/platforms/sumo_logic/const.py +++ /dev/null @@ -1,9 +0,0 @@ - -SUMO_LOGIC_QUERY_DETAILS = { - "siem_type": "sumologic", - "name": "Sumo Logic Query", - "group_name": "Sumo Logic", - "platform_name": "Query", - "first_choice": 0, - "group_id": "sumologic" -} diff --git a/siem-converter/app/converter/platforms/sumo_logic/mappings/__init__.py b/siem-converter/app/converter/platforms/sumo_logic/mappings/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sumo_logic/mappings/sumologic_cti.py b/siem-converter/app/converter/platforms/sumo_logic/mappings/sumologic_cti.py deleted file mode 100644 index 8d022e97..00000000 --- a/siem-converter/app/converter/platforms/sumo_logic/mappings/sumologic_cti.py +++ /dev/null @@ -1,12 +0,0 @@ -DEFAULT_SUMOLOGIC_MAPPING = { - "SourceIP": "src_ip", - "DestinationIP": "dst_ip", - "Domain": "host", - "URL": "url", - "HashMd5": "fileHash", - "HashSha1": "fileHash", - "HashSha256": "fileHash", - "HashSha512": "fileHash", - "Emails": "flattened_destinations", - "Files": "files" -} diff --git a/siem-converter/app/converter/platforms/sumo_logic/renders/__init__.py b/siem-converter/app/converter/platforms/sumo_logic/renders/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/platforms/sumo_logic/renders/sumologic_cti.py b/siem-converter/app/converter/platforms/sumo_logic/renders/sumologic_cti.py deleted file mode 100644 index 33773088..00000000 --- a/siem-converter/app/converter/platforms/sumo_logic/renders/sumologic_cti.py +++ /dev/null @@ -1,36 +0,0 @@ -""" -Uncoder IO Community Edition License ------------------------------------------------------------------ -Copyright (c) 2023 SOC Prime, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ------------------------------------------------------------------ -""" - -from app.converter.platforms.sumo_logic.const import SUMO_LOGIC_QUERY_DETAILS -from app.converter.platforms.sumo_logic.mappings.sumologic_cti import DEFAULT_SUMOLOGIC_MAPPING -from app.converter.core.models.platform_details import PlatformDetails -from app.converter.core.render_cti import RenderCTI - - -class SumologicCTI(RenderCTI): - details: PlatformDetails = PlatformDetails(**SUMO_LOGIC_QUERY_DETAILS) - - data_map: str = '{key} = "{value}"' - or_operator: str = " OR " - group_or_operator: str = " OR " - or_group: str = "({or_group})" - result_join: str = "" - final_result_for_many: str = "({result})\n" - final_result_for_one: str = "{result}\n" - default_mapping = DEFAULT_SUMOLOGIC_MAPPING diff --git a/siem-converter/app/converter/tools/__init__.py b/siem-converter/app/converter/tools/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/converter/tools/const.py b/siem-converter/app/converter/tools/const.py deleted file mode 100644 index 1616189a..00000000 --- a/siem-converter/app/converter/tools/const.py +++ /dev/null @@ -1,25 +0,0 @@ -from typing import Literal - -IP_IOC_REGEXP_PATTERN = r"(?:^|[ \/\[(\"',;>|])((?:25[0-5]|2[0-4]\d|[0-1]?\d{1,2})(?:\.(?:25[0-5]|2[0-4]\d|[0-1]?\d{1,2})){3})(?=[\s)\]\"',;:\/?\n<|]|$)" -DOMAIN_IOC_REGEXP_PATTERN = r"(?:^|[\s\/\[\]@(\"',;{>|])(?:(?:http[s]?|ftp):\/\/?)?([^:\\\/\s({\[\]@\"'`,]+\.[a-zA-Z]+)(?:(?:(?:[/|:]\w+)*\/)(?:[\w\-.]+[^#?\s]+)?(?:[\w/\-&?=%.#]+(?:\(\))?)?)?(?=[\s)\]\"',;<|]|$)" -URL_IOC_REGEXP_PATTERN = r"(?:^|[\s\/\[\]@(\"',;{>|])((?:(?:http[s]?|ftp):\/\/?)+(?:[^:\\\/\s({\[\]@\"'`,]+\.[a-zA-Z0-9]+)(?:(?:(?:[/|:]\w+)*\/)(?:[\w\-.]+[^#?\s<']+)?(?:[\w/\-&?=%.#]+(?:\(\))?)?)?)(?=[\s)\]\"',;<|]|$)" - -IOCType = Literal["ip", "domain", "url", "hash"] -HashType = Literal["md5", "sha1", "sha256", "sha512"] -IocParsingRule = Literal[ - "replace_dots", "remove_private_and_reserved_ips", "replace_hxxp" -] - -HASH_MAP = { - "md5": "HashMd5", - "sha1": "HashSha1", - "sha256": "HashSha256", - "sha512": "HashSha512" -} - -hash_regexes = { - "md5": r"(?:^|[\s\/\[(\"',;{>|])([A-Fa-f0-9]{32})(?=[\s)\]\"',;\n<|]|$)", - "sha1": r"(?:^|[\s\/\[(\"',;{>|])([A-Fa-f0-9]{40})(?=[\s)\]\"',;\n<|]|$)", - "sha256": r"(?:^|[\s\/\[(\"',;{>|])([A-Fa-f0-9]{64})(?=[\s)\]\"',;\n<|]|$)", - "sha512": r"(?:^|[\s\/\[(\"',;{>|])([A-Fa-f0-9]{128})(?=[\s)\]\"',;\n<|]|$)", -} diff --git a/siem-converter/app/converter/tools/custom_enum.py b/siem-converter/app/converter/tools/custom_enum.py deleted file mode 100644 index 9d685c7d..00000000 --- a/siem-converter/app/converter/tools/custom_enum.py +++ /dev/null @@ -1,29 +0,0 @@ -class CustomEnumMeta(type): - def __new__(mcs, name: str, bases: tuple, attrs: dict): - _attrs, _enum = {}, {} - for base in bases: - if isinstance(base, mcs): - _enum.update(getattr(base, "_enum", {})) - - for key, value in attrs.items(): - if key[:2] == key[-2:] == "__" or hasattr(value, "__get__"): - _attrs[key] = value - else: - _enum[key] = value - - _attrs["_enum"] = _enum - - return super().__new__(mcs, name, bases, _attrs) - - def __contains__(cls, name): - return name in cls._enum.values() - - def __getattr__(cls, name): - try: - return cls._enum[name] - except KeyError as e: - raise AttributeError(name) from e - - -class CustomEnum(metaclass=CustomEnumMeta): - pass diff --git a/siem-converter/app/converter/tools/decorators.py b/siem-converter/app/converter/tools/decorators.py deleted file mode 100644 index d917563d..00000000 --- a/siem-converter/app/converter/tools/decorators.py +++ /dev/null @@ -1,23 +0,0 @@ -from app.converter.core.exceptions.core import BasePlatformException -from app.converter.core.exceptions.iocs import BaseIOCsException -from app.converter.core.exceptions.parser import BaseParserException -from app.converter.core.exceptions.render import BaseRenderException - - -def handle_translation_exceptions(func): - def exception_handler(*args, **kwargs): - try: - result = func(*args, **kwargs) - if result: - print("Translated successfully.") - return True, result - else: - print("Unexpected error.") - return False, f"Unexpected error. To resolve it, please, contact us via GitHub." - except (BaseParserException, BasePlatformException, BaseRenderException, BaseIOCsException) as err: - print(f"Unexpected error. {str(err)}") - return False, str(err) - except Exception as err: - print(f"Unexpected error. {str(err)}") - return False, f"Unexpected error. To resolve it, please, contact us via GitHub." - return exception_handler diff --git a/siem-converter/app/converter/tools/utils.py b/siem-converter/app/converter/tools/utils.py deleted file mode 100644 index ae632a4b..00000000 --- a/siem-converter/app/converter/tools/utils.py +++ /dev/null @@ -1,33 +0,0 @@ -import re -from typing import Optional, List - - -def get_match_group(match: re.Match, group_name: str) -> Optional[str]: - try: - return match.group(group_name) - except IndexError: - return - - -def concatenate_str(str1: str, str2: str) -> str: - return str1 + " " + str2 if str1 else str2 - - -def get_mitre_attack_str(mitre_attack: List[str]) -> str: - return f"MITRE ATT&CK: {', '.join(mitre_attack).upper()}." if mitre_attack else "" - - -def get_author_str(author: str) -> str: - return f"Author: {author}." if author else "" - - -def get_licence_str(licence: str) -> str: - return f"Licence: {licence}." - - -def get_rule_id_str(rule_id: str) -> str: - return f"Rule ID: {rule_id}." - - -def get_references_str(references: List[str]) -> str: - return f"References: {', '.join(references)}." if references else "" diff --git a/siem-converter/app/models/__init__.py b/siem-converter/app/models/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/routers/__init__.py b/siem-converter/app/routers/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/siem-converter/app/routers/assistance.py b/siem-converter/app/routers/assistance.py deleted file mode 100644 index 35078e97..00000000 --- a/siem-converter/app/routers/assistance.py +++ /dev/null @@ -1,44 +0,0 @@ -import datetime -import json -from datetime import datetime -from typing import List, Dict -import os - -from contextlib import asynccontextmanager - -from fastapi import APIRouter, FastAPI - -from const import ROOT_PROJECT_PATH - - -assistance_router = APIRouter() - -suggestions = {} - - -@asynccontextmanager -async def lifespan(app: FastAPI): - with open(os.path.join(ROOT_PROJECT_PATH, 'app/dictionaries/uncoder_meta_info_roota.json'), 'r') as file: - json_f = json.load(file) - suggestions['roota'] = json_f - with open(os.path.join(ROOT_PROJECT_PATH, 'app/dictionaries/uncoder_meta_info_sigma.json'), 'r') as file: - json_f = json.load(file) - suggestions['sigma'] = json_f - yield - - -@assistance_router.get( - '/suggestions/{parser_id}', - tags=["assistance"], - description="Get suggestions" -) -async def get_suggestions(parser_id: str) -> List[Dict]: - parser_dict = suggestions.get(parser_id, []) - if parser_id == 'roota': - today = datetime.today().strftime('%Y-%m-%d') - for i in parser_dict: - if i['title'] == 'Date': - for v in i['dictionary']: - v['name'] = today - return parser_dict - return parser_dict diff --git a/siem-converter/app/routers/ioc_translate.py b/siem-converter/app/routers/ioc_translate.py deleted file mode 100644 index d7f151fa..00000000 --- a/siem-converter/app/routers/ioc_translate.py +++ /dev/null @@ -1,46 +0,0 @@ -from typing import Optional, List -from fastapi import APIRouter, Body - -from app.converter.tools.const import IOCType, HashType, IocParsingRule -from app.converter.cti_converter import CTIConverter -from app.models.ioc_translation import CTIPlatform, OneTranslationCTIData -from app.models.translation import InfoMessage - -iocs_router = APIRouter() -converter = CTIConverter() - - -@iocs_router.post( - "/iocs/translate", - description="Parse IOCs from text.", -) -@iocs_router.post("/iocs/translate", include_in_schema=False) -def parse_and_translate_iocs( - text: str = Body(..., description="Text to parse IOCs from", embed=True), - iocs_per_query: int = Body(25, description="Platforms to parse IOCs to", embed=True), - platform: CTIPlatform = Body(..., description="Platforms to parse IOCs to", embed=True), - include_ioc_types: Optional[List[IOCType]] = Body( - None, description="List of IOC types to include. By default all types are enabled.", embed=True), - include_hash_types: Optional[List[HashType]] = Body( - None, description="List of hash types to include. By default all hash types are enabled.", embed=True), - exceptions: Optional[List[str]] = Body( - None, description="List of exceptions. IOC is ignored if it contains one of exception values.", embed=True), - ioc_parsing_rules: Optional[List[IocParsingRule]] = Body( - None, embed=True, description="Additional parsing parameters."), - include_source_ip: Optional[bool] = Body( - False, description="Include source IP in query. By default it is false." - ) -) -> OneTranslationCTIData: - status, translations = converter.convert(text=text, - platform_data=platform, - iocs_per_query=iocs_per_query, - include_ioc_types=include_ioc_types, - include_hash_types=include_hash_types, - exceptions=exceptions, - ioc_parsing_rules=ioc_parsing_rules, - include_source_ip=include_source_ip) - if status: - return OneTranslationCTIData(status=status, translations=translations, target_siem_type=platform.name) - else: - info_message = InfoMessage(message=translations, severity="error") - return OneTranslationCTIData(info=info_message, status=status, target_siem_type=platform.name) diff --git a/siem-converter/app/routers/translate.py b/siem-converter/app/routers/translate.py deleted file mode 100644 index 0fd21f6f..00000000 --- a/siem-converter/app/routers/translate.py +++ /dev/null @@ -1,101 +0,0 @@ -from fastapi import APIRouter, Body - -from app.converter.converter import SiemConverter -from app.converter.cti_converter import CTIConverter -from app.models.translation import OneTranslationData, ConvertorPlatforms, Platform, InfoMessage - -st_router = APIRouter() - -converter = SiemConverter() - - -@st_router.post( - "/translate", - tags=["siem_translate"], - description="Generate target translation", -) -@st_router.post("/translate/", include_in_schema=False) -def generate_one_translation( - source_siem: str = Body(..., embed=True), - source_scheme: str = Body(None, embed=True), - target_siem: str = Body(..., embed=True), - target_scheme: str = Body(None, embed=True), - text: str = Body(..., embed=True), - -) -> OneTranslationData: - status, data = converter.generate_translation( - text=text, - source=source_siem, - target=target_siem - ) - if status: - return OneTranslationData( - status=status, - translation=data, - target_siem_type=target_siem) - else: - info_message = InfoMessage(message=data, severity="error") - return OneTranslationData( - info=info_message, - status=status, - target_siem_type=target_siem) - - -@st_router.post( - "/translate/all", - tags=["siem_translate"], - description="Generate all translations", -) -@st_router.post("/translate/all/", include_in_schema=False) -def generate_all_translations( - source_siem: str = Body(..., embed=True), - source_scheme: str = Body(None, embed=True), - text: str = Body(..., embed=True), -) -> list[OneTranslationData]: - result = converter.generate_all_translation( - text=text, - source=source_siem - ) - translations = [] - for siem_result in result: - if siem_result.get("status"): - translations.append(OneTranslationData( - status=siem_result.get("status", True), - translation=siem_result.get("result"), - target_siem_type=siem_result.get("siem_type")) - ) - else: - translations.append(OneTranslationData( - status=siem_result.get("status", False), - info=InfoMessage(message=siem_result.get("result"), severity="error"), - target_siem_type=siem_result.get("siem_type")) - ) - return translations - - -@st_router.get( - "/platforms", - tags=["siem_translate"], - description="Get converter platforms", -) -@st_router.get("/platforms/", include_in_schema=False) -def get_convertor_platforms() -> ConvertorPlatforms: - renders, parsers = converter.get_all_platforms() - return ConvertorPlatforms(renders=renders, parsers=parsers) - - -@st_router.get( - "/all_platforms", - description="Get Sigma, RootA and iocs platforms", -) -@st_router.get("/all_platforms/", include_in_schema=False) -def get_all_platforms() -> list: - converter_renders, converter_platforms = converter.get_all_platforms() - return [ - Platform(id="roota", name="RootA", code="roota", group_name="RootA", group_id="roota", - renders=converter_renders, parsers=converter_platforms), - Platform(id="sigma", name="Sigma", code="sigma", group_name="Sigma", group_id="sigma", - renders=[render for render in converter_renders if render.code != "sigma"]), - Platform(id="ioc", name="IOCs", code="ioc", group_name="IOCs", group_id="ioc", - renders=CTIConverter().get_renders()) - ] diff --git a/siem-converter/const.py b/siem-converter/const.py deleted file mode 100644 index 024c8f39..00000000 --- a/siem-converter/const.py +++ /dev/null @@ -1,3 +0,0 @@ -import os - -ROOT_PROJECT_PATH = os.path.abspath(os.path.dirname(__file__)) diff --git a/siem-converter/requirements.txt b/siem-converter/requirements.txt deleted file mode 100644 index 56522342..00000000 --- a/siem-converter/requirements.txt +++ /dev/null @@ -1,5 +0,0 @@ -fastapi>=0.95.0,<0.96.0 -uvicorn[standard]>=0.21.0,<0.22.0 -pydantic~=1.10.13 -PyYAML~=6.0.1 -colorama~=0.4.6 \ No newline at end of file diff --git a/siem-converter/.gitignore b/uncoder-core/.gitignore similarity index 98% rename from siem-converter/.gitignore rename to uncoder-core/.gitignore index a887c9dc..3ec6e06e 100644 --- a/siem-converter/.gitignore +++ b/uncoder-core/.gitignore @@ -197,7 +197,3 @@ test_data # backup_logs .backup_logs - -# sigmac tests stuff -tactics.json -techniques.json \ No newline at end of file diff --git a/siem-converter/Dockerfile b/uncoder-core/Dockerfile similarity index 100% rename from siem-converter/Dockerfile rename to uncoder-core/Dockerfile diff --git a/siem-converter/app/__init__.py b/uncoder-core/app/__init__.py similarity index 100% rename from siem-converter/app/__init__.py rename to uncoder-core/app/__init__.py diff --git a/uncoder-core/app/dictionaries/tactics.json b/uncoder-core/app/dictionaries/tactics.json new file mode 100644 index 00000000..7c20e614 --- /dev/null +++ b/uncoder-core/app/dictionaries/tactics.json @@ -0,0 +1,72 @@ +{ + "credential_access": { + "external_id": "TA0006", + "url": "https://attack.mitre.org/tactics/TA0006", + "tactic": "Credential Access" + }, + "execution": { + "external_id": "TA0002", + "url": "https://attack.mitre.org/tactics/TA0002", + "tactic": "Execution" + }, + "impact": { + "external_id": "TA0040", + "url": "https://attack.mitre.org/tactics/TA0040", + "tactic": "Impact" + }, + "persistence": { + "external_id": "TA0003", + "url": "https://attack.mitre.org/tactics/TA0003", + "tactic": "Persistence" + }, + "privilege_escalation": { + "external_id": "TA0004", + "url": "https://attack.mitre.org/tactics/TA0004", + "tactic": "Privilege Escalation" + }, + "lateral_movement": { + "external_id": "TA0008", + "url": "https://attack.mitre.org/tactics/TA0008", + "tactic": "Lateral Movement" + }, + "defense_evasion": { + "external_id": "TA0005", + "url": "https://attack.mitre.org/tactics/TA0005", + "tactic": "Defense Evasion" + }, + "exfiltration": { + "external_id": "TA0010", + "url": "https://attack.mitre.org/tactics/TA0010", + "tactic": "Exfiltration" + }, + "discovery": { + "external_id": "TA0007", + "url": "https://attack.mitre.org/tactics/TA0007", + "tactic": "Discovery" + }, + "collection": { + "external_id": "TA0009", + "url": "https://attack.mitre.org/tactics/TA0009", + "tactic": "Collection" + }, + "resource_development": { + "external_id": "TA0042", + "url": "https://attack.mitre.org/tactics/TA0042", + "tactic": "Resource Development" + }, + "reconnaissance": { + "external_id": "TA0043", + "url": "https://attack.mitre.org/tactics/TA0043", + "tactic": "Reconnaissance" + }, + "command_and_control": { + "external_id": "TA0011", + "url": "https://attack.mitre.org/tactics/TA0011", + "tactic": "Command and Control" + }, + "initial_access": { + "external_id": "TA0001", + "url": "https://attack.mitre.org/tactics/TA0001", + "tactic": "Initial Access" + } +} \ No newline at end of file diff --git a/uncoder-core/app/dictionaries/techniques.json b/uncoder-core/app/dictionaries/techniques.json new file mode 100644 index 00000000..83323af3 --- /dev/null +++ b/uncoder-core/app/dictionaries/techniques.json @@ -0,0 +1,3763 @@ +{ + "t1047": { + "technique_id": "T1047", + "technique": "Windows Management Instrumentation", + "url": "https://attack.mitre.org/techniques/T1047", + "tactic": [ + "Execution" + ] + }, + "t1113": { + "technique_id": "T1113", + "technique": "Screen Capture", + "url": "https://attack.mitre.org/techniques/T1113", + "tactic": [ + "Collection" + ] + }, + "t1037": { + "technique_id": "T1037", + "technique": "Boot or Logon Initialization Scripts", + "url": "https://attack.mitre.org/techniques/T1037", + "tactic": [ + "Persistence", + "Privilege Escalation" + ] + }, + "t1557": { + "technique_id": "T1557", + "technique": "Adversary-in-the-Middle", + "url": "https://attack.mitre.org/techniques/T1557", + "tactic": [ + "Credential Access", + "Collection" + ] + }, + "t1033": { + "technique_id": "T1033", + "technique": "System Owner/User Discovery", + "url": "https://attack.mitre.org/techniques/T1033", + "tactic": [ + "Discovery" + ] + }, + "t1583": { + "technique_id": "T1583", + "technique": "Acquire Infrastructure", + "url": "https://attack.mitre.org/techniques/T1583", + "tactic": [ + "Resource Development" + ] + }, + "t1613": { + "technique_id": "T1613", + "technique": "Container and Resource Discovery", + "url": "https://attack.mitre.org/techniques/T1613", + "tactic": [ + "Discovery" + ] + }, + "t1592": { + "technique_id": "T1592", + "technique": "Gather Victim Host Information", + "url": "https://attack.mitre.org/techniques/T1592", + "tactic": [ + "Reconnaissance" + ] + }, + "t1003": { + "technique_id": "T1003", + "technique": "OS Credential Dumping", + "url": "https://attack.mitre.org/techniques/T1003", + "tactic": [ + "Credential Access" + ] + }, + "t1129": { + "technique_id": "T1129", + "technique": "Shared Modules", + "url": "https://attack.mitre.org/techniques/T1129", + "tactic": [ + "Execution" + ] + }, + "t1602": { + "technique_id": "T1602", + "technique": "Data from Configuration Repository", + "url": "https://attack.mitre.org/techniques/T1602", + "tactic": [ + "Collection" + ] + }, + "t1006": { + "technique_id": "T1006", + "technique": "Direct Volume Access", + "url": "https://attack.mitre.org/techniques/T1006", + "tactic": [ + "Defense Evasion" + ] + }, + "t1014": { + "technique_id": "T1014", + "technique": "Rootkit", + "url": "https://attack.mitre.org/techniques/T1014", + "tactic": [ + "Defense Evasion" + ] + }, + "t1123": { + "technique_id": "T1123", + "technique": "Audio Capture", + "url": "https://attack.mitre.org/techniques/T1123", + "tactic": [ + "Collection" + ] + }, + "t1543": { + "technique_id": "T1543", + "technique": "Create or Modify System Process", + "url": "https://attack.mitre.org/techniques/T1543", + "tactic": [ + "Persistence", + "Privilege Escalation" + ] + }, + "t1133": { + "technique_id": "T1133", + "technique": "External Remote Services", + "url": "https://attack.mitre.org/techniques/T1133", + "tactic": [ + "Persistence", + "Initial Access" + ] + }, + "t1539": { + "technique_id": "T1539", + "technique": "Steal Web Session Cookie", + "url": "https://attack.mitre.org/techniques/T1539", + "tactic": [ + "Credential Access" + ] + }, + "t1578": { + "technique_id": "T1578", + "technique": "Modify Cloud Compute Infrastructure", + "url": "https://attack.mitre.org/techniques/T1578", + "tactic": [ + "Defense Evasion" + ] + }, + "t1069": { + "technique_id": "T1069", + "technique": "Permission Groups Discovery", + "url": "https://attack.mitre.org/techniques/T1069", + "tactic": [ + "Discovery" + ] + }, + "t1114": { + "technique_id": "T1114", + "technique": "Email Collection", + "url": "https://attack.mitre.org/techniques/T1114", + "tactic": [ + "Collection" + ] + }, + "t1594": { + "technique_id": "T1594", + "technique": "Search Victim-Owned Websites", + "url": "https://attack.mitre.org/techniques/T1594", + "tactic": [ + "Reconnaissance" + ] + }, + "t1561": { + "technique_id": "T1561", + "technique": "Disk Wipe", + "url": "https://attack.mitre.org/techniques/T1561", + "tactic": [ + "Impact" + ] + }, + "t1615": { + "technique_id": "T1615", + "technique": "Group Policy Discovery", + "url": "https://attack.mitre.org/techniques/T1615", + "tactic": [ + "Discovery" + ] + }, + "t1025": { + "technique_id": "T1025", + "technique": "Data from Removable Media", + "url": "https://attack.mitre.org/techniques/T1025", + "tactic": [ + "Collection" + ] + }, + "t1547": { + "technique_id": "T1547", + "technique": "Boot or Logon Autostart Execution", + "url": "https://attack.mitre.org/techniques/T1547", + "tactic": [ + "Persistence", + "Privilege Escalation" + ] + }, + "t1600": { + "technique_id": "T1600", + "technique": "Weaken Encryption", + "url": "https://attack.mitre.org/techniques/T1600", + "tactic": [ + "Defense Evasion" + ] + }, + "t1489": { + "technique_id": "T1489", + "technique": "Service Stop", + "url": "https://attack.mitre.org/techniques/T1489", + "tactic": [ + "Impact" + ] + }, + "t1652": { + "technique_id": "T1652", + "technique": "Device Driver Discovery", + "url": "https://attack.mitre.org/techniques/T1652", + "tactic": [ + "Discovery" + ] + }, + "t1564": { + "technique_id": "T1564", + "technique": "Hide Artifacts", + "url": "https://attack.mitre.org/techniques/T1564", + "tactic": [ + "Defense Evasion" + ] + }, + "t1080": { + "technique_id": "T1080", + "technique": "Taint Shared Content", + "url": "https://attack.mitre.org/techniques/T1080", + "tactic": [ + "Lateral Movement" + ] + }, + "t1137": { + "technique_id": "T1137", + "technique": "Office Application Startup", + "url": "https://attack.mitre.org/techniques/T1137", + "tactic": [ + "Persistence" + ] + }, + "t1119": { + "technique_id": "T1119", + "technique": "Automated Collection", + "url": "https://attack.mitre.org/techniques/T1119", + "tactic": [ + "Collection" + ] + }, + "t1115": { + "technique_id": "T1115", + "technique": "Clipboard Data", + "url": "https://attack.mitre.org/techniques/T1115", + "tactic": [ + "Collection" + ] + }, + "t1007": { + "technique_id": "T1007", + "technique": "System Service Discovery", + "url": "https://attack.mitre.org/techniques/T1007", + "tactic": [ + "Discovery" + ] + }, + "t1040": { + "technique_id": "T1040", + "technique": "Network Sniffing", + "url": "https://attack.mitre.org/techniques/T1040", + "tactic": [ + "Credential Access", + "Discovery" + ] + }, + "t1530": { + "technique_id": "T1530", + "technique": "Data from Cloud Storage", + "url": "https://attack.mitre.org/techniques/T1530", + "tactic": [ + "Collection" + ] + }, + "t1135": { + "technique_id": "T1135", + "technique": "Network Share Discovery", + "url": "https://attack.mitre.org/techniques/T1135", + "tactic": [ + "Discovery" + ] + }, + "t1120": { + "technique_id": "T1120", + "technique": "Peripheral Device Discovery", + "url": "https://attack.mitre.org/techniques/T1120", + "tactic": [ + "Discovery" + ] + }, + "t1082": { + "technique_id": "T1082", + "technique": "System Information Discovery", + "url": "https://attack.mitre.org/techniques/T1082", + "tactic": [ + "Discovery" + ] + }, + "t1071": { + "technique_id": "T1071", + "technique": "Application Layer Protocol", + "url": "https://attack.mitre.org/techniques/T1071", + "tactic": [ + "Command and Control" + ] + }, + "t1053": { + "technique_id": "T1053", + "technique": "Scheduled Task/Job", + "url": "https://attack.mitre.org/techniques/T1053", + "tactic": [ + "Execution", + "Persistence", + "Privilege Escalation" + ] + }, + "t1176": { + "technique_id": "T1176", + "technique": "Browser Extensions", + "url": "https://attack.mitre.org/techniques/T1176", + "tactic": [ + "Persistence" + ] + }, + "t1106": { + "technique_id": "T1106", + "technique": "Native API", + "url": "https://attack.mitre.org/techniques/T1106", + "tactic": [ + "Execution" + ] + }, + "t1202": { + "technique_id": "T1202", + "technique": "Indirect Command Execution", + "url": "https://attack.mitre.org/techniques/T1202", + "tactic": [ + "Defense Evasion" + ] + }, + "t1091": { + "technique_id": "T1091", + "technique": "Replication Through Removable Media", + "url": "https://attack.mitre.org/techniques/T1091", + "tactic": [ + "Lateral Movement", + "Initial Access" + ] + }, + "t1005": { + "technique_id": "T1005", + "technique": "Data from Local System", + "url": "https://attack.mitre.org/techniques/T1005", + "tactic": [ + "Collection" + ] + }, + "t1140": { + "technique_id": "T1140", + "technique": "Deobfuscate/Decode Files or Information", + "url": "https://attack.mitre.org/techniques/T1140", + "tactic": [ + "Defense Evasion" + ] + }, + "t1562": { + "technique_id": "T1562", + "technique": "Impair Defenses", + "url": "https://attack.mitre.org/techniques/T1562", + "tactic": [ + "Defense Evasion" + ] + }, + "t1195": { + "technique_id": "T1195", + "technique": "Supply Chain Compromise", + "url": "https://attack.mitre.org/techniques/T1195", + "tactic": [ + "Initial Access" + ] + }, + "t1190": { + "technique_id": "T1190", + "technique": "Exploit Public-Facing Application", + "url": "https://attack.mitre.org/techniques/T1190", + "tactic": [ + "Initial Access" + ] + }, + "t1558": { + "technique_id": "T1558", + "technique": "Steal or Forge Kerberos Tickets", + "url": "https://attack.mitre.org/techniques/T1558", + "tactic": [ + "Credential Access" + ] + }, + "t1555": { + "technique_id": "T1555", + "technique": "Credentials from Password Stores", + "url": "https://attack.mitre.org/techniques/T1555", + "tactic": [ + "Credential Access" + ] + }, + "t1567": { + "technique_id": "T1567", + "technique": "Exfiltration Over Web Service", + "url": "https://attack.mitre.org/techniques/T1567", + "tactic": [ + "Exfiltration" + ] + }, + "t1219": { + "technique_id": "T1219", + "technique": "Remote Access Software", + "url": "https://attack.mitre.org/techniques/T1219", + "tactic": [ + "Command and Control" + ] + }, + "t1036": { + "technique_id": "T1036", + "technique": "Masquerading", + "url": "https://attack.mitre.org/techniques/T1036", + "tactic": [ + "Defense Evasion" + ] + }, + "t1552": { + "technique_id": "T1552", + "technique": "Unsecured Credentials", + "url": "https://attack.mitre.org/techniques/T1552", + "tactic": [ + "Credential Access" + ] + }, + "t1659": { + "technique_id": "T1659", + "technique": "Content Injection", + "url": "https://attack.mitre.org/techniques/T1659", + "tactic": [ + "Initial Access", + "Command and Control" + ] + }, + "t1055": { + "technique_id": "T1055", + "technique": "Process Injection", + "url": "https://attack.mitre.org/techniques/T1055", + "tactic": [ + "Defense Evasion", + "Privilege Escalation" + ] + }, + "t1205": { + "technique_id": "T1205", + "technique": "Traffic Signaling", + "url": "https://attack.mitre.org/techniques/T1205", + "tactic": [ + "Defense Evasion", + "Persistence", + "Command and Control" + ] + }, + "t1218": { + "technique_id": "T1218", + "technique": "System Binary Proxy Execution", + "url": "https://attack.mitre.org/techniques/T1218", + "tactic": [ + "Defense Evasion" + ] + }, + "t1620": { + "technique_id": "T1620", + "technique": "Reflective Code Loading", + "url": "https://attack.mitre.org/techniques/T1620", + "tactic": [ + "Defense Evasion" + ] + }, + "t1611": { + "technique_id": "T1611", + "technique": "Escape to Host", + "url": "https://attack.mitre.org/techniques/T1611", + "tactic": [ + "Privilege Escalation" + ] + }, + "t1010": { + "technique_id": "T1010", + "technique": "Application Window Discovery", + "url": "https://attack.mitre.org/techniques/T1010", + "tactic": [ + "Discovery" + ] + }, + "t1029": { + "technique_id": "T1029", + "technique": "Scheduled Transfer", + "url": "https://attack.mitre.org/techniques/T1029", + "tactic": [ + "Exfiltration" + ] + }, + "t1525": { + "technique_id": "T1525", + "technique": "Implant Internal Image", + "url": "https://attack.mitre.org/techniques/T1525", + "tactic": [ + "Persistence" + ] + }, + "t1572": { + "technique_id": "T1572", + "technique": "Protocol Tunneling", + "url": "https://attack.mitre.org/techniques/T1572", + "tactic": [ + "Command and Control" + ] + }, + "t1550": { + "technique_id": "T1550", + "technique": "Use Alternate Authentication Material", + "url": "https://attack.mitre.org/techniques/T1550", + "tactic": [ + "Defense Evasion", + "Lateral Movement" + ] + }, + "t1011": { + "technique_id": "T1011", + "technique": "Exfiltration Over Other Network Medium", + "url": "https://attack.mitre.org/techniques/T1011", + "tactic": [ + "Exfiltration" + ] + }, + "t1589": { + "technique_id": "T1589", + "technique": "Gather Victim Identity Information", + "url": "https://attack.mitre.org/techniques/T1589", + "tactic": [ + "Reconnaissance" + ] + }, + "t1560": { + "technique_id": "T1560", + "technique": "Archive Collected Data", + "url": "https://attack.mitre.org/techniques/T1560", + "tactic": [ + "Collection" + ] + }, + "t1185": { + "technique_id": "T1185", + "technique": "Browser Session Hijacking", + "url": "https://attack.mitre.org/techniques/T1185", + "tactic": [ + "Collection" + ] + }, + "t1021": { + "technique_id": "T1021", + "technique": "Remote Services", + "url": "https://attack.mitre.org/techniques/T1021", + "tactic": [ + "Lateral Movement" + ] + }, + "t1596": { + "technique_id": "T1596", + "technique": "Search Open Technical Databases", + "url": "https://attack.mitre.org/techniques/T1596", + "tactic": [ + "Reconnaissance" + ] + }, + "t1207": { + "technique_id": "T1207", + "technique": "Rogue Domain Controller", + "url": "https://attack.mitre.org/techniques/T1207", + "tactic": [ + "Defense Evasion" + ] + }, + "t1610": { + "technique_id": "T1610", + "technique": "Deploy Container", + "url": "https://attack.mitre.org/techniques/T1610", + "tactic": [ + "Defense Evasion", + "Execution" + ] + }, + "t1112": { + "technique_id": "T1112", + "technique": "Modify Registry", + "url": "https://attack.mitre.org/techniques/T1112", + "tactic": [ + "Defense Evasion" + ] + }, + "t1580": { + "technique_id": "T1580", + "technique": "Cloud Infrastructure Discovery", + "url": "https://attack.mitre.org/techniques/T1580", + "tactic": [ + "Discovery" + ] + }, + "t1491": { + "technique_id": "T1491", + "technique": "Defacement", + "url": "https://attack.mitre.org/techniques/T1491", + "tactic": [ + "Impact" + ] + }, + "t1535": { + "technique_id": "T1535", + "technique": "Unused/Unsupported Cloud Regions", + "url": "https://attack.mitre.org/techniques/T1535", + "tactic": [ + "Defense Evasion" + ] + }, + "t1563": { + "technique_id": "T1563", + "technique": "Remote Service Session Hijacking", + "url": "https://attack.mitre.org/techniques/T1563", + "tactic": [ + "Lateral Movement" + ] + }, + "t1217": { + "technique_id": "T1217", + "technique": "Browser Information Discovery", + "url": "https://attack.mitre.org/techniques/T1217", + "tactic": [ + "Discovery" + ] + }, + "t1092": { + "technique_id": "T1092", + "technique": "Communication Through Removable Media", + "url": "https://attack.mitre.org/techniques/T1092", + "tactic": [ + "Command and Control" + ] + }, + "t1222": { + "technique_id": "T1222", + "technique": "File and Directory Permissions Modification", + "url": "https://attack.mitre.org/techniques/T1222", + "tactic": [ + "Defense Evasion" + ] + }, + "t1595": { + "technique_id": "T1595", + "technique": "Active Scanning", + "url": "https://attack.mitre.org/techniques/T1595", + "tactic": [ + "Reconnaissance" + ] + }, + "t1548": { + "technique_id": "T1548", + "technique": "Abuse Elevation Control Mechanism", + "url": "https://attack.mitre.org/techniques/T1548", + "tactic": [ + "Privilege Escalation", + "Defense Evasion" + ] + }, + "t1125": { + "technique_id": "T1125", + "technique": "Video Capture", + "url": "https://attack.mitre.org/techniques/T1125", + "tactic": [ + "Collection" + ] + }, + "t1016": { + "technique_id": "T1016", + "technique": "System Network Configuration Discovery", + "url": "https://attack.mitre.org/techniques/T1016", + "tactic": [ + "Discovery" + ] + }, + "t1087": { + "technique_id": "T1087", + "technique": "Account Discovery", + "url": "https://attack.mitre.org/techniques/T1087", + "tactic": [ + "Discovery" + ] + }, + "t1090": { + "technique_id": "T1090", + "technique": "Proxy", + "url": "https://attack.mitre.org/techniques/T1090", + "tactic": [ + "Command and Control" + ] + }, + "t1059": { + "technique_id": "T1059", + "technique": "Command and Scripting Interpreter", + "url": "https://attack.mitre.org/techniques/T1059", + "tactic": [ + "Execution" + ] + }, + "t1482": { + "technique_id": "T1482", + "technique": "Domain Trust Discovery", + "url": "https://attack.mitre.org/techniques/T1482", + "tactic": [ + "Discovery" + ] + }, + "t1020": { + "technique_id": "T1020", + "technique": "Automated Exfiltration", + "url": "https://attack.mitre.org/techniques/T1020", + "tactic": [ + "Exfiltration" + ] + }, + "t1070": { + "technique_id": "T1070", + "technique": "Indicator Removal", + "url": "https://attack.mitre.org/techniques/T1070", + "tactic": [ + "Defense Evasion" + ] + }, + "t1609": { + "technique_id": "T1609", + "technique": "Container Administration Command", + "url": "https://attack.mitre.org/techniques/T1609", + "tactic": [ + "Execution" + ] + }, + "t1083": { + "technique_id": "T1083", + "technique": "File and Directory Discovery", + "url": "https://attack.mitre.org/techniques/T1083", + "tactic": [ + "Discovery" + ] + }, + "t1568": { + "technique_id": "T1568", + "technique": "Dynamic Resolution", + "url": "https://attack.mitre.org/techniques/T1568", + "tactic": [ + "Command and Control" + ] + }, + "t1647": { + "technique_id": "T1647", + "technique": "Plist File Modification", + "url": "https://attack.mitre.org/techniques/T1647", + "tactic": [ + "Defense Evasion" + ] + }, + "t1074": { + "technique_id": "T1074", + "technique": "Data Staged", + "url": "https://attack.mitre.org/techniques/T1074", + "tactic": [ + "Collection" + ] + }, + "t1649": { + "technique_id": "T1649", + "technique": "Steal or Forge Authentication Certificates", + "url": "https://attack.mitre.org/techniques/T1649", + "tactic": [ + "Credential Access" + ] + }, + "t1049": { + "technique_id": "T1049", + "technique": "System Network Connections Discovery", + "url": "https://attack.mitre.org/techniques/T1049", + "tactic": [ + "Discovery" + ] + }, + "t1584": { + "technique_id": "T1584", + "technique": "Compromise Infrastructure", + "url": "https://attack.mitre.org/techniques/T1584", + "tactic": [ + "Resource Development" + ] + }, + "t1542": { + "technique_id": "T1542", + "technique": "Pre-OS Boot", + "url": "https://attack.mitre.org/techniques/T1542", + "tactic": [ + "Defense Evasion", + "Persistence" + ] + }, + "t1612": { + "technique_id": "T1612", + "technique": "Build Image on Host", + "url": "https://attack.mitre.org/techniques/T1612", + "tactic": [ + "Defense Evasion" + ] + }, + "t1586": { + "technique_id": "T1586", + "technique": "Compromise Accounts", + "url": "https://attack.mitre.org/techniques/T1586", + "tactic": [ + "Resource Development" + ] + }, + "t1497": { + "technique_id": "T1497", + "technique": "Virtualization/Sandbox Evasion", + "url": "https://attack.mitre.org/techniques/T1497", + "tactic": [ + "Defense Evasion", + "Discovery" + ] + }, + "t1102": { + "technique_id": "T1102", + "technique": "Web Service", + "url": "https://attack.mitre.org/techniques/T1102", + "tactic": [ + "Command and Control" + ] + }, + "t1608": { + "technique_id": "T1608", + "technique": "Stage Capabilities", + "url": "https://attack.mitre.org/techniques/T1608", + "tactic": [ + "Resource Development" + ] + }, + "t1104": { + "technique_id": "T1104", + "technique": "Multi-Stage Channels", + "url": "https://attack.mitre.org/techniques/T1104", + "tactic": [ + "Command and Control" + ] + }, + "t1657": { + "technique_id": "T1657", + "technique": "Financial Theft", + "url": "https://attack.mitre.org/techniques/T1657", + "tactic": [ + "Impact" + ] + }, + "t1480": { + "technique_id": "T1480", + "technique": "Execution Guardrails", + "url": "https://attack.mitre.org/techniques/T1480", + "tactic": [ + "Defense Evasion" + ] + }, + "t1619": { + "technique_id": "T1619", + "technique": "Cloud Storage Object Discovery", + "url": "https://attack.mitre.org/techniques/T1619", + "tactic": [ + "Discovery" + ] + }, + "t1654": { + "technique_id": "T1654", + "technique": "Log Enumeration", + "url": "https://attack.mitre.org/techniques/T1654", + "tactic": [ + "Discovery" + ] + }, + "t1528": { + "technique_id": "T1528", + "technique": "Steal Application Access Token", + "url": "https://attack.mitre.org/techniques/T1528", + "tactic": [ + "Credential Access" + ] + }, + "t1204": { + "technique_id": "T1204", + "technique": "User Execution", + "url": "https://attack.mitre.org/techniques/T1204", + "tactic": [ + "Execution" + ] + }, + "t1057": { + "technique_id": "T1057", + "technique": "Process Discovery", + "url": "https://attack.mitre.org/techniques/T1057", + "tactic": [ + "Discovery" + ] + }, + "t1072": { + "technique_id": "T1072", + "technique": "Software Deployment Tools", + "url": "https://attack.mitre.org/techniques/T1072", + "tactic": [ + "Execution", + "Lateral Movement" + ] + }, + "t1041": { + "technique_id": "T1041", + "technique": "Exfiltration Over C2 Channel", + "url": "https://attack.mitre.org/techniques/T1041", + "tactic": [ + "Exfiltration" + ] + }, + "t1591": { + "technique_id": "T1591", + "technique": "Gather Victim Org Information", + "url": "https://attack.mitre.org/techniques/T1591", + "tactic": [ + "Reconnaissance" + ] + }, + "t1606": { + "technique_id": "T1606", + "technique": "Forge Web Credentials", + "url": "https://attack.mitre.org/techniques/T1606", + "tactic": [ + "Credential Access" + ] + }, + "t1621": { + "technique_id": "T1621", + "technique": "Multi-Factor Authentication Request Generation", + "url": "https://attack.mitre.org/techniques/T1621", + "tactic": [ + "Credential Access" + ] + }, + "t1554": { + "technique_id": "T1554", + "technique": "Compromise Client Software Binary", + "url": "https://attack.mitre.org/techniques/T1554", + "tactic": [ + "Persistence" + ] + }, + "t1212": { + "technique_id": "T1212", + "technique": "Exploitation for Credential Access", + "url": "https://attack.mitre.org/techniques/T1212", + "tactic": [ + "Credential Access" + ] + }, + "t1590": { + "technique_id": "T1590", + "technique": "Gather Victim Network Information", + "url": "https://attack.mitre.org/techniques/T1590", + "tactic": [ + "Reconnaissance" + ] + }, + "t1210": { + "technique_id": "T1210", + "technique": "Exploitation of Remote Services", + "url": "https://attack.mitre.org/techniques/T1210", + "tactic": [ + "Lateral Movement" + ] + }, + "t1534": { + "technique_id": "T1534", + "technique": "Internal Spearphishing", + "url": "https://attack.mitre.org/techniques/T1534", + "tactic": [ + "Lateral Movement" + ] + }, + "t1199": { + "technique_id": "T1199", + "technique": "Trusted Relationship", + "url": "https://attack.mitre.org/techniques/T1199", + "tactic": [ + "Initial Access" + ] + }, + "t1593": { + "technique_id": "T1593", + "technique": "Search Open Websites/Domains", + "url": "https://attack.mitre.org/techniques/T1593", + "tactic": [ + "Reconnaissance" + ] + }, + "t1098": { + "technique_id": "T1098", + "technique": "Account Manipulation", + "url": "https://attack.mitre.org/techniques/T1098", + "tactic": [ + "Persistence", + "Privilege Escalation" + ] + }, + "t1048": { + "technique_id": "T1048", + "technique": "Exfiltration Over Alternative Protocol", + "url": "https://attack.mitre.org/techniques/T1048", + "tactic": [ + "Exfiltration" + ] + }, + "t1597": { + "technique_id": "T1597", + "technique": "Search Closed Sources", + "url": "https://attack.mitre.org/techniques/T1597", + "tactic": [ + "Reconnaissance" + ] + }, + "t1566": { + "technique_id": "T1566", + "technique": "Phishing", + "url": "https://attack.mitre.org/techniques/T1566", + "tactic": [ + "Initial Access" + ] + }, + "t1110": { + "technique_id": "T1110", + "technique": "Brute Force", + "url": "https://attack.mitre.org/techniques/T1110", + "tactic": [ + "Credential Access" + ] + }, + "t1565": { + "technique_id": "T1565", + "technique": "Data Manipulation", + "url": "https://attack.mitre.org/techniques/T1565", + "tactic": [ + "Impact" + ] + }, + "t1559": { + "technique_id": "T1559", + "technique": "Inter-Process Communication", + "url": "https://attack.mitre.org/techniques/T1559", + "tactic": [ + "Execution" + ] + }, + "t1001": { + "technique_id": "T1001", + "technique": "Data Obfuscation", + "url": "https://attack.mitre.org/techniques/T1001", + "tactic": [ + "Command and Control" + ] + }, + "t1039": { + "technique_id": "T1039", + "technique": "Data from Network Shared Drive", + "url": "https://attack.mitre.org/techniques/T1039", + "tactic": [ + "Collection" + ] + }, + "t1601": { + "technique_id": "T1601", + "technique": "Modify System Image", + "url": "https://attack.mitre.org/techniques/T1601", + "tactic": [ + "Defense Evasion" + ] + }, + "t1574": { + "technique_id": "T1574", + "technique": "Hijack Execution Flow", + "url": "https://attack.mitre.org/techniques/T1574", + "tactic": [ + "Persistence", + "Privilege Escalation", + "Defense Evasion" + ] + }, + "t1078": { + "technique_id": "T1078", + "technique": "Valid Accounts", + "url": "https://attack.mitre.org/techniques/T1078", + "tactic": [ + "Defense Evasion", + "Persistence", + "Privilege Escalation", + "Initial Access" + ] + }, + "t1571": { + "technique_id": "T1571", + "technique": "Non-Standard Port", + "url": "https://attack.mitre.org/techniques/T1571", + "tactic": [ + "Command and Control" + ] + }, + "t1068": { + "technique_id": "T1068", + "technique": "Exploitation for Privilege Escalation", + "url": "https://attack.mitre.org/techniques/T1068", + "tactic": [ + "Privilege Escalation" + ] + }, + "t1531": { + "technique_id": "T1531", + "technique": "Account Access Removal", + "url": "https://attack.mitre.org/techniques/T1531", + "tactic": [ + "Impact" + ] + }, + "t1027": { + "technique_id": "T1027", + "technique": "Obfuscated Files or Information", + "url": "https://attack.mitre.org/techniques/T1027", + "tactic": [ + "Defense Evasion" + ] + }, + "t1201": { + "technique_id": "T1201", + "technique": "Password Policy Discovery", + "url": "https://attack.mitre.org/techniques/T1201", + "tactic": [ + "Discovery" + ] + }, + "t1546": { + "technique_id": "T1546", + "technique": "Event Triggered Execution", + "url": "https://attack.mitre.org/techniques/T1546", + "tactic": [ + "Privilege Escalation", + "Persistence" + ] + }, + "t1187": { + "technique_id": "T1187", + "technique": "Forced Authentication", + "url": "https://attack.mitre.org/techniques/T1187", + "tactic": [ + "Credential Access" + ] + }, + "t1599": { + "technique_id": "T1599", + "technique": "Network Boundary Bridging", + "url": "https://attack.mitre.org/techniques/T1599", + "tactic": [ + "Defense Evasion" + ] + }, + "t1486": { + "technique_id": "T1486", + "technique": "Data Encrypted for Impact", + "url": "https://attack.mitre.org/techniques/T1486", + "tactic": [ + "Impact" + ] + }, + "t1553": { + "technique_id": "T1553", + "technique": "Subvert Trust Controls", + "url": "https://attack.mitre.org/techniques/T1553", + "tactic": [ + "Defense Evasion" + ] + }, + "t1573": { + "technique_id": "T1573", + "technique": "Encrypted Channel", + "url": "https://attack.mitre.org/techniques/T1573", + "tactic": [ + "Command and Control" + ] + }, + "t1056": { + "technique_id": "T1056", + "technique": "Input Capture", + "url": "https://attack.mitre.org/techniques/T1056", + "tactic": [ + "Collection", + "Credential Access" + ] + }, + "t1203": { + "technique_id": "T1203", + "technique": "Exploitation for Client Execution", + "url": "https://attack.mitre.org/techniques/T1203", + "tactic": [ + "Execution" + ] + }, + "t1570": { + "technique_id": "T1570", + "technique": "Lateral Tool Transfer", + "url": "https://attack.mitre.org/techniques/T1570", + "tactic": [ + "Lateral Movement" + ] + }, + "t1095": { + "technique_id": "T1095", + "technique": "Non-Application Layer Protocol", + "url": "https://attack.mitre.org/techniques/T1095", + "tactic": [ + "Command and Control" + ] + }, + "t1012": { + "technique_id": "T1012", + "technique": "Query Registry", + "url": "https://attack.mitre.org/techniques/T1012", + "tactic": [ + "Discovery" + ] + }, + "t1030": { + "technique_id": "T1030", + "technique": "Data Transfer Size Limits", + "url": "https://attack.mitre.org/techniques/T1030", + "tactic": [ + "Exfiltration" + ] + }, + "t1499": { + "technique_id": "T1499", + "technique": "Endpoint Denial of Service", + "url": "https://attack.mitre.org/techniques/T1499", + "tactic": [ + "Impact" + ] + }, + "t1614": { + "technique_id": "T1614", + "technique": "System Location Discovery", + "url": "https://attack.mitre.org/techniques/T1614", + "tactic": [ + "Discovery" + ] + }, + "t1197": { + "technique_id": "T1197", + "technique": "BITS Jobs", + "url": "https://attack.mitre.org/techniques/T1197", + "tactic": [ + "Defense Evasion", + "Persistence" + ] + }, + "t1656": { + "technique_id": "T1656", + "technique": "Impersonation", + "url": "https://attack.mitre.org/techniques/T1656", + "tactic": [ + "Defense Evasion" + ] + }, + "t1132": { + "technique_id": "T1132", + "technique": "Data Encoding", + "url": "https://attack.mitre.org/techniques/T1132", + "tactic": [ + "Command and Control" + ] + }, + "t1598": { + "technique_id": "T1598", + "technique": "Phishing for Information", + "url": "https://attack.mitre.org/techniques/T1598", + "tactic": [ + "Reconnaissance" + ] + }, + "t1496": { + "technique_id": "T1496", + "technique": "Resource Hijacking", + "url": "https://attack.mitre.org/techniques/T1496", + "tactic": [ + "Impact" + ] + }, + "t1585": { + "technique_id": "T1585", + "technique": "Establish Accounts", + "url": "https://attack.mitre.org/techniques/T1585", + "tactic": [ + "Resource Development" + ] + }, + "t1588": { + "technique_id": "T1588", + "technique": "Obtain Capabilities", + "url": "https://attack.mitre.org/techniques/T1588", + "tactic": [ + "Resource Development" + ] + }, + "t1569": { + "technique_id": "T1569", + "technique": "System Services", + "url": "https://attack.mitre.org/techniques/T1569", + "tactic": [ + "Execution" + ] + }, + "t1650": { + "technique_id": "T1650", + "technique": "Acquire Access", + "url": "https://attack.mitre.org/techniques/T1650", + "tactic": [ + "Resource Development" + ] + }, + "t1213": { + "technique_id": "T1213", + "technique": "Data from Information Repositories", + "url": "https://attack.mitre.org/techniques/T1213", + "tactic": [ + "Collection" + ] + }, + "t1200": { + "technique_id": "T1200", + "technique": "Hardware Additions", + "url": "https://attack.mitre.org/techniques/T1200", + "tactic": [ + "Initial Access" + ] + }, + "t1505": { + "technique_id": "T1505", + "technique": "Server Software Component", + "url": "https://attack.mitre.org/techniques/T1505", + "tactic": [ + "Persistence" + ] + }, + "t1485": { + "technique_id": "T1485", + "technique": "Data Destruction", + "url": "https://attack.mitre.org/techniques/T1485", + "tactic": [ + "Impact" + ] + }, + "t1537": { + "technique_id": "T1537", + "technique": "Transfer Data to Cloud Account", + "url": "https://attack.mitre.org/techniques/T1537", + "tactic": [ + "Exfiltration" + ] + }, + "t1189": { + "technique_id": "T1189", + "technique": "Drive-by Compromise", + "url": "https://attack.mitre.org/techniques/T1189", + "tactic": [ + "Initial Access" + ] + }, + "t1498": { + "technique_id": "T1498", + "technique": "Network Denial of Service", + "url": "https://attack.mitre.org/techniques/T1498", + "tactic": [ + "Impact" + ] + }, + "t1651": { + "technique_id": "T1651", + "technique": "Cloud Administration Command", + "url": "https://attack.mitre.org/techniques/T1651", + "tactic": [ + "Execution" + ] + }, + "t1221": { + "technique_id": "T1221", + "technique": "Template Injection", + "url": "https://attack.mitre.org/techniques/T1221", + "tactic": [ + "Defense Evasion" + ] + }, + "t1134": { + "technique_id": "T1134", + "technique": "Access Token Manipulation", + "url": "https://attack.mitre.org/techniques/T1134", + "tactic": [ + "Defense Evasion", + "Privilege Escalation" + ] + }, + "t1111": { + "technique_id": "T1111", + "technique": "Multi-Factor Authentication Interception", + "url": "https://attack.mitre.org/techniques/T1111", + "tactic": [ + "Credential Access" + ] + }, + "t1136": { + "technique_id": "T1136", + "technique": "Create Account", + "url": "https://attack.mitre.org/techniques/T1136", + "tactic": [ + "Persistence" + ] + }, + "t1526": { + "technique_id": "T1526", + "technique": "Cloud Service Discovery", + "url": "https://attack.mitre.org/techniques/T1526", + "tactic": [ + "Discovery" + ] + }, + "t1018": { + "technique_id": "T1018", + "technique": "Remote System Discovery", + "url": "https://attack.mitre.org/techniques/T1018", + "tactic": [ + "Discovery" + ] + }, + "t1046": { + "technique_id": "T1046", + "technique": "Network Service Discovery", + "url": "https://attack.mitre.org/techniques/T1046", + "tactic": [ + "Discovery" + ] + }, + "t1518": { + "technique_id": "T1518", + "technique": "Software Discovery", + "url": "https://attack.mitre.org/techniques/T1518", + "tactic": [ + "Discovery" + ] + }, + "t1538": { + "technique_id": "T1538", + "technique": "Cloud Service Dashboard", + "url": "https://attack.mitre.org/techniques/T1538", + "tactic": [ + "Discovery" + ] + }, + "t1622": { + "technique_id": "T1622", + "technique": "Debugger Evasion", + "url": "https://attack.mitre.org/techniques/T1622", + "tactic": [ + "Defense Evasion", + "Discovery" + ] + }, + "t1052": { + "technique_id": "T1052", + "technique": "Exfiltration Over Physical Medium", + "url": "https://attack.mitre.org/techniques/T1052", + "tactic": [ + "Exfiltration" + ] + }, + "t1105": { + "technique_id": "T1105", + "technique": "Ingress Tool Transfer", + "url": "https://attack.mitre.org/techniques/T1105", + "tactic": [ + "Command and Control" + ] + }, + "t1648": { + "technique_id": "T1648", + "technique": "Serverless Execution", + "url": "https://attack.mitre.org/techniques/T1648", + "tactic": [ + "Execution" + ] + }, + "t1653": { + "technique_id": "T1653", + "technique": "Power Settings", + "url": "https://attack.mitre.org/techniques/T1653", + "tactic": [ + "Persistence" + ] + }, + "t1484": { + "technique_id": "T1484", + "technique": "Domain Policy Modification", + "url": "https://attack.mitre.org/techniques/T1484", + "tactic": [ + "Defense Evasion", + "Privilege Escalation" + ] + }, + "t1220": { + "technique_id": "T1220", + "technique": "XSL Script Processing", + "url": "https://attack.mitre.org/techniques/T1220", + "tactic": [ + "Defense Evasion" + ] + }, + "t1587": { + "technique_id": "T1587", + "technique": "Develop Capabilities", + "url": "https://attack.mitre.org/techniques/T1587", + "tactic": [ + "Resource Development" + ] + }, + "t1008": { + "technique_id": "T1008", + "technique": "Fallback Channels", + "url": "https://attack.mitre.org/techniques/T1008", + "tactic": [ + "Command and Control" + ] + }, + "t1124": { + "technique_id": "T1124", + "technique": "System Time Discovery", + "url": "https://attack.mitre.org/techniques/T1124", + "tactic": [ + "Discovery" + ] + }, + "t1556": { + "technique_id": "T1556", + "technique": "Modify Authentication Process", + "url": "https://attack.mitre.org/techniques/T1556", + "tactic": [ + "Credential Access", + "Defense Evasion", + "Persistence" + ] + }, + "t1495": { + "technique_id": "T1495", + "technique": "Firmware Corruption", + "url": "https://attack.mitre.org/techniques/T1495", + "tactic": [ + "Impact" + ] + }, + "t1490": { + "technique_id": "T1490", + "technique": "Inhibit System Recovery", + "url": "https://attack.mitre.org/techniques/T1490", + "tactic": [ + "Impact" + ] + }, + "t1216": { + "technique_id": "T1216", + "technique": "System Script Proxy Execution", + "url": "https://attack.mitre.org/techniques/T1216", + "tactic": [ + "Defense Evasion" + ] + }, + "t1211": { + "technique_id": "T1211", + "technique": "Exploitation for Defense Evasion", + "url": "https://attack.mitre.org/techniques/T1211", + "tactic": [ + "Defense Evasion" + ] + }, + "t1127": { + "technique_id": "T1127", + "technique": "Trusted Developer Utilities Proxy Execution", + "url": "https://attack.mitre.org/techniques/T1127", + "tactic": [ + "Defense Evasion" + ] + }, + "t1529": { + "technique_id": "T1529", + "technique": "System Shutdown/Reboot", + "url": "https://attack.mitre.org/techniques/T1529", + "tactic": [ + "Impact" + ] + }, + "t1055.011": { + "technique_id": "T1055.011", + "technique": "Process Injection : Extra Window Memory Injection", + "url": "https://attack.mitre.org/techniques/T1055/011" + }, + "t1053.005": { + "technique_id": "T1053.005", + "technique": "Scheduled Task/Job : Scheduled Task", + "url": "https://attack.mitre.org/techniques/T1053/005" + }, + "t1205.002": { + "technique_id": "T1205.002", + "technique": "Traffic Signaling : Socket Filters", + "url": "https://attack.mitre.org/techniques/T1205/002" + }, + "t1560.001": { + "technique_id": "T1560.001", + "technique": "Archive Collected Data : Archive via Utility", + "url": "https://attack.mitre.org/techniques/T1560/001" + }, + "t1021.005": { + "technique_id": "T1021.005", + "technique": "Remote Services : VNC", + "url": "https://attack.mitre.org/techniques/T1021/005" + }, + "t1027.011": { + "technique_id": "T1027.011", + "technique": "Obfuscated Files or Information : Fileless Storage", + "url": "https://attack.mitre.org/techniques/T1027/011" + }, + "t1218.011": { + "technique_id": "T1218.011", + "technique": "System Binary Proxy Execution : Rundll32", + "url": "https://attack.mitre.org/techniques/T1218/011" + }, + "t1583.007": { + "technique_id": "T1583.007", + "technique": "Acquire Infrastructure : Serverless", + "url": "https://attack.mitre.org/techniques/T1583/007" + }, + "t1132.001": { + "technique_id": "T1132.001", + "technique": "Data Encoding : Standard Encoding", + "url": "https://attack.mitre.org/techniques/T1132/001" + }, + "t1027.009": { + "technique_id": "T1027.009", + "technique": "Obfuscated Files or Information : Embedded Payloads", + "url": "https://attack.mitre.org/techniques/T1027/009" + }, + "t1556.003": { + "technique_id": "T1556.003", + "technique": "Modify Authentication Process : Pluggable Authentication Modules", + "url": "https://attack.mitre.org/techniques/T1556/003" + }, + "t1578.004": { + "technique_id": "T1578.004", + "technique": "Modify Cloud Compute Infrastructure : Revert Cloud Instance", + "url": "https://attack.mitre.org/techniques/T1578/004" + }, + "t1596.003": { + "technique_id": "T1596.003", + "technique": "Search Open Technical Databases : Digital Certificates", + "url": "https://attack.mitre.org/techniques/T1596/003" + }, + "t1056.001": { + "technique_id": "T1056.001", + "technique": "Input Capture : Keylogging", + "url": "https://attack.mitre.org/techniques/T1056/001" + }, + "t1222.002": { + "technique_id": "T1222.002", + "technique": "File and Directory Permissions Modification : Linux and Mac File and Directory Permissions Modification", + "url": "https://attack.mitre.org/techniques/T1222/002" + }, + "t1110.001": { + "technique_id": "T1110.001", + "technique": "Brute Force : Password Guessing", + "url": "https://attack.mitre.org/techniques/T1110/001" + }, + "t1216.001": { + "technique_id": "T1216.001", + "technique": "System Script Proxy Execution : PubPrn", + "url": "https://attack.mitre.org/techniques/T1216/001" + }, + "t1597.002": { + "technique_id": "T1597.002", + "technique": "Search Closed Sources : Purchase Technical Data", + "url": "https://attack.mitre.org/techniques/T1597/002" + }, + "t1561.002": { + "technique_id": "T1561.002", + "technique": "Disk Wipe : Disk Structure Wipe", + "url": "https://attack.mitre.org/techniques/T1561/002" + }, + "t1498.001": { + "technique_id": "T1498.001", + "technique": "Network Denial of Service : Direct Network Flood", + "url": "https://attack.mitre.org/techniques/T1498/001" + }, + "t1574.007": { + "technique_id": "T1574.007", + "technique": "Hijack Execution Flow : Path Interception by PATH Environment Variable", + "url": "https://attack.mitre.org/techniques/T1574/007" + }, + "t1213.002": { + "technique_id": "T1213.002", + "technique": "Data from Information Repositories : Sharepoint", + "url": "https://attack.mitre.org/techniques/T1213/002" + }, + "t1564.008": { + "technique_id": "T1564.008", + "technique": "Hide Artifacts : Email Hiding Rules", + "url": "https://attack.mitre.org/techniques/T1564/008" + }, + "t1491.002": { + "technique_id": "T1491.002", + "technique": "Defacement : External Defacement", + "url": "https://attack.mitre.org/techniques/T1491/002" + }, + "t1590.005": { + "technique_id": "T1590.005", + "technique": "Gather Victim Network Information : IP Addresses", + "url": "https://attack.mitre.org/techniques/T1590/005" + }, + "t1499.001": { + "technique_id": "T1499.001", + "technique": "Endpoint Denial of Service : OS Exhaustion Flood", + "url": "https://attack.mitre.org/techniques/T1499/001" + }, + "t1546.013": { + "technique_id": "T1546.013", + "technique": "Event Triggered Execution : PowerShell Profile", + "url": "https://attack.mitre.org/techniques/T1546/013" + }, + "t1059.007": { + "technique_id": "T1059.007", + "technique": "Command and Scripting Interpreter : JavaScript", + "url": "https://attack.mitre.org/techniques/T1059/007" + }, + "t1590.002": { + "technique_id": "T1590.002", + "technique": "Gather Victim Network Information : DNS", + "url": "https://attack.mitre.org/techniques/T1590/002" + }, + "t1546.006": { + "technique_id": "T1546.006", + "technique": "Event Triggered Execution : LC_LOAD_DYLIB Addition", + "url": "https://attack.mitre.org/techniques/T1546/006" + }, + "t1053.007": { + "technique_id": "T1053.007", + "technique": "Scheduled Task/Job : Container Orchestration Job", + "url": "https://attack.mitre.org/techniques/T1053/007" + }, + "t1568.002": { + "technique_id": "T1568.002", + "technique": "Dynamic Resolution : Domain Generation Algorithms", + "url": "https://attack.mitre.org/techniques/T1568/002" + }, + "t1036.007": { + "technique_id": "T1036.007", + "technique": "Masquerading : Double File Extension", + "url": "https://attack.mitre.org/techniques/T1036/007" + }, + "t1548.002": { + "technique_id": "T1548.002", + "technique": "Abuse Elevation Control Mechanism : Bypass User Account Control", + "url": "https://attack.mitre.org/techniques/T1548/002" + }, + "t1016.001": { + "technique_id": "T1016.001", + "technique": "System Network Configuration Discovery : Internet Connection Discovery", + "url": "https://attack.mitre.org/techniques/T1016/001" + }, + "t1548.003": { + "technique_id": "T1548.003", + "technique": "Abuse Elevation Control Mechanism : Sudo and Sudo Caching", + "url": "https://attack.mitre.org/techniques/T1548/003" + }, + "t1560.003": { + "technique_id": "T1560.003", + "technique": "Archive Collected Data : Archive via Custom Method", + "url": "https://attack.mitre.org/techniques/T1560/003" + }, + "t1583.008": { + "technique_id": "T1583.008", + "technique": "Acquire Infrastructure : Malvertising", + "url": "https://attack.mitre.org/techniques/T1583/008" + }, + "t1003.002": { + "technique_id": "T1003.002", + "technique": "OS Credential Dumping : Security Account Manager", + "url": "https://attack.mitre.org/techniques/T1003/002" + }, + "t1596.002": { + "technique_id": "T1596.002", + "technique": "Search Open Technical Databases : WHOIS", + "url": "https://attack.mitre.org/techniques/T1596/002" + }, + "t1542.001": { + "technique_id": "T1542.001", + "technique": "Pre-OS Boot : System Firmware", + "url": "https://attack.mitre.org/techniques/T1542/001" + }, + "t1069.003": { + "technique_id": "T1069.003", + "technique": "Permission Groups Discovery : Cloud Groups", + "url": "https://attack.mitre.org/techniques/T1069/003" + }, + "t1574.011": { + "technique_id": "T1574.011", + "technique": "Hijack Execution Flow : Services Registry Permissions Weakness", + "url": "https://attack.mitre.org/techniques/T1574/011" + }, + "t1596.001": { + "technique_id": "T1596.001", + "technique": "Search Open Technical Databases : DNS/Passive DNS", + "url": "https://attack.mitre.org/techniques/T1596/001" + }, + "t1499.003": { + "technique_id": "T1499.003", + "technique": "Endpoint Denial of Service : Application Exhaustion Flood", + "url": "https://attack.mitre.org/techniques/T1499/003" + }, + "t1195.001": { + "technique_id": "T1195.001", + "technique": "Supply Chain Compromise : Compromise Software Dependencies and Development Tools", + "url": "https://attack.mitre.org/techniques/T1195/001" + }, + "t1588.004": { + "technique_id": "T1588.004", + "technique": "Obtain Capabilities : Digital Certificates", + "url": "https://attack.mitre.org/techniques/T1588/004" + }, + "t1583.002": { + "technique_id": "T1583.002", + "technique": "Acquire Infrastructure : DNS Server", + "url": "https://attack.mitre.org/techniques/T1583/002" + }, + "t1071.004": { + "technique_id": "T1071.004", + "technique": "Application Layer Protocol : DNS", + "url": "https://attack.mitre.org/techniques/T1071/004" + }, + "t1552.005": { + "technique_id": "T1552.005", + "technique": "Unsecured Credentials : Cloud Instance Metadata API", + "url": "https://attack.mitre.org/techniques/T1552/005" + }, + "t1555.002": { + "technique_id": "T1555.002", + "technique": "Credentials from Password Stores : Securityd Memory", + "url": "https://attack.mitre.org/techniques/T1555/002" + }, + "t1542.003": { + "technique_id": "T1542.003", + "technique": "Pre-OS Boot : Bootkit", + "url": "https://attack.mitre.org/techniques/T1542/003" + }, + "t1218.013": { + "technique_id": "T1218.013", + "technique": "System Binary Proxy Execution : Mavinject", + "url": "https://attack.mitre.org/techniques/T1218/013" + }, + "t1074.001": { + "technique_id": "T1074.001", + "technique": "Data Staged : Local Data Staging", + "url": "https://attack.mitre.org/techniques/T1074/001" + }, + "t1036.005": { + "technique_id": "T1036.005", + "technique": "Masquerading : Match Legitimate Name or Location", + "url": "https://attack.mitre.org/techniques/T1036/005" + }, + "t1587.003": { + "technique_id": "T1587.003", + "technique": "Develop Capabilities : Digital Certificates", + "url": "https://attack.mitre.org/techniques/T1587/003" + }, + "t1565.001": { + "technique_id": "T1565.001", + "technique": "Data Manipulation : Stored Data Manipulation", + "url": "https://attack.mitre.org/techniques/T1565/001" + }, + "t1110.002": { + "technique_id": "T1110.002", + "technique": "Brute Force : Password Cracking", + "url": "https://attack.mitre.org/techniques/T1110/002" + }, + "t1114.001": { + "technique_id": "T1114.001", + "technique": "Email Collection : Local Email Collection", + "url": "https://attack.mitre.org/techniques/T1114/001" + }, + "t1555.001": { + "technique_id": "T1555.001", + "technique": "Credentials from Password Stores : Keychain", + "url": "https://attack.mitre.org/techniques/T1555/001" + }, + "t1003.004": { + "technique_id": "T1003.004", + "technique": "OS Credential Dumping : LSA Secrets", + "url": "https://attack.mitre.org/techniques/T1003/004" + }, + "t1606.002": { + "technique_id": "T1606.002", + "technique": "Forge Web Credentials : SAML Tokens", + "url": "https://attack.mitre.org/techniques/T1606/002" + }, + "t1036.008": { + "technique_id": "T1036.008", + "technique": "Masquerading : Masquerade File Type", + "url": "https://attack.mitre.org/techniques/T1036/008" + }, + "t1587.001": { + "technique_id": "T1587.001", + "technique": "Develop Capabilities : Malware", + "url": "https://attack.mitre.org/techniques/T1587/001" + }, + "t1087.002": { + "technique_id": "T1087.002", + "technique": "Account Discovery : Domain Account", + "url": "https://attack.mitre.org/techniques/T1087/002" + }, + "t1547.014": { + "technique_id": "T1547.014", + "technique": "Boot or Logon Autostart Execution : Active Setup", + "url": "https://attack.mitre.org/techniques/T1547/014" + }, + "t1559.002": { + "technique_id": "T1559.002", + "technique": "Inter-Process Communication : Dynamic Data Exchange", + "url": "https://attack.mitre.org/techniques/T1559/002" + }, + "t1204.002": { + "technique_id": "T1204.002", + "technique": "User Execution : Malicious File", + "url": "https://attack.mitre.org/techniques/T1204/002" + }, + "t1591.003": { + "technique_id": "T1591.003", + "technique": "Gather Victim Org Information : Identify Business Tempo", + "url": "https://attack.mitre.org/techniques/T1591/003" + }, + "t1592.001": { + "technique_id": "T1592.001", + "technique": "Gather Victim Host Information : Hardware", + "url": "https://attack.mitre.org/techniques/T1592/001" + }, + "t1484.002": { + "technique_id": "T1484.002", + "technique": "Domain Policy Modification : Domain Trust Modification", + "url": "https://attack.mitre.org/techniques/T1484/002" + }, + "t1573.001": { + "technique_id": "T1573.001", + "technique": "Encrypted Channel : Symmetric Cryptography", + "url": "https://attack.mitre.org/techniques/T1573/001" + }, + "t1087.001": { + "technique_id": "T1087.001", + "technique": "Account Discovery : Local Account", + "url": "https://attack.mitre.org/techniques/T1087/001" + }, + "t1586.001": { + "technique_id": "T1586.001", + "technique": "Compromise Accounts : Social Media Accounts", + "url": "https://attack.mitre.org/techniques/T1586/001" + }, + "t1562.009": { + "technique_id": "T1562.009", + "technique": "Impair Defenses : Safe Mode Boot", + "url": "https://attack.mitre.org/techniques/T1562/009" + }, + "t1542.005": { + "technique_id": "T1542.005", + "technique": "Pre-OS Boot : TFTP Boot", + "url": "https://attack.mitre.org/techniques/T1542/005" + }, + "t1543.003": { + "technique_id": "T1543.003", + "technique": "Create or Modify System Process : Windows Service", + "url": "https://attack.mitre.org/techniques/T1543/003" + }, + "t1568.001": { + "technique_id": "T1568.001", + "technique": "Dynamic Resolution : Fast Flux DNS", + "url": "https://attack.mitre.org/techniques/T1568/001" + }, + "t1497.001": { + "technique_id": "T1497.001", + "technique": "Virtualization/Sandbox Evasion : System Checks", + "url": "https://attack.mitre.org/techniques/T1497/001" + }, + "t1053.003": { + "technique_id": "T1053.003", + "technique": "Scheduled Task/Job : Cron", + "url": "https://attack.mitre.org/techniques/T1053/003" + }, + "t1069.002": { + "technique_id": "T1069.002", + "technique": "Permission Groups Discovery : Domain Groups", + "url": "https://attack.mitre.org/techniques/T1069/002" + }, + "t1588.006": { + "technique_id": "T1588.006", + "technique": "Obtain Capabilities : Vulnerabilities", + "url": "https://attack.mitre.org/techniques/T1588/006" + }, + "t1566.002": { + "technique_id": "T1566.002", + "technique": "Phishing : Spearphishing Link", + "url": "https://attack.mitre.org/techniques/T1566/002" + }, + "t1070.002": { + "technique_id": "T1070.002", + "technique": "Indicator Removal : Clear Linux or Mac System Logs", + "url": "https://attack.mitre.org/techniques/T1070/002" + }, + "t1499.004": { + "technique_id": "T1499.004", + "technique": "Endpoint Denial of Service : Application or System Exploitation", + "url": "https://attack.mitre.org/techniques/T1499/004" + }, + "t1218.004": { + "technique_id": "T1218.004", + "technique": "System Binary Proxy Execution : InstallUtil", + "url": "https://attack.mitre.org/techniques/T1218/004" + }, + "t1598.003": { + "technique_id": "T1598.003", + "technique": "Phishing for Information : Spearphishing Link", + "url": "https://attack.mitre.org/techniques/T1598/003" + }, + "t1021.004": { + "technique_id": "T1021.004", + "technique": "Remote Services : SSH", + "url": "https://attack.mitre.org/techniques/T1021/004" + }, + "t1098.003": { + "technique_id": "T1098.003", + "technique": "Account Manipulation : Additional Cloud Roles", + "url": "https://attack.mitre.org/techniques/T1098/003" + }, + "t1547.012": { + "technique_id": "T1547.012", + "technique": "Boot or Logon Autostart Execution : Print Processors", + "url": "https://attack.mitre.org/techniques/T1547/012" + }, + "t1566.001": { + "technique_id": "T1566.001", + "technique": "Phishing : Spearphishing Attachment", + "url": "https://attack.mitre.org/techniques/T1566/001" + }, + "t1027.008": { + "technique_id": "T1027.008", + "technique": "Obfuscated Files or Information : Stripped Payloads", + "url": "https://attack.mitre.org/techniques/T1027/008" + }, + "t1559.001": { + "technique_id": "T1559.001", + "technique": "Inter-Process Communication : Component Object Model", + "url": "https://attack.mitre.org/techniques/T1559/001" + }, + "t1574.001": { + "technique_id": "T1574.001", + "technique": "Hijack Execution Flow : DLL Search Order Hijacking", + "url": "https://attack.mitre.org/techniques/T1574/001" + }, + "t1003.007": { + "technique_id": "T1003.007", + "technique": "OS Credential Dumping : Proc Filesystem", + "url": "https://attack.mitre.org/techniques/T1003/007" + }, + "t1583.005": { + "technique_id": "T1583.005", + "technique": "Acquire Infrastructure : Botnet", + "url": "https://attack.mitre.org/techniques/T1583/005" + }, + "t1555.005": { + "technique_id": "T1555.005", + "technique": "Credentials from Password Stores : Password Managers", + "url": "https://attack.mitre.org/techniques/T1555/005" + }, + "t1553.001": { + "technique_id": "T1553.001", + "technique": "Subvert Trust Controls : Gatekeeper Bypass", + "url": "https://attack.mitre.org/techniques/T1553/001" + }, + "t1608.004": { + "technique_id": "T1608.004", + "technique": "Stage Capabilities : Drive-by Target", + "url": "https://attack.mitre.org/techniques/T1608/004" + }, + "t1553.002": { + "technique_id": "T1553.002", + "technique": "Subvert Trust Controls : Code Signing", + "url": "https://attack.mitre.org/techniques/T1553/002" + }, + "t1565.003": { + "technique_id": "T1565.003", + "technique": "Data Manipulation : Runtime Data Manipulation", + "url": "https://attack.mitre.org/techniques/T1565/003" + }, + "t1552.002": { + "technique_id": "T1552.002", + "technique": "Unsecured Credentials : Credentials in Registry", + "url": "https://attack.mitre.org/techniques/T1552/002" + }, + "t1036.009": { + "technique_id": "T1036.009", + "technique": "Masquerading : Break Process Trees", + "url": "https://attack.mitre.org/techniques/T1036/009" + }, + "t1590.004": { + "technique_id": "T1590.004", + "technique": "Gather Victim Network Information : Network Topology", + "url": "https://attack.mitre.org/techniques/T1590/004" + }, + "t1587.002": { + "technique_id": "T1587.002", + "technique": "Develop Capabilities : Code Signing Certificates", + "url": "https://attack.mitre.org/techniques/T1587/002" + }, + "t1222.001": { + "technique_id": "T1222.001", + "technique": "File and Directory Permissions Modification : Windows File and Directory Permissions Modification", + "url": "https://attack.mitre.org/techniques/T1222/001" + }, + "t1137.006": { + "technique_id": "T1137.006", + "technique": "Office Application Startup : Add-ins", + "url": "https://attack.mitre.org/techniques/T1137/006" + }, + "t1505.002": { + "technique_id": "T1505.002", + "technique": "Server Software Component : Transport Agent", + "url": "https://attack.mitre.org/techniques/T1505/002" + }, + "t1074.002": { + "technique_id": "T1074.002", + "technique": "Data Staged : Remote Data Staging", + "url": "https://attack.mitre.org/techniques/T1074/002" + }, + "t1098.006": { + "technique_id": "T1098.006", + "technique": "Account Manipulation : Additional Container Cluster Roles", + "url": "https://attack.mitre.org/techniques/T1098/006" + }, + "t1218.007": { + "technique_id": "T1218.007", + "technique": "System Binary Proxy Execution : Msiexec", + "url": "https://attack.mitre.org/techniques/T1218/007" + }, + "t1590.003": { + "technique_id": "T1590.003", + "technique": "Gather Victim Network Information : Network Trust Dependencies", + "url": "https://attack.mitre.org/techniques/T1590/003" + }, + "t1498.002": { + "technique_id": "T1498.002", + "technique": "Network Denial of Service : Reflection Amplification", + "url": "https://attack.mitre.org/techniques/T1498/002" + }, + "t1556.002": { + "technique_id": "T1556.002", + "technique": "Modify Authentication Process : Password Filter DLL", + "url": "https://attack.mitre.org/techniques/T1556/002" + }, + "t1505.005": { + "technique_id": "T1505.005", + "technique": "Server Software Component : Terminal Services DLL", + "url": "https://attack.mitre.org/techniques/T1505/005" + }, + "t1059.002": { + "technique_id": "T1059.002", + "technique": "Command and Scripting Interpreter : AppleScript", + "url": "https://attack.mitre.org/techniques/T1059/002" + }, + "t1499.002": { + "technique_id": "T1499.002", + "technique": "Endpoint Denial of Service : Service Exhaustion Flood", + "url": "https://attack.mitre.org/techniques/T1499/002" + }, + "t1195.003": { + "technique_id": "T1195.003", + "technique": "Supply Chain Compromise : Compromise Hardware Supply Chain", + "url": "https://attack.mitre.org/techniques/T1195/003" + }, + "t1070.007": { + "technique_id": "T1070.007", + "technique": "Indicator Removal : Clear Network Connection History and Configurations", + "url": "https://attack.mitre.org/techniques/T1070/007" + }, + "t1558.004": { + "technique_id": "T1558.004", + "technique": "Steal or Forge Kerberos Tickets : AS-REP Roasting", + "url": "https://attack.mitre.org/techniques/T1558/004" + }, + "t1584.003": { + "technique_id": "T1584.003", + "technique": "Compromise Infrastructure : Virtual Private Server", + "url": "https://attack.mitre.org/techniques/T1584/003" + }, + "t1600.001": { + "technique_id": "T1600.001", + "technique": "Weaken Encryption : Reduce Key Space", + "url": "https://attack.mitre.org/techniques/T1600/001" + }, + "t1070.003": { + "technique_id": "T1070.003", + "technique": "Indicator Removal : Clear Command History", + "url": "https://attack.mitre.org/techniques/T1070/003" + }, + "t1137.005": { + "technique_id": "T1137.005", + "technique": "Office Application Startup : Outlook Rules", + "url": "https://attack.mitre.org/techniques/T1137/005" + }, + "t1586.003": { + "technique_id": "T1586.003", + "technique": "Compromise Accounts : Cloud Accounts", + "url": "https://attack.mitre.org/techniques/T1586/003" + }, + "t1586.002": { + "technique_id": "T1586.002", + "technique": "Compromise Accounts : Email Accounts", + "url": "https://attack.mitre.org/techniques/T1586/002" + }, + "t1608.001": { + "technique_id": "T1608.001", + "technique": "Stage Capabilities : Upload Malware", + "url": "https://attack.mitre.org/techniques/T1608/001" + }, + "t1583.001": { + "technique_id": "T1583.001", + "technique": "Acquire Infrastructure : Domains", + "url": "https://attack.mitre.org/techniques/T1583/001" + }, + "t1560.002": { + "technique_id": "T1560.002", + "technique": "Archive Collected Data : Archive via Library", + "url": "https://attack.mitre.org/techniques/T1560/002" + }, + "t1055.003": { + "technique_id": "T1055.003", + "technique": "Process Injection : Thread Execution Hijacking", + "url": "https://attack.mitre.org/techniques/T1055/003" + }, + "t1546.011": { + "technique_id": "T1546.011", + "technique": "Event Triggered Execution : Application Shimming", + "url": "https://attack.mitre.org/techniques/T1546/011" + }, + "t1547.010": { + "technique_id": "T1547.010", + "technique": "Boot or Logon Autostart Execution : Port Monitors", + "url": "https://attack.mitre.org/techniques/T1547/010" + }, + "t1070.008": { + "technique_id": "T1070.008", + "technique": "Indicator Removal : Clear Mailbox Data", + "url": "https://attack.mitre.org/techniques/T1070/008" + }, + "t1037.002": { + "technique_id": "T1037.002", + "technique": "Boot or Logon Initialization Scripts : Login Hook", + "url": "https://attack.mitre.org/techniques/T1037/002" + }, + "t1567.004": { + "technique_id": "T1567.004", + "technique": "Exfiltration Over Web Service : Exfiltration Over Webhook", + "url": "https://attack.mitre.org/techniques/T1567/004" + }, + "t1021.008": { + "technique_id": "T1021.008", + "technique": "Remote Services : Direct Cloud VM Connections", + "url": "https://attack.mitre.org/techniques/T1021/008" + }, + "t1070.006": { + "technique_id": "T1070.006", + "technique": "Indicator Removal : Timestomp", + "url": "https://attack.mitre.org/techniques/T1070/006" + }, + "t1016.002": { + "technique_id": "T1016.002", + "technique": "System Network Configuration Discovery : Wi-Fi Discovery", + "url": "https://attack.mitre.org/techniques/T1016/002" + }, + "t1564.011": { + "technique_id": "T1564.011", + "technique": "Hide Artifacts : Ignore Process Interrupts", + "url": "https://attack.mitre.org/techniques/T1564/011" + }, + "t1547.009": { + "technique_id": "T1547.009", + "technique": "Boot or Logon Autostart Execution : Shortcut Modification", + "url": "https://attack.mitre.org/techniques/T1547/009" + }, + "t1087.003": { + "technique_id": "T1087.003", + "technique": "Account Discovery : Email Account", + "url": "https://attack.mitre.org/techniques/T1087/003" + }, + "t1497.003": { + "technique_id": "T1497.003", + "technique": "Virtualization/Sandbox Evasion : Time Based Evasion", + "url": "https://attack.mitre.org/techniques/T1497/003" + }, + "t1218.003": { + "technique_id": "T1218.003", + "technique": "System Binary Proxy Execution : CMSTP", + "url": "https://attack.mitre.org/techniques/T1218/003" + }, + "t1563.001": { + "technique_id": "T1563.001", + "technique": "Remote Service Session Hijacking : SSH Hijacking", + "url": "https://attack.mitre.org/techniques/T1563/001" + }, + "t1562.002": { + "technique_id": "T1562.002", + "technique": "Impair Defenses : Disable Windows Event Logging", + "url": "https://attack.mitre.org/techniques/T1562/002" + }, + "t1021.002": { + "technique_id": "T1021.002", + "technique": "Remote Services : SMB/Windows Admin Shares", + "url": "https://attack.mitre.org/techniques/T1021/002" + }, + "t1218.002": { + "technique_id": "T1218.002", + "technique": "System Binary Proxy Execution : Control Panel", + "url": "https://attack.mitre.org/techniques/T1218/002" + }, + "t1599.001": { + "technique_id": "T1599.001", + "technique": "Network Boundary Bridging : Network Address Translation Traversal", + "url": "https://attack.mitre.org/techniques/T1599/001" + }, + "t1608.002": { + "technique_id": "T1608.002", + "technique": "Stage Capabilities : Upload Tool", + "url": "https://attack.mitre.org/techniques/T1608/002" + }, + "t1547.005": { + "technique_id": "T1547.005", + "technique": "Boot or Logon Autostart Execution : Security Support Provider", + "url": "https://attack.mitre.org/techniques/T1547/005" + }, + "t1597.001": { + "technique_id": "T1597.001", + "technique": "Search Closed Sources : Threat Intel Vendors", + "url": "https://attack.mitre.org/techniques/T1597/001" + }, + "t1602.002": { + "technique_id": "T1602.002", + "technique": "Data from Configuration Repository : Network Device Configuration Dump", + "url": "https://attack.mitre.org/techniques/T1602/002" + }, + "t1562.004": { + "technique_id": "T1562.004", + "technique": "Impair Defenses : Disable or Modify System Firewall", + "url": "https://attack.mitre.org/techniques/T1562/004" + }, + "t1553.003": { + "technique_id": "T1553.003", + "technique": "Subvert Trust Controls : SIP and Trust Provider Hijacking", + "url": "https://attack.mitre.org/techniques/T1553/003" + }, + "t1071.003": { + "technique_id": "T1071.003", + "technique": "Application Layer Protocol : Mail Protocols", + "url": "https://attack.mitre.org/techniques/T1071/003" + }, + "t1556.007": { + "technique_id": "T1556.007", + "technique": "Modify Authentication Process : Hybrid Identity", + "url": "https://attack.mitre.org/techniques/T1556/007" + }, + "t1595.002": { + "technique_id": "T1595.002", + "technique": "Active Scanning : Vulnerability Scanning", + "url": "https://attack.mitre.org/techniques/T1595/002" + }, + "t1059.009": { + "technique_id": "T1059.009", + "technique": "Command and Scripting Interpreter : Cloud API", + "url": "https://attack.mitre.org/techniques/T1059/009" + }, + "t1562.012": { + "technique_id": "T1562.012", + "technique": "Impair Defenses : Disable or Modify Linux Audit System", + "url": "https://attack.mitre.org/techniques/T1562/012" + }, + "t1553.006": { + "technique_id": "T1553.006", + "technique": "Subvert Trust Controls : Code Signing Policy Modification", + "url": "https://attack.mitre.org/techniques/T1553/006" + }, + "t1543.004": { + "technique_id": "T1543.004", + "technique": "Create or Modify System Process : Launch Daemon", + "url": "https://attack.mitre.org/techniques/T1543/004" + }, + "t1555.003": { + "technique_id": "T1555.003", + "technique": "Credentials from Password Stores : Credentials from Web Browsers", + "url": "https://attack.mitre.org/techniques/T1555/003" + }, + "t1574.008": { + "technique_id": "T1574.008", + "technique": "Hijack Execution Flow : Path Interception by Search Order Hijacking", + "url": "https://attack.mitre.org/techniques/T1574/008" + }, + "t1557.003": { + "technique_id": "T1557.003", + "technique": "Adversary-in-the-Middle : DHCP Spoofing", + "url": "https://attack.mitre.org/techniques/T1557/003" + }, + "t1027.001": { + "technique_id": "T1027.001", + "technique": "Obfuscated Files or Information : Binary Padding", + "url": "https://attack.mitre.org/techniques/T1027/001" + }, + "t1505.003": { + "technique_id": "T1505.003", + "technique": "Server Software Component : Web Shell", + "url": "https://attack.mitre.org/techniques/T1505/003" + }, + "t1484.001": { + "technique_id": "T1484.001", + "technique": "Domain Policy Modification : Group Policy Modification", + "url": "https://attack.mitre.org/techniques/T1484/001" + }, + "t1552.004": { + "technique_id": "T1552.004", + "technique": "Unsecured Credentials : Private Keys", + "url": "https://attack.mitre.org/techniques/T1552/004" + }, + "t1583.004": { + "technique_id": "T1583.004", + "technique": "Acquire Infrastructure : Server", + "url": "https://attack.mitre.org/techniques/T1583/004" + }, + "t1021.006": { + "technique_id": "T1021.006", + "technique": "Remote Services : Windows Remote Management", + "url": "https://attack.mitre.org/techniques/T1021/006" + }, + "t1011.001": { + "technique_id": "T1011.001", + "technique": "Exfiltration Over Other Network Medium : Exfiltration Over Bluetooth", + "url": "https://attack.mitre.org/techniques/T1011/001" + }, + "t1078.001": { + "technique_id": "T1078.001", + "technique": "Valid Accounts : Default Accounts", + "url": "https://attack.mitre.org/techniques/T1078/001" + }, + "t1547.003": { + "technique_id": "T1547.003", + "technique": "Boot or Logon Autostart Execution : Time Providers", + "url": "https://attack.mitre.org/techniques/T1547/003" + }, + "t1546.005": { + "technique_id": "T1546.005", + "technique": "Event Triggered Execution : Trap", + "url": "https://attack.mitre.org/techniques/T1546/005" + }, + "t1574.006": { + "technique_id": "T1574.006", + "technique": "Hijack Execution Flow : Dynamic Linker Hijacking", + "url": "https://attack.mitre.org/techniques/T1574/006" + }, + "t1136.001": { + "technique_id": "T1136.001", + "technique": "Create Account : Local Account", + "url": "https://attack.mitre.org/techniques/T1136/001" + }, + "t1070.001": { + "technique_id": "T1070.001", + "technique": "Indicator Removal : Clear Windows Event Logs", + "url": "https://attack.mitre.org/techniques/T1070/001" + }, + "t1585.002": { + "technique_id": "T1585.002", + "technique": "Establish Accounts : Email Accounts", + "url": "https://attack.mitre.org/techniques/T1585/002" + }, + "t1557.001": { + "technique_id": "T1557.001", + "technique": "Adversary-in-the-Middle : LLMNR/NBT-NS Poisoning and SMB Relay", + "url": "https://attack.mitre.org/techniques/T1557/001" + }, + "t1003.001": { + "technique_id": "T1003.001", + "technique": "OS Credential Dumping : LSASS Memory", + "url": "https://attack.mitre.org/techniques/T1003/001" + }, + "t1134.002": { + "technique_id": "T1134.002", + "technique": "Access Token Manipulation : Create Process with Token", + "url": "https://attack.mitre.org/techniques/T1134/002" + }, + "t1548.001": { + "technique_id": "T1548.001", + "technique": "Abuse Elevation Control Mechanism : Setuid and Setgid", + "url": "https://attack.mitre.org/techniques/T1548/001" + }, + "t1547.004": { + "technique_id": "T1547.004", + "technique": "Boot or Logon Autostart Execution : Winlogon Helper DLL", + "url": "https://attack.mitre.org/techniques/T1547/004" + }, + "t1021.003": { + "technique_id": "T1021.003", + "technique": "Remote Services : Distributed Component Object Model", + "url": "https://attack.mitre.org/techniques/T1021/003" + }, + "t1110.003": { + "technique_id": "T1110.003", + "technique": "Brute Force : Password Spraying", + "url": "https://attack.mitre.org/techniques/T1110/003" + }, + "t1090.002": { + "technique_id": "T1090.002", + "technique": "Proxy : External Proxy", + "url": "https://attack.mitre.org/techniques/T1090/002" + }, + "t1056.003": { + "technique_id": "T1056.003", + "technique": "Input Capture : Web Portal Capture", + "url": "https://attack.mitre.org/techniques/T1056/003" + }, + "t1589.002": { + "technique_id": "T1589.002", + "technique": "Gather Victim Identity Information : Email Addresses", + "url": "https://attack.mitre.org/techniques/T1589/002" + }, + "t1598.004": { + "technique_id": "T1598.004", + "technique": "Phishing for Information : Spearphishing Voice", + "url": "https://attack.mitre.org/techniques/T1598/004" + }, + "t1003.005": { + "technique_id": "T1003.005", + "technique": "OS Credential Dumping : Cached Domain Credentials", + "url": "https://attack.mitre.org/techniques/T1003/005" + }, + "t1098.004": { + "technique_id": "T1098.004", + "technique": "Account Manipulation : SSH Authorized Keys", + "url": "https://attack.mitre.org/techniques/T1098/004" + }, + "t1590.006": { + "technique_id": "T1590.006", + "technique": "Gather Victim Network Information : Network Security Appliances", + "url": "https://attack.mitre.org/techniques/T1590/006" + }, + "t1546.012": { + "technique_id": "T1546.012", + "technique": "Event Triggered Execution : Image File Execution Options Injection", + "url": "https://attack.mitre.org/techniques/T1546/012" + }, + "t1218.008": { + "technique_id": "T1218.008", + "technique": "System Binary Proxy Execution : Odbcconf", + "url": "https://attack.mitre.org/techniques/T1218/008" + }, + "t1593.002": { + "technique_id": "T1593.002", + "technique": "Search Open Websites/Domains : Search Engines", + "url": "https://attack.mitre.org/techniques/T1593/002" + }, + "t1591.002": { + "technique_id": "T1591.002", + "technique": "Gather Victim Org Information : Business Relationships", + "url": "https://attack.mitre.org/techniques/T1591/002" + }, + "t1548.005": { + "technique_id": "T1548.005", + "technique": "Abuse Elevation Control Mechanism : Temporary Elevated Cloud Access", + "url": "https://attack.mitre.org/techniques/T1548/005" + }, + "t1055.013": { + "technique_id": "T1055.013", + "technique": "Process Injection : Process Doppelg\u00e4nging", + "url": "https://attack.mitre.org/techniques/T1055/013" + }, + "t1578.003": { + "technique_id": "T1578.003", + "technique": "Modify Cloud Compute Infrastructure : Delete Cloud Instance", + "url": "https://attack.mitre.org/techniques/T1578/003" + }, + "t1593.003": { + "technique_id": "T1593.003", + "technique": "Search Open Websites/Domains : Code Repositories", + "url": "https://attack.mitre.org/techniques/T1593/003" + }, + "t1574.005": { + "technique_id": "T1574.005", + "technique": "Hijack Execution Flow : Executable Installer File Permissions Weakness", + "url": "https://attack.mitre.org/techniques/T1574/005" + }, + "t1546.008": { + "technique_id": "T1546.008", + "technique": "Event Triggered Execution : Accessibility Features", + "url": "https://attack.mitre.org/techniques/T1546/008" + }, + "t1562.006": { + "technique_id": "T1562.006", + "technique": "Impair Defenses : Indicator Blocking", + "url": "https://attack.mitre.org/techniques/T1562/006" + }, + "t1136.002": { + "technique_id": "T1136.002", + "technique": "Create Account : Domain Account", + "url": "https://attack.mitre.org/techniques/T1136/002" + }, + "t1589.003": { + "technique_id": "T1589.003", + "technique": "Gather Victim Identity Information : Employee Names", + "url": "https://attack.mitre.org/techniques/T1589/003" + }, + "t1558.001": { + "technique_id": "T1558.001", + "technique": "Steal or Forge Kerberos Tickets : Golden Ticket", + "url": "https://attack.mitre.org/techniques/T1558/001" + }, + "t1592.004": { + "technique_id": "T1592.004", + "technique": "Gather Victim Host Information : Client Configurations", + "url": "https://attack.mitre.org/techniques/T1592/004" + }, + "t1562.007": { + "technique_id": "T1562.007", + "technique": "Impair Defenses : Disable or Modify Cloud Firewall", + "url": "https://attack.mitre.org/techniques/T1562/007" + }, + "t1036.002": { + "technique_id": "T1036.002", + "technique": "Masquerading : Right-to-Left Override", + "url": "https://attack.mitre.org/techniques/T1036/002" + }, + "t1588.001": { + "technique_id": "T1588.001", + "technique": "Obtain Capabilities : Malware", + "url": "https://attack.mitre.org/techniques/T1588/001" + }, + "t1542.002": { + "technique_id": "T1542.002", + "technique": "Pre-OS Boot : Component Firmware", + "url": "https://attack.mitre.org/techniques/T1542/002" + }, + "t1048.001": { + "technique_id": "T1048.001", + "technique": "Exfiltration Over Alternative Protocol : Exfiltration Over Symmetric Encrypted Non-C2 Protocol", + "url": "https://attack.mitre.org/techniques/T1048/001" + }, + "t1137.001": { + "technique_id": "T1137.001", + "technique": "Office Application Startup : Office Template Macros", + "url": "https://attack.mitre.org/techniques/T1137/001" + }, + "t1583.003": { + "technique_id": "T1583.003", + "technique": "Acquire Infrastructure : Virtual Private Server", + "url": "https://attack.mitre.org/techniques/T1583/003" + }, + "t1213.001": { + "technique_id": "T1213.001", + "technique": "Data from Information Repositories : Confluence", + "url": "https://attack.mitre.org/techniques/T1213/001" + }, + "t1550.003": { + "technique_id": "T1550.003", + "technique": "Use Alternate Authentication Material : Pass the Ticket", + "url": "https://attack.mitre.org/techniques/T1550/003" + }, + "t1036.004": { + "technique_id": "T1036.004", + "technique": "Masquerading : Masquerade Task or Service", + "url": "https://attack.mitre.org/techniques/T1036/004" + }, + "t1055.004": { + "technique_id": "T1055.004", + "technique": "Process Injection : Asynchronous Procedure Call", + "url": "https://attack.mitre.org/techniques/T1055/004" + }, + "t1020.001": { + "technique_id": "T1020.001", + "technique": "Automated Exfiltration : Traffic Duplication", + "url": "https://attack.mitre.org/techniques/T1020/001" + }, + "t1546.009": { + "technique_id": "T1546.009", + "technique": "Event Triggered Execution : AppCert DLLs", + "url": "https://attack.mitre.org/techniques/T1546/009" + }, + "t1114.003": { + "technique_id": "T1114.003", + "technique": "Email Collection : Email Forwarding Rule", + "url": "https://attack.mitre.org/techniques/T1114/003" + }, + "t1098.005": { + "technique_id": "T1098.005", + "technique": "Account Manipulation : Device Registration", + "url": "https://attack.mitre.org/techniques/T1098/005" + }, + "t1553.005": { + "technique_id": "T1553.005", + "technique": "Subvert Trust Controls : Mark-of-the-Web Bypass", + "url": "https://attack.mitre.org/techniques/T1553/005" + }, + "t1600.002": { + "technique_id": "T1600.002", + "technique": "Weaken Encryption : Disable Crypto Hardware", + "url": "https://attack.mitre.org/techniques/T1600/002" + }, + "t1055.002": { + "technique_id": "T1055.002", + "technique": "Process Injection : Portable Executable Injection", + "url": "https://attack.mitre.org/techniques/T1055/002" + }, + "t1218.012": { + "technique_id": "T1218.012", + "technique": "System Binary Proxy Execution : Verclsid", + "url": "https://attack.mitre.org/techniques/T1218/012" + }, + "t1569.001": { + "technique_id": "T1569.001", + "technique": "System Services : Launchctl", + "url": "https://attack.mitre.org/techniques/T1569/001" + }, + "t1584.005": { + "technique_id": "T1584.005", + "technique": "Compromise Infrastructure : Botnet", + "url": "https://attack.mitre.org/techniques/T1584/005" + }, + "t1059.008": { + "technique_id": "T1059.008", + "technique": "Command and Scripting Interpreter : Network Device CLI", + "url": "https://attack.mitre.org/techniques/T1059/008" + }, + "t1552.003": { + "technique_id": "T1552.003", + "technique": "Unsecured Credentials : Bash History", + "url": "https://attack.mitre.org/techniques/T1552/003" + }, + "t1562.010": { + "technique_id": "T1562.010", + "technique": "Impair Defenses : Downgrade Attack", + "url": "https://attack.mitre.org/techniques/T1562/010" + }, + "t1559.003": { + "technique_id": "T1559.003", + "technique": "Inter-Process Communication : XPC Services", + "url": "https://attack.mitre.org/techniques/T1559/003" + }, + "t1552.001": { + "technique_id": "T1552.001", + "technique": "Unsecured Credentials : Credentials In Files", + "url": "https://attack.mitre.org/techniques/T1552/001" + }, + "t1568.003": { + "technique_id": "T1568.003", + "technique": "Dynamic Resolution : DNS Calculation", + "url": "https://attack.mitre.org/techniques/T1568/003" + }, + "t1218.005": { + "technique_id": "T1218.005", + "technique": "System Binary Proxy Execution : Mshta", + "url": "https://attack.mitre.org/techniques/T1218/005" + }, + "t1547.015": { + "technique_id": "T1547.015", + "technique": "Boot or Logon Autostart Execution : Login Items", + "url": "https://attack.mitre.org/techniques/T1547/015" + }, + "t1608.005": { + "technique_id": "T1608.005", + "technique": "Stage Capabilities : Link Target", + "url": "https://attack.mitre.org/techniques/T1608/005" + }, + "t1606.001": { + "technique_id": "T1606.001", + "technique": "Forge Web Credentials : Web Cookies", + "url": "https://attack.mitre.org/techniques/T1606/001" + }, + "t1134.001": { + "technique_id": "T1134.001", + "technique": "Access Token Manipulation : Token Impersonation/Theft", + "url": "https://attack.mitre.org/techniques/T1134/001" + }, + "t1567.001": { + "technique_id": "T1567.001", + "technique": "Exfiltration Over Web Service : Exfiltration to Code Repository", + "url": "https://attack.mitre.org/techniques/T1567/001" + }, + "t1021.007": { + "technique_id": "T1021.007", + "technique": "Remote Services : Cloud Services", + "url": "https://attack.mitre.org/techniques/T1021/007" + }, + "t1205.001": { + "technique_id": "T1205.001", + "technique": "Traffic Signaling : Port Knocking", + "url": "https://attack.mitre.org/techniques/T1205/001" + }, + "t1027.012": { + "technique_id": "T1027.012", + "technique": "Obfuscated Files or Information : LNK Icon Smuggling", + "url": "https://attack.mitre.org/techniques/T1027/012" + }, + "t1583.006": { + "technique_id": "T1583.006", + "technique": "Acquire Infrastructure : Web Services", + "url": "https://attack.mitre.org/techniques/T1583/006" + }, + "t1598.002": { + "technique_id": "T1598.002", + "technique": "Phishing for Information : Spearphishing Attachment", + "url": "https://attack.mitre.org/techniques/T1598/002" + }, + "t1098.001": { + "technique_id": "T1098.001", + "technique": "Account Manipulation : Additional Cloud Credentials", + "url": "https://attack.mitre.org/techniques/T1098/001" + }, + "t1491.001": { + "technique_id": "T1491.001", + "technique": "Defacement : Internal Defacement", + "url": "https://attack.mitre.org/techniques/T1491/001" + }, + "t1564.002": { + "technique_id": "T1564.002", + "technique": "Hide Artifacts : Hidden Users", + "url": "https://attack.mitre.org/techniques/T1564/002" + }, + "t1134.003": { + "technique_id": "T1134.003", + "technique": "Access Token Manipulation : Make and Impersonate Token", + "url": "https://attack.mitre.org/techniques/T1134/003" + }, + "t1552.006": { + "technique_id": "T1552.006", + "technique": "Unsecured Credentials : Group Policy Preferences", + "url": "https://attack.mitre.org/techniques/T1552/006" + }, + "t1048.002": { + "technique_id": "T1048.002", + "technique": "Exfiltration Over Alternative Protocol : Exfiltration Over Asymmetric Encrypted Non-C2 Protocol", + "url": "https://attack.mitre.org/techniques/T1048/002" + }, + "t1087.004": { + "technique_id": "T1087.004", + "technique": "Account Discovery : Cloud Account", + "url": "https://attack.mitre.org/techniques/T1087/004" + }, + "t1562.003": { + "technique_id": "T1562.003", + "technique": "Impair Defenses : Impair Command History Logging", + "url": "https://attack.mitre.org/techniques/T1562/003" + }, + "t1556.008": { + "technique_id": "T1556.008", + "technique": "Modify Authentication Process : Network Provider DLL", + "url": "https://attack.mitre.org/techniques/T1556/008" + }, + "t1546.003": { + "technique_id": "T1546.003", + "technique": "Event Triggered Execution : Windows Management Instrumentation Event Subscription", + "url": "https://attack.mitre.org/techniques/T1546/003" + }, + "t1596.004": { + "technique_id": "T1596.004", + "technique": "Search Open Technical Databases : CDNs", + "url": "https://attack.mitre.org/techniques/T1596/004" + }, + "t1497.002": { + "technique_id": "T1497.002", + "technique": "Virtualization/Sandbox Evasion : User Activity Based Checks", + "url": "https://attack.mitre.org/techniques/T1497/002" + }, + "t1585.003": { + "technique_id": "T1585.003", + "technique": "Establish Accounts : Cloud Accounts", + "url": "https://attack.mitre.org/techniques/T1585/003" + }, + "t1134.004": { + "technique_id": "T1134.004", + "technique": "Access Token Manipulation : Parent PID Spoofing", + "url": "https://attack.mitre.org/techniques/T1134/004" + }, + "t1552.008": { + "technique_id": "T1552.008", + "technique": "Unsecured Credentials : Chat Messages", + "url": "https://attack.mitre.org/techniques/T1552/008" + }, + "t1059.001": { + "technique_id": "T1059.001", + "technique": "Command and Scripting Interpreter : PowerShell", + "url": "https://attack.mitre.org/techniques/T1059/001" + }, + "t1546.001": { + "technique_id": "T1546.001", + "technique": "Event Triggered Execution : Change Default File Association", + "url": "https://attack.mitre.org/techniques/T1546/001" + }, + "t1055.014": { + "technique_id": "T1055.014", + "technique": "Process Injection : VDSO Hijacking", + "url": "https://attack.mitre.org/techniques/T1055/014" + }, + "t1071.002": { + "technique_id": "T1071.002", + "technique": "Application Layer Protocol : File Transfer Protocols", + "url": "https://attack.mitre.org/techniques/T1071/002" + }, + "t1546.014": { + "technique_id": "T1546.014", + "technique": "Event Triggered Execution : Emond", + "url": "https://attack.mitre.org/techniques/T1546/014" + }, + "t1102.003": { + "technique_id": "T1102.003", + "technique": "Web Service : One-Way Communication", + "url": "https://attack.mitre.org/techniques/T1102/003" + }, + "t1574.010": { + "technique_id": "T1574.010", + "technique": "Hijack Execution Flow : Services File Permissions Weakness", + "url": "https://attack.mitre.org/techniques/T1574/010" + }, + "t1547.001": { + "technique_id": "T1547.001", + "technique": "Boot or Logon Autostart Execution : Registry Run Keys / Startup Folder", + "url": "https://attack.mitre.org/techniques/T1547/001" + }, + "t1136.003": { + "technique_id": "T1136.003", + "technique": "Create Account : Cloud Account", + "url": "https://attack.mitre.org/techniques/T1136/003" + }, + "t1069.001": { + "technique_id": "T1069.001", + "technique": "Permission Groups Discovery : Local Groups", + "url": "https://attack.mitre.org/techniques/T1069/001" + }, + "t1547.006": { + "technique_id": "T1547.006", + "technique": "Boot or Logon Autostart Execution : Kernel Modules and Extensions", + "url": "https://attack.mitre.org/techniques/T1547/006" + }, + "t1056.002": { + "technique_id": "T1056.002", + "technique": "Input Capture : GUI Input Capture", + "url": "https://attack.mitre.org/techniques/T1056/002" + }, + "t1588.002": { + "technique_id": "T1588.002", + "technique": "Obtain Capabilities : Tool", + "url": "https://attack.mitre.org/techniques/T1588/002" + }, + "t1052.001": { + "technique_id": "T1052.001", + "technique": "Exfiltration Over Physical Medium : Exfiltration over USB", + "url": "https://attack.mitre.org/techniques/T1052/001" + }, + "t1574.013": { + "technique_id": "T1574.013", + "technique": "Hijack Execution Flow : KernelCallbackTable", + "url": "https://attack.mitre.org/techniques/T1574/013" + }, + "t1053.006": { + "technique_id": "T1053.006", + "technique": "Scheduled Task/Job : Systemd Timers", + "url": "https://attack.mitre.org/techniques/T1053/006" + }, + "t1542.004": { + "technique_id": "T1542.004", + "technique": "Pre-OS Boot : ROMMONkit", + "url": "https://attack.mitre.org/techniques/T1542/004" + }, + "t1218.001": { + "technique_id": "T1218.001", + "technique": "System Binary Proxy Execution : Compiled HTML File", + "url": "https://attack.mitre.org/techniques/T1218/001" + }, + "t1070.005": { + "technique_id": "T1070.005", + "technique": "Indicator Removal : Network Share Connection Removal", + "url": "https://attack.mitre.org/techniques/T1070/005" + }, + "t1090.003": { + "technique_id": "T1090.003", + "technique": "Proxy : Multi-hop Proxy", + "url": "https://attack.mitre.org/techniques/T1090/003" + }, + "t1059.004": { + "technique_id": "T1059.004", + "technique": "Command and Scripting Interpreter : Unix Shell", + "url": "https://attack.mitre.org/techniques/T1059/004" + }, + "t1137.003": { + "technique_id": "T1137.003", + "technique": "Office Application Startup : Outlook Forms", + "url": "https://attack.mitre.org/techniques/T1137/003" + }, + "t1562.001": { + "technique_id": "T1562.001", + "technique": "Impair Defenses : Disable or Modify Tools", + "url": "https://attack.mitre.org/techniques/T1562/001" + }, + "t1584.006": { + "technique_id": "T1584.006", + "technique": "Compromise Infrastructure : Web Services", + "url": "https://attack.mitre.org/techniques/T1584/006" + }, + "t1027.005": { + "technique_id": "T1027.005", + "technique": "Obfuscated Files or Information : Indicator Removal from Tools", + "url": "https://attack.mitre.org/techniques/T1027/005" + }, + "t1204.003": { + "technique_id": "T1204.003", + "technique": "User Execution : Malicious Image", + "url": "https://attack.mitre.org/techniques/T1204/003" + }, + "t1585.001": { + "technique_id": "T1585.001", + "technique": "Establish Accounts : Social Media Accounts", + "url": "https://attack.mitre.org/techniques/T1585/001" + }, + "t1055.012": { + "technique_id": "T1055.012", + "technique": "Process Injection : Process Hollowing", + "url": "https://attack.mitre.org/techniques/T1055/012" + }, + "t1564.009": { + "technique_id": "T1564.009", + "technique": "Hide Artifacts : Resource Forking", + "url": "https://attack.mitre.org/techniques/T1564/009" + }, + "t1110.004": { + "technique_id": "T1110.004", + "technique": "Brute Force : Credential Stuffing", + "url": "https://attack.mitre.org/techniques/T1110/004" + }, + "t1556.006": { + "technique_id": "T1556.006", + "technique": "Modify Authentication Process : Multi-Factor Authentication", + "url": "https://attack.mitre.org/techniques/T1556/006" + }, + "t1114.002": { + "technique_id": "T1114.002", + "technique": "Email Collection : Remote Email Collection", + "url": "https://attack.mitre.org/techniques/T1114/002" + }, + "t1505.004": { + "technique_id": "T1505.004", + "technique": "Server Software Component : IIS Components", + "url": "https://attack.mitre.org/techniques/T1505/004" + }, + "t1036.001": { + "technique_id": "T1036.001", + "technique": "Masquerading : Invalid Code Signature", + "url": "https://attack.mitre.org/techniques/T1036/001" + }, + "t1564.006": { + "technique_id": "T1564.006", + "technique": "Hide Artifacts : Run Virtual Instance", + "url": "https://attack.mitre.org/techniques/T1564/006" + }, + "t1546.004": { + "technique_id": "T1546.004", + "technique": "Event Triggered Execution : Unix Shell Configuration Modification", + "url": "https://attack.mitre.org/techniques/T1546/004" + }, + "t1134.005": { + "technique_id": "T1134.005", + "technique": "Access Token Manipulation : SID-History Injection", + "url": "https://attack.mitre.org/techniques/T1134/005" + }, + "t1548.004": { + "technique_id": "T1548.004", + "technique": "Abuse Elevation Control Mechanism : Elevated Execution with Prompt", + "url": "https://attack.mitre.org/techniques/T1548/004" + }, + "t1592.003": { + "technique_id": "T1592.003", + "technique": "Gather Victim Host Information : Firmware", + "url": "https://attack.mitre.org/techniques/T1592/003" + }, + "t1547.002": { + "technique_id": "T1547.002", + "technique": "Boot or Logon Autostart Execution : Authentication Package", + "url": "https://attack.mitre.org/techniques/T1547/002" + }, + "t1218.010": { + "technique_id": "T1218.010", + "technique": "System Binary Proxy Execution : Regsvr32", + "url": "https://attack.mitre.org/techniques/T1218/010" + }, + "t1567.003": { + "technique_id": "T1567.003", + "technique": "Exfiltration Over Web Service : Exfiltration to Text Storage Sites", + "url": "https://attack.mitre.org/techniques/T1567/003" + }, + "t1592.002": { + "technique_id": "T1592.002", + "technique": "Gather Victim Host Information : Software", + "url": "https://attack.mitre.org/techniques/T1592/002" + }, + "t1566.004": { + "technique_id": "T1566.004", + "technique": "Phishing : Spearphishing Voice", + "url": "https://attack.mitre.org/techniques/T1566/004" + }, + "t1587.004": { + "technique_id": "T1587.004", + "technique": "Develop Capabilities : Exploits", + "url": "https://attack.mitre.org/techniques/T1587/004" + }, + "t1593.001": { + "technique_id": "T1593.001", + "technique": "Search Open Websites/Domains : Social Media", + "url": "https://attack.mitre.org/techniques/T1593/001" + }, + "t1546.015": { + "technique_id": "T1546.015", + "technique": "Event Triggered Execution : Component Object Model Hijacking", + "url": "https://attack.mitre.org/techniques/T1546/015" + }, + "t1589.001": { + "technique_id": "T1589.001", + "technique": "Gather Victim Identity Information : Credentials", + "url": "https://attack.mitre.org/techniques/T1589/001" + }, + "t1195.002": { + "technique_id": "T1195.002", + "technique": "Supply Chain Compromise : Compromise Software Supply Chain", + "url": "https://attack.mitre.org/techniques/T1195/002" + }, + "t1036.003": { + "technique_id": "T1036.003", + "technique": "Masquerading : Rename System Utilities", + "url": "https://attack.mitre.org/techniques/T1036/003" + }, + "t1102.002": { + "technique_id": "T1102.002", + "technique": "Web Service : Bidirectional Communication", + "url": "https://attack.mitre.org/techniques/T1102/002" + }, + "t1595.003": { + "technique_id": "T1595.003", + "technique": "Active Scanning : Wordlist Scanning", + "url": "https://attack.mitre.org/techniques/T1595/003" + }, + "t1562.011": { + "technique_id": "T1562.011", + "technique": "Impair Defenses : Spoof Security Alerting", + "url": "https://attack.mitre.org/techniques/T1562/011" + }, + "t1137.004": { + "technique_id": "T1137.004", + "technique": "Office Application Startup : Outlook Home Page", + "url": "https://attack.mitre.org/techniques/T1137/004" + }, + "t1573.002": { + "technique_id": "T1573.002", + "technique": "Encrypted Channel : Asymmetric Cryptography", + "url": "https://attack.mitre.org/techniques/T1573/002" + }, + "t1567.002": { + "technique_id": "T1567.002", + "technique": "Exfiltration Over Web Service : Exfiltration to Cloud Storage", + "url": "https://attack.mitre.org/techniques/T1567/002" + }, + "t1574.009": { + "technique_id": "T1574.009", + "technique": "Hijack Execution Flow : Path Interception by Unquoted Path", + "url": "https://attack.mitre.org/techniques/T1574/009" + }, + "t1608.003": { + "technique_id": "T1608.003", + "technique": "Stage Capabilities : Install Digital Certificate", + "url": "https://attack.mitre.org/techniques/T1608/003" + }, + "t1037.005": { + "technique_id": "T1037.005", + "technique": "Boot or Logon Initialization Scripts : Startup Items", + "url": "https://attack.mitre.org/techniques/T1037/005" + }, + "t1614.001": { + "technique_id": "T1614.001", + "technique": "System Location Discovery : System Language Discovery", + "url": "https://attack.mitre.org/techniques/T1614/001" + }, + "t1027.003": { + "technique_id": "T1027.003", + "technique": "Obfuscated Files or Information : Steganography", + "url": "https://attack.mitre.org/techniques/T1027/003" + }, + "t1584.002": { + "technique_id": "T1584.002", + "technique": "Compromise Infrastructure : DNS Server", + "url": "https://attack.mitre.org/techniques/T1584/002" + }, + "t1001.003": { + "technique_id": "T1001.003", + "technique": "Data Obfuscation : Protocol Impersonation", + "url": "https://attack.mitre.org/techniques/T1001/003" + }, + "t1550.004": { + "technique_id": "T1550.004", + "technique": "Use Alternate Authentication Material : Web Session Cookie", + "url": "https://attack.mitre.org/techniques/T1550/004" + }, + "t1078.002": { + "technique_id": "T1078.002", + "technique": "Valid Accounts : Domain Accounts", + "url": "https://attack.mitre.org/techniques/T1078/002" + }, + "t1218.009": { + "technique_id": "T1218.009", + "technique": "System Binary Proxy Execution : Regsvcs/Regasm", + "url": "https://attack.mitre.org/techniques/T1218/009" + }, + "t1553.004": { + "technique_id": "T1553.004", + "technique": "Subvert Trust Controls : Install Root Certificate", + "url": "https://attack.mitre.org/techniques/T1553/004" + }, + "t1037.003": { + "technique_id": "T1037.003", + "technique": "Boot or Logon Initialization Scripts : Network Logon Script", + "url": "https://attack.mitre.org/techniques/T1037/003" + }, + "t1027.004": { + "technique_id": "T1027.004", + "technique": "Obfuscated Files or Information : Compile After Delivery", + "url": "https://attack.mitre.org/techniques/T1027/004" + }, + "t1564.007": { + "technique_id": "T1564.007", + "technique": "Hide Artifacts : VBA Stomping", + "url": "https://attack.mitre.org/techniques/T1564/007" + }, + "t1127.001": { + "technique_id": "T1127.001", + "technique": "Trusted Developer Utilities Proxy Execution : MSBuild", + "url": "https://attack.mitre.org/techniques/T1127/001" + }, + "t1578.005": { + "technique_id": "T1578.005", + "technique": "Modify Cloud Compute Infrastructure : Modify Cloud Compute Configurations", + "url": "https://attack.mitre.org/techniques/T1578/005" + }, + "t1090.004": { + "technique_id": "T1090.004", + "technique": "Proxy : Domain Fronting", + "url": "https://attack.mitre.org/techniques/T1090/004" + }, + "t1557.002": { + "technique_id": "T1557.002", + "technique": "Adversary-in-the-Middle : ARP Cache Poisoning", + "url": "https://attack.mitre.org/techniques/T1557/002" + }, + "t1562.008": { + "technique_id": "T1562.008", + "technique": "Impair Defenses : Disable or Modify Cloud Logs", + "url": "https://attack.mitre.org/techniques/T1562/008" + }, + "t1518.001": { + "technique_id": "T1518.001", + "technique": "Software Discovery : Security Software Discovery", + "url": "https://attack.mitre.org/techniques/T1518/001" + }, + "t1564.003": { + "technique_id": "T1564.003", + "technique": "Hide Artifacts : Hidden Window", + "url": "https://attack.mitre.org/techniques/T1564/003" + }, + "t1059.006": { + "technique_id": "T1059.006", + "technique": "Command and Scripting Interpreter : Python", + "url": "https://attack.mitre.org/techniques/T1059/006" + }, + "t1591.004": { + "technique_id": "T1591.004", + "technique": "Gather Victim Org Information : Identify Roles", + "url": "https://attack.mitre.org/techniques/T1591/004" + }, + "t1546.010": { + "technique_id": "T1546.010", + "technique": "Event Triggered Execution : AppInit DLLs", + "url": "https://attack.mitre.org/techniques/T1546/010" + }, + "t1546.002": { + "technique_id": "T1546.002", + "technique": "Event Triggered Execution : Screensaver", + "url": "https://attack.mitre.org/techniques/T1546/002" + }, + "t1578.002": { + "technique_id": "T1578.002", + "technique": "Modify Cloud Compute Infrastructure : Create Cloud Instance", + "url": "https://attack.mitre.org/techniques/T1578/002" + }, + "t1555.006": { + "technique_id": "T1555.006", + "technique": "Credentials from Password Stores : Cloud Secrets Management Stores", + "url": "https://attack.mitre.org/techniques/T1555/006" + }, + "t1213.003": { + "technique_id": "T1213.003", + "technique": "Data from Information Repositories : Code Repositories", + "url": "https://attack.mitre.org/techniques/T1213/003" + }, + "t1565.002": { + "technique_id": "T1565.002", + "technique": "Data Manipulation : Transmitted Data Manipulation", + "url": "https://attack.mitre.org/techniques/T1565/002" + }, + "t1003.008": { + "technique_id": "T1003.008", + "technique": "OS Credential Dumping : /etc/passwd and /etc/shadow", + "url": "https://attack.mitre.org/techniques/T1003/008" + }, + "t1543.001": { + "technique_id": "T1543.001", + "technique": "Create or Modify System Process : Launch Agent", + "url": "https://attack.mitre.org/techniques/T1543/001" + }, + "t1059.003": { + "technique_id": "T1059.003", + "technique": "Command and Scripting Interpreter : Windows Command Shell", + "url": "https://attack.mitre.org/techniques/T1059/003" + }, + "t1055.009": { + "technique_id": "T1055.009", + "technique": "Process Injection : Proc Memory", + "url": "https://attack.mitre.org/techniques/T1055/009" + }, + "t1601.001": { + "technique_id": "T1601.001", + "technique": "Modify System Image : Patch System Image", + "url": "https://attack.mitre.org/techniques/T1601/001" + }, + "t1558.002": { + "technique_id": "T1558.002", + "technique": "Steal or Forge Kerberos Tickets : Silver Ticket", + "url": "https://attack.mitre.org/techniques/T1558/002" + }, + "t1070.009": { + "technique_id": "T1070.009", + "technique": "Indicator Removal : Clear Persistence", + "url": "https://attack.mitre.org/techniques/T1070/009" + }, + "t1555.004": { + "technique_id": "T1555.004", + "technique": "Credentials from Password Stores : Windows Credential Manager", + "url": "https://attack.mitre.org/techniques/T1555/004" + }, + "t1132.002": { + "technique_id": "T1132.002", + "technique": "Data Encoding : Non-Standard Encoding", + "url": "https://attack.mitre.org/techniques/T1132/002" + }, + "t1556.001": { + "technique_id": "T1556.001", + "technique": "Modify Authentication Process : Domain Controller Authentication", + "url": "https://attack.mitre.org/techniques/T1556/001" + }, + "t1027.006": { + "technique_id": "T1027.006", + "technique": "Obfuscated Files or Information : HTML Smuggling", + "url": "https://attack.mitre.org/techniques/T1027/006" + }, + "t1556.005": { + "technique_id": "T1556.005", + "technique": "Modify Authentication Process : Reversible Encryption", + "url": "https://attack.mitre.org/techniques/T1556/005" + }, + "t1027.010": { + "technique_id": "T1027.010", + "technique": "Obfuscated Files or Information : Command Obfuscation", + "url": "https://attack.mitre.org/techniques/T1027/010" + }, + "t1070.004": { + "technique_id": "T1070.004", + "technique": "Indicator Removal : File Deletion", + "url": "https://attack.mitre.org/techniques/T1070/004" + }, + "t1546.016": { + "technique_id": "T1546.016", + "technique": "Event Triggered Execution : Installer Packages", + "url": "https://attack.mitre.org/techniques/T1546/016" + }, + "t1595.001": { + "technique_id": "T1595.001", + "technique": "Active Scanning : Scanning IP Blocks", + "url": "https://attack.mitre.org/techniques/T1595/001" + }, + "t1037.004": { + "technique_id": "T1037.004", + "technique": "Boot or Logon Initialization Scripts : RC Scripts", + "url": "https://attack.mitre.org/techniques/T1037/004" + }, + "t1027.002": { + "technique_id": "T1027.002", + "technique": "Obfuscated Files or Information : Software Packing", + "url": "https://attack.mitre.org/techniques/T1027/002" + }, + "t1584.007": { + "technique_id": "T1584.007", + "technique": "Compromise Infrastructure : Serverless", + "url": "https://attack.mitre.org/techniques/T1584/007" + }, + "t1071.001": { + "technique_id": "T1071.001", + "technique": "Application Layer Protocol : Web Protocols", + "url": "https://attack.mitre.org/techniques/T1071/001" + }, + "t1059.005": { + "technique_id": "T1059.005", + "technique": "Command and Scripting Interpreter : Visual Basic", + "url": "https://attack.mitre.org/techniques/T1059/005" + }, + "t1564.005": { + "technique_id": "T1564.005", + "technique": "Hide Artifacts : Hidden File System", + "url": "https://attack.mitre.org/techniques/T1564/005" + }, + "t1543.002": { + "technique_id": "T1543.002", + "technique": "Create or Modify System Process : Systemd Service", + "url": "https://attack.mitre.org/techniques/T1543/002" + }, + "t1563.002": { + "technique_id": "T1563.002", + "technique": "Remote Service Session Hijacking : RDP Hijacking", + "url": "https://attack.mitre.org/techniques/T1563/002" + }, + "t1547.013": { + "technique_id": "T1547.013", + "technique": "Boot or Logon Autostart Execution : XDG Autostart Entries", + "url": "https://attack.mitre.org/techniques/T1547/013" + }, + "t1584.004": { + "technique_id": "T1584.004", + "technique": "Compromise Infrastructure : Server", + "url": "https://attack.mitre.org/techniques/T1584/004" + }, + "t1590.001": { + "technique_id": "T1590.001", + "technique": "Gather Victim Network Information : Domain Properties", + "url": "https://attack.mitre.org/techniques/T1590/001" + }, + "t1055.005": { + "technique_id": "T1055.005", + "technique": "Process Injection : Thread Local Storage", + "url": "https://attack.mitre.org/techniques/T1055/005" + }, + "t1036.006": { + "technique_id": "T1036.006", + "technique": "Masquerading : Space after Filename", + "url": "https://attack.mitre.org/techniques/T1036/006" + }, + "t1547.007": { + "technique_id": "T1547.007", + "technique": "Boot or Logon Autostart Execution : Re-opened Applications", + "url": "https://attack.mitre.org/techniques/T1547/007" + }, + "t1608.006": { + "technique_id": "T1608.006", + "technique": "Stage Capabilities : SEO Poisoning", + "url": "https://attack.mitre.org/techniques/T1608/006" + }, + "t1550.002": { + "technique_id": "T1550.002", + "technique": "Use Alternate Authentication Material : Pass the Hash", + "url": "https://attack.mitre.org/techniques/T1550/002" + }, + "t1574.002": { + "technique_id": "T1574.002", + "technique": "Hijack Execution Flow : DLL Side-Loading", + "url": "https://attack.mitre.org/techniques/T1574/002" + }, + "t1098.002": { + "technique_id": "T1098.002", + "technique": "Account Manipulation : Additional Email Delegate Permissions", + "url": "https://attack.mitre.org/techniques/T1098/002" + }, + "t1588.003": { + "technique_id": "T1588.003", + "technique": "Obtain Capabilities : Code Signing Certificates", + "url": "https://attack.mitre.org/techniques/T1588/003" + }, + "t1055.008": { + "technique_id": "T1055.008", + "technique": "Process Injection : Ptrace System Calls", + "url": "https://attack.mitre.org/techniques/T1055/008" + }, + "t1027.007": { + "technique_id": "T1027.007", + "technique": "Obfuscated Files or Information : Dynamic API Resolution", + "url": "https://attack.mitre.org/techniques/T1027/007" + }, + "t1021.001": { + "technique_id": "T1021.001", + "technique": "Remote Services : Remote Desktop Protocol", + "url": "https://attack.mitre.org/techniques/T1021/001" + }, + "t1037.001": { + "technique_id": "T1037.001", + "technique": "Boot or Logon Initialization Scripts : Logon Script (Windows)", + "url": "https://attack.mitre.org/techniques/T1037/001" + }, + "t1055.015": { + "technique_id": "T1055.015", + "technique": "Process Injection : ListPlanting", + "url": "https://attack.mitre.org/techniques/T1055/015" + }, + "t1596.005": { + "technique_id": "T1596.005", + "technique": "Search Open Technical Databases : Scan Databases", + "url": "https://attack.mitre.org/techniques/T1596/005" + }, + "t1564.001": { + "technique_id": "T1564.001", + "technique": "Hide Artifacts : Hidden Files and Directories", + "url": "https://attack.mitre.org/techniques/T1564/001" + }, + "t1578.001": { + "technique_id": "T1578.001", + "technique": "Modify Cloud Compute Infrastructure : Create Snapshot", + "url": "https://attack.mitre.org/techniques/T1578/001" + }, + "t1591.001": { + "technique_id": "T1591.001", + "technique": "Gather Victim Org Information : Determine Physical Locations", + "url": "https://attack.mitre.org/techniques/T1591/001" + }, + "t1137.002": { + "technique_id": "T1137.002", + "technique": "Office Application Startup : Office Test", + "url": "https://attack.mitre.org/techniques/T1137/002" + }, + "t1003.003": { + "technique_id": "T1003.003", + "technique": "OS Credential Dumping : NTDS", + "url": "https://attack.mitre.org/techniques/T1003/003" + }, + "t1602.001": { + "technique_id": "T1602.001", + "technique": "Data from Configuration Repository : SNMP (MIB Dump)", + "url": "https://attack.mitre.org/techniques/T1602/001" + }, + "t1001.002": { + "technique_id": "T1001.002", + "technique": "Data Obfuscation : Steganography", + "url": "https://attack.mitre.org/techniques/T1001/002" + }, + "t1204.001": { + "technique_id": "T1204.001", + "technique": "User Execution : Malicious Link", + "url": "https://attack.mitre.org/techniques/T1204/001" + }, + "t1550.001": { + "technique_id": "T1550.001", + "technique": "Use Alternate Authentication Material : Application Access Token", + "url": "https://attack.mitre.org/techniques/T1550/001" + }, + "t1547.008": { + "technique_id": "T1547.008", + "technique": "Boot or Logon Autostart Execution : LSASS Driver", + "url": "https://attack.mitre.org/techniques/T1547/008" + }, + "t1569.002": { + "technique_id": "T1569.002", + "technique": "System Services : Service Execution", + "url": "https://attack.mitre.org/techniques/T1569/002" + }, + "t1078.004": { + "technique_id": "T1078.004", + "technique": "Valid Accounts : Cloud Accounts", + "url": "https://attack.mitre.org/techniques/T1078/004" + }, + "t1480.001": { + "technique_id": "T1480.001", + "technique": "Execution Guardrails : Environmental Keying", + "url": "https://attack.mitre.org/techniques/T1480/001" + }, + "t1564.004": { + "technique_id": "T1564.004", + "technique": "Hide Artifacts : NTFS File Attributes", + "url": "https://attack.mitre.org/techniques/T1564/004" + }, + "t1558.003": { + "technique_id": "T1558.003", + "technique": "Steal or Forge Kerberos Tickets : Kerberoasting", + "url": "https://attack.mitre.org/techniques/T1558/003" + }, + "t1003.006": { + "technique_id": "T1003.006", + "technique": "OS Credential Dumping : DCSync", + "url": "https://attack.mitre.org/techniques/T1003/006" + }, + "t1053.002": { + "technique_id": "T1053.002", + "technique": "Scheduled Task/Job : At", + "url": "https://attack.mitre.org/techniques/T1053/002" + }, + "t1055.001": { + "technique_id": "T1055.001", + "technique": "Process Injection : Dynamic-link Library Injection", + "url": "https://attack.mitre.org/techniques/T1055/001" + }, + "t1588.005": { + "technique_id": "T1588.005", + "technique": "Obtain Capabilities : Exploits", + "url": "https://attack.mitre.org/techniques/T1588/005" + }, + "t1056.004": { + "technique_id": "T1056.004", + "technique": "Input Capture : Credential API Hooking", + "url": "https://attack.mitre.org/techniques/T1056/004" + }, + "t1546.007": { + "technique_id": "T1546.007", + "technique": "Event Triggered Execution : Netsh Helper DLL", + "url": "https://attack.mitre.org/techniques/T1546/007" + }, + "t1566.003": { + "technique_id": "T1566.003", + "technique": "Phishing : Spearphishing via Service", + "url": "https://attack.mitre.org/techniques/T1566/003" + }, + "t1090.001": { + "technique_id": "T1090.001", + "technique": "Proxy : Internal Proxy", + "url": "https://attack.mitre.org/techniques/T1090/001" + }, + "t1102.001": { + "technique_id": "T1102.001", + "technique": "Web Service : Dead Drop Resolver", + "url": "https://attack.mitre.org/techniques/T1102/001" + }, + "t1001.001": { + "technique_id": "T1001.001", + "technique": "Data Obfuscation : Junk Data", + "url": "https://attack.mitre.org/techniques/T1001/001" + }, + "t1598.001": { + "technique_id": "T1598.001", + "technique": "Phishing for Information : Spearphishing Service", + "url": "https://attack.mitre.org/techniques/T1598/001" + }, + "t1552.007": { + "technique_id": "T1552.007", + "technique": "Unsecured Credentials : Container API", + "url": "https://attack.mitre.org/techniques/T1552/007" + }, + "t1584.001": { + "technique_id": "T1584.001", + "technique": "Compromise Infrastructure : Domains", + "url": "https://attack.mitre.org/techniques/T1584/001" + }, + "t1505.001": { + "technique_id": "T1505.001", + "technique": "Server Software Component : SQL Stored Procedures", + "url": "https://attack.mitre.org/techniques/T1505/001" + }, + "t1556.004": { + "technique_id": "T1556.004", + "technique": "Modify Authentication Process : Network Device Authentication", + "url": "https://attack.mitre.org/techniques/T1556/004" + }, + "t1561.001": { + "technique_id": "T1561.001", + "technique": "Disk Wipe : Disk Content Wipe", + "url": "https://attack.mitre.org/techniques/T1561/001" + }, + "t1048.003": { + "technique_id": "T1048.003", + "technique": "Exfiltration Over Alternative Protocol : Exfiltration Over Unencrypted Non-C2 Protocol", + "url": "https://attack.mitre.org/techniques/T1048/003" + }, + "t1574.004": { + "technique_id": "T1574.004", + "technique": "Hijack Execution Flow : Dylib Hijacking", + "url": "https://attack.mitre.org/techniques/T1574/004" + }, + "t1601.002": { + "technique_id": "T1601.002", + "technique": "Modify System Image : Downgrade System Image", + "url": "https://attack.mitre.org/techniques/T1601/002" + }, + "t1078.003": { + "technique_id": "T1078.003", + "technique": "Valid Accounts : Local Accounts", + "url": "https://attack.mitre.org/techniques/T1078/003" + }, + "t1218.014": { + "technique_id": "T1218.014", + "technique": "System Binary Proxy Execution : MMC", + "url": "https://attack.mitre.org/techniques/T1218/014" + }, + "t1564.010": { + "technique_id": "T1564.010", + "technique": "Hide Artifacts : Process Argument Spoofing", + "url": "https://attack.mitre.org/techniques/T1564/010" + }, + "t1574.012": { + "technique_id": "T1574.012", + "technique": "Hijack Execution Flow : COR_PROFILER", + "url": "https://attack.mitre.org/techniques/T1574/012" + } +} \ No newline at end of file diff --git a/siem-converter/app/dictionaries/uncoder_meta_info_roota.json b/uncoder-core/app/dictionaries/uncoder_meta_info_roota.json similarity index 99% rename from siem-converter/app/dictionaries/uncoder_meta_info_roota.json rename to uncoder-core/app/dictionaries/uncoder_meta_info_roota.json index 09289f96..0ab2a1a4 100644 --- a/siem-converter/app/dictionaries/uncoder_meta_info_roota.json +++ b/uncoder-core/app/dictionaries/uncoder_meta_info_roota.json @@ -55,7 +55,6 @@ { "name": "logscale-lql-query", "description": "Falcon LogScale Query" }, { "name": "mde-kql-query", "description": "Microsoft Defender for Endpoint Query" }, { "name": "qradar-aql-query", "description": "IBM QRadar Query" }, - { "name": "sigma-yml-rule", "description": "Sigma Rule" }, { "name": "athena-sql-query", "description": "AWS Athena Query (Security Lake)" }, { "name": "chronicle-yaral-query", "description": "Chronicle Security Query" } ] diff --git a/siem-converter/app/dictionaries/uncoder_meta_info_sigma.json b/uncoder-core/app/dictionaries/uncoder_meta_info_sigma.json similarity index 100% rename from siem-converter/app/dictionaries/uncoder_meta_info_sigma.json rename to uncoder-core/app/dictionaries/uncoder_meta_info_sigma.json diff --git a/siem-converter/app/converter/__init__.py b/uncoder-core/app/models/__init__.py similarity index 100% rename from siem-converter/app/converter/__init__.py rename to uncoder-core/app/models/__init__.py diff --git a/siem-converter/app/models/ioc_translation.py b/uncoder-core/app/models/ioc_translation.py similarity index 87% rename from siem-converter/app/models/ioc_translation.py rename to uncoder-core/app/models/ioc_translation.py index 18ba5b15..80c0a682 100644 --- a/siem-converter/app/models/ioc_translation.py +++ b/uncoder-core/app/models/ioc_translation.py @@ -6,11 +6,11 @@ class CTIPlatform(BaseModel): - name: str + id: str class OneTranslationCTIData(BaseModel): info: Optional[InfoMessage] = None status: bool translations: Optional[list] = None - target_siem_type: str + target_platform_id: str diff --git a/siem-converter/app/models/translation.py b/uncoder-core/app/models/translation.py similarity index 86% rename from siem-converter/app/models/translation.py rename to uncoder-core/app/models/translation.py index fdd89b7e..42ba1ae6 100644 --- a/siem-converter/app/models/translation.py +++ b/uncoder-core/app/models/translation.py @@ -12,10 +12,10 @@ class OneTranslationData(BaseModel): info: Optional[InfoMessage] = None status: bool translation: Optional[str] = None - target_siem_type: str + target_platform_id: str -class ConvertorPlatform(BaseModel): +class TranslatorPlatform(BaseModel): name: str id: str code: str @@ -28,7 +28,7 @@ class ConvertorPlatform(BaseModel): first_choice: int = 1 -class ConvertorPlatforms(BaseModel): +class TranslatorPlatforms(BaseModel): renders: list parsers: list diff --git a/siem-converter/app/converter/core/__init__.py b/uncoder-core/app/routers/__init__.py similarity index 100% rename from siem-converter/app/converter/core/__init__.py rename to uncoder-core/app/routers/__init__.py diff --git a/uncoder-core/app/routers/assistance.py b/uncoder-core/app/routers/assistance.py new file mode 100644 index 00000000..fb89cdee --- /dev/null +++ b/uncoder-core/app/routers/assistance.py @@ -0,0 +1,37 @@ +import json +import os +from collections.abc import Generator +from contextlib import asynccontextmanager +from datetime import datetime + +from fastapi import APIRouter, FastAPI + +from app.translator.core.mitre import MitreConfig +from const import ROOT_PROJECT_PATH + +assistance_router = APIRouter() + +suggestions = {} + + +@asynccontextmanager +async def lifespan(app: FastAPI) -> Generator[None, None, None]: # noqa: ARG001 + MitreConfig(server=True).update_mitre_config() + with open(os.path.join(ROOT_PROJECT_PATH, "app/dictionaries/uncoder_meta_info_roota.json")) as file: + suggestions["roota"] = json.load(file) + with open(os.path.join(ROOT_PROJECT_PATH, "app/dictionaries/uncoder_meta_info_sigma.json")) as file: + suggestions["sigma"] = json.load(file) + yield + + +@assistance_router.get("/suggestions/{parser_id}", tags=["assistance"], description="Get suggestions") +async def get_suggestions(parser_id: str) -> list[dict]: + parser_dict = suggestions.get(parser_id, []) + if parser_id == "roota": + today = datetime.today().strftime("%Y-%m-%d") + for i in parser_dict: + if i["title"] == "Date": + for v in i["dictionary"]: + v["name"] = today + return parser_dict + return parser_dict diff --git a/uncoder-core/app/routers/ioc_translate.py b/uncoder-core/app/routers/ioc_translate.py new file mode 100644 index 00000000..3e78125d --- /dev/null +++ b/uncoder-core/app/routers/ioc_translate.py @@ -0,0 +1,47 @@ +from typing import Optional + +from fastapi import APIRouter, Body + +from app.models.ioc_translation import CTIPlatform, OneTranslationCTIData +from app.models.translation import InfoMessage +from app.translator.cti_translator import cti_translator +from app.translator.tools.const import HashType, IocParsingRule, IOCType + +iocs_router = APIRouter() + + +@iocs_router.post("/iocs/translate", description="Parse IOCs from text.") +@iocs_router.post("/iocs/translate", include_in_schema=False) +def parse_and_translate_iocs( + text: str = Body(..., description="Text to parse IOCs from", embed=True), + iocs_per_query: int = Body(25, description="IOCs per query limit", embed=True), + platform: CTIPlatform = Body(..., description="Platform to parse IOCs to", embed=True), + include_ioc_types: Optional[list[IOCType]] = Body( + None, description="List of IOC types to include. By default all types are enabled.", embed=True + ), + include_hash_types: Optional[list[HashType]] = Body( + None, description="List of hash types to include. By default all hash types are enabled.", embed=True + ), + exceptions: Optional[list[str]] = Body( + None, description="List of exceptions. IOC is ignored if it contains one of exception values.", embed=True + ), + ioc_parsing_rules: Optional[list[IocParsingRule]] = Body( + None, embed=True, description="Additional parsing parameters." + ), + include_source_ip: Optional[bool] = Body(False, description="Include source IP in query. By default it is false."), +) -> OneTranslationCTIData: + status, translations = cti_translator.translate( + text=text, + platform_data=platform, + iocs_per_query=iocs_per_query, + include_ioc_types=include_ioc_types, + include_hash_types=include_hash_types, + exceptions=exceptions, + ioc_parsing_rules=ioc_parsing_rules, + include_source_ip=include_source_ip, + ) + if status: + return OneTranslationCTIData(status=status, translations=translations, target_platform_id=platform.id) + + info_message = InfoMessage(message=translations, severity="error") + return OneTranslationCTIData(info=info_message, status=status, target_platform_id=platform.id) diff --git a/uncoder-core/app/routers/translate.py b/uncoder-core/app/routers/translate.py new file mode 100644 index 00000000..009cab03 --- /dev/null +++ b/uncoder-core/app/routers/translate.py @@ -0,0 +1,92 @@ +from fastapi import APIRouter, Body + +from app.models.translation import InfoMessage, OneTranslationData, Platform, TranslatorPlatforms +from app.translator.core.context_vars import return_only_first_query_ctx_var +from app.translator.cti_translator import CTITranslator +from app.translator.translator import Translator + +st_router = APIRouter() + +translator = Translator() + + +@st_router.post("/translate", tags=["translator"], description="Generate target translation") +@st_router.post("/translate/", include_in_schema=False) +def translate_one( + source_platform_id: str = Body(..., embed=True), + target_platform_id: str = Body(..., embed=True), + text: str = Body(..., embed=True), + return_only_first_query: bool = False, +) -> OneTranslationData: + return_only_first_query_ctx_var.set(return_only_first_query) + status, data = translator.translate_one(text=text, source=source_platform_id, target=target_platform_id) + if status: + return OneTranslationData(status=status, translation=data, target_platform_id=target_platform_id) + + info_message = InfoMessage(message=data, severity="error") + return OneTranslationData(info=info_message, status=status, target_platform_id=target_platform_id) + + +@st_router.post("/translate/all", tags=["translator"], description="Generate all translations") +@st_router.post("/translate/all/", include_in_schema=False) +def translate_all( + source_platform_id: str = Body(..., embed=True), + text: str = Body(..., embed=True), + return_only_first_query: bool = False, +) -> list[OneTranslationData]: + return_only_first_query_ctx_var.set(return_only_first_query) + result = translator.translate_all(text=text, source=source_platform_id) + translations = [] + for platform_result in result: + if platform_result.get("status"): + translations.append( + OneTranslationData( + status=platform_result.get("status", True), + translation=platform_result.get("result"), + target_platform_id=platform_result.get("platform_id"), + ) + ) + else: + translations.append( + OneTranslationData( + status=platform_result.get("status", False), + info=InfoMessage(message=platform_result.get("result"), severity="error"), + target_platform_id=platform_result.get("platform_id"), + ) + ) + return translations + + +@st_router.get("/platforms", tags=["translator"], description="Get translator platforms") +@st_router.get("/platforms/", include_in_schema=False) +def get_translator_platforms() -> TranslatorPlatforms: + renders, parsers = translator.get_all_platforms() + return TranslatorPlatforms(renders=renders, parsers=parsers) + + +@st_router.get("/all_platforms", description="Get Sigma, RootA and iocs platforms") +@st_router.get("/all_platforms/", include_in_schema=False) +def get_all_platforms() -> list: + translator_renders, translator_parsers = translator.get_all_platforms() + return [ + Platform( + id="roota", + name="RootA", + code="roota", + group_name="RootA", + group_id="roota", + renders=translator_renders, + parsers=translator_parsers, + ), + Platform( + id="sigma", + name="Sigma", + code="sigma", + group_name="Sigma", + group_id="sigma", + renders=[render for render in translator_renders if render.code != "sigma"], + ), + Platform( + id="ioc", name="IOCs", code="ioc", group_name="IOCs", group_id="ioc", renders=CTITranslator().get_renders() + ), + ] diff --git a/siem-converter/app/converter/core/exceptions/__init__.py b/uncoder-core/app/translator/__init__.py similarity index 100% rename from siem-converter/app/converter/core/exceptions/__init__.py rename to uncoder-core/app/translator/__init__.py diff --git a/uncoder-core/app/translator/const.py b/uncoder-core/app/translator/const.py new file mode 100644 index 00000000..767fe882 --- /dev/null +++ b/uncoder-core/app/translator/const.py @@ -0,0 +1,12 @@ +from os.path import abspath, dirname +from typing import Union + +from app.translator.core.str_value_manager import StrValue + +APP_PATH = dirname(abspath(__file__)) + +CTI_MIN_LIMIT_QUERY = 10000 + +CTI_IOCS_PER_QUERY_LIMIT = 25 + +DEFAULT_VALUE_TYPE = Union[bool, int, str, StrValue, list[Union[int, str, StrValue]]] diff --git a/siem-converter/app/converter/core/mixins/__init__.py b/uncoder-core/app/translator/core/__init__.py similarity index 100% rename from siem-converter/app/converter/core/mixins/__init__.py rename to uncoder-core/app/translator/core/__init__.py diff --git a/uncoder-core/app/translator/core/const.py b/uncoder-core/app/translator/core/const.py new file mode 100644 index 00000000..c8fd16ce --- /dev/null +++ b/uncoder-core/app/translator/core/const.py @@ -0,0 +1,10 @@ +from typing import Union + +from app.translator.core.models.query_tokens.field import Alias, Field +from app.translator.core.models.query_tokens.field_field import FieldField +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.function_value import FunctionValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.keyword import Keyword + +QUERY_TOKEN_TYPE = Union[FieldField, FieldValue, FunctionValue, Keyword, Identifier, Field, Alias] diff --git a/uncoder-core/app/translator/core/context_vars.py b/uncoder-core/app/translator/core/context_vars.py new file mode 100644 index 00000000..8ff6ccfb --- /dev/null +++ b/uncoder-core/app/translator/core/context_vars.py @@ -0,0 +1,10 @@ +from contextvars import ContextVar +from typing import Optional + +return_only_first_query_ctx_var: ContextVar[bool] = ContextVar("return_only_first_query_ctx_var", default=False) +"""Set to True to return only first query if rendered multiple options""" + +wrap_query_with_meta_info_ctx_var: ContextVar[bool] = ContextVar("wrap_query_with_meta_info_ctx_var", default=True) +"""Set to False not to wrap query with meta info commentary""" + +preset_log_source_str_ctx_var: ContextVar[Optional[str]] = ContextVar("preset_log_source_str_ctx_var", default=None) diff --git a/siem-converter/app/converter/core/models/__init__.py b/uncoder-core/app/translator/core/custom_types/__init__.py similarity index 100% rename from siem-converter/app/converter/core/models/__init__.py rename to uncoder-core/app/translator/core/custom_types/__init__.py diff --git a/uncoder-core/app/translator/core/custom_types/functions.py b/uncoder-core/app/translator/core/custom_types/functions.py new file mode 100644 index 00000000..8cecc010 --- /dev/null +++ b/uncoder-core/app/translator/core/custom_types/functions.py @@ -0,0 +1,41 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class FunctionType(CustomEnum): + avg = "avg" + count = "count" + distinct_count = "distinct_count" + max = "max" + min = "min" + sum = "sum" + + values = "values" + + earliest = "earliest" + latest = "latest" + + divide = "divide" + multiply = "multiply" + + lower = "lower" + split = "split" + upper = "upper" + + array_length = "array_length" + extract_time = "extract_time" + ipv4_is_in_range = "ipv4_is_in_range" + + bin = "bin" + eval = "eval" + fields = "fields" + iploc = "iploc" + join = "join" + rename = "rename" + search = "search" + sort_limit = "sort_limit" + stats = "stats" + table = "table" + timeframe = "timeframe" + union = "union" + + aggregation_data_function = "aggregation_data_function" diff --git a/uncoder-core/app/translator/core/custom_types/meta_info.py b/uncoder-core/app/translator/core/custom_types/meta_info.py new file mode 100644 index 00000000..f90dfed4 --- /dev/null +++ b/uncoder-core/app/translator/core/custom_types/meta_info.py @@ -0,0 +1,9 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class SeverityType(CustomEnum): + critical = "critical" + high = "high" + medium = "medium" + low = "low" + informational = "informational" diff --git a/uncoder-core/app/translator/core/custom_types/predefined_fields.py b/uncoder-core/app/translator/core/custom_types/predefined_fields.py new file mode 100644 index 00000000..077850af --- /dev/null +++ b/uncoder-core/app/translator/core/custom_types/predefined_fields.py @@ -0,0 +1,16 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class IPLocationType(CustomEnum): + asn = "ip_loc_asn" + asn_org = "ip_loc_asn_org" + city = "ip_loc_city" + continent = "ip_loc_continent" + country = "ip_loc_country" + lat_lon = "ip_loc_lat_lon" + region = "ip_loc_region" + timezone = "ip_loc_timezone" + + +class TimeType(CustomEnum): + timestamp = "timestamp" diff --git a/uncoder-core/app/translator/core/custom_types/time.py b/uncoder-core/app/translator/core/custom_types/time.py new file mode 100644 index 00000000..4cdc71fe --- /dev/null +++ b/uncoder-core/app/translator/core/custom_types/time.py @@ -0,0 +1,23 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class TimeFrameType(CustomEnum): + years = "years" + months = "months" + days = "days" + hours = "hours" + minutes = "minutes" + + +class TimePartType(CustomEnum): + day = "day" + day_of_week = "day_of_week" + day_of_year = "day_of_year" + hour = "hour" + microsecond = "microsecond" + millisecond = "millisecond" + minute = "minute" + month = "month" + quarter = "quarter" + second = "second" + year = "year" diff --git a/uncoder-core/app/translator/core/custom_types/tokens.py b/uncoder-core/app/translator/core/custom_types/tokens.py new file mode 100644 index 00000000..663e5d33 --- /dev/null +++ b/uncoder-core/app/translator/core/custom_types/tokens.py @@ -0,0 +1,44 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class LogicalOperatorType(CustomEnum): + AND = "and" + OR = "or" + NOT = "not" + + +class OperatorType(CustomEnum): + LT = "<" + LTE = "<=" + GT = ">" + GTE = ">=" + EQ = "=" + NOT_EQ = "!=" + CONTAINS = "contains" + NOT_CONTAINS = "not contains" + STARTSWITH = "startswith" + NOT_STARTSWITH = "not startswith" + ENDSWITH = "endswith" + NOT_ENDSWITH = "not endswith" + REGEX = "re" + NOT_REGEX = "not re" + KEYWORD = "keyword" + IS_NONE = "is none" + IS_NOT_NONE = "is not none" + + +class GroupType(CustomEnum): + L_PAREN = "(" + R_PAREN = ")" + + +STR_SEARCH_OPERATORS = ( + OperatorType.CONTAINS, + OperatorType.NOT_CONTAINS, + OperatorType.ENDSWITH, + OperatorType.NOT_ENDSWITH, + OperatorType.STARTSWITH, + OperatorType.NOT_STARTSWITH, + OperatorType.REGEX, + OperatorType.NOT_REGEX, +) diff --git a/uncoder-core/app/translator/core/custom_types/values.py b/uncoder-core/app/translator/core/custom_types/values.py new file mode 100644 index 00000000..440123dd --- /dev/null +++ b/uncoder-core/app/translator/core/custom_types/values.py @@ -0,0 +1,16 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class ValueType(CustomEnum): + value = "value" + number_value = "num_value" + double_quotes_value = "d_q_value" + single_quotes_value = "s_q_value" + back_quotes_value = "b_q_value" + no_quotes_value = "no_q_value" + bool_value = "bool_value" + regex_value = "re_value" + greater_than_or_equal = "gte_value" + less_than_or_equal = "lte_value" + multi_value = "multi_value" + ip = "ip" diff --git a/uncoder-core/app/translator/core/escape_manager.py b/uncoder-core/app/translator/core/escape_manager.py new file mode 100644 index 00000000..4c3861c7 --- /dev/null +++ b/uncoder-core/app/translator/core/escape_manager.py @@ -0,0 +1,25 @@ +import re +from abc import ABC +from typing import ClassVar, Union + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.models.escape_details import EscapeDetails + + +class EscapeManager(ABC): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = {} + + def escape(self, value: Union[str, int], value_type: str = ValueType.value) -> Union[str, int]: + if isinstance(value, int): + return value + if escape_details := self.escape_map.get(value_type): + for escape_detail in escape_details: + symbols_pattern = re.compile(escape_detail.pattern) + value = symbols_pattern.sub(escape_detail.escape_symbols, value) + return value + + @staticmethod + def remove_escape(value: Union[str, int]) -> Union[str, int]: + if isinstance(value, int): + return value + return value.encode().decode("unicode_escape") diff --git a/siem-converter/app/converter/core/models/functions/__init__.py b/uncoder-core/app/translator/core/exceptions/__init__.py similarity index 100% rename from siem-converter/app/converter/core/models/functions/__init__.py rename to uncoder-core/app/translator/core/exceptions/__init__.py diff --git a/uncoder-core/app/translator/core/exceptions/core.py b/uncoder-core/app/translator/core/exceptions/core.py new file mode 100644 index 00000000..425c1ff0 --- /dev/null +++ b/uncoder-core/app/translator/core/exceptions/core.py @@ -0,0 +1,98 @@ +from typing import Optional + + +class BasePlatformException(BaseException): + ... + + +class StrictPlatformException(BasePlatformException): + def __init__(self, platform_name: str, fields: list[str], mapping: Optional[str] = None): + message = ( + f"Platform {platform_name} has strict mapping. " + f"Source fields: {', '.join(fields)} have no mapping." + f" Mapping file: {mapping}." + if mapping + else "" + ) + super().__init__(message) + + +class UnsupportedMappingsException(BasePlatformException): + def __init__(self, platform_name: str, mappings: list[str]): + message = f"Platform {platform_name} does not support these mappings: {mappings}." + super().__init__(message) + + +class StrictPlatformFieldException(BasePlatformException): + def __init__(self, platform_name: str, field_name: str): + message = f"Source field `{field_name}` has no mapping for platform {platform_name}." + super().__init__(message) + + +class UnsupportedPlatform(BasePlatformException): + def __init__(self, platform: str, is_parser: bool = False): + direction = "input" if is_parser else "output" + if platform: + message = ( + f"The selected {direction} language `{platform}` is not supported. " + f"Please, select an option in the dropdown." + ) + else: + message = f"Please, select an {direction} language." + super().__init__(message) + + +class UnsupportedRootAParser(BasePlatformException): + def __init__(self, parser: str): + message = ( + f"The platform `{parser}` specified in the body is not supported. " + f"Please, use platform names listed in the readme.md" + ) + super().__init__(message) + + +class RootAParserException(BasePlatformException): + def __init__(self, platform_name: str, error: str): + message = f"The body part of the detection section has invalid {platform_name} syntax. Error: {error}" + super().__init__(message) + + +class YamlRuleValidationException(BasePlatformException): + rule = None + + def __init__(self, missing_fields: list[str]): + prepared_missing_fields = "\n".join(f"- {field}" for field in missing_fields) + message = f"Invalid {self.rule} rule structure. Some required fields are missing: {prepared_missing_fields}." + super().__init__(message) + + +class RootARuleValidationException(YamlRuleValidationException): + rule = "RootA" + + +class SigmaRuleValidationException(YamlRuleValidationException): + rule = "Sigma" + + +class InvalidRuleStructure(BasePlatformException): + rule_type = None + + def __init__(self, error: str): + message = f"Invalid {self.rule_type} structure. Issue: {error}." + super().__init__(message) + + +class InvalidYamlStructure(InvalidRuleStructure): + rule_type: str = "YAML" + + +class InvalidJSONStructure(InvalidRuleStructure): + rule_type: str = "JSON" + + +class InvalidTOMLStructure(InvalidRuleStructure): + rule_type: str = "TOML" + + +class InvalidXMLStructure(InvalidRuleStructure): + rule_type: str = "XML" diff --git a/uncoder-core/app/translator/core/exceptions/functions.py b/uncoder-core/app/translator/core/exceptions/functions.py new file mode 100644 index 00000000..17a956d3 --- /dev/null +++ b/uncoder-core/app/translator/core/exceptions/functions.py @@ -0,0 +1,14 @@ +class BaseFunctionException(Exception): + ... + + +class InternalFunctionException(Exception): + pass + + +class NotSupportedFunctionException(InternalFunctionException): + pass + + +class InvalidFunctionSignature(InternalFunctionException): + pass diff --git a/siem-converter/app/converter/core/exceptions/iocs.py b/uncoder-core/app/translator/core/exceptions/iocs.py similarity index 99% rename from siem-converter/app/converter/core/exceptions/iocs.py rename to uncoder-core/app/translator/core/exceptions/iocs.py index 161551fa..6ed9b988 100644 --- a/siem-converter/app/converter/core/exceptions/iocs.py +++ b/uncoder-core/app/translator/core/exceptions/iocs.py @@ -1,4 +1,3 @@ - class BaseIOCsException(BaseException): ... @@ -8,7 +7,6 @@ class IocsLimitExceededException(BaseIOCsException): class EmptyIOCSException(BaseIOCsException): - def __init__(self): message = "No Indicators of Compromise recognized in the input." super().__init__(message) diff --git a/uncoder-core/app/translator/core/exceptions/parser.py b/uncoder-core/app/translator/core/exceptions/parser.py new file mode 100644 index 00000000..6c9a8e08 --- /dev/null +++ b/uncoder-core/app/translator/core/exceptions/parser.py @@ -0,0 +1,38 @@ +class BaseParserException(BaseException): + ... + + +class TokenizerGeneralException(BaseParserException): + def __init__(self, error: str): + message = ( + f"It looks like there's been an issue during translation. {error}. " + f"Resolve the issue and try again. If you think there's no issue, please contact us. " + f"We are improving the translation capabilities for this language, " + f"and some operators may be not supported yet." + ) + super().__init__(message) + + +class UnsupportedOperatorException(BaseParserException): + def __init__(self, operator: str): + message = f"Cannot translate. Operator {operator} is not supported." + super().__init__(message) + + +class QueryParenthesesException(BaseParserException): + def __init__(self): + message = ( + "The query logic is broken. In the input, the numbers of opening and closing parentheses " + "do not match. If you think there's no error, please contact us." + ) + super().__init__(message) + + +class FieldMappingException(BaseParserException): + def __init__(self, field_name: str): + message = ( + f"Cannot translate. The field {field_name} does not exist in the output language or " + f"its translation is not supported yet. If you want to contribute to the open-source translation " + f"capabilities into this language, do it via our UncoderIO GitHub project." + ) + super().__init__(message) diff --git a/uncoder-core/app/translator/core/exceptions/render.py b/uncoder-core/app/translator/core/exceptions/render.py new file mode 100644 index 00000000..65117d59 --- /dev/null +++ b/uncoder-core/app/translator/core/exceptions/render.py @@ -0,0 +1,18 @@ +class BaseRenderException(BaseException): + ... + + +class UnexpectedLogsourceException(BaseRenderException): + def __init__(self, platform_name: str, log_source: str): + message = f"{platform_name} can't convert query with logsources: {log_source}" + super().__init__(message) + + +class FunctionRenderException(BaseRenderException): + ... + + +class UnsupportedRenderMethod(BaseRenderException): + def __init__(self, platform_name: str, method: str): + message = f'Cannot translate. {platform_name} backend does not support "{method}".' + super().__init__(message) diff --git a/uncoder-core/app/translator/core/functions.py b/uncoder-core/app/translator/core/functions.py new file mode 100644 index 00000000..1ac217bb --- /dev/null +++ b/uncoder-core/app/translator/core/functions.py @@ -0,0 +1,211 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from __future__ import annotations + +import re +from abc import ABC, abstractmethod +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, ClassVar, Optional, Union + +from app.translator.core.exceptions.functions import NotSupportedFunctionException +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.functions.base import Function, ParsedFunctions, RenderedFunctions +from app.translator.core.models.query_tokens.field import Alias, Field, PredefinedField +from app.translator.tools.utils import execute_module +from settings import INIT_FUNCTIONS + +if TYPE_CHECKING: + from app.translator.core.render import PlatformQueryRender + + +@dataclass +class FunctionMatchContainer: + name: str + match: re.Match + + +class BaseFunctionParser(ABC): + function_names_map: ClassVar[dict[str, str]] = {} + functions_group_name: str = None + manager: PlatformFunctionsManager = None + + def set_functions_manager(self, manager: PlatformFunctionsManager) -> BaseFunctionParser: + self.manager = manager + return self + + @abstractmethod + def parse(self, *args, **kwargs) -> Function: + raise NotImplementedError + + def tokenize_body(self, func_body: str) -> list[Any]: + tokenized = [] + while func_body: + identifier, func_body = self._get_next_token(func_body=func_body) + tokenized.append(identifier) + return tokenized + + def _get_next_token(self, func_body: str) -> tuple[Any, str]: + raise NotImplementedError + + +class FunctionParser(BaseFunctionParser): + @abstractmethod + def parse(self, func_name: str, match: re.Match) -> Function: + raise NotImplementedError + + @abstractmethod + def get_func_match(self, higher_order_func_body: str) -> Optional[FunctionMatchContainer]: + raise NotImplementedError + + +class HigherOrderFunctionParser(BaseFunctionParser): # for highest level functions e.g. | stats, | search, | sort, etc. + @abstractmethod + def parse(self, func_body: str, raw: str) -> Function: + raise NotImplementedError + + +class FunctionRender(ABC): + function_names_map: ClassVar[dict[str, str]] = {} + order_to_render: int = 0 + render_to_prefix: bool = False + manager: PlatformFunctionsManager = None + + def set_functions_manager(self, manager: PlatformFunctionsManager) -> FunctionRender: + self.manager = manager + return self + + @abstractmethod + def render(self, function: Function, source_mapping: SourceMapping) -> str: + raise NotImplementedError + + def map_field(self, field: Union[Alias, Field], source_mapping: SourceMapping) -> str: + if isinstance(field, Alias): + return field.name + + if isinstance(field, Field): + mappings = self.manager.platform_functions.platform_query_render.mappings + mapped_fields = mappings.map_field(field, source_mapping) + return mapped_fields[0] + + if isinstance(field, PredefinedField): + return self.manager.platform_functions.platform_query_render.map_predefined_field(field) + + raise NotSupportedFunctionException + + +class PlatformFunctionsManager: + platform_functions: PlatformFunctions = None + + def __init__(self): + # {platform_func_name: HigherOrderFunctionParser} + self._hof_parsers_map: dict[str, HigherOrderFunctionParser] = {} + self._parsers_map: dict[str, FunctionParser] = {} # {platform_func_name: FunctionParser} + + self._renders_map: dict[str, FunctionRender] = {} # {generic_func_name: FunctionRender} + self._order_to_render: dict[str, int] = {} # {generic_func_name: int} + + def register_render(self, render_class: type[FunctionRender]) -> type[FunctionRender]: + render = render_class() + render.manager = self + for generic_function_name in render.function_names_map: + self._renders_map[generic_function_name] = render + self._order_to_render[generic_function_name] = render.order_to_render + + return render_class + + def register_parser(self, parser_class: type[BaseFunctionParser]) -> type[BaseFunctionParser]: + parser = parser_class() + parser.manager = self + parsers_map = self._hof_parsers_map if isinstance(parser, HigherOrderFunctionParser) else self._parsers_map + for platform_function_name in parser.function_names_map: + parsers_map[platform_function_name] = parser + + if parser.functions_group_name: + parsers_map[parser.functions_group_name] = parser + + return parser_class + + def get_hof_parser(self, platform_func_name: str) -> HigherOrderFunctionParser: + if INIT_FUNCTIONS and (parser := self._hof_parsers_map.get(platform_func_name)): + return parser + + raise NotSupportedFunctionException + + def get_parser(self, platform_func_name: str) -> Optional[FunctionParser]: + if INIT_FUNCTIONS and (parser := self._parsers_map.get(platform_func_name)): + return parser + + def get_render(self, generic_func_name: str) -> FunctionRender: + if INIT_FUNCTIONS and (render := self._renders_map.get(generic_func_name)): + return render + + raise NotSupportedFunctionException + + @property + def order_to_render(self) -> dict[str, int]: + if INIT_FUNCTIONS: + return self._order_to_render + + return {} + + @property + def supported_render_names(self) -> set[str]: + return set(self._renders_map) + + +class PlatformFunctions: + dir_path: str = None + platform_query_render: PlatformQueryRender = None + manager: PlatformFunctionsManager = PlatformFunctionsManager() + + function_delimiter = "|" + + def __init__(self): + self.manager.platform_functions = self + if self.dir_path: + execute_module(f"{self.dir_path}/parsers/__init__.py") + execute_module(f"{self.dir_path}/renders/__init__.py") + + def parse(self, query: str) -> ParsedFunctions: # noqa: ARG002 + return ParsedFunctions() + + def _sort_functions_to_render(self, functions: list[Function]) -> list[Function]: + return sorted(functions, key=lambda func: self.manager.order_to_render.get(func.name, 0)) + + def render(self, functions: list[Function], source_mapping: SourceMapping) -> RenderedFunctions: + rendered = "" + rendered_prefix = "" + not_supported = [] + functions = self._sort_functions_to_render(functions) + for func in functions: + try: + func_render = self.manager.get_render(func.name) + _rendered = func_render.render(func, source_mapping) + if func_render.render_to_prefix: + rendered_prefix += _rendered + else: + rendered += self.wrap_function_with_delimiter(_rendered) + except NotSupportedFunctionException: + not_supported.append(func.raw) + + not_supported = [self.wrap_function_with_delimiter(func.strip()) for func in not_supported] + return RenderedFunctions(rendered_prefix=rendered_prefix, rendered=rendered, not_supported=not_supported) + + def wrap_function_with_delimiter(self, func: str) -> str: + return f" {self.function_delimiter} {func}" diff --git a/uncoder-core/app/translator/core/mapping.py b/uncoder-core/app/translator/core/mapping.py new file mode 100644 index 00000000..1c4d2070 --- /dev/null +++ b/uncoder-core/app/translator/core/mapping.py @@ -0,0 +1,279 @@ +from __future__ import annotations + +from abc import ABC, abstractmethod +from typing import TYPE_CHECKING, Optional, TypeVar, Union + +from app.translator.core.exceptions.core import StrictPlatformException, UnsupportedMappingsException +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.mappings.utils.load_from_files import LoaderFileMappings + +if TYPE_CHECKING: + from app.translator.core.models.query_tokens.field import Field + + +DEFAULT_MAPPING_NAME = "default" + + +class LogSourceSignature(ABC): + _default_source: dict + wildcard_symbol = "*" + + @abstractmethod + def is_suitable(self, **kwargs) -> bool: + raise NotImplementedError("Abstract method") + + def is_probably_suitable(self, **kwargs) -> bool: + """ + Performs check with more options, but the result is less accurate than the "is_suitable" method + """ + raise NotImplementedError("Abstract method") + + @staticmethod + def _check_conditions(conditions: list[Union[bool, None]]) -> bool: + conditions = [condition for condition in conditions if condition is not None] + return bool(conditions) and all(conditions) + + @abstractmethod + def __str__(self) -> str: + raise NotImplementedError("Abstract method") + + @property + def default_source(self) -> dict: + return self._default_source + + +class FieldMapping: + def __init__(self, generic_field_name: str, platform_field_name: str): + self.generic_field_name = generic_field_name + self.platform_field_name = platform_field_name + + +class FieldsMapping: + def __init__(self, fields_mapping: list): + self.__parser_mapping = self.__build_parser_mapping(fields_mapping) + self.__render_mapping = self.__build_render_mapping(fields_mapping) + + @staticmethod + def __build_parser_mapping(fields: list[FieldMapping]) -> dict[str, FieldMapping]: + result = {} + for field in fields: + if isinstance(field.platform_field_name, list): + for f in field.platform_field_name: + result.update({f: field}) + else: + result.update({field.platform_field_name: field}) + return result + + @staticmethod + def __build_render_mapping(fields: list[FieldMapping]) -> dict[str, FieldMapping]: + return {field.generic_field_name: field for field in fields} + + def get_generic_field_name(self, platform_field_name: str) -> Optional[str]: + field = self.__parser_mapping.get(platform_field_name) + return field and field.generic_field_name + + def get_platform_field_name(self, generic_field_name: str) -> Optional[str]: + field = self.__render_mapping.get(generic_field_name) + return field and field.platform_field_name + + def update(self, fields_mapping: FieldsMapping) -> None: + self.__parser_mapping.update(fields_mapping.__parser_mapping) + self.__render_mapping.update(fields_mapping.__render_mapping) + + def is_suitable(self, field_names: list[str]) -> bool: + return bool(field_names) and set(field_names).issubset(set(self.__parser_mapping.keys())) + + +_LogSourceSignatureType = TypeVar("_LogSourceSignatureType", bound=LogSourceSignature) + + +class SourceMapping: + def __init__( + self, + source_id: str, + log_source_signature: _LogSourceSignatureType = None, + fields_mapping: Optional[FieldsMapping] = None, + raw_log_fields: Optional[dict] = None, + conditions: Optional[dict] = None, + ): + self.source_id = source_id + self.log_source_signature = log_source_signature + self.fields_mapping = fields_mapping or FieldsMapping([]) + self.raw_log_fields = raw_log_fields + self.conditions = conditions + + +class BasePlatformMappings: + details: PlatformDetails = None + + is_strict_mapping: bool = False + skip_load_default_mappings: bool = True + extend_default_mapping_with_all_fields: bool = False + + def __init__(self, platform_dir: str, platform_details: PlatformDetails): + self._loader = LoaderFileMappings() + self._platform_dir = platform_dir + self.details = platform_details + self._source_mappings = self.prepare_mapping() + + def update_default_source_mapping(self, default_mapping: SourceMapping, fields_mapping: FieldsMapping) -> None: + default_mapping.fields_mapping.update(fields_mapping) + + def prepare_mapping(self) -> dict[str, SourceMapping]: + source_mappings = {} + default_mapping = SourceMapping(source_id=DEFAULT_MAPPING_NAME) + for mapping_dict in self._loader.load_platform_mappings(self._platform_dir): + log_source_signature = self.prepare_log_source_signature(mapping=mapping_dict) + if (source_id := mapping_dict["source"]) == DEFAULT_MAPPING_NAME: + default_mapping.log_source_signature = log_source_signature + if self.skip_load_default_mappings: + continue + + field_mappings_dict = mapping_dict.get("field_mapping", {}) + raw_log_fields = mapping_dict.get("raw_log_fields", {}) + conditions = mapping_dict.get("conditions", {}) + field_mappings_dict.update({field: field for field in raw_log_fields}) + fields_mapping = self.prepare_fields_mapping(field_mapping=field_mappings_dict) + self.update_default_source_mapping(default_mapping=default_mapping, fields_mapping=fields_mapping) + source_mappings[source_id] = SourceMapping( + source_id=source_id, + log_source_signature=log_source_signature, + fields_mapping=fields_mapping, + raw_log_fields=raw_log_fields, + conditions=conditions, + ) + + if self.skip_load_default_mappings: + source_mappings[DEFAULT_MAPPING_NAME] = default_mapping + + if self.extend_default_mapping_with_all_fields: + source_mappings[DEFAULT_MAPPING_NAME].fields_mapping.update(default_mapping.fields_mapping) + + return source_mappings + + @staticmethod + def prepare_fields_mapping(field_mapping: dict) -> FieldsMapping: + fields = [] + for generic_field_name, platform_field_name in field_mapping.items(): + fields.append(FieldMapping(generic_field_name=generic_field_name, platform_field_name=platform_field_name)) + return FieldsMapping(fields_mapping=fields) + + @abstractmethod + def prepare_log_source_signature(self, mapping: dict) -> LogSourceSignature: + raise NotImplementedError("Abstract method") + + def get_source_mappings_by_fields_and_log_sources( + self, field_names: list[str], log_sources: dict[str, list[Union[int, str]]] + ) -> list[SourceMapping]: + by_log_sources_and_fields = [] + by_fields = [] + for source_mapping in self._source_mappings.values(): + if source_mapping.source_id == DEFAULT_MAPPING_NAME: + continue + + if source_mapping.fields_mapping.is_suitable(field_names): + by_fields.append(source_mapping) + + log_source_signature: LogSourceSignature = source_mapping.log_source_signature + if log_source_signature and log_source_signature.is_suitable(**log_sources): + by_log_sources_and_fields.append(source_mapping) + + return by_log_sources_and_fields or by_fields or [self._source_mappings[DEFAULT_MAPPING_NAME]] + + def get_source_mapping(self, source_id: str) -> Optional[SourceMapping]: + return self._source_mappings.get(source_id) + + def get_source_mappings_by_ids( + self, source_mapping_ids: list[str], return_default: bool = True + ) -> list[SourceMapping]: + source_mappings = [] + for source_mapping_id in source_mapping_ids: + if source_mapping_id == DEFAULT_MAPPING_NAME: + continue + if source_mapping := self.get_source_mapping(source_mapping_id): + source_mappings.append(source_mapping) + + if not source_mappings and return_default: + source_mappings = [self.get_source_mapping(DEFAULT_MAPPING_NAME)] + + return source_mappings + + def get_source_mappings_by_log_sources(self, log_sources: dict) -> Optional[list[str]]: + raise NotImplementedError("Abstract method") + + @property + def default_mapping(self) -> SourceMapping: + return self._source_mappings[DEFAULT_MAPPING_NAME] + + def check_fields_mapping_existence( + self, + query_field_tokens: list[Field], + function_field_tokens_map: dict[str, list[Field]], + supported_func_render_names: set[str], + source_mapping: SourceMapping, + ) -> list[str]: + unmapped = [] + + for field in query_field_tokens: + self._check_field_mapping_existence(field, source_mapping, unmapped) + + for func_name, function_field_tokens in function_field_tokens_map.items(): + if func_name in supported_func_render_names: + for field in function_field_tokens: + self._check_field_mapping_existence(field, source_mapping, unmapped) + + if self.is_strict_mapping and unmapped: + raise StrictPlatformException( + platform_name=self.details.name, fields=unmapped, mapping=source_mapping.source_id + ) + + return unmapped + + @staticmethod + def _check_field_mapping_existence(field: Field, source_mapping: SourceMapping, unmapped: list[str]) -> None: + generic_field_name = field.get_generic_field_name(source_mapping.source_id) + mapped_field = source_mapping.fields_mapping.get_platform_field_name(generic_field_name=generic_field_name) + if not mapped_field and field.source_name not in unmapped: + unmapped.append(field.source_name) + + @staticmethod + def map_field(field: Field, source_mapping: SourceMapping) -> list[str]: + generic_field_name = field.get_generic_field_name(source_mapping.source_id) + # field can be mapped to corresponding platform field name or list of platform field names + mapped_field = source_mapping.fields_mapping.get_platform_field_name(generic_field_name=generic_field_name) + + if isinstance(mapped_field, str): + mapped_field = [mapped_field] + + return mapped_field if mapped_field else [generic_field_name] if generic_field_name else [field.source_name] + + +class BaseCommonPlatformMappings(ABC, BasePlatformMappings): + def prepare_mapping(self) -> dict[str, SourceMapping]: + source_mappings = {} + common_field_mapping = self._loader.load_common_mapping(self._platform_dir).get("field_mapping", {}) + + for mapping_dict in self._loader.load_platform_mappings(self._platform_dir): + source_id = mapping_dict["source"] + log_source_signature = self.prepare_log_source_signature(mapping=mapping_dict) + fields_mapping = self.prepare_fields_mapping(field_mapping=common_field_mapping) + source_mappings[source_id] = SourceMapping( + source_id=source_id, log_source_signature=log_source_signature, fields_mapping=fields_mapping + ) + + return source_mappings + + +class BaseStrictLogSourcesPlatformMappings(ABC, BasePlatformMappings): + def get_source_mappings_by_ids(self, source_mapping_ids: list[str]) -> list[SourceMapping]: + source_mappings = [] + for source_mapping_id in source_mapping_ids: + if source_mapping_id == DEFAULT_MAPPING_NAME: + continue + if source_mapping := self.get_source_mapping(source_mapping_id): + source_mappings.append(source_mapping) + + if not source_mappings: + raise UnsupportedMappingsException(platform_name=self.details.name, mappings=source_mapping_ids) + + return source_mappings diff --git a/uncoder-core/app/translator/core/mitre.py b/uncoder-core/app/translator/core/mitre.py new file mode 100644 index 00000000..2e86a3be --- /dev/null +++ b/uncoder-core/app/translator/core/mitre.py @@ -0,0 +1,218 @@ +import json +import os +import ssl +import urllib.request +from json import JSONDecodeError +from typing import Optional, Union +from urllib.error import HTTPError + +from app.translator.core.models.query_container import MitreInfoContainer, MitreTacticContainer, MitreTechniqueContainer +from app.translator.tools.singleton_meta import SingletonMeta +from const import ROOT_PROJECT_PATH + + +class TrieNode: + def __init__(self): + self.children = {} + self.is_end_of_word = False + self.result = None + + +class Trie: + """ + Trie (prefix tree) data structure for storing and searching Mitre ATT&CK Techniques and Tactics strings. + + This class handles the insertion and searching of strings related to Mitre ATT&CK Techniques and Tactics, even when + the strings have variations in spacing, case, or underscores. By normalizing the text—converting it to lowercase and + removing spaces and underscores—different variations of the same logical string are treated as equivalent. + + It means strings 'CredentialAccess', 'credential Access', and 'credential_access' will be processed identically, + leading to the same result. + """ + + def __init__(self): + self.root = TrieNode() + + def normalize_text(self, text: str) -> str: + return text.replace(" ", "").lower().replace("_", "").lower() + + def insert(self, text: str, result: Union[MitreTacticContainer, MitreTechniqueContainer]) -> None: + node = self.root + normalized_text = self.normalize_text(text) + + for char in normalized_text: + if char not in node.children: + node.children[char] = TrieNode() + node = node.children[char] + + node.is_end_of_word = True + node.result = result + + +class TacticsTrie(Trie): + def __init__(self): + self.root = TrieNode() + + def search(self, text: str) -> Optional[MitreTacticContainer]: + node: TrieNode = self.root + normalized_text = self.normalize_text(text) + + for char in normalized_text: + if char not in node.children: + return + node = node.children[char] + + if node.is_end_of_word: + return node.result + + +class TechniquesTrie(Trie): + def search(self, text: str) -> Optional[MitreTechniqueContainer]: + node: TrieNode = self.root + normalized_text = self.normalize_text(text) + + for char in normalized_text: + if char not in node.children: + return + node = node.children[char] + + if node.is_end_of_word: + return node.result + + +class MitreConfig(metaclass=SingletonMeta): + config_url: str = "https://raw.githubusercontent.com/mitre/cti/master/enterprise-attack/enterprise-attack.json" + mitre_source_types: tuple = ("mitre-attack",) + + def __init__(self, server: bool = False): + self.tactics: TacticsTrie = TacticsTrie() + self.techniques: TechniquesTrie = TechniquesTrie() + if not server: + self.__load_mitre_configs_from_files() + + @staticmethod + def __revoked_or_deprecated(entry: dict) -> bool: + if entry.get("revoked") or entry.get("x_mitre_deprecated"): + return True + return False + + def __get_mitre_json(self) -> dict: + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + + try: + with urllib.request.urlopen(self.config_url, context=ctx) as cti_json: + return json.loads(cti_json.read().decode()) + except HTTPError: + return {} + + def update_mitre_config(self) -> None: # noqa: PLR0912 + if not (mitre_json := self.__get_mitre_json()): + self.__load_mitre_configs_from_files() + return + + tactic_map = {} + + # Map the tactics + for entry in mitre_json["objects"]: + if entry["type"] != "x-mitre-tactic" or self.__revoked_or_deprecated(entry): + continue + for ref in entry["external_references"]: + if ref["source_name"] == "mitre-attack": + tactic_map[entry["x_mitre_shortname"]] = entry["name"] + + tactic_data = MitreTacticContainer( + external_id=ref["external_id"], url=ref["url"], name=entry["name"] + ) + self.tactics.insert(entry["name"], tactic_data) + + break + + # Map the techniques + for entry in mitre_json["objects"]: + if entry["type"] != "attack-pattern" or self.__revoked_or_deprecated(entry): + continue + if entry.get("x_mitre_is_subtechnique"): + continue + for ref in entry["external_references"]: + if ref["source_name"] in self.mitre_source_types: + sub_tactics = [] + for tactic in entry["kill_chain_phases"]: + if tactic["kill_chain_name"] in self.mitre_source_types: + sub_tactics.append(tactic_map[tactic["phase_name"]]) + + technique_data = MitreTechniqueContainer( + technique_id=ref["external_id"], name=entry["name"], url=ref["url"], tactic=sub_tactics + ) + self.techniques.insert(ref["external_id"], technique_data) + break + + # Map the sub-techniques + for entry in mitre_json["objects"]: + if entry["type"] != "attack-pattern" or self.__revoked_or_deprecated(entry): + continue + if entry.get("x_mitre_is_subtechnique"): + for ref in entry["external_references"]: + if ref["source_name"] in self.mitre_source_types: + sub_technique_id = ref["external_id"] + sub_technique_name = entry["name"] + if parent_technique := self.techniques.search(sub_technique_id.split(".")[0]): + sub_technique_name = f"{parent_technique.name} : {sub_technique_name}" + sub_technique_data = MitreTechniqueContainer( + technique_id=ref["external_id"], + name=sub_technique_name, + url=ref["url"], + tactic=parent_technique.tactic, + ) + self.techniques.insert(sub_technique_id, sub_technique_data) + break + + def __load_mitre_configs_from_files(self) -> None: + try: + with open(os.path.join(ROOT_PROJECT_PATH, "app/dictionaries/tactics.json")) as file: + loaded = json.load(file) + + for tactic_name, tactic_data in loaded.items(): + tactic = MitreTacticContainer( + external_id=tactic_data["external_id"], url=tactic_data["url"], name=tactic_data["tactic"] + ) + self.tactics.insert(tactic_name, tactic) + except JSONDecodeError: + print("Unable to load MITRE Tactics") + + try: + with open(os.path.join(ROOT_PROJECT_PATH, "app/dictionaries/techniques.json")) as file: + loaded = json.load(file) + for technique_id, technique_data in loaded.items(): + technique = MitreTechniqueContainer( + technique_id=technique_data["technique_id"], + name=technique_data["technique"], + url=technique_data["url"], + tactic=technique_data.get("tactic", []), + ) + self.techniques.insert(technique_id, technique) + except JSONDecodeError: + print("Unable to load MITRE Techniques") + + def get_tactic(self, tactic: str) -> Optional[MitreTacticContainer]: + return self.tactics.search(tactic) + + def get_technique(self, technique_id: str) -> Optional[MitreTechniqueContainer]: + return self.techniques.search(technique_id) + + def get_mitre_info( + self, tactics: Optional[list[str]] = None, techniques: Optional[list[str]] = None + ) -> MitreInfoContainer: + tactics_list = [] + techniques_list = [] + for tactic in tactics or []: + if tactic_found := self.tactics.search(tactic): + tactics_list.append(tactic_found) + for technique in techniques or []: + if technique_found := self.techniques.search(technique): + techniques_list.append(technique_found) + return MitreInfoContainer( + tactics=sorted(tactics_list, key=lambda x: x.name), + techniques=sorted(techniques_list, key=lambda x: x.technique_id), + ) diff --git a/siem-converter/app/converter/core/operator_types/__init__.py b/uncoder-core/app/translator/core/mixins/__init__.py similarity index 100% rename from siem-converter/app/converter/core/operator_types/__init__.py rename to uncoder-core/app/translator/core/mixins/__init__.py diff --git a/uncoder-core/app/translator/core/mixins/logic.py b/uncoder-core/app/translator/core/mixins/logic.py new file mode 100644 index 00000000..7002e847 --- /dev/null +++ b/uncoder-core/app/translator/core/mixins/logic.py @@ -0,0 +1,32 @@ +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.tokens import GroupType, LogicalOperatorType +from app.translator.core.models.query_tokens.field_field import FieldField +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.function_value import FunctionValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.keyword import Keyword + + +class ANDLogicOperatorMixin: + @staticmethod + def get_missed_and_token_indices(tokens: list[QUERY_TOKEN_TYPE]) -> list[int]: + missed_and_indices = [] + for index in range(len(tokens) - 1): + token = tokens[index] + next_token = tokens[index + 1] + if ( + isinstance(token, (FieldField, FieldValue, FunctionValue, Keyword)) + or isinstance(token, Identifier) + and token.token_type == GroupType.R_PAREN + ) and not ( + isinstance(next_token, Identifier) + and (next_token.token_type in (LogicalOperatorType.AND, LogicalOperatorType.OR, GroupType.R_PAREN)) + ): + missed_and_indices.append(index + 1) + return list(reversed(missed_and_indices)) + + def add_and_token_if_missed(self, tokens: list[QUERY_TOKEN_TYPE]) -> list[QUERY_TOKEN_TYPE]: + indices = self.get_missed_and_token_indices(tokens=tokens) + for index in indices: + tokens.insert(index, Identifier(token_type=LogicalOperatorType.AND)) + return tokens diff --git a/uncoder-core/app/translator/core/mixins/operator.py b/uncoder-core/app/translator/core/mixins/operator.py new file mode 100644 index 00000000..dec9e3f4 --- /dev/null +++ b/uncoder-core/app/translator/core/mixins/operator.py @@ -0,0 +1,61 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from typing import Optional, Union + +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.models.query_tokens.identifier import Identifier + + +class WildCardMixin: + @staticmethod + def _clean_value(value: str, wildcard_symbol: str) -> str: + return value.strip(wildcard_symbol) + + def get_field_value_operator( + self, value: Union[str, list[str]], operator: Optional[str] = None, wildcard_symbol: Optional[str] = None + ) -> tuple[Union[list[str], str], Identifier]: + if wildcard_symbol: + value = self._clean_value(value, wildcard_symbol) + return value, Identifier(token_type=operator) + + +class OperatorBasedMixin(WildCardMixin): + def get_field_value_operator( + self, value: Union[str, list[str]], operator: Optional[str] = None, wildcard_symbol: Optional[str] = None + ) -> tuple[Union[list[str], str], Identifier]: + value, operator = self.__get_value_operator(operator=operator, value=value) + if wildcard_symbol: + value = self._clean_value(value, wildcard_symbol) + return value, operator + + @staticmethod + def __get_value_operator(operator: str, value: Union[str, list[str]]) -> tuple[Union[list[str], str], Identifier]: + if operator == OperatorType.CONTAINS: + operator = Identifier(token_type=OperatorType.CONTAINS) + elif operator == OperatorType.ENDSWITH: + operator = Identifier(token_type=OperatorType.ENDSWITH) + elif operator == OperatorType.STARTSWITH: + operator = Identifier(token_type=OperatorType.STARTSWITH) + elif operator == OperatorType.REGEX: + operator = Identifier(token_type=OperatorType.REGEX) + elif operator == OperatorType.EQ: + operator = Identifier(token_type=OperatorType.EQ) + else: + raise Exception("Unknown operator") + return value, operator diff --git a/uncoder-core/app/translator/core/mixins/rule.py b/uncoder-core/app/translator/core/mixins/rule.py new file mode 100644 index 00000000..52e648de --- /dev/null +++ b/uncoder-core/app/translator/core/mixins/rule.py @@ -0,0 +1,69 @@ +import json +from typing import Union + +import toml +import xmltodict +import yaml + +from app.translator.core.exceptions.core import ( + InvalidJSONStructure, + InvalidTOMLStructure, + InvalidXMLStructure, + InvalidYamlStructure, +) +from app.translator.core.mitre import MitreConfig, MitreInfoContainer + + +class JsonRuleMixin: + mitre_config: MitreConfig = MitreConfig() + + @staticmethod + def load_rule(text: str) -> dict: + try: + return json.loads(text) + except json.JSONDecodeError as err: + raise InvalidJSONStructure(error=str(err)) from err + + +class YamlRuleMixin: + mitre_config: MitreConfig = MitreConfig() + + @staticmethod + def load_rule(text: str) -> dict: + try: + return yaml.safe_load(text) + except yaml.YAMLError as err: + raise InvalidYamlStructure(error=str(err)) from err + + def parse_mitre_attack(self, tags: list[str]) -> MitreInfoContainer: + parsed_techniques = [] + parsed_tactics = [] + for tag in set(tags): + tag = tag.lower() + if tag.startswith("attack."): + tag = tag[7::] + if tag[-1].isdigit(): + parsed_techniques.append(tag) + else: + parsed_tactics.append(tag) + return self.mitre_config.get_mitre_info(tactics=parsed_tactics, techniques=parsed_techniques) + + +class XMLRuleMixin: + @staticmethod + def load_rule(text: Union[str, bytes]) -> dict: + try: + return xmltodict.parse(text) + except Exception as err: + raise InvalidXMLStructure(error=str(err)) from err + + +class TOMLRuleMixin: + mitre_config: MitreConfig = MitreConfig() + + @staticmethod + def load_rule(text: str) -> dict: + try: + return toml.loads(text) + except toml.TomlDecodeError as err: + raise InvalidTOMLStructure(error=str(err)) from err diff --git a/uncoder-core/app/translator/core/mixins/tokens.py b/uncoder-core/app/translator/core/mixins/tokens.py new file mode 100644 index 00000000..09bbe266 --- /dev/null +++ b/uncoder-core/app/translator/core/mixins/tokens.py @@ -0,0 +1,20 @@ +from typing import Union + +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.tokens import LogicalOperatorType, OperatorType +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.identifier import Identifier + + +class ExtraConditionMixin: + def generate_extra_conditions(self, source_mapping: SourceMapping) -> list[QUERY_TOKEN_TYPE]: + extra_tokens = [] + for field, value in source_mapping.conditions.items(): + extra_tokens.extend( + [ + FieldValue(source_name=field, operator=Identifier(token_type=OperatorType.EQ), value=value), + Identifier(token_type=LogicalOperatorType.AND), + ] + ) + return extra_tokens diff --git a/siem-converter/app/converter/mappings/__init__.py b/uncoder-core/app/translator/core/models/__init__.py similarity index 100% rename from siem-converter/app/converter/mappings/__init__.py rename to uncoder-core/app/translator/core/models/__init__.py diff --git a/uncoder-core/app/translator/core/models/escape_details.py b/uncoder-core/app/translator/core/models/escape_details.py new file mode 100644 index 00000000..79f6d71a --- /dev/null +++ b/uncoder-core/app/translator/core/models/escape_details.py @@ -0,0 +1,7 @@ +from dataclasses import dataclass + + +@dataclass +class EscapeDetails: + pattern: str = None + escape_symbols: str = "\\\\\g<1>" diff --git a/siem-converter/app/converter/mappings/utils/__init__.py b/uncoder-core/app/translator/core/models/functions/__init__.py similarity index 100% rename from siem-converter/app/converter/mappings/utils/__init__.py rename to uncoder-core/app/translator/core/models/functions/__init__.py diff --git a/uncoder-core/app/translator/core/models/functions/base.py b/uncoder-core/app/translator/core/models/functions/base.py new file mode 100644 index 00000000..2290810e --- /dev/null +++ b/uncoder-core/app/translator/core/models/functions/base.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Optional, Union + +from app.translator.core.models.query_tokens.field import Alias, BaseFieldsGetter, Field +from app.translator.core.models.query_tokens.field_field import FieldField +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.keyword import Keyword + +if TYPE_CHECKING: + from app.translator.core.models.query_tokens.function_value import FunctionValue + + +@dataclass +class Function(BaseFieldsGetter): + name: str = None + args: list[ + Union[Alias, Field, FieldField, FieldValue, FunctionValue, Keyword, Function, Identifier, int, str, bool] + ] = field(default_factory=list) + alias: Optional[Alias] = None + raw: str = "" + + @property + def fields(self) -> list[Field]: + fields = [] + for arg in self.args: + if isinstance(arg, Field): + fields.append(arg) + elif isinstance(arg, (BaseFieldsGetter, Function)): + fields.extend(arg.fields) + + return fields + + +@dataclass +class ParsedFunctions: + functions: list[Function] = field(default_factory=list) + not_supported: list[str] = field(default_factory=list) + invalid: list[str] = field(default_factory=list) + aliases: dict[str, Function] = field(default_factory=dict) + + +@dataclass +class RenderedFunctions: + rendered_prefix: str = "" + rendered: str = "" + not_supported: list[str] = field(default_factory=list) diff --git a/uncoder-core/app/translator/core/models/functions/bin.py b/uncoder-core/app/translator/core/models/functions/bin.py new file mode 100644 index 00000000..edfdcdf1 --- /dev/null +++ b/uncoder-core/app/translator/core/models/functions/bin.py @@ -0,0 +1,25 @@ +from dataclasses import dataclass +from typing import Optional + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.core.custom_types.time import TimeFrameType +from app.translator.core.models.functions.base import Function +from app.translator.core.models.query_tokens.field import Field + + +@dataclass +class Span: + value: str = "1" + type_: str = TimeFrameType.days + + +@dataclass +class BinFunction(Function): + name: str = FunctionType.bin + span: Optional[Span] = None + field: Optional[Field] = None + bins: Optional[int] = None + + @property + def fields(self) -> list[Field]: + return [self.field] if self.field else [] diff --git a/uncoder-core/app/translator/core/models/functions/eval.py b/uncoder-core/app/translator/core/models/functions/eval.py new file mode 100644 index 00000000..146f5c96 --- /dev/null +++ b/uncoder-core/app/translator/core/models/functions/eval.py @@ -0,0 +1,32 @@ +from dataclasses import dataclass, field +from typing import Union + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.core.models.functions.base import Function +from app.translator.core.models.query_tokens.field import Alias, Field +from app.translator.core.models.query_tokens.identifier import Identifier + + +@dataclass +class EvalArg: + field_: Union[Alias, Field] = None + expression: list[Union[Field, Function, Identifier, int, float, str]] = field(default_factory=list) + + +@dataclass +class EvalFunction(Function): + name: str = FunctionType.eval + args: list[EvalArg] = None + + @property + def fields(self) -> list[Field]: + fields = [] + for arg in self.args: + if isinstance(arg.field_, Field): + fields.append(arg.field_) + for el in arg.expression: + if isinstance(el, Field): + fields.append(el) + if isinstance(el, Function): + fields.extend(el.fields) + return fields diff --git a/uncoder-core/app/translator/core/models/functions/group_by.py b/uncoder-core/app/translator/core/models/functions/group_by.py new file mode 100644 index 00000000..63def63e --- /dev/null +++ b/uncoder-core/app/translator/core/models/functions/group_by.py @@ -0,0 +1,27 @@ +from dataclasses import dataclass, field +from typing import Union + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.core.models.functions.base import Function +from app.translator.core.models.query_tokens.field import Alias, Field, PredefinedField + + +@dataclass +class GroupByFunction(Function): + name: str = FunctionType.stats + args: list[Function] = field(default_factory=list) + by_clauses: list[Union[Alias, Field, PredefinedField]] = field(default_factory=list) + filter_: Function = None + + @property + def fields(self) -> list[Field]: + fields = [] + for arg in self.args: + fields.extend(arg.fields) + for by_clause in self.by_clauses: + if isinstance(by_clause, Field): + fields.append(by_clause) + if self.filter_: + fields.extend(self.filter_.fields) + + return fields diff --git a/uncoder-core/app/translator/core/models/functions/join.py b/uncoder-core/app/translator/core/models/functions/join.py new file mode 100644 index 00000000..909f8ebd --- /dev/null +++ b/uncoder-core/app/translator/core/models/functions/join.py @@ -0,0 +1,38 @@ +from dataclasses import dataclass, field +from typing import Union + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.core.models.functions.base import Function +from app.translator.core.models.query_container import TokenizedQueryContainer +from app.translator.core.models.query_tokens.field import Alias, BaseFieldsGetter, Field +from app.translator.core.models.query_tokens.field_field import FieldField +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.function_value import FunctionValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.tools.custom_enum import CustomEnum + + +class JoinType(CustomEnum): + inner = "inner" + left = "left" + right = "right" + cross = "cross" + + +@dataclass +class JoinFunction(Function): + name: str = FunctionType.join + alias: Alias = None + type_: str = JoinType.inner + tokenized_query_container: TokenizedQueryContainer = None + condition: list[Union[FieldField, FieldValue, FunctionValue, Identifier]] = field(default_factory=list) + preset_log_source_str: str = None + + @property + def fields(self) -> list[Field]: + fields = [] + for arg in self.condition: + if isinstance(arg, BaseFieldsGetter): + fields.extend(arg.fields) + + return fields diff --git a/uncoder-core/app/translator/core/models/functions/rename.py b/uncoder-core/app/translator/core/models/functions/rename.py new file mode 100644 index 00000000..dafb90a5 --- /dev/null +++ b/uncoder-core/app/translator/core/models/functions/rename.py @@ -0,0 +1,25 @@ +from dataclasses import dataclass + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.core.models.functions.base import Function +from app.translator.core.models.query_tokens.field import Alias, Field + + +@dataclass +class RenameArg: + field_: Field = None + alias: Alias = None + + +@dataclass +class RenameFunction(Function): + name: str = FunctionType.rename + args: list[RenameArg] = None + + @property + def fields(self) -> list[Field]: + fields = [] + for arg in self.args: + fields.append(arg.field_) + + return fields diff --git a/uncoder-core/app/translator/core/models/functions/sort.py b/uncoder-core/app/translator/core/models/functions/sort.py new file mode 100644 index 00000000..0dded9f6 --- /dev/null +++ b/uncoder-core/app/translator/core/models/functions/sort.py @@ -0,0 +1,38 @@ +from dataclasses import dataclass +from typing import Union + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.core.models.functions.base import Function +from app.translator.core.models.query_tokens.field import Alias, Field +from app.translator.tools.custom_enum import CustomEnum + + +class SortOrder(CustomEnum): + asc = "asc" + desc = "desc" + + +@dataclass +class SortArg: + field: Union[Alias, Field] = None + function: Function = None + sort_order: str = SortOrder.asc + + +@dataclass +class SortLimitFunction(Function): + name: str = FunctionType.sort_limit + args: list[SortArg] = None + limit: str = None + + @property + def fields(self) -> list[Field]: + fields = [] + for arg in self.args: + if isinstance(arg.field, Field): + fields.append(arg.field) + + if arg.function: + fields.extend(arg.function.fields) + + return fields diff --git a/uncoder-core/app/translator/core/models/functions/time.py b/uncoder-core/app/translator/core/models/functions/time.py new file mode 100644 index 00000000..eb6a1229 --- /dev/null +++ b/uncoder-core/app/translator/core/models/functions/time.py @@ -0,0 +1,12 @@ +from dataclasses import dataclass + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.core.custom_types.time import TimeFrameType +from app.translator.core.models.functions.base import Function + + +@dataclass +class TimeFrameFunction(Function): + name: str = FunctionType.timeframe + timeframe_value: str = "1" + timeframe_type: str = TimeFrameType.days diff --git a/uncoder-core/app/translator/core/models/functions/union.py b/uncoder-core/app/translator/core/models/functions/union.py new file mode 100644 index 00000000..a8932f76 --- /dev/null +++ b/uncoder-core/app/translator/core/models/functions/union.py @@ -0,0 +1,17 @@ +from dataclasses import dataclass + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.core.models.functions.base import Function +from app.translator.core.models.query_container import TokenizedQueryContainer +from app.translator.core.models.query_tokens.field import Field + + +@dataclass +class UnionFunction(Function): + name: str = FunctionType.union + tokenized_query_container: TokenizedQueryContainer = None + preset_log_source_str: str = None + + @property + def fields(self) -> list[Field]: + return [] diff --git a/siem-converter/app/converter/core/models/iocs.py b/uncoder-core/app/translator/core/models/iocs.py similarity index 100% rename from siem-converter/app/converter/core/models/iocs.py rename to uncoder-core/app/translator/core/models/iocs.py diff --git a/siem-converter/app/converter/core/models/platform_details.py b/uncoder-core/app/translator/core/models/platform_details.py similarity index 93% rename from siem-converter/app/converter/core/models/platform_details.py rename to uncoder-core/app/translator/core/models/platform_details.py index 07794881..def40bae 100644 --- a/siem-converter/app/converter/core/models/platform_details.py +++ b/uncoder-core/app/translator/core/models/platform_details.py @@ -4,7 +4,7 @@ @dataclass class PlatformDetails: - siem_type: str = "" + platform_id: str = "" name: str = "" platform_name: str = "" group_id: Optional[str] = None diff --git a/uncoder-core/app/translator/core/models/query_container.py b/uncoder-core/app/translator/core/models/query_container.py new file mode 100644 index 00000000..ad866b51 --- /dev/null +++ b/uncoder-core/app/translator/core/models/query_container.py @@ -0,0 +1,142 @@ +import uuid +from dataclasses import dataclass, field +from datetime import datetime, timedelta +from typing import Optional + +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.mapping import DEFAULT_MAPPING_NAME +from app.translator.core.models.functions.base import ParsedFunctions +from app.translator.core.models.query_tokens.field import Field + + +@dataclass +class MitreTechniqueContainer: + technique_id: str + name: str + url: str + tactic: list[str] + + +@dataclass +class MitreTacticContainer: + external_id: str + url: str + name: str + + +@dataclass +class MitreInfoContainer: + tactics: list[MitreTacticContainer] = field(default_factory=list) + techniques: list[MitreTechniqueContainer] = field(default_factory=list) + + +class RawMetaInfoContainer: + def __init__( + self, + *, + trigger_operator: Optional[str] = None, + trigger_threshold: Optional[str] = None, + query_frequency: Optional[str] = None, + query_period: Optional[str] = None, + from_: Optional[str] = None, + interval: Optional[str] = None, + ) -> None: + self.trigger_operator = trigger_operator + self.trigger_threshold = trigger_threshold + self.query_frequency = query_frequency + self.query_period = query_period + self.from_ = from_ + self.interval = interval + + +class MetaInfoContainer: + def __init__( + self, + *, + id_: Optional[str] = None, + index: Optional[list[str]] = None, + language: Optional[str] = None, + risk_score: Optional[int] = None, + type_: Optional[str] = None, + title: Optional[str] = None, + description: Optional[str] = None, + author: Optional[list[str]] = None, + date: Optional[str] = None, + output_table_fields: Optional[list[Field]] = None, + query_fields: Optional[list[Field]] = None, + function_fields: Optional[list[Field]] = None, + function_fields_map: Optional[dict[str, list[Field]]] = None, + license_: Optional[str] = None, + severity: Optional[str] = None, + references: Optional[list[str]] = None, + tags: Optional[list[str]] = None, + raw_mitre_attack: Optional[list[str]] = None, + status: Optional[str] = None, + false_positives: Optional[list[str]] = None, + source_mapping_ids: Optional[list[str]] = None, + parsed_logsources: Optional[dict] = None, + timeframe: Optional[timedelta] = None, + query_period: Optional[timedelta] = None, + mitre_attack: Optional[MitreInfoContainer] = None, + raw_metainfo_container: Optional[RawMetaInfoContainer] = None, + ) -> None: + self.id = id_ or str(uuid.uuid4()) + self.title = title or "" + self.index = index or [] + self.language = language or "" + self.risk_score = risk_score + self.type_ = type_ or "" + self.description = description or "" + self.author = [v.strip() for v in author] if author and author != [None] else [] + self.date = date or datetime.now().date().strftime("%Y-%m-%d") + self.output_table_fields = output_table_fields or [] + self.query_fields = query_fields or [] + self.function_fields = function_fields or [] + self.function_fields_map = function_fields_map or {} + self.license = license_ or "DRL 1.1" + self.severity = severity or SeverityType.low + self.references = references or [] + self.tags = tags or [] + self.mitre_attack = mitre_attack or MitreInfoContainer() + self.raw_mitre_attack = raw_mitre_attack or [] + self.status = status or "stable" + self.false_positives = false_positives or [] + self._source_mapping_ids = source_mapping_ids or [DEFAULT_MAPPING_NAME] + self.parsed_logsources = parsed_logsources or {} + self.timeframe = timeframe + self.query_period = query_period + self.raw_metainfo_container = raw_metainfo_container or RawMetaInfoContainer() + + @property + def author_str(self) -> str: + return ", ".join(self.author) + + @property + def source_mapping_ids(self) -> list[str]: + return sorted(self._source_mapping_ids) + + @source_mapping_ids.setter + def source_mapping_ids(self, source_mapping_ids: list[str]) -> None: + self._source_mapping_ids = source_mapping_ids + + +@dataclass +class RawQueryContainer: + query: str + language: str + meta_info: MetaInfoContainer = field(default_factory=MetaInfoContainer) + + +@dataclass +class RawQueryDictContainer: + query: dict + language: str + meta_info: dict + + +@dataclass +class TokenizedQueryContainer: + tokens: list[QUERY_TOKEN_TYPE] + meta_info: MetaInfoContainer + functions: ParsedFunctions = field(default_factory=ParsedFunctions) diff --git a/siem-converter/app/converter/platforms/athena/__init__.py b/uncoder-core/app/translator/core/models/query_tokens/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/athena/__init__.py rename to uncoder-core/app/translator/core/models/query_tokens/__init__.py diff --git a/uncoder-core/app/translator/core/models/query_tokens/field.py b/uncoder-core/app/translator/core/models/query_tokens/field.py new file mode 100644 index 00000000..557c9d90 --- /dev/null +++ b/uncoder-core/app/translator/core/models/query_tokens/field.py @@ -0,0 +1,47 @@ +from abc import ABC, abstractmethod +from typing import Optional + +from app.translator.core.mapping import DEFAULT_MAPPING_NAME, SourceMapping + + +class Alias: + def __init__(self, name: str): + self.name = name + + +class Field: + def __init__(self, source_name: str): + self.source_name = source_name + self.__generic_names_map = {} + + def get_generic_field_name(self, source_id: str) -> Optional[str]: + return self.__generic_names_map.get(source_id) + + def add_generic_names_map(self, generic_names_map: dict) -> None: + self.__generic_names_map = generic_names_map + + def set_generic_names_map(self, source_mappings: list[SourceMapping], default_mapping: SourceMapping) -> None: + generic_names_map = { + source_mapping.source_id: source_mapping.fields_mapping.get_generic_field_name(self.source_name) + or self.source_name + for source_mapping in source_mappings + } + if DEFAULT_MAPPING_NAME not in generic_names_map: + fields_mapping = default_mapping.fields_mapping + generic_names_map[DEFAULT_MAPPING_NAME] = ( + fields_mapping.get_generic_field_name(self.source_name) or self.source_name + ) + + self.__generic_names_map = generic_names_map + + +class PredefinedField: + def __init__(self, name: str): + self.name = name + + +class BaseFieldsGetter(ABC): + @property + @abstractmethod + def fields(self) -> list[Field]: + raise NotImplementedError("Abstract method") diff --git a/uncoder-core/app/translator/core/models/query_tokens/field_field.py b/uncoder-core/app/translator/core/models/query_tokens/field_field.py new file mode 100644 index 00000000..41d285c7 --- /dev/null +++ b/uncoder-core/app/translator/core/models/query_tokens/field_field.py @@ -0,0 +1,28 @@ +from app.translator.core.models.query_tokens.field import Alias, BaseFieldsGetter, Field +from app.translator.core.models.query_tokens.identifier import Identifier + + +class FieldField(BaseFieldsGetter): + def __init__( + self, + source_name_left: str, + operator: Identifier, + source_name_right: str, + is_alias_left: bool = False, + is_alias_right: bool = False, + ): + self.field_left = Field(source_name=source_name_left) if not is_alias_left else None + self.alias_left = Alias(name=source_name_left) if is_alias_left else None + self.operator = operator + self.field_right = Field(source_name=source_name_right) if not is_alias_right else None + self.alias_right = Alias(name=source_name_right) if is_alias_right else None + + @property + def fields(self) -> list[Field]: + fields = [] + if self.field_left: + fields.append(self.field_left) + if self.field_right: + fields.append(self.field_right) + + return fields diff --git a/uncoder-core/app/translator/core/models/query_tokens/field_value.py b/uncoder-core/app/translator/core/models/query_tokens/field_value.py new file mode 100644 index 00000000..79dfb720 --- /dev/null +++ b/uncoder-core/app/translator/core/models/query_tokens/field_value.py @@ -0,0 +1,39 @@ +from typing import Union + +from app.translator.core.custom_types.tokens import STR_SEARCH_OPERATORS +from app.translator.core.models.query_tokens.field import Alias, BaseFieldsGetter, Field, PredefinedField +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.value import Value +from app.translator.core.str_value_manager import StrValue + + +class FieldValue(BaseFieldsGetter, Value): + def __init__( + self, + source_name: str, + operator: Identifier, + value: Union[bool, int, str, StrValue, list, tuple], + is_alias: bool = False, + is_predefined_field: bool = False, + ): + super().__init__(value, cast_to_int=operator.token_type not in STR_SEARCH_OPERATORS) + # mapped by platform fields mapping + self.field = Field(source_name=source_name) if not (is_alias or is_predefined_field) else None + # not mapped + self.alias = Alias(name=source_name) if is_alias else None + # mapped by platform predefined fields mapping + self.predefined_field = PredefinedField(name=source_name) if is_predefined_field else None + self.operator = operator + + def __repr__(self): + if self.alias: + return f"{self.alias.name} {self.operator.token_type} {self.values}" + + if self.predefined_field: + return f"{self.predefined_field.name} {self.operator.token_type} {self.values}" + + return f"{self.field.source_name} {self.operator.token_type} {self.values}" + + @property + def fields(self) -> list[Field]: + return [self.field] if self.field else [] diff --git a/uncoder-core/app/translator/core/models/query_tokens/function_value.py b/uncoder-core/app/translator/core/models/query_tokens/function_value.py new file mode 100644 index 00000000..89bcd63b --- /dev/null +++ b/uncoder-core/app/translator/core/models/query_tokens/function_value.py @@ -0,0 +1,19 @@ +from typing import Union + +from app.translator.core.custom_types.tokens import STR_SEARCH_OPERATORS +from app.translator.core.models.functions.base import Function +from app.translator.core.models.query_tokens.field import BaseFieldsGetter, Field +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.value import Value +from app.translator.core.str_value_manager import StrValue + + +class FunctionValue(BaseFieldsGetter, Value): + def __init__(self, function: Function, operator: Identifier, value: Union[int, str, StrValue, list, tuple]): + super().__init__(value, cast_to_int=operator.token_type not in STR_SEARCH_OPERATORS) + self.function = function + self.operator = operator + + @property + def fields(self) -> list[Field]: + return self.function.fields diff --git a/uncoder-core/app/translator/core/models/query_tokens/identifier.py b/uncoder-core/app/translator/core/models/query_tokens/identifier.py new file mode 100644 index 00000000..462db2d1 --- /dev/null +++ b/uncoder-core/app/translator/core/models/query_tokens/identifier.py @@ -0,0 +1,18 @@ +from app.translator.core.custom_types.tokens import GroupType, LogicalOperatorType, OperatorType + + +class _IdentifierTokenType(LogicalOperatorType, GroupType, OperatorType): + pass + + +class Identifier: + valid_token_types = _IdentifierTokenType + + def __init__(self, *, token_type: str) -> None: + if token_type not in self.valid_token_types: + raise Exception(f"Unexpected token type: {token_type}") + + self.token_type = token_type + + def __repr__(self): + return f"{self.token_type}" diff --git a/uncoder-core/app/translator/core/models/query_tokens/keyword.py b/uncoder-core/app/translator/core/models/query_tokens/keyword.py new file mode 100644 index 00000000..09382791 --- /dev/null +++ b/uncoder-core/app/translator/core/models/query_tokens/keyword.py @@ -0,0 +1,21 @@ +from typing import Union + +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.value import Value + + +class Keyword(Value): + def __init__(self, value: Union[str, list[str]]): + super().__init__(value) + self.operator: Identifier = Identifier(token_type=OperatorType.KEYWORD) + self.name = "keyword" + + def _add_value(self, value: Union[str, list[str]]) -> None: + if value and isinstance(value, (list, tuple)): + self.values.extend(value) + elif value and isinstance(value, str): + self.values.append(value) + + def __repr__(self): + return f"{self.name} {self.operator.token_type} {self.values}" diff --git a/uncoder-core/app/translator/core/models/query_tokens/value.py b/uncoder-core/app/translator/core/models/query_tokens/value.py new file mode 100644 index 00000000..d3d77eb0 --- /dev/null +++ b/uncoder-core/app/translator/core/models/query_tokens/value.py @@ -0,0 +1,30 @@ +from typing import Optional, Union + +from app.translator.core.str_value_manager import StrValue + + +class Value: + def __init__(self, value: Union[bool, int, str, StrValue, list, tuple], cast_to_int: bool = False): + self.values = [] + self.__cast_to_int = cast_to_int + self._add_value(value) + + @property + def value(self) -> Union[bool, int, str, StrValue, list[Union[int, str, StrValue]]]: + if isinstance(self.values, list) and len(self.values) == 1: + return self.values[0] + return self.values + + @value.setter + def value(self, new_value: Union[bool, int, str, StrValue, list[Union[int, str, StrValue]]]) -> None: + self.values = [] + self._add_value(new_value) + + def _add_value(self, value: Optional[Union[bool, int, str, StrValue, list, tuple]]) -> None: + if value and isinstance(value, (list, tuple)): + for v in value: + self._add_value(v) + elif value and isinstance(value, str) and value.isnumeric() and self.__cast_to_int: + self.values.append(int(value)) + elif value is not None and isinstance(value, (bool, int, str)): + self.values.append(value) diff --git a/uncoder-core/app/translator/core/parser.py b/uncoder-core/app/translator/core/parser.py new file mode 100644 index 00000000..da7330eb --- /dev/null +++ b/uncoder-core/app/translator/core/parser.py @@ -0,0 +1,90 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from abc import ABC, abstractmethod +from typing import Optional, Union + +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.exceptions.parser import TokenizerGeneralException +from app.translator.core.functions import PlatformFunctions +from app.translator.core.mapping import BasePlatformMappings, SourceMapping +from app.translator.core.models.functions.base import Function +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.models.query_tokens.field import Field +from app.translator.core.models.query_tokens.field_field import FieldField +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.function_value import FunctionValue +from app.translator.core.tokenizer import QueryTokenizer + + +class QueryParser(ABC): + wrapped_with_comment_pattern: str = None + details: PlatformDetails = None + + def remove_comments(self, text: str) -> str: + if self.wrapped_with_comment_pattern: + return re.sub(self.wrapped_with_comment_pattern, "\n", text, flags=re.MULTILINE).strip() + + return text + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + return RawQueryContainer(query=text, language=language) + + @abstractmethod + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + raise NotImplementedError("Abstract method") + + +class PlatformQueryParser(QueryParser, ABC): + mappings: BasePlatformMappings = None + tokenizer: QueryTokenizer = None + platform_functions: PlatformFunctions = None + + def get_query_tokens(self, query: str) -> list[QUERY_TOKEN_TYPE]: + if not query: + raise TokenizerGeneralException("Can't translate empty query. Please provide more details") + return self.tokenizer.tokenize(query=query) + + @staticmethod + def get_field_tokens( + query_tokens: list[QUERY_TOKEN_TYPE], functions: Optional[list[Function]] = None + ) -> tuple[list[Field], list[Field], dict[str, list[Field]]]: + query_field_tokens = [] + function_field_tokens = [] + function_field_tokens_map = {} + for token in query_tokens: + if isinstance(token, (FieldField, FieldValue, FunctionValue)): + query_field_tokens.extend(token.fields) + + for func in functions or []: + function_field_tokens.extend(func.fields) + function_field_tokens_map[func.name] = func.fields + + return query_field_tokens, function_field_tokens, function_field_tokens_map + + def get_source_mappings( + self, field_tokens: list[Field], log_sources: dict[str, list[Union[int, str]]] + ) -> list[SourceMapping]: + field_names = [field.source_name for field in field_tokens] + source_mappings = self.mappings.get_source_mappings_by_fields_and_log_sources( + field_names=field_names, log_sources=log_sources + ) + self.tokenizer.set_field_tokens_generic_names_map(field_tokens, source_mappings, self.mappings.default_mapping) + return source_mappings diff --git a/uncoder-core/app/translator/core/parser_cti.py b/uncoder-core/app/translator/core/parser_cti.py new file mode 100644 index 00000000..05ca14fa --- /dev/null +++ b/uncoder-core/app/translator/core/parser_cti.py @@ -0,0 +1,137 @@ +import ipaddress +import re +from typing import Optional + +from pydantic import BaseModel + +from app.translator.core.exceptions.iocs import EmptyIOCSException, IocsLimitExceededException +from app.translator.tools.const import ( + DOMAIN_IOC_REGEXP_PATTERN, + HASH_MAP, + IP_IOC_REGEXP_PATTERN, + URL_IOC_REGEXP_PATTERN, + HashType, + IocParsingRule, + IOCType, + hash_regexes, +) + + +class Iocs(BaseModel): + ip: list[str] = [] + url: list[str] = [] + domain: list[str] = [] + hash_dict: dict = {} + + def get_total_count(self) -> int: + hash_len = 0 + for value in self.hash_dict.values(): + hash_len += len(value) + return len(self.ip) + len(self.url) + len(self.domain) + hash_len + + def return_iocs(self, include_source_ip: bool = False) -> dict: + if all(not value for value in [self.ip, self.url, self.domain, self.hash_dict]): + raise EmptyIOCSException + result = {"DestinationIP": self.ip, "URL": self.url, "Domain": self.domain} + if include_source_ip: + result["SourceIP"] = self.ip + for key, value in self.hash_dict.items(): + result[HASH_MAP[key]] = value + return result + + +class CTIParser: + def get_iocs_from_string( + self, + string: str, + include_ioc_types: Optional[list[IOCType]] = None, + include_hash_types: Optional[list[HashType]] = None, + exceptions: Optional[list[str]] = None, + ioc_parsing_rules: Optional[list[IocParsingRule]] = None, + limit: Optional[int] = None, + include_source_ip: Optional[bool] = False, + ) -> dict: + iocs = Iocs() + string = self.replace_dots_hxxp(string, ioc_parsing_rules) + if not include_ioc_types or "ip" in include_ioc_types: + iocs.ip.extend(self._find_all_str_by_regex(string, IP_IOC_REGEXP_PATTERN)) + if not include_ioc_types or "domain" in include_ioc_types: + iocs.domain.extend(self._find_all_str_by_regex(string, DOMAIN_IOC_REGEXP_PATTERN)) + if not include_ioc_types or "url" in include_ioc_types: + iocs.url.extend(self._find_all_str_by_regex(string, URL_IOC_REGEXP_PATTERN)) + if not include_ioc_types or "hash" in include_ioc_types: + if not include_hash_types: + include_hash_types = list(hash_regexes.keys()) + for hash_type in include_hash_types: + iocs.hash_dict[hash_type] = self._find_all_str_by_regex(string, hash_regexes[hash_type]) + iocs = self.remove_duplicates(iocs) + iocs = self.remove_exceptions(iocs, exceptions) + if ioc_parsing_rules is None or "remove_private_and_reserved_ips" in ioc_parsing_rules: + self._exclude_private_and_reserved_ips(iocs) + if limit is not None: + total_count = iocs.get_total_count() + if total_count > limit: + raise IocsLimitExceededException(f"IOCs count {total_count} exceeds limit {limit}.") + return iocs.return_iocs(include_source_ip) + + def replace_dots_hxxp(self, string: str, ioc_parsing_rules: Optional[list[IocParsingRule]]) -> str: + if ioc_parsing_rules is None or "replace_dots" in ioc_parsing_rules: + string = self._replace_dots(string) + if ioc_parsing_rules is None or "replace_hxxp" in ioc_parsing_rules: + string = self._replace_hxxp(string) + return string + + def remove_duplicates(self, iocs: Iocs) -> Iocs: + iocs.ip = self._remove_duplicates_from_list(iocs.ip) + iocs.domain = self._remove_duplicates_from_list(iocs.domain) + iocs.url = self._remove_duplicates_from_list(iocs.url) + for key, value in iocs.hash_dict.items(): + iocs.hash_dict[key] = self._remove_duplicates_from_list(value) + return iocs + + def remove_exceptions(self, iocs: Iocs, exceptions: Optional[list[str]] = None) -> Iocs: + iocs.ip = self._remove_exceptions(iocs.ip, exceptions) + iocs.domain = self._remove_exceptions(iocs.domain, exceptions) + iocs.url = self._remove_exceptions(iocs.url, exceptions) + for key, value in iocs.hash_dict.items(): + iocs.hash_dict[key] = self._remove_exceptions(value, exceptions) + return iocs + + @staticmethod + def _find_all_str_by_regex(s: str, pattern: str) -> list[str]: + return re.findall(pattern, s, re.MULTILINE) + + @staticmethod + def _remove_duplicates_from_list(ls: list) -> list: + return list(dict.fromkeys(ls)) + + @classmethod + def _remove_exceptions(cls, values: list[str], exceptions: Optional[list[str]]) -> list[str]: + if not exceptions: + return values + return [v for v in values if not cls._str_contain_one_of_exceptions(v, exceptions)] + + @staticmethod + def _str_contain_one_of_exceptions(s: str, exceptions: list[str]) -> bool: + return any(e for e in exceptions if e in s) + + @staticmethod + def _replace_dots(s: str) -> str: + return s.replace("(.)", ".").replace("[.]", ".").replace("{.}", ".") + + @staticmethod + def _replace_hxxp(s: str) -> str: + return s.replace("hxxp", "http") + + @classmethod + def _exclude_private_and_reserved_ips(cls, iocs: Iocs) -> None: + iocs.ip = [ip for ip in iocs.ip if not cls._is_private_or_reserved(ip)] + + @staticmethod + def _is_private_or_reserved(ip: str) -> bool: + try: + addr = ipaddress.ip_address(ip) + except Exception: + return False + else: + return addr.is_private or addr.is_reserved diff --git a/uncoder-core/app/translator/core/render.py b/uncoder-core/app/translator/core/render.py new file mode 100644 index 00000000..673cc6fa --- /dev/null +++ b/uncoder-core/app/translator/core/render.py @@ -0,0 +1,489 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +import itertools +from abc import ABC, abstractmethod +from collections.abc import Callable +from typing import ClassVar, Optional, Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.context_vars import return_only_first_query_ctx_var, wrap_query_with_meta_info_ctx_var +from app.translator.core.custom_types.tokens import LogicalOperatorType, OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.exceptions.core import StrictPlatformException +from app.translator.core.exceptions.parser import UnsupportedOperatorException +from app.translator.core.exceptions.render import UnsupportedRenderMethod +from app.translator.core.functions import PlatformFunctions +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature, SourceMapping +from app.translator.core.models.functions.base import Function, RenderedFunctions +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, RawQueryContainer, TokenizedQueryContainer +from app.translator.core.models.query_tokens.field import Field, PredefinedField +from app.translator.core.models.query_tokens.field_field import FieldField +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.function_value import FunctionValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.keyword import Keyword +from app.translator.core.str_value_manager import StrValue, StrValueManager + + +class BaseFieldValueRender(ABC): + details: PlatformDetails = None + escape_manager: EscapeManager = None + str_value_manager: StrValueManager = None + + def __init__(self, or_token: str): + self.modifiers_map: dict[str, Callable[[str, DEFAULT_VALUE_TYPE], str]] = { + OperatorType.EQ: self.equal_modifier, + OperatorType.NOT_EQ: self.not_equal_modifier, + OperatorType.LT: self.less_modifier, + OperatorType.LTE: self.less_or_equal_modifier, + OperatorType.GT: self.greater_modifier, + OperatorType.GTE: self.greater_or_equal_modifier, + OperatorType.CONTAINS: self.contains_modifier, + OperatorType.NOT_CONTAINS: self.not_contains_modifier, + OperatorType.ENDSWITH: self.endswith_modifier, + OperatorType.NOT_ENDSWITH: self.not_endswith_modifier, + OperatorType.STARTSWITH: self.startswith_modifier, + OperatorType.NOT_STARTSWITH: self.not_startswith_modifier, + OperatorType.REGEX: self.regex_modifier, + OperatorType.NOT_REGEX: self.not_regex_modifier, + OperatorType.KEYWORD: self.keywords, + OperatorType.IS_NONE: self.is_none, + OperatorType.IS_NOT_NONE: self.is_not_none, + } + self.or_token = f" {or_token} " + + @staticmethod + def _get_value_type(field_name: str, value: Union[int, str, StrValue], value_type: Optional[str] = None) -> str: # noqa: ARG004 + return value_type or ValueType.value + + @staticmethod + def _wrap_str_value(value: str) -> str: + return value + + @staticmethod + def _wrap_int_value(value: int) -> str: + return str(value) + + @staticmethod + def _map_bool_value(value: bool) -> str: + return "true" if value else "false" + + def _pre_process_value( + self, + field: str, + value: Union[bool, int, str, StrValue], + value_type: str = ValueType.value, + wrap_str: bool = False, + wrap_int: bool = False, + ) -> Union[int, str]: + value_type = self._get_value_type(field, value, value_type) + if isinstance(value, StrValue): + value = self.str_value_manager.from_container_to_str(value, value_type) + return self._wrap_str_value(value) if wrap_str else value + if isinstance(value, str): + value = self.str_value_manager.escape_manager.escape(value, value_type) + return self._wrap_str_value(value) if wrap_str else value + if isinstance(value, bool): + return self._map_bool_value(value) + if isinstance(value, int): + return self._wrap_int_value(value) if wrap_int else value + return value + + def _pre_process_values_list( + self, field: str, values: list[Union[int, str, StrValue]], value_type: str = ValueType.value + ) -> list[str]: + processed = [] + for val in values: + value_type = self._get_value_type(field, val, value_type) + if isinstance(val, StrValue): + processed.append(self.str_value_manager.from_container_to_str(val, value_type)) + elif isinstance(val, str): + processed.append(self.str_value_manager.escape_manager.escape(val, value_type)) + else: + processed.append(str(val)) + return processed + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.EQ.capitalize()) + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.NOT_EQ.capitalize()) + + def less_modifier(self, field: str, value: Union[int, str]) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.LT.capitalize()) + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.LTE.capitalize()) + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.GT.capitalize()) + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.GTE.capitalize()) + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.CONTAINS.capitalize()) + + def not_contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.NOT_CONTAINS.capitalize()) + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.ENDSWITH.capitalize()) + + def not_endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.NOT_ENDSWITH.capitalize()) + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.STARTSWITH.capitalize()) + + def not_startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.NOT_STARTSWITH.capitalize()) + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.REGEX.capitalize()) + + def not_regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.NOT_REGEX.capitalize()) + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.KEYWORD.capitalize()) + + def is_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.IS_NONE.capitalize()) + + def is_not_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method=OperatorType.IS_NOT_NONE.capitalize()) + + def apply_value(self, value: Union[str, int], value_type: str = ValueType.value) -> Union[str, int]: + return self.escape_manager.escape(value, value_type) + + def apply_field_value(self, field: str, operator: Identifier, value: DEFAULT_VALUE_TYPE) -> str: + if modifier_function := self.modifiers_map.get(operator.token_type): + return modifier_function(field, value) + raise UnsupportedOperatorException(operator.token_type) + + +class BaseFieldFieldRender(ABC): + operators_map: ClassVar[dict[str, str]] = {} + + def apply_field_field(self, field_left: str, operator: Identifier, field_right: str) -> str: + if mapped_operator := self.operators_map.get(operator.token_type): + return f"{field_left} {mapped_operator} {field_right}" + raise UnsupportedOperatorException(operator.token_type) + + +class QueryRender(ABC): + comment_symbol: str = None + details: PlatformDetails = None + is_single_line_comment: bool = False + unsupported_functions_text = "Unsupported functions were excluded from the result query:" + unmapped_fields_text = "Unmapped fields: " + + platform_functions: PlatformFunctions = None + + def __init__(self): + self.init_platform_functions() + + def init_platform_functions(self) -> None: + self.platform_functions = PlatformFunctions() + self.platform_functions.platform_query_render = self + + def render_not_supported_functions(self, not_supported_functions: list) -> str: + line_template = f"{self.comment_symbol} " if self.comment_symbol and self.is_single_line_comment else "" + not_supported_functions_str = "\n".join(line_template + func.lstrip() for func in not_supported_functions) + return "\n\n" + self.wrap_with_comment(f"{self.unsupported_functions_text}\n{not_supported_functions_str}") + + def wrap_with_not_supported_functions(self, query: str, not_supported_functions: Optional[list] = None) -> str: + if not_supported_functions and wrap_query_with_meta_info_ctx_var.get(): + rendered_not_supported = self.render_not_supported_functions(not_supported_functions) + return query + rendered_not_supported + + return query + + def wrap_with_unmapped_fields(self, query: str, fields: Optional[list[str]]) -> str: + if wrap_query_with_meta_info_ctx_var.get() and fields: + return query + "\n\n" + self.wrap_with_comment(f"{self.unmapped_fields_text}{', '.join(fields)}") + return query + + def wrap_with_comment(self, value: str) -> str: + return f"{self.comment_symbol} {value}" + + @abstractmethod + def generate( + self, raw_query_container: RawQueryContainer, tokenized_query_container: Optional[TokenizedQueryContainer] + ) -> str: + raise NotImplementedError("Abstract method") + + +class PlatformQueryRender(QueryRender): + mappings: BasePlatformMappings = None + + or_token = "or" + and_token = "and" + not_token = "not" + + group_token = "(%s)" + query_parts_delimiter = " " + + field_field_render = BaseFieldFieldRender() + field_value_render = BaseFieldValueRender(or_token=or_token) + + predefined_fields_map: ClassVar[dict[str, str]] = {} + raw_log_field_patterns_map: ClassVar[dict[str, str]] = {} + + def __init__(self): + super().__init__() + self.logical_operators_map = { + LogicalOperatorType.AND: f" {self.and_token} ", + LogicalOperatorType.OR: f" {self.or_token} ", + LogicalOperatorType.NOT: f" {self.not_token} ", + } + + def generate_prefix(self, log_source_signature: Optional[LogSourceSignature], functions_prefix: str = "") -> str: # noqa: ARG002 + if log_source_signature and str(log_source_signature): + return f"{log_source_signature} {self.and_token}" + return "" + + def generate_functions(self, functions: list[Function], source_mapping: SourceMapping) -> RenderedFunctions: + return self.platform_functions.render(functions, source_mapping) + + def map_predefined_field(self, predefined_field: PredefinedField) -> str: + if not (mapped_predefined_field_name := self.predefined_fields_map.get(predefined_field.name)): + if self.mappings.is_strict_mapping: + raise StrictPlatformException(platform_name=self.details.name, fields=[predefined_field.name]) + + return predefined_field.name + + return mapped_predefined_field_name + + def apply_token(self, token: QUERY_TOKEN_TYPE, source_mapping: SourceMapping) -> str: + if isinstance(token, FieldValue): + if token.alias: + mapped_fields = [token.alias.name] + elif token.predefined_field: + mapped_fields = [self.map_predefined_field(token.predefined_field)] + else: + mapped_fields = self.mappings.map_field(token.field, source_mapping) + joined = self.logical_operators_map[LogicalOperatorType.OR].join( + [ + self.field_value_render.apply_field_value(field=field, operator=token.operator, value=token.value) + for field in mapped_fields + ] + ) + return self.group_token % joined if len(mapped_fields) > 1 else joined + if isinstance(token, FieldField): + alias_left, field_left = token.alias_left, token.field_left + mapped_fields_left = ( + [alias_left.name] if alias_left else self.mappings.map_field(field_left, source_mapping) + ) + alias_right, field_right = token.alias_right, token.field_right + mapped_fields_right = ( + [alias_right.name] if alias_right else self.mappings.map_field(field_right, source_mapping) + ) + cross_paired_fields = list(itertools.product(mapped_fields_left, mapped_fields_right)) + joined = self.logical_operators_map[LogicalOperatorType.OR].join( + [ + self.field_field_render.apply_field_field(pair[0], token.operator, pair[1]) + for pair in cross_paired_fields + ] + ) + return self.group_token % joined if len(cross_paired_fields) > 1 else joined + if isinstance(token, FunctionValue): + func_render = self.platform_functions.manager.get_render(token.function.name) + rendered_func = func_render.render(token.function, source_mapping) + return self.field_value_render.apply_field_value( + field=rendered_func, operator=token.operator, value=token.value + ) + if isinstance(token, Keyword): + return self.field_value_render.apply_field_value(field="", operator=token.operator, value=token.value) + if token.token_type in LogicalOperatorType: + return self.logical_operators_map.get(token.token_type) + + return token.token_type + + def generate_query(self, tokens: list[QUERY_TOKEN_TYPE], source_mapping: SourceMapping) -> str: + result_values = [] + for token in tokens: + result_values.append(self.apply_token(token=token, source_mapping=source_mapping)) + + return "".join(result_values) + + def wrap_with_meta_info(self, query: str, meta_info: Optional[MetaInfoContainer]) -> str: + if wrap_query_with_meta_info_ctx_var.get() and meta_info and (meta_info.id or meta_info.title): + meta_info_dict = { + "name: ": meta_info.title, + "uuid: ": meta_info.id, + "author: ": meta_info.author_str or "not defined in query/rule", + "licence: ": meta_info.license, + } + query_meta_info = "\n".join( + self.wrap_with_comment(f"{key}{value}") for key, value in meta_info_dict.items() if value + ) + query = f"{query}\n\n{query_meta_info}" + return query + + @staticmethod + def _finalize_search_query(query: str) -> str: + return query + + def _join_query_parts(self, prefix: str, query: str, functions: str) -> str: + parts = filter(lambda s: bool(s), map(str.strip, [prefix, self._finalize_search_query(query), functions])) + return self.query_parts_delimiter.join(parts) + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = self._join_query_parts(prefix, query, functions) + query = self.wrap_with_meta_info(query, meta_info) + query = self.wrap_with_unmapped_fields(query, unmapped_fields) + return self.wrap_with_not_supported_functions(query, not_supported_functions) + + @staticmethod + def unique_queries(queries_map: dict[str, str]) -> dict[str, dict[str]]: + unique_queries = {} + for source_id, query in queries_map.items(): + unique_queries.setdefault(query, []).append(source_id) + + return unique_queries + + def finalize(self, queries_map: dict[str, str]) -> str: + if len(set(queries_map.values())) == 1: + return next(iter(queries_map.values())) + unique_queries = self.unique_queries(queries_map=queries_map) + result = "" + for query, source_ids in unique_queries.items(): + result = result + self.wrap_with_comment(", ".join(source_ids)) + f"\n{query}\n\n" + + return result + + def generate_from_raw_query_container(self, query_container: RawQueryContainer) -> str: + return self.finalize_query( + prefix="", query=query_container.query, functions="", meta_info=query_container.meta_info + ) + + def process_raw_log_field(self, field: str, field_type: str) -> Optional[str]: + if raw_log_field_pattern := self.raw_log_field_patterns_map.get(field_type): + return raw_log_field_pattern.format(field=field) + + def process_raw_log_field_prefix(self, field: str, source_mapping: SourceMapping) -> Optional[list]: + if isinstance(field, list): + prefix_list = [] + for f in field: + if _prefix_list := self.process_raw_log_field_prefix(field=f, source_mapping=source_mapping): + prefix_list.extend(_prefix_list) + return prefix_list + if raw_log_field_type := source_mapping.raw_log_fields.get(field): + return [self.process_raw_log_field(field=field, field_type=raw_log_field_type)] + + def generate_extra_conditions(self, source_mapping: SourceMapping) -> list[QUERY_TOKEN_TYPE]: # noqa: ARG002 + return [] + + def generate_raw_log_fields(self, fields: list[Field], source_mapping: SourceMapping) -> str: + if not self.raw_log_field_patterns_map: + return "" + defined_raw_log_fields = [] + for field in fields: + mapped_field = source_mapping.fields_mapping.get_platform_field_name(generic_field_name=field.source_name) + if not mapped_field: + generic_field_name = field.get_generic_field_name(source_mapping.source_id) + mapped_field = source_mapping.fields_mapping.get_platform_field_name( + generic_field_name=generic_field_name + ) + if not mapped_field and self.mappings.is_strict_mapping: + raise StrictPlatformException( + platform_name=self.details.name, fields=[field.source_name], mapping=source_mapping.source_id + ) + if prefix_list := self.process_raw_log_field_prefix(field=mapped_field, source_mapping=source_mapping): + for prefix in prefix_list: + if prefix not in defined_raw_log_fields: + defined_raw_log_fields.append(prefix) + return "\n".join(defined_raw_log_fields) + + def _generate_from_tokenized_query_container_by_source_mapping( + self, query_container: TokenizedQueryContainer, source_mapping: SourceMapping + ) -> str: + unmapped_fields = self.mappings.check_fields_mapping_existence( + query_container.meta_info.query_fields, + query_container.meta_info.function_fields_map, + self.platform_functions.manager.supported_render_names, + source_mapping, + ) + rendered_functions = self.generate_functions(query_container.functions.functions, source_mapping) + prefix = self.generate_prefix(source_mapping.log_source_signature, rendered_functions.rendered_prefix) + + if source_mapping.raw_log_fields: + defined_raw_log_fields = self.generate_raw_log_fields( + fields=query_container.meta_info.query_fields + query_container.meta_info.function_fields, + source_mapping=source_mapping, + ) + prefix += f"\n{defined_raw_log_fields}" + if source_mapping.conditions: + extra_tokens = self.generate_extra_conditions(source_mapping=source_mapping) + query_container.tokens = [*extra_tokens, *query_container.tokens] + query = self.generate_query(tokens=query_container.tokens, source_mapping=source_mapping) + not_supported_functions = query_container.functions.not_supported + rendered_functions.not_supported + return self.finalize_query( + prefix=prefix, + query=query, + functions=rendered_functions.rendered, + not_supported_functions=not_supported_functions, + unmapped_fields=unmapped_fields, + meta_info=query_container.meta_info, + source_mapping=source_mapping, + ) + + def generate_from_tokenized_query_container(self, query_container: TokenizedQueryContainer) -> str: + queries_map = {} + errors = [] + source_mappings = self.mappings.get_source_mappings_by_ids(query_container.meta_info.source_mapping_ids) + + for source_mapping in source_mappings: + try: + finalized_query = self._generate_from_tokenized_query_container_by_source_mapping( + query_container, source_mapping + ) + if return_only_first_query_ctx_var.get() is True: + return finalized_query + queries_map[source_mapping.source_id] = finalized_query + except StrictPlatformException as err: + errors.append(err) + continue + + if not queries_map and errors: + raise errors[0] + return self.finalize(queries_map) + + def generate( + self, raw_query_container: RawQueryContainer, tokenized_query_container: Optional[TokenizedQueryContainer] + ) -> str: + if tokenized_query_container: + return self.generate_from_tokenized_query_container(tokenized_query_container) + return self.generate_from_raw_query_container(raw_query_container) diff --git a/uncoder-core/app/translator/core/render_cti.py b/uncoder-core/app/translator/core/render_cti.py new file mode 100644 index 00000000..20bfb7bf --- /dev/null +++ b/uncoder-core/app/translator/core/render_cti.py @@ -0,0 +1,69 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.iocs import IocsChunkValue +from app.translator.core.models.platform_details import PlatformDetails + + +class RenderCTI: + # EXAMPLE OF A BACKEND ATTRIBUTES + field_value_template: str = "{key} =~ '{value}'" + or_operator: str = " or " + group_or_operator: str = " or " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "union * | where ({result})\n" + final_result_for_one: str = "union * | where {result}\n" + default_mapping = None + details: PlatformDetails = None + + def create_field_value(self, field: str, value: str, generic_field: str) -> str: # noqa: ARG002 + return self.field_value_template.format(key=field, value=value) + + def render(self, data: list[list[IocsChunkValue]]) -> list[str]: + final_result = [] + for iocs_chunk in data: + data_values = self.collect_data_values(iocs_chunk) + if len(data_values) > 1: + final_result.append(self.final_result_for_many.format(result=self.group_or_operator.join(data_values))) + else: + final_result.append(self.final_result_for_one.format(result=data_values[0])) + return final_result + + def collect_data_values(self, chunk: list[IocsChunkValue]) -> list[str]: + data_values = [] + key_chunk = [] + processing_key = chunk[0].platform_field + for value in chunk: + if processing_key != value.platform_field: + data_values.append( + self.or_group.format(or_group=self.or_operator.join(key_chunk), processing_key=processing_key) + ) + key_chunk = [] + processing_key = value.platform_field + key_chunk.append( + self.create_field_value( + field=value.platform_field, value=value.value, generic_field=value.generic_field + ) + ) + if key_chunk: + data_values.append( + self.or_group.format(or_group=self.or_operator.join(key_chunk), processing_key=processing_key) + ) + return data_values diff --git a/uncoder-core/app/translator/core/str_value_manager.py b/uncoder-core/app/translator/core/str_value_manager.py new file mode 100644 index 00000000..1151d7a3 --- /dev/null +++ b/uncoder-core/app/translator/core/str_value_manager.py @@ -0,0 +1,258 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import ClassVar, Optional, TypeVar, Union + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager + + +class BaseSpecSymbol: + ... + + +SpecSymbolType = TypeVar("SpecSymbolType", bound=BaseSpecSymbol) + + +class SingleSymbolWildCard(BaseSpecSymbol): + ... + + +class UnboundLenWildCard(BaseSpecSymbol): + ... + + +class ReEndOfStrSymbol(BaseSpecSymbol): + ... + + +class ReWordBoundarySymbol(BaseSpecSymbol): + ... + + +class ReWordSymbol(BaseSpecSymbol): + ... + + +class ReDigitalSymbol(BaseSpecSymbol): + ... + + +class ReAnySymbol(BaseSpecSymbol): + ... + + +class ReWhiteSpaceSymbol(BaseSpecSymbol): + ... + + +class ReOneOrMoreQuantifier(BaseSpecSymbol): + ... + + +class ReZeroOrMoreQuantifier(BaseSpecSymbol): + ... + + +class ReZeroOrOneQuantifier(BaseSpecSymbol): + ... + + +class ReLeftParenthesis(BaseSpecSymbol): + ... + + +class ReRightParenthesis(BaseSpecSymbol): + ... + + +class ReLeftSquareBracket(BaseSpecSymbol): + ... + + +class ReRightSquareBracket(BaseSpecSymbol): + ... + + +class ReLeftCurlyBracket(BaseSpecSymbol): + ... + + +class ReRightCurlyBracket(BaseSpecSymbol): + ... + + +class ReOrOperator(BaseSpecSymbol): + ... + + +class ReCaretSymbol(BaseSpecSymbol): + ... + + +class ReCommaSymbol(BaseSpecSymbol): + ... + + +class ReHyphenSymbol(BaseSpecSymbol): + ... + + +class StrValue(str): + def __new__(cls, value: str, split_value: Optional[list[Union[str, SpecSymbolType]]] = None): # noqa: ARG003 + return super().__new__(cls, value) + + def __init__( + self, + value: str, # noqa: ARG002 + split_value: Optional[list[Union[str, SpecSymbolType]]] = None, + ) -> None: + self.split_value = split_value or [] + + @property + def has_spec_symbols(self) -> bool: + return any(isinstance(el, BaseSpecSymbol) for el in self.split_value) + + +RE_STR_SPEC_SYMBOLS_MAP = { + "?": ReZeroOrOneQuantifier, + "*": ReZeroOrMoreQuantifier, + "+": ReOneOrMoreQuantifier, + "^": ReCaretSymbol, + "$": ReEndOfStrSymbol, + ".": ReAnySymbol, + "[": ReLeftSquareBracket, + "]": ReRightSquareBracket, + "(": ReLeftParenthesis, + ")": ReRightParenthesis, + "{": ReLeftCurlyBracket, + "}": ReRightCurlyBracket, + "|": ReOrOperator, + ",": ReCommaSymbol, + "-": ReHyphenSymbol, +} + + +CONTAINER_SPEC_SYMBOLS_MAP = { + SingleSymbolWildCard: "?", + UnboundLenWildCard: "*", + ReAnySymbol: ".", + ReWordBoundarySymbol: r"\b", + ReWordSymbol: r"\w", + ReDigitalSymbol: r"\d", + ReWhiteSpaceSymbol: r"\s", + ReZeroOrMoreQuantifier: "*", + ReOneOrMoreQuantifier: "+", + ReZeroOrOneQuantifier: "?", + ReLeftSquareBracket: "[", + ReRightSquareBracket: "]", + ReLeftParenthesis: "(", + ReRightParenthesis: ")", + ReLeftCurlyBracket: "{", + ReRightCurlyBracket: "}", + ReOrOperator: "|", + ReCaretSymbol: "^", + ReEndOfStrSymbol: "$", + ReCommaSymbol: ",", + ReHyphenSymbol: "-", +} + + +class StrValueManager: + escape_manager: EscapeManager = None + str_spec_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = {} + re_str_alpha_num_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = {} + re_str_spec_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = {} + container_spec_symbols_map: ClassVar[dict[type[BaseSpecSymbol], str]] = CONTAINER_SPEC_SYMBOLS_MAP + + @staticmethod + def from_str_to_container( + value: str, + value_type: str = ValueType.value, # noqa: ARG004 + escape_symbol: Optional[str] = None, # noqa: ARG004 + ) -> StrValue: + return StrValue(value=value, split_value=[value]) + + def from_re_str_to_container(self, value: str) -> StrValue: + split = [] + prev_char = None + inside_curly_brackets = False + inside_square_brackets = False + for char in value: + if prev_char == "\\": + if char == "\\": + split.append(char) + prev_char = None + continue + if char in self.re_str_alpha_num_symbols_map: + split.append(self.re_str_alpha_num_symbols_map[char]()) + else: + split.append(char) + elif char in self.re_str_spec_symbols_map: + if char == "{": + inside_curly_brackets = True + elif char == "}": + inside_curly_brackets = False + elif char == "[": + inside_square_brackets = True + elif char == "]": + inside_square_brackets = False + elif ( + char == "," + and not inside_curly_brackets + or char == "-" + and (not inside_square_brackets or isinstance(split[-1], ReLeftSquareBracket)) + ): + split.append(char) + continue + split.append(self.re_str_spec_symbols_map[char]()) + elif char != "\\": + split.append(char) + + prev_char = char + + return StrValue(value, self._concat(split)) + + def from_container_to_str(self, container: StrValue, value_type: str = ValueType.value) -> str: + result = "" + for el in container.split_value: + if isinstance(el, str): + result += self.escape_manager.escape(el, value_type) + elif isinstance(el, BaseSpecSymbol) and (pattern := self.container_spec_symbols_map.get(type(el))): + result += pattern + + return result + + @staticmethod + def _concat(split: list[Union[str, SpecSymbolType]]) -> list[Union[str, SpecSymbolType]]: + result = [] + sub_str = "" + for el in split: + if isinstance(el, str): + sub_str += el + elif isinstance(el, BaseSpecSymbol): + if sub_str: + result.append(sub_str) + result.append(el) + sub_str = "" + + if sub_str: + result.append(sub_str) + + return result diff --git a/uncoder-core/app/translator/core/tokenizer.py b/uncoder-core/app/translator/core/tokenizer.py new file mode 100644 index 00000000..9a963473 --- /dev/null +++ b/uncoder-core/app/translator/core/tokenizer.py @@ -0,0 +1,356 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from abc import ABC, abstractmethod +from typing import Any, ClassVar, Optional, Union + +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.tokens import GroupType, LogicalOperatorType, OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.exceptions.parser import ( + QueryParenthesesException, + TokenizerGeneralException, + UnsupportedOperatorException, +) +from app.translator.core.functions import PlatformFunctions +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.query_tokens.field import Field +from app.translator.core.models.query_tokens.field_field import FieldField +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.function_value import FunctionValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.keyword import Keyword +from app.translator.core.str_value_manager import StrValue, StrValueManager +from app.translator.tools.utils import get_match_group + + +class BaseTokenizer(ABC): + @abstractmethod + def tokenize(self, query: str) -> list[QUERY_TOKEN_TYPE]: + raise NotImplementedError + + +class QueryTokenizer(BaseTokenizer): + # used to generate re pattern. so the keys order is important + single_value_operators_map: ClassVar[dict[str, str]] = {} + # used to generate re pattern. so the keys order is important + multi_value_operators_map: ClassVar[dict[str, str]] = {} + # used to generate re pattern. so the keys order is important + fields_operator_map: ClassVar[dict[str, str]] = {} + operators_map: ClassVar[dict[str, str]] = {} # used to generate re pattern. so the keys order is important + + logical_operators_map: ClassVar[dict[str, str]] = { + "and": LogicalOperatorType.AND, + "AND": LogicalOperatorType.AND, + "or": LogicalOperatorType.OR, + "OR": LogicalOperatorType.OR, + "not": LogicalOperatorType.NOT, + "NOT": LogicalOperatorType.NOT, + } + _logical_operator_pattern = f"(?P{'|'.join(logical_operators_map)})" + logical_operator_pattern = rf"^{_logical_operator_pattern}\s+" + field_value_pattern = r"""^___field___\s*___operator___\s*___value___""" + base_value_pattern = r"(?:___value_pattern___)" + + # do not modify, use subclasses to define this attribute + field_pattern: str = None + function_pattern: str = None + _value_pattern: str = None + value_pattern: str = None + multi_value_pattern: str = None + keyword_pattern: str = None + multi_value_delimiter_pattern = r"," + + wildcard_symbol = None + escape_manager: EscapeManager = None + str_value_manager: StrValueManager = None + platform_functions: PlatformFunctions = None + + def __init_subclass__(cls, **kwargs): + cls._validate_re_patterns() + cls.value_pattern = cls.base_value_pattern.replace("___value_pattern___", cls._value_pattern) + cls.operators_map = { + **cls.single_value_operators_map, + **cls.multi_value_operators_map, + **cls.fields_operator_map, + } + cls.operator_pattern = rf"""(?:___field___\s*(?P(?:{'|'.join(cls.operators_map)})))\s*""" + + @classmethod + def _validate_re_patterns(cls) -> None: + if not all([cls.field_pattern, cls._value_pattern]): + raise ValueError(f"{cls.__name__} re patterns must be set") + + def map_operator(self, operator: str) -> str: + try: + return self.operators_map[operator.lower()] + except KeyError as e: + raise UnsupportedOperatorException(operator) from e + + def search_field(self, query: str) -> str: + field_search = re.search(self.field_pattern, query) + if field_search is None: + raise TokenizerGeneralException(error=f"Field couldn't be found in query part: {query}") + return field_search.group("field_name") + + def escape_field_name(self, field_name: str) -> str: + return field_name.replace(".", r"\.") + + def search_operator(self, query: str, field_name: str) -> str: + field_name = self.escape_field_name(field_name) + operator_pattern = self.operator_pattern.replace("___field___", field_name) + compiled_operator_regex = re.compile(operator_pattern, re.IGNORECASE) + if (operator_search := re.search(compiled_operator_regex, query)) is None: + raise TokenizerGeneralException(error=f"Operator couldn't be found in query part: {query}") + + operator = operator_search.group("operator") + return operator.strip(" ") + + def get_operator_and_value( + self, + match: re.Match, + mapped_operator: str = OperatorType.EQ, + operator: Optional[str] = None, # noqa: ARG002 + ) -> tuple[str, Any]: + return mapped_operator, get_match_group(match, group_name=ValueType.value) + + @staticmethod + def clean_multi_value(value: str) -> str: + value = value.strip(" ") + if value.startswith("'") and value.endswith("'") or value.startswith('"') and value.endswith('"'): + value = value[1:-1] + + return value + + def search_single_value(self, query: str, operator: str, field_name: str) -> tuple[str, str, Union[str, StrValue]]: + field_value_match = self._get_field_value_match(query, operator, field_name, self.value_pattern) + mapped_operator, value = self.get_operator_and_value(field_value_match, self.map_operator(operator), operator) + if self.should_process_value_wildcards(operator): + mapped_operator, value = self.process_value_wildcards(value, mapped_operator) + + pos = field_value_match.end() + return query[pos:], mapped_operator, value + + def search_multi_value( + self, query: str, operator: str, field_name: str + ) -> tuple[str, dict[str, list[Union[str, StrValue]]]]: + field_value_match = self._get_field_value_match(query, operator, field_name, self.multi_value_pattern) + if (multi_value := get_match_group(field_value_match, group_name=ValueType.multi_value)) is None: + raise TokenizerGeneralException(error=f"Value couldn't be found in query part: {query}") + + values = [self.clean_multi_value(v) for v in re.split(self.multi_value_delimiter_pattern, multi_value)] + grouped_values = self.group_values_by_operator(values, operator) + pos = field_value_match.end() + return query[pos:], grouped_values + + def _get_field_value_match(self, query: str, operator: str, field_name: str, value_pattern: str) -> re.Match: + field_value_pattern = self.get_field_value_pattern(operator, field_name, value_pattern) + field_value_match = re.match(field_value_pattern, query, re.IGNORECASE) + if field_value_match is None: + raise TokenizerGeneralException(error=f"Value couldn't be found in query part: {query}") + + return field_value_match + + def group_values_by_operator(self, values: list[str], operator: str) -> dict[str, list[str]]: + mapped_operator = self.map_operator(operator) + result = {} + for value in values: + op = mapped_operator + if self.should_process_value_wildcards(operator): + op, value = self.process_value_wildcards(value, mapped_operator) + result.setdefault(op, []).append(value) + + return result + + def search_keyword(self, query: str) -> tuple[Keyword, str]: + keyword_search = re.search(self.keyword_pattern, query) + _, value = self.get_operator_and_value(keyword_search) + keyword = Keyword(value=value) + pos = keyword_search.end() + return keyword, query[pos:] + + def get_field_value_pattern(self, operator: str, field_name: str, value_pattern: str) -> str: + field_value_pattern = self.field_value_pattern.replace("___field___", self.escape_field_name(field_name)) + field_value_pattern = field_value_pattern.replace("___operator___", operator) + return field_value_pattern.replace("___value___", value_pattern) + + @staticmethod + def should_process_value_wildcards(operator: Optional[str]) -> bool: # noqa: ARG004 + return True + + def process_value_wildcards( + self, value: Union[str, StrValue], operator: str = OperatorType.EQ + ) -> tuple[str, Union[str, StrValue]]: + if not (wildcard := self.wildcard_symbol) or operator == OperatorType.REGEX or len(value) == 1: + return operator, value + + if value.startswith(wildcard) and value.endswith(wildcard): + if isinstance(value, StrValue): + value = StrValue(value.strip(wildcard), value.split_value[1:-1]) + else: + value = value.strip(wildcard) + return OperatorType.CONTAINS, value + + if value.startswith(wildcard): + if isinstance(value, StrValue): + value = StrValue(value.lstrip(wildcard), value.split_value[1:]) + else: + value = value.lstrip(wildcard) + return OperatorType.ENDSWITH, value + + if value.endswith(wildcard): + if isinstance(value, StrValue): + value = StrValue(value.rstrip(wildcard), value.split_value[:-1]) + else: + value = value.rstrip(wildcard) + return OperatorType.STARTSWITH, value + + return operator, value + + @staticmethod + def create_field_value(field_name: str, operator: Identifier, value: Union[str, list]) -> FieldValue: + return FieldValue(source_name=field_name, operator=operator, value=value) + + @staticmethod + def concat_field_value_tokens(tokens: list[FieldValue]) -> list[Union[FieldValue, Identifier]]: + result = [tokens[0]] + for token in tokens[1:]: + result.append(Identifier(token_type=LogicalOperatorType.OR)) + result.append(token) + + return result + + def is_multi_value_flow(self, field_name: str, operator: str, query: str) -> bool: # noqa: ARG002 + return operator.lower() in self.multi_value_operators_map + + def search_field_value(self, query: str) -> tuple[Union[FieldValue, list[Union[FieldValue, Identifier]]], str]: + field_name = self.search_field(query) + operator = self.search_operator(query, field_name) + if self.is_multi_value_flow(field_name, operator, query): + query, grouped_values = self.search_multi_value(query=query, operator=operator, field_name=field_name) + tokens = [ + self.create_field_value(field_name=field_name, operator=Identifier(token_type=op), value=values) + for op, values in grouped_values.items() + ] + if len(tokens) > 1: + l_paren = Identifier(token_type=GroupType.L_PAREN) + r_paren = Identifier(token_type=GroupType.R_PAREN) + tokens = [l_paren, *self.concat_field_value_tokens(tokens), r_paren] + + return tokens, query + + query, operator, value = self.search_single_value(query=query, operator=operator, field_name=field_name) + operator_token = Identifier(token_type=operator) + field_value = self.create_field_value(field_name=field_name, operator=operator_token, value=value) + return field_value, query + + def _check_field_value_match(self, query: str, white_space_pattern: str = r"\s+") -> bool: + single_value_operator_group = rf"(?:{'|'.join(self.single_value_operators_map)})" + single_value_pattern = rf"""{self.field_pattern}\s*{single_value_operator_group}\s*{self.value_pattern}\s*""" + if re.match(single_value_pattern, query, re.IGNORECASE): + return True + + if self.multi_value_operators_map: + multi_value_operator_group = rf"(?:{'|'.join(self.multi_value_operators_map)})" + pattern = f"{self.field_pattern}{white_space_pattern}{multi_value_operator_group}{white_space_pattern}" + multi_value_pattern = rf"{pattern}{self.multi_value_pattern}" + if re.match(multi_value_pattern, query, re.IGNORECASE): + return True + + return False + + def search_function_value(self, query: str) -> tuple[FunctionValue, str]: + ... + + @staticmethod + def _check_function_value_match(query: str) -> bool: # noqa: ARG004 + return False + + def _get_next_token( + self, query: str + ) -> tuple[Union[FieldValue, FunctionValue, Keyword, Identifier, list[Union[FieldValue, Identifier]]], str]: + query = query.strip("\n").strip(" ").strip("\n") + if query.startswith(GroupType.L_PAREN): + return Identifier(token_type=GroupType.L_PAREN), query[1:] + if query.startswith(GroupType.R_PAREN): + return Identifier(token_type=GroupType.R_PAREN), query[1:] + if logical_operator_search := re.match(self.logical_operator_pattern, query): + logical_operator = logical_operator_search.group("logical_operator") + pos = logical_operator_search.end() + return Identifier(token_type=logical_operator.lower()), query[pos:] + if self.platform_functions and self._check_function_value_match(query): # noqa: SIM102 + if search_result := self.search_function_value(query): + return search_result + if self._check_field_value_match(query): + return self.search_field_value(query) + if self.keyword_pattern and re.match(self.keyword_pattern, query): + return self.search_keyword(query) + + unsupported_query_entry = self._get_unsupported_query_entry(query) + raise TokenizerGeneralException(f"Unsupported query entry: {unsupported_query_entry}") + + def _get_unsupported_query_entry(self, query: str) -> str: + split_by_logical_operator = re.split(rf"\s+{self._logical_operator_pattern}\s+", query, maxsplit=1) + return split_by_logical_operator[0] + + @staticmethod + def _validate_parentheses(tokens: list[QUERY_TOKEN_TYPE]) -> None: + parentheses = [] + for token in tokens: + if isinstance(token, Identifier) and token.token_type in (GroupType.L_PAREN, GroupType.R_PAREN): + if token.token_type == GroupType.L_PAREN: + parentheses.append(token) + elif not parentheses or parentheses[-1].token_type == GroupType.R_PAREN: + raise QueryParenthesesException + else: + parentheses.pop() + if parentheses: + raise QueryParenthesesException + + def tokenize(self, query: str) -> list[QUERY_TOKEN_TYPE]: + tokenized = [] + while query: + next_token, sliced_query = self._get_next_token(query=query) + if isinstance(next_token, list): + tokenized.extend(next_token) + else: + tokenized.append(next_token) + + if len(sliced_query) >= len(query): + raise TokenizerGeneralException("Unsupported query entry. Infinite loop") + + query = sliced_query + + self._validate_parentheses(tokenized) + return tokenized + + @staticmethod + def filter_tokens( + tokens: list[QUERY_TOKEN_TYPE], + token_type: Union[type[FieldValue], type[Field], type[FieldField], type[Keyword], type[Identifier]], + ) -> list[QUERY_TOKEN_TYPE]: + return [token for token in tokens if isinstance(token, token_type)] + + @staticmethod + def set_field_tokens_generic_names_map( + tokens: list[Field], source_mappings: list[SourceMapping], default_mapping: SourceMapping + ) -> None: + for token in tokens: + token.set_generic_names_map(source_mappings, default_mapping) diff --git a/uncoder-core/app/translator/cti_translator.py b/uncoder-core/app/translator/cti_translator.py new file mode 100644 index 00000000..740839cc --- /dev/null +++ b/uncoder-core/app/translator/cti_translator.py @@ -0,0 +1,91 @@ +import logging +from typing import Optional + +from app.models.ioc_translation import CTIPlatform +from app.translator.const import CTI_IOCS_PER_QUERY_LIMIT, CTI_MIN_LIMIT_QUERY +from app.translator.core.models.iocs import IocsChunkValue +from app.translator.core.parser_cti import CTIParser +from app.translator.managers import RenderCTIManager, render_cti_manager +from app.translator.tools.decorators import handle_translation_exceptions + + +class CTITranslator: + render_manager: RenderCTIManager = render_cti_manager + + def __init__(self): + self.logger = logging.getLogger("cti_translator") + self.parser = CTIParser() + + @handle_translation_exceptions + def __parse_iocs_from_string( + self, + text: str, + include_ioc_types: Optional[list] = None, + include_hash_types: Optional[list] = None, + exceptions: Optional[list] = None, + ioc_parsing_rules: Optional[list] = None, + include_source_ip: bool = False, + ) -> dict: + return self.parser.get_iocs_from_string( + string=text, + include_ioc_types=include_ioc_types, + include_hash_types=include_hash_types, + exceptions=exceptions, + ioc_parsing_rules=ioc_parsing_rules, + limit=CTI_MIN_LIMIT_QUERY, + include_source_ip=include_source_ip, + ) + + @handle_translation_exceptions + def __render_translation(self, parsed_data: dict, platform_data: CTIPlatform, iocs_per_query: int) -> list[str]: + render_cti = self.render_manager.get(platform_data.id) + + chunked_iocs = self.__get_iocs_chunk( + chunks_size=iocs_per_query, data=parsed_data, mapping=render_cti.default_mapping + ) + return render_cti.render(chunked_iocs) + + def translate( + self, + text: str, + platform_data: CTIPlatform, + iocs_per_query: int = CTI_IOCS_PER_QUERY_LIMIT, + include_ioc_types: Optional[list] = None, + include_hash_types: Optional[list] = None, + exceptions: Optional[list] = None, + ioc_parsing_rules: Optional[list] = None, + include_source_ip: bool = False, + ) -> (bool, list[str]): + status, parsed_data = self.__parse_iocs_from_string( + text=text, + include_ioc_types=include_ioc_types, + include_hash_types=include_hash_types, + exceptions=exceptions, + ioc_parsing_rules=ioc_parsing_rules, + include_source_ip=include_source_ip, + ) + if status: + return self.__render_translation( + parsed_data=parsed_data, platform_data=platform_data, iocs_per_query=iocs_per_query + ) + return status, parsed_data + + @staticmethod + def __get_iocs_chunk( + chunks_size: int, data: dict[str, list[str]], mapping: dict[str, str] + ) -> list[list[IocsChunkValue]]: + result = [] + for generic_field, iocs_list in data.items(): + for ioc in iocs_list: + if mapping.get(generic_field): + result.append( + IocsChunkValue(generic_field=generic_field, platform_field=mapping[generic_field], value=ioc) + ) + return [result[i : i + chunks_size] for i in range(0, len(result), chunks_size)] + + @classmethod + def get_renders(cls) -> list: + return cls.render_manager.get_platforms_details + + +cti_translator = CTITranslator() diff --git a/uncoder-core/app/translator/managers.py b/uncoder-core/app/translator/managers.py new file mode 100644 index 00000000..38fb1e63 --- /dev/null +++ b/uncoder-core/app/translator/managers.py @@ -0,0 +1,97 @@ +from abc import ABC +from functools import cached_property +from typing import ClassVar, Union + +from app.models.translation import TranslatorPlatform +from app.translator.core.exceptions.core import UnsupportedPlatform, UnsupportedRootAParser +from app.translator.core.parser import QueryParser +from app.translator.core.render import QueryRender +from app.translator.core.render_cti import RenderCTI + + +class PlatformManager(ABC): + platforms: ClassVar[dict[str, Union[QueryParser, QueryRender, RenderCTI]]] = {} + + def all_platforms(self) -> list: + return list(self.platforms.keys()) + + @cached_property + def get_platforms_details(self) -> list[TranslatorPlatform]: + platforms = [ + TranslatorPlatform( + id=platform.details.platform_id, + name=platform.details.name, + code=platform.details.platform_id, + group_name=platform.details.group_name, + group_id=platform.details.group_id, + platform_name=platform.details.platform_name, + platform_id=platform.details.platform_id, + alt_platform_name=platform.details.alt_platform_name, + alt_platform=platform.details.alt_platform, + first_choice=platform.details.first_choice, + ) + for platform in self.platforms.values() + ] + return sorted(platforms, key=lambda platform: platform.group_name) + + +class ParserManager(PlatformManager): + supported_by_roota_platforms: ClassVar[dict[str, QueryParser]] = {} + main_platforms: ClassVar[dict[str, QueryParser]] = {} + + def get(self, platform_id: str) -> QueryParser: + if platform := self.platforms.get(platform_id): + return platform + raise UnsupportedPlatform(platform=platform_id, is_parser=True) + + def register(self, cls: type[QueryParser]) -> type[QueryParser]: + self.platforms[cls.details.platform_id] = cls() + return cls + + def get_supported_by_roota(self, platform_id: str) -> QueryParser: + if platform := self.supported_by_roota_platforms.get(platform_id): + return platform + raise UnsupportedRootAParser(parser=platform_id) + + def register_supported_by_roota(self, cls: type[QueryParser]) -> type[QueryParser]: + parser = cls() + self.supported_by_roota_platforms[cls.details.platform_id] = parser + self.platforms[cls.details.platform_id] = parser + return cls + + def register_main(self, cls: type[QueryParser]) -> type[QueryParser]: + parser = cls() + self.main_platforms[cls.details.platform_id] = parser + self.platforms[cls.details.platform_id] = parser + return cls + + +class RenderManager(PlatformManager): + platforms: ClassVar[dict[str, QueryRender]] = {} + + def get(self, platform_id: str) -> QueryRender: + if platform := self.platforms.get(platform_id): + return platform + raise UnsupportedPlatform(platform=platform_id) + + def register(self, cls: type[QueryRender]) -> type[QueryRender]: + self.platforms[cls.details.platform_id] = cls() + return cls + + +class RenderCTIManager(PlatformManager): + platforms: ClassVar[dict[str, RenderCTI]] = {} + + def get(self, platform_id: str) -> RenderCTI: + if platform := self.platforms.get(platform_id): + return platform + raise UnsupportedPlatform(platform=platform_id) + + def register(self, cls: type[RenderCTI]) -> type[RenderCTI]: + self.platforms[cls.details.platform_id] = cls() + return cls + + +parser_manager = ParserManager() +render_manager = RenderManager() +render_cti_manager = RenderCTIManager() diff --git a/siem-converter/app/converter/platforms/athena/mappings/__init__.py b/uncoder-core/app/translator/mappings/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/athena/mappings/__init__.py rename to uncoder-core/app/translator/mappings/__init__.py diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/default.yml b/uncoder-core/app/translator/mappings/platforms/anomali/default.yml new file mode 100644 index 00000000..fed2954e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/default.yml @@ -0,0 +1,5 @@ +platform: Anomali +source: default + + +default_log_source: {} diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/proxy.yml b/uncoder-core/app/translator/mappings/platforms/anomali/proxy.yml new file mode 100644 index 00000000..5cecdbe9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/proxy.yml @@ -0,0 +1,41 @@ +platform: Anomali +source: proxy + +field_mapping: + c-uri-query: url + c-useragent: user_agent + c-uri: url + cs-method: http_method + cs-bytes: bytes_out + cs-referrer: http_referrer + sc-status: return_code + + dns-query: query + dns-answer: answer + dns-record: record_type + + CommandLine: command_line + DestinationHostname: dest + DestinationIp: dest_ip + DestinationPort: dest_port + Details: reg_value_data + dst_ip: dest_ip + dst_port: dest_port + EventID: event_id + EventName: event_name + FileName: file_name + FilePath: file_path + Image: image + NewProcessName: image + OriginalFileName: original_file_name + ParentCommandLine: parent_command_line + ParentImage: parent_image + ParentProcessID: parent_process_id + Platform: platform + ProcessCommandLine: command_line + ProcessID: process_id + SourceImage: parent_image + SourcePort: src_port + TargetFilename: file_name + TargetObject: reg_key + UserAgent: user_agent diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/webserver.yml b/uncoder-core/app/translator/mappings/platforms/anomali/webserver.yml new file mode 100644 index 00000000..a3065010 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/webserver.yml @@ -0,0 +1,41 @@ +platform: Anomali +source: webserver + +field_mapping: + c-uri-query: url + c-useragent: user_agent + c-uri: url + cs-method: http_method + cs-bytes: bytes_out + cs-referrer: http_referrer + sc-status: return_code + + dns-query: query + dns-answer: answer + dns-record: record_type + + CommandLine: command_line + DestinationHostname: dest + DestinationIp: dest_ip + DestinationPort: dest_port + Details: reg_value_data + dst_ip: dest_ip + dst_port: dest_port + EventID: event_id + EventName: event_name + FileName: file_name + FilePath: file_path + Image: image + NewProcessName: image + OriginalFileName: original_file_name + ParentCommandLine: parent_command_line + ParentImage: parent_image + ParentProcessID: parent_process_id + Platform: platform + ProcessCommandLine: command_line + ProcessID: process_id + SourceImage: parent_image + SourcePort: src_port + TargetFilename: file_name + TargetObject: reg_key + UserAgent: user_agent diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_image_load.yml new file mode 100644 index 00000000..d3aa7544 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_image_load.yml @@ -0,0 +1,18 @@ +platform: Anomali +source: windows_image_load + + +log_source: + product: [windows] + category: [image_load] + +default_log_source: + product: windows + category: image_load + +field_mapping: + Image: image + #ImageLoaded: ImageLoaded + #SignatureStatus: SignatureStatus + OriginalFileName: original_file_name + #Signed: Signed \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_network_connection.yml new file mode 100644 index 00000000..c18cc5c3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_network_connection.yml @@ -0,0 +1,20 @@ +platform: Anomali +source: windows_network_connection + + +log_source: + product: [windows] + category: [network_connection] + +default_log_source: + product: windows + category: network_connection + +field_mapping: + Image: image + DestinationHostname: dest + DestinationIp: dest_ip + DestinationPort: dest_port + SourceIp: src_ip + SourcePort: src_port + #Initiated: Initiated \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_pipe_created.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_pipe_created.yml new file mode 100644 index 00000000..9144d683 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_pipe_created.yml @@ -0,0 +1,16 @@ +platform: Anomali +source: windows_pipe_created + + +log_source: + product: [windows] + category: [pipe_created] + +default_log_source: + product: windows + category: pipe_created + +field_mapping: + EventID: event_id + #PipeName: PipeName + Image: image \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_process_access.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_process_access.yml new file mode 100644 index 00000000..5f105eb0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_process_access.yml @@ -0,0 +1,24 @@ + +platform: Anomali +source: windows_process_access + + +log_source: + product: [windows] + category: [process_access] + +default_log_source: + product: windows + category: process_access + +field_mapping: + #SourceProcessGUID: SourceProcessGUID + #SourceProcessId: SourceProcessId + #SourceThreadId: SourceThreadId + #ourceImage: SourceImage + #TargetProcessGUID: TargetProcessGUID + #TargerProcessId: TargerProcessId + #TargetImage: TargetImage + #GrantedAccess: GrantedAccess + #CallTrace: CallTrace + User: user \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_process_creation.yml new file mode 100644 index 00000000..8af5bdbe --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_process_creation.yml @@ -0,0 +1,23 @@ +platform: Anomali +source: windows_process_creation + + +log_source: + product: [windows] + category: [process_creation] + +default_log_source: + product: windows + category: process_creation + +field_mapping: + CommandLine: command_line + #CurrentDirectory: CurrentDirectory + Hashes: file_hash + Image: image + #IntegrityLevel: IntegrityLevel + ParentCommandLine: parent_command_line + ParentImage: parent_image + #ParentUser: ParentUser + #Product: Product + User: user \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_registry_event.yml new file mode 100644 index 00000000..aa91e179 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_registry_event.yml @@ -0,0 +1,31 @@ +platform: Anomali +source: windows_registry_event + +log_source: + product: [windows] + category: [registry_event, registry_set, registry_delete, registry_add] + +default_log_source: + product: windows + category: registry_event + +field_mapping: + TargetObject: reg_key + Image: image + Details: reg_value_data + EventType: event_name + CommandLine: command_line + #LogonId: LogonId + #Product: Product + #Company: Company + #IntegrityLevel: IntegrityLevel + #CurrentDirectory: CurrentDirectory + ProcessId: process_id + ParentProcessId: parent_process_id + ParentCommandLine: parent_command_line + ParentImage: parent_image + #ParentUser: ParentUser + #ParentIntegrityLevel: ParentIntegrityLevel + #ParentLogonId: ParentLogonId + #ParentProduct: ParentProduct + #ParentCompany: ParentCompany \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_security.yml new file mode 100644 index 00000000..6809de3c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_security.yml @@ -0,0 +1,147 @@ +platform: Anomali +source: windows_security + + +log_source: + product: [windows] + service: [security] + +default_log_source: + product: windows + service: security + +field_mapping: + EventID: event_id + ParentImage: parent_image + #AccessMask: AccessMask + AccountName: user + #AllowedToDelegateTo: AllowedToDelegateTo + #AttributeLDAPDisplayName: AttributeLDAPDisplayName + #AuditPolicyChanges: AuditPolicyChanges + #AuthenticationPackageName: AuthenticationPackageName + #CallingProcessName: CallingProcessName + #Channel: Channel + #ComputerName: ComputerName + #EventType: EventType + #FailureReason: FailureReason + #FileName: FileName + #GrantedAccess: GrantedAccess + #Hashes: Hashes + #HiveName: HiveName + #IpAddress: IpAddress + #IpPort: IpPort + #KeyLength: KeyLength + #LogonProcessName: LogonProcessName + #LogonType: LogonType + #LinkName: LinkName + #MemberName: MemberName + #MemberSid: MemberSid + #NewProcessName: NewProcessName + #ObjectClass: ObjectClass + #ObjectType: ObjectType + #ObjectValueName: ObjectValueName + #Path: Path + #CommandLine: CommandLine + #OldUacValue: OldUacValue + #CertIssuerName: CertIssuerName + #SubStatus: SubStatus + #DisplayName: DisplayName + #TaskContent: TaskContent + #ServiceSid: ServiceSid + #CertThumbprint: CertThumbprint + #ObjectName: ObjectName + #ClassName: ClassName + #NotificationPackageName: NotificationPackageName + #NewSd: NewSd + #TestSigning: TestSigning + #TargetInfo: TargetInfo + #ParentProcessId: ParentProcessId + #AccessList: AccessList + #GroupMembership: GroupMembership + #FilterName: FilterName + #ChangeType: ChangeType + #LayerName: LayerName + #ServiceAccount: ServiceAccount + #ClientProcessId: ClientProcessId + #AttributeValue: AttributeValue + #SessionName: SessionName + #TaskName: TaskName + #ObjectDN: ObjectDN + #TemplateContent: TemplateContent + #NewTemplateContent: NewTemplateContent + #SourcePort: SourcePort + #PasswordLastSet: PasswordLastSet + #PrivilegeList: PrivilegeList + #DeviceDescription: DeviceDescription + #TargetServerName: TargetServerName + #NewTargetUserName: NewTargetUserName + #OperationType: OperationType + #DestPort: DestPort + #ServiceStartType: ServiceStartType + #OldTargetUserName: OldTargetUserName + #UserPrincipalName: UserPrincipalName + #Accesses: Accesses + #DnsHostName: DnsHostName + #DisableIntegrityChecks: DisableIntegrityChecks + #AuditSourceName: AuditSourceName + #Workstation: Workstation + #DestAddress: DestAddress + #PreAuthType: PreAuthType + #SecurityPackageName: SecurityPackageName + #SubjectLogonId: SubjectLogonId + #NewUacValue: NewUacValue + #EnabledPrivilegeList: EnabledPrivilegeList + #RelativeTargetName: RelativeTargetName + #CertSerialNumber: CertSerialNumber + #SidHistory: SidHistory + #TargetLogonId: TargetLogonId + #KernelDebug: KernelDebug + #CallerProcessName: CallerProcessName + #Properties: Properties + #UserAccountControl: UserAccountControl + #RegistryValue: RegistryValue + #SecurityID: SecurityID + #ServiceFileName: ServiceFileName + #SecurityDescriptor: SecurityDescriptor + #ServiceName: ServiceName + #ShareName: ShareName + #NewValue: NewValue + #Source: Source + #Status: Status + #SubjectDomainName: SubjectDomainName + #SubjectUserName: SubjectUserName + #SubjectUserSid: SubjectUserSid + #SourceAddr: SourceAddr + #SourceAddress: SourceAddress + #TargetName: TargetName + #ServicePrincipalNames: ServicePrincipalNames + #TargetDomainName: TargetDomainName + #TargetSid: TargetSid + #TargetUserName: TargetUserName + #ObjectServer: ObjectServer + #TargetUserSid: TargetUserSid + #TicketEncryptionType: TicketEncryptionType + #TicketOptions: TicketOptions + #WorkstationName: WorkstationName + #TransmittedServices: TransmittedServices + #AuthenticationAlgorithm: AuthenticationAlgorithm + #LayerRTID: LayerRTID + #BSSID: BSSID + #BSSType: BSSType + #CipherAlgorithm: CipherAlgorithm + #ConnectionId: ConnectionId + #ConnectionMode: ConnectionMode + #InterfaceDescription: InterfaceDescription + #InterfaceGuid: InterfaceGuid + #OnexEnabled: OnexEnabled + #PHYType: PHYType + #ProfileName: ProfileName + #SSID: SSID + #Domain: Domain + #ServiceType: ServiceType + #SourceName: SourceName + #StartType: StartType + #UserID: UserID + #ParentProcessName: ParentProcessName + #Service: Service + #ProcessName: ProcessName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_sysmon.yml new file mode 100644 index 00000000..284c2685 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_sysmon.yml @@ -0,0 +1,63 @@ +platform: Anomali +source: windows_sysmon + + +log_source: + product: [windows] + service: [sysmon] + +default_log_source: + product: windows + service: sysmon + +field_mapping: + CommandLine: command_line + Image: image + ParentImage: parent_image + EventID: event_id + #CallTrace: CallTrace + #Company: Company + #CurrentDirectory: CurrentDirectory + #Description: Description + DestinationHostname: dest + DestinationIp: dest_ip + #DestinationIsIpv6: DestinationIsIpv6 + DestinationPort: dest_port + #DestinationPortName: DestinationPortName + Hashes: file_hash + #Initiated: Initiated + #IntegrityLevel: IntegrityLevel + ParentCommandLine: parent_command_line + #Product: Product + #Protocol: Protocol + #RuleName: RuleName + SourceHostname: src + SourceIp: src_ip + #SourceIsIpv6: SourceIsIpv6 + SourcePort: src_port + #SourcePortName: SourcePortName + TargetFilename: file_name + User: user + OriginalFileName: original_file_name + #Signed: Signed + #Signature: Signature + #SignatureStatus: SignatureStatus + TargetObject: reg_key + Details: reg_value_data + QueryName: query + QueryResults: record_type + #QueryStatus: QueryStatus + #IsExecutable: IsExecutable + #PipeName: PipeName + #ImageLoaded: ImageLoaded + #ImagePath: ImagePath + #Imphash: Imphash + #SourceImage: SourceImage + #StartModule: StartModule + #TargetImage: TargetImage + Device: dvc_name + ProcessID: process_id + #FileVersion: FileVersion + #StartAddress: StartAddress + #StartFunction: StartFunction + EventType: event_name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_system.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_system.yml new file mode 100644 index 00000000..d64ced48 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_system.yml @@ -0,0 +1,27 @@ +platform: Anomali +source: windows_system + + +log_source: + product: [windows] + service: [system] + +default_log_source: + product: windows + service: system + +field_mapping: + EventID: event_id + #AccountName: AccountName + #ImagePath: ImagePath + #ServiceName: ServiceName + #ServiceType: ServiceType + #StartType: StartType + #Provider_Name: Provider_Name + #Origin: Origin + #HiveName: HiveName + #Caption: Caption + #param1: param1 + #param2: param2 + #Channel: Channel + #DeviceName: DeviceName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/anomali/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/anomali/windows_wmi_event.yml new file mode 100644 index 00000000..58cbcb9e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/anomali/windows_wmi_event.yml @@ -0,0 +1,15 @@ +platform: Anomali +source: windows_wmi_event + + +log_source: + product: [windows] + category: [wmi_event] + +default_log_source: + product: windows + category: wmi_event + +field_mapping: +# Destination: Destination + EventID: event_id diff --git a/uncoder-core/app/translator/mappings/platforms/arcsight/default.yml b/uncoder-core/app/translator/mappings/platforms/arcsight/default.yml new file mode 100644 index 00000000..ef7bc834 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/arcsight/default.yml @@ -0,0 +1,5 @@ +platform: ArcSight +source: default + + +default_log_source: {} diff --git a/uncoder-core/app/translator/mappings/platforms/arcsight/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/arcsight/linux_network_connection.yml new file mode 100644 index 00000000..d720251d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/arcsight/linux_network_connection.yml @@ -0,0 +1,9 @@ +platform: ArcSight +source: linux_network_connection + + +default_log_source: {} + + +field_mapping: + SourceHostname: sourceHostName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/arcsight/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/arcsight/macos_network_connection.yml new file mode 100644 index 00000000..85370b92 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/arcsight/macos_network_connection.yml @@ -0,0 +1,9 @@ +platform: ArcSight +source: macos_network_connection + + +default_log_source: {} + + +field_mapping: + SourceHostname: sourceHostName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/arcsight/windows_create_remote_thread.yml b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_create_remote_thread.yml new file mode 100644 index 00000000..4b9f3e91 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_create_remote_thread.yml @@ -0,0 +1,13 @@ +platform: ArcSight +source: windows_create_remote_thread + + +default_log_source: {} + + +field_mapping: + SourceImage: sourceProcessName + TargetImage: destinationProcessName + StartModule: deviceCustomString3 + StartAddress: deviceCustomString3 + StartFunction: deviceCustomString3 \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/arcsight/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_network_connection.yml new file mode 100644 index 00000000..32d52d69 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_network_connection.yml @@ -0,0 +1,9 @@ +platform: ArcSight +source: windows_network_connection + + +default_log_source: {} + + +field_mapping: + SourceHostname: sourceHostName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/arcsight/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_process_creation.yml new file mode 100644 index 00000000..356466c8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_process_creation.yml @@ -0,0 +1,9 @@ +platform: ArcSight +source: windows_process_creation + + +default_log_source: {} + + +field_mapping: + OriginalFileName: oldFileName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/arcsight/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_security.yml new file mode 100644 index 00000000..57803a9b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_security.yml @@ -0,0 +1,54 @@ +platform: ArcSight +source: windows_security + + +default_log_source: {} + +conditions: + deviceVendor: Microsoft + deviceProduct: Microsoft Windows + + +field_mapping: + EventID: externalId + AccessMask: deviceCustomString1 + AccountName: destinationUserName + AuditPolicyChanges: deviceAction + AuthenticationPackageName: deviceCustomString5 + EventType: deviceSeverity + FailureReason: deviceCustomString4 + IpAddress: sourceAddress + IpPort: sourcePort + LogonProcessName: + - destinationProcessName + - sourceProcessName + LogonType: deviceCustomNumber1 + MemberName: destinationUserId + MemberSid: destinationUserName + NewProcessName: destinationProcessName + ObjectClass: deviceCustomString5 + ObjectName: fileName + ObjectType: fileType + ObjectValueName: deviceCustomString6 + CommandLine: deviceCustomString4 + ProcessName: destinationProcessName + Properties: deviceCustomString6 + ServiceFileName: filePath + ServiceName: destinationServiceName + ShareName: + - filePath + - deviceCustomString6 + Status: eventOutcome + SubjectDomainName: destinationNTDomain + SubjectUserName: destinationUserName + SubjectUserSid: destinationUserName + TargetDomainName: destinationNTDomain + TargetSid: destinationNTDomain + TargetUserName: destinationUserName + TargetUserSid: destinationUserName + TicketEncryptionType: deviceCustomString5 + TicketOptions: deviceCustomString1 + WorkstationName: sourceHostName + ServiceType: fileType + StartType: deviceCustomString5 + ParentProcessName: filePath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/arcsight/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_sysmon.yml new file mode 100644 index 00000000..e92d02a8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/arcsight/windows_sysmon.yml @@ -0,0 +1,50 @@ +platform: ArcSight +source: windows_sysmon + + +default_log_source: {} + +conditions: + deviceVendor: Microsoft + deviceProduct: Sysmon + +field_mapping: + CommandLine: deviceCustomString1 + Image: destinationProcessName + ParentImage: sourceProcessName + EventID: externalId + CallTrace: deviceCustomString3 + Company: oldFileType + CurrentDirectory: deviceCustomString3 + Description: oldFilePermission + DestinationHostname: destinationHostName + DestinationIp: destinationAddress + DestinationPort: destinationPort + Initiated: deviceCustomString4 + IntegrityLevel: deviceCustomString5 + ParentCommandLine: deviceCustomString2 + Product: destinationServiceName + Protocol: transportProtocol + RuleName: deviceFacility + SourceHostname: sourceHostName + SourceIp: sourceAddress + SourcePort: sourcePort + TargetFilename: fileName + User: sourceUserName + OriginalFileName: oldFileName + Signed: deviceCustomString1 + Signature: deviceCustomString2 + SignatureStatus: deviceCustomString3 + TargetObject: fileName + Details: deviceCustomString1 + QueryName: + - requestUrl + - destinationHostName + QueryResults: deviceCustomString1 + QueryStatus: deviceCustomNumber1 + PipeName: fileName + ImageLoaded: destinationProcessName + SourceImage: sourceProcessName + StartModule: deviceCustomString3 + TargetImage: destinationProcessName + EventType: deviceAction \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/athena/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/athena/aws_cloudtrail.yml new file mode 100644 index 00000000..81c7451d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/aws_cloudtrail.yml @@ -0,0 +1,23 @@ +platform: Athena +source: aws_cloudtrail + + +log_source: + table: + - cloudtrail_logs_* + +default_log_source: + table: cloudtrail_logs_* + +field_mapping: + eventSource: api.service.name + eventName: api.operation + AdditionalEventData: additionalEventData + additionalEventData.MFAUsed: JSON_VALUE(unmapped['additionalEventData'], 'strict$.MFAUsed') + requestParameters.attribute: unmapped['eventType'] + responseElements.pendingModifiedValues.masterUserPassword: unmapped['responseElements'] + responseElements.publiclyAccessible: JSON_VALUE(unmapped['responseElements'], 'strict$.ConsoleLogin') + userIdentity.principalId: http_request.user_agent + userIdentity.sessionContext.sessionIssuer.type: unmapped['userIdentity_sessionContext_sessionIssuer_arn'] + userIdentity.type: unmapped['userIdentity_sessionContext_sessionIssuer_principalId'] + userIdentity.userName: unmapped['userIdentity_sessionContext_sessionIssuer_type'] diff --git a/uncoder-core/app/translator/mappings/platforms/athena/default.yml b/uncoder-core/app/translator/mappings/platforms/athena/default.yml new file mode 100644 index 00000000..48514132 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/default.yml @@ -0,0 +1,7 @@ +platform: Athena +source: default + + + +default_log_source: + table: eventlog \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/athena/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/athena/linux_file_event.yml new file mode 100644 index 00000000..99b38772 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/linux_file_event.yml @@ -0,0 +1,10 @@ +platform: Athena +source: linux_file_event + + +default_log_source: + table: eventlog + +field_mapping: + Image: actor.process.file.path + TargetFilename : file.name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/athena/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/athena/linux_process_creation.yml new file mode 100644 index 00000000..7b1f3535 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/linux_process_creation.yml @@ -0,0 +1,18 @@ +platform: Athena +source: linux_process_creation + + + +default_log_source: + table: eventlog + + +field_mapping: + CommandLine: process.cmd_line + Image: process.file.path + ParentCommandLine: + - process.parent_process.cmd_line + - actor.process.cmd_line + ParentImage: + - process.parent_process.file.path + - actor.process.file.path \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/athena/macos_file_event.yml b/uncoder-core/app/translator/mappings/platforms/athena/macos_file_event.yml new file mode 100644 index 00000000..e969e6c4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/macos_file_event.yml @@ -0,0 +1,10 @@ +platform: Athena +source: macos_file_event + + +default_log_source: + table: eventlog + +field_mapping: + Image: actor.process.file.path + TargetFilename : file.name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/athena/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/athena/macos_process_creation.yml new file mode 100644 index 00000000..53462d06 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/macos_process_creation.yml @@ -0,0 +1,13 @@ +platform: Athena +source: macos_process_creation + + + +default_log_source: + table: eventlog + +field_mapping: + CommandLine: process.cmd_line + Image: process.file.path + ParentCommandLine: process.parent_process.cmd_line;actor.process.cmd_line + ParentImage: process.parent_process.file.path;actor.process.file.path \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/athena/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/athena/windows_file_event.yml new file mode 100644 index 00000000..4535a4d7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/windows_file_event.yml @@ -0,0 +1,16 @@ +platform: Microsoft Sentinel +source: windows_file_event + + +default_log_source: + table: eventlog + +field_mapping: + CreationUtcTime: file.created_time + FileName: file.name + Image: actor.process.file.path + OriginalFilename: file.name + TargetFilename: file.name + User: + - actor.user.name + - actor.process.user.name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/athena/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/athena/windows_image_load.yml new file mode 100644 index 00000000..901b2888 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/windows_image_load.yml @@ -0,0 +1,11 @@ +platform: Athena +source: windows_image_load + + +default_log_source: + table: eventlog + +field_mapping: + Image: actor.process.file.path + ImageLoaded: module.file.path + OriginalFileName: module.file.path diff --git a/uncoder-core/app/translator/mappings/platforms/athena/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/athena/windows_process_creation.yml new file mode 100644 index 00000000..6d75881d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/windows_process_creation.yml @@ -0,0 +1,25 @@ +platform: Athena +source: windows_process_creation + + +default_log_source: + table: eventlog + +field_mapping: + CommandLine: process.cmd_line + Hashes: + - proces.file.fingerprints.value + - actor.proces.file.fingerprints.value + Image: process.file.path + IntegrityLevel: process.integrity + ParentCommandLine: + - process.parent_process.cmd_line + - actor.process.cmd_line + ParentImage: + - process.parent_process.file.path + - actor.process.file.path + ParentUser: process.parent_process.user + Product: + - process.file.product.name + - process.file.product.vendor_name + User: process.user.name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/athena/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/athena/windows_registry_event.yml new file mode 100644 index 00000000..59dbaaf4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/windows_registry_event.yml @@ -0,0 +1,14 @@ +platform: Athena +source: windows_registry_event + + +default_log_source: + table: eventlog + +field_mapping: + TargetObject: + - reg_key.path + - reg_value.name + - reg_value.path + Image: actor.process.file.path + User: actor.process.user.name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/athena/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/athena/windows_security.yml new file mode 100644 index 00000000..ef24c377 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/athena/windows_security.yml @@ -0,0 +1,16 @@ +platform: Athena +source: windows_security + + +default_log_source: + table: eventlog + +field_mapping: + AccessMask: unmapped.Access Mask + ObjectName: unmapped.win_resource.name + ObjectType: unmapped.win_resource.type + CommandLine: process.cmd_line + PrivilegeList: unmapped.Service Request Information.Privileges + ProcessName: actor.process.file.path + SubjectUserName: actor.user.name + Service: unmapped.win_resource.name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/default.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/default.yml new file mode 100644 index 00000000..a1db3852 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/default.yml @@ -0,0 +1,2 @@ +platform: CarbonBlack +source: default diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/linux_dns_query.yml new file mode 100644 index 00000000..e23d35bf --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/linux_dns_query.yml @@ -0,0 +1,8 @@ +platform: CarbonBlack +source: linux_dns_query + + +field_mapping: + User: + - childproc_username + - process_username diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/linux_network_connection.yml new file mode 100644 index 00000000..5c6eda13 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/linux_network_connection.yml @@ -0,0 +1,9 @@ +platform: CarbonBlack +source: linux_network_connection + + +field_mapping: + DestinationHostname: + - netconn_domain + - netconn_proxy_domain + DestinationPort: netconn_port diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/macos_dns_query.yml new file mode 100644 index 00000000..ddff23a5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/macos_dns_query.yml @@ -0,0 +1,8 @@ +platform: CarbonBlack +source: macos_dns_query + + +field_mapping: + User: + - childproc_username + - process_username diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/macos_network_connection.yml new file mode 100644 index 00000000..d61abbf4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/macos_network_connection.yml @@ -0,0 +1,9 @@ +platform: CarbonBlack +source: macos_network_connection + + +field_mapping: + DestinationHostname: + - netconn_domain + - netconn_proxy_domain + DestinationPort: netconn_port diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_create_remote_thread.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_create_remote_thread.yml new file mode 100644 index 00000000..11a6cf67 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_create_remote_thread.yml @@ -0,0 +1,7 @@ +platform: CarbonBlack +source: windows_create_remote_thread + + +field_mapping: + SourceImage: parent_name + StartModule: modload_name diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_dns_query.yml new file mode 100644 index 00000000..8f1a84b9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_dns_query.yml @@ -0,0 +1,8 @@ +platform: CarbonBlack +source: windows_dns_query + + +field_mapping: + User: + - childproc_username + - process_username diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_file_event.yml new file mode 100644 index 00000000..86fcf3a5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_file_event.yml @@ -0,0 +1,8 @@ +platform: CarbonBlack +source: windows_file_event + + +field_mapping: + User: + - childproc_username + - process_username diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_image_load.yml new file mode 100644 index 00000000..11199a15 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_image_load.yml @@ -0,0 +1,6 @@ +platform: CarbonBlack +source: windows_image_load + + +field_mapping: + OriginalFileName: process_original_filename diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_network_connection.yml new file mode 100644 index 00000000..8017db4f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_network_connection.yml @@ -0,0 +1,9 @@ +platform: CarbonBlack +source: windows_network_connection + + +field_mapping: + DestinationHostname: + - netconn_domain + - netconn_proxy_domain + DestinationPort: netconn_port diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_process_creation.yml new file mode 100644 index 00000000..cb4fc2c8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_process_creation.yml @@ -0,0 +1,14 @@ +platform: CarbonBlack +source: windows_process_creation + + +field_mapping: + Hashes: + - md5 + - filewrite_md5 + - childproc_md5 + - parent_md5 + User: + - childproc_username + - process_username + OriginalFileName: process_original_filename \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_registry_event.yml new file mode 100644 index 00000000..ff1b0aee --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_registry_event.yml @@ -0,0 +1,9 @@ +platform: CarbonBlack +source: windows_registry_event + + +field_mapping: + TargetObject: regmod_name + User: + - childproc_username + - process_username diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_security.yml new file mode 100644 index 00000000..6e288c0b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_security.yml @@ -0,0 +1,17 @@ +platform: CarbonBlack +source: windows_security + + +field_mapping: + AccountName: + - process_username + - childproc_username + ComputerName: device_name + NewProcessName: process_name + DeviceDescription: + - process_product_name + - process_product_version + - process_publisher + - process_file_description + DestPort: netconn_port + UserID: parent_name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_sysmon.yml new file mode 100644 index 00000000..65778b83 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/carbonblack/windows_sysmon.yml @@ -0,0 +1,51 @@ +platform: CarbonBlack +source: windows_sysmon + + + +field_mapping: + CommandLine: process_cmdline + Image: process_name + ParentImage: parent_name + Company: process_publisher + Description: + - process_product_name + - process_product_version + - process_publisher + - process_file_description + DestinationHostname: + - netconn_domain + - netconn_proxy_domain + DestinationIp: + - netconn_ipv4 + - netconn_ipv6 + DestinationIsIpv6: ipaddr + Hashes: + - md5 + - filewrite_md5 + - childproc_md5 + - parent_md5 + IntegrityLevel: process_integrity_level + ParentCommandLine: parent_cmdline + Product: + - process_product_name + - process_file_description + SourceIp: + - netconn_ipv4 + - netconn_ipv6 + - netconn_local_ipv4 + - netconn_local_ipv6 + SourcePort: netconn_port + TargetFilename: filemod_name + User: childproc_username;process_username + OriginalFileName: process_original_filename + Signature: + - childproc_publisher + - filemod_publisher + - modload_publisher + - parent_publisher + - process_publisher + ImageLoaded: modload_name + StartModule: modload_name + TargetImage: filemod_name + FileVersion: process_product_version \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/default.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/default.yml new file mode 100644 index 00000000..6af5af20 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/default.yml @@ -0,0 +1,2 @@ +platform: Chronicle +source: default diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_create_remote_thread.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_create_remote_thread.yml new file mode 100644 index 00000000..94fe955a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_create_remote_thread.yml @@ -0,0 +1,11 @@ +platform: Chronicle +source: windows_create_remote_thread + + + +field_mapping: + SourceImage: principal.process.file.full_path + TargetImage: target.process.file.full_path + StartModule: target.resource.name + StartAddress: target.resource.name + StartFunction: target.resource.name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_dns_query.yml new file mode 100644 index 00000000..31889bcd --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_dns_query.yml @@ -0,0 +1,9 @@ +platform: Chronicle +source: windows_dns_query + + + +field_mapping: + Image: principal.process.file.full_path + QueryName: network.dns.questions + QueryResults: network.dns.answers.data \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_file_event.yml new file mode 100644 index 00000000..3e277768 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_file_event.yml @@ -0,0 +1,10 @@ +platform: Chronicle +source: windows_file_event + + + +field_mapping: + CreationUtcTime: target.resource.name + Image: principal.process.file.full_path + PreviousCreationUtcTime: target.resource.attribute.labels.value + TargetFilename: target.file.full_path \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_image_load.yml new file mode 100644 index 00000000..d494e165 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_image_load.yml @@ -0,0 +1,15 @@ +platform: Chronicle +source: windows_image_load + + + +field_mapping: + Image: principal.process.file.full_path + ImageLoaded: + - target.process.file.full_path + - principal.process.file.full_path + SignatureStatus: target.resource.attribute.labels[2].value + Signed: target.resource.attribute.labels[0].value + Hashes: + - target.process.file.md5 + - target.process.file.sha256;target.process.file.sha1 \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_network_connection.yml new file mode 100644 index 00000000..6c3836de --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_network_connection.yml @@ -0,0 +1,12 @@ +platform: Chronicle +source: windows_network_connection + + + +field_mapping: + Image: principal.process.file.full_path + DestinationHostname: target.hostname + DestinationIp: target.ip + DestinationPort: target.port + SourceIp: principal.ip + SourcePort: principal.port diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_pipe_created.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_pipe_created.yml new file mode 100644 index 00000000..95e5eab3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_pipe_created.yml @@ -0,0 +1,7 @@ +platform: Chronicle +source: windows_pipe_created + + +field_mapping: + PipeName: target.resource.name + Image: target.process.file.full_path \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_process_access.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_process_access.yml new file mode 100644 index 00000000..5d571baa --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_process_access.yml @@ -0,0 +1,8 @@ +platform: Chronicle +source: windows_process_access + + +field_mapping: + TargetImage: target.process.file.full_path + SourceImage: principal.process.file.full_path + GrantedAccess: target.resource.name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_process_creation.yml new file mode 100644 index 00000000..9ae0cde1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_process_creation.yml @@ -0,0 +1,12 @@ +platform: Chronicle +source: windows_process_creation + + + +field_mapping: + CommandLine: target.process.command_line + CurrentDirectory: src.file.full_path + Hashes: target.process.file.md5 + Image: target.process.file.full_path + ParentCommandLine: principal.process.command_line + ParentImage: principal.process.file.full_path diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_registry_event.yml new file mode 100644 index 00000000..0955c33d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_registry_event.yml @@ -0,0 +1,9 @@ +platform: Chronicle +source: windows_registry_event + + + +field_mapping: + TargetObject: target.registry.registry_key + Image: principal.process.file.full_path + Details: target.registry.registry_value_data \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_security.yml new file mode 100644 index 00000000..b5d85f96 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_security.yml @@ -0,0 +1,12 @@ +platform: Chronicle +source: windows_security + + + +field_mapping: + EventID: metadata.product_event_type + AccessMask: security_result.about.resource.name + AccountName: principal.user.role_name + AuthenticationPackageName: security_result.about.resource.name + ComputerName: principal.hostname + GrantedAccess: resource.name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/chronicle/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_sysmon.yml new file mode 100644 index 00000000..bb2bbcdc --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/chronicle/windows_sysmon.yml @@ -0,0 +1,34 @@ +platform: Chronicle +source: windows_sysmon + + + +field_mapping: + CommandLine: target.process.command_line + Image: target.process.file.full_path + ParentImage: principal.process.file.full_path + EventID: metadata.product_event_type + CallTrace: principal.process.file.full_path + Description: metadata.description + DestinationHostname: target.hostname + DestinationIp: target.ip + DestinationPort: target.port + ParentCommandLine: principal.process.command_line + Protocol: network.ip_protocol + RuleName: security_result.rule_name + SourceHostname: principal.hostname + SourceIp: principal.ip + SourcePort: principal.port + TargetFilename: target.file.full_path + TargetObject: target.registry.registry_key + Details: target.registry.registry_value_data + QueryName: network.dns.questions + QueryResults: network.dns.answers + PipeName: target.resource.name + ImageLoaded: principal.process.file.full_path + Imphash: target.process.file.file_metadata.pe.import_hash + SourceImage: principal.process.file.full_path + StartModule: target.resource.name + TargetImage: target.process.file.full_path + StartFunction: ScriptBlockText + event.Technique: security_result.detection_fields.value diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/default.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/default.yml new file mode 100644 index 00000000..e682d14f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/default.yml @@ -0,0 +1,10 @@ +platform: Crowdstrike +source: default + + +log_source: + event_simpleName: + - ProcessRollup2 + +default_log_source: + event_simpleName: ProcessRollup2 diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/linux_dns_query.yml new file mode 100644 index 00000000..b8ba86b4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/linux_dns_query.yml @@ -0,0 +1,15 @@ +platform: Crowdstrike +source: linux_dns_query + + +log_source: + event_simpleName: [DnsRequest] + +default_log_source: + event_simpleName: DnsRequest + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/linux_network_connection.yml new file mode 100644 index 00000000..550fe0d5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/linux_network_connection.yml @@ -0,0 +1,19 @@ +platform: Crowdstrike +source: linux_network_connection + + +log_source: + event_simpleName: [NetworkConnectIP4] + +default_log_source: + event_simpleName: NetworkConnectIP4 + +field_mapping: + CommandLine: CommandLine + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + ParentImage: ParentImage \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/linux_process_creation.yml new file mode 100644 index 00000000..b517a534 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/linux_process_creation.yml @@ -0,0 +1,15 @@ +platform: Crowdstrike +source: linux_process_creation + + +log_source: + event_simpleName: [ProcessRollup2] + +default_log_source: + event_simpleName: ProcessRollup2 + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/macos_dns_query.yml new file mode 100644 index 00000000..811c741e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/macos_dns_query.yml @@ -0,0 +1,15 @@ +platform: Crowdstrike +source: macos_dns_query + + +log_source: + event_simpleName: [DnsRequest] + +default_log_source: + event_simpleName: DnsRequest + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/macos_network_connection.yml new file mode 100644 index 00000000..3bdc540d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/macos_network_connection.yml @@ -0,0 +1,19 @@ +platform: Crowdstrike +source: macos_network_connection + + +log_source: + event_simpleName: [NetworkConnectIP4] + +default_log_source: + event_simpleName: NetworkConnectIP4 + +field_mapping: + CommandLine: CommandLine + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + ParentImage: ParentImage \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/macos_process_creation.yml new file mode 100644 index 00000000..efbeb5be --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/macos_process_creation.yml @@ -0,0 +1,15 @@ +platform: Crowdstrike +source: macos_process_creation + + +log_source: + event_simpleName: [ProcessRollup2] + +default_log_source: + event_simpleName: ProcessRollup2 + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_dns_query.yml new file mode 100644 index 00000000..dc460135 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_dns_query.yml @@ -0,0 +1,15 @@ +platform: Crowdstrike +source: windows_dns_query + + +log_source: + event_simpleName: [DnsRequest] + +default_log_source: + event_simpleName: DnsRequest + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_driver_load.yml new file mode 100644 index 00000000..12e37fa3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_driver_load.yml @@ -0,0 +1,14 @@ +platform: Crowdstrike +source: windows_driver_load + + +log_source: + event_simpleName: [DriverLoad] + +default_log_source: + event_simpleName: DriverLoad + +field_mapping: + ImageLoaded: ImageLoaded + ImagePath: ImagePath + Hashes: Hashes \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_image_load.yml new file mode 100644 index 00000000..fa60f7d0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_image_load.yml @@ -0,0 +1,16 @@ +platform: Crowdstrike +source: windows_image_load + + +log_source: + event_simpleName: [LoadImage] + +default_log_source: + event_simpleName: LoadImage + +field_mapping: + Image: Image + ImageLoaded: ImageLoaded + SignatureStatus: SignatureStatus + OriginalFileName: OriginalFileName + Signed: Signed \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_network_connection.yml new file mode 100644 index 00000000..6e8d4921 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_network_connection.yml @@ -0,0 +1,18 @@ +platform: Crowdstrike +source: windows_network_connection + + +log_source: + event_simpleName: [NetworkConnectIP4] + +default_log_source: + event_simpleName: NetworkConnectIP4 + +field_mapping: + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + Initiated: Initiated \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_process_creation.yml new file mode 100644 index 00000000..2b317059 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_process_creation.yml @@ -0,0 +1,21 @@ +platform: Crowdstrike +source: windows_process_creation + + +log_source: + event_simpleName: [ProcessRollup2] + +default_log_source: + event_simpleName: ProcessRollup2 + +field_mapping: + CommandLine: CommandLine + CurrentDirectory: CurrentDirectory + Hashes: Hashes + Image: Image + IntegrityLevel: IntegrityLevel + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage + ParentUser: ParentUser + Product: Product + User: User diff --git a/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_registry_event.yml new file mode 100644 index 00000000..0886c504 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_registry_event.yml @@ -0,0 +1,16 @@ +platform: Crowdstrike +source: windows_registry_event + + +log_source: + event_simpleName: [RegistryOperationDetectInfo] + +default_log_source: + event_simpleName: RegistryOperationDetectInfo + +field_mapping: + TargetObject: TargetObject + Image: Image + Details: Details + EventType: EventType + User: User diff --git a/siem-converter/app/converter/mappings/platforms/crowdstrike/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_sysmon.yml similarity index 100% rename from siem-converter/app/converter/mappings/platforms/crowdstrike/windows_sysmon.yml rename to uncoder-core/app/translator/mappings/platforms/crowdstrike/windows_sysmon.yml diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/aws_cloudtrail.yml new file mode 100644 index 00000000..77ce3355 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/aws_cloudtrail.yml @@ -0,0 +1,35 @@ +platform: ElasticSearch +source: aws_cloudtrail + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + eventSource: event.provider + eventName: event.action + AdditionalEventData: AdditionalEventData + additionalEventData.MFAUsed: additionalEventData.MFAUsed + errorCode: errorCode + errorMessage: errorMessage + eventType: eventType + requestParameters: requestParameters + requestParameters.attribute: requestParameters.attribute + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP + requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort + requestParameters.userData: requestParameters.userData + responseElements: responseElements + responseElements.ConsoleLogin: responseElements.ConsoleLogin + responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword + responseElements.publiclyAccessible: responseElements.publiclyAccessible + status: status + terminatingRuleId: terminatingRuleId + userAgent: userAgent + userIdentity.arn: userIdentity.arn + userIdentity.principalId: userIdentity.principalId + userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type + userIdentity.type: userIdentity.type + userIdentity.userName: userIdentity.userName diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/aws_eks.yml new file mode 100644 index 00000000..f7b89081 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/aws_eks.yml @@ -0,0 +1,25 @@ +platform: ElasticSearch +source: aws_eks + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy + aws_node_type: aws_node_type + objectRef.namespace: objectRef.namespace + objectRef.resource: objectRef.resource + objectRef.subresource: objectRef.subresource + requestObject.rules.resources: requestObject.rules.resources + requestObject.rules.verbs: requestObject.rules.verbs + requestObject.spec.containers.image: requestObject.spec.containers.image + requestURI: requestURI + stage: stage + user.groups: user.groups + user.username: user.username + verb: verb diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_AzureDiagnostics.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_AzureDiagnostics.yml new file mode 100644 index 00000000..b830ced9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_AzureDiagnostics.yml @@ -0,0 +1,13 @@ +platform: ElasticSearch +source: azure_AzureDiagnostics + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ResultDescription: ResultDescription + Category: Category \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_BehaviorAnalytics.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_BehaviorAnalytics.yml new file mode 100644 index 00000000..2cde670a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_BehaviorAnalytics.yml @@ -0,0 +1,20 @@ +platform: ElasticSearch +source: azure_BehaviorAnalytics + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ActionType: ActionType + ActivityInsights: ActivityInsights + ActivityType: ActivityType + EventSource: EventSource + DevicesInsights: DevicesInsights + RiskDetail: RiskDetail + UsersInsights: UsersInsights + UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount + UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_aadnoninteractiveusersigninlogs.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_aadnoninteractiveusersigninlogs.yml new file mode 100644 index 00000000..9d9a04f7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_aadnoninteractiveusersigninlogs.yml @@ -0,0 +1,13 @@ +platform: ElasticSearch +source: azure_aadnoninteractiveusersigninlogs + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + UserAgent: UserAgent + Type: Type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_azureactivity.yml new file mode 100644 index 00000000..188fdff3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_azureactivity.yml @@ -0,0 +1,22 @@ +platform: ElasticSearch +source: azure_azureactivity + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ActivityStatus: ActivityStatus + ActivityStatusValue: ActivityStatusValue + ActivitySubstatusValue: ActivitySubstatusValue + Authorization: Authorization + Category: Category + CategoryValue: CategoryValue + OperationName: OperationName + OperationNameValue: OperationNameValue + ResourceId: ResourceId + ResourceProviderValue: ResourceProviderValue + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_azuread.yml new file mode 100644 index 00000000..38b6404d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_azuread.yml @@ -0,0 +1,23 @@ +platform: ElasticSearch +source: azure_azuread + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ActivityDisplayName: event.action + Category: azure.auditlogs.properties.category + LoggedByService: azure.auditlogs.properties.logged_by_service + Result: event.outcome + OperationName: OperationName + TargetResources: TargetResources + AADOperationType: AADOperationType + InitiatedBy: InitiatedBy + ResultReason: ResultReason + Status: Status + Status.errorCode: Status.errorCode + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_m365.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_m365.yml new file mode 100644 index 00000000..a8680631 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_m365.yml @@ -0,0 +1,23 @@ +platform: ElasticSearch +source: azure_m365 + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ClientInfoString: ClientInfoString + LogonError: LogonError + ModifiedProperties: ModifiedProperties + OfficeObjectId: OfficeObjectId + OfficeWorkload: OfficeWorkload + Operation: Operation + Parameters: Parameters + RecordType: RecordType + ResultStatus: ResultStatus + SourceFileExtension: SourceFileExtension + SourceFileName: SourceFileName + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_signinlogs.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_signinlogs.yml new file mode 100644 index 00000000..dfc0dd8e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/azure_signinlogs.yml @@ -0,0 +1,29 @@ +platform: ElasticSearch +source: azure_signinlogs + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + AppDisplayName: AppDisplayName + AppId: AppId + AuthenticationRequirement: AuthenticationRequirement + Category: Category + ConditionalAccessStatus: ConditionalAccessStatus + DeviceDetail: DeviceDetail + IsInteractive: IsInteractive + NetworkLocationDetails: NetworkLocationDetails + ResourceDisplayName: ResourceDisplayName + ResourceIdentity: ResourceIdentity + ResultDescription: ResultDescription + ResultType: ResultType + Status.errorCode: Status.errorCode + Status: Status + Status.failureReason: Status.failureReason + TokenIssuerType: TokenIssuerType + UserAgent: UserAgent + UserPrincipalName: UserPrincipalName diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/default.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/default.yml new file mode 100644 index 00000000..87685092 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/default.yml @@ -0,0 +1,6 @@ +platform: ElasticSearch +source: default + + +default_log_source: + index: "*" \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/dns.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/dns.yml new file mode 100644 index 00000000..a8ceaba8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/dns.yml @@ -0,0 +1,17 @@ +platform: ElasticSearch +source: dns + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + dns-query: dns.question.name + parent-domain: + - url.registered_domain + - destination.registered_domain + dns-answer: dns.answers.name + dns-record: dns.question.type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/firewall.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/firewall.yml new file mode 100644 index 00000000..84877506 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/firewall.yml @@ -0,0 +1,19 @@ +platform: ElasticSearch +source: firewall + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + src-ip: + - source.address + - source.ip + src-port: source.port + dst-ip: + - destination.address + - destination.ip + dst-port: destination.port \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/gcp_gcp.audit.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/gcp_gcp.audit.yml new file mode 100644 index 00000000..ee7c50f9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/gcp_gcp.audit.yml @@ -0,0 +1,21 @@ +platform: ElasticSearch +source: gcp_gcp.audit + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + backupConfiguration.enabled: backupConfiguration.enabled + gcp.audit.method_name: gcp.audit.method_name + jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction + jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue + message: message + methodName: methodName + protoPayload.methodName: protoPayload.methodName + protoPayload.request.canIpForward: protoPayload.request.canIpForward + protoPayload.serviceName: protoPayload.serviceName + serviceName: serviceName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/gcp_pubsub.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/gcp_pubsub.yml new file mode 100644 index 00000000..9c853aed --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/gcp_pubsub.yml @@ -0,0 +1,16 @@ +platform: ElasticSearch +source: gcp_pubsub + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + data.protoPayload.methodName: data.protoPayload.methodName + data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member + data.resource.type: data.resource.type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/linux_auditd.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/linux_auditd.yml new file mode 100644 index 00000000..bc3077d9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/linux_auditd.yml @@ -0,0 +1,16 @@ +platform: ElasticSearch +source: linux_auditd + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + a0: process.args + a1: process.args + a2: process.args + a3: process.args + exe: process.executable diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/linux_dns_query.yml new file mode 100644 index 00000000..276da305 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/linux_dns_query.yml @@ -0,0 +1,15 @@ +platform: ElasticSearch +source: linux_dns_query + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + Image: process.executable + User: user.name + QueryName: dns.question.name + QueryResults: dns.answers \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/linux_process_creation.yml new file mode 100644 index 00000000..608acfac --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/linux_process_creation.yml @@ -0,0 +1,15 @@ +platform: ElasticSearch +source: linux_process_creation + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/macos_dns_query.yml new file mode 100644 index 00000000..ff6cd11c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/macos_dns_query.yml @@ -0,0 +1,15 @@ +platform: ElasticSearch +source: macos_dns_query + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + Image: process.executable + User: user.name + QueryName: dns.question.name + QueryResults: dns.answers \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/macos_network_connection.yml new file mode 100644 index 00000000..952f6935 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/macos_network_connection.yml @@ -0,0 +1,18 @@ +platform: ElasticSearch +source: macos_network_connection + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + Image: process.executable + DestinationHostname: destination.domain + DestinationIp: destination.ip + DestinationPort: destination.port + SourceIp: source.ip + SourcePort: source.port + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/macos_process_creation.yml new file mode 100644 index 00000000..42c08125 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/macos_process_creation.yml @@ -0,0 +1,15 @@ +platform: ElasticSearch +source: macos_process_creation + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/okta_okta.yml new file mode 100644 index 00000000..2755ba61 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/okta_okta.yml @@ -0,0 +1,33 @@ +platform: ElasticSearch +source: okta_okta + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + client.user.id: okta.actor.id + source.user.id: okta.actor.id + User: okta.actor.type + alternateId: okta.actor.alternate_id + client.user.full_name: okta.actor.display_name + source.user.full_name: okta.actor.display_name + related.user: okta.actor.display_name + client.ip: okta.client.ip + source.ip: okta.client.ip + user_agent.original: okta.client.user_agent.raw_user_agent + userAgent.os: okta.client.user_agent.os + userAgent.browser: okta.client.user_agent.browser + client.zone: okta.client.zone + client.device: okta.client.device + client.id: okta.client.id + event.action: okta.event_type + outcome.reason: okta.outcome.reason + event.outcome: okta.event.outcome + client.as.number: okta.security_context.as.number + client.as.organization.name: okta.security_context.as.organization.name + client.domain: okta.security_context.isp + source.domain: okta.security_context.domain diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/proxy.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/proxy.yml new file mode 100644 index 00000000..cd67e710 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/proxy.yml @@ -0,0 +1,30 @@ +platform: ElasticSearch +source: proxy + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + c-uri: url.original + c-useragent: user_agent.original + cs-method: http.request.method + cs-bytes: http.request.body.bytes + cs-cookie-vars: http.cookie_vars + c-uri-extension: url.extension + c-uri-query: url.query + cs-cookie: http.cookie_vars + cs-host: + - url.domain + - destination.domain + + cs-referrer: http.request.referrer + cs-version: http.version + r-dns: + - destination.domain + - url.domain + sc-status: http.response.status_code + post-body: http.post_body \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/webserver.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/webserver.yml new file mode 100644 index 00000000..0011f16c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/webserver.yml @@ -0,0 +1,29 @@ +platform: ElasticSearch +source: webserver + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + c-uri: url.original + c-useragent: user_agent.original + cs-method: http.request.method + cs-bytes: http.request.body.bytes + cs-cookie-vars: http.cookie_vars + c-uri-extension: url.extension + c-uri-query: url.query + cs-cookie: http.cookie_vars + cs-host: + - url.domain + - destination.domain + cs-referrer: http.request.referrer + cs-version: http.version + r-dns: + - destination.domain + - url.domain + sc-status: http.response.status_code + post-body: http.post_body \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_bits_client.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_bits_client.yml new file mode 100644 index 00000000..5ea62c7e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_bits_client.yml @@ -0,0 +1,19 @@ +platform: ElasticSearch +source: windows_bits_client + +conditions: + winlog.channel: 'Microsoft-Windows-Bits-Client/Operational' + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + LocalName: winlog.event_data.LocalName + EventID: + - event.code + - winlog.event_id + RemoteName: winlog.event_data.RemoteName + processPath: winlog.event_data.processPath diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_dns_query.yml new file mode 100644 index 00000000..f175f14e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_dns_query.yml @@ -0,0 +1,18 @@ +platform: ElasticSearch +source: macos_dns_query + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + Image: process.executable + User: + - user.name + - winlog.user.domain + - winlog.user.name + QueryName: dns.question.name + QueryResults: dns.answers \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_driver_load.yml new file mode 100644 index 00000000..ab06904a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_driver_load.yml @@ -0,0 +1,14 @@ +platform: ElasticSearch +source: windows_driver_load + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + ImageLoaded: file.path + ImagePath: file.path + Hashes: winlog.event_data.Hashes \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_image_load.yml new file mode 100644 index 00000000..38e615c1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_image_load.yml @@ -0,0 +1,20 @@ +platform: ElasticSearch +source: windows_image_load + +conditions: + event.action: + - 'Image loaded (rule: ImageLoad)' + - 'load' + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + Image: process.executable + ImageLoaded: file.path + SignatureStatus: SignatureStatus + OriginalFileName: OriginalFileName + Signed: Signed \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_ldap_debug.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_ldap_debug.yml new file mode 100644 index 00000000..1a8d1b6b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_ldap_debug.yml @@ -0,0 +1,17 @@ +platform: ElasticSearch +source: windows_ldap_debug + +conditions: + winlog.channel: 'Microsoft-Windows-LDAP-Client/Debug' + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + EventID: + - event.code + - winlog.event_id + SearchFilter: winlog.event_data.SearchFilter \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_network_connection.yml new file mode 100644 index 00000000..dcac9463 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_network_connection.yml @@ -0,0 +1,19 @@ +platform: ElasticSearch +source: windows_network_connection + +conditions: + event.category: 'network' + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + Image: process.executable + DestinationHostname: destination.domain + DestinationIp: destination.ip + DestinationPort: destination.port + SourceIp: source.ip + SourcePort: source.port diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_ntlm.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_ntlm.yml new file mode 100644 index 00000000..d8de2ac2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_ntlm.yml @@ -0,0 +1,19 @@ +platform: ElasticSearch +source: windows_ntlm + +conditions: + winlog.provider_name: 'Microsoft-Windows-NTLM/Operational' + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + ProcessName: ProcessName + WorkstationName: winlog.event_data.WorkstationName + EventID: + - event.code + - winlog.event_id + TargetName: winlog.event_data.TargetName diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_powershell.yml new file mode 100644 index 00000000..25be0b51 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_powershell.yml @@ -0,0 +1,25 @@ +platform: ElasticSearch +source: windows_powershell + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + EventID: + - event.code + - winlog.event_id + CommandLine: powershell.command.value + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + HostApplication: process.command_line + ContextInfo: winlog.event_data.ContextInfo + HostName: process.title + EngineVersion: + - powershell.engine.version + - powershell.process.executable_version \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_process_creation.yml new file mode 100644 index 00000000..c0c1dea8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_process_creation.yml @@ -0,0 +1,23 @@ +platform: ElasticSearch +source: windows_process_creation + +conditions: + event.category: 'process' + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + CommandLine: process.command_line + CurrentDirectory: CurrentDirectory + Hashes: Hashes + Image: process.executable + IntegrityLevel: IntegrityLevel + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable + ParentUser: ParentUser + Product: Product + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_security.yml new file mode 100644 index 00000000..5a01016a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_security.yml @@ -0,0 +1,151 @@ +platform: ElasticSearch +source: windows_security + +conditions: + winlog.channel : "Security" + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + EventID: + - event.code + - winlog.event_id + ParentImage: process.parent.executable + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + Channel: winlog.channel + ComputerName: winlog.ComputerName + EventType: winlog.event_data.EventType + FailureReason: winlog.event_data.FailureCode + FileName: file.path + GrantedAccess: winlog.event_data.GrantedAccess + Hashes: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + IpAddress: source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + LinkName: LinkName + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + Path: winlog.event_data.Path + CommandLine: winlog.event_data.ProcessCommandLine + OldUacValue: winlog.event_data.OldUacValue + CertIssuerName: winlog.event_data.CertIssuerName + SubStatus: winlog.event_data.SubStatus + DisplayName: winlog.event_data.DisplayName + TaskContent: winlog.event_data.TaskContent + ServiceSid: winlog.event_data.ServiceSid + CertThumbprint: winlog.event_data.CertThumbprint + ClassName: winlog.event_data.ClassName + NotificationPackageName: winlog.event_data.NotificationPackageName + NewSd: winlog.event_data.NewSd + TestSigning: winlog.event_data.TestSigning + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ServiceAccount: winlog.event_data.ServiceAccount + AttributeValue: winlog.event_data.AttributeValue + SessionName: winlog.event_data.SessionName + TaskName: winlog.event_data.TaskName + ObjectDN: winlog.event_data.ObjectDN + TemplateContent: winlog.event_data.TemplateContent + NewTemplateContent: winlog.event_data.NewTemplateContent + SourcePort: winlog.event_data.SourcePort + PasswordLastSet: winlog.event_data.PasswordLastSet + PrivilegeList: winlog.event_data.PrivilegeList + DeviceDescription: winlog.event_data.DeviceDescription + TargetServerName: winlog.event_data.TargetServerName + NewTargetUserName: winlog.event_data.NewTargetUserName + OperationType: winlog.event_data.OperationType + DestPort: winlog.event_data.DestPort + ServiceStartType: winlog.event_data.ServiceStartType + OldTargetUserName: winlog.event_data.OldTargetUserName + UserPrincipalName: winlog.event_data.UserPrincipalName + Accesses: winlog.event_data.Accesses + DnsHostName: winlog.event_data.DnsHostName + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + AuditSourceName: winlog.event_data.AuditSourceName + Workstation: winlog.event_data.Workstation + DestAddress: winlog.event_data.DestAddress + PreAuthType: winlog.event_data.PreAuthType + SecurityPackageName: winlog.event_data.SecurityPackageName + SubjectLogonId: winlog.event_data.SubjectLogonId + NewUacValue: winlog.event_data.NewUacValue + EnabledPrivilegeList: winlog.event_data.EnabledPrivilegeList + RelativeTargetName: winlog.event_data.RelativeTargetName + CertSerialNumber: winlog.event_data.CertSerialNumber + SidHistory: winlog.event_data.SidHistory + TargetLogonId: winlog.event_data.TargetLogonId + KernelDebug: winlog.event_data.KernelDebug + CallerProcessName: winlog.event_data.CallerProcessName + ProcessName: process.executable + Properties: winlog.event_data.Properties + UserAccountControl: winlog.event_data.UserAccountControl + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + SecurityDescriptor: + - winlog.event_data.SecurityDescriptor + - winlog.event_data.TaskContentNew + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + NewValue: winlog.event_data.NewValue + Source: winlog.event_data.Source + Status: winlog.event_data.Status + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + SourceAddr: SourceAddr + SourceAddress: SourceAddress + TargetName: TargetName + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + TargetDomainName: winlog.event_data.TargetDomainName + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + ObjectServer: + TargetUserSid: winlog.event_data.TargetUserSid + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + LayerRTID: LayerRTID + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + ParentProcessName: process.parent.name + Service: Service diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_sysmon.yml new file mode 100644 index 00000000..edb7e997 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_sysmon.yml @@ -0,0 +1,69 @@ +platform: ElasticSearch +source: windows_sysmon + +conditions: + winlog.channel: 'Microsoft-Windows-Sysmon/Operational' + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentImage: process.parent.executable + EventID: + - event.code + - winlog.event_id + CallTrace: winlog.event_data.CallTrace + Company: winlog.event_data.Company + CurrentDirectory: process.working_directory + Description: winlog.event_data.Description + DestinationHostname: destination.domain + DestinationIp: destination.ip + DestinationIsIpv6: winlog.event_data.DestinationIsIpv6 + DestinationPort: destination.port + DestinationPortName: network.protocol + Hashes: winlog.event_data.Hashes + Initiated: winlog.event_data.Initiated + IntegrityLevel: winlog.event_data.IntegrityLevel + ParentCommandLine: process.parent.command_line + Product: winlog.event_data.Product + Protocol: winlog.event_data.Protocol + RuleName: winlog.event_data.RuleName + SourceHostname: source.domain + SourceIp: source.ip + SourceIsIpv6: winlog.event_data.SourceIsIpv6 + SourcePort: source.port + SourcePortName: winlog.event_data.SourcePortName + TargetFilename: file.path + User: + - user.name + - winlog.user.domain + - winlog.user.name + OriginalFileName: winlog.event_data.OriginalFileName + Signed: winlog.event_data.Signed + Signature: winlog.event_data.Signature + SignatureStatus: winlog.event_data.SignatureStatus + TargetObject: winlog.event_data.TargetObject + Details: winlog.event_data.Details + QueryName: dns.question.name + QueryResults: dns.answers + QueryStatus: winlog.event_data.QueryStatus + IsExecutable: winlog.event_data.IsExecutable + PipeName: file.name + ImageLoaded: file.path + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + SourceImage: process.executable + StartModule: winlog.event_data.StartModule + TargetImage: winlog.event_data.TargetImage + Device: Device + ProcessID: ProcessID + FileVersion: FileVersion + StartAddress: StartAddress + StartFunction: StartFunction + EventType: winlog.event_data.EventType \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_wmi_event.yml new file mode 100644 index 00000000..a31347b6 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch/windows_wmi_event.yml @@ -0,0 +1,15 @@ +platform: ElasticSearch +source: windows_wmi_event + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + Destination: Destination + EventID: + - event.code + - winlog.event_id diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/aws_cloudtrail.yml new file mode 100644 index 00000000..81bf4594 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/aws_cloudtrail.yml @@ -0,0 +1,41 @@ +platform: ElasticSearch ES|QL +source: aws_cloudtrail +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + additionalEventdata: aws.cloudtrail.additional_eventdata + apiVersion: aws.cloudtrail.api_version + awsRegion: cloud.region + errorCode: aws.cloudtrail.error_code + errorMessage: aws.cloudtrail.error_message + eventID: event.id + eventName: event.action + eventSource: event.provider + eventTime: '@timestamp' + eventType: aws.cloudtrail.event_type + eventVersion: aws.cloudtrail.event_version + managementEvent: aws.cloudtrail.management_event + readOnly: aws.cloudtrail.read_only + requestID: aws.cloudtrail.request_id + requestParameters: aws.cloudtrail.request_parameters + resources.accountId: aws.cloudtrail.resources.account_id + resources.ARN: aws.cloudtrail.resources.arn + resources.type: aws.cloudtrail.resources.type + responseElements: aws.cloudtrail.response_elements + serviceEventDetails: aws.cloudtrail.service_event_details + sharedEventId: aws.cloudtrail.shared_event_id + sourceIPAddress: source.address + userAgent: user_agent + userIdentity.accessKeyId: aws.cloudtrail.user_identity.access_key_id + userIdentity.accountId: cloud.account.id + userIdentity.arn: aws.cloudtrail.user_identity.arn + userIdentity.invokedBy: aws.cloudtrail.user_identity.invoked_by + userIdentity.principalId: user.id + userIdentity.sessionContext.attributes.creationDate: aws.cloudtrail.user_identity.session_context.creation_date + userIdentity.sessionContext.attributes.mfaAuthenticated: aws.cloudtrail.user_identity.session_context.mfa_authenticated + userIdentity.sessionContext.sessionIssuer.userName: role.name + userIdentity.type: aws.cloudtrail.user_identity.type + userIdentity.userName: user.name + vpcEndpointId: aws.cloudtrail.vpc_endpoint_id diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/azure_mcas.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/azure_mcas.yml new file mode 100644 index 00000000..6cd2c7f8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/azure_mcas.yml @@ -0,0 +1,301 @@ +platform: ElasticSearch ES|QL +source: azure_mcas +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + Name: o365.audit.Name + ProviderName: + - winlog.event_data.ProviderName + - winlog.provider_name + dns_query_name: dns.question.name + EventID: winlog.event_id + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AttributeValue: winlog.event_data.AttributeValue + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + CallTrace: winlog.event_data.CallTrace + Channel: winlog.channel + CommandLine: process.command_line.text + Command_Line: process.command_line.text + Commandline: process.command_line.text + commandline: process.command_line.text + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + ComputerName: winlog.ComputerName + CurrentDirectory: process.working_directory.text + Description: winlog.event_data.Description + DestinationHostname: + - destination.domain + - dns.question.name + - dns.question.subdomain + DestinationIp: destination.address + dst_ip: destination.address + DestinationPort: destination.port + dst_port: destination.port + DestinationPortName: network.protocol + Details: winlog.event_data.Details + EngineVersion: winlog.event_data.EngineVersion + EventType: winlog.event_data.EventType + FailureCode: winlog.event_data.FailureCode + FileName: file.path.text + GrantedAccess: winlog.event_data.GrantedAccess + GroupName: + - winlog.event_data.GroupName + - group.name + GroupSid: + - group.id + - winlog.event_data.GroupSid + Hashes: winlog.event_data.Hashes + file_hash: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + HostVersion: winlog.event_data.HostVersion + Image: process.executable.text + ImageLoaded: dll.path + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + IpAddress: source.address + ClientAddress: + - winlog.event_data.ClientAddress + - source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + ParentCommandLine: process.parent.command_line.text + ParentProcessName: process.parent.name.text + ParentImage: process.parent.executable.text + Path: winlog.event_data.Path + PipeName: file.name + ProcessCommandLine: winlog.event_data.ProcessCommandLine + ProcessName: process.executable.text + Properties: winlog.event_data.Properties + RuleName: winlog.event_data.RuleName + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + Signature: winlog.event_data.Signature + Signed: winlog.event_data.Signed + Source: winlog.event_data.Source + SourceHostname: source.domain + SourceImage: process.executable.text + SourceIp: source.address + src_ip: source.address + SourcePort: source.port + src_port: source.port + StartModule: winlog.event_data.StartModule + Status: winlog.event_data.Status + SubStatus: winlog.event_data.SubStatus + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + TargetDomainName: winlog.event_data.TargetDomainName + TargetFilename: file.path.text + TargetImage: winlog.event_data.TargetImage + TargetObject: winlog.event_data.TargetObject + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + TargetUserSid: winlog.event_data.TargetUserSid + QueryName: dns.question.name + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + User: user.name + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + QueryResults: dns.answers + OriginalFileName: winlog.event_data.OriginalFileName + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + Initiated: winlog.event_data.Initiated + NewUACList: winlog.event_data.NewUACList + UserAccountControl: winlog.event_data.UserAccountControl + NewUacValue: winlog.event_data.NewUacValue + OldUacValue: winlog.event_data.OldUacValue + AccountExpires: winlog.event_data.AccountExpires + DisplayName: winlog.event_data.DisplayName + DnsHostName: winlog.event_data.DnsHostName + HomeDirectory: winlog.event_data.HomeDirectory + HomePath: winlog.event_data.HomePath + LogonHours: winlog.event_data.LogonHours + PasswordLastSet: winlog.event_data.PasswordLastSet + PrimaryGroupId: winlog.event_data.PrimaryGroupId + PrivilegeList: winlog.event_data.PrivilegeList + ProfilePath: winlog.event_data.ProfilePath + SamAccountName: winlog.event_data.SamAccountName + ScriptPath: winlog.event_data.ScriptPath + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + SidHistory: winlog.event_data.SidHistory + UserParameters: winlog.event_data.UserParameters + UserPrincipalName: winlog.event_data.UserPrincipalName + UserWorkstations: winlog.event_data.UserWorkstations + RelativeTargetName: winlog.event_data.RelativeTargetName + NotificationPackageName: winlog.event_data.NotificationPackageName + SecurityPackageName: winlog.event_data.SecurityPackageName + HostApplication: process.command_line.text + TaskName: winlog.event_data.TaskName + TaskContent: winlog.event_data.TaskContent + ObjectServer: winlog.event_data.ObjectServer + NewSd: winlog.event_data.NewSd + OldSd: winlog.event_data.OldSd + TestSigning: winlog.event_data.TestSigning + AdvancedOptions: winlog.event_data.AdvancedOptions + ConfigAccessPolicy: winlog.event_data.ConfigAccessPolicy + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + FlightSigning: winlog.event_data.FlightSigning + HypervisorDebug: winlog.event_data.HypervisorDebug + HypervisorLaunchType: winlog.event_data.HypervisorLaunchType + HypervisorLoadOptions: winlog.event_data.HypervisorLoadOptions + KernelDebug: winlog.event_data.KernelDebug + LoadOptions: winlog.event_data.LoadOptions + RemoteEventLogging: winlog.event_data.RemoteEventLogging + ExceptionCode: winlog.event_data.ExceptionCode + CertSerialNumber: winlog.event_data.CertSerialNumber + CertThumbprint: winlog.event_data.CertThumbprint + CertIssuerName: winlog.event_data.CertIssuerName + TicketOptionsDescription: winlog.event_data.TicketOptionsDescription + keywords: winlog.keywords + StartAddress: winlog.event_data.StartAddress + ServiceSid: winlog.event_data.ServiceSid + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ProcessId: winlog.event_data.ProcessId + ProcessID: winlog.event_data.ProcessID + SubjectLogonId: winlog.event_data.SubjectLogonId + ElevatedToken: winlog.event_data.ElevatedToken + PublishURLs: winlog.event_data.PublishURLs + VMUserAuthenticationEvent: horizon.user_authentication_event + VMUserAuthenticationUser: horizon.user_authentication_user + VMUserAuthenticationSourceIp: horizon.user_authentication_source_ip + VMUserAuthenticationTimeStamp: horizon.user_authentication_time_stamp + VMDesktopSessionStartEvent: horizon.desktop_session_start_event + VMDesktopSessionStartUser: horizon.desktop_session_start_user + VMDesktopSessionStartDesktopID: horizon.desktop_session_start_desktop_id + VMDesktopSessionStartTimeStamp: horizon.desktop_session_time_stamp + VMApplicationLaunchEvent: horizon.application_launch_event + VMApplicationLaunchUser: horizon.application_launch_user + VMApplicationLaunchAppId: horizon.application_launch_app_id + VMApplicationLaunchAppName: horizon.application_launch_app_name + VMApplicationLaunchTimeStamp: horizon.application_launch_time_stamp + VMConnectionServerStatusEvent: horizon.connection_server_status_event + VMConnectionServerStatusServer: horizon.connection_server_status_server + VMConnectionServerStatus: horizon.connection_Server_status + VMConnectionServerStatusTimeStamp: horizon.connection_server_status_time_stamp + VMVirtualDesktopPoolManagmentEvent: horizon.virtual_desktop_pool_managment_event + VMVirtualDesktopPoolManagmentPoolId: horizon.virtual_desktop_pool_managment_pool_id + VMVirtualDesktopPoolManagmentPoolName: horizon.virtual_desktop_pool_managment_pool_name + VMVirtualDesktopPoolManagmentTimeStamp: horizon.virtual_desktop_pool_managment_time_stamp + VMLoadBalancingEvent: horizon.load_balancing_event + VMLoadBalancingStatus: horizon.load_balancing_status + VMLoadBalancingAlgorithm: horizon.load_balancing_algorithm + VMLoadBalancingTimeStamp: horizon.load_balancing_time_stamp + VMBlastProtocolEvent: horizon.blast_protocol_event + VMBlastProtocolUser: horizon.blast_protocol_user + VMBlastProtocolProtocolVersion: horizon.blast_protocol_protocol_version + VMBlastProtocolTimeStamp: horizon.blast_protocol_time_stamp + VMSecurityEventName: horizon.security_event_name + VMSecurityEventUser: horizon.security_event_user + VMSecurityEventAlertType: horizon.security_event_alert_type + VMSecurityEventSourceIp: horizon.security_event_source_ip + VMSecurityEventTimeStamp: horizon.security_event_time_stamp + VMLicensingInformationEvent: horizon.licensing_information_event + VMLicensingInformationLicenseType: horizon.licensing_information_license_type + VMLicensingInformationExpiryDate: horizon.licensing_information_expiry_date + VMLicensingInformationTimeStamp: horizon.licensing_information_time_stamp + VMConnectionBrokeringEvent: horizon.connection_brokering_event + VMConnectionBrokeringUser: horizon.connection_brokering_user + VMConnectionBrokeringDesktopId: horizon.connection_brokering_desktop_id + VMConnectionBrokeringStatus: horizon.connection_brokering_status + VMConnectionBrokeringTimeStamp: horizon.connection_brokering_time_stamp + DatastoreName: vsphere.datastore_name + FilesystemType: vsphere.datastore_fstype + DatastoreBytes: vsphere.datastore_capacity_free_bytes + DatastoreBytesUsed: vsphere.datastore_capacity_used_pct + HostName: vsphere.host_name + UsedCPUmhz: vsphere.host_cpu_free_mhz + UsedMemoryBites: vsphere.host_memory_total_bytes + FreeMemoryBites: vsphere.host_memory_free_bytes + VMHostID: vsphere.virtualmachine_host_id + VMHostName: + - vsphere.virtualmachine_host_hostname + - esxi.vmhost_name + VMName: + - vsphere.virtualmachine_name + - esxi.vmname + VMOperatingSystem: vsphere.virtualmachine_os + VMUsedCPU: vsphere.virtualmachine_cpu_used_mhz + VMTotalCPU: vsphere.virtualmachine_cpu_free_mhz + VMMemoryGuestUsed: vsphere.virtualmachine_memory_used_guest_bytes + VMUMemoryHostUsed: vsphere.virtualmachine_memory_used_host_bytes + VMTotalMemoryGuestBytes: vsphere.virtualmachine_memory_total_guest_bytes + VMMemoryGuestFree: vsphere.virtualmachine_memory_free_guest_bytes + VMCustomFields: vsphere.virtualmachine_custom_fields + VMNetworkNames: vsphere.virtualmachine_network_names + VMLogicalSwitchingEvent: nsxv.vmlogical_switching_event + VMLogicalSwitchingEventID: nsxv.vmlogical_switching_event_id + VMLogicalSwitchingName: nsxv.vmlogical_switching_name + VMDistributedFirewallEvent: nsxv.distributed_firewall_event_type + VMDistributedFirewallRuleID: nsxv.distributed_firewall_rule_id + VMDistributedFirewallAction: nsxv.distributed_firewall_action + VMDistributedFirewallSourceIp: nsxv.distributed_firewall_source_ip + VMDistributedFirewallDestinationIp: nsxv.distributed_firewall_destination_ip + VMSecurityGroupEventType: nsxv.security_group_event_type + VMSecurityGroupId: nsxv.security_group_id + VMSecurityGroupName: nsxv.security_group_name + VMEdgeServicesGatewayEventType: nsxv.security_edge_services_gateway_event_type + VMEdgeServicesGatewayESGID: nsxv.security_edge_services_gateway_esgid + VMEdgeServicesGatewayStatus: nsxv.security_edge_services_gateway_status + VMLoadBalancingEventType: nsxv.load_balancing_event_type + VMLoadBalancingId: nsxv.load_balancing_id + VMLoadBalancingVirtualServer: nsxv.load_balancing_virtual_server + VMNSXManagerEventType: nsxv.nsx_manager_event_type + VMNSXManagerEventDescription: nsxv.nsx_manager_event_description + VMEdgeFirewallEventType: nsxv.edge_firewall_event_type + VMEdgeFirewallSourceIP: nsxv.edge_firewall_source_ip + VMEdgeFirewallDestinationIP: nsxv.edge_firewall_destination_ip + VMEdgeFirewallRuleID: nsxv.edge_firewall_rule_id + VMSSLVPNEventType: nsxv.ssl_vpn_event_type + VMSSLVPNUserName: nsxv.ssl_vpn_user_name + VMSSLVPNSourceIp: nsxv.ssl_vpn_source_ip + VMNSXControllerEventType: nsxv.nsx_controller_event_type + VMNSXControllerID: nsxv.nsx_controller_id + VMNSXControllerStatus: nsxv.nsx_controller_status + VMLogicalRoutingEventType: nsxv.logical_routing_event_type + VMLogicalRoutingRouterID: nsxv.logical_routing_router_id + VMLogicalRoutingRouterName: nsxv.logical_routing_router_name diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/azure_office365.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/azure_office365.yml new file mode 100644 index 00000000..3a3aabbe --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/azure_office365.yml @@ -0,0 +1,387 @@ +platform: ElasticSearch ES|QL +source: azure_office365 +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + GroupName: o365.audit.GroupName + LogonType: o365.audit.LogonType + Source: o365.audit.Source + Status: o365.audit.Status + Actor.ID: o365.audit.Actor.ID + Actor.Type: o365.audit.Actor.Type + ActorContextId: o365.audit.ActorContextId + ActorIpAddress: o365.audit.ActorIpAddress + ActorUserId: o365.audit.ActorUserId + ActorYammerUserId: o365.audit.ActorYammerUserId + AlertEntityId: o365.audit.AlertEntityId + AlertId: o365.audit.AlertId + AlertLinks: o365.audit.AlertLinks + AlertType: o365.audit.AlertType + AppId: o365.audit.AppId + ApplicationDisplayName: o365.audit.ApplicationDisplayName + ApplicationId: o365.audit.ApplicationId + AzureActiveDirectoryEventType: o365.audit.AzureActiveDirectoryEventType + Category: o365.audit.Category + ClientAppId: o365.audit.ClientAppId + ClientIP: o365.audit.ClientIP + ClientIPAddress: o365.audit.ClientIPAddress + ClientInfoString: o365.audit.ClientInfoString + ClientRequestId: o365.audit.ClientRequestId + Comments: o365.audit.Comments + CorrelationId: o365.audit.CorrelationId + CreationTime: o365.audit.CreationTime + CustomUniqueId: o365.audit.CustomUniqueId + Data: o365.audit.Data + DataType: o365.audit.DataType + EntityType: o365.audit.EntityType + ErrorNumber: o365.audit.ErrorNumber + EventData: o365.audit.EventData + EventSource: o365.audit.EventSource + ExceptionInfo: o365.audit.ExceptionInfo + ExchangeMetaData: o365.audit.ExchangeMetaData + ExtendedProperties: o365.audit.ExtendedProperties + ExternalAccess: o365.audit.ExternalAccess + Id: o365.audit.Id + ImplicitShare: o365.audit.ImplicitShare + IncidentId: o365.audit.IncidentId + InterSystemsId: o365.audit.InterSystemsId + InternalLogonType: o365.audit.InternalLogonType + IntraSystemId: o365.audit.IntraSystemId + Item: o365.audit.Item + ItemName: o365.audit.ItemName + ItemType: o365.audit.ItemType + ListId: o365.audit.ListId + ListItemUniqueId: o365.audit.ListItemUniqueId + LogonError: o365.audit.LogonError + LogonUserSid: o365.audit.LogonUserSid + MailboxGuid: o365.audit.MailboxGuid + MailboxOwnerMasterAccountSid: o365.audit.MailboxOwnerMasterAccountSid + MailboxOwnerSid: o365.audit.MailboxOwnerSid + MailboxOwnerUPN: o365.audit.MailboxOwnerUPN + Members: o365.audit.Members + ModifiedProperties: o365.audit.ModifiedProperties + ObjectId: o365.audit.ObjectId + Operation: o365.audit.Operation + OrganizationId: o365.audit.OrganizationId + OrganizationName: o365.audit.OrganizationName + OriginatingServer: o365.audit.OriginatingServer + Parameters: o365.audit.Parameters + PolicyDetails: o365.audit.PolicyDetails + PolicyId: o365.audit.PolicyId + RecordType: o365.audit.RecordType + ResultStatus: o365.audit.ResultStatus + SensitiveInfoDetectionIsIncluded: o365.audit.SensitiveInfoDetectionIsIncluded + SessionId: o365.audit.SessionId + Severity: o365.audit.Severity + SharePointMetaData: o365.audit.SharePointMetaData + Site: o365.audit.Site + SiteUrl: o365.audit.SiteUrl + SourceFileExtension: o365.audit.SourceFileExtension + SourceFileName: o365.audit.SourceFileName + SourceRelativeUrl: o365.audit.SourceRelativeUrl + SupportTicketId: o365.audit.SupportTicketId + Target.ID: o365.audit.Target.ID + Target.Type: o365.audit.Target.Type + TargetContextId: o365.audit.TargetContextId + TargetUserOrGroupName: o365.audit.TargetUserOrGroupName + TargetUserOrGroupType: o365.audit.TargetUserOrGroupType + TeamGuid: o365.audit.TeamGuid + TeamName: o365.audit.TeamName + UniqueSharingId: o365.audit.UniqueSharingId + UserAgent: o365.audit.UserAgent + UserId: o365.audit.UserId + UserKey: o365.audit.UserKey + UserType: o365.audit.UserType + Version: o365.audit.Version + WebId: o365.audit.WebId + Workload: o365.audit.Workload + YammerNetworkId: o365.audit.YammerNetworkId + ProviderName: + - winlog.event_data.ProviderName + - winlog.provider_name + dns_query_name: dns.question.name + EventID: winlog.event_id + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AttributeValue: winlog.event_data.AttributeValue + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + CallTrace: winlog.event_data.CallTrace + Channel: winlog.channel + CommandLine: process.command_line.text + Command_Line: process.command_line.text + Commandline: process.command_line.text + commandline: process.command_line.text + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + ComputerName: winlog.ComputerName + CurrentDirectory: process.working_directory.text + Description: winlog.event_data.Description + DestinationHostname: + - destination.domain + - dns.question.name + - dns.question.subdomain + DestinationIp: destination.address + dst_ip: destination.address + DestinationPort: destination.port + dst_port: destination.port + DestinationPortName: network.protocol + Details: winlog.event_data.Details + EngineVersion: winlog.event_data.EngineVersion + EventType: winlog.event_data.EventType + FailureCode: winlog.event_data.FailureCode + FileName: file.path.text + GrantedAccess: winlog.event_data.GrantedAccess + GroupSid: + - group.id + - winlog.event_data.GroupSid + Hashes: winlog.event_data.Hashes + file_hash: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + HostVersion: winlog.event_data.HostVersion + Image: process.executable.text + ImageLoaded: dll.path + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + IpAddress: source.address + ClientAddress: + - winlog.event_data.ClientAddress + - source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + ParentCommandLine: process.parent.command_line.text + ParentProcessName: process.parent.name.text + ParentImage: process.parent.executable.text + Path: winlog.event_data.Path + PipeName: file.name + ProcessCommandLine: winlog.event_data.ProcessCommandLine + ProcessName: process.executable.text + Properties: winlog.event_data.Properties + RuleName: winlog.event_data.RuleName + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + Signature: winlog.event_data.Signature + Signed: winlog.event_data.Signed + SourceHostname: source.domain + SourceImage: process.executable.text + SourceIp: source.address + src_ip: source.address + SourcePort: source.port + src_port: source.port + StartModule: winlog.event_data.StartModule + SubStatus: winlog.event_data.SubStatus + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + TargetDomainName: winlog.event_data.TargetDomainName + TargetFilename: file.path.text + TargetImage: winlog.event_data.TargetImage + TargetObject: winlog.event_data.TargetObject + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + TargetUserSid: winlog.event_data.TargetUserSid + QueryName: dns.question.name + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + User: user.name + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + QueryResults: dns.answers + OriginalFileName: winlog.event_data.OriginalFileName + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + Initiated: winlog.event_data.Initiated + NewUACList: winlog.event_data.NewUACList + UserAccountControl: winlog.event_data.UserAccountControl + NewUacValue: winlog.event_data.NewUacValue + OldUacValue: winlog.event_data.OldUacValue + AccountExpires: winlog.event_data.AccountExpires + DisplayName: winlog.event_data.DisplayName + DnsHostName: winlog.event_data.DnsHostName + HomeDirectory: winlog.event_data.HomeDirectory + HomePath: winlog.event_data.HomePath + LogonHours: winlog.event_data.LogonHours + PasswordLastSet: winlog.event_data.PasswordLastSet + PrimaryGroupId: winlog.event_data.PrimaryGroupId + PrivilegeList: winlog.event_data.PrivilegeList + ProfilePath: winlog.event_data.ProfilePath + SamAccountName: winlog.event_data.SamAccountName + ScriptPath: winlog.event_data.ScriptPath + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + SidHistory: winlog.event_data.SidHistory + UserParameters: winlog.event_data.UserParameters + UserPrincipalName: winlog.event_data.UserPrincipalName + UserWorkstations: winlog.event_data.UserWorkstations + RelativeTargetName: winlog.event_data.RelativeTargetName + NotificationPackageName: winlog.event_data.NotificationPackageName + SecurityPackageName: winlog.event_data.SecurityPackageName + HostApplication: process.command_line.text + TaskName: winlog.event_data.TaskName + TaskContent: winlog.event_data.TaskContent + ObjectServer: winlog.event_data.ObjectServer + NewSd: winlog.event_data.NewSd + OldSd: winlog.event_data.OldSd + TestSigning: winlog.event_data.TestSigning + AdvancedOptions: winlog.event_data.AdvancedOptions + ConfigAccessPolicy: winlog.event_data.ConfigAccessPolicy + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + FlightSigning: winlog.event_data.FlightSigning + HypervisorDebug: winlog.event_data.HypervisorDebug + HypervisorLaunchType: winlog.event_data.HypervisorLaunchType + HypervisorLoadOptions: winlog.event_data.HypervisorLoadOptions + KernelDebug: winlog.event_data.KernelDebug + LoadOptions: winlog.event_data.LoadOptions + RemoteEventLogging: winlog.event_data.RemoteEventLogging + ExceptionCode: winlog.event_data.ExceptionCode + CertSerialNumber: winlog.event_data.CertSerialNumber + CertThumbprint: winlog.event_data.CertThumbprint + CertIssuerName: winlog.event_data.CertIssuerName + TicketOptionsDescription: winlog.event_data.TicketOptionsDescription + keywords: winlog.keywords + StartAddress: winlog.event_data.StartAddress + ServiceSid: winlog.event_data.ServiceSid + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ProcessId: winlog.event_data.ProcessId + ProcessID: winlog.event_data.ProcessID + SubjectLogonId: winlog.event_data.SubjectLogonId + ElevatedToken: winlog.event_data.ElevatedToken + PublishURLs: winlog.event_data.PublishURLs + VMUserAuthenticationEvent: horizon.user_authentication_event + VMUserAuthenticationUser: horizon.user_authentication_user + VMUserAuthenticationSourceIp: horizon.user_authentication_source_ip + VMUserAuthenticationTimeStamp: horizon.user_authentication_time_stamp + VMDesktopSessionStartEvent: horizon.desktop_session_start_event + VMDesktopSessionStartUser: horizon.desktop_session_start_user + VMDesktopSessionStartDesktopID: horizon.desktop_session_start_desktop_id + VMDesktopSessionStartTimeStamp: horizon.desktop_session_time_stamp + VMApplicationLaunchEvent: horizon.application_launch_event + VMApplicationLaunchUser: horizon.application_launch_user + VMApplicationLaunchAppId: horizon.application_launch_app_id + VMApplicationLaunchAppName: horizon.application_launch_app_name + VMApplicationLaunchTimeStamp: horizon.application_launch_time_stamp + VMConnectionServerStatusEvent: horizon.connection_server_status_event + VMConnectionServerStatusServer: horizon.connection_server_status_server + VMConnectionServerStatus: horizon.connection_Server_status + VMConnectionServerStatusTimeStamp: horizon.connection_server_status_time_stamp + VMVirtualDesktopPoolManagmentEvent: horizon.virtual_desktop_pool_managment_event + VMVirtualDesktopPoolManagmentPoolId: horizon.virtual_desktop_pool_managment_pool_id + VMVirtualDesktopPoolManagmentPoolName: horizon.virtual_desktop_pool_managment_pool_name + VMVirtualDesktopPoolManagmentTimeStamp: horizon.virtual_desktop_pool_managment_time_stamp + VMLoadBalancingEvent: horizon.load_balancing_event + VMLoadBalancingStatus: horizon.load_balancing_status + VMLoadBalancingAlgorithm: horizon.load_balancing_algorithm + VMLoadBalancingTimeStamp: horizon.load_balancing_time_stamp + VMBlastProtocolEvent: horizon.blast_protocol_event + VMBlastProtocolUser: horizon.blast_protocol_user + VMBlastProtocolProtocolVersion: horizon.blast_protocol_protocol_version + VMBlastProtocolTimeStamp: horizon.blast_protocol_time_stamp + VMSecurityEventName: horizon.security_event_name + VMSecurityEventUser: horizon.security_event_user + VMSecurityEventAlertType: horizon.security_event_alert_type + VMSecurityEventSourceIp: horizon.security_event_source_ip + VMSecurityEventTimeStamp: horizon.security_event_time_stamp + VMLicensingInformationEvent: horizon.licensing_information_event + VMLicensingInformationLicenseType: horizon.licensing_information_license_type + VMLicensingInformationExpiryDate: horizon.licensing_information_expiry_date + VMLicensingInformationTimeStamp: horizon.licensing_information_time_stamp + VMConnectionBrokeringEvent: horizon.connection_brokering_event + VMConnectionBrokeringUser: horizon.connection_brokering_user + VMConnectionBrokeringDesktopId: horizon.connection_brokering_desktop_id + VMConnectionBrokeringStatus: horizon.connection_brokering_status + VMConnectionBrokeringTimeStamp: horizon.connection_brokering_time_stamp + DatastoreName: vsphere.datastore_name + FilesystemType: vsphere.datastore_fstype + DatastoreBytes: vsphere.datastore_capacity_free_bytes + DatastoreBytesUsed: vsphere.datastore_capacity_used_pct + HostName: vsphere.host_name + UsedCPUmhz: vsphere.host_cpu_free_mhz + UsedMemoryBites: vsphere.host_memory_total_bytes + FreeMemoryBites: vsphere.host_memory_free_bytes + VMHostID: vsphere.virtualmachine_host_id + VMHostName: + - vsphere.virtualmachine_host_hostname + - esxi.vmhost_name + VMName: + - vsphere.virtualmachine_name + - esxi.vmname + VMOperatingSystem: vsphere.virtualmachine_os + VMUsedCPU: vsphere.virtualmachine_cpu_used_mhz + VMTotalCPU: vsphere.virtualmachine_cpu_free_mhz + VMMemoryGuestUsed: vsphere.virtualmachine_memory_used_guest_bytes + VMUMemoryHostUsed: vsphere.virtualmachine_memory_used_host_bytes + VMTotalMemoryGuestBytes: vsphere.virtualmachine_memory_total_guest_bytes + VMMemoryGuestFree: vsphere.virtualmachine_memory_free_guest_bytes + VMCustomFields: vsphere.virtualmachine_custom_fields + VMNetworkNames: vsphere.virtualmachine_network_names + VMLogicalSwitchingEvent: nsxv.vmlogical_switching_event + VMLogicalSwitchingEventID: nsxv.vmlogical_switching_event_id + VMLogicalSwitchingName: nsxv.vmlogical_switching_name + VMDistributedFirewallEvent: nsxv.distributed_firewall_event_type + VMDistributedFirewallRuleID: nsxv.distributed_firewall_rule_id + VMDistributedFirewallAction: nsxv.distributed_firewall_action + VMDistributedFirewallSourceIp: nsxv.distributed_firewall_source_ip + VMDistributedFirewallDestinationIp: nsxv.distributed_firewall_destination_ip + VMSecurityGroupEventType: nsxv.security_group_event_type + VMSecurityGroupId: nsxv.security_group_id + VMSecurityGroupName: nsxv.security_group_name + VMEdgeServicesGatewayEventType: nsxv.security_edge_services_gateway_event_type + VMEdgeServicesGatewayESGID: nsxv.security_edge_services_gateway_esgid + VMEdgeServicesGatewayStatus: nsxv.security_edge_services_gateway_status + VMLoadBalancingEventType: nsxv.load_balancing_event_type + VMLoadBalancingId: nsxv.load_balancing_id + VMLoadBalancingVirtualServer: nsxv.load_balancing_virtual_server + VMNSXManagerEventType: nsxv.nsx_manager_event_type + VMNSXManagerEventDescription: nsxv.nsx_manager_event_description + VMEdgeFirewallEventType: nsxv.edge_firewall_event_type + VMEdgeFirewallSourceIP: nsxv.edge_firewall_source_ip + VMEdgeFirewallDestinationIP: nsxv.edge_firewall_destination_ip + VMEdgeFirewallRuleID: nsxv.edge_firewall_rule_id + VMSSLVPNEventType: nsxv.ssl_vpn_event_type + VMSSLVPNUserName: nsxv.ssl_vpn_user_name + VMSSLVPNSourceIp: nsxv.ssl_vpn_source_ip + VMNSXControllerEventType: nsxv.nsx_controller_event_type + VMNSXControllerID: nsxv.nsx_controller_id + VMNSXControllerStatus: nsxv.nsx_controller_status + VMLogicalRoutingEventType: nsxv.logical_routing_event_type + VMLogicalRoutingRouterID: nsxv.logical_routing_router_id + VMLogicalRoutingRouterName: nsxv.logical_routing_router_name diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/cyberark.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/cyberark.yml new file mode 100644 index 00000000..590ddd86 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/cyberark.yml @@ -0,0 +1,303 @@ +platform: ElasticSearch ES|QL +source: cyberark +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + Issuer: source.user.name + Hostname: agent.hostname + Action: event.action + ProviderName: + - winlog.event_data.ProviderName + - winlog.provider_name + dns_query_name: dns.question.name + EventID: winlog.event_id + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AttributeValue: winlog.event_data.AttributeValue + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + CallTrace: winlog.event_data.CallTrace + Channel: winlog.channel + CommandLine: process.command_line.text + Command_Line: process.command_line.text + Commandline: process.command_line.text + commandline: process.command_line.text + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + ComputerName: winlog.ComputerName + CurrentDirectory: process.working_directory.text + Description: winlog.event_data.Description + DestinationHostname: + - destination.domain + - dns.question.name + - dns.question.subdomain + DestinationIp: destination.address + dst_ip: destination.address + DestinationPort: destination.port + dst_port: destination.port + DestinationPortName: network.protocol + Details: winlog.event_data.Details + EngineVersion: winlog.event_data.EngineVersion + EventType: winlog.event_data.EventType + FailureCode: winlog.event_data.FailureCode + FileName: file.path.text + GrantedAccess: winlog.event_data.GrantedAccess + GroupName: + - winlog.event_data.GroupName + - group.name + GroupSid: + - group.id + - winlog.event_data.GroupSid + Hashes: winlog.event_data.Hashes + file_hash: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + HostVersion: winlog.event_data.HostVersion + Image: process.executable.text + ImageLoaded: dll.path + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + IpAddress: source.address + ClientAddress: + - winlog.event_data.ClientAddress + - source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + ParentCommandLine: process.parent.command_line.text + ParentProcessName: process.parent.name.text + ParentImage: process.parent.executable.text + Path: winlog.event_data.Path + PipeName: file.name + ProcessCommandLine: winlog.event_data.ProcessCommandLine + ProcessName: process.executable.text + Properties: winlog.event_data.Properties + RuleName: winlog.event_data.RuleName + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + Signature: winlog.event_data.Signature + Signed: winlog.event_data.Signed + Source: winlog.event_data.Source + SourceHostname: source.domain + SourceImage: process.executable.text + SourceIp: source.address + src_ip: source.address + SourcePort: source.port + src_port: source.port + StartModule: winlog.event_data.StartModule + Status: winlog.event_data.Status + SubStatus: winlog.event_data.SubStatus + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + TargetDomainName: winlog.event_data.TargetDomainName + TargetFilename: file.path.text + TargetImage: winlog.event_data.TargetImage + TargetObject: winlog.event_data.TargetObject + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + TargetUserSid: winlog.event_data.TargetUserSid + QueryName: dns.question.name + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + User: user.name + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + QueryResults: dns.answers + OriginalFileName: winlog.event_data.OriginalFileName + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + Initiated: winlog.event_data.Initiated + NewUACList: winlog.event_data.NewUACList + UserAccountControl: winlog.event_data.UserAccountControl + NewUacValue: winlog.event_data.NewUacValue + OldUacValue: winlog.event_data.OldUacValue + AccountExpires: winlog.event_data.AccountExpires + DisplayName: winlog.event_data.DisplayName + DnsHostName: winlog.event_data.DnsHostName + HomeDirectory: winlog.event_data.HomeDirectory + HomePath: winlog.event_data.HomePath + LogonHours: winlog.event_data.LogonHours + PasswordLastSet: winlog.event_data.PasswordLastSet + PrimaryGroupId: winlog.event_data.PrimaryGroupId + PrivilegeList: winlog.event_data.PrivilegeList + ProfilePath: winlog.event_data.ProfilePath + SamAccountName: winlog.event_data.SamAccountName + ScriptPath: winlog.event_data.ScriptPath + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + SidHistory: winlog.event_data.SidHistory + UserParameters: winlog.event_data.UserParameters + UserPrincipalName: winlog.event_data.UserPrincipalName + UserWorkstations: winlog.event_data.UserWorkstations + RelativeTargetName: winlog.event_data.RelativeTargetName + NotificationPackageName: winlog.event_data.NotificationPackageName + SecurityPackageName: winlog.event_data.SecurityPackageName + HostApplication: process.command_line.text + TaskName: winlog.event_data.TaskName + TaskContent: winlog.event_data.TaskContent + ObjectServer: winlog.event_data.ObjectServer + NewSd: winlog.event_data.NewSd + OldSd: winlog.event_data.OldSd + TestSigning: winlog.event_data.TestSigning + AdvancedOptions: winlog.event_data.AdvancedOptions + ConfigAccessPolicy: winlog.event_data.ConfigAccessPolicy + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + FlightSigning: winlog.event_data.FlightSigning + HypervisorDebug: winlog.event_data.HypervisorDebug + HypervisorLaunchType: winlog.event_data.HypervisorLaunchType + HypervisorLoadOptions: winlog.event_data.HypervisorLoadOptions + KernelDebug: winlog.event_data.KernelDebug + LoadOptions: winlog.event_data.LoadOptions + RemoteEventLogging: winlog.event_data.RemoteEventLogging + ExceptionCode: winlog.event_data.ExceptionCode + CertSerialNumber: winlog.event_data.CertSerialNumber + CertThumbprint: winlog.event_data.CertThumbprint + CertIssuerName: winlog.event_data.CertIssuerName + TicketOptionsDescription: winlog.event_data.TicketOptionsDescription + keywords: winlog.keywords + StartAddress: winlog.event_data.StartAddress + ServiceSid: winlog.event_data.ServiceSid + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ProcessId: winlog.event_data.ProcessId + ProcessID: winlog.event_data.ProcessID + SubjectLogonId: winlog.event_data.SubjectLogonId + ElevatedToken: winlog.event_data.ElevatedToken + PublishURLs: winlog.event_data.PublishURLs + VMUserAuthenticationEvent: horizon.user_authentication_event + VMUserAuthenticationUser: horizon.user_authentication_user + VMUserAuthenticationSourceIp: horizon.user_authentication_source_ip + VMUserAuthenticationTimeStamp: horizon.user_authentication_time_stamp + VMDesktopSessionStartEvent: horizon.desktop_session_start_event + VMDesktopSessionStartUser: horizon.desktop_session_start_user + VMDesktopSessionStartDesktopID: horizon.desktop_session_start_desktop_id + VMDesktopSessionStartTimeStamp: horizon.desktop_session_time_stamp + VMApplicationLaunchEvent: horizon.application_launch_event + VMApplicationLaunchUser: horizon.application_launch_user + VMApplicationLaunchAppId: horizon.application_launch_app_id + VMApplicationLaunchAppName: horizon.application_launch_app_name + VMApplicationLaunchTimeStamp: horizon.application_launch_time_stamp + VMConnectionServerStatusEvent: horizon.connection_server_status_event + VMConnectionServerStatusServer: horizon.connection_server_status_server + VMConnectionServerStatus: horizon.connection_Server_status + VMConnectionServerStatusTimeStamp: horizon.connection_server_status_time_stamp + VMVirtualDesktopPoolManagmentEvent: horizon.virtual_desktop_pool_managment_event + VMVirtualDesktopPoolManagmentPoolId: horizon.virtual_desktop_pool_managment_pool_id + VMVirtualDesktopPoolManagmentPoolName: horizon.virtual_desktop_pool_managment_pool_name + VMVirtualDesktopPoolManagmentTimeStamp: horizon.virtual_desktop_pool_managment_time_stamp + VMLoadBalancingEvent: horizon.load_balancing_event + VMLoadBalancingStatus: horizon.load_balancing_status + VMLoadBalancingAlgorithm: horizon.load_balancing_algorithm + VMLoadBalancingTimeStamp: horizon.load_balancing_time_stamp + VMBlastProtocolEvent: horizon.blast_protocol_event + VMBlastProtocolUser: horizon.blast_protocol_user + VMBlastProtocolProtocolVersion: horizon.blast_protocol_protocol_version + VMBlastProtocolTimeStamp: horizon.blast_protocol_time_stamp + VMSecurityEventName: horizon.security_event_name + VMSecurityEventUser: horizon.security_event_user + VMSecurityEventAlertType: horizon.security_event_alert_type + VMSecurityEventSourceIp: horizon.security_event_source_ip + VMSecurityEventTimeStamp: horizon.security_event_time_stamp + VMLicensingInformationEvent: horizon.licensing_information_event + VMLicensingInformationLicenseType: horizon.licensing_information_license_type + VMLicensingInformationExpiryDate: horizon.licensing_information_expiry_date + VMLicensingInformationTimeStamp: horizon.licensing_information_time_stamp + VMConnectionBrokeringEvent: horizon.connection_brokering_event + VMConnectionBrokeringUser: horizon.connection_brokering_user + VMConnectionBrokeringDesktopId: horizon.connection_brokering_desktop_id + VMConnectionBrokeringStatus: horizon.connection_brokering_status + VMConnectionBrokeringTimeStamp: horizon.connection_brokering_time_stamp + DatastoreName: vsphere.datastore_name + FilesystemType: vsphere.datastore_fstype + DatastoreBytes: vsphere.datastore_capacity_free_bytes + DatastoreBytesUsed: vsphere.datastore_capacity_used_pct + HostName: vsphere.host_name + UsedCPUmhz: vsphere.host_cpu_free_mhz + UsedMemoryBites: vsphere.host_memory_total_bytes + FreeMemoryBites: vsphere.host_memory_free_bytes + VMHostID: vsphere.virtualmachine_host_id + VMHostName: + - vsphere.virtualmachine_host_hostname + - esxi.vmhost_name + VMName: + - vsphere.virtualmachine_name + - esxi.vmname + VMOperatingSystem: vsphere.virtualmachine_os + VMUsedCPU: vsphere.virtualmachine_cpu_used_mhz + VMTotalCPU: vsphere.virtualmachine_cpu_free_mhz + VMMemoryGuestUsed: vsphere.virtualmachine_memory_used_guest_bytes + VMUMemoryHostUsed: vsphere.virtualmachine_memory_used_host_bytes + VMTotalMemoryGuestBytes: vsphere.virtualmachine_memory_total_guest_bytes + VMMemoryGuestFree: vsphere.virtualmachine_memory_free_guest_bytes + VMCustomFields: vsphere.virtualmachine_custom_fields + VMNetworkNames: vsphere.virtualmachine_network_names + VMLogicalSwitchingEvent: nsxv.vmlogical_switching_event + VMLogicalSwitchingEventID: nsxv.vmlogical_switching_event_id + VMLogicalSwitchingName: nsxv.vmlogical_switching_name + VMDistributedFirewallEvent: nsxv.distributed_firewall_event_type + VMDistributedFirewallRuleID: nsxv.distributed_firewall_rule_id + VMDistributedFirewallAction: nsxv.distributed_firewall_action + VMDistributedFirewallSourceIp: nsxv.distributed_firewall_source_ip + VMDistributedFirewallDestinationIp: nsxv.distributed_firewall_destination_ip + VMSecurityGroupEventType: nsxv.security_group_event_type + VMSecurityGroupId: nsxv.security_group_id + VMSecurityGroupName: nsxv.security_group_name + VMEdgeServicesGatewayEventType: nsxv.security_edge_services_gateway_event_type + VMEdgeServicesGatewayESGID: nsxv.security_edge_services_gateway_esgid + VMEdgeServicesGatewayStatus: nsxv.security_edge_services_gateway_status + VMLoadBalancingEventType: nsxv.load_balancing_event_type + VMLoadBalancingId: nsxv.load_balancing_id + VMLoadBalancingVirtualServer: nsxv.load_balancing_virtual_server + VMNSXManagerEventType: nsxv.nsx_manager_event_type + VMNSXManagerEventDescription: nsxv.nsx_manager_event_description + VMEdgeFirewallEventType: nsxv.edge_firewall_event_type + VMEdgeFirewallSourceIP: nsxv.edge_firewall_source_ip + VMEdgeFirewallDestinationIP: nsxv.edge_firewall_destination_ip + VMEdgeFirewallRuleID: nsxv.edge_firewall_rule_id + VMSSLVPNEventType: nsxv.ssl_vpn_event_type + VMSSLVPNUserName: nsxv.ssl_vpn_user_name + VMSSLVPNSourceIp: nsxv.ssl_vpn_source_ip + VMNSXControllerEventType: nsxv.nsx_controller_event_type + VMNSXControllerID: nsxv.nsx_controller_id + VMNSXControllerStatus: nsxv.nsx_controller_status + VMLogicalRoutingEventType: nsxv.logical_routing_event_type + VMLogicalRoutingRouterID: nsxv.logical_routing_router_id + VMLogicalRoutingRouterName: nsxv.logical_routing_router_name diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/default.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/default.yml new file mode 100644 index 00000000..e8be2b1e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/default.yml @@ -0,0 +1,8 @@ +platform: ElasticSearch ES|QL +source: default + + +default_log_source: + index: "logs-*" + + diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_image_load.yml new file mode 100644 index 00000000..b1734d59 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_image_load.yml @@ -0,0 +1,302 @@ +platform: ElasticSearch ES|QL +source: windows_image_load +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + ImageLoaded: + - dll.path + - file.path + ProviderName: + - winlog.event_data.ProviderName + - winlog.provider_name + dns_query_name: dns.question.name + EventID: winlog.event_id + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AttributeValue: winlog.event_data.AttributeValue + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + CallTrace: winlog.event_data.CallTrace + Channel: winlog.channel + CommandLine: process.command_line.text + Command_Line: process.command_line.text + Commandline: process.command_line.text + commandline: process.command_line.text + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + ComputerName: winlog.ComputerName + CurrentDirectory: process.working_directory.text + Description: winlog.event_data.Description + DestinationHostname: + - destination.domain + - dns.question.name + - dns.question.subdomain + DestinationIp: destination.address + dst_ip: destination.address + DestinationPort: destination.port + dst_port: destination.port + DestinationPortName: network.protocol + Details: winlog.event_data.Details + EngineVersion: winlog.event_data.EngineVersion + EventType: winlog.event_data.EventType + FailureCode: winlog.event_data.FailureCode + FileName: file.path.text + GrantedAccess: winlog.event_data.GrantedAccess + GroupName: + - winlog.event_data.GroupName + - group.name + GroupSid: + - group.id + - winlog.event_data.GroupSid + Hashes: winlog.event_data.Hashes + file_hash: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + HostVersion: winlog.event_data.HostVersion + Image: process.executable.text + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + IpAddress: source.address + ClientAddress: + - winlog.event_data.ClientAddress + - source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + ParentCommandLine: process.parent.command_line.text + ParentProcessName: process.parent.name.text + ParentImage: process.parent.executable.text + Path: winlog.event_data.Path + PipeName: file.name + ProcessCommandLine: winlog.event_data.ProcessCommandLine + ProcessName: process.executable.text + Properties: winlog.event_data.Properties + RuleName: winlog.event_data.RuleName + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + Signature: winlog.event_data.Signature + Signed: winlog.event_data.Signed + Source: winlog.event_data.Source + SourceHostname: source.domain + SourceImage: process.executable.text + SourceIp: source.address + src_ip: source.address + SourcePort: source.port + src_port: source.port + StartModule: winlog.event_data.StartModule + Status: winlog.event_data.Status + SubStatus: winlog.event_data.SubStatus + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + TargetDomainName: winlog.event_data.TargetDomainName + TargetFilename: file.path.text + TargetImage: winlog.event_data.TargetImage + TargetObject: winlog.event_data.TargetObject + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + TargetUserSid: winlog.event_data.TargetUserSid + QueryName: dns.question.name + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + User: user.name + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + QueryResults: dns.answers + OriginalFileName: winlog.event_data.OriginalFileName + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + Initiated: winlog.event_data.Initiated + NewUACList: winlog.event_data.NewUACList + UserAccountControl: winlog.event_data.UserAccountControl + NewUacValue: winlog.event_data.NewUacValue + OldUacValue: winlog.event_data.OldUacValue + AccountExpires: winlog.event_data.AccountExpires + DisplayName: winlog.event_data.DisplayName + DnsHostName: winlog.event_data.DnsHostName + HomeDirectory: winlog.event_data.HomeDirectory + HomePath: winlog.event_data.HomePath + LogonHours: winlog.event_data.LogonHours + PasswordLastSet: winlog.event_data.PasswordLastSet + PrimaryGroupId: winlog.event_data.PrimaryGroupId + PrivilegeList: winlog.event_data.PrivilegeList + ProfilePath: winlog.event_data.ProfilePath + SamAccountName: winlog.event_data.SamAccountName + ScriptPath: winlog.event_data.ScriptPath + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + SidHistory: winlog.event_data.SidHistory + UserParameters: winlog.event_data.UserParameters + UserPrincipalName: winlog.event_data.UserPrincipalName + UserWorkstations: winlog.event_data.UserWorkstations + RelativeTargetName: winlog.event_data.RelativeTargetName + NotificationPackageName: winlog.event_data.NotificationPackageName + SecurityPackageName: winlog.event_data.SecurityPackageName + HostApplication: process.command_line.text + TaskName: winlog.event_data.TaskName + TaskContent: winlog.event_data.TaskContent + ObjectServer: winlog.event_data.ObjectServer + NewSd: winlog.event_data.NewSd + OldSd: winlog.event_data.OldSd + TestSigning: winlog.event_data.TestSigning + AdvancedOptions: winlog.event_data.AdvancedOptions + ConfigAccessPolicy: winlog.event_data.ConfigAccessPolicy + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + FlightSigning: winlog.event_data.FlightSigning + HypervisorDebug: winlog.event_data.HypervisorDebug + HypervisorLaunchType: winlog.event_data.HypervisorLaunchType + HypervisorLoadOptions: winlog.event_data.HypervisorLoadOptions + KernelDebug: winlog.event_data.KernelDebug + LoadOptions: winlog.event_data.LoadOptions + RemoteEventLogging: winlog.event_data.RemoteEventLogging + ExceptionCode: winlog.event_data.ExceptionCode + CertSerialNumber: winlog.event_data.CertSerialNumber + CertThumbprint: winlog.event_data.CertThumbprint + CertIssuerName: winlog.event_data.CertIssuerName + TicketOptionsDescription: winlog.event_data.TicketOptionsDescription + keywords: winlog.keywords + StartAddress: winlog.event_data.StartAddress + ServiceSid: winlog.event_data.ServiceSid + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ProcessId: winlog.event_data.ProcessId + ProcessID: winlog.event_data.ProcessID + SubjectLogonId: winlog.event_data.SubjectLogonId + ElevatedToken: winlog.event_data.ElevatedToken + PublishURLs: winlog.event_data.PublishURLs + VMUserAuthenticationEvent: horizon.user_authentication_event + VMUserAuthenticationUser: horizon.user_authentication_user + VMUserAuthenticationSourceIp: horizon.user_authentication_source_ip + VMUserAuthenticationTimeStamp: horizon.user_authentication_time_stamp + VMDesktopSessionStartEvent: horizon.desktop_session_start_event + VMDesktopSessionStartUser: horizon.desktop_session_start_user + VMDesktopSessionStartDesktopID: horizon.desktop_session_start_desktop_id + VMDesktopSessionStartTimeStamp: horizon.desktop_session_time_stamp + VMApplicationLaunchEvent: horizon.application_launch_event + VMApplicationLaunchUser: horizon.application_launch_user + VMApplicationLaunchAppId: horizon.application_launch_app_id + VMApplicationLaunchAppName: horizon.application_launch_app_name + VMApplicationLaunchTimeStamp: horizon.application_launch_time_stamp + VMConnectionServerStatusEvent: horizon.connection_server_status_event + VMConnectionServerStatusServer: horizon.connection_server_status_server + VMConnectionServerStatus: horizon.connection_Server_status + VMConnectionServerStatusTimeStamp: horizon.connection_server_status_time_stamp + VMVirtualDesktopPoolManagmentEvent: horizon.virtual_desktop_pool_managment_event + VMVirtualDesktopPoolManagmentPoolId: horizon.virtual_desktop_pool_managment_pool_id + VMVirtualDesktopPoolManagmentPoolName: horizon.virtual_desktop_pool_managment_pool_name + VMVirtualDesktopPoolManagmentTimeStamp: horizon.virtual_desktop_pool_managment_time_stamp + VMLoadBalancingEvent: horizon.load_balancing_event + VMLoadBalancingStatus: horizon.load_balancing_status + VMLoadBalancingAlgorithm: horizon.load_balancing_algorithm + VMLoadBalancingTimeStamp: horizon.load_balancing_time_stamp + VMBlastProtocolEvent: horizon.blast_protocol_event + VMBlastProtocolUser: horizon.blast_protocol_user + VMBlastProtocolProtocolVersion: horizon.blast_protocol_protocol_version + VMBlastProtocolTimeStamp: horizon.blast_protocol_time_stamp + VMSecurityEventName: horizon.security_event_name + VMSecurityEventUser: horizon.security_event_user + VMSecurityEventAlertType: horizon.security_event_alert_type + VMSecurityEventSourceIp: horizon.security_event_source_ip + VMSecurityEventTimeStamp: horizon.security_event_time_stamp + VMLicensingInformationEvent: horizon.licensing_information_event + VMLicensingInformationLicenseType: horizon.licensing_information_license_type + VMLicensingInformationExpiryDate: horizon.licensing_information_expiry_date + VMLicensingInformationTimeStamp: horizon.licensing_information_time_stamp + VMConnectionBrokeringEvent: horizon.connection_brokering_event + VMConnectionBrokeringUser: horizon.connection_brokering_user + VMConnectionBrokeringDesktopId: horizon.connection_brokering_desktop_id + VMConnectionBrokeringStatus: horizon.connection_brokering_status + VMConnectionBrokeringTimeStamp: horizon.connection_brokering_time_stamp + DatastoreName: vsphere.datastore_name + FilesystemType: vsphere.datastore_fstype + DatastoreBytes: vsphere.datastore_capacity_free_bytes + DatastoreBytesUsed: vsphere.datastore_capacity_used_pct + HostName: vsphere.host_name + UsedCPUmhz: vsphere.host_cpu_free_mhz + UsedMemoryBites: vsphere.host_memory_total_bytes + FreeMemoryBites: vsphere.host_memory_free_bytes + VMHostID: vsphere.virtualmachine_host_id + VMHostName: + - vsphere.virtualmachine_host_hostname + - esxi.vmhost_name + VMName: + - vsphere.virtualmachine_name + - esxi.vmname + VMOperatingSystem: vsphere.virtualmachine_os + VMUsedCPU: vsphere.virtualmachine_cpu_used_mhz + VMTotalCPU: vsphere.virtualmachine_cpu_free_mhz + VMMemoryGuestUsed: vsphere.virtualmachine_memory_used_guest_bytes + VMUMemoryHostUsed: vsphere.virtualmachine_memory_used_host_bytes + VMTotalMemoryGuestBytes: vsphere.virtualmachine_memory_total_guest_bytes + VMMemoryGuestFree: vsphere.virtualmachine_memory_free_guest_bytes + VMCustomFields: vsphere.virtualmachine_custom_fields + VMNetworkNames: vsphere.virtualmachine_network_names + VMLogicalSwitchingEvent: nsxv.vmlogical_switching_event + VMLogicalSwitchingEventID: nsxv.vmlogical_switching_event_id + VMLogicalSwitchingName: nsxv.vmlogical_switching_name + VMDistributedFirewallEvent: nsxv.distributed_firewall_event_type + VMDistributedFirewallRuleID: nsxv.distributed_firewall_rule_id + VMDistributedFirewallAction: nsxv.distributed_firewall_action + VMDistributedFirewallSourceIp: nsxv.distributed_firewall_source_ip + VMDistributedFirewallDestinationIp: nsxv.distributed_firewall_destination_ip + VMSecurityGroupEventType: nsxv.security_group_event_type + VMSecurityGroupId: nsxv.security_group_id + VMSecurityGroupName: nsxv.security_group_name + VMEdgeServicesGatewayEventType: nsxv.security_edge_services_gateway_event_type + VMEdgeServicesGatewayESGID: nsxv.security_edge_services_gateway_esgid + VMEdgeServicesGatewayStatus: nsxv.security_edge_services_gateway_status + VMLoadBalancingEventType: nsxv.load_balancing_event_type + VMLoadBalancingId: nsxv.load_balancing_id + VMLoadBalancingVirtualServer: nsxv.load_balancing_virtual_server + VMNSXManagerEventType: nsxv.nsx_manager_event_type + VMNSXManagerEventDescription: nsxv.nsx_manager_event_description + VMEdgeFirewallEventType: nsxv.edge_firewall_event_type + VMEdgeFirewallSourceIP: nsxv.edge_firewall_source_ip + VMEdgeFirewallDestinationIP: nsxv.edge_firewall_destination_ip + VMEdgeFirewallRuleID: nsxv.edge_firewall_rule_id + VMSSLVPNEventType: nsxv.ssl_vpn_event_type + VMSSLVPNUserName: nsxv.ssl_vpn_user_name + VMSSLVPNSourceIp: nsxv.ssl_vpn_source_ip + VMNSXControllerEventType: nsxv.nsx_controller_event_type + VMNSXControllerID: nsxv.nsx_controller_id + VMNSXControllerStatus: nsxv.nsx_controller_status + VMLogicalRoutingEventType: nsxv.logical_routing_event_type + VMLogicalRoutingRouterID: nsxv.logical_routing_router_id + VMLogicalRoutingRouterName: nsxv.logical_routing_router_name diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_powershell.yml new file mode 100644 index 00000000..dacb1357 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_powershell.yml @@ -0,0 +1,302 @@ +platform: ElasticSearch ES|QL +source: windows_powershell +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + CommandLine: + - powershell.command.value + - process.command_line.text + ProviderName: + - winlog.event_data.ProviderName + - winlog.provider_name + dns_query_name: dns.question.name + EventID: winlog.event_id + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AttributeValue: winlog.event_data.AttributeValue + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + CallTrace: winlog.event_data.CallTrace + Channel: winlog.channel + Command_Line: process.command_line.text + Commandline: process.command_line.text + commandline: process.command_line.text + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + ComputerName: winlog.ComputerName + CurrentDirectory: process.working_directory.text + Description: winlog.event_data.Description + DestinationHostname: + - destination.domain + - dns.question.name + - dns.question.subdomain + DestinationIp: destination.address + dst_ip: destination.address + DestinationPort: destination.port + dst_port: destination.port + DestinationPortName: network.protocol + Details: winlog.event_data.Details + EngineVersion: winlog.event_data.EngineVersion + EventType: winlog.event_data.EventType + FailureCode: winlog.event_data.FailureCode + FileName: file.path.text + GrantedAccess: winlog.event_data.GrantedAccess + GroupName: + - winlog.event_data.GroupName + - group.name + GroupSid: + - group.id + - winlog.event_data.GroupSid + Hashes: winlog.event_data.Hashes + file_hash: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + HostVersion: winlog.event_data.HostVersion + Image: process.executable.text + ImageLoaded: dll.path + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + IpAddress: source.address + ClientAddress: + - winlog.event_data.ClientAddress + - source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + ParentCommandLine: process.parent.command_line.text + ParentProcessName: process.parent.name.text + ParentImage: process.parent.executable.text + Path: winlog.event_data.Path + PipeName: file.name + ProcessCommandLine: winlog.event_data.ProcessCommandLine + ProcessName: process.executable.text + Properties: winlog.event_data.Properties + RuleName: winlog.event_data.RuleName + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + Signature: winlog.event_data.Signature + Signed: winlog.event_data.Signed + Source: winlog.event_data.Source + SourceHostname: source.domain + SourceImage: process.executable.text + SourceIp: source.address + src_ip: source.address + SourcePort: source.port + src_port: source.port + StartModule: winlog.event_data.StartModule + Status: winlog.event_data.Status + SubStatus: winlog.event_data.SubStatus + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + TargetDomainName: winlog.event_data.TargetDomainName + TargetFilename: file.path.text + TargetImage: winlog.event_data.TargetImage + TargetObject: winlog.event_data.TargetObject + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + TargetUserSid: winlog.event_data.TargetUserSid + QueryName: dns.question.name + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + User: user.name + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + QueryResults: dns.answers + OriginalFileName: winlog.event_data.OriginalFileName + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + Initiated: winlog.event_data.Initiated + NewUACList: winlog.event_data.NewUACList + UserAccountControl: winlog.event_data.UserAccountControl + NewUacValue: winlog.event_data.NewUacValue + OldUacValue: winlog.event_data.OldUacValue + AccountExpires: winlog.event_data.AccountExpires + DisplayName: winlog.event_data.DisplayName + DnsHostName: winlog.event_data.DnsHostName + HomeDirectory: winlog.event_data.HomeDirectory + HomePath: winlog.event_data.HomePath + LogonHours: winlog.event_data.LogonHours + PasswordLastSet: winlog.event_data.PasswordLastSet + PrimaryGroupId: winlog.event_data.PrimaryGroupId + PrivilegeList: winlog.event_data.PrivilegeList + ProfilePath: winlog.event_data.ProfilePath + SamAccountName: winlog.event_data.SamAccountName + ScriptPath: winlog.event_data.ScriptPath + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + SidHistory: winlog.event_data.SidHistory + UserParameters: winlog.event_data.UserParameters + UserPrincipalName: winlog.event_data.UserPrincipalName + UserWorkstations: winlog.event_data.UserWorkstations + RelativeTargetName: winlog.event_data.RelativeTargetName + NotificationPackageName: winlog.event_data.NotificationPackageName + SecurityPackageName: winlog.event_data.SecurityPackageName + HostApplication: process.command_line.text + TaskName: winlog.event_data.TaskName + TaskContent: winlog.event_data.TaskContent + ObjectServer: winlog.event_data.ObjectServer + NewSd: winlog.event_data.NewSd + OldSd: winlog.event_data.OldSd + TestSigning: winlog.event_data.TestSigning + AdvancedOptions: winlog.event_data.AdvancedOptions + ConfigAccessPolicy: winlog.event_data.ConfigAccessPolicy + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + FlightSigning: winlog.event_data.FlightSigning + HypervisorDebug: winlog.event_data.HypervisorDebug + HypervisorLaunchType: winlog.event_data.HypervisorLaunchType + HypervisorLoadOptions: winlog.event_data.HypervisorLoadOptions + KernelDebug: winlog.event_data.KernelDebug + LoadOptions: winlog.event_data.LoadOptions + RemoteEventLogging: winlog.event_data.RemoteEventLogging + ExceptionCode: winlog.event_data.ExceptionCode + CertSerialNumber: winlog.event_data.CertSerialNumber + CertThumbprint: winlog.event_data.CertThumbprint + CertIssuerName: winlog.event_data.CertIssuerName + TicketOptionsDescription: winlog.event_data.TicketOptionsDescription + keywords: winlog.keywords + StartAddress: winlog.event_data.StartAddress + ServiceSid: winlog.event_data.ServiceSid + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ProcessId: winlog.event_data.ProcessId + ProcessID: winlog.event_data.ProcessID + SubjectLogonId: winlog.event_data.SubjectLogonId + ElevatedToken: winlog.event_data.ElevatedToken + PublishURLs: winlog.event_data.PublishURLs + VMUserAuthenticationEvent: horizon.user_authentication_event + VMUserAuthenticationUser: horizon.user_authentication_user + VMUserAuthenticationSourceIp: horizon.user_authentication_source_ip + VMUserAuthenticationTimeStamp: horizon.user_authentication_time_stamp + VMDesktopSessionStartEvent: horizon.desktop_session_start_event + VMDesktopSessionStartUser: horizon.desktop_session_start_user + VMDesktopSessionStartDesktopID: horizon.desktop_session_start_desktop_id + VMDesktopSessionStartTimeStamp: horizon.desktop_session_time_stamp + VMApplicationLaunchEvent: horizon.application_launch_event + VMApplicationLaunchUser: horizon.application_launch_user + VMApplicationLaunchAppId: horizon.application_launch_app_id + VMApplicationLaunchAppName: horizon.application_launch_app_name + VMApplicationLaunchTimeStamp: horizon.application_launch_time_stamp + VMConnectionServerStatusEvent: horizon.connection_server_status_event + VMConnectionServerStatusServer: horizon.connection_server_status_server + VMConnectionServerStatus: horizon.connection_Server_status + VMConnectionServerStatusTimeStamp: horizon.connection_server_status_time_stamp + VMVirtualDesktopPoolManagmentEvent: horizon.virtual_desktop_pool_managment_event + VMVirtualDesktopPoolManagmentPoolId: horizon.virtual_desktop_pool_managment_pool_id + VMVirtualDesktopPoolManagmentPoolName: horizon.virtual_desktop_pool_managment_pool_name + VMVirtualDesktopPoolManagmentTimeStamp: horizon.virtual_desktop_pool_managment_time_stamp + VMLoadBalancingEvent: horizon.load_balancing_event + VMLoadBalancingStatus: horizon.load_balancing_status + VMLoadBalancingAlgorithm: horizon.load_balancing_algorithm + VMLoadBalancingTimeStamp: horizon.load_balancing_time_stamp + VMBlastProtocolEvent: horizon.blast_protocol_event + VMBlastProtocolUser: horizon.blast_protocol_user + VMBlastProtocolProtocolVersion: horizon.blast_protocol_protocol_version + VMBlastProtocolTimeStamp: horizon.blast_protocol_time_stamp + VMSecurityEventName: horizon.security_event_name + VMSecurityEventUser: horizon.security_event_user + VMSecurityEventAlertType: horizon.security_event_alert_type + VMSecurityEventSourceIp: horizon.security_event_source_ip + VMSecurityEventTimeStamp: horizon.security_event_time_stamp + VMLicensingInformationEvent: horizon.licensing_information_event + VMLicensingInformationLicenseType: horizon.licensing_information_license_type + VMLicensingInformationExpiryDate: horizon.licensing_information_expiry_date + VMLicensingInformationTimeStamp: horizon.licensing_information_time_stamp + VMConnectionBrokeringEvent: horizon.connection_brokering_event + VMConnectionBrokeringUser: horizon.connection_brokering_user + VMConnectionBrokeringDesktopId: horizon.connection_brokering_desktop_id + VMConnectionBrokeringStatus: horizon.connection_brokering_status + VMConnectionBrokeringTimeStamp: horizon.connection_brokering_time_stamp + DatastoreName: vsphere.datastore_name + FilesystemType: vsphere.datastore_fstype + DatastoreBytes: vsphere.datastore_capacity_free_bytes + DatastoreBytesUsed: vsphere.datastore_capacity_used_pct + HostName: vsphere.host_name + UsedCPUmhz: vsphere.host_cpu_free_mhz + UsedMemoryBites: vsphere.host_memory_total_bytes + FreeMemoryBites: vsphere.host_memory_free_bytes + VMHostID: vsphere.virtualmachine_host_id + VMHostName: + - vsphere.virtualmachine_host_hostname + - esxi.vmhost_name + VMName: + - vsphere.virtualmachine_name + - esxi.vmname + VMOperatingSystem: vsphere.virtualmachine_os + VMUsedCPU: vsphere.virtualmachine_cpu_used_mhz + VMTotalCPU: vsphere.virtualmachine_cpu_free_mhz + VMMemoryGuestUsed: vsphere.virtualmachine_memory_used_guest_bytes + VMUMemoryHostUsed: vsphere.virtualmachine_memory_used_host_bytes + VMTotalMemoryGuestBytes: vsphere.virtualmachine_memory_total_guest_bytes + VMMemoryGuestFree: vsphere.virtualmachine_memory_free_guest_bytes + VMCustomFields: vsphere.virtualmachine_custom_fields + VMNetworkNames: vsphere.virtualmachine_network_names + VMLogicalSwitchingEvent: nsxv.vmlogical_switching_event + VMLogicalSwitchingEventID: nsxv.vmlogical_switching_event_id + VMLogicalSwitchingName: nsxv.vmlogical_switching_name + VMDistributedFirewallEvent: nsxv.distributed_firewall_event_type + VMDistributedFirewallRuleID: nsxv.distributed_firewall_rule_id + VMDistributedFirewallAction: nsxv.distributed_firewall_action + VMDistributedFirewallSourceIp: nsxv.distributed_firewall_source_ip + VMDistributedFirewallDestinationIp: nsxv.distributed_firewall_destination_ip + VMSecurityGroupEventType: nsxv.security_group_event_type + VMSecurityGroupId: nsxv.security_group_id + VMSecurityGroupName: nsxv.security_group_name + VMEdgeServicesGatewayEventType: nsxv.security_edge_services_gateway_event_type + VMEdgeServicesGatewayESGID: nsxv.security_edge_services_gateway_esgid + VMEdgeServicesGatewayStatus: nsxv.security_edge_services_gateway_status + VMLoadBalancingEventType: nsxv.load_balancing_event_type + VMLoadBalancingId: nsxv.load_balancing_id + VMLoadBalancingVirtualServer: nsxv.load_balancing_virtual_server + VMNSXManagerEventType: nsxv.nsx_manager_event_type + VMNSXManagerEventDescription: nsxv.nsx_manager_event_description + VMEdgeFirewallEventType: nsxv.edge_firewall_event_type + VMEdgeFirewallSourceIP: nsxv.edge_firewall_source_ip + VMEdgeFirewallDestinationIP: nsxv.edge_firewall_destination_ip + VMEdgeFirewallRuleID: nsxv.edge_firewall_rule_id + VMSSLVPNEventType: nsxv.ssl_vpn_event_type + VMSSLVPNUserName: nsxv.ssl_vpn_user_name + VMSSLVPNSourceIp: nsxv.ssl_vpn_source_ip + VMNSXControllerEventType: nsxv.nsx_controller_event_type + VMNSXControllerID: nsxv.nsx_controller_id + VMNSXControllerStatus: nsxv.nsx_controller_status + VMLogicalRoutingEventType: nsxv.logical_routing_event_type + VMLogicalRoutingRouterID: nsxv.logical_routing_router_id + VMLogicalRoutingRouterName: nsxv.logical_routing_router_name diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_ps_module.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_ps_module.yml new file mode 100644 index 00000000..da8b0ae7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_ps_module.yml @@ -0,0 +1,302 @@ +platform: ElasticSearch ES|QL +source: windows_ps_module +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + CommandLine: + - powershell.command.value + - process.command_line.text + ProviderName: + - winlog.event_data.ProviderName + - winlog.provider_name + dns_query_name: dns.question.name + EventID: winlog.event_id + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AttributeValue: winlog.event_data.AttributeValue + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + CallTrace: winlog.event_data.CallTrace + Channel: winlog.channel + Command_Line: process.command_line.text + Commandline: process.command_line.text + commandline: process.command_line.text + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + ComputerName: winlog.ComputerName + CurrentDirectory: process.working_directory.text + Description: winlog.event_data.Description + DestinationHostname: + - destination.domain + - dns.question.name + - dns.question.subdomain + DestinationIp: destination.address + dst_ip: destination.address + DestinationPort: destination.port + dst_port: destination.port + DestinationPortName: network.protocol + Details: winlog.event_data.Details + EngineVersion: winlog.event_data.EngineVersion + EventType: winlog.event_data.EventType + FailureCode: winlog.event_data.FailureCode + FileName: file.path.text + GrantedAccess: winlog.event_data.GrantedAccess + GroupName: + - winlog.event_data.GroupName + - group.name + GroupSid: + - group.id + - winlog.event_data.GroupSid + Hashes: winlog.event_data.Hashes + file_hash: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + HostVersion: winlog.event_data.HostVersion + Image: process.executable.text + ImageLoaded: dll.path + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + IpAddress: source.address + ClientAddress: + - winlog.event_data.ClientAddress + - source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + ParentCommandLine: process.parent.command_line.text + ParentProcessName: process.parent.name.text + ParentImage: process.parent.executable.text + Path: winlog.event_data.Path + PipeName: file.name + ProcessCommandLine: winlog.event_data.ProcessCommandLine + ProcessName: process.executable.text + Properties: winlog.event_data.Properties + RuleName: winlog.event_data.RuleName + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + Signature: winlog.event_data.Signature + Signed: winlog.event_data.Signed + Source: winlog.event_data.Source + SourceHostname: source.domain + SourceImage: process.executable.text + SourceIp: source.address + src_ip: source.address + SourcePort: source.port + src_port: source.port + StartModule: winlog.event_data.StartModule + Status: winlog.event_data.Status + SubStatus: winlog.event_data.SubStatus + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + TargetDomainName: winlog.event_data.TargetDomainName + TargetFilename: file.path.text + TargetImage: winlog.event_data.TargetImage + TargetObject: winlog.event_data.TargetObject + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + TargetUserSid: winlog.event_data.TargetUserSid + QueryName: dns.question.name + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + User: user.name + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + QueryResults: dns.answers + OriginalFileName: winlog.event_data.OriginalFileName + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + Initiated: winlog.event_data.Initiated + NewUACList: winlog.event_data.NewUACList + UserAccountControl: winlog.event_data.UserAccountControl + NewUacValue: winlog.event_data.NewUacValue + OldUacValue: winlog.event_data.OldUacValue + AccountExpires: winlog.event_data.AccountExpires + DisplayName: winlog.event_data.DisplayName + DnsHostName: winlog.event_data.DnsHostName + HomeDirectory: winlog.event_data.HomeDirectory + HomePath: winlog.event_data.HomePath + LogonHours: winlog.event_data.LogonHours + PasswordLastSet: winlog.event_data.PasswordLastSet + PrimaryGroupId: winlog.event_data.PrimaryGroupId + PrivilegeList: winlog.event_data.PrivilegeList + ProfilePath: winlog.event_data.ProfilePath + SamAccountName: winlog.event_data.SamAccountName + ScriptPath: winlog.event_data.ScriptPath + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + SidHistory: winlog.event_data.SidHistory + UserParameters: winlog.event_data.UserParameters + UserPrincipalName: winlog.event_data.UserPrincipalName + UserWorkstations: winlog.event_data.UserWorkstations + RelativeTargetName: winlog.event_data.RelativeTargetName + NotificationPackageName: winlog.event_data.NotificationPackageName + SecurityPackageName: winlog.event_data.SecurityPackageName + HostApplication: process.command_line.text + TaskName: winlog.event_data.TaskName + TaskContent: winlog.event_data.TaskContent + ObjectServer: winlog.event_data.ObjectServer + NewSd: winlog.event_data.NewSd + OldSd: winlog.event_data.OldSd + TestSigning: winlog.event_data.TestSigning + AdvancedOptions: winlog.event_data.AdvancedOptions + ConfigAccessPolicy: winlog.event_data.ConfigAccessPolicy + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + FlightSigning: winlog.event_data.FlightSigning + HypervisorDebug: winlog.event_data.HypervisorDebug + HypervisorLaunchType: winlog.event_data.HypervisorLaunchType + HypervisorLoadOptions: winlog.event_data.HypervisorLoadOptions + KernelDebug: winlog.event_data.KernelDebug + LoadOptions: winlog.event_data.LoadOptions + RemoteEventLogging: winlog.event_data.RemoteEventLogging + ExceptionCode: winlog.event_data.ExceptionCode + CertSerialNumber: winlog.event_data.CertSerialNumber + CertThumbprint: winlog.event_data.CertThumbprint + CertIssuerName: winlog.event_data.CertIssuerName + TicketOptionsDescription: winlog.event_data.TicketOptionsDescription + keywords: winlog.keywords + StartAddress: winlog.event_data.StartAddress + ServiceSid: winlog.event_data.ServiceSid + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ProcessId: winlog.event_data.ProcessId + ProcessID: winlog.event_data.ProcessID + SubjectLogonId: winlog.event_data.SubjectLogonId + ElevatedToken: winlog.event_data.ElevatedToken + PublishURLs: winlog.event_data.PublishURLs + VMUserAuthenticationEvent: horizon.user_authentication_event + VMUserAuthenticationUser: horizon.user_authentication_user + VMUserAuthenticationSourceIp: horizon.user_authentication_source_ip + VMUserAuthenticationTimeStamp: horizon.user_authentication_time_stamp + VMDesktopSessionStartEvent: horizon.desktop_session_start_event + VMDesktopSessionStartUser: horizon.desktop_session_start_user + VMDesktopSessionStartDesktopID: horizon.desktop_session_start_desktop_id + VMDesktopSessionStartTimeStamp: horizon.desktop_session_time_stamp + VMApplicationLaunchEvent: horizon.application_launch_event + VMApplicationLaunchUser: horizon.application_launch_user + VMApplicationLaunchAppId: horizon.application_launch_app_id + VMApplicationLaunchAppName: horizon.application_launch_app_name + VMApplicationLaunchTimeStamp: horizon.application_launch_time_stamp + VMConnectionServerStatusEvent: horizon.connection_server_status_event + VMConnectionServerStatusServer: horizon.connection_server_status_server + VMConnectionServerStatus: horizon.connection_Server_status + VMConnectionServerStatusTimeStamp: horizon.connection_server_status_time_stamp + VMVirtualDesktopPoolManagmentEvent: horizon.virtual_desktop_pool_managment_event + VMVirtualDesktopPoolManagmentPoolId: horizon.virtual_desktop_pool_managment_pool_id + VMVirtualDesktopPoolManagmentPoolName: horizon.virtual_desktop_pool_managment_pool_name + VMVirtualDesktopPoolManagmentTimeStamp: horizon.virtual_desktop_pool_managment_time_stamp + VMLoadBalancingEvent: horizon.load_balancing_event + VMLoadBalancingStatus: horizon.load_balancing_status + VMLoadBalancingAlgorithm: horizon.load_balancing_algorithm + VMLoadBalancingTimeStamp: horizon.load_balancing_time_stamp + VMBlastProtocolEvent: horizon.blast_protocol_event + VMBlastProtocolUser: horizon.blast_protocol_user + VMBlastProtocolProtocolVersion: horizon.blast_protocol_protocol_version + VMBlastProtocolTimeStamp: horizon.blast_protocol_time_stamp + VMSecurityEventName: horizon.security_event_name + VMSecurityEventUser: horizon.security_event_user + VMSecurityEventAlertType: horizon.security_event_alert_type + VMSecurityEventSourceIp: horizon.security_event_source_ip + VMSecurityEventTimeStamp: horizon.security_event_time_stamp + VMLicensingInformationEvent: horizon.licensing_information_event + VMLicensingInformationLicenseType: horizon.licensing_information_license_type + VMLicensingInformationExpiryDate: horizon.licensing_information_expiry_date + VMLicensingInformationTimeStamp: horizon.licensing_information_time_stamp + VMConnectionBrokeringEvent: horizon.connection_brokering_event + VMConnectionBrokeringUser: horizon.connection_brokering_user + VMConnectionBrokeringDesktopId: horizon.connection_brokering_desktop_id + VMConnectionBrokeringStatus: horizon.connection_brokering_status + VMConnectionBrokeringTimeStamp: horizon.connection_brokering_time_stamp + DatastoreName: vsphere.datastore_name + FilesystemType: vsphere.datastore_fstype + DatastoreBytes: vsphere.datastore_capacity_free_bytes + DatastoreBytesUsed: vsphere.datastore_capacity_used_pct + HostName: vsphere.host_name + UsedCPUmhz: vsphere.host_cpu_free_mhz + UsedMemoryBites: vsphere.host_memory_total_bytes + FreeMemoryBites: vsphere.host_memory_free_bytes + VMHostID: vsphere.virtualmachine_host_id + VMHostName: + - vsphere.virtualmachine_host_hostname + - esxi.vmhost_name + VMName: + - vsphere.virtualmachine_name + - esxi.vmname + VMOperatingSystem: vsphere.virtualmachine_os + VMUsedCPU: vsphere.virtualmachine_cpu_used_mhz + VMTotalCPU: vsphere.virtualmachine_cpu_free_mhz + VMMemoryGuestUsed: vsphere.virtualmachine_memory_used_guest_bytes + VMUMemoryHostUsed: vsphere.virtualmachine_memory_used_host_bytes + VMTotalMemoryGuestBytes: vsphere.virtualmachine_memory_total_guest_bytes + VMMemoryGuestFree: vsphere.virtualmachine_memory_free_guest_bytes + VMCustomFields: vsphere.virtualmachine_custom_fields + VMNetworkNames: vsphere.virtualmachine_network_names + VMLogicalSwitchingEvent: nsxv.vmlogical_switching_event + VMLogicalSwitchingEventID: nsxv.vmlogical_switching_event_id + VMLogicalSwitchingName: nsxv.vmlogical_switching_name + VMDistributedFirewallEvent: nsxv.distributed_firewall_event_type + VMDistributedFirewallRuleID: nsxv.distributed_firewall_rule_id + VMDistributedFirewallAction: nsxv.distributed_firewall_action + VMDistributedFirewallSourceIp: nsxv.distributed_firewall_source_ip + VMDistributedFirewallDestinationIp: nsxv.distributed_firewall_destination_ip + VMSecurityGroupEventType: nsxv.security_group_event_type + VMSecurityGroupId: nsxv.security_group_id + VMSecurityGroupName: nsxv.security_group_name + VMEdgeServicesGatewayEventType: nsxv.security_edge_services_gateway_event_type + VMEdgeServicesGatewayESGID: nsxv.security_edge_services_gateway_esgid + VMEdgeServicesGatewayStatus: nsxv.security_edge_services_gateway_status + VMLoadBalancingEventType: nsxv.load_balancing_event_type + VMLoadBalancingId: nsxv.load_balancing_id + VMLoadBalancingVirtualServer: nsxv.load_balancing_virtual_server + VMNSXManagerEventType: nsxv.nsx_manager_event_type + VMNSXManagerEventDescription: nsxv.nsx_manager_event_description + VMEdgeFirewallEventType: nsxv.edge_firewall_event_type + VMEdgeFirewallSourceIP: nsxv.edge_firewall_source_ip + VMEdgeFirewallDestinationIP: nsxv.edge_firewall_destination_ip + VMEdgeFirewallRuleID: nsxv.edge_firewall_rule_id + VMSSLVPNEventType: nsxv.ssl_vpn_event_type + VMSSLVPNUserName: nsxv.ssl_vpn_user_name + VMSSLVPNSourceIp: nsxv.ssl_vpn_source_ip + VMNSXControllerEventType: nsxv.nsx_controller_event_type + VMNSXControllerID: nsxv.nsx_controller_id + VMNSXControllerStatus: nsxv.nsx_controller_status + VMLogicalRoutingEventType: nsxv.logical_routing_event_type + VMLogicalRoutingRouterID: nsxv.logical_routing_router_id + VMLogicalRoutingRouterName: nsxv.logical_routing_router_name diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_ps_script.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_ps_script.yml new file mode 100644 index 00000000..e6f7a247 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_ps_script.yml @@ -0,0 +1,302 @@ +platform: ElasticSearch ES|QL +source: windows_ps_script +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + CommandLine: + - powershell.command.value + - process.command_line.text + ProviderName: + - winlog.event_data.ProviderName + - winlog.provider_name + dns_query_name: dns.question.name + EventID: winlog.event_id + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AttributeValue: winlog.event_data.AttributeValue + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + CallTrace: winlog.event_data.CallTrace + Channel: winlog.channel + Command_Line: process.command_line.text + Commandline: process.command_line.text + commandline: process.command_line.text + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + ComputerName: winlog.ComputerName + CurrentDirectory: process.working_directory.text + Description: winlog.event_data.Description + DestinationHostname: + - destination.domain + - dns.question.name + - dns.question.subdomain + DestinationIp: destination.address + dst_ip: destination.address + DestinationPort: destination.port + dst_port: destination.port + DestinationPortName: network.protocol + Details: winlog.event_data.Details + EngineVersion: winlog.event_data.EngineVersion + EventType: winlog.event_data.EventType + FailureCode: winlog.event_data.FailureCode + FileName: file.path.text + GrantedAccess: winlog.event_data.GrantedAccess + GroupName: + - winlog.event_data.GroupName + - group.name + GroupSid: + - group.id + - winlog.event_data.GroupSid + Hashes: winlog.event_data.Hashes + file_hash: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + HostVersion: winlog.event_data.HostVersion + Image: process.executable.text + ImageLoaded: dll.path + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + IpAddress: source.address + ClientAddress: + - winlog.event_data.ClientAddress + - source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + ParentCommandLine: process.parent.command_line.text + ParentProcessName: process.parent.name.text + ParentImage: process.parent.executable.text + Path: winlog.event_data.Path + PipeName: file.name + ProcessCommandLine: winlog.event_data.ProcessCommandLine + ProcessName: process.executable.text + Properties: winlog.event_data.Properties + RuleName: winlog.event_data.RuleName + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + Signature: winlog.event_data.Signature + Signed: winlog.event_data.Signed + Source: winlog.event_data.Source + SourceHostname: source.domain + SourceImage: process.executable.text + SourceIp: source.address + src_ip: source.address + SourcePort: source.port + src_port: source.port + StartModule: winlog.event_data.StartModule + Status: winlog.event_data.Status + SubStatus: winlog.event_data.SubStatus + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + TargetDomainName: winlog.event_data.TargetDomainName + TargetFilename: file.path.text + TargetImage: winlog.event_data.TargetImage + TargetObject: winlog.event_data.TargetObject + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + TargetUserSid: winlog.event_data.TargetUserSid + QueryName: dns.question.name + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + User: user.name + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + QueryResults: dns.answers + OriginalFileName: winlog.event_data.OriginalFileName + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + Initiated: winlog.event_data.Initiated + NewUACList: winlog.event_data.NewUACList + UserAccountControl: winlog.event_data.UserAccountControl + NewUacValue: winlog.event_data.NewUacValue + OldUacValue: winlog.event_data.OldUacValue + AccountExpires: winlog.event_data.AccountExpires + DisplayName: winlog.event_data.DisplayName + DnsHostName: winlog.event_data.DnsHostName + HomeDirectory: winlog.event_data.HomeDirectory + HomePath: winlog.event_data.HomePath + LogonHours: winlog.event_data.LogonHours + PasswordLastSet: winlog.event_data.PasswordLastSet + PrimaryGroupId: winlog.event_data.PrimaryGroupId + PrivilegeList: winlog.event_data.PrivilegeList + ProfilePath: winlog.event_data.ProfilePath + SamAccountName: winlog.event_data.SamAccountName + ScriptPath: winlog.event_data.ScriptPath + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + SidHistory: winlog.event_data.SidHistory + UserParameters: winlog.event_data.UserParameters + UserPrincipalName: winlog.event_data.UserPrincipalName + UserWorkstations: winlog.event_data.UserWorkstations + RelativeTargetName: winlog.event_data.RelativeTargetName + NotificationPackageName: winlog.event_data.NotificationPackageName + SecurityPackageName: winlog.event_data.SecurityPackageName + HostApplication: process.command_line.text + TaskName: winlog.event_data.TaskName + TaskContent: winlog.event_data.TaskContent + ObjectServer: winlog.event_data.ObjectServer + NewSd: winlog.event_data.NewSd + OldSd: winlog.event_data.OldSd + TestSigning: winlog.event_data.TestSigning + AdvancedOptions: winlog.event_data.AdvancedOptions + ConfigAccessPolicy: winlog.event_data.ConfigAccessPolicy + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + FlightSigning: winlog.event_data.FlightSigning + HypervisorDebug: winlog.event_data.HypervisorDebug + HypervisorLaunchType: winlog.event_data.HypervisorLaunchType + HypervisorLoadOptions: winlog.event_data.HypervisorLoadOptions + KernelDebug: winlog.event_data.KernelDebug + LoadOptions: winlog.event_data.LoadOptions + RemoteEventLogging: winlog.event_data.RemoteEventLogging + ExceptionCode: winlog.event_data.ExceptionCode + CertSerialNumber: winlog.event_data.CertSerialNumber + CertThumbprint: winlog.event_data.CertThumbprint + CertIssuerName: winlog.event_data.CertIssuerName + TicketOptionsDescription: winlog.event_data.TicketOptionsDescription + keywords: winlog.keywords + StartAddress: winlog.event_data.StartAddress + ServiceSid: winlog.event_data.ServiceSid + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ProcessId: winlog.event_data.ProcessId + ProcessID: winlog.event_data.ProcessID + SubjectLogonId: winlog.event_data.SubjectLogonId + ElevatedToken: winlog.event_data.ElevatedToken + PublishURLs: winlog.event_data.PublishURLs + VMUserAuthenticationEvent: horizon.user_authentication_event + VMUserAuthenticationUser: horizon.user_authentication_user + VMUserAuthenticationSourceIp: horizon.user_authentication_source_ip + VMUserAuthenticationTimeStamp: horizon.user_authentication_time_stamp + VMDesktopSessionStartEvent: horizon.desktop_session_start_event + VMDesktopSessionStartUser: horizon.desktop_session_start_user + VMDesktopSessionStartDesktopID: horizon.desktop_session_start_desktop_id + VMDesktopSessionStartTimeStamp: horizon.desktop_session_time_stamp + VMApplicationLaunchEvent: horizon.application_launch_event + VMApplicationLaunchUser: horizon.application_launch_user + VMApplicationLaunchAppId: horizon.application_launch_app_id + VMApplicationLaunchAppName: horizon.application_launch_app_name + VMApplicationLaunchTimeStamp: horizon.application_launch_time_stamp + VMConnectionServerStatusEvent: horizon.connection_server_status_event + VMConnectionServerStatusServer: horizon.connection_server_status_server + VMConnectionServerStatus: horizon.connection_Server_status + VMConnectionServerStatusTimeStamp: horizon.connection_server_status_time_stamp + VMVirtualDesktopPoolManagmentEvent: horizon.virtual_desktop_pool_managment_event + VMVirtualDesktopPoolManagmentPoolId: horizon.virtual_desktop_pool_managment_pool_id + VMVirtualDesktopPoolManagmentPoolName: horizon.virtual_desktop_pool_managment_pool_name + VMVirtualDesktopPoolManagmentTimeStamp: horizon.virtual_desktop_pool_managment_time_stamp + VMLoadBalancingEvent: horizon.load_balancing_event + VMLoadBalancingStatus: horizon.load_balancing_status + VMLoadBalancingAlgorithm: horizon.load_balancing_algorithm + VMLoadBalancingTimeStamp: horizon.load_balancing_time_stamp + VMBlastProtocolEvent: horizon.blast_protocol_event + VMBlastProtocolUser: horizon.blast_protocol_user + VMBlastProtocolProtocolVersion: horizon.blast_protocol_protocol_version + VMBlastProtocolTimeStamp: horizon.blast_protocol_time_stamp + VMSecurityEventName: horizon.security_event_name + VMSecurityEventUser: horizon.security_event_user + VMSecurityEventAlertType: horizon.security_event_alert_type + VMSecurityEventSourceIp: horizon.security_event_source_ip + VMSecurityEventTimeStamp: horizon.security_event_time_stamp + VMLicensingInformationEvent: horizon.licensing_information_event + VMLicensingInformationLicenseType: horizon.licensing_information_license_type + VMLicensingInformationExpiryDate: horizon.licensing_information_expiry_date + VMLicensingInformationTimeStamp: horizon.licensing_information_time_stamp + VMConnectionBrokeringEvent: horizon.connection_brokering_event + VMConnectionBrokeringUser: horizon.connection_brokering_user + VMConnectionBrokeringDesktopId: horizon.connection_brokering_desktop_id + VMConnectionBrokeringStatus: horizon.connection_brokering_status + VMConnectionBrokeringTimeStamp: horizon.connection_brokering_time_stamp + DatastoreName: vsphere.datastore_name + FilesystemType: vsphere.datastore_fstype + DatastoreBytes: vsphere.datastore_capacity_free_bytes + DatastoreBytesUsed: vsphere.datastore_capacity_used_pct + HostName: vsphere.host_name + UsedCPUmhz: vsphere.host_cpu_free_mhz + UsedMemoryBites: vsphere.host_memory_total_bytes + FreeMemoryBites: vsphere.host_memory_free_bytes + VMHostID: vsphere.virtualmachine_host_id + VMHostName: + - vsphere.virtualmachine_host_hostname + - esxi.vmhost_name + VMName: + - vsphere.virtualmachine_name + - esxi.vmname + VMOperatingSystem: vsphere.virtualmachine_os + VMUsedCPU: vsphere.virtualmachine_cpu_used_mhz + VMTotalCPU: vsphere.virtualmachine_cpu_free_mhz + VMMemoryGuestUsed: vsphere.virtualmachine_memory_used_guest_bytes + VMUMemoryHostUsed: vsphere.virtualmachine_memory_used_host_bytes + VMTotalMemoryGuestBytes: vsphere.virtualmachine_memory_total_guest_bytes + VMMemoryGuestFree: vsphere.virtualmachine_memory_free_guest_bytes + VMCustomFields: vsphere.virtualmachine_custom_fields + VMNetworkNames: vsphere.virtualmachine_network_names + VMLogicalSwitchingEvent: nsxv.vmlogical_switching_event + VMLogicalSwitchingEventID: nsxv.vmlogical_switching_event_id + VMLogicalSwitchingName: nsxv.vmlogical_switching_name + VMDistributedFirewallEvent: nsxv.distributed_firewall_event_type + VMDistributedFirewallRuleID: nsxv.distributed_firewall_rule_id + VMDistributedFirewallAction: nsxv.distributed_firewall_action + VMDistributedFirewallSourceIp: nsxv.distributed_firewall_source_ip + VMDistributedFirewallDestinationIp: nsxv.distributed_firewall_destination_ip + VMSecurityGroupEventType: nsxv.security_group_event_type + VMSecurityGroupId: nsxv.security_group_id + VMSecurityGroupName: nsxv.security_group_name + VMEdgeServicesGatewayEventType: nsxv.security_edge_services_gateway_event_type + VMEdgeServicesGatewayESGID: nsxv.security_edge_services_gateway_esgid + VMEdgeServicesGatewayStatus: nsxv.security_edge_services_gateway_status + VMLoadBalancingEventType: nsxv.load_balancing_event_type + VMLoadBalancingId: nsxv.load_balancing_id + VMLoadBalancingVirtualServer: nsxv.load_balancing_virtual_server + VMNSXManagerEventType: nsxv.nsx_manager_event_type + VMNSXManagerEventDescription: nsxv.nsx_manager_event_description + VMEdgeFirewallEventType: nsxv.edge_firewall_event_type + VMEdgeFirewallSourceIP: nsxv.edge_firewall_source_ip + VMEdgeFirewallDestinationIP: nsxv.edge_firewall_destination_ip + VMEdgeFirewallRuleID: nsxv.edge_firewall_rule_id + VMSSLVPNEventType: nsxv.ssl_vpn_event_type + VMSSLVPNUserName: nsxv.ssl_vpn_user_name + VMSSLVPNSourceIp: nsxv.ssl_vpn_source_ip + VMNSXControllerEventType: nsxv.nsx_controller_event_type + VMNSXControllerID: nsxv.nsx_controller_id + VMNSXControllerStatus: nsxv.nsx_controller_status + VMLogicalRoutingEventType: nsxv.logical_routing_event_type + VMLogicalRoutingRouterID: nsxv.logical_routing_router_id + VMLogicalRoutingRouterName: nsxv.logical_routing_router_name diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_security.yml new file mode 100644 index 00000000..6c07164e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_security.yml @@ -0,0 +1,300 @@ +platform: ElasticSearch ES|QL +source: windows_security +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + IpAddress: source.ip winlog.event_data.ClientAddress + ProcessName: winlog.event_data.ProcessName process.executable.text + ProviderName: + - winlog.event_data.ProviderName + - winlog.provider_name + dns_query_name: dns.question.name + EventID: winlog.event_id + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AttributeValue: winlog.event_data.AttributeValue + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + CallTrace: winlog.event_data.CallTrace + Channel: winlog.channel + CommandLine: process.command_line.text + Command_Line: process.command_line.text + Commandline: process.command_line.text + commandline: process.command_line.text + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + ComputerName: winlog.ComputerName + CurrentDirectory: process.working_directory.text + Description: winlog.event_data.Description + DestinationHostname: + - destination.domain + - dns.question.name + - dns.question.subdomain + DestinationIp: destination.address + dst_ip: destination.address + DestinationPort: destination.port + dst_port: destination.port + DestinationPortName: network.protocol + Details: winlog.event_data.Details + EngineVersion: winlog.event_data.EngineVersion + EventType: winlog.event_data.EventType + FailureCode: winlog.event_data.FailureCode + FileName: file.path.text + GrantedAccess: winlog.event_data.GrantedAccess + GroupName: + - winlog.event_data.GroupName + - group.name + GroupSid: + - group.id + - winlog.event_data.GroupSid + Hashes: winlog.event_data.Hashes + file_hash: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + HostVersion: winlog.event_data.HostVersion + Image: process.executable.text + ImageLoaded: dll.path + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + ClientAddress: + - winlog.event_data.ClientAddress + - source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + ParentCommandLine: process.parent.command_line.text + ParentProcessName: process.parent.name.text + ParentImage: process.parent.executable.text + Path: winlog.event_data.Path + PipeName: file.name + ProcessCommandLine: winlog.event_data.ProcessCommandLine + Properties: winlog.event_data.Properties + RuleName: winlog.event_data.RuleName + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + Signature: winlog.event_data.Signature + Signed: winlog.event_data.Signed + Source: winlog.event_data.Source + SourceHostname: source.domain + SourceImage: process.executable.text + SourceIp: source.address + src_ip: source.address + SourcePort: source.port + src_port: source.port + StartModule: winlog.event_data.StartModule + Status: winlog.event_data.Status + SubStatus: winlog.event_data.SubStatus + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + TargetDomainName: winlog.event_data.TargetDomainName + TargetFilename: file.path.text + TargetImage: winlog.event_data.TargetImage + TargetObject: winlog.event_data.TargetObject + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + TargetUserSid: winlog.event_data.TargetUserSid + QueryName: dns.question.name + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + User: user.name + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + QueryResults: dns.answers + OriginalFileName: winlog.event_data.OriginalFileName + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + Initiated: winlog.event_data.Initiated + NewUACList: winlog.event_data.NewUACList + UserAccountControl: winlog.event_data.UserAccountControl + NewUacValue: winlog.event_data.NewUacValue + OldUacValue: winlog.event_data.OldUacValue + AccountExpires: winlog.event_data.AccountExpires + DisplayName: winlog.event_data.DisplayName + DnsHostName: winlog.event_data.DnsHostName + HomeDirectory: winlog.event_data.HomeDirectory + HomePath: winlog.event_data.HomePath + LogonHours: winlog.event_data.LogonHours + PasswordLastSet: winlog.event_data.PasswordLastSet + PrimaryGroupId: winlog.event_data.PrimaryGroupId + PrivilegeList: winlog.event_data.PrivilegeList + ProfilePath: winlog.event_data.ProfilePath + SamAccountName: winlog.event_data.SamAccountName + ScriptPath: winlog.event_data.ScriptPath + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + SidHistory: winlog.event_data.SidHistory + UserParameters: winlog.event_data.UserParameters + UserPrincipalName: winlog.event_data.UserPrincipalName + UserWorkstations: winlog.event_data.UserWorkstations + RelativeTargetName: winlog.event_data.RelativeTargetName + NotificationPackageName: winlog.event_data.NotificationPackageName + SecurityPackageName: winlog.event_data.SecurityPackageName + HostApplication: process.command_line.text + TaskName: winlog.event_data.TaskName + TaskContent: winlog.event_data.TaskContent + ObjectServer: winlog.event_data.ObjectServer + NewSd: winlog.event_data.NewSd + OldSd: winlog.event_data.OldSd + TestSigning: winlog.event_data.TestSigning + AdvancedOptions: winlog.event_data.AdvancedOptions + ConfigAccessPolicy: winlog.event_data.ConfigAccessPolicy + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + FlightSigning: winlog.event_data.FlightSigning + HypervisorDebug: winlog.event_data.HypervisorDebug + HypervisorLaunchType: winlog.event_data.HypervisorLaunchType + HypervisorLoadOptions: winlog.event_data.HypervisorLoadOptions + KernelDebug: winlog.event_data.KernelDebug + LoadOptions: winlog.event_data.LoadOptions + RemoteEventLogging: winlog.event_data.RemoteEventLogging + ExceptionCode: winlog.event_data.ExceptionCode + CertSerialNumber: winlog.event_data.CertSerialNumber + CertThumbprint: winlog.event_data.CertThumbprint + CertIssuerName: winlog.event_data.CertIssuerName + TicketOptionsDescription: winlog.event_data.TicketOptionsDescription + keywords: winlog.keywords + StartAddress: winlog.event_data.StartAddress + ServiceSid: winlog.event_data.ServiceSid + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ProcessId: winlog.event_data.ProcessId + ProcessID: winlog.event_data.ProcessID + SubjectLogonId: winlog.event_data.SubjectLogonId + ElevatedToken: winlog.event_data.ElevatedToken + PublishURLs: winlog.event_data.PublishURLs + VMUserAuthenticationEvent: horizon.user_authentication_event + VMUserAuthenticationUser: horizon.user_authentication_user + VMUserAuthenticationSourceIp: horizon.user_authentication_source_ip + VMUserAuthenticationTimeStamp: horizon.user_authentication_time_stamp + VMDesktopSessionStartEvent: horizon.desktop_session_start_event + VMDesktopSessionStartUser: horizon.desktop_session_start_user + VMDesktopSessionStartDesktopID: horizon.desktop_session_start_desktop_id + VMDesktopSessionStartTimeStamp: horizon.desktop_session_time_stamp + VMApplicationLaunchEvent: horizon.application_launch_event + VMApplicationLaunchUser: horizon.application_launch_user + VMApplicationLaunchAppId: horizon.application_launch_app_id + VMApplicationLaunchAppName: horizon.application_launch_app_name + VMApplicationLaunchTimeStamp: horizon.application_launch_time_stamp + VMConnectionServerStatusEvent: horizon.connection_server_status_event + VMConnectionServerStatusServer: horizon.connection_server_status_server + VMConnectionServerStatus: horizon.connection_Server_status + VMConnectionServerStatusTimeStamp: horizon.connection_server_status_time_stamp + VMVirtualDesktopPoolManagmentEvent: horizon.virtual_desktop_pool_managment_event + VMVirtualDesktopPoolManagmentPoolId: horizon.virtual_desktop_pool_managment_pool_id + VMVirtualDesktopPoolManagmentPoolName: horizon.virtual_desktop_pool_managment_pool_name + VMVirtualDesktopPoolManagmentTimeStamp: horizon.virtual_desktop_pool_managment_time_stamp + VMLoadBalancingEvent: horizon.load_balancing_event + VMLoadBalancingStatus: horizon.load_balancing_status + VMLoadBalancingAlgorithm: horizon.load_balancing_algorithm + VMLoadBalancingTimeStamp: horizon.load_balancing_time_stamp + VMBlastProtocolEvent: horizon.blast_protocol_event + VMBlastProtocolUser: horizon.blast_protocol_user + VMBlastProtocolProtocolVersion: horizon.blast_protocol_protocol_version + VMBlastProtocolTimeStamp: horizon.blast_protocol_time_stamp + VMSecurityEventName: horizon.security_event_name + VMSecurityEventUser: horizon.security_event_user + VMSecurityEventAlertType: horizon.security_event_alert_type + VMSecurityEventSourceIp: horizon.security_event_source_ip + VMSecurityEventTimeStamp: horizon.security_event_time_stamp + VMLicensingInformationEvent: horizon.licensing_information_event + VMLicensingInformationLicenseType: horizon.licensing_information_license_type + VMLicensingInformationExpiryDate: horizon.licensing_information_expiry_date + VMLicensingInformationTimeStamp: horizon.licensing_information_time_stamp + VMConnectionBrokeringEvent: horizon.connection_brokering_event + VMConnectionBrokeringUser: horizon.connection_brokering_user + VMConnectionBrokeringDesktopId: horizon.connection_brokering_desktop_id + VMConnectionBrokeringStatus: horizon.connection_brokering_status + VMConnectionBrokeringTimeStamp: horizon.connection_brokering_time_stamp + DatastoreName: vsphere.datastore_name + FilesystemType: vsphere.datastore_fstype + DatastoreBytes: vsphere.datastore_capacity_free_bytes + DatastoreBytesUsed: vsphere.datastore_capacity_used_pct + HostName: vsphere.host_name + UsedCPUmhz: vsphere.host_cpu_free_mhz + UsedMemoryBites: vsphere.host_memory_total_bytes + FreeMemoryBites: vsphere.host_memory_free_bytes + VMHostID: vsphere.virtualmachine_host_id + VMHostName: + - vsphere.virtualmachine_host_hostname + - esxi.vmhost_name + VMName: + - vsphere.virtualmachine_name + - esxi.vmname + VMOperatingSystem: vsphere.virtualmachine_os + VMUsedCPU: vsphere.virtualmachine_cpu_used_mhz + VMTotalCPU: vsphere.virtualmachine_cpu_free_mhz + VMMemoryGuestUsed: vsphere.virtualmachine_memory_used_guest_bytes + VMUMemoryHostUsed: vsphere.virtualmachine_memory_used_host_bytes + VMTotalMemoryGuestBytes: vsphere.virtualmachine_memory_total_guest_bytes + VMMemoryGuestFree: vsphere.virtualmachine_memory_free_guest_bytes + VMCustomFields: vsphere.virtualmachine_custom_fields + VMNetworkNames: vsphere.virtualmachine_network_names + VMLogicalSwitchingEvent: nsxv.vmlogical_switching_event + VMLogicalSwitchingEventID: nsxv.vmlogical_switching_event_id + VMLogicalSwitchingName: nsxv.vmlogical_switching_name + VMDistributedFirewallEvent: nsxv.distributed_firewall_event_type + VMDistributedFirewallRuleID: nsxv.distributed_firewall_rule_id + VMDistributedFirewallAction: nsxv.distributed_firewall_action + VMDistributedFirewallSourceIp: nsxv.distributed_firewall_source_ip + VMDistributedFirewallDestinationIp: nsxv.distributed_firewall_destination_ip + VMSecurityGroupEventType: nsxv.security_group_event_type + VMSecurityGroupId: nsxv.security_group_id + VMSecurityGroupName: nsxv.security_group_name + VMEdgeServicesGatewayEventType: nsxv.security_edge_services_gateway_event_type + VMEdgeServicesGatewayESGID: nsxv.security_edge_services_gateway_esgid + VMEdgeServicesGatewayStatus: nsxv.security_edge_services_gateway_status + VMLoadBalancingEventType: nsxv.load_balancing_event_type + VMLoadBalancingId: nsxv.load_balancing_id + VMLoadBalancingVirtualServer: nsxv.load_balancing_virtual_server + VMNSXManagerEventType: nsxv.nsx_manager_event_type + VMNSXManagerEventDescription: nsxv.nsx_manager_event_description + VMEdgeFirewallEventType: nsxv.edge_firewall_event_type + VMEdgeFirewallSourceIP: nsxv.edge_firewall_source_ip + VMEdgeFirewallDestinationIP: nsxv.edge_firewall_destination_ip + VMEdgeFirewallRuleID: nsxv.edge_firewall_rule_id + VMSSLVPNEventType: nsxv.ssl_vpn_event_type + VMSSLVPNUserName: nsxv.ssl_vpn_user_name + VMSSLVPNSourceIp: nsxv.ssl_vpn_source_ip + VMNSXControllerEventType: nsxv.nsx_controller_event_type + VMNSXControllerID: nsxv.nsx_controller_id + VMNSXControllerStatus: nsxv.nsx_controller_status + VMLogicalRoutingEventType: nsxv.logical_routing_event_type + VMLogicalRoutingRouterID: nsxv.logical_routing_router_id + VMLogicalRoutingRouterName: nsxv.logical_routing_router_name diff --git a/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_sysmon.yml new file mode 100644 index 00000000..bde8fc38 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/elasticsearch_esql/windows_sysmon.yml @@ -0,0 +1,302 @@ +platform: ElasticSearch ES|QL +source: windows_sysmon +log_source: + index: [logs-*] +default_log_source: + index: logs-* +field_mapping: + ImageLoaded: + - winlog.event_data.ImageLoaded + - file.path + ProviderName: + - winlog.event_data.ProviderName + - winlog.provider_name + dns_query_name: dns.question.name + EventID: winlog.event_id + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AttributeValue: winlog.event_data.AttributeValue + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + CallTrace: winlog.event_data.CallTrace + Channel: winlog.channel + CommandLine: process.command_line.text + Command_Line: process.command_line.text + Commandline: process.command_line.text + commandline: process.command_line.text + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + ComputerName: winlog.ComputerName + CurrentDirectory: process.working_directory.text + Description: winlog.event_data.Description + DestinationHostname: + - destination.domain + - dns.question.name + - dns.question.subdomain + DestinationIp: destination.address + dst_ip: destination.address + DestinationPort: destination.port + dst_port: destination.port + DestinationPortName: network.protocol + Details: winlog.event_data.Details + EngineVersion: winlog.event_data.EngineVersion + EventType: winlog.event_data.EventType + FailureCode: winlog.event_data.FailureCode + FileName: file.path.text + GrantedAccess: winlog.event_data.GrantedAccess + GroupName: + - winlog.event_data.GroupName + - group.name + GroupSid: + - group.id + - winlog.event_data.GroupSid + Hashes: winlog.event_data.Hashes + file_hash: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + HostVersion: winlog.event_data.HostVersion + Image: process.executable.text + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + IpAddress: source.address + ClientAddress: + - winlog.event_data.ClientAddress + - source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + ParentCommandLine: process.parent.command_line.text + ParentProcessName: process.parent.name.text + ParentImage: process.parent.executable.text + Path: winlog.event_data.Path + PipeName: file.name + ProcessCommandLine: winlog.event_data.ProcessCommandLine + ProcessName: process.executable.text + Properties: winlog.event_data.Properties + RuleName: winlog.event_data.RuleName + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + Signature: winlog.event_data.Signature + Signed: winlog.event_data.Signed + Source: winlog.event_data.Source + SourceHostname: source.domain + SourceImage: process.executable.text + SourceIp: source.address + src_ip: source.address + SourcePort: source.port + src_port: source.port + StartModule: winlog.event_data.StartModule + Status: winlog.event_data.Status + SubStatus: winlog.event_data.SubStatus + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + TargetDomainName: winlog.event_data.TargetDomainName + TargetFilename: file.path.text + TargetImage: winlog.event_data.TargetImage + TargetObject: winlog.event_data.TargetObject + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + TargetUserSid: winlog.event_data.TargetUserSid + QueryName: dns.question.name + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + User: user.name + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + QueryResults: dns.answers + OriginalFileName: winlog.event_data.OriginalFileName + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + Initiated: winlog.event_data.Initiated + NewUACList: winlog.event_data.NewUACList + UserAccountControl: winlog.event_data.UserAccountControl + NewUacValue: winlog.event_data.NewUacValue + OldUacValue: winlog.event_data.OldUacValue + AccountExpires: winlog.event_data.AccountExpires + DisplayName: winlog.event_data.DisplayName + DnsHostName: winlog.event_data.DnsHostName + HomeDirectory: winlog.event_data.HomeDirectory + HomePath: winlog.event_data.HomePath + LogonHours: winlog.event_data.LogonHours + PasswordLastSet: winlog.event_data.PasswordLastSet + PrimaryGroupId: winlog.event_data.PrimaryGroupId + PrivilegeList: winlog.event_data.PrivilegeList + ProfilePath: winlog.event_data.ProfilePath + SamAccountName: winlog.event_data.SamAccountName + ScriptPath: winlog.event_data.ScriptPath + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + SidHistory: winlog.event_data.SidHistory + UserParameters: winlog.event_data.UserParameters + UserPrincipalName: winlog.event_data.UserPrincipalName + UserWorkstations: winlog.event_data.UserWorkstations + RelativeTargetName: winlog.event_data.RelativeTargetName + NotificationPackageName: winlog.event_data.NotificationPackageName + SecurityPackageName: winlog.event_data.SecurityPackageName + HostApplication: process.command_line.text + TaskName: winlog.event_data.TaskName + TaskContent: winlog.event_data.TaskContent + ObjectServer: winlog.event_data.ObjectServer + NewSd: winlog.event_data.NewSd + OldSd: winlog.event_data.OldSd + TestSigning: winlog.event_data.TestSigning + AdvancedOptions: winlog.event_data.AdvancedOptions + ConfigAccessPolicy: winlog.event_data.ConfigAccessPolicy + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + FlightSigning: winlog.event_data.FlightSigning + HypervisorDebug: winlog.event_data.HypervisorDebug + HypervisorLaunchType: winlog.event_data.HypervisorLaunchType + HypervisorLoadOptions: winlog.event_data.HypervisorLoadOptions + KernelDebug: winlog.event_data.KernelDebug + LoadOptions: winlog.event_data.LoadOptions + RemoteEventLogging: winlog.event_data.RemoteEventLogging + ExceptionCode: winlog.event_data.ExceptionCode + CertSerialNumber: winlog.event_data.CertSerialNumber + CertThumbprint: winlog.event_data.CertThumbprint + CertIssuerName: winlog.event_data.CertIssuerName + TicketOptionsDescription: winlog.event_data.TicketOptionsDescription + keywords: winlog.keywords + StartAddress: winlog.event_data.StartAddress + ServiceSid: winlog.event_data.ServiceSid + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ProcessId: winlog.event_data.ProcessId + ProcessID: winlog.event_data.ProcessID + SubjectLogonId: winlog.event_data.SubjectLogonId + ElevatedToken: winlog.event_data.ElevatedToken + PublishURLs: winlog.event_data.PublishURLs + VMUserAuthenticationEvent: horizon.user_authentication_event + VMUserAuthenticationUser: horizon.user_authentication_user + VMUserAuthenticationSourceIp: horizon.user_authentication_source_ip + VMUserAuthenticationTimeStamp: horizon.user_authentication_time_stamp + VMDesktopSessionStartEvent: horizon.desktop_session_start_event + VMDesktopSessionStartUser: horizon.desktop_session_start_user + VMDesktopSessionStartDesktopID: horizon.desktop_session_start_desktop_id + VMDesktopSessionStartTimeStamp: horizon.desktop_session_time_stamp + VMApplicationLaunchEvent: horizon.application_launch_event + VMApplicationLaunchUser: horizon.application_launch_user + VMApplicationLaunchAppId: horizon.application_launch_app_id + VMApplicationLaunchAppName: horizon.application_launch_app_name + VMApplicationLaunchTimeStamp: horizon.application_launch_time_stamp + VMConnectionServerStatusEvent: horizon.connection_server_status_event + VMConnectionServerStatusServer: horizon.connection_server_status_server + VMConnectionServerStatus: horizon.connection_Server_status + VMConnectionServerStatusTimeStamp: horizon.connection_server_status_time_stamp + VMVirtualDesktopPoolManagmentEvent: horizon.virtual_desktop_pool_managment_event + VMVirtualDesktopPoolManagmentPoolId: horizon.virtual_desktop_pool_managment_pool_id + VMVirtualDesktopPoolManagmentPoolName: horizon.virtual_desktop_pool_managment_pool_name + VMVirtualDesktopPoolManagmentTimeStamp: horizon.virtual_desktop_pool_managment_time_stamp + VMLoadBalancingEvent: horizon.load_balancing_event + VMLoadBalancingStatus: horizon.load_balancing_status + VMLoadBalancingAlgorithm: horizon.load_balancing_algorithm + VMLoadBalancingTimeStamp: horizon.load_balancing_time_stamp + VMBlastProtocolEvent: horizon.blast_protocol_event + VMBlastProtocolUser: horizon.blast_protocol_user + VMBlastProtocolProtocolVersion: horizon.blast_protocol_protocol_version + VMBlastProtocolTimeStamp: horizon.blast_protocol_time_stamp + VMSecurityEventName: horizon.security_event_name + VMSecurityEventUser: horizon.security_event_user + VMSecurityEventAlertType: horizon.security_event_alert_type + VMSecurityEventSourceIp: horizon.security_event_source_ip + VMSecurityEventTimeStamp: horizon.security_event_time_stamp + VMLicensingInformationEvent: horizon.licensing_information_event + VMLicensingInformationLicenseType: horizon.licensing_information_license_type + VMLicensingInformationExpiryDate: horizon.licensing_information_expiry_date + VMLicensingInformationTimeStamp: horizon.licensing_information_time_stamp + VMConnectionBrokeringEvent: horizon.connection_brokering_event + VMConnectionBrokeringUser: horizon.connection_brokering_user + VMConnectionBrokeringDesktopId: horizon.connection_brokering_desktop_id + VMConnectionBrokeringStatus: horizon.connection_brokering_status + VMConnectionBrokeringTimeStamp: horizon.connection_brokering_time_stamp + DatastoreName: vsphere.datastore_name + FilesystemType: vsphere.datastore_fstype + DatastoreBytes: vsphere.datastore_capacity_free_bytes + DatastoreBytesUsed: vsphere.datastore_capacity_used_pct + HostName: vsphere.host_name + UsedCPUmhz: vsphere.host_cpu_free_mhz + UsedMemoryBites: vsphere.host_memory_total_bytes + FreeMemoryBites: vsphere.host_memory_free_bytes + VMHostID: vsphere.virtualmachine_host_id + VMHostName: + - vsphere.virtualmachine_host_hostname + - esxi.vmhost_name + VMName: + - vsphere.virtualmachine_name + - esxi.vmname + VMOperatingSystem: vsphere.virtualmachine_os + VMUsedCPU: vsphere.virtualmachine_cpu_used_mhz + VMTotalCPU: vsphere.virtualmachine_cpu_free_mhz + VMMemoryGuestUsed: vsphere.virtualmachine_memory_used_guest_bytes + VMUMemoryHostUsed: vsphere.virtualmachine_memory_used_host_bytes + VMTotalMemoryGuestBytes: vsphere.virtualmachine_memory_total_guest_bytes + VMMemoryGuestFree: vsphere.virtualmachine_memory_free_guest_bytes + VMCustomFields: vsphere.virtualmachine_custom_fields + VMNetworkNames: vsphere.virtualmachine_network_names + VMLogicalSwitchingEvent: nsxv.vmlogical_switching_event + VMLogicalSwitchingEventID: nsxv.vmlogical_switching_event_id + VMLogicalSwitchingName: nsxv.vmlogical_switching_name + VMDistributedFirewallEvent: nsxv.distributed_firewall_event_type + VMDistributedFirewallRuleID: nsxv.distributed_firewall_rule_id + VMDistributedFirewallAction: nsxv.distributed_firewall_action + VMDistributedFirewallSourceIp: nsxv.distributed_firewall_source_ip + VMDistributedFirewallDestinationIp: nsxv.distributed_firewall_destination_ip + VMSecurityGroupEventType: nsxv.security_group_event_type + VMSecurityGroupId: nsxv.security_group_id + VMSecurityGroupName: nsxv.security_group_name + VMEdgeServicesGatewayEventType: nsxv.security_edge_services_gateway_event_type + VMEdgeServicesGatewayESGID: nsxv.security_edge_services_gateway_esgid + VMEdgeServicesGatewayStatus: nsxv.security_edge_services_gateway_status + VMLoadBalancingEventType: nsxv.load_balancing_event_type + VMLoadBalancingId: nsxv.load_balancing_id + VMLoadBalancingVirtualServer: nsxv.load_balancing_virtual_server + VMNSXManagerEventType: nsxv.nsx_manager_event_type + VMNSXManagerEventDescription: nsxv.nsx_manager_event_description + VMEdgeFirewallEventType: nsxv.edge_firewall_event_type + VMEdgeFirewallSourceIP: nsxv.edge_firewall_source_ip + VMEdgeFirewallDestinationIP: nsxv.edge_firewall_destination_ip + VMEdgeFirewallRuleID: nsxv.edge_firewall_rule_id + VMSSLVPNEventType: nsxv.ssl_vpn_event_type + VMSSLVPNUserName: nsxv.ssl_vpn_user_name + VMSSLVPNSourceIp: nsxv.ssl_vpn_source_ip + VMNSXControllerEventType: nsxv.nsx_controller_event_type + VMNSXControllerID: nsxv.nsx_controller_id + VMNSXControllerStatus: nsxv.nsx_controller_status + VMLogicalRoutingEventType: nsxv.logical_routing_event_type + VMLogicalRoutingRouterID: nsxv.logical_routing_router_id + VMLogicalRoutingRouterName: nsxv.logical_routing_router_name diff --git a/uncoder-core/app/translator/mappings/platforms/falco/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/falco/aws_cloudtrail.yml new file mode 100644 index 00000000..9c680168 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/falco/aws_cloudtrail.yml @@ -0,0 +1,8 @@ +platform: Falco +source: aws_cloudtrail + +field_mapping: + eventSource: ct.src + eventName: ct.name + errorCode: ct.error + RequestParameters: json.value[/requestParameters] diff --git a/uncoder-core/app/translator/mappings/platforms/falco/default.yml b/uncoder-core/app/translator/mappings/platforms/falco/default.yml new file mode 100644 index 00000000..cbf5326f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/falco/default.yml @@ -0,0 +1,6 @@ +platform: Falco +source: default + + +field_mapping: + {} \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/common.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/common.yml new file mode 100644 index 00000000..859ce0cb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/common.yml @@ -0,0 +1,257 @@ +platform: FortiSiem +description: Common field mapping + +field_mapping: + TargetDetails: details + Account_Name: user + Computer_Name: computer + Originating_Computer: srcName + FileHash: hashCode + FilePath: filePath + Fqbn: hostName + RuleId: ruleId # int type + RuleName: ruleName + CallTrace: procPath + IntegrityLevel: integrityLevel + ParentIntegrityLevel: procTrustLevel # int type + Company: company + ParentProcessGuid: procOwner + LogonGuid: uuid + ParentUser: userGrp + Hashes: hashCode + Imphash: hashIMP + OriginalFilename: srcFileName + OriginalFileName: srcFileName + ParentProcess: parentProcName + Product: product + sha1: hashSHA1 + DestPort: destIpPort # int type + Destination: destIpAddr # ip type + destination.port: destIpPort # int type + HostApplication: appName + TargetName: targetName + TargetProcessAddress: destMACAddr + Service: serviceName + Source: eventSource + ImagePath: serviceFileName + Path: procPath + Payload: dataPayload + Properties: propName + QueryName: queryId + QueryResults: actionResult + QueryStatus: status + LogonProcessName: winLogonProc + ServicePrincipalNames: principal + HostVersion: version + FailureCode: winKerbFailCode + EngineVersion: version + DeviceClassName: deviceType + DeviceDescription: description + Status: winLogonFailCode2 + AccessList: fileAccess + AccessMask: fileAccess + AttributeLDAPDisplayName: propName + ContextInfo: lineContent + AttributeValue: propValue + GroupSid: groupID + AuditPolicyChanges: actionName + CallingProcessName: procName + GrantedAccess: accessKeyId + KeyLength: msgLen + keywords: msg + Keywords: msg + LayerRTID: permissionLevelID + Level: permissionLevelType + LDAPDisplayName: propName + Value: propValue + ObjectClass: osObjType + ObjectServer: serverName + ObjectValueName: osObjValue + PipeName: vpnTunnelName + PrivilegeList: privName + RelativeTargetName: targetName + SAMAccountName: accountName + ScriptBlockText: script + ShareName: fileName + SidHistory: essId + Signed: authResult + StartFunction: funName + StartModule: module + TicketEncryptionType: encryptAlgo + TicketOptions: kerbTicketOption + IpAddress: srcIpAddr # ip type + HiveName: procName + DestinationIsIpv6: isIpv6 + AllowedToDelegateTo: _win_isAllowedToDelegateTo_removed + CallerProcessName: procName + Caption: description + CertThumbprint: certInfo + ClassName: deviceType + ModifyingApplication: appName + OldTargetUserName: oldTargetUser + ommandLine: command + OriginalName: originalProcName + Provider_Name: _win_providerName_removed + param1: paraName + param2: otherParaName + RemoteAddress: remoteAddress + SamAccountName: targetUser + TargetUserSid: userId + TargetSid: groupID + TemplateContent: templateContent + NewTemplateContent: newTemplateContent + NewTargetUserName: targetUser + RemoteName: targetName + LocalName: filePath + processPath: procName + Action: eventAction # int type + ApplicationPath: procPath + SearchFilter: queryFilter + Address: remoteAddress + Origin: origLocation + PasswordLastSet: passwordLastSet # Dara type + TargetServerName: targetName + ServerName: serverName + DeviceName: hostName + AuditSourceName: resourceName + TargetLogonId: winLogonId + ComputerName: computer + CurrentDirectory: dirName + Description: description + FileVersion: fileVersion + GroupName: targetUserGrp + LogonId: winLogonId + NewName: newObjValue + ProcessName: procName + QNAME: destName + TargetFilename: fileName + User: user + Image: procName + ParentImage: parentProcName + CommandLine: command + TaskName: task + ServiceName: serviceName + TargetObject: regKeyPath + EventType: osObjAction + EventID: eventType + EventCode: eventType + Details: details + ParentCommandLine: parentCommand + Message: msg + HostName: hostName + FileName: fileName + TargetImage: targetProcName + Accesses: osObjAccessType + AccountName: user + DestinationIp: destIpAddr # ip type + DestinationPort: destIpPort # int type + DestinationHostname: destName + DestinationAddress: destIpAddr # ip type + ObjectType: osObjType + ObjectName: osObjName + SourceImage: procName + SourceAddress: srcIpAddr # ip type + SourcePort: srcIpPort # int type + SourceNetworkAddress: srcIpAddr # ip type + SourceWorkstation: srcName + TargetUserName: user + UserName: user + SubjectDomainName: domain + SubjectLogonId: winLogonId + SubjectUserName: user + SubjectUserSid: securityId + Workstation: computer + WorkstationName: computer + ServiceFileName: serviceFileName + Signature: signatureName + ImageLoaded: loadedProcName + LogonType: winLogonType # int type + AuthenticationPackage: procName + AuthenticationPackageName: authenMethod + Device: deviceIdentification + PolicyName: policyName + TargetProcessId: targetProcId + TargetUser: targetUser + NewValue: newObjValue + SubjectAccountName: user + ClientAddress: srcIpAddr # ip type + ProcessID: procId + TargetFileName: fileName + AccountDomain: domain + Computer: computer + DomainName: targetDomain + Initiated: initiated + Commandline: command + ProcessCommandLine: command + Channel: activityType + ServiceType: serviceType + ServiceStartType: serviceStartType + #network + dst_ip: destIpAddr # ip type + src_ip: srcIpAddr # ip type + dst_port: destIpPort # int type + src_port: srcIpPort # int type + dns_query: uriQuery + uri_query: uriQuery + parent_domain: domain + record_type: type + query: queryId + action: activityName + operation: opName + c-useragent: httpUserAgent + c-uri: httpEndUri + endpoint: targetName + service: serviceName + path: procPath + name: procName + cipher: password + request_type: type + answer: actionResult + resp_mime_types: type + message_size: msgLen # int type + question_length: size # int type + cs-method: httpMethod + sc-status: status + method: httpMethod + referer: httpReferrer + useragent: httpUserAgent + clientip: srcIpAddr # int type + MachineName: hostName + TargetPort: destIpPort # int type + DestAddress: destIpAddr # ip type + SourceIp: srcIpAddr # ip type + AppName: appName + Binary: code + ClientProcessId: procId + ParentProcessId: parentProcId + Data: lineContent + ErrorCode: errorCode + FileNameBuffer: fileName + ProcessNameBuffer: procName + RequestedPolicy: policyIdentity + ValidatedPolicy: recipientPolicyId + FilterOrigin: filter + Application: appName + AppID: appName + ImageName: procName + ImpersonationLevel: permissionLevelID + Name: procName + PackageFullName: procName + PackagePath: procPath + ProcessId: procId + ProcessPath: procPath + Protocol: srcProto + SidList: userId + SourceCommandLine: command + SourceHostname: srcName + TargetOutboundUserName: user + TaskContent: task + TaskContentNew: task + md5: hashMD5 + param3: swParam + process: procName + sha256: hashSHA256 + subjectName: subjectContainsWords + ExceptionCode: errorCode + SignatureStatus: signatureStatus + payload: dataPayload diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/default.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/default.yml new file mode 100644 index 00000000..a4ab18c6 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/default.yml @@ -0,0 +1,5 @@ +platform: FortiSiem +source: default + + +default_log_source: {} diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/linux_file_event.yml new file mode 100644 index 00000000..6819ad48 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/linux_file_event.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: linux_file_event + + +log_source: + eventType: + - FSM_LINUX_FILE_* + +default_log_source: + eventType: FSM_LINUX_FILE_* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_app.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_app.yml new file mode 100644 index 00000000..c9d6c8e5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_app.yml @@ -0,0 +1,6 @@ +platform: FortiSiem +source: windows_app + + +default_log_source: + eventType: Win-App-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_application.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_application.yml new file mode 100644 index 00000000..203e22f8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_application.yml @@ -0,0 +1,6 @@ +platform: FortiSiem +source: windows_application + + +default_log_source: + eventType: Win-App-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_appxdeployment.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_appxdeployment.yml new file mode 100644 index 00000000..0266ede0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_appxdeployment.yml @@ -0,0 +1,12 @@ +platform: FortiSiem +source: windows_appxdeployment + + +log_source: + eventType: + - Win-AppXDeployment-Server-400 + - Win-AppXDeployment-Server-401 + - Win-AppXDeployment-Server-157 + +default_log_source: + eventType: Win-AppXDeployment-Server-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_appxdeployment_server.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_appxdeployment_server.yml new file mode 100644 index 00000000..57f1a911 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_appxdeployment_server.yml @@ -0,0 +1,12 @@ +platform: FortiSiem +source: windows_appxdeployment_server + + +log_source: + eventType: + - Win-AppXDeployment-Server-400 + - Win-AppXDeployment-Server-401 + - Win-AppXDeployment-Server-157 + +default_log_source: + eventType: Win-AppXDeployment-Server-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_appxpackaging_om.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_appxpackaging_om.yml new file mode 100644 index 00000000..30aba090 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_appxpackaging_om.yml @@ -0,0 +1,12 @@ +platform: FortiSiem +source: windows_appxpackaging_om + + +log_source: + eventType: + - Win-AppXDeployment-Server-400 + - Win-AppXDeployment-Server-401 + - Win-AppXDeployment-Server-157 + +default_log_source: + eventType: Win-AppXDeployment-Server-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_bits_client.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_bits_client.yml new file mode 100644 index 00000000..0366c9bb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_bits_client.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_bits_client + + +log_source: + eventType: + - Win-App-Microsoft-Windows-Bits-Client-16403 + +default_log_source: + eventType: Win-App-Microsoft-Windows-Bits-Client-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_codeintegrity_operational.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_codeintegrity_operational.yml new file mode 100644 index 00000000..29f38f1f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_codeintegrity_operational.yml @@ -0,0 +1,12 @@ +platform: FortiSiem +source: windows_codeintegrity_operational + + +log_source: + eventType: + - Win-CodeIntegrity-Operational-3023 + - Win-CodeIntegrity-Operational-3033 + - Win-CodeIntegrity-Operational-3077 + +default_log_source: + eventType: Win-CodeIntegrity-Operational-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_diagnosis_scripted.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_diagnosis_scripted.yml new file mode 100644 index 00000000..eac2b48b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_diagnosis_scripted.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_diagnosis_scripted + + +log_source: + eventType: + - Win-Diagnostics-Scripted-101 + +default_log_source: + eventType: Win-Diagnostics-Scripted* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_dns_client.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_dns_client.yml new file mode 100644 index 00000000..f8c6ced2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_dns_client.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_dns_client + + +log_source: + eventType: + - Win-DNS-Client-3008 + +default_log_source: + eventType: Win-DNS-Client-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_dns_query.yml new file mode 100644 index 00000000..335940e2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_dns_query.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_dns_query + + +log_source: + eventType: + - Win-Sysmon-22-DNS-Query + +default_log_source: + eventType: Win-Sysmon-22-DNS-Query diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_driver_load.yml new file mode 100644 index 00000000..3333ddb7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_driver_load.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_driver_load + + +log_source: + eventType: + - Win-Sysmon-6-Driver-Loaded + +default_log_source: + eventType: Win-Sysmon-6-Driver-Loaded diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_file_block.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_file_block.yml new file mode 100644 index 00000000..f27a3dd6 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_file_block.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_file_block + + +log_source: + eventType: + - Win-Sysmon-27-FileBlockExecutable + +default_log_source: + eventType: Win-Sysmon-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_file_event.yml new file mode 100644 index 00000000..d51e2082 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_file_event.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_file_event + + +log_source: + eventType: + - Win-Sysmon-11-FileCreate + +default_log_source: + eventType: Win-Sysmon-11-FileCreate diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_firewall_as.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_firewall_as.yml new file mode 100644 index 00000000..c794a5a2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_firewall_as.yml @@ -0,0 +1,14 @@ +platform: FortiSiem +source: windows_firewall_as + + +log_source: + eventType: + - Win-Firewall-AS-2002 + - Win-Firewall-AS-2083 + - Win-Firewall-AS-2003 + - Win-Firewall-AS-2082 + - Win-Firewall-AS-2008 + +default_log_source: + eventType: Win-Firewall-AS-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_image_load.yml new file mode 100644 index 00000000..817295a5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_image_load.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_image_load + + +log_source: + eventType: + - Win-Sysmon-7-Image-Loaded + +default_log_source: + eventType: Win-Sysmon-7-Image-Loaded diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_msexchange_management.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_msexchange_management.yml new file mode 100644 index 00000000..f9b2fc5c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_msexchange_management.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_msexchange_management + + +log_source: + eventType: + - Win-MSExchange-Management-6 + +default_log_source: + eventType: Win-MSExchange-Management-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_network_connection.yml new file mode 100644 index 00000000..e4423aaf --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_network_connection.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_network_connection + + +log_source: + eventType: + - Win-Sysmon-3-Network-Connect* + +default_log_source: + eventType: Win-Sysmon-3-Network-Connect* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_openssh.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_openssh.yml new file mode 100644 index 00000000..28ccf6b3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_openssh.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_openssh + + +log_source: + eventType: + - Win-OpenSSH-4 + +default_log_source: + eventType: Win-OpenSSH-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_powershell.yml new file mode 100644 index 00000000..4c673fbc --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_powershell.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_powershell + + +log_source: + eventType: + - Win-PowerShell-* + +default_log_source: + eventType: Win-PowerShell-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_powershell_classic.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_powershell_classic.yml new file mode 100644 index 00000000..eaebbb89 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_powershell_classic.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_powershell_classic + + +log_source: + eventType: + - Win-PowerShell-* + +default_log_source: + eventType: Win-PowerShell-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_process_access.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_process_access.yml new file mode 100644 index 00000000..c6587f80 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_process_access.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_process_access + + +log_source: + eventType: + - Win-Sysmon-10-ProcessAccess + +default_log_source: + eventType: Win-Sysmon-10-ProcessAccess diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_process_creation.yml new file mode 100644 index 00000000..d89fc796 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_process_creation.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_process_creation + + +log_source: + eventType: + - Win-Sysmon-1-Create-Process + +default_log_source: + eventType: Win-Sysmon-1-Create-Process diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_process_termination.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_process_termination.yml new file mode 100644 index 00000000..eb0beb95 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_process_termination.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_process_termination + + +log_source: + eventType: + - Win-Sysmon-5-Process-Terminated + +default_log_source: + eventType: Win-Sysmon-5-Process-Terminated diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_provider_name.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_provider_name.yml new file mode 100644 index 00000000..1212585b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_provider_name.yml @@ -0,0 +1,12 @@ +platform: FortiSiem +source: windows_provider_name + + +log_source: + eventType: + - Win-PrintService-Setup-Succeeded + - Win-PrintService-Remove-Printer-Failed + - Win-PrintService-Publish-Printer-Failed + +default_log_source: + eventType: Win-PrintService-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_registry_event.yml new file mode 100644 index 00000000..c3cdb11b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_registry_event.yml @@ -0,0 +1,12 @@ +platform: FortiSiem +source: windows_registry_event + + +log_source: + eventType: + - Win-Sysmon-12-Registry-* + - Win-Sysmon-13-Registry-* + - Win-Sysmon-14-Registry-* + +default_log_source: + eventType: Win-Sysmon-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_security.yml new file mode 100644 index 00000000..50598307 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_security.yml @@ -0,0 +1,13 @@ +platform: FortiSiem +source: windows_security + + +log_source: + eventType: + - Win-Security-4768-success + - Win-Security-4768-failure + - Win-Security-4769-failure + - Win-Security-4769-success + +default_log_source: + eventType: Win-Security-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_security_mitigations.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_security_mitigations.yml new file mode 100644 index 00000000..3be71e9c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_security_mitigations.yml @@ -0,0 +1,11 @@ +platform: FortiSiem +source: windows_security_mitigations + + +log_source: + eventType: + - Win-Security-Mitigation-11 + - Win-Security-Mitigation-12 + +default_log_source: + eventType: Win-Security-Mitigation-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_shell_core.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_shell_core.yml new file mode 100644 index 00000000..fee9af04 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_shell_core.yml @@ -0,0 +1,10 @@ +platform: FortiSiem +source: windows_shell_core + + +log_source: + eventType: + - Win-Shell-Core-28115 + +default_log_source: + eventType: Win-Shell-Core-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_sysmon.yml new file mode 100644 index 00000000..71dc40a8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_sysmon.yml @@ -0,0 +1,16 @@ +platform: FortiSiem +source: windows_sysmon + + +log_source: + eventType: + - Win-Sysmon-2-FileCreation-Time-Changed + - Win-Sysmon-8-CreateRemoteThread + - Win-Sysmon-9-RawAccessRead + - Win-Sysmon-15-FileCreateStreamHash + - Win-Sysmon-16-Config-State-Changed + - Win-Sysmon-17-PipeCreated + - Win-Sysmon-18-PipeConnected + +default_log_source: + eventType: Win-Sysmon-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_system.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_system.yml new file mode 100644 index 00000000..7997e07e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_system.yml @@ -0,0 +1,12 @@ +platform: FortiSiem +source: windows_system + + +log_source: + eventType: + - Win-System-Microsoft-Windows-Eventlog-104 + - Win-System-Service-Control-Manager-7036-* + - Win-System-Service-Control-Manager-7045 + +default_log_source: + eventType: Win-System-* diff --git a/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_wmi_event.yml new file mode 100644 index 00000000..fc0c41f2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/forti_siem/windows_wmi_event.yml @@ -0,0 +1,12 @@ +platform: FortiSiem +source: windows_wmi_event + + +log_source: + eventType: + - Win-Sysmon-19-WMI-Event-Filter-Activity + - Win-Sysmon-20-WMI-Event-Consumer-Activity + - Win-Sysmon-21-WMI-Event-ConsumerToFilter-Activity + +default_log_source: + eventType: Win-Sysmon-* diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/graylog/aws_cloudtrail.yml new file mode 100644 index 00000000..d52c7870 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/aws_cloudtrail.yml @@ -0,0 +1,29 @@ +platform: Graylog +source: aws_cloudtrail + + +field_mapping: + eventSource: event.provider + eventName: event.action + AdditionalEventData: AdditionalEventData + additionalEventData.MFAUsed: additionalEventData.MFAUsed + errorCode: errorCode + errorMessage: errorMessage + eventType: eventType + requestParameters: requestParameters + requestParameters.attribute: requestParameters.attribute + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP + requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort + requestParameters.userData: requestParameters.userData + responseElements: responseElements + responseElements.ConsoleLogin: responseElements.ConsoleLogin + responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword + responseElements.publiclyAccessible: responseElements.publiclyAccessible + status: status + terminatingRuleId: terminatingRuleId + userAgent: userAgent + userIdentity.arn: userIdentity.arn + userIdentity.principalId: userIdentity.principalId + userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type + userIdentity.type: userIdentity.type + userIdentity.userName: userIdentity.userName diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/graylog/aws_eks.yml new file mode 100644 index 00000000..ccf031da --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/aws_eks.yml @@ -0,0 +1,19 @@ +platform: Graylog +source: aws_eks + + +field_mapping: + annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy + aws_node_type: aws_node_type + objectRef.namespace: objectRef.namespace + objectRef.resource: objectRef.resource + objectRef.subresource: objectRef.subresource + requestObject.rules.resources: requestObject.rules.resources + requestObject.rules.verbs: requestObject.rules.verbs + requestObject.spec.containers.image: requestObject.spec.containers.image + requestURI: requestURI + stage: stage + user.groups: user.groups + user.username: user.username + verb: verb diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/azure_AzureDiagnostics.yml b/uncoder-core/app/translator/mappings/platforms/graylog/azure_AzureDiagnostics.yml new file mode 100644 index 00000000..535855f1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/azure_AzureDiagnostics.yml @@ -0,0 +1,7 @@ +platform: Graylog +source: azure_AzureDiagnostics + + +field_mapping: + ResultDescription: ResultDescription + Category: Category diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/azure_BehaviorAnalytics.yml b/uncoder-core/app/translator/mappings/platforms/graylog/azure_BehaviorAnalytics.yml new file mode 100644 index 00000000..172ce072 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/azure_BehaviorAnalytics.yml @@ -0,0 +1,14 @@ +platform: Graylog +source: azure_BehaviorAnalytics + + +field_mapping: + ActionType: ActionType + ActivityInsights: ActivityInsights + ActivityType: ActivityType + EventSource: EventSource + DevicesInsights: DevicesInsights + RiskDetail: RiskDetail + UsersInsights: UsersInsights + UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount + UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/azure_aadnoninteractiveusersigninlogs.yml b/uncoder-core/app/translator/mappings/platforms/graylog/azure_aadnoninteractiveusersigninlogs.yml new file mode 100644 index 00000000..5421cd39 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/azure_aadnoninteractiveusersigninlogs.yml @@ -0,0 +1,7 @@ +platform: Graylog +source: azure_aadnoninteractiveusersigninlogs + + +field_mapping: + UserAgent: UserAgent + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/graylog/azure_azureactivity.yml new file mode 100644 index 00000000..9b10bdd1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/azure_azureactivity.yml @@ -0,0 +1,16 @@ +platform: Graylog +source: azure_azureactivity + + +field_mapping: + ActivityStatus: ActivityStatus + ActivityStatusValue: ActivityStatusValue + ActivitySubstatusValue: ActivitySubstatusValue + Authorization: Authorization + Category: Category + CategoryValue: CategoryValue + OperationName: OperationName + OperationNameValue: OperationNameValue + ResourceId: ResourceId + ResourceProviderValue: ResourceProviderValue + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/graylog/azure_azuread.yml new file mode 100644 index 00000000..599eaaec --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/azure_azuread.yml @@ -0,0 +1,17 @@ +platform: Graylog +source: azure_azuread + + +field_mapping: + ActivityDisplayName: event.action + Category: azure.auditlogs.properties.category + LoggedByService: azure.auditlogs.properties.logged_by_service + Result: event.outcome + OperationName: OperationName + TargetResources: TargetResources + AADOperationType: AADOperationType + InitiatedBy: InitiatedBy + ResultReason: ResultReason + Status: Status + Status.errorCode: Status.errorCode + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/azure_m365.yml b/uncoder-core/app/translator/mappings/platforms/graylog/azure_m365.yml new file mode 100644 index 00000000..436a2b2b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/azure_m365.yml @@ -0,0 +1,17 @@ +platform: Graylog +source: azure_m365 + + +field_mapping: + ClientInfoString: ClientInfoString + LogonError: LogonError + ModifiedProperties: ModifiedProperties + OfficeObjectId: OfficeObjectId + OfficeWorkload: OfficeWorkload + Operation: Operation + Parameters: Parameters + RecordType: RecordType + ResultStatus: ResultStatus + SourceFileExtension: SourceFileExtension + SourceFileName: SourceFileName + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/azure_signinlogs.yml b/uncoder-core/app/translator/mappings/platforms/graylog/azure_signinlogs.yml new file mode 100644 index 00000000..eead8072 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/azure_signinlogs.yml @@ -0,0 +1,23 @@ +platform: Graylog +source: azure_signinlogs + + +field_mapping: + AppDisplayName: AppDisplayName + AppId: AppId + AuthenticationRequirement: AuthenticationRequirement + Category: Category + ConditionalAccessStatus: ConditionalAccessStatus + DeviceDetail: DeviceDetail + IsInteractive: IsInteractive + NetworkLocationDetails: NetworkLocationDetails + ResourceDisplayName: ResourceDisplayName + ResourceIdentity: ResourceIdentity + ResultDescription: ResultDescription + ResultType: ResultType + Status.errorCode: Status.errorCode + Status: Status + Status.failureReason: Status.failureReason + TokenIssuerType: TokenIssuerType + UserAgent: UserAgent + UserPrincipalName: UserPrincipalName diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/default.yml b/uncoder-core/app/translator/mappings/platforms/graylog/default.yml new file mode 100644 index 00000000..9a0041de --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/default.yml @@ -0,0 +1,6 @@ +platform: Graylog +source: default + + +default_log_source: + index: "" diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/dns.yml b/uncoder-core/app/translator/mappings/platforms/graylog/dns.yml new file mode 100644 index 00000000..98962061 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/dns.yml @@ -0,0 +1,11 @@ +platform: Graylog +source: dns + + +field_mapping: + dns-query: dns.question.name + parent-domain: + - url.registered_domain + - destination.registered_domain + dns-answer: dns.answers.name + dns-record: dns.question.type diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/firewall.yml b/uncoder-core/app/translator/mappings/platforms/graylog/firewall.yml new file mode 100644 index 00000000..3ea75ec8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/firewall.yml @@ -0,0 +1,13 @@ +platform: Graylog +source: firewall + + +field_mapping: + src-ip: + - source.address + - source.ip + src-port: source.port + dst-ip: + - destination.address + - destination.ip + dst-port: destination.port diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/gcp_gcp.audit.yml b/uncoder-core/app/translator/mappings/platforms/graylog/gcp_gcp.audit.yml new file mode 100644 index 00000000..313cf6ca --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/gcp_gcp.audit.yml @@ -0,0 +1,15 @@ +platform: Graylog +source: gcp_gcp.audit + + +field_mapping: + backupConfiguration.enabled: backupConfiguration.enabled + gcp.audit.method_name: gcp.audit.method_name + jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction + jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue + message: message + methodName: methodName + protoPayload.methodName: protoPayload.methodName + protoPayload.request.canIpForward: protoPayload.request.canIpForward + protoPayload.serviceName: protoPayload.serviceName + serviceName: serviceName diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/gcp_pubsub.yml b/uncoder-core/app/translator/mappings/platforms/graylog/gcp_pubsub.yml new file mode 100644 index 00000000..3deb08a0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/gcp_pubsub.yml @@ -0,0 +1,10 @@ +platform: Graylog +source: gcp_pubsub + + +field_mapping: + data.protoPayload.methodName: data.protoPayload.methodName + data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member + data.resource.type: data.resource.type diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/linux_auditd.yml b/uncoder-core/app/translator/mappings/platforms/graylog/linux_auditd.yml new file mode 100644 index 00000000..e34e6a51 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/linux_auditd.yml @@ -0,0 +1,10 @@ +platform: Graylog +source: linux_auditd + + +field_mapping: + a0: process.args + a1: process.args + a2: process.args + a3: process.args + exe: process.executable diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/graylog/linux_dns_query.yml new file mode 100644 index 00000000..85d34836 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/linux_dns_query.yml @@ -0,0 +1,9 @@ +platform: Graylog +source: linux_dns_query + + +field_mapping: + Image: process.executable + User: user.name + QueryName: dns.question.name + QueryResults: dns.answers diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/graylog/linux_process_creation.yml new file mode 100644 index 00000000..076b4fd7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/linux_process_creation.yml @@ -0,0 +1,9 @@ +platform: Graylog +source: linux_process_creation + + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/graylog/macos_dns_query.yml new file mode 100644 index 00000000..0ac7ad07 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/macos_dns_query.yml @@ -0,0 +1,9 @@ +platform: Graylog +source: macos_dns_query + + +field_mapping: + Image: process.executable + User: user.name + QueryName: dns.question.name + QueryResults: dns.answers diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/graylog/macos_network_connection.yml new file mode 100644 index 00000000..faa62008 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/macos_network_connection.yml @@ -0,0 +1,12 @@ +platform: Graylog +source: macos_network_connection + + +field_mapping: + Image: process.executable + DestinationHostname: destination.domain + DestinationIp: destination.ip + DestinationPort: destination.port + SourceIp: source.ip + SourcePort: source.port + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/graylog/macos_process_creation.yml new file mode 100644 index 00000000..3fab3123 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/macos_process_creation.yml @@ -0,0 +1,9 @@ +platform: Graylog +source: macos_process_creation + + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/graylog/okta_okta.yml new file mode 100644 index 00000000..0481c85c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/okta_okta.yml @@ -0,0 +1,27 @@ +platform: Graylog +source: okta_okta + + +field_mapping: + client.user.id: okta.actor.id + source.user.id: okta.actor.id + User: okta.actor.type + alternateId: okta.actor.alternate_id + client.user.full_name: okta.actor.display_name + source.user.full_name: okta.actor.display_name + related.user: okta.actor.display_name + client.ip: okta.client.ip + source.ip: okta.client.ip + user_agent.original: okta.client.user_agent.raw_user_agent + userAgent.os: okta.client.user_agent.os + userAgent.browser: okta.client.user_agent.browser + client.zone: okta.client.zone + client.device: okta.client.device + client.id: okta.client.id + event.action: okta.event_type + outcome.reason: okta.outcome.reason + event.outcome: okta.event.outcome + client.as.number: okta.security_context.as.number + client.as.organization.name: okta.security_context.as.organization.name + client.domain: okta.security_context.isp + source.domain: okta.security_context.domain diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/proxy.yml b/uncoder-core/app/translator/mappings/platforms/graylog/proxy.yml new file mode 100644 index 00000000..2cfacb63 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/proxy.yml @@ -0,0 +1,24 @@ +platform: Graylog +source: proxy + + +field_mapping: + c-uri: url.original + c-useragent: user_agent.original + cs-method: http.request.method + cs-bytes: http.request.body.bytes + cs-cookie-vars: http.cookie_vars + c-uri-extension: url.extension + c-uri-query: url.query + cs-cookie: http.cookie_vars + cs-host: + - url.domain + - destination.domain + + cs-referrer: http.request.referrer + cs-version: http.version + r-dns: + - destination.domain + - url.domain + sc-status: http.response.status_code + post-body: http.post_body diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/webserver.yml b/uncoder-core/app/translator/mappings/platforms/graylog/webserver.yml new file mode 100644 index 00000000..0475f0ea --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/webserver.yml @@ -0,0 +1,23 @@ +platform: Graylog +source: webserver + + +field_mapping: + c-uri: url.original + c-useragent: user_agent.original + cs-method: http.request.method + cs-bytes: http.request.body.bytes + cs-cookie-vars: http.cookie_vars + c-uri-extension: url.extension + c-uri-query: url.query + cs-cookie: http.cookie_vars + cs-host: + - url.domain + - destination.domain + cs-referrer: http.request.referrer + cs-version: http.version + r-dns: + - destination.domain + - url.domain + sc-status: http.response.status_code + post-body: http.post_body diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_application.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_application.yml new file mode 100644 index 00000000..7451e139 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_application.yml @@ -0,0 +1,10 @@ +platform: Graylog +source: windows_application + + +field_mapping: + Provider_Name: winlog.provider_name + EventID: winlog.event_id + Data: winlog.event_data.Data + Message: winlog.event_data.Message + Level: winlog.opcode diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_bits_client.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_bits_client.yml new file mode 100644 index 00000000..72e1c4dc --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_bits_client.yml @@ -0,0 +1,9 @@ +platform: Graylog +source: windows_bits_client + + +field_mapping: + LocalName: winlog.event_data.LocalName + EventID: winlog.event_id + RemoteName: winlog.event_data.RemoteName + processPath: winlog.event_data.processPath diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_create_remote_thread.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_create_remote_thread.yml new file mode 100644 index 00000000..8df765eb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_create_remote_thread.yml @@ -0,0 +1,10 @@ +platform: Graylog +source: windows_create_remote_thread + + +field_mapping: + SourceImage: winlog.event_data.SourceImage + TargetImage: winlog.event_data.TargetImage + StartModule: winlog.event_data.StartModule + StartAddress: winlog.event_data.StartAddress + StartFunction: winlog.event_data.StartFunction diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_create_stream_hash.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_create_stream_hash.yml new file mode 100644 index 00000000..0c3b8db1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_create_stream_hash.yml @@ -0,0 +1,7 @@ +platform: Graylog +source: windows_create_stream_hash + + +field_mapping: + Image: winlog.event_data.Image + Hashes: winlog.event_data.Hashes diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_dns_query.yml new file mode 100644 index 00000000..2ee09b72 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_dns_query.yml @@ -0,0 +1,9 @@ +platform: Graylog +source: windows_dns_query + + +field_mapping: + Image: winlog.event_data.Image + User: winlog.event_data.User + QueryName: winlog.event_data.QueryName + QueryResults: winlog.event_data.QueryResults diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_driver_load.yml new file mode 100644 index 00000000..bb5ebafa --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_driver_load.yml @@ -0,0 +1,8 @@ +platform: Graylog +source: windows_driver_load + + +field_mapping: + ImageLoaded: winlog.event_data.ImageLoaded + ImagePath: winlog.event_data.ImagePath + Hashes: winlog.event_data.Hashes diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_file_event.yml new file mode 100644 index 00000000..bc2bc062 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_file_event.yml @@ -0,0 +1,10 @@ +platform: Graylog +source: windows_file_event + + +field_mapping: + CreationUtcTime: winlog.event_data.CreationUtcTime + Image: winlog.event_data.Image + PreviousCreationUtcTime: winlog.event_data.PreviousCreationUtcTime + TargetFilename: winlog.event_data.TargetFilename + User: winlog.event_data.User diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_image_load.yml new file mode 100644 index 00000000..2e0b0197 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_image_load.yml @@ -0,0 +1,10 @@ +platform: Graylog +source: windows_image_load + + +field_mapping: + Image: winlog.event_data.Image + ImageLoaded: winlog.event_data.ImageLoaded + SignatureStatus: winlog.event_data.SignatureStatus + OriginalFileName: winlog.event_data.OriginalFileName + Signed: winlog.event_data.Signed diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_ldap_debug.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_ldap_debug.yml new file mode 100644 index 00000000..3d955950 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_ldap_debug.yml @@ -0,0 +1,7 @@ +platform: Graylog +source: windows_ldap_debug + + +field_mapping: + EventID: winlog.event_id + SearchFilter: winlog.event_data.SearchFilter diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_network_connection.yml new file mode 100644 index 00000000..d71c09bc --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_network_connection.yml @@ -0,0 +1,12 @@ +platform: Graylog +source: windows_network_connection + + +field_mapping: + Image: winlog.event_data.Image + DestinationHostname: winlog.event_data.DestinationHostname + DestinationIp: winlog.event_data.DestinationIp + DestinationPort: winlog.event_data.DestinationPort + SourceIp: winlog.event_data.SourceIp + SourcePort: winlog.event_data.SourcePort + Initiated: winlog.event_data.Initiated diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_ntlm.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_ntlm.yml new file mode 100644 index 00000000..ffa7f571 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_ntlm.yml @@ -0,0 +1,9 @@ +platform: Graylog +source: windows_ntlm + + +field_mapping: + WorkstationName: winlog.event_data.WorkstationName + ProcessName: winlog.event_data.ProcessName + EventID: winlog.event_id + TargetName: winlog.event_data.TargetName diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_pipe_created.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_pipe_created.yml new file mode 100644 index 00000000..2c3396bb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_pipe_created.yml @@ -0,0 +1,7 @@ +platform: Graylog +source: windows_pipe_created + + +field_mapping: + PipeName: winlog.event_data.PipeName + Image: winlog.event_data.Image diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_powershell.yml new file mode 100644 index 00000000..03b9d0ca --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_powershell.yml @@ -0,0 +1,13 @@ +platform: Graylog +source: windows_powershell + + +field_mapping: + EventID: winlog.event_id + ScriptBlockText: winlog.event_data.ScriptBlockText + Payload: winlog.event_data.Payload + HostApplication: winlog.event_data.HostApplication + ContextInfo: winlog.event_data.ContextInfo + HostName: winlog.event_data.HostName + EngineVersion: winlog.event_data.EngineVersion + HostVersion: winlog.event_data.HostVersion diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_process_access.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_process_access.yml new file mode 100644 index 00000000..a3fb330d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_process_access.yml @@ -0,0 +1,9 @@ +platform: Graylog +source: windows_process_access + + +field_mapping: + TargetImage: winlog.event_data.TargetImage + SourceImage: winlog.event_data.SourceImage + CallTrace: winlog.event_data.CallTrace + GrantedAccess: winlog.event_data.GrantedAccess diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_process_creation.yml new file mode 100644 index 00000000..cce3e416 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_process_creation.yml @@ -0,0 +1,15 @@ +platform: Graylog +source: windows_process_creation + + +field_mapping: + CommandLine: winlog.event_data.CommandLine + CurrentDirectory: winlog.event_data.CurrentDirectory + Hashes: winlog.event_data.Hashes + Image: winlog.event_data.Image + IntegrityLevel: winlog.event_data.IntegrityLevel + ParentCommandLine: winlog.event_data.ParentCommandLine + ParentImage: winlog.event_data.ParentImage + ParentUser: winlog.event_data.ParentUser + Product: winlog.event_data.Product + User: winlog.event_data.User diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_raw_access_thread.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_raw_access_thread.yml new file mode 100644 index 00000000..d61a1c6c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_raw_access_thread.yml @@ -0,0 +1,7 @@ +platform: Graylog +source: windows_raw_access_thread + + +field_mapping: + Image: winlog.event_data.Image + Device: winlog.event_data.Device diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_registry_event.yml new file mode 100644 index 00000000..b7736fbb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_registry_event.yml @@ -0,0 +1,10 @@ +platform: Graylog +source: windows_registry_event + + +field_mapping: + TargetObject: winlog.event_data.TargetObject + Image: winlog.event_data.Image + Details: winlog.event_data.Details + EventType: winlog.event_data.EventType + User: winlog.event_data.User diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_security.yml new file mode 100644 index 00000000..5c38b0f8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_security.yml @@ -0,0 +1,139 @@ +platform: Graylog +source: windows_security + + +field_mapping: + EventID: winlog.event_id + ParentImage: winlog.event_data.ParentImage + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + Channel: winlog.channel + ComputerName: winlog.computer_name + EventType: winlog.event_data.EventType + FailureReason: winlog.event_data.FailureCode + FileName: winlog.event_data.FileName + GrantedAccess: winlog.event_data.GrantedAccess + Hashes: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + IpAddress: winlog.event_data.IpAddress + IpPort: winlog.event_data.IpPort + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + LinkName: winlog.event_data.LinkName + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + Path: winlog.event_data.Path + CommandLine: winlog.event_data.ProcessCommandLine + OldUacValue: winlog.event_data.OldUacValue + CertIssuerName: winlog.event_data.CertIssuerName + SubStatus: winlog.event_data.SubStatus + DisplayName: winlog.event_data.DisplayName + TaskContent: winlog.event_data.TaskContent + ServiceSid: winlog.event_data.ServiceSid + CertThumbprint: winlog.event_data.CertThumbprint + ClassName: winlog.event_data.ClassName + NotificationPackageName: winlog.event_data.NotificationPackageName + NewSd: winlog.event_data.NewSd + TestSigning: winlog.event_data.TestSigning + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ServiceAccount: winlog.event_data.ServiceAccount + AttributeValue: winlog.event_data.AttributeValue + SessionName: winlog.event_data.SessionName + TaskName: winlog.event_data.TaskName + ObjectDN: winlog.event_data.ObjectDN + TemplateContent: winlog.event_data.TemplateContent + NewTemplateContent: winlog.event_data.NewTemplateContent + SourcePort: winlog.event_data.SourcePort + PasswordLastSet: winlog.event_data.PasswordLastSet + PrivilegeList: winlog.event_data.PrivilegeList + DeviceDescription: winlog.event_data.DeviceDescription + TargetServerName: winlog.event_data.TargetServerName + NewTargetUserName: winlog.event_data.NewTargetUserName + OperationType: winlog.event_data.OperationType + DestPort: winlog.event_data.DestPort + ServiceStartType: winlog.event_data.ServiceStartType + OldTargetUserName: winlog.event_data.OldTargetUserName + UserPrincipalName: winlog.event_data.UserPrincipalName + Accesses: winlog.event_data.Accesses + DnsHostName: winlog.event_data.DnsHostName + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + AuditSourceName: winlog.event_data.AuditSourceName + Workstation: winlog.event_data.Workstation + DestAddress: winlog.event_data.DestAddress + PreAuthType: winlog.event_data.PreAuthType + SecurityPackageName: winlog.event_data.SecurityPackageName + SubjectLogonId: winlog.event_data.SubjectLogonId + NewUacValue: winlog.event_data.NewUacValue + EnabledPrivilegeList: winlog.event_data.EnabledPrivilegeList + RelativeTargetName: winlog.event_data.RelativeTargetName + CertSerialNumber: winlog.event_data.CertSerialNumber + SidHistory: winlog.event_data.SidHistory + TargetLogonId: winlog.event_data.TargetLogonId + KernelDebug: winlog.event_data.KernelDebug + CallerProcessName: winlog.event_data.CallerProcessName + ProcessName: winlog.event_data.ProcessName + Properties: winlog.event_data.Properties + UserAccountControl: winlog.event_data.UserAccountControl + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + SecurityDescriptor: winlog.event_data.SecurityDescriptor + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + NewValue: winlog.event_data.NewValue + Source: winlog.event_data.Source + Status: winlog.event_data.Status + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + SourceAddr: winlog.event_data.SourceAddr + SourceAddress: winlog.event_data.SourceAddress + TargetName: winlog.event_data.TargetName + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + TargetDomainName: winlog.event_data.TargetDomainName + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + ObjectServer: winlog.event_data.ObjectServer + TargetUserSid: winlog.event_data.TargetUserSid + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + WorkstationName: winlog.event_data.WorkstationName + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + LayerRTID: winlog.event_data.LayerRTID + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + ParentProcessName: winlog.event_data.ParentProcessName + Service: winlog.event_data.Service diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_sysmon.yml new file mode 100644 index 00000000..eaccde12 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_sysmon.yml @@ -0,0 +1,55 @@ +platform: Graylog +source: windows_sysmon + + +field_mapping: + CommandLine: winlog.event_data.CommandLine + Image: winlog.event_data.Image + ParentImage: winlog.event_data.ParentImage + EventID: winlog.event_id + CallTrace: winlog.event_data.CallTrace + Company: winlog.event_data.Company + CurrentDirectory: winlog.event_data.CurrentDirectory + Description: winlog.event_data.Description + DestinationHostname: winlog.event_data.DestinationHostname + DestinationIp: winlog.event_data.DestinationIp + DestinationIsIpv6: winlog.event_data.DestinationIsIpv6 + DestinationPort: winlog.event_data.DestinationPort + DestinationPortName: winlog.event_data.DestinationPortName + Hashes: winlog.event_data.Hashes + Initiated: winlog.event_data.Initiated + IntegrityLevel: winlog.event_data.IntegrityLevel + ParentCommandLine: winlog.event_data.ParentCommandLine + Product: winlog.event_data.Product + Protocol: winlog.event_data.Protocol + RuleName: winlog.event_data.RuleName + SourceHostname: winlog.event_data.SourceHostname + SourceIp: winlog.event_data.SourceIp + SourceIsIpv6: winlog.event_data.SourceIsIpv6 + SourcePort: winlog.event_data.SourcePort + SourcePortName: winlog.event_data.SourcePortName + TargetFilename: winlog.event_data.TargetFilename + User: winlog.event_data.User + OriginalFileName: winlog.event_data.OriginalFileName + Signed: winlog.event_data.Signed + Signature: winlog.event_data.Signature + SignatureStatus: winlog.event_data.SignatureStatus + TargetObject: winlog.event_data.TargetObject + Details: winlog.event_data.Details + QueryName: winlog.event_data.QueryName + QueryResults: winlog.event_data.QueryResults + QueryStatus: winlog.event_data.QueryStatus + IsExecutable: winlog.event_data.IsExecutable + PipeName: winlog.event_data.PipeName + ImageLoaded: winlog.event_data.ImageLoaded + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + SourceImage: winlog.event_data.SourceImage + StartModule: winlog.event_data.StartModule + TargetImage: winlog.event_data.TargetImage + Device: winlog.event_data.Device + ProcessID: winlog.event_data.ProcessID + FileVersion: winlog.event_data.FileVersion + StartAddress: winlog.event_data.StartAddress + StartFunction: winlog.event_data.StartFunction + EventType: winlog.event_data.EventType diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_system.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_system.yml new file mode 100644 index 00000000..50bce90e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_system.yml @@ -0,0 +1,19 @@ +platform: Graylog +source: windows_system + + +field_mapping: + EventID: winlog.event_id + AccountName: winlog.event_data.AccountName + ImagePath: winlog.event_data.ImagePath + ServiceName: winlog.event_data.ServiceName + ServiceType: winlog.event_data.ServiceType + StartType: winlog.event_data.StartType + Provider_Name: winlog.provider_name + Origin: winlog.event_data.Origin + HiveName: winlog.event_data.HiveName + Caption: winlog.event_data.Caption + param1: winlog.event_data.param1 + param2: winlog.event_data.param2 + Channel: winlog.channel + DeviceName: winlog.event_data.DeviceName diff --git a/uncoder-core/app/translator/mappings/platforms/graylog/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/graylog/windows_wmi_event.yml new file mode 100644 index 00000000..bc25e741 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/graylog/windows_wmi_event.yml @@ -0,0 +1,7 @@ +platform: Graylog +source: windows_wmi_event + + +field_mapping: + Destination: winlog.event_data.Destination + EventID: winlog.event_id diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/hunters/aws_cloudtrail.yml new file mode 100644 index 00000000..ff73fdf9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/aws_cloudtrail.yml @@ -0,0 +1,31 @@ +platform: Hunters +source: aws_cloudtrail + + +default_log_source: + table: table_name + +field_mapping: + eventSource: eventSource + eventName: eventName + AdditionalEventData: additionalEventData.MFAUsed + errorCode: errorCode + errorMessage: errorMessage + eventType: eventType + requestParameters: requestParameters + requestParameters.attribute: requestParameters.attribute + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP + requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort + requestParameters.userData: requestParameters.userData + responseElements: responseElements + responseElements.ConsoleLogin: responseElements.ConsoleLogin + responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword + responseElements.publiclyAccessible: responseElements.publiclyAccessible + status: status + terminatingRuleId: terminatingRuleId + userAgent: userAgent + userIdentity.arn: userIdentity.arn + userIdentity.principalId: userIdentity.principalId + userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type + userIdentity.type: userIdentity.type + userIdentity.userName: userIdentity.userName diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/hunters/aws_eks.yml new file mode 100644 index 00000000..a94b4142 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/aws_eks.yml @@ -0,0 +1,22 @@ +platform: Hunters +source: aws_eks + + +default_log_source: + table: table_name + +field_mapping: + annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy + aws_node_type: aws_node_type + objectRef.namespace: objectRef.namespace + objectRef.resource: objectRef.resource + objectRef.subresource: objectRef.subresource + requestObject.rules.resources: requestObject.rules.resources + requestObject.rules.verbs: requestObject.rules.verbs + requestObject.spec.containers.image: requestObject.spec.containers.image + requestURI: requestURI + stage: stage + user.groups: user.groups + user.username: user.username + verb: verb diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/azure_AzureDiagnostics.yml b/uncoder-core/app/translator/mappings/platforms/hunters/azure_AzureDiagnostics.yml new file mode 100644 index 00000000..69b0be38 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/azure_AzureDiagnostics.yml @@ -0,0 +1,10 @@ +platform: Hunters +source: azure_AzureDiagnostics + + +default_log_source: + table: table_name + +field_mapping: + ResultDescription: ResultDescription + Category: Category diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/azure_BehaviorAnalytics.yml b/uncoder-core/app/translator/mappings/platforms/hunters/azure_BehaviorAnalytics.yml new file mode 100644 index 00000000..34505850 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/azure_BehaviorAnalytics.yml @@ -0,0 +1,17 @@ +platform: Hunters +source: azure_BehaviorAnalytics + + +default_log_source: + table: table_name + +field_mapping: + ActionType: ActionType + ActivityInsights: ActivityInsights + ActivityType: ActivityType + EventSource: EventSource + DevicesInsights: DevicesInsights + RiskDetail: RiskDetail + UsersInsights: UsersInsights + UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount + UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/azure_aadnoninteractiveusersigninlogs.yml b/uncoder-core/app/translator/mappings/platforms/hunters/azure_aadnoninteractiveusersigninlogs.yml new file mode 100644 index 00000000..dece4af7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/azure_aadnoninteractiveusersigninlogs.yml @@ -0,0 +1,10 @@ +platform: Hunters +source: azure_aadnoninteractiveusersigninlogs + + +default_log_source: + table: table_name + +field_mapping: + UserAgent: UserAgent + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/hunters/azure_azureactivity.yml new file mode 100644 index 00000000..2982eda2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/azure_azureactivity.yml @@ -0,0 +1,19 @@ +platform: Hunters +source: azure_azureactivity + + +default_log_source: + table: table_name + +field_mapping: + ActivityStatus: ActivityStatus + ActivityStatusValue: ActivityStatusValue + ActivitySubstatusValue: ActivitySubstatusValue + Authorization: Authorization + Category: Category + CategoryValue: CategoryValue + OperationName: OperationName + OperationNameValue: OperationNameValue + ResourceId: ResourceId + ResourceProviderValue: ResourceProviderValue + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/hunters/azure_azuread.yml new file mode 100644 index 00000000..2de79fba --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/azure_azuread.yml @@ -0,0 +1,20 @@ +platform: Hunters +source: azure_azuread + + +default_log_source: + table: table_name + +field_mapping: + ActivityDisplayName: ActivityDisplayName + Category: Category + LoggedByService: LoggedByService + Result: Result + OperationName: OperationName + TargetResources: TargetResources + AADOperationType: AADOperationType + InitiatedBy: InitiatedBy + ResultReason: ResultReason + Status: Status + Status.errorCode: Status.errorCode + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/azure_m365.yml b/uncoder-core/app/translator/mappings/platforms/hunters/azure_m365.yml new file mode 100644 index 00000000..602b0c63 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/azure_m365.yml @@ -0,0 +1,20 @@ +platform: Hunters +source: azure_m365 + + +default_log_source: + table: table_name + +field_mapping: + ClientInfoString: ClientInfoString + LogonError: LogonError + ModifiedProperties: ModifiedProperties + OfficeObjectId: OfficeObjectId + OfficeWorkload: OfficeWorkload + Operation: Operation + Parameters: Parameters + RecordType: RecordType + ResultStatus: ResultStatus + SourceFileExtension: SourceFileExtension + SourceFileName: SourceFileName + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/azure_signinlogs.yml b/uncoder-core/app/translator/mappings/platforms/hunters/azure_signinlogs.yml new file mode 100644 index 00000000..f70bb5ae --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/azure_signinlogs.yml @@ -0,0 +1,26 @@ +platform: Hunters +source: azure_signinlogs + + +default_log_source: + table: table_name + +field_mapping: + AppDisplayName: AppDisplayName + AppId: AppId + AuthenticationRequirement: AuthenticationRequirement + Category: Category + ConditionalAccessStatus: ConditionalAccessStatus + DeviceDetail: DeviceDetail + IsInteractive: IsInteractive + NetworkLocationDetails: NetworkLocationDetails + ResourceDisplayName: ResourceDisplayName + ResourceIdentity: ResourceIdentity + ResultDescription: ResultDescription + ResultType: ResultType + Status.errorCode: Status.errorCode + Status: Status + Status.failureReason: Status.failureReason + TokenIssuerType: TokenIssuerType + UserAgent: UserAgent + UserPrincipalName: UserPrincipalName diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/default.yml b/uncoder-core/app/translator/mappings/platforms/hunters/default.yml new file mode 100644 index 00000000..d465c4b2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/default.yml @@ -0,0 +1,6 @@ +platform: Hunters +source: default + + +default_log_source: + table: table_name diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/dns.yml b/uncoder-core/app/translator/mappings/platforms/hunters/dns.yml new file mode 100644 index 00000000..e876dc40 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/dns.yml @@ -0,0 +1,12 @@ +platform: Hunters +source: dns + + +default_log_source: + table: table_name + +field_mapping: + dns-query: dns-query + parent-domain: parent-domain + dns-answer: dns-answer + dns-record: dns-record diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/firewall.yml b/uncoder-core/app/translator/mappings/platforms/hunters/firewall.yml new file mode 100644 index 00000000..ec028f10 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/firewall.yml @@ -0,0 +1,12 @@ +platform: Hunters +source: firewall + + +default_log_source: + table: table_name + +field_mapping: + src-ip: src-ip + src-port: src-port + dst-ip: dst-ip + dst-port: dst-port diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/gcp_gcp.audit.yml b/uncoder-core/app/translator/mappings/platforms/hunters/gcp_gcp.audit.yml new file mode 100644 index 00000000..0e3e5555 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/gcp_gcp.audit.yml @@ -0,0 +1,18 @@ +platform: Hunters +source: gcp_gcp.audit + + +default_log_source: + table: table_name + +field_mapping: + backupConfiguration.enabled: backupConfiguration.enabled + gcp.audit.method_name: gcp.audit.method_name + jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction + jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue + message: message + methodName: methodName + protoPayload.methodName: protoPayload.methodName + protoPayload.request.canIpForward: protoPayload.request.canIpForward + protoPayload.serviceName: protoPayload.serviceName + serviceName: serviceName diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/gcp_pubsub.yml b/uncoder-core/app/translator/mappings/platforms/hunters/gcp_pubsub.yml new file mode 100644 index 00000000..e784c7f3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/gcp_pubsub.yml @@ -0,0 +1,13 @@ +platform: Hunters +source: gcp_pubsub + + +default_log_source: + table: table_name + +field_mapping: + data.protoPayload.methodName: data.protoPayload.methodName + data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member + data.resource.type: data.resource.type diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/linux_auditd.yml b/uncoder-core/app/translator/mappings/platforms/hunters/linux_auditd.yml new file mode 100644 index 00000000..f172b1f7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/linux_auditd.yml @@ -0,0 +1,13 @@ +platform: Hunters +source: linux_auditd + + +default_log_source: + table: table_name + +field_mapping: + a0: a0 + a1: a1 + a2: a2 + a3: a3 + exe: exe diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/hunters/linux_dns_query.yml new file mode 100644 index 00000000..95b133f1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/linux_dns_query.yml @@ -0,0 +1,12 @@ +platform: Hunters +source: linux_dns_query + + +default_log_source: + table: table_name + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/hunters/linux_file_event.yml new file mode 100644 index 00000000..aa7489d1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/linux_file_event.yml @@ -0,0 +1,9 @@ +platform: Hunters +source: linux_file_event + +default_log_source: + table: table_name + +field_mapping: + Image: Image + TargetFilename: TargetFilename diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/hunters/linux_network_connection.yml new file mode 100644 index 00000000..f1dec9c4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/linux_network_connection.yml @@ -0,0 +1,15 @@ +platform: Hunters +source: linux_network_connection + + +default_log_source: + table: table_name + +field_mapping: + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + ParentImage: ParentImage diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/hunters/linux_process_creation.yml new file mode 100644 index 00000000..d8170cef --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/linux_process_creation.yml @@ -0,0 +1,12 @@ +platform: Hunters +source: linux_process_creation + + +default_log_source: + table: table_name + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/hunters/macos_dns_query.yml new file mode 100644 index 00000000..a38dae1d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/macos_dns_query.yml @@ -0,0 +1,12 @@ +platform: Hunters +source: macos_dns_query + + +default_log_source: + table: table_name + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/hunters/macos_network_connection.yml new file mode 100644 index 00000000..b1cadb4c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/macos_network_connection.yml @@ -0,0 +1,15 @@ +platform: Hunters +source: macos_network_connection + + +default_log_source: + table: table_name + +field_mapping: + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + ParentImage: ParentImage diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/hunters/macos_process_creation.yml new file mode 100644 index 00000000..46c1bc7c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/macos_process_creation.yml @@ -0,0 +1,12 @@ +platform: Hunters +source: macos_process_creation + + +default_log_source: + table: table_name + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/hunters/okta_okta.yml new file mode 100644 index 00000000..707fad2e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/okta_okta.yml @@ -0,0 +1,30 @@ +platform: Hunters +source: okta_okta + + +default_log_source: + table: table_name + +field_mapping: + client.user.id: client.user.id + source.user.id: source.user.id + User: User + alternateId: alternateId + client.user.full_name: client.user.full_name + source.user.full_name: source.user.full_name + related.user: related.user + client.ip: client.ip + source.ip: source.ip + user_agent.original: user_agent.original + userAgent.os: userAgent.os + userAgent.browser: userAgent.browser + client.zone: client.zone + client.device: client.device + client.id: client.id + event.action: event.action + outcome.reason: outcome.reason + event.outcome: event.outcome + client.as.number: client.as.number + client.as.organization.name: client.as.organization.name + client.domain: client.domain + source.domain: source.domain diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/proxy.yml b/uncoder-core/app/translator/mappings/platforms/hunters/proxy.yml new file mode 100644 index 00000000..2386f389 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/proxy.yml @@ -0,0 +1,22 @@ +platform: Hunters +source: proxy + + +default_log_source: + table: table_name + +field_mapping: + c-uri: c-uri + c-useragent: c-useragent + cs-method: cs-method + cs-bytes: cs-bytes + cs-cookie-vars: cs-cookie-vars + c-uri-extension: c-uri-extension + c-uri-query: c-uri-query + cs-cookie: cs-cookie + cs-host: cs-host + cs-referrer: cs-referrer + cs-version: cs-version + r-dns: r-dns + sc-status: sc-status + post-body: post-body diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/webserver.yml b/uncoder-core/app/translator/mappings/platforms/hunters/webserver.yml new file mode 100644 index 00000000..0b7c4a80 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/webserver.yml @@ -0,0 +1,22 @@ +platform: Hunters +source: webserver + + +default_log_source: + table: table_name + +field_mapping: + c-uri: c-uri + c-useragent: c-useragent + cs-method: cs-method + cs-bytes: cs-bytes + cs-cookie-vars: cs-cookie-vars + c-uri-extension: c-uri-extension + c-uri-query: c-uri-query + cs-cookie: cs-cookie + cs-host: cs-host + cs-referrer: cs-referrer + cs-version: cs-version + r-dns: r-dns + sc-status: sc-status + post-body: post-body diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_application.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_application.yml new file mode 100644 index 00000000..6e68cc20 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_application.yml @@ -0,0 +1,22 @@ +platform: Hunters +source: windows_application + +default_log_source: + table: table_name + +field_mapping: + EventID: EventID + src_ip: src_ip + source: source + additional_information: additional_information + EventData: EventData + Channel: Channel + statement: statement + Faulting application path: Faulting application path + object_name: object_name + class_type: class_type + action_id: action_id + Provider_Name: Provider_Name + Data: Data + Message: Message + Level: Level diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_bits_client.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_bits_client.yml new file mode 100644 index 00000000..1355aa11 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_bits_client.yml @@ -0,0 +1,12 @@ +platform: Hunters +source: windows_bits_client + + +default_log_source: + table: table_name + +field_mapping: + LocalName: LocalName + EventID: EventID + RemoteName: RemoteName + processPath: processPath diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_dns_query.yml new file mode 100644 index 00000000..dc0a766d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_dns_query.yml @@ -0,0 +1,12 @@ +platform: Hunters +source: windows_dns_query + + +default_log_source: + table: table_name + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_driver_load.yml new file mode 100644 index 00000000..fd677986 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_driver_load.yml @@ -0,0 +1,11 @@ +platform: Hunters +source: windows_driver_load + + +default_log_source: + table: table_name + +field_mapping: + ImageLoaded: ImageLoaded + ImagePath: ImagePath + Hashes: Hashes diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_image_load.yml new file mode 100644 index 00000000..d5b05711 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_image_load.yml @@ -0,0 +1,13 @@ +platform: Hunters +source: windows_image_load + + +default_log_source: + table: table_name + +field_mapping: + Image: Image + ImageLoaded: ImageLoaded + SignatureStatus: SignatureStatus + OriginalFileName: OriginalFileName + Signed: Signed diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_ldap_debug.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_ldap_debug.yml new file mode 100644 index 00000000..613d6d0d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_ldap_debug.yml @@ -0,0 +1,10 @@ +platform: Hunters +source: windows_ldap_debug + + +default_log_source: + table: table_name + +field_mapping: + EventID: EventID + SearchFilter: SearchFilter diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_network_connection.yml new file mode 100644 index 00000000..997706fb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_network_connection.yml @@ -0,0 +1,15 @@ +platform: Hunters +source: windows_network_connection + + +default_log_source: + table: table_name + +field_mapping: + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + Initiated: Initiated diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_ntlm.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_ntlm.yml new file mode 100644 index 00000000..4d549131 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_ntlm.yml @@ -0,0 +1,12 @@ +platform: Hunters +source: windows_ntlm + + +default_log_source: + table: table_name + +field_mapping: + ProcessName: ProcessName + TargetName: TargetName + WorkstationName: WorkstationName + EventID: EventID diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_powershell.yml new file mode 100644 index 00000000..7fdb87b2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_powershell.yml @@ -0,0 +1,16 @@ +platform: Hunters +source: windows_powershell + + +default_log_source: + table: table_name + +field_mapping: + EventID: EventID + CommandLine: CommandLine + ScriptBlockText: ScriptBlockText + Payload: Payload + HostApplication: HostApplication + ContextInfo: ContextInfo + HostName: HostName + EngineVersion: EngineVersion diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_process_creation.yml new file mode 100644 index 00000000..9a575b1e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_process_creation.yml @@ -0,0 +1,18 @@ +platform: Hunters +source: windows_process_creation + + +default_log_source: + table: table_name + +field_mapping: + CommandLine: CommandLine + CurrentDirectory: CurrentDirectory + Hashes: Hashes + Image: Image + IntegrityLevel: IntegrityLevel + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage + ParentUser: ParentUser + Product: Product + User: User diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_security.yml new file mode 100644 index 00000000..569d03fe --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_security.yml @@ -0,0 +1,142 @@ +platform: Hunters +source: windows_security + + +default_log_source: + table: table_name + +field_mapping: + EventID: EventID + ParentImage: ParentImage + AccessMask: AccessMask + AccountName: AccountName + AllowedToDelegateTo: AllowedToDelegateTo + AttributeLDAPDisplayName: AttributeLDAPDisplayName + AuditPolicyChanges: AuditPolicyChanges + AuthenticationPackageName: AuthenticationPackageName + CallingProcessName: CallingProcessName + Channel: Channel + ComputerName: ComputerName + EventType: EventType + FailureReason: FailureReason + FileName: FileName + GrantedAccess: GrantedAccess + Hashes: Hashes + HiveName: HiveName + IpAddress: IpAddress + IpPort: IpPort + KeyLength: KeyLength + LogonProcessName: LogonProcessName + LogonType: LogonType + LinkName: LinkName + MemberName: MemberName + MemberSid: MemberSid + NewProcessName: NewProcessName + ObjectClass: ObjectClass + ObjectType: ObjectType + ObjectValueName: ObjectValueName + Path: Path + CommandLine: CommandLine + OldUacValue: OldUacValue + CertIssuerName: CertIssuerName + SubStatus: SubStatus + DisplayName: DisplayName + TaskContent: TaskContent + ServiceSid: ServiceSid + CertThumbprint: CertThumbprint + ObjectName: ObjectName + ClassName: ClassName + NotificationPackageName: NotificationPackageName + NewSd: NewSd + TestSigning: TestSigning + TargetInfo: TargetInfo + ParentProcessId: ParentProcessId + AccessList: AccessList + GroupMembership: GroupMembership + FilterName: FilterName + ChangeType: ChangeType + LayerName: LayerName + ServiceAccount: ServiceAccount + ClientProcessId: ClientProcessId + AttributeValue: AttributeValue + SessionName: SessionName + TaskName: TaskName + ObjectDN: ObjectDN + TemplateContent: TemplateContent + NewTemplateContent: NewTemplateContent + SourcePort: SourcePort + PasswordLastSet: PasswordLastSet + PrivilegeList: PrivilegeList + DeviceDescription: DeviceDescription + TargetServerName: TargetServerName + NewTargetUserName: NewTargetUserName + OperationType: OperationType + DestPort: DestPort + ServiceStartType: ServiceStartType + OldTargetUserName: OldTargetUserName + UserPrincipalName: UserPrincipalName + Accesses: Accesses + DnsHostName: DnsHostName + DisableIntegrityChecks: DisableIntegrityChecks + AuditSourceName: AuditSourceName + Workstation: Workstation + DestAddress: DestAddress + PreAuthType: PreAuthType + SecurityPackageName: SecurityPackageName + SubjectLogonId: SubjectLogonId + NewUacValue: NewUacValue + EnabledPrivilegeList: EnabledPrivilegeList + RelativeTargetName: RelativeTargetName + CertSerialNumber: CertSerialNumber + SidHistory: SidHistory + TargetLogonId: TargetLogonId + KernelDebug: KernelDebug + CallerProcessName: CallerProcessName + Properties: Properties + UserAccountControl: UserAccountControl + RegistryValue: RegistryValue + SecurityID: SecurityID + ServiceFileName: ServiceFileName + SecurityDescriptor: SecurityDescriptor + ServiceName: ServiceName + ShareName: ShareName + NewValue: NewValue + Source: Source + Status: Status + SubjectDomainName: SubjectDomainName + SubjectUserName: SubjectUserName + SubjectUserSid: SubjectUserSid + SourceAddr: SourceAddr + SourceAddress: SourceAddress + TargetName: TargetName + ServicePrincipalNames: ServicePrincipalNames + TargetDomainName: TargetDomainName + TargetSid: TargetSid + TargetUserName: TargetUserName + ObjectServer: ObjectServer + TargetUserSid: TargetUserSid + TicketEncryptionType: TicketEncryptionType + TicketOptions: TicketOptions + WorkstationName: WorkstationName + TransmittedServices: TransmittedServices + AuthenticationAlgorithm: AuthenticationAlgorithm + LayerRTID: LayerRTID + BSSID: BSSID + BSSType: BSSType + CipherAlgorithm: CipherAlgorithm + ConnectionId: ConnectionId + ConnectionMode: ConnectionMode + InterfaceDescription: InterfaceDescription + InterfaceGuid: InterfaceGuid + OnexEnabled: OnexEnabled + PHYType: PHYType + ProfileName: ProfileName + SSID: SSID + Domain: Domain + ServiceType: ServiceType + SourceName: SourceName + StartType: StartType + UserID: UserID + ParentProcessName: ParentProcessName + Service: Service + ProcessName: ProcessName diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_sysmon.yml new file mode 100644 index 00000000..c1b8c593 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_sysmon.yml @@ -0,0 +1,58 @@ +platform: Hunters +source: windows_sysmon + + +default_log_source: + table: table_name + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentImage: ParentImage + EventID: EventID + CallTrace: CallTrace + Company: Company + CurrentDirectory: CurrentDirectory + Description: Description + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationIsIpv6: DestinationIsIpv6 + DestinationPort: DestinationPort + DestinationPortName: DestinationPortName + Hashes: Hashes + Initiated: Initiated + IntegrityLevel: IntegrityLevel + ParentCommandLine: ParentCommandLine + Product: Product + Protocol: Protocol + RuleName: RuleName + SourceHostname: SourceHostname + SourceIp: SourceIp + SourceIsIpv6: SourceIsIpv6 + SourcePort: SourcePort + SourcePortName: SourcePortName + TargetFilename: TargetFilename + User: User + OriginalFileName: OriginalFileName + Signed: Signed + Signature: Signature + SignatureStatus: SignatureStatus + TargetObject: TargetObject + Details: Details + QueryName: QueryName + QueryResults: QueryResults + QueryStatus: QueryStatus + IsExecutable: IsExecutable + PipeName: PipeName + ImageLoaded: ImageLoaded + ImagePath: ImagePath + Imphash: Imphash + SourceImage: SourceImage + StartModule: StartModule + TargetImage: TargetImage + Device: Device + ProcessID: ProcessID + FileVersion: FileVersion + StartAddress: StartAddress + StartFunction: StartFunction + EventType: EventType diff --git a/uncoder-core/app/translator/mappings/platforms/hunters/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/hunters/windows_wmi_event.yml new file mode 100644 index 00000000..10f776e3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/hunters/windows_wmi_event.yml @@ -0,0 +1,10 @@ +platform: Hunters +source: windows_wmi_event + + +default_log_source: + table: table_name + +field_mapping: + Destination: Destination + EventID: EventID diff --git a/uncoder-core/app/translator/mappings/platforms/logrhythm_axon/default.yml b/uncoder-core/app/translator/mappings/platforms/logrhythm_axon/default.yml new file mode 100644 index 00000000..864f0420 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logrhythm_axon/default.yml @@ -0,0 +1,309 @@ +platform: LogRhythm Axon +source: default + + +field_mapping: + EventID: vendor_information.id + Channel: general_information.log_source.type_name + ComputerName: origin.host.name + FileName: object.file.name + ProcessId: object.process.id + Image: object.process.name + AccountEmail: unattributed.account.email_address + ContextInfo: general_information.raw_message + CurrentDirectory: object.process.path + ParentProcessId: object.process.parent_process.id + ParentImage: object.process.parent_process.path + ParentCommandLine: object.process.parent_process.command_line + TargetFilename: object.file.name + SourceIp: origin.host.ip_address.value + SourceHostname: origin.host.name + SourcePort: origin.host.network_port.value + DestinationIp: target.host.ip_address.value + DestinationHostname: + - target.host.name + - target.host.domain + DestinationPort: target.host.network_port.value + DestinationPortName: action.network.protocol.name + ImageLoaded: object.file.path + SignatureStatus: object.process.signature.status + SourceProcessId: object.process.id + SourceImage: object.process.name + Device: object.process.path + Destination: object.process.name + QueryName: action.dns.query + QueryStatus: action.dns.result + CommandName: object.process.command_line + CommandPath: object.process.path + HostApplication: object.script.command_line + HostName: origin.host.name + ScriptName: object.script.name + ScriptBlockText: object.script.command_line + ScriptBlockId: object.script.id + Application: object.process.name + ClientAddress: origin.host.ip_address.value + ClientName: origin.host.domain.name + DestAddress: target.host.ip_address.value + DestPort: target.host.network_port.value + IpAddress: origin.host.ip_address.value + IpPort: origin.host.network_port.value + NewProcessId: object.process.id + NewProcessName: object.process.name + ParentProcessName: object.process.parent_process.name + ProcessName: object.process.name + SourceAddress: origin.host.ip_address.value + WorkstationName: origin.host.name + destination.port: target.host.network_port.value + dst: target.host.ip_address.value + dst_ip: target.host.ip_address.value + dst_port: target.host.network_port.value + network_application: + - action.network.protocol.name + - object.url.protocol + network_protocol: action.network.protocol.name + proto: action.network.protocol.name + src: origin.host.ip_address.value + src_ip: origin.host.ip_address.value + src_port: origin.host.network_port.value + action: action.command + mqtt_action: action.command + smb_action: action.command + tunnel_action: action.command + arg: object.process.command_args + ftp_arg: object.process.command_args + mysql_arg: object.process.command_args + pop3_arg: object.process.command_args + client: origin.host.ip_address.value + command: action.command + ftp_command: action.command + irc_command: action.command + pop3_command: action.command + duration: action.duration + from: origin.account.email_address + kerberos_from: origin.account.email_address + smtp_from: origin.account.email_address + method: action.network.http_method + http_method: action.network.http_method + sip_method: action.network.http_method + name: object.file.name + smb_files_name: object.file.name + software_name: object.file.name + weird_name: object.file.name + path: object.file.path + smb_mapping_path: object.file.path + smb_files_path: object.file.path + smtp_files_path: object.file.path + password: object.file.name + reply_to: target.account.email_address + response_body_len: action.network.byte_information.received + request_body_len: action.network.byte_information.sent + rtt: action.duration + status_code: action.result.code + known_certs_subject: object.certificate.subject + sip_subject: object.email_message.subject + smtp_subject: object.email_message.subject + ssl_subject: object.certificate.subject + username: origin.account.name + uri: object.url.path + user: origin.account.name + user_agent: action.user_agent + http_user_agent: action.user_agent + gquic_user_agent: action.user_agent + sip_user_agent: action.user_agent + smtp_user_agent: action.user_agent + version: object.file.version + gquic_version: object.file.version + http_version: object.file.version + ntp_version: object.file.version + socks_version: object.file.version + snmp_version: object.file.version + ssh_version: object.file.version + tls_version: object.file.version + answer: action.dns.result + question_length: action.network.byte_information.total + record_type: action.dns.record_type + parent_domain: target.host.domain + cs-bytes: action.network.byte_information.received + r-dns: target.host.domain + sc-bytes: action.network.byte_information.received + sc-status: action.result.code + c-uri: object.url.complete + c-uri-extension: object.url.type + c-uri-query: object.url.query + c-uri-stem: object.url.path + c-useragent: action.user_agent + cs-host: + - target.host.name + - target.host.domain + cs-method: action.network.http_method + cs-version: object.file.version + uid: action.session.id + endpoint: origin.host.name + domain: target.host.domain + host_name: target.host.name + client_fqdn: origin.host.name + requested_addr: target.host.ip_address.value + server_addr: target.host.ip_address.value + qtype: action.dns.record_type + qtype_name: action.dns.record_type + query: action.dns.query + rcode_name: action.dns.result + md5: unattributed.hash.md5 + sha1: unattributed.hash.sha1 + sha256: unattributed.hash.sha256 + sha512: unattributed.hash.sha512 + filename: object.file.name + host: + - unattributed.host.name + - unattributed.host.ip_address.value + domainname: unattributed.host.name + hostname: unattributed.host.name + server_nb_computer_name: unattributed.host.name + server_tree_name: unattributed.host.name + server_dns_computer_name: unattributed.host.name + machine: unattributed.host.name + os: origin.host.os.platform + mac: unattributed.host.mac_address + result: + - action.result.message + - action.result.code + - action.result.reason + mailfrom: origin.account.email_address + rcptto: target.account.email_address + second_received: target.account.email_address + server_name: unattributed.host.name + c-ip: origin.host.ip_address.value + cs-uri: object.url.path + cs-uri-query: object.url.query + cs-uri-stem: object.url.path + clientip: origin.host.ip_address.value + clientIP: origin.host.ip_address.value + dest_domain: + - target.host.name + - target.host.domain + dest_ip: target.host.ip_address.value + dest_port: target.host.network_port.value + agent.version: object.file.version + destination.hostname: + - target.host.name + - target.host.domain + DestinationAddress: + - target.host.name + - target.host.domain + - target.host.ip_address.value + DestinationIP: target.host.ip_address.value + dst-ip: target.host.ip_address.value + dstip: target.host.ip_address.value + dstport: target.host.ip_address.value + Host: target.host.name + HostVersion: object.file.version + http_host: + - target.host.name + - target.host.domain + - target.host.ip_address.value + http_uri: object.url.path + http_url: object.url.complete + http.request.url-query-params: object.url.query + HttpMethod: action.network.http_method + in_url: object.url.path + post_url_parameter: object.url.path + Request_Url: object.url.complete + request_url: object.url.complete + request_URL: object.url.complete + RequestUrl: object.url.complete + resource.url: object.url.path + resource.URL: object.url.path + sc_status: action.result.code + sender_domain: + - target.host.name + - target.host.domain + service.response_code: action.result.code + source: + - origin.host.name + - origin.host.domain.name + - origin.host.ip_address.value + SourceAddr: origin.host.ip_address.value + SourceIP: origin.host.ip_address.value + SourceNetworkAddress: origin.host.ip_address.value + srcip: origin.host.ip_address.value + Status: action.result.code + status: action.result.code + url: object.url.path + URL: object.url.path + url_query: object.url.query + url.query: object.url.query + uri_path: object.url.path + user_agent.name: action.user_agent + user-agent: action.user_agent + User-Agent: action.user_agent + useragent: action.user_agent + UserAgent: action.user_agent + User_Agent: action.user_agent + web_dest: + - target.host.name + - target.host.domain + - target.host.ip_address.value + - object.url.domain + web.dest: + - target.host.name + - target.host.domain + - target.host.ip_address.value + - object.url.domain + Web.dest: + - target.host.name + - target.host.domain + - target.host.ip_address.value + - object.url.domain + web.host: + - target.host.name + - target.host.domain + - target.host.ip_address.value + - object.url.domain + Web.host: + - target.host.name + - target.host.domain + - target.host.ip_address.value + - object.url.domain + web_method: action.network.http_method + Web_method: action.network.http_method + web.method: action.network.http_method + Web.method: action.network.http_method + web_src: origin.host.ip_address.value + web_status: action.result.code + Web_status: action.result.code + web.status: action.result.code + Web.status: action.result.code + web_uri: object.url.path + web_url: object.url.complete + destination.ip: target.host.ip_address.value + source.ip: origin.host.ip_address.value + source.port: origin.host.ip_address.value + Computer: + - target.host.name + - target.host.domain + - target.host.ip_address.value + OriginalFileName: object.file.name + User: origin.account.name + EventType: action.command + TargetObject: + - object.registry_object.key + - object.registry_object.path + - object.resource.name + CommandLine: object.process.command_line + type: + - action.command + - action.type + - action.session.type + a0: + - object.process.command_line + - object.process.command_args + - object.process.name + cs-user-agent: action.user_agent + blocked: + - action.message + - action.result.reason + cs-ip: origin.host.ip_address.value + SubjectLogonId: action.session.id + SubjectUserName: origin.account.name + SubjectUserSid: origin.account.id + SubjectDomainName: origin.account.domain diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/logscale/aws_cloudtrail.yml new file mode 100644 index 00000000..15b24317 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/aws_cloudtrail.yml @@ -0,0 +1,30 @@ +platform: LogScale +source: aws_cloudtrail + + + +field_mapping: + eventSource: event.provider + eventName: event.action + AdditionalEventData: AdditionalEventData + additionalEventData.MFAUsed: additionalEventData.MFAUsed + errorCode: errorCode + errorMessage: errorMessage + eventType: eventType + requestParameters: requestParameters + requestParameters.attribute: requestParameters.attribute + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP + requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort + requestParameters.userData: requestParameters.userData + responseElements: responseElements + responseElements.ConsoleLogin: responseElements.ConsoleLogin + responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword + responseElements.publiclyAccessible: responseElements.publiclyAccessible + status: status + terminatingRuleId: terminatingRuleId + userAgent: userAgent + userIdentity.arn: userIdentity.arn + userIdentity.principalId: userIdentity.principalId + userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type + userIdentity.type: userIdentity.type + userIdentity.userName: userIdentity.userName diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/logscale/aws_eks.yml new file mode 100644 index 00000000..dc59d369 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/aws_eks.yml @@ -0,0 +1,20 @@ +platform: LogScale +source: aws_eks + + + +field_mapping: + annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy + aws_node_type: aws_node_type + objectRef.namespace: objectRef.namespace + objectRef.resource: objectRef.resource + objectRef.subresource: objectRef.subresource + requestObject.rules.resources: requestObject.rules.resources + requestObject.rules.verbs: requestObject.rules.verbs + requestObject.spec.containers.image: requestObject.spec.containers.image + requestURI: requestURI + stage: stage + user.groups: user.groups + user.username: user.username + verb: verb diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/azure_AzureDiagnostics.yml b/uncoder-core/app/translator/mappings/platforms/logscale/azure_AzureDiagnostics.yml new file mode 100644 index 00000000..5f2dcf48 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/azure_AzureDiagnostics.yml @@ -0,0 +1,8 @@ +platform: LogScale +source: azure_AzureDiagnostics + + + +field_mapping: + ResultDescription: ResultDescription + Category: Category \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/azure_BehaviorAnalytics.yml b/uncoder-core/app/translator/mappings/platforms/logscale/azure_BehaviorAnalytics.yml new file mode 100644 index 00000000..5d4e2189 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/azure_BehaviorAnalytics.yml @@ -0,0 +1,15 @@ +platform: LogScale +source: azure_BehaviorAnalytics + + + +field_mapping: + ActionType: ActionType + ActivityInsights: ActivityInsights + ActivityType: ActivityType + EventSource: EventSource + DevicesInsights: DevicesInsights + RiskDetail: RiskDetail + UsersInsights: UsersInsights + UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount + UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/azure_aadnoninteractiveusersigninlogs.yml b/uncoder-core/app/translator/mappings/platforms/logscale/azure_aadnoninteractiveusersigninlogs.yml new file mode 100644 index 00000000..5ad81783 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/azure_aadnoninteractiveusersigninlogs.yml @@ -0,0 +1,8 @@ +platform: LogScale +source: azure_aadnoninteractiveusersigninlogs + + + +field_mapping: + UserAgent: UserAgent + Type: Type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/logscale/azure_azureactivity.yml new file mode 100644 index 00000000..e62ad8a7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/azure_azureactivity.yml @@ -0,0 +1,17 @@ +platform: LogScale +source: azure_azureactivity + + + +field_mapping: + ActivityStatus: ActivityStatus + ActivityStatusValue: ActivityStatusValue + ActivitySubstatusValue: ActivitySubstatusValue + Authorization: Authorization + Category: Category + CategoryValue: CategoryValue + OperationName: OperationName + OperationNameValue: OperationNameValue + ResourceId: ResourceId + ResourceProviderValue: ResourceProviderValue + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/logscale/azure_azuread.yml new file mode 100644 index 00000000..d9c401bf --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/azure_azuread.yml @@ -0,0 +1,18 @@ +platform: LogScale +source: azure_azuread + + + +field_mapping: + ActivityDisplayName: event.action + Category: azure.auditlogs.properties.category + LoggedByService: azure.auditlogs.properties.logged_by_service + Result: event.outcome + OperationName: OperationName + TargetResources: TargetResources + AADOperationType: AADOperationType + InitiatedBy: InitiatedBy + ResultReason: ResultReason + Status: Status + Status.errorCode: Status.errorCode + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/azure_m365.yml b/uncoder-core/app/translator/mappings/platforms/logscale/azure_m365.yml new file mode 100644 index 00000000..91161cfb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/azure_m365.yml @@ -0,0 +1,18 @@ +platform: LogScale +source: azure_m365 + + + +field_mapping: + ClientInfoString: ClientInfoString + LogonError: LogonError + ModifiedProperties: ModifiedProperties + OfficeObjectId: OfficeObjectId + OfficeWorkload: OfficeWorkload + Operation: Operation + Parameters: Parameters + RecordType: RecordType + ResultStatus: ResultStatus + SourceFileExtension: SourceFileExtension + SourceFileName: SourceFileName + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/azure_signinlogs.yml b/uncoder-core/app/translator/mappings/platforms/logscale/azure_signinlogs.yml new file mode 100644 index 00000000..5b55f211 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/azure_signinlogs.yml @@ -0,0 +1,24 @@ +platform: LogScale +source: azure_signinlogs + + + +field_mapping: + AppDisplayName: AppDisplayName + AppId: AppId + AuthenticationRequirement: AuthenticationRequirement + Category: Category + ConditionalAccessStatus: ConditionalAccessStatus + DeviceDetail: DeviceDetail + IsInteractive: IsInteractive + NetworkLocationDetails: NetworkLocationDetails + ResourceDisplayName: ResourceDisplayName + ResourceIdentity: ResourceIdentity + ResultDescription: ResultDescription + ResultType: ResultType + Status.errorCode: Status.errorCode + Status: Status + Status.failureReason: Status.failureReason + TokenIssuerType: TokenIssuerType + UserAgent: UserAgent + UserPrincipalName: UserPrincipalName diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/default.yml b/uncoder-core/app/translator/mappings/platforms/logscale/default.yml new file mode 100644 index 00000000..347978fa --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/default.yml @@ -0,0 +1,6 @@ +platform: LogScale +source: default + + +default_log_source: + index: "" \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/dns.yml b/uncoder-core/app/translator/mappings/platforms/logscale/dns.yml new file mode 100644 index 00000000..975aec99 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/dns.yml @@ -0,0 +1,12 @@ +platform: LogScale +source: dns + + + +field_mapping: + dns-query: dns.question.name + parent-domain: + - url.registered_domain + - destination.registered_domain + dns-answer: dns.answers.name + dns-record: dns.question.type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/firewall.yml b/uncoder-core/app/translator/mappings/platforms/logscale/firewall.yml new file mode 100644 index 00000000..8d2d8db2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/firewall.yml @@ -0,0 +1,14 @@ +platform: LogScale +source: firewall + + + +field_mapping: + src-ip: + - source.address + - source.ip + src-port: source.port + dst-ip: + - destination.address + - destination.ip + dst-port: destination.port \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/gcp_gcp.audit.yml b/uncoder-core/app/translator/mappings/platforms/logscale/gcp_gcp.audit.yml new file mode 100644 index 00000000..001c74bc --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/gcp_gcp.audit.yml @@ -0,0 +1,16 @@ +platform: LogScale +source: gcp_gcp.audit + + + +field_mapping: + backupConfiguration.enabled: backupConfiguration.enabled + gcp.audit.method_name: gcp.audit.method_name + jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction + jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue + message: message + methodName: methodName + protoPayload.methodName: protoPayload.methodName + protoPayload.request.canIpForward: protoPayload.request.canIpForward + protoPayload.serviceName: protoPayload.serviceName + serviceName: serviceName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/gcp_pubsub.yml b/uncoder-core/app/translator/mappings/platforms/logscale/gcp_pubsub.yml new file mode 100644 index 00000000..0ba30634 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/gcp_pubsub.yml @@ -0,0 +1,11 @@ +platform: LogScale +source: gcp_pubsub + + + +field_mapping: + data.protoPayload.methodName: data.protoPayload.methodName + data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member + data.resource.type: data.resource.type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/linux_auditd.yml b/uncoder-core/app/translator/mappings/platforms/logscale/linux_auditd.yml new file mode 100644 index 00000000..ce47d122 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/linux_auditd.yml @@ -0,0 +1,11 @@ +platform: LogScale +source: linux_auditd + + + +field_mapping: + a0: process.args + a1: process.args + a2: process.args + a3: process.args + exe: process.executable diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/logscale/linux_dns_query.yml new file mode 100644 index 00000000..66444688 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/linux_dns_query.yml @@ -0,0 +1,10 @@ +platform: LogScale +source: linux_dns_query + + + +field_mapping: + Image: process.executable + User: user.name + QueryName: dns.question.name + QueryResults: dns.answers \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/logscale/linux_process_creation.yml new file mode 100644 index 00000000..48b8f224 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/linux_process_creation.yml @@ -0,0 +1,10 @@ +platform: LogScale +source: linux_process_creation + + + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/logscale/macos_dns_query.yml new file mode 100644 index 00000000..0c3daea9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/macos_dns_query.yml @@ -0,0 +1,10 @@ +platform: LogScale +source: macos_dns_query + + + +field_mapping: + Image: process.executable + User: user.name + QueryName: dns.question.name + QueryResults: dns.answers \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/logscale/macos_network_connection.yml new file mode 100644 index 00000000..4ac7b1be --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/macos_network_connection.yml @@ -0,0 +1,13 @@ +platform: LogScale +source: macos_network_connection + + + +field_mapping: + Image: process.executable + DestinationHostname: destination.domain + DestinationIp: destination.ip + DestinationPort: destination.port + SourceIp: source.ip + SourcePort: source.port + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/logscale/macos_process_creation.yml new file mode 100644 index 00000000..15bde398 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/macos_process_creation.yml @@ -0,0 +1,10 @@ +platform: LogScale +source: macos_process_creation + + + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/logscale/okta_okta.yml new file mode 100644 index 00000000..50d35393 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/okta_okta.yml @@ -0,0 +1,28 @@ +platform: LogScale +source: okta_okta + + + +field_mapping: + client.user.id: okta.actor.id + source.user.id: okta.actor.id + User: okta.actor.type + alternateId: okta.actor.alternate_id + client.user.full_name: okta.actor.display_name + source.user.full_name: okta.actor.display_name + related.user: okta.actor.display_name + client.ip: okta.client.ip + source.ip: okta.client.ip + user_agent.original: okta.client.user_agent.raw_user_agent + userAgent.os: okta.client.user_agent.os + userAgent.browser: okta.client.user_agent.browser + client.zone: okta.client.zone + client.device: okta.client.device + client.id: okta.client.id + event.action: okta.event_type + outcome.reason: okta.outcome.reason + event.outcome: okta.event.outcome + client.as.number: okta.security_context.as.number + client.as.organization.name: okta.security_context.as.organization.name + client.domain: okta.security_context.isp + source.domain: okta.security_context.domain diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/proxy.yml b/uncoder-core/app/translator/mappings/platforms/logscale/proxy.yml new file mode 100644 index 00000000..61e7916d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/proxy.yml @@ -0,0 +1,25 @@ +platform: LogScale +source: proxy + + + +field_mapping: + c-uri: url.original + c-useragent: user_agent.original + cs-method: http.request.method + cs-bytes: http.request.body.bytes + cs-cookie-vars: http.cookie_vars + c-uri-extension: url.extension + c-uri-query: url.query + cs-cookie: http.cookie_vars + cs-host: + - url.domain + - destination.domain + + cs-referrer: http.request.referrer + cs-version: http.version + r-dns: + - destination.domain + - url.domain + sc-status: http.response.status_code + post-body: http.post_body \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/webserver.yml b/uncoder-core/app/translator/mappings/platforms/logscale/webserver.yml new file mode 100644 index 00000000..a23c8ee7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/webserver.yml @@ -0,0 +1,24 @@ +platform: LogScale +source: webserver + + + +field_mapping: + c-uri: url.original + c-useragent: user_agent.original + cs-method: http.request.method + cs-bytes: http.request.body.bytes + cs-cookie-vars: http.cookie_vars + c-uri-extension: url.extension + c-uri-query: url.query + cs-cookie: http.cookie_vars + cs-host: + - url.domain + - destination.domain + cs-referrer: http.request.referrer + cs-version: http.version + r-dns: + - destination.domain + - url.domain + sc-status: http.response.status_code + post-body: http.post_body \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_application.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_application.yml new file mode 100644 index 00000000..a7cff027 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_application.yml @@ -0,0 +1,11 @@ +platform: LogScale +source: windows_application + + + +field_mapping: + Provider_Name: winlog.provider_name + EventID: winlog.event_id + Data: winlog.event_data.Data + Message: winlog.event_data.Message + Level: winlog.opcode \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_bits_client.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_bits_client.yml new file mode 100644 index 00000000..255dab8b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_bits_client.yml @@ -0,0 +1,10 @@ +platform: LogScale +source: windows_bits_client + + + +field_mapping: + LocalName: winlog.event_data.LocalName + EventID: winlog.event_id + RemoteName: winlog.event_data.RemoteName + processPath: winlog.event_data.processPath diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_create_remote_thread.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_create_remote_thread.yml new file mode 100644 index 00000000..b5984bb5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_create_remote_thread.yml @@ -0,0 +1,11 @@ +platform: LogScale +source: windows_create_remote_thread + + + +field_mapping: + SourceImage: winlog.event_data.SourceImage + TargetImage: winlog.event_data.TargetImage + StartModule: winlog.event_data.StartModule + StartAddress: winlog.event_data.StartAddress + StartFunction: winlog.event_data.StartFunction \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_create_stream_hash.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_create_stream_hash.yml new file mode 100644 index 00000000..08259a0e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_create_stream_hash.yml @@ -0,0 +1,8 @@ +platform: LogScale +source: windows_create_stream_hash + + + +field_mapping: + Image: winlog.event_data.Image + Hashes: winlog.event_data.Hashes \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_dns_query.yml new file mode 100644 index 00000000..554eb52f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_dns_query.yml @@ -0,0 +1,10 @@ +platform: LogScale +source: windows_dns_query + + + +field_mapping: + Image: winlog.event_data.Image + User: winlog.event_data.User + QueryName: winlog.event_data.QueryName + QueryResults: winlog.event_data.QueryResults \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_driver_load.yml new file mode 100644 index 00000000..ad8a0e97 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_driver_load.yml @@ -0,0 +1,9 @@ +platform: LogScale +source: windows_driver_load + + + +field_mapping: + ImageLoaded: winlog.event_data.ImageLoaded + ImagePath: winlog.event_data.ImagePath + Hashes: winlog.event_data.Hashes \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_file_event.yml new file mode 100644 index 00000000..e6704d5b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_file_event.yml @@ -0,0 +1,11 @@ +platform: LogScale +source: windows_file_event + + + +field_mapping: + CreationUtcTime: winlog.event_data.CreationUtcTime + Image: winlog.event_data.Image + PreviousCreationUtcTime: winlog.event_data.PreviousCreationUtcTime + TargetFilename: winlog.event_data.TargetFilename + User: winlog.event_data.User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_image_load.yml new file mode 100644 index 00000000..a1f5b38a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_image_load.yml @@ -0,0 +1,11 @@ +platform: LogScale +source: windows_image_load + + + +field_mapping: + Image: winlog.event_data.Image + ImageLoaded: winlog.event_data.ImageLoaded + SignatureStatus: winlog.event_data.SignatureStatus + OriginalFileName: winlog.event_data.OriginalFileName + Signed: winlog.event_data.Signed diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_ldap_debug.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_ldap_debug.yml new file mode 100644 index 00000000..f0d398b8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_ldap_debug.yml @@ -0,0 +1,8 @@ +platform: LogScale +source: windows_ldap_debug + + + +field_mapping: + EventID: winlog.event_id + SearchFilter: winlog.event_data.SearchFilter \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_network_connection.yml new file mode 100644 index 00000000..3fdc0a66 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_network_connection.yml @@ -0,0 +1,13 @@ +platform: LogScale +source: windows_network_connection + + + +field_mapping: + Image: winlog.event_data.Image + DestinationHostname: winlog.event_data.DestinationHostname + DestinationIp: winlog.event_data.DestinationIp + DestinationPort: winlog.event_data.DestinationPort + SourceIp: winlog.event_data.SourceIp + SourcePort: winlog.event_data.SourcePort + Initiated: winlog.event_data.Initiated \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_ntlm.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_ntlm.yml new file mode 100644 index 00000000..79faa582 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_ntlm.yml @@ -0,0 +1,10 @@ +platform: LogScale +source: windows_ntlm + + + +field_mapping: + WorkstationName: winlog.event_data.WorkstationName + ProcessName: winlog.event_data.ProcessName + EventID: winlog.event_id + TargetName: winlog.event_data.TargetName diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_pipe_created.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_pipe_created.yml new file mode 100644 index 00000000..5b21c6e8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_pipe_created.yml @@ -0,0 +1,8 @@ +platform: LogScale +source: windows_pipe_created + + + +field_mapping: + PipeName: winlog.event_data.PipeName + Image: winlog.event_data.Image \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_powershell.yml new file mode 100644 index 00000000..a24ce046 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_powershell.yml @@ -0,0 +1,14 @@ +platform: LogScale +source: windows_powershell + + + +field_mapping: + EventID: winlog.event_id + ScriptBlockText: winlog.event_data.ScriptBlockText + Payload: winlog.event_data.Payload + HostApplication: winlog.event_data.HostApplication + ContextInfo: winlog.event_data.ContextInfo + HostName: winlog.event_data.HostName + EngineVersion: winlog.event_data.EngineVersion + HostVersion: winlog.event_data.HostVersion \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_process_access.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_process_access.yml new file mode 100644 index 00000000..9a6a899f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_process_access.yml @@ -0,0 +1,10 @@ +platform: LogScale +source: windows_process_access + + + +field_mapping: + TargetImage: winlog.event_data.TargetImage + SourceImage: winlog.event_data.SourceImage + CallTrace: winlog.event_data.CallTrace + GrantedAccess: winlog.event_data.GrantedAccess \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_process_creation.yml new file mode 100644 index 00000000..c3ca438c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_process_creation.yml @@ -0,0 +1,16 @@ +platform: LogScale +source: windows_process_creation + + + +field_mapping: + CommandLine: winlog.event_data.CommandLine + CurrentDirectory: winlog.event_data.CurrentDirectory + Hashes: winlog.event_data.Hashes + Image: winlog.event_data.Image + IntegrityLevel: winlog.event_data.IntegrityLevel + ParentCommandLine: winlog.event_data.ParentCommandLine + ParentImage: winlog.event_data.ParentImage + ParentUser: winlog.event_data.ParentUser + Product: winlog.event_data.Product + User: winlog.event_data.User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_raw_access_thread.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_raw_access_thread.yml new file mode 100644 index 00000000..51d1deff --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_raw_access_thread.yml @@ -0,0 +1,8 @@ +platform: LogScale +source: windows_raw_access_thread + + + +field_mapping: + Image: winlog.event_data.Image + Device: winlog.event_data.Device \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_registry_event.yml new file mode 100644 index 00000000..2c5f1a98 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_registry_event.yml @@ -0,0 +1,11 @@ +platform: LogScale +source: windows_registry_event + + + +field_mapping: + TargetObject: winlog.event_data.TargetObject + Image: winlog.event_data.Image + Details: winlog.event_data.Details + EventType: winlog.event_data.EventType + User: winlog.event_data.User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_security.yml new file mode 100644 index 00000000..45d670b4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_security.yml @@ -0,0 +1,140 @@ +platform: LogScale +source: windows_security + + + +field_mapping: + EventID: winlog.event_id + ParentImage: winlog.event_data.ParentImage + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + Channel: winlog.channel + ComputerName: winlog.computer_name + EventType: winlog.event_data.EventType + FailureReason: winlog.event_data.FailureCode + FileName: winlog.event_data.FileName + GrantedAccess: winlog.event_data.GrantedAccess + Hashes: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + IpAddress: winlog.event_data.IpAddress + IpPort: winlog.event_data.IpPort + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + LinkName: winlog.event_data.LinkName + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + Path: winlog.event_data.Path + CommandLine: winlog.event_data.ProcessCommandLine + OldUacValue: winlog.event_data.OldUacValue + CertIssuerName: winlog.event_data.CertIssuerName + SubStatus: winlog.event_data.SubStatus + DisplayName: winlog.event_data.DisplayName + TaskContent: winlog.event_data.TaskContent + ServiceSid: winlog.event_data.ServiceSid + CertThumbprint: winlog.event_data.CertThumbprint + ClassName: winlog.event_data.ClassName + NotificationPackageName: winlog.event_data.NotificationPackageName + NewSd: winlog.event_data.NewSd + TestSigning: winlog.event_data.TestSigning + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ServiceAccount: winlog.event_data.ServiceAccount + AttributeValue: winlog.event_data.AttributeValue + SessionName: winlog.event_data.SessionName + TaskName: winlog.event_data.TaskName + ObjectDN: winlog.event_data.ObjectDN + TemplateContent: winlog.event_data.TemplateContent + NewTemplateContent: winlog.event_data.NewTemplateContent + SourcePort: winlog.event_data.SourcePort + PasswordLastSet: winlog.event_data.PasswordLastSet + PrivilegeList: winlog.event_data.PrivilegeList + DeviceDescription: winlog.event_data.DeviceDescription + TargetServerName: winlog.event_data.TargetServerName + NewTargetUserName: winlog.event_data.NewTargetUserName + OperationType: winlog.event_data.OperationType + DestPort: winlog.event_data.DestPort + ServiceStartType: winlog.event_data.ServiceStartType + OldTargetUserName: winlog.event_data.OldTargetUserName + UserPrincipalName: winlog.event_data.UserPrincipalName + Accesses: winlog.event_data.Accesses + DnsHostName: winlog.event_data.DnsHostName + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + AuditSourceName: winlog.event_data.AuditSourceName + Workstation: winlog.event_data.Workstation + DestAddress: winlog.event_data.DestAddress + PreAuthType: winlog.event_data.PreAuthType + SecurityPackageName: winlog.event_data.SecurityPackageName + SubjectLogonId: winlog.event_data.SubjectLogonId + NewUacValue: winlog.event_data.NewUacValue + EnabledPrivilegeList: winlog.event_data.EnabledPrivilegeList + RelativeTargetName: winlog.event_data.RelativeTargetName + CertSerialNumber: winlog.event_data.CertSerialNumber + SidHistory: winlog.event_data.SidHistory + TargetLogonId: winlog.event_data.TargetLogonId + KernelDebug: winlog.event_data.KernelDebug + CallerProcessName: winlog.event_data.CallerProcessName + ProcessName: winlog.event_data.ProcessName + Properties: winlog.event_data.Properties + UserAccountControl: winlog.event_data.UserAccountControl + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + SecurityDescriptor: winlog.event_data.SecurityDescriptor + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + NewValue: winlog.event_data.NewValue + Source: winlog.event_data.Source + Status: winlog.event_data.Status + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + SourceAddr: winlog.event_data.SourceAddr + SourceAddress: winlog.event_data.SourceAddress + TargetName: winlog.event_data.TargetName + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + TargetDomainName: winlog.event_data.TargetDomainName + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + ObjectServer: winlog.event_data.ObjectServer + TargetUserSid: winlog.event_data.TargetUserSid + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + WorkstationName: winlog.event_data.WorkstationName + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + LayerRTID: winlog.event_data.LayerRTID + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + ParentProcessName: winlog.event_data.ParentProcessName + Service: winlog.event_data.Service \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_sysmon.yml new file mode 100644 index 00000000..083c9b53 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_sysmon.yml @@ -0,0 +1,57 @@ +platform: LogScale +source: windows_sysmon + + + + +field_mapping: + CommandLine: winlog.event_data.CommandLine + Image: winlog.event_data.Image + ParentImage: winlog.event_data.ParentImage + EventID: winlog.event_id + CallTrace: winlog.event_data.CallTrace + Company: winlog.event_data.Company + CurrentDirectory: winlog.event_data.CurrentDirectory + Description: winlog.event_data.Description + DestinationHostname: winlog.event_data.DestinationHostname + DestinationIp: winlog.event_data.DestinationIp + DestinationIsIpv6: winlog.event_data.DestinationIsIpv6 + DestinationPort: winlog.event_data.DestinationPort + DestinationPortName: winlog.event_data.DestinationPortName + Hashes: winlog.event_data.Hashes + Initiated: winlog.event_data.Initiated + IntegrityLevel: winlog.event_data.IntegrityLevel + ParentCommandLine: winlog.event_data.ParentCommandLine + Product: winlog.event_data.Product + Protocol: winlog.event_data.Protocol + RuleName: winlog.event_data.RuleName + SourceHostname: winlog.event_data.SourceHostname + SourceIp: winlog.event_data.SourceIp + SourceIsIpv6: winlog.event_data.SourceIsIpv6 + SourcePort: winlog.event_data.SourcePort + SourcePortName: winlog.event_data.SourcePortName + TargetFilename: winlog.event_data.TargetFilename + User: winlog.event_data.User + OriginalFileName: winlog.event_data.OriginalFileName + Signed: winlog.event_data.Signed + Signature: winlog.event_data.Signature + SignatureStatus: winlog.event_data.SignatureStatus + TargetObject: winlog.event_data.TargetObject + Details: winlog.event_data.Details + QueryName: winlog.event_data.QueryName + QueryResults: winlog.event_data.QueryResults + QueryStatus: winlog.event_data.QueryStatus + IsExecutable: winlog.event_data.IsExecutable + PipeName: winlog.event_data.PipeName + ImageLoaded: winlog.event_data.ImageLoaded + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + SourceImage: winlog.event_data.SourceImage + StartModule: winlog.event_data.StartModule + TargetImage: winlog.event_data.TargetImage + Device: winlog.event_data.Device + ProcessID: winlog.event_data.ProcessID + FileVersion: winlog.event_data.FileVersion + StartAddress: winlog.event_data.StartAddress + StartFunction: winlog.event_data.StartFunction + EventType: winlog.event_data.EventType \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_system.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_system.yml new file mode 100644 index 00000000..f6a620bd --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_system.yml @@ -0,0 +1,20 @@ +platform: LogScale +source: windows_system + + + +field_mapping: + EventID: winlog.event_id + AccountName: winlog.event_data.AccountName + ImagePath: winlog.event_data.ImagePath + ServiceName: winlog.event_data.ServiceName + ServiceType: winlog.event_data.ServiceType + StartType: winlog.event_data.StartType + Provider_Name: winlog.provider_name + Origin: winlog.event_data.Origin + HiveName: winlog.event_data.HiveName + Caption: winlog.event_data.Caption + param1: winlog.event_data.param1 + param2: winlog.event_data.param2 + Channel: winlog.channel + DeviceName: winlog.event_data.DeviceName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/logscale/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/logscale/windows_wmi_event.yml new file mode 100644 index 00000000..3b340800 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/logscale/windows_wmi_event.yml @@ -0,0 +1,8 @@ +platform: LogScale +source: windows_wmi_event + + + +field_mapping: + Destination: winlog.event_data.Destination + EventID: winlog.event_id diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/default.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/default.yml new file mode 100644 index 00000000..4c6beda4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/default.yml @@ -0,0 +1,9 @@ +platform: Microsoft Defender +source: default + + +log_source: + table: [union *] + +default_log_source: + table: union * \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/linux_file_event.yml new file mode 100644 index 00000000..100b37d7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/linux_file_event.yml @@ -0,0 +1,13 @@ +platform: Microsoft Defender +source: linux_file_event + + +log_source: + table: [DeviceFileEvents] + +default_log_source: + table: DeviceFileEvents + +field_mapping: + Image: InitiatingProcessFolderPath + TargetFilename : FolderPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/linux_network_connection.yml new file mode 100644 index 00000000..a3d6fe5e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/linux_network_connection.yml @@ -0,0 +1,14 @@ +platform: Microsoft Defender +source: linux_network_connection + + +log_source: + table: [DeviceNetworkEvents] + +default_log_source: + table: DeviceNetworkEvents + +field_mapping: + CommandLine: ProcessCommandLine + Image: FolderPath + ParentImage: InitiatingProcessFolderPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/macos_file_event.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/macos_file_event.yml new file mode 100644 index 00000000..37326a22 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/macos_file_event.yml @@ -0,0 +1,13 @@ +platform: Microsoft Defender +source: macos_file_event + + +log_source: + table: [DeviceFileEvents] + +default_log_source: + table: DeviceFileEvents + +field_mapping: + Image: InitiatingProcessFolderPath + TargetFilename : FolderPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/macos_network_connection.yml new file mode 100644 index 00000000..8abbd1b0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/macos_network_connection.yml @@ -0,0 +1,14 @@ +platform: Microsoft Defender +source: macos_network_connection + + +log_source: + table: [DeviceNetworkEvents] + +default_log_source: + table: DeviceNetworkEvents + +field_mapping: + CommandLine: ProcessCommandLine + Image: FolderPath + ParentImage: InitiatingProcessFolderPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/macos_process_creation.yml new file mode 100644 index 00000000..50aec451 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/macos_process_creation.yml @@ -0,0 +1,15 @@ +platform: Microsoft Defender +source: macos_process_creation + + +log_source: + table: [DeviceProcessEvents] + +default_log_source: + table: DeviceProcessEvents + +field_mapping: + CommandLine: ProcessCommandLine + Image: FolderPath + ParentCommandLine: InitiatingProcessCommandLine + ParentImage: InitiatingProcessFolderPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_file_event.yml new file mode 100644 index 00000000..abe6f95f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_file_event.yml @@ -0,0 +1,13 @@ +platform: Microsoft Defender +source: windows_file_event + + +log_source: + table: [DeviceFileEvents] + +default_log_source: + table: DeviceFileEvents + +field_mapping: + Image: InitiatingProcessFolderPath + TargetFilename: FolderPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_image_load.yml new file mode 100644 index 00000000..37d3bdeb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_image_load.yml @@ -0,0 +1,13 @@ +platform: Microsoft Defender +source: windows_image_load + + +log_source: + table: [DeviceImageLoadEvents] + +default_log_source: + table: DeviceImageLoadEvents + +field_mapping: + Image: InitiatingProcessFolderPath + ImageLoaded: FolderPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_network_connection.yml new file mode 100644 index 00000000..2864671e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_network_connection.yml @@ -0,0 +1,12 @@ +platform: Microsoft Defender +source: windows_network_connection + + +log_source: + table: [DeviceNetworkEvents] + +default_log_source: + table: DeviceNetworkEvents + +field_mapping: + Image: FolderPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_process_creation.yml new file mode 100644 index 00000000..ddf6e71e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_process_creation.yml @@ -0,0 +1,15 @@ +platform: Microsoft Defender +source: windows_process_creation + + +log_source: + table: [DeviceProcessEvents] + +default_log_source: + table: DeviceProcessEvents + +field_mapping: + CommandLine: ProcessCommandLine + Image: FolderPath + ParentCommandLine: InitiatingProcessCommandLine + ParentImage: InitiatingProcessFolderPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_registry_event.yml new file mode 100644 index 00000000..fc9306a7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_registry_event.yml @@ -0,0 +1,16 @@ +platform: Microsoft Defender +source: windows_registry_event + + +log_source: + table: [DeviceRegistryEvents] + +default_log_source: + table: DeviceRegistryEvents + +field_mapping: + TargetObject: RegistryKey + Image: InitiatingProcessFolderPath + Details: + - RegistryValueData + - RegistryValueType \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_sysmon.yml new file mode 100644 index 00000000..b651f6f6 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_defender/windows_sysmon.yml @@ -0,0 +1,15 @@ +platform: Microsoft Defender +source: windows_sysmon + + +log_source: + table: [DeviceProcessEvents] + +default_log_source: + table: DeviceProcessEvents + + +field_mapping: + CommandLine: ProcessCommandLine + Image: FolderPath + ParentImage: InitiatingProcessFolderPath diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/aws_cloudtrail.yml new file mode 100644 index 00000000..dd8aa813 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/aws_cloudtrail.yml @@ -0,0 +1,35 @@ +platform: Microsoft Sentinel +source: aws_cloudtrail + + +log_source: + table: [AWSCloudTrail] + +default_log_source: + table: AWSCloudTrail + +field_mapping: + eventSource: EventSource + eventName: EventName + AdditionalEventData: AdditionalEventData + additionalEventData.MFAUsed: additionalEventData.MFAUsed + errorCode: ErrorCode + errorMessage: ErrorMessage + eventType: EventTypeName + requestParameters: RequestParameters + requestParameters.attribute: requestParameters.attribute + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP + requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort + requestParameters.userData: requestParameters.userData + responseElements: ResponseElements + responseElements.ConsoleLogin: responseElements.ConsoleLogin + responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword + responseElements.publiclyAccessible: responseElements.publiclyAccessible + status: status + terminatingRuleId: terminatingRuleId + userAgent: UserAgent + userIdentity.arn: UserIdentityArn + userIdentity.principalId: UserIdentityPrincipalid + userIdentity.sessionContext.sessionIssuer.type: SessionIssuerType + userIdentity.type: UserIdentityType + userIdentity.userName: UserIdentityUserName diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/aws_eks.yml new file mode 100644 index 00000000..7c084b52 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/aws_eks.yml @@ -0,0 +1,25 @@ +platform: Microsoft Sentinel +source: aws_eks + + +log_source: + table: [AWSCloudTrail] + +default_log_source: + table: AWSCloudTrail + +field_mapping: + annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy + aws_node_type: aws_node_type + objectRef.namespace: objectRef.namespace + objectRef.resource: objectRef.resource + objectRef.subresource: objectRef.subresource + requestObject.rules.resources: requestObject.rules.resources + requestObject.rules.verbs: requestObject.rules.verbs + requestObject.spec.containers.image: requestObject.spec.containers.image + requestURI: requestURI + stage: stage + user.groups: user.groups + user.username: user.username + verb: verb diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_AzureDiagnostics.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_AzureDiagnostics.yml new file mode 100644 index 00000000..0e2dc391 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_AzureDiagnostics.yml @@ -0,0 +1,13 @@ +platform: Microsoft Sentinel +source: azure_AzureDiagnostics + + +log_source: + table: [AzureDiagnostics] + +default_log_source: + table: AzureDiagnostics + +field_mapping: + ResultDescription: ResultDescription + Category: Category diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_BehaviorAnalytics.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_BehaviorAnalytics.yml new file mode 100644 index 00000000..75c8b1ec --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_BehaviorAnalytics.yml @@ -0,0 +1,20 @@ +platform: Microsoft Sentinel +source: azure_BehaviorAnalytics + + +log_source: + table: [BehaviorAnalytics] + +default_log_source: + table: BehaviorAnalytics + +field_mapping: + ActionType: ActionType + ActivityInsights: ActivityInsights + ActivityType: ActivityType + EventSource: EventSource + DevicesInsights: DevicesInsights + RiskDetail: RiskDetail + UsersInsights: UsersInsights + UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount + UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_aadnoninteractiveusersigninlogs.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_aadnoninteractiveusersigninlogs.yml new file mode 100644 index 00000000..459f4bd0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_aadnoninteractiveusersigninlogs.yml @@ -0,0 +1,13 @@ +platform: Microsoft Sentinel +source: azure_aadnoninteractiveusersigninlogs + + +log_source: + table: [AADNonInteractiveUserSignInLogs] + +default_log_source: + table: AADNonInteractiveUserSignInLogs + +field_mapping: + UserAgent: UserAgent + Type: Type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_azureactivity.yml new file mode 100644 index 00000000..ae03fe99 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_azureactivity.yml @@ -0,0 +1,22 @@ +platform: Microsoft Sentinel +source: azure_azureactivity + + +log_source: + table: [AzureActivity] + +default_log_source: + table: AzureActivity + +field_mapping: + ActivityStatus: ActivityStatus + ActivityStatusValue: ActivityStatusValue + ActivitySubstatusValue: ActivitySubstatusValue + Authorization: Authorization + Category: Category + CategoryValue: CategoryValue + OperationName: OperationName + OperationNameValue: OperationNameValue + ResourceId: ResourceId + ResourceProviderValue: ResourceProviderValue + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_azuread.yml new file mode 100644 index 00000000..14bf7a3c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_azuread.yml @@ -0,0 +1,23 @@ +platform: Microsoft Sentinel +source: azure_azuread + + +log_source: + table: [AuditLogs] + +default_log_source: + table: AuditLogs + +field_mapping: + ActivityDisplayName: ActivityDisplayName + Category: Category + LoggedByService: LoggedByService + Result: Result + OperationName: OperationName + TargetResources: TargetResources + AADOperationType: AADOperationType + InitiatedBy: InitiatedBy + ResultReason: ResultReason + Status: Status + Status.errorCode: Status.errorCode + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_m365.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_m365.yml new file mode 100644 index 00000000..67444f16 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_m365.yml @@ -0,0 +1,23 @@ +platform: Microsoft Sentinel +source: azure_m365 + + +log_source: + table: [OfficeActivity] + +default_log_source: + table: OfficeActivity + +field_mapping: + ClientInfoString: ClientInfoString + LogonError: LogonError + ModifiedProperties: ModifiedProperties + OfficeObjectId: OfficeObjectId + OfficeWorkload: OfficeWorkload + Operation: Operation + Parameters: Parameters + RecordType: RecordType + ResultStatus: ResultStatus + SourceFileExtension: SourceFileExtension + SourceFileName: SourceFileName + UserAgent: UserAgent diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_o365.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_o365.yml similarity index 91% rename from siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_o365.yml rename to uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_o365.yml index 6ba31c1a..26832f7c 100644 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_o365.yml +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_o365.yml @@ -1,6 +1,6 @@ platform: Microsoft Sentinel source: azure_o365 -description: Text that describe current mapping + log_source: table: [OfficeActivity] diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_office365.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_office365.yml new file mode 100644 index 00000000..c2e2c4f8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_office365.yml @@ -0,0 +1,23 @@ +platform: Microsoft Sentinel +source: azure_office365 + + +log_source: + table: [OfficeActivity] + +default_log_source: + table: OfficeActivity + +field_mapping: + ClientInfoString: ClientInfoString + LogonError: LogonError + ModifiedProperties: ModifiedProperties + OfficeObjectId: OfficeObjectId + OfficeWorkload: OfficeWorkload + Operation: Operation + Parameters: Parameters + RecordType: RecordType + ResultStatus: ResultStatus + SourceFileExtension: SourceFileExtension + SourceFileName: SourceFileName + UserAgent: UserAgent \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_signlogs.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_signlogs.yml similarity index 94% rename from siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_signlogs.yml rename to uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_signlogs.yml index d3127e6d..068c3ece 100644 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/azure_signlogs.yml +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/azure_signlogs.yml @@ -1,6 +1,6 @@ platform: Microsoft Sentinel source: azure_signinlogs -description: Text that describe current mapping + log_source: table: [SigninLogs] diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/default.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/default.yml new file mode 100644 index 00000000..6ae2dc90 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/default.yml @@ -0,0 +1,7 @@ +platform: Microsoft Sentinel +source: default + + + +default_log_source: + table: union * \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_auidt.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_auidt.yml similarity index 79% rename from siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_auidt.yml rename to uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_auidt.yml index 2e7a1f12..5cc7d35e 100644 --- a/siem-converter/app/converter/mappings/platforms/microsoft_sentinel/linux_auidt.yml +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_auidt.yml @@ -1,6 +1,6 @@ platform: Microsoft Sentinel source: linux_auditd -description: Text that describe current mapping + log_source: table: [Syslog] diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_dns_query.yml new file mode 100644 index 00000000..21e060e2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_dns_query.yml @@ -0,0 +1,15 @@ +platform: Microsoft Sentinel +source: linux_dns_query + + +log_source: + table: [SysmonEvent] + +default_log_source: + table: SysmonEvent + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_file_event.yml new file mode 100644 index 00000000..c5d1c65b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_file_event.yml @@ -0,0 +1,13 @@ +platform: Microsoft Sentinel +source: linux_file_event + + +log_source: + table: [SysmonEvent, DeviceFileEvents] + +default_log_source: + table: SysmonEvent + +field_mapping: + Image: Image + TargetFilename : TargetFilename \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_network_connection.yml new file mode 100644 index 00000000..44aa5d7a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_network_connection.yml @@ -0,0 +1,19 @@ +platform: Microsoft Sentinel +source: linux_network_connection + + +log_source: + table: [SysmonEvent, DeviceNetworkEvents] + +default_log_source: + table: SysmonEvent + +field_mapping: + CommandLine: CommandLine + Image : Image + DestinationHostname: DestinationHostname + DestinationIp : DestinationIp + DestinationPort : DestinationPort + SourceIp : SourceIp + SourcePort : SourcePort + ParentImage : ParentImage \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_process_creation.yml new file mode 100644 index 00000000..e1e7eed7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/linux_process_creation.yml @@ -0,0 +1,15 @@ +platform: Microsoft Sentinel +source: linux_process_creation + + +log_source: + table: [SysmonEvent] + +default_log_source: + table: SysmonEvent + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/macos_file_event.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/macos_file_event.yml new file mode 100644 index 00000000..b418e9d8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/macos_file_event.yml @@ -0,0 +1,13 @@ +platform: Microsoft Sentinel +source: macos_file_event + + +log_source: + table: [DeviceFileEvents] + +default_log_source: + table: DeviceFileEvents + +field_mapping: + Image: Image + TargetFilename : TargetFilename \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/macos_network_connection.yml new file mode 100644 index 00000000..ff90178f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/macos_network_connection.yml @@ -0,0 +1,19 @@ +platform: Microsoft Sentinel +source: macos_network_connection + + +log_source: + table: [DeviceNetworkEvents] + +default_log_source: + table: DeviceNetworkEvents + +field_mapping: + CommandLine: CommandLine + Image : Image + DestinationHostname: DestinationHostname + DestinationIp : DestinationIp + DestinationPort : DestinationPort + SourceIp : SourceIp + SourcePort : SourcePort + ParentImage : ParentImage diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/macos_process_creation.yml new file mode 100644 index 00000000..0a3e154e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/macos_process_creation.yml @@ -0,0 +1,15 @@ +platform: Microsoft Sentinel +source: macos_process_creation + + +log_source: + table: [DeviceProcessEvents] + +default_log_source: + table: DeviceProcessEvents + +field_mapping: + Image: FolderPath + ParentCommandLine: InitiatingProcessCommandLine + ParentImage: InitiatingProcessFolderPath + CommandLine: ProcessCommandLine \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/okta_okta.yml new file mode 100644 index 00000000..b1492d38 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/okta_okta.yml @@ -0,0 +1,33 @@ +platform: Microsoft Sentinel +source: okta_okta + + +log_source: + table: [Okta_CL] + +default_log_source: + table: Okta_CL + +field_mapping: + client.user.id: client.user.id + source.user.id: source.user.id + User: User + alternateId: alternateId + client.user.full_name: client.user.full_name + source.user.full_name: source.user.full_name + related.user: related.user + client.ip: client.ip + source.ip: source.ip + user_agent.original: user_agent.original + userAgent.os: userAgent.os + userAgent.browser: userAgent.browser + client.zone: client.zone + client.device: client.device + client.id: client.id + event.action: event.action + outcome.reason: outcome.reason + event.outcome: event.outcome + client.as.number: client.as.number + client.as.organization.name: client.as.organization.name + client.domain: client.domain + source.domain: source.domain diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_bits_client.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_bits_client.yml new file mode 100644 index 00000000..3730b67c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_bits_client.yml @@ -0,0 +1,15 @@ +platform: Microsoft Sentinel +source: windows_bits_client + + +log_source: + table: [Event] + +default_log_source: + table: Event + +field_mapping: + LocalName: LocalName + EventID: EventID + RemoteName: RemoteName + processPath: processPath diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_dns_query.yml new file mode 100644 index 00000000..5b06f785 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_dns_query.yml @@ -0,0 +1,15 @@ +platform: Microsoft Sentinel +source: windows_dns_load + + +log_source: + table: [SysmonEvent, DeviceNetworkEvents] + +default_log_source: + table: SysmonEvent + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_driver_load.yml new file mode 100644 index 00000000..d5b121e8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_driver_load.yml @@ -0,0 +1,14 @@ +platform: Microsoft Sentinel +source: windows_driver_load + + +log_source: + table: [SysmonEvent] + +default_log_source: + table: SysmonEvent + +field_mapping: + ImageLoaded: ImageLoaded + ImagePath: ImagePath + Hashes: Hashes diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_file_event.yml new file mode 100644 index 00000000..5c7932b4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_file_event.yml @@ -0,0 +1,18 @@ +platform: Microsoft Sentinel +source: windows_file_event + + +log_source: + table: [SysmonEvent, DeviceFileEvents] + +default_log_source: + table: SysmonEvent + +field_mapping: + CreationUtcTime: CreationUtcTime +# FileName + Image: Image +# OriginalFilename: PreviousCreationUtcTime + PreviousCreationUtcTime: PreviousCreationUtcTime + TargetFilename: TargetFilename + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_image_load.yml new file mode 100644 index 00000000..f464b934 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_image_load.yml @@ -0,0 +1,16 @@ +platform: Microsoft Sentinel +source: windows_image_load + + +log_source: + table: [SysmonEvent, DeviceImageLoadEvents] + +default_log_source: + table: SysmonEvent + +field_mapping: + Image: Image + ImageLoaded: ImageLoaded + SignatureStatus: SignatureStatus + OriginalFileName: OriginalFileName + Signed: Signed \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_ldap_debug.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_ldap_debug.yml new file mode 100644 index 00000000..f2bb1c50 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_ldap_debug.yml @@ -0,0 +1,13 @@ +platform: Microsoft Sentinel +source: windows_ldap_debug + + +log_source: + table: [Event] + +default_log_source: + table: Event | where EventLog in~ ('Microsoft-Windows-LDAP-Client/Debug') + +field_mapping: + EventID: EventID + SearchFilter: SearchFilter diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_network_connection.yml new file mode 100644 index 00000000..82147cb9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_network_connection.yml @@ -0,0 +1,18 @@ +platform: Microsoft Sentinel +source: windows_network_connection + + +log_source: + table: [SysmonEvent, DeviceNetworkEvents] + +default_log_source: + table: SysmonEvent + +field_mapping: + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + Initiated: Initiated \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_ntlm.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_ntlm.yml new file mode 100644 index 00000000..0e01bd8b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_ntlm.yml @@ -0,0 +1,15 @@ +platform: Microsoft Sentinel +source: windows_ntlm + + +log_source: + table: [Event] + +default_log_source: + table: Event + +field_mapping: + WorkstationName: WorkstationName + ProcessName: ProcessName + EventID: EventID + TargetName: TargetName diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_powershell.yml new file mode 100644 index 00000000..2495c511 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_powershell.yml @@ -0,0 +1,19 @@ +platform: Microsoft Sentinel +source: windows_powershell + + +log_source: + table: [Event] + +default_log_source: + table: "Event | where EventLog in~ ('Microsoft-Windows-PowerShell/Operational', 'Windows PowerShell')" + +field_mapping: + EventID: EventID + CommandLine: CommandLine + ScriptBlockText: EventData + Payload: Payload + HostApplication: HostApplication + ContextInfo: ContextInfo + HostName: HostName + EngineVersion: EngineVersion \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_process_creation.yml new file mode 100644 index 00000000..a240c945 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_process_creation.yml @@ -0,0 +1,21 @@ +platform: Microsoft Sentinel +source: windows_process_creation + + +log_source: + table: [SysmonEvent, SecurityEvent, WindowsEvent, DeviceProcessEvents] + +default_log_source: + table: SecurityEvent + +field_mapping: + CommandLine: CommandLine + CurrentDirectory: CurrentDirectory + Hashes: Hashes + Image: NewProcessName + IntegrityLevel: IntegrityLevel + ParentCommandLine: ParentCommandLine + ParentImage: ParentProcessName + ParentUser: ParentUser + Product: Product + User: Account \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_registry_event.yml new file mode 100644 index 00000000..4baf179b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_registry_event.yml @@ -0,0 +1,16 @@ +platform: Microsoft Sentinel +source: windows_registry_event + + +log_source: + table: [SysmonEvent, DeviceRegistryEvents] + +default_log_source: + table: SysmonEvent + +field_mapping: + TargetObject: TargetObject + Image: Image + Details: Details + EventType: EventType + User: User diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_security.yml new file mode 100644 index 00000000..cec940ed --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_security.yml @@ -0,0 +1,145 @@ +platform: Microsoft Sentinel +source: windows_security + + +log_source: + table: [SecurityEvent, WindowsEvent] + +default_log_source: + table: SecurityEvent + +field_mapping: + EventID: EventID + ParentImage: ParentProcessName + AccessMask: AccessMask + AccountName: AccountName + AllowedToDelegateTo: AllowedToDelegateTo + AttributeLDAPDisplayName: AttributeLDAPDisplayName + AuditPolicyChanges: AuditPolicyChanges + AuthenticationPackageName: AuthenticationPackageName + CallingProcessName: CallingProcessName + Channel: Channel + ComputerName: Computer + EventType: EventType + FailureReason: FailureReason + FileName: FileName + GrantedAccess: GrantedAccess + Hashes: FileHash + HiveName: HiveName + IpAddress: IpAddress + IpPort: IpPort + KeyLength: KeyLength + LogonProcessName: LogonProcessName + LogonType: LogonType + LinkName: LinkName + MemberName: MemberName + MemberSid: MemberSid + NewProcessName: NewProcessName + ObjectClass: ObjectClass + ObjectName: ObjectName + ObjectType: ObjectType + ObjectValueName: ObjectValueName + Path: Path + CommandLine: CommandLine + OldUacValue: OldUacValue + CertIssuerName: CertIssuerName + SubStatus: SubStatus + DisplayName: DisplayName + TaskContent: TaskContent + ServiceSid: ServiceSid + CertThumbprint: CertThumbprint + ClassName: ClassName + NotificationPackageName: NotificationPackageName + NewSd: NewSd + TestSigning: TestSigning + TargetInfo: TargetInfo + ClientProcessId: ClientProcessId + ParentProcessId: ParentProcessId + AccessList: AccessList + GroupMembership: GroupMembership + FilterName: FilterName + ChangeType: ChangeType + LayerName: LayerName + ServiceAccount: ServiceAccount + AttributeValue: AttributeValue + SessionName: SessionName + TaskName: TaskName + ObjectDN: ObjectDN + TemplateContent: TemplateContent + NewTemplateContent: NewTemplateContent + SourcePort: SourcePort + PasswordLastSet: PasswordLastSet + PrivilegeList: PrivilegeList + DeviceDescription: DeviceDescription + TargetServerName: TargetServerName + NewTargetUserName: NewTargetUserName + OperationType: OperationType + DestPort: DestPort + ServiceStartType: ServiceStartType + OldTargetUserName: OldTargetUserName + UserPrincipalName: UserPrincipalName + Accesses: Accesses + DnsHostName: DnsHostName + DisableIntegrityChecks: DisableIntegrityChecks + AuditSourceName: AuditSourceName + Workstation: Workstation + DestAddress: DestAddress + PreAuthType: PreAuthType + SecurityPackageName: SecurityPackageName + SubjectLogonId: SubjectLogonId + NewUacValue: NewUacValue + EnabledPrivilegeList: EnabledPrivilegeList + RelativeTargetName: RelativeTargetName + CertSerialNumber: CertSerialNumber + SidHistory: SidHistory + TargetLogonId: TargetLogonId + KernelDebug: KernelDebug + CallerProcessName: CallerProcessName + ProcessName: ProcessName + Properties: Properties + UserAccountControl: UserAccountControl + RegistryValue: RegistryValue + SecurityID: SecurityID + ServiceFileName: ServiceFileName + SecurityDescriptor: SecurityDescriptor + ServiceName: ServiceName + ShareName: ShareName + NewValue: NewValue + Source: Source + Status: Status + SubjectDomainName: SubjectDomainName + SubjectUserName: SubjectUserName + SubjectUserSid: SubjectUserSid + SourceAddr: SourceAddr + SourceAddress: SourceAddress + TargetName: TargetName + ServicePrincipalNames: ServicePrincipalNames + TargetDomainName: TargetDomainName + TargetSid: TargetSid + TargetUserName: TargetUserName + ObjectServer: ObjectServer + TargetUserSid: TargetUserSid + TicketEncryptionType: TicketEncryptionType + TicketOptions: TicketOptions + WorkstationName: WorkstationName + TransmittedServices: TransmittedServices + AuthenticationAlgorithm: AuthenticationAlgorithm + LayerRTID: LayerRTID + BSSID: BSSID + BSSType: BSSType + CipherAlgorithm: CipherAlgorithm + ConnectionId: ConnectionId + ConnectionMode: ConnectionMode + InterfaceDescription: InterfaceDescription + InterfaceGuid: InterfaceGuid + OnexEnabled: OnexEnabled + PHYType: PHYType + ProfileName: ProfileName + SSID: SSID + Domain: Domain + ServiceType: ServiceType + SourceName: SourceName + StartType: StartType + UserID: UserID + ParentProcessName: ParentProcessName + Service: Service diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_sysmon.yml new file mode 100644 index 00000000..f7cba6b8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_sysmon.yml @@ -0,0 +1,62 @@ +platform: Microsoft Sentinel +source: windows_sysmon + + +log_source: + table: [SysmonEvent] + +default_log_source: + table: SysmonEvent + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentImage: ParentImage + EventID: EventID + CallTrace: CallTrace + Company: Company + CurrentDirectory: CurrentDirectory + Description: Description + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationIsIpv6: DestinationIsIpv6 + DestinationPort: DestinationPort + DestinationPortName: DestinationPortName + Hashes: Hashes + Initiated: Initiated + IntegrityLevel: IntegrityLevel + ParentCommandLine: ParentCommandLine + Product: Product + Protocol: Protocol + RuleName: RuleName + SourceHostname: SourceHostname + SourceIp: SourceIp + SourceIsIpv6: SourceIsIpv6 + SourcePort: SourcePort + SourcePortName: SourcePortName + TargetFilename: TargetFilename + User: User + OriginalFileName: OriginalFileName + Signed: Signed + Signature: Signature + SignatureStatus: SignatureStatus + TargetObject: TargetObject + Details: Details + QueryName: QueryName + QueryResults: QueryResults + QueryStatus: QueryStatus + IsExecutable: IsExecutable + PipeName: PipeName + ImageLoaded: ImageLoaded + ImagePath: ImagePath + Imphash: Imphash + SourceImage: SourceImage + StartModule: StartModule + TargetImage: TargetImage + ProcessID: ProcessID + FileVersion: FileVersion + StartAddress: StartAddress + StartFunction: StartFunction + EventType: EventType + timestamp: TimeGenerated + Device: Computer \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_wmi_event.yml new file mode 100644 index 00000000..a5e5aa53 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/microsoft_sentinel/windows_wmi_event.yml @@ -0,0 +1,13 @@ +platform: Microsoft Sentinel +source: windows_wmi_event + + +log_source: + table: [Event] + +default_log_source: + table: Event + +field_mapping: + Destination: Destination + EventID: EventID \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/aws_cloudtrail.yml new file mode 100644 index 00000000..009746b8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/aws_cloudtrail.yml @@ -0,0 +1,35 @@ +platform: OpenSearch +source: aws_cloudtrail + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + eventSource: event.provider + eventName: event.action + AdditionalEventData: AdditionalEventData + additionalEventData.MFAUsed: additionalEventData.MFAUsed + errorCode: errorCode + errorMessage: errorMessage + eventType: eventType + requestParameters: requestParameters + requestParameters.attribute: requestParameters.attribute + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP + requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort + requestParameters.userData: requestParameters.userData + responseElements: responseElements + responseElements.ConsoleLogin: responseElements.ConsoleLogin + responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword + responseElements.publiclyAccessible: responseElements.publiclyAccessible + status: status + terminatingRuleId: terminatingRuleId + userAgent: userAgent + userIdentity.arn: userIdentity.arn + userIdentity.principalId: userIdentity.principalId + userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type + userIdentity.type: userIdentity.type + userIdentity.userName: userIdentity.userName diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/aws_eks.yml new file mode 100644 index 00000000..b2161ca5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/aws_eks.yml @@ -0,0 +1,25 @@ +platform: OpenSearch +source: aws_eks + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy + aws_node_type: aws_node_type + objectRef.namespace: objectRef.namespace + objectRef.resource: objectRef.resource + objectRef.subresource: objectRef.subresource + requestObject.rules.resources: requestObject.rules.resources + requestObject.rules.verbs: requestObject.rules.verbs + requestObject.spec.containers.image: requestObject.spec.containers.image + requestURI: requestURI + stage: stage + user.groups: user.groups + user.username: user.username + verb: verb diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/azure_AzureDiagnostics.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_AzureDiagnostics.yml new file mode 100644 index 00000000..115636ee --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_AzureDiagnostics.yml @@ -0,0 +1,13 @@ +platform: OpenSearch +source: azure_AzureDiagnostics + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ResultDescription: ResultDescription + Category: Category \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/azure_BehaviorAnalytics.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_BehaviorAnalytics.yml new file mode 100644 index 00000000..dffbd5ea --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_BehaviorAnalytics.yml @@ -0,0 +1,20 @@ +platform: OpenSearch +source: azure_BehaviorAnalytics + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ActionType: ActionType + ActivityInsights: ActivityInsights + ActivityType: ActivityType + EventSource: EventSource + DevicesInsights: DevicesInsights + RiskDetail: RiskDetail + UsersInsights: UsersInsights + UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount + UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/azure_aadnoninteractiveusersigninlogs.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_aadnoninteractiveusersigninlogs.yml new file mode 100644 index 00000000..533e7eeb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_aadnoninteractiveusersigninlogs.yml @@ -0,0 +1,13 @@ +platform: OpenSearch +source: azure_aadnoninteractiveusersigninlogs + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + UserAgent: UserAgent + Type: Type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_azureactivity.yml new file mode 100644 index 00000000..6058b234 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_azureactivity.yml @@ -0,0 +1,22 @@ +platform: OpenSearch +source: azure_azureactivity + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ActivityStatus: ActivityStatus + ActivityStatusValue: ActivityStatusValue + ActivitySubstatusValue: ActivitySubstatusValue + Authorization: Authorization + Category: Category + CategoryValue: CategoryValue + OperationName: OperationName + OperationNameValue: OperationNameValue + ResourceId: ResourceId + ResourceProviderValue: ResourceProviderValue + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_azuread.yml new file mode 100644 index 00000000..62b4c2ae --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_azuread.yml @@ -0,0 +1,23 @@ +platform: OpenSearch +source: azure_azuread + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ActivityDisplayName: event.action + Category: azure.auditlogs.properties.category + LoggedByService: azure.auditlogs.properties.logged_by_service + Result: event.outcome + OperationName: OperationName + TargetResources: TargetResources + AADOperationType: AADOperationType + InitiatedBy: InitiatedBy + ResultReason: ResultReason + Status: Status + Status.errorCode: Status.errorCode + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/azure_m365.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_m365.yml new file mode 100644 index 00000000..838af96b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_m365.yml @@ -0,0 +1,23 @@ +platform: OpenSearch +source: azure_m365 + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + ClientInfoString: ClientInfoString + LogonError: LogonError + ModifiedProperties: ModifiedProperties + OfficeObjectId: OfficeObjectId + OfficeWorkload: OfficeWorkload + Operation: Operation + Parameters: Parameters + RecordType: RecordType + ResultStatus: ResultStatus + SourceFileExtension: SourceFileExtension + SourceFileName: SourceFileName + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/azure_signinlogs.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_signinlogs.yml new file mode 100644 index 00000000..ed4d7bfd --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/azure_signinlogs.yml @@ -0,0 +1,29 @@ +platform: OpenSearch +source: azure_signinlogs + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + AppDisplayName: AppDisplayName + AppId: AppId + AuthenticationRequirement: AuthenticationRequirement + Category: Category + ConditionalAccessStatus: ConditionalAccessStatus + DeviceDetail: DeviceDetail + IsInteractive: IsInteractive + NetworkLocationDetails: NetworkLocationDetails + ResourceDisplayName: ResourceDisplayName + ResourceIdentity: ResourceIdentity + ResultDescription: ResultDescription + ResultType: ResultType + Status.errorCode: Status.errorCode + Status: Status + Status.failureReason: Status.failureReason + TokenIssuerType: TokenIssuerType + UserAgent: UserAgent + UserPrincipalName: UserPrincipalName diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/default.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/default.yml new file mode 100644 index 00000000..501a2cf3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/default.yml @@ -0,0 +1,6 @@ +platform: OpenSearch +source: default + + +default_log_source: + index: "*" \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/dns.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/dns.yml new file mode 100644 index 00000000..f80fe642 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/dns.yml @@ -0,0 +1,17 @@ +platform: OpenSearch +source: dns + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + dns-query: dns.question.name + parent-domain: + - url.registered_domain + - destination.registered_domain + dns-answer: dns.answers.name + dns-record: dns.question.type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/firewall.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/firewall.yml new file mode 100644 index 00000000..d52e40de --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/firewall.yml @@ -0,0 +1,19 @@ +platform: OpenSearch +source: firewall + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + src-ip: + - source.address + - source.ip + src-port: source.port + dst-ip: + - destination.address + - destination.ip + dst-port: destination.port \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/gcp_gcp.audit.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/gcp_gcp.audit.yml new file mode 100644 index 00000000..014b63a1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/gcp_gcp.audit.yml @@ -0,0 +1,21 @@ +platform: OpenSearch +source: gcp_gcp.audit + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + backupConfiguration.enabled: backupConfiguration.enabled + gcp.audit.method_name: gcp.audit.method_name + jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction + jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue + message: message + methodName: methodName + protoPayload.methodName: protoPayload.methodName + protoPayload.request.canIpForward: protoPayload.request.canIpForward + protoPayload.serviceName: protoPayload.serviceName + serviceName: serviceName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/gcp_pubsub.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/gcp_pubsub.yml new file mode 100644 index 00000000..e9fa5687 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/gcp_pubsub.yml @@ -0,0 +1,16 @@ +platform: OpenSearch +source: gcp_pubsub + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + data.protoPayload.methodName: data.protoPayload.methodName + data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member + data.resource.type: data.resource.type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/linux_auditd.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/linux_auditd.yml new file mode 100644 index 00000000..6717b1ad --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/linux_auditd.yml @@ -0,0 +1,16 @@ +platform: OpenSearch +source: linux_auditd + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + a0: process.args + a1: process.args + a2: process.args + a3: process.args + exe: process.executable diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/linux_dns_query.yml new file mode 100644 index 00000000..86502539 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/linux_dns_query.yml @@ -0,0 +1,15 @@ +platform: OpenSearch +source: linux_dns_query + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + Image: process.executable + User: user.name + QueryName: dns.question.name + QueryResults: dns.answers \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/linux_process_creation.yml new file mode 100644 index 00000000..46471ed8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/linux_process_creation.yml @@ -0,0 +1,15 @@ +platform: OpenSearch +source: linux_process_creation + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/macos_dns_query.yml new file mode 100644 index 00000000..ac98cef1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/macos_dns_query.yml @@ -0,0 +1,15 @@ +platform: OpenSearch +source: macos_dns_query + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + Image: process.executable + User: user.name + QueryName: dns.question.name + QueryResults: dns.answers \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/macos_network_connection.yml new file mode 100644 index 00000000..37d11f61 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/macos_network_connection.yml @@ -0,0 +1,18 @@ +platform: OpenSearch +source: macos_network_connection + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + Image: process.executable + DestinationHostname: destination.domain + DestinationIp: destination.ip + DestinationPort: destination.port + SourceIp: source.ip + SourcePort: source.port + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/macos_process_creation.yml new file mode 100644 index 00000000..bb418d93 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/macos_process_creation.yml @@ -0,0 +1,15 @@ +platform: OpenSearch +source: macos_process_creation + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/okta_okta.yml new file mode 100644 index 00000000..5a2c244b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/okta_okta.yml @@ -0,0 +1,33 @@ +platform: OpenSearch +source: okta_okta + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + client.user.id: okta.actor.id + source.user.id: okta.actor.id + User: okta.actor.type + alternateId: okta.actor.alternate_id + client.user.full_name: okta.actor.display_name + source.user.full_name: okta.actor.display_name + related.user: okta.actor.display_name + client.ip: okta.client.ip + source.ip: okta.client.ip + user_agent.original: okta.client.user_agent.raw_user_agent + userAgent.os: okta.client.user_agent.os + userAgent.browser: okta.client.user_agent.browser + client.zone: okta.client.zone + client.device: okta.client.device + client.id: okta.client.id + event.action: okta.event_type + outcome.reason: okta.outcome.reason + event.outcome: okta.event.outcome + client.as.number: okta.security_context.as.number + client.as.organization.name: okta.security_context.as.organization.name + client.domain: okta.security_context.isp + source.domain: okta.security_context.domain diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/proxy.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/proxy.yml new file mode 100644 index 00000000..ee0a2c91 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/proxy.yml @@ -0,0 +1,30 @@ +platform: OpenSearch +source: proxy + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + c-uri: url.original + c-useragent: user_agent.original + cs-method: http.request.method + cs-bytes: http.request.body.bytes + cs-cookie-vars: http.cookie_vars + c-uri-extension: url.extension + c-uri-query: url.query + cs-cookie: http.cookie_vars + cs-host: + - url.domain + - destination.domain + + cs-referrer: http.request.referrer + cs-version: http.version + r-dns: + - destination.domain + - url.domain + sc-status: http.response.status_code + post-body: http.post_body \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/webserver.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/webserver.yml new file mode 100644 index 00000000..fbe336bb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/webserver.yml @@ -0,0 +1,29 @@ +platform: OpenSearch +source: webserver + + +log_source: + index: [logs-*] + +default_log_source: + index: logs-* + +field_mapping: + c-uri: url.original + c-useragent: user_agent.original + cs-method: http.request.method + cs-bytes: http.request.body.bytes + cs-cookie-vars: http.cookie_vars + c-uri-extension: url.extension + c-uri-query: url.query + cs-cookie: http.cookie_vars + cs-host: + - url.domain + - destination.domain + cs-referrer: http.request.referrer + cs-version: http.version + r-dns: + - destination.domain + - url.domain + sc-status: http.response.status_code + post-body: http.post_body \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_bits_client.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_bits_client.yml new file mode 100644 index 00000000..a1561ddf --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_bits_client.yml @@ -0,0 +1,17 @@ +platform: OpenSearch +source: windows_bits_client + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + LocalName: winlog.event_data.LocalName + EventID: + - event.code + - winlog.event_id + RemoteName: winlog.event_data.RemoteName + processPath: winlog.event_data.processPath diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_dns_query.yml new file mode 100644 index 00000000..169118bc --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_dns_query.yml @@ -0,0 +1,18 @@ +platform: OpenSearch +source: macos_dns_query + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + Image: process.executable + User: + - user.name + - winlog.user.domain + - winlog.user.name + QueryName: dns.question.name + QueryResults: dns.answers \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_driver_load.yml new file mode 100644 index 00000000..106ca241 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_driver_load.yml @@ -0,0 +1,14 @@ +platform: OpenSearch +source: windows_driver_load + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + ImageLoaded: file.path + ImagePath: file.path + Hashes: winlog.event_data.Hashes \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_image_load.yml new file mode 100644 index 00000000..b36ba107 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_image_load.yml @@ -0,0 +1,16 @@ +platform: OpenSearch +source: windows_image_load + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + Image: process.executable + ImageLoaded: file.path + SignatureStatus: SignatureStatus + OriginalFileName: OriginalFileName + Signed: Signed \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_ldap_debug.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_ldap_debug.yml new file mode 100644 index 00000000..94cd7e7b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_ldap_debug.yml @@ -0,0 +1,15 @@ +platform: OpenSearch +source: windows_ldap_debug + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + EventID: + - event.code + - winlog.event_id + SearchFilter: winlog.event_data.SearchFilter \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_network_connection.yml new file mode 100644 index 00000000..f2a67599 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_network_connection.yml @@ -0,0 +1,17 @@ +platform: OpenSearch +source: windows_network_connection + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + Image: process.executable + DestinationHostname: destination.domain + DestinationIp: destination.ip + DestinationPort: destination.port + SourceIp: source.ip + SourcePort: source.port diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_ntlm.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_ntlm.yml new file mode 100644 index 00000000..4f575b69 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_ntlm.yml @@ -0,0 +1,17 @@ +platform: OpenSearch +source: windows_ntlm + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + ProcessName: ProcessName + WorkstationName: winlog.event_data.WorkstationName + EventID: + - event.code + - winlog.event_id + TargetName: winlog.event_data.TargetName diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_powershell.yml new file mode 100644 index 00000000..43d10067 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_powershell.yml @@ -0,0 +1,25 @@ +platform: OpenSearch +source: windows_powershell + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + EventID: + - event.code + - winlog.event_id + CommandLine: powershell.command.value + ScriptBlockText: powershell.file.script_block_text + Payload: + - powershell.command.invocation_details + - winlog.event_data.Payload + HostApplication: process.command_line + ContextInfo: winlog.event_data.ContextInfo + HostName: process.title + EngineVersion: + - powershell.engine.version + - powershell.process.executable_version \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_process_creation.yml new file mode 100644 index 00000000..1ffe5dc8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_process_creation.yml @@ -0,0 +1,21 @@ +platform: OpenSearch +source: windows_process_creation + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + CommandLine: process.command_line + CurrentDirectory: CurrentDirectory + Hashes: Hashes + Image: process.executable + IntegrityLevel: IntegrityLevel + ParentCommandLine: process.parent.command_line + ParentImage: process.parent.executable + ParentUser: ParentUser + Product: Product + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_security.yml new file mode 100644 index 00000000..10dd8cb2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_security.yml @@ -0,0 +1,149 @@ +platform: OpenSearch +source: windows_security + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + EventID: + - event.code + - winlog.event_id + ParentImage: process.parent.executable + AccessMask: winlog.event_data.AccessMask + AccountName: winlog.event_data.AccountName + AllowedToDelegateTo: winlog.event_data.AllowedToDelegateTo + AttributeLDAPDisplayName: winlog.event_data.AttributeLDAPDisplayName + AuditPolicyChanges: winlog.event_data.AuditPolicyChanges + AuthenticationPackageName: winlog.event_data.AuthenticationPackageName + CallingProcessName: winlog.event_data.CallingProcessName + Channel: winlog.channel + ComputerName: winlog.ComputerName + EventType: winlog.event_data.EventType + FailureReason: winlog.event_data.FailureCode + FileName: file.path + GrantedAccess: winlog.event_data.GrantedAccess + Hashes: winlog.event_data.Hashes + HiveName: winlog.event_data.HiveName + IpAddress: source.ip + IpPort: source.port + KeyLength: winlog.event_data.KeyLength + LogonProcessName: winlog.event_data.LogonProcessName + LogonType: winlog.event_data.LogonType + LinkName: LinkName + MemberName: winlog.event_data.MemberName + MemberSid: winlog.event_data.MemberSid + NewProcessName: winlog.event_data.NewProcessName + ObjectClass: winlog.event_data.ObjectClass + ObjectName: winlog.event_data.ObjectName + ObjectType: winlog.event_data.ObjectType + ObjectValueName: winlog.event_data.ObjectValueName + Path: winlog.event_data.Path + CommandLine: winlog.event_data.ProcessCommandLine + OldUacValue: winlog.event_data.OldUacValue + CertIssuerName: winlog.event_data.CertIssuerName + SubStatus: winlog.event_data.SubStatus + DisplayName: winlog.event_data.DisplayName + TaskContent: winlog.event_data.TaskContent + ServiceSid: winlog.event_data.ServiceSid + CertThumbprint: winlog.event_data.CertThumbprint + ClassName: winlog.event_data.ClassName + NotificationPackageName: winlog.event_data.NotificationPackageName + NewSd: winlog.event_data.NewSd + TestSigning: winlog.event_data.TestSigning + TargetInfo: winlog.event_data.TargetInfo + ClientProcessId: winlog.event_data.ClientProcessId + ParentProcessId: winlog.event_data.ParentProcessId + AccessList: winlog.event_data.AccessList + GroupMembership: winlog.event_data.GroupMembership + FilterName: winlog.event_data.FilterName + ChangeType: winlog.event_data.ChangeType + LayerName: winlog.event_data.LayerName + ServiceAccount: winlog.event_data.ServiceAccount + AttributeValue: winlog.event_data.AttributeValue + SessionName: winlog.event_data.SessionName + TaskName: winlog.event_data.TaskName + ObjectDN: winlog.event_data.ObjectDN + TemplateContent: winlog.event_data.TemplateContent + NewTemplateContent: winlog.event_data.NewTemplateContent + SourcePort: winlog.event_data.SourcePort + PasswordLastSet: winlog.event_data.PasswordLastSet + PrivilegeList: winlog.event_data.PrivilegeList + DeviceDescription: winlog.event_data.DeviceDescription + TargetServerName: winlog.event_data.TargetServerName + NewTargetUserName: winlog.event_data.NewTargetUserName + OperationType: winlog.event_data.OperationType + DestPort: winlog.event_data.DestPort + ServiceStartType: winlog.event_data.ServiceStartType + OldTargetUserName: winlog.event_data.OldTargetUserName + UserPrincipalName: winlog.event_data.UserPrincipalName + Accesses: winlog.event_data.Accesses + DnsHostName: winlog.event_data.DnsHostName + DisableIntegrityChecks: winlog.event_data.DisableIntegrityChecks + AuditSourceName: winlog.event_data.AuditSourceName + Workstation: winlog.event_data.Workstation + DestAddress: winlog.event_data.DestAddress + PreAuthType: winlog.event_data.PreAuthType + SecurityPackageName: winlog.event_data.SecurityPackageName + SubjectLogonId: winlog.event_data.SubjectLogonId + NewUacValue: winlog.event_data.NewUacValue + EnabledPrivilegeList: winlog.event_data.EnabledPrivilegeList + RelativeTargetName: winlog.event_data.RelativeTargetName + CertSerialNumber: winlog.event_data.CertSerialNumber + SidHistory: winlog.event_data.SidHistory + TargetLogonId: winlog.event_data.TargetLogonId + KernelDebug: winlog.event_data.KernelDebug + CallerProcessName: winlog.event_data.CallerProcessName + ProcessName: process.executable + Properties: winlog.event_data.Properties + UserAccountControl: winlog.event_data.UserAccountControl + RegistryValue: winlog.event_data.RegistryValue + SecurityID: winlog.event_data.SecurityID + ServiceFileName: winlog.event_data.ServiceFileName + SecurityDescriptor: + - winlog.event_data.SecurityDescriptor + - winlog.event_data.TaskContentNew + ServiceName: winlog.event_data.ServiceName + ShareName: winlog.event_data.ShareName + NewValue: winlog.event_data.NewValue + Source: winlog.event_data.Source + Status: winlog.event_data.Status + SubjectDomainName: winlog.event_data.SubjectDomainName + SubjectUserName: winlog.event_data.SubjectUserName + SubjectUserSid: winlog.event_data.SubjectUserSid + SourceAddr: SourceAddr + SourceAddress: SourceAddress + TargetName: TargetName + ServicePrincipalNames: winlog.event_data.ServicePrincipalNames + TargetDomainName: winlog.event_data.TargetDomainName + TargetSid: winlog.event_data.TargetSid + TargetUserName: winlog.event_data.TargetUserName + ObjectServer: + TargetUserSid: winlog.event_data.TargetUserSid + TicketEncryptionType: winlog.event_data.TicketEncryptionType + TicketOptions: winlog.event_data.TicketOptions + WorkstationName: source.domain + TransmittedServices: winlog.event_data.TransmittedServices + AuthenticationAlgorithm: winlog.event_data.AuthenticationAlgorithm + LayerRTID: LayerRTID + BSSID: winlog.event_data.BSSID + BSSType: winlog.event_data.BSSType + CipherAlgorithm: winlog.event_data.CipherAlgorithm + ConnectionId: winlog.event_data.ConnectionId + ConnectionMode: winlog.event_data.ConnectionMode + InterfaceDescription: winlog.event_data.InterfaceDescription + InterfaceGuid: winlog.event_data.InterfaceGuid + OnexEnabled: winlog.event_data.OnexEnabled + PHYType: winlog.event_data.PHYType + ProfileName: winlog.event_data.ProfileName + SSID: winlog.event_data.SSID + Domain: winlog.event_data.Domain + ServiceType: winlog.event_data.ServiceType + SourceName: winlog.event_data.SourceName + StartType: winlog.event_data.StartType + UserID: winlog.event_data.UserID + ParentProcessName: process.parent.name + Service: Service diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_sysmon.yml new file mode 100644 index 00000000..5db381e3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_sysmon.yml @@ -0,0 +1,67 @@ +platform: OpenSearch +source: windows_sysmon + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + + +field_mapping: + CommandLine: process.command_line + Image: process.executable + ParentImage: process.parent.executable + EventID: + - event.code + - winlog.event_id + CallTrace: winlog.event_data.CallTrace + Company: winlog.event_data.Company + CurrentDirectory: process.working_directory + Description: winlog.event_data.Description + DestinationHostname: destination.domain + DestinationIp: destination.ip + DestinationIsIpv6: winlog.event_data.DestinationIsIpv6 + DestinationPort: destination.port + DestinationPortName: network.protocol + Hashes: winlog.event_data.Hashes + Initiated: winlog.event_data.Initiated + IntegrityLevel: winlog.event_data.IntegrityLevel + ParentCommandLine: process.parent.command_line + Product: winlog.event_data.Product + Protocol: winlog.event_data.Protocol + RuleName: winlog.event_data.RuleName + SourceHostname: source.domain + SourceIp: source.ip + SourceIsIpv6: winlog.event_data.SourceIsIpv6 + SourcePort: source.port + SourcePortName: winlog.event_data.SourcePortName + TargetFilename: file.path + User: + - user.name + - winlog.user.domain + - winlog.user.name + OriginalFileName: winlog.event_data.OriginalFileName + Signed: winlog.event_data.Signed + Signature: winlog.event_data.Signature + SignatureStatus: winlog.event_data.SignatureStatus + TargetObject: winlog.event_data.TargetObject + Details: winlog.event_data.Details + QueryName: dns.question.name + QueryResults: dns.answers + QueryStatus: winlog.event_data.QueryStatus + IsExecutable: winlog.event_data.IsExecutable + PipeName: file.name + ImageLoaded: file.path + ImagePath: winlog.event_data.ImagePath + Imphash: winlog.event_data.Imphash + SourceImage: process.executable + StartModule: winlog.event_data.StartModule + TargetImage: winlog.event_data.TargetImage + Device: Device + ProcessID: ProcessID + FileVersion: FileVersion + StartAddress: StartAddress + StartFunction: StartFunction + EventType: winlog.event_data.EventType \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/opensearch/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_wmi_event.yml new file mode 100644 index 00000000..6ff2c4f3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/opensearch/windows_wmi_event.yml @@ -0,0 +1,15 @@ +platform: OpenSearch +source: windows_wmi_event + + +log_source: + index: [winlogbeat-*, logs-*] + +default_log_source: + index: winlogbeat-* + +field_mapping: + Destination: Destination + EventID: + - event.code + - winlog.event_id diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/default.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/default.yml new file mode 100644 index 00000000..3bb33181 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/default.yml @@ -0,0 +1,6 @@ +platform: Palo Alto Cortex XDR +source: default + + +default_log_source: + datamodel: datamodel diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/linux_file_event.yml new file mode 100644 index 00000000..48cd3530 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/linux_file_event.yml @@ -0,0 +1,29 @@ +platform: Palo Alto Cortex XDR +source: linux_file_event + +log_source: + preset: xdr_file + +default_log_source: + preset: xdr_file + +field_mapping: + TargetFilename: action_file_name + SourceFilename: action_file_previous_file_name + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/linux_process_creation.yml new file mode 100644 index 00000000..683d4b90 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/linux_process_creation.yml @@ -0,0 +1,30 @@ +platform: Palo Alto Cortex XDR +source: linux_process_creation + +log_source: + preset: xdr_process + +default_log_source: + preset: xdr_process + +field_mapping: + User: action_process_username + CommandLine: action_process_image_command_line + Image: action_process_image_path + LogonId: action_process_logon_id + Product: action_process_signature_product + Company: action_process_signature_vendor + IntegrityLevel: action_process_integrity_level + CurrentDirectory: action_process_cwd + ProcessId: action_process_os_pid + ParentProcessId: actor_process_os_pid + ParentCommandLine: actor_process_image_command_line + ParentImage: actor_process_image_path + ParentUser: actor_effective_username + ParentIntegrityLevel: actor_process_integrity_level + ParentLogonId: actor_process_logon_id + ParentProduct: actor_process_signature_product + ParentCompany: actor_process_signature_vendor + md5: action_process_image_md5 + sha256: action_process_image_sha256 + EventID: action_evtlog_event_id \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/macos_file_event.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/macos_file_event.yml new file mode 100644 index 00000000..28639263 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/macos_file_event.yml @@ -0,0 +1,29 @@ +platform: Palo Alto Cortex XDR +source: macos_file_event + +log_source: + preset: xdr_file + +default_log_source: + preset: xdr_file + +field_mapping: + TargetFilename: action_file_name + SourceFilename: action_file_previous_file_name + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/macos_process_creation.yml new file mode 100644 index 00000000..72d368f7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/macos_process_creation.yml @@ -0,0 +1,29 @@ +platform: Palo Alto Cortex XDR +source: macos_process_creation + +log_source: + preset: xdr_process + +default_log_source: + preset: xdr_process + +field_mapping: + User: action_process_username + CommandLine: action_process_image_command_line + Image: action_process_image_path + LogonId: action_process_logon_id + Product: action_process_signature_product + Company: action_process_signature_vendor + IntegrityLevel: action_process_integrity_level + CurrentDirectory: action_process_cwd + ProcessId: action_process_os_pid + ParentProcessId: actor_process_os_pid + ParentCommandLine: actor_process_image_command_line + ParentImage: actor_process_image_path + ParentUser: actor_effective_username + ParentIntegrityLevel: actor_process_integrity_level + ParentLogonId: actor_process_logon_id + ParentProduct: actor_process_signature_product + ParentCompany: actor_process_signature_vendor + md5: action_process_image_md5 + sha256: action_process_image_sha256 \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/windows_file_event.yml new file mode 100644 index 00000000..10065aac --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/windows_file_event.yml @@ -0,0 +1,29 @@ +platform: Palo Alto Cortex XDR +source: windows_file_event + +log_source: + preset: xdr_file + +default_log_source: + preset: xdr_file + +field_mapping: + TargetFilename: action_file_name + SourceFilename: action_file_previous_file_name + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/windows_process_creation.yml new file mode 100644 index 00000000..b3201f3d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/windows_process_creation.yml @@ -0,0 +1,30 @@ +platform: Palo Alto Cortex XDR +source: windows_process_creation + +log_source: + preset: xdr_process + +default_log_source: + preset: xdr_process + +field_mapping: + User: action_process_username + CommandLine: action_process_image_command_line + Image: action_process_image_path + LogonId: action_process_logon_id + Product: action_process_signature_product + Company: action_process_signature_vendor + IntegrityLevel: action_process_integrity_level + CurrentDirectory: action_process_cwd + ProcessId: action_process_os_pid + ParentProcessId: actor_process_os_pid + ParentCommandLine: actor_process_image_command_line + ParentImage: actor_process_image_path + ParentUser: actor_effective_username + ParentIntegrityLevel: actor_process_integrity_level + ParentLogonId: actor_process_logon_id + ParentProduct: actor_process_signature_product + ParentCompany: actor_process_signature_vendor + md5: action_process_image_md5 + sha256: action_process_image_sha256 + OriginalFileName: actor_process_file_original_name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/windows_registry_event.yml new file mode 100644 index 00000000..dbcddfef --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xdr/windows_registry_event.yml @@ -0,0 +1,32 @@ +platform: Palo Alto Cortex XDR +source: windows_registry_event + +log_source: + preset: xdr_registry + +default_log_source: + preset: xdr_registry + +field_mapping: + Details: + - action_registry_value_name + - action_registry_data + TargetObject: action_registry_key_name + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor + EventType: event_sub_type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/apache_httpd.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/apache_httpd.yml new file mode 100644 index 00000000..ee859e86 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/apache_httpd.yml @@ -0,0 +1,16 @@ +platform: Palo Alto Cortex XSIAM +source: apache_httpd + + +default_log_source: + datamodel: datamodel + dataset: apache_httpd_raw + +field_mapping: + c-uri: xdm.network.http.url + c-useragent: xdm.source.user_agent + cs-method: xdm.network.http.method + cs-bytes: xdm.target.sent_bytes + c-uri-query: xdm.network.http.url + cs-referrer: xdm.network.http.referrer + sc-status: xdm.network.http.response_code \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/apache_tomcat.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/apache_tomcat.yml new file mode 100644 index 00000000..821fa0d4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/apache_tomcat.yml @@ -0,0 +1,14 @@ +platform: Palo Alto Cortex XSIAM +source: apache_tomcat + + +default_log_source: + datamodel: datamodel + dataset: apache_tomcat_raw + +field_mapping: + c-uri: xdm.network.http.url + c-useragent: User_agent + cs-method: xdm.network.http.method + cs-bytes: xdm.target.sent_bytes + sc-status: xdm.network.http.response_code \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/aws_cloudtrail.yml new file mode 100644 index 00000000..7e1b6ac9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/aws_cloudtrail.yml @@ -0,0 +1,36 @@ +platform: Palo Alto Cortex XSIAM +source: aws_cloudtrail + + +default_log_source: + dataset: amazon_aws_raw + +field_mapping: + eventSource: eventSource + eventName: eventName + errorCode: errorCode + errorMessage: errorMessage + eventType: eventType + requestParameters: requestParameters + responseElements: responseElements + status: status + terminatingRuleId: terminatingRuleId + userAgent: userAgent + AdditionalEventData.MFAUsed: additionalEventData.MFAUsed + + +raw_log_fields: + additionalEventData.MFAUsed: object + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: object + requestParameters.ipPermissions.items.ipRanges.items.fromPort: object + requestParameters.attribute: object + requestParameters.userData: list + responseElements.ConsoleLogin: object + responseElements.pendingModifiedValues.masterUserPassword: object + responseElements.publiclyAccessible: object + userIdentity.arn: object + userIdentity.principalId: object + userIdentity.sessionContext.sessionIssuer.type: object + userIdentity.type: object + userIdentity.userName: object + requestParameters.publiclyAccessible: object \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/aws_eks.yml new file mode 100644 index 00000000..c7159587 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/aws_eks.yml @@ -0,0 +1,25 @@ +platform: Palo Alto Cortex XSIAM +source: aws_eks + + +default_log_source: + dataset: amazon_aws_raw + +field_mapping: + aws_node_type: aws_node_type + requestURI: requestURI + stage: stage + verb: verb + + +raw_log_fields: + annotations.authorization.k8s.io\/decision: object + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: object + objectRef.namespace: object + objectRef.resource: object + objectRef.subresource: object + requestObject.rules.resources: object + requestObject.rules.verbs: object + requestObject.spec.containers.image: object + user.groups: object + user.username: object \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_aadnoninteractiveusersigninlogs.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_aadnoninteractiveusersigninlogs.yml new file mode 100644 index 00000000..40d419d9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_aadnoninteractiveusersigninlogs.yml @@ -0,0 +1,16 @@ +platform: Palo Alto Cortex XSIAM +source: azure_aadnoninteractiveusersigninlogs + + +default_log_source: + dataset: msft_azure_raw + +field_mapping: + UserAgent: properties.userAgent + Type: properties.type + AuthenticationProcessingDetails: properties.authenticationProcessingDetails + +raw_log_fields: + properties.userAgent: object + properties.type: object + properties.authenticationProcessingDetails: list \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_azureactivity.yml new file mode 100644 index 00000000..78cb3137 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_azureactivity.yml @@ -0,0 +1,33 @@ +platform: Palo Alto Cortex XSIAM +source: azure_azureactivity + + +default_log_source: + dataset: msft_azure_raw + +field_mapping: + ActivityStatus: properties.activityStatus + ActivityStatusValue: properties.activityStatusValue + ActivitySubstatusValue: properties.activitySubstatusValue + Authorization: properties.authorization + Category: properties.category + CategoryValue: properties.categoryValue + OperationName: properties.operationName + OperationNameValue: oproperties.perationNameValue + ResourceId: properties.resourceId + ResourceProviderValue: properties.resourceProviderValue + Type: properties.type + operationName: properties.operationName + +raw_log_fields: + properties.activityStatus: object + properties.activityStatusValue: object + properties.activitySubstatusValue: object + properties.authorization: object + properties.category: object + properties.categoryValue: object + properties.operationName: object + properties.operationNameValue: object + properties.resourceId: object + properties.resourceProviderValue: object + properties.type: object \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_azuread.yml new file mode 100644 index 00000000..6044b336 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_azuread.yml @@ -0,0 +1,34 @@ +platform: Palo Alto Cortex XSIAM +source: azure_azuread + + +default_log_source: + dataset: msft_azure_raw + +field_mapping: + ActivityDisplayName: properties.activityDisplayName + Category: properties.category + LoggedByService: properties.loggedByService + Result: properties.result + OperationName: properties.operationName + TargetResources: properties.targetResources + AADOperationType: properties.AADOperationType + InitiatedBy: properties.initiatedBy + ResultReason: properties.resultReason + Status: properties.status + #Status.errorCode: properties.status_errorCode + UserAgent: properties.userAgent + +raw_log_fields: + properties.activityDisplayName: object + properties.category: object + properties.loggedByService: object + properties.result: object + properties.operationName: object + properties.targetResources: object + properties.AADOperationType: object + properties.initiatedBy: object + properties.resultReason: object + properties.status: object + properties.status_errorCode: object + properties.userAgent: object \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_m365.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_m365.yml new file mode 100644 index 00000000..94e7a832 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_m365.yml @@ -0,0 +1,34 @@ +platform: Palo Alto Cortex XSIAM +source: azure_m365 + + +default_log_source: + dataset: msft_azure_raw + +field_mapping: + ClientInfoString: properties.clientInfoString + LogonError: properties.logonError + ModifiedProperties: properties.modifiedProperties + OfficeObjectId: properties.officeObjectId + OfficeWorkload: properties.officeWorkload + Operation: properties.operation + Parameters: properties.parameters + RecordType: properties.recordType + ResultStatus: properties.resultStatus + SourceFileExtension: properties.sourceFileExtension + SourceFileName: properties.sourceFileName + UserAgent: properties.userAgent + +raw_log_fields: + properties.clientInfoString: object + properties.logonError: object + properties.modifiedProperties: object + properties.officeObjectId: object + properties.officeWorkload: object + properties.operation: object + properties.parameters: object + properties.recordType: object + properties.resultStatus: object + properties.sourceFileExtension: object + properties.sourceFileName: object + properties.userAgent: object \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_signinlogs.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_signinlogs.yml new file mode 100644 index 00000000..5aafbe6a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/azure_signinlogs.yml @@ -0,0 +1,46 @@ +platform: Palo Alto Cortex XSIAM +source: azure_signinlogs + + +default_log_source: + dataset: msft_azure_raw + +field_mapping: + AppDisplayName: properties.appDisplayName + AppId: properties.appId + AuthenticationRequirement: properties.authenticationRequirement + Category: properties.category + ConditionalAccessStatus: properties.conditionalAccessStatus + DeviceDetail: properties.deviceDetail + IsInteractive: properties.isInteractive + NetworkLocationDetails: properties.networkLocationDetails + ResourceDisplayName: properties.resourceDisplayName + ResourceIdentity: properties.resourceIdentity + ResultDescription: properties.resultDescription + ResultType: properties.resultType + Status.errorCode: properties.status.errorCode + Status: properties.status + Status.failureReason: properties.status.failureReason + TokenIssuerType: properties.tokenIssuerType + UserAgent: properties.userAgent + UserPrincipalName: properties.userPrincipalName + +raw_log_fields: + properties.appDisplayName: object + properties.appId: object + properties.authenticationRequirement: object + properties.category: object + properties.conditionalAccessStatus: object + properties.deviceDetail: object + properties.isInteractive: object + properties.networkLocationDetails: object + properties.resourceDisplayName: object + properties.resourceIdentity: object + properties.resultDescription: object + properties.resultType: object + properties.status.errorCode: object + properties.status: object + properties.status.failureReason: object + properties.tokenIssuerType: object + properties.userAgent: object + properties.userPrincipalName: object \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/default.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/default.yml new file mode 100644 index 00000000..7405d27b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/default.yml @@ -0,0 +1,141 @@ +platform: Palo Alto Cortex XSIAM +source: default + + +default_log_source: + datamodel: datamodel + + +field_mapping: + CommandLine: xdm.target.process.command_line + Image: + - xdm.target.process.name + - xdm.source.process.name + ProcessName: + - xdm.target.process.name + - xdm.source.process.name + ProcessPath: xdm.target.process.executable.path + ImageLoaded: + - xdm.target.process.executable.filename + - xdm.source.process.executable.filename + ParentCommandLine: xdm.source.process.command_line + ParentImage: xdm.source.process.name + User: xdm.source.user.username + TargetFilename: xdm.target.file.filename + TargetImage: xdm.target.process.name + SourceImage: xdm.source.process.name + EventID: xdm.event.id + Protocol: xdm.network.ip_protocol + src-ip: xdm.source.ipv4 + SourceIp: xdm.source.ipv4 + src-packets: xdm.source.sent_packets + dst-packets: xdm.target.sent_packets + src-port: xdm.source.port + SourcePort: xdm.source.port + dst-ip: xdm.target.ipv4 + DestinationIp: xdm.target.ipv4 + dst-port: xdm.target.port + DestinationPort: xdm.target.port + destinationPort: xdm.target.port + src-bytes: xdm.source.sent_bytes + dst-bytes: xdm.target.sent_bytes + src-hostname: xdm.source.host.hostname + dst-hostname: xdm.target.host.hostname + icmp.type: xdm.network.icmp.type + icmp.code: xdm.network.icmp.code + c-uri: xdm.network.http.url + c-uri-query: xdm.network.http.url + QueryName: xdm.network.dns.dns_question.name + Application: xdm.network.application_protocol + sourceNetwork: xdm.source.subnet + SourceHostName: xdm.source.host.hostname + DestinationHostname: xdm.target.host.hostname + Hashes: + - xdm.source.process.executable.sha256 + - xdm.source.process.executable.md5 + IntegrityLevel: xdm.source.process.integrity_level + SourceHostname: xdm.source.host.hostname + Signed: xdm.source.process.executable.is_signed + Signature: xdm.source.process.executable.signer + SignatureStatus: xdm.source.process.executable.signature_status + QueryResults: xdm.network.dns.dns_resource_record.value + QueryStatus: xdm.network.dns.response_code + ImagePath: xdm.target.process.executable.path + ProcessID: xdm.target.process.pid + EventType: xdm.event.original_event_type + dns-query: xdm.network.dns.dns_question.name + dns-answer: xdm.network.dns.dns_resource_record.value + dns-record: xdm.network.dns.dns_question.name + FileName: xdm.target.file.filename + IpAddress: xdm.source.ipv4 + IpPort: xdm.source.port + LogonProcessName: xdm.target.process.executable.path + ProcessId: xdm.target.process.pid + NewProcessName: xdm.target.process.executable.path + Path: xdm.target.process.executable.path + ParentProcessId: xdm.source.process.pid + GroupMembership: xdm.target.user.groups + NewTargetUserName: xdm.target.user.username + OldTargetUserName: xdm.target.user.username + UserPrincipalName: xdm.source.user.username + DestAddress: xdm.target.ipv4 + SubjectAccountName: xdm.source.user.username + SubjectUserName: xdm.source.user.username + SubjectUserSid: xdm.source.user.identifier + SourceAddr: xdm.source.ipv4 + SourceAddress: xdm.source.ipv4 + TargetSid: xdm.target.user.identifier + TargetUserName: xdm.target.user.username + SourceUserName: xdm.source.user.username + ParentProcessName: xdm.source.process.executable.path + client.user.full_name: xdm.target.user.username + source.user.full_name: xdm.source.user.username + client.ip: xdm.source.ipv4 + user_agent.original: xdm.network.http.browser + userAgent.os: xdm.network.http.browser + userAgent.browser: xdm.network.http.browser + eventtype: xdm.event.type + a0: xdm.target.process.command_line + a1: xdm.target.process.command_line + a2: xdm.target.process.command_line + a3: xdm.target.process.command_line + exe: xdm.target.process.executable.path + userAgent: xdm.network.http.browser + userIdentity.userName: userIdentity.userName + user.username: xdm.source.user.username + AccountName: xdm.target.user.username + processPath: xdm.target.process.executable.path + QNAME: xdm.network.dns.dns_question.name + ImageName: xdm.target.process.name + UserName: xdm.source.user.username + Address: xdm.source.ipv4 + Process_Name: xdm.target.process.executable.path + uri: xdm.network.http.url + query: xdm.network.dns.dns_question.name + referrer: xdm.network.http.referrer + user_agent: xdm.network.http.browser + status_code: xdm.network.http.response_code + qtype_name: xdm.network.dns.opcode + http.method: xdm.network.http.method + method: xdm.network.http.method + notice.user_agent: xdm.network.http.browser + hasIdentity: xdm.source.user.identity_type + ComputerName: xdm.source.host.hostname + ExternalSeverity: xdm.alert.severity + SourceMAC: xdm.source.host.mac_addresses + DestinationMAC: xdm.target.host.mac_addresses + SourceOS: xdm.source.host.os + DestinationOS: xdm.target.host.os + url_category: xdm.network.http.url_category + EventSeverity: xdm.alert.severity + duration: xdm.event.duration + ThreatName: xdm.alert.original_threat_id + AnalyzerName: xdm.observer.type + Classification: xdm.alert.category + ResultCode: xdm.event.outcome_reason + Technique: xdm.alert.mitre_techniques + Action: xdm.event.outcome + FileExtension: xdm.target.file.extension + Workstation: xdm.source.host.hostname + RegistryKey: xdm.target.registry.key + RegistryValue: xdm.target.registry.value diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/dns.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/dns.yml new file mode 100644 index 00000000..ceb20d2d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/dns.yml @@ -0,0 +1,14 @@ +platform: Palo Alto Cortex XSIAM +source: dns + +default_log_source: + datamodel: datamodel + +field_mapping: + dns-query: xdm.network.dns.dns_question.name + dns-answer: xdm.network.dns.dns_resource_record.value + #dns-record: dns-record + dns_query_name: xdm.network.dns.dns_question.name + QueryName: xdm.network.dns.dns_question.name + query: xdm.network.dns.dns_question.name + dns-record-type: xdm.network.dns.dns_question.type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/firewall.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/firewall.yml new file mode 100644 index 00000000..b85d5706 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/firewall.yml @@ -0,0 +1,56 @@ +platform: Palo Alto Cortex XSIAM +source: firewall + +log_source: + preset: network_story + +default_log_source: + preset: network_story + +field_mapping: + DestinationPort: + - action_local_port + - action_remote_port + DestinationIp: + - action_local_ip + - action_remote_ip + SourcePort: + - action_local_port + - action_remote_port + SourceIp: + - action_local_ip + - action_remote_ip + dst-ip: + - action_local_ip + - action_remote_ip + dst-port: + - action_local_port + - action_remote_port + src-ip: + - action_local_ip + - action_remote_ip + src-port: + - action_local_port + - action_remote_port + Protocol: action_network_protocol + DestinationHostname: action_external_hostname + SourceHostname: agent_hostname + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor + Application: xdm.network.application_protocol + application: xdm.network.application_protocol \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/linux_file_event.yml new file mode 100644 index 00000000..92223940 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/linux_file_event.yml @@ -0,0 +1,29 @@ +platform: Palo Alto Cortex XSIAM +source: linux_file_event + +log_source: + preset: xdr_file + +default_log_source: + preset: xdr_file + +field_mapping: + TargetFilename: action_file_name + SourceFilename: action_file_previous_file_name + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/linux_network_connection.yml new file mode 100644 index 00000000..1e1933e7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/linux_network_connection.yml @@ -0,0 +1,54 @@ +platform: Palo Alto Cortex XSIAM +source: linux_network_connection + +log_source: + preset: network_story + +default_log_source: + preset: network_story + +field_mapping: + DestinationPort: + - action_local_port + - action_remote_port + DestinationIp: + - action_local_ip + - action_remote_ip + SourcePort: + - action_local_port + - action_remote_port + SourceIp: + - action_local_ip + - action_remote_ip + dst_ip: + - action_local_ip + - action_remote_ip + dst_port: + - action_local_port + - action_remote_port + src_ip: + - action_local_ip + - action_remote_ip + src_port: + - action_local_port + - action_remote_port + Protocol: action_network_protocol + DestinationHostname: action_external_hostname + SourceHostname: agent_hostname + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/linux_process_creation.yml new file mode 100644 index 00000000..1245f22f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/linux_process_creation.yml @@ -0,0 +1,30 @@ +platform: Palo Alto Cortex XSIAM +source: linux_process_creation + +log_source: + preset: xdr_process + +default_log_source: + preset: xdr_process + +field_mapping: + User: action_process_username + CommandLine: action_process_image_command_line + Image: action_process_image_path + LogonId: action_process_logon_id + Product: action_process_signature_product + Company: action_process_signature_vendor + IntegrityLevel: action_process_integrity_level + CurrentDirectory: action_process_cwd + ProcessId: action_process_os_pid + ParentProcessId: actor_process_os_pid + ParentCommandLine: actor_process_image_command_line + ParentImage: actor_process_image_path + ParentUser: actor_effective_username + ParentIntegrityLevel: actor_process_integrity_level + ParentLogonId: actor_process_logon_id + ParentProduct: actor_process_signature_product + ParentCompany: actor_process_signature_vendor + md5: action_process_image_md5 + sha256: action_process_image_sha256 + EventID: action_evtlog_event_id \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/macos_file_event.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/macos_file_event.yml new file mode 100644 index 00000000..60899029 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/macos_file_event.yml @@ -0,0 +1,29 @@ +platform: Palo Alto Cortex XSIAM +source: macos_file_event + +log_source: + preset: xdr_file + +default_log_source: + preset: xdr_file + +field_mapping: + TargetFilename: action_file_name + SourceFilename: action_file_previous_file_name + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/macos_network_connection.yml new file mode 100644 index 00000000..727a1a8d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/macos_network_connection.yml @@ -0,0 +1,54 @@ +platform: Palo Alto Cortex XSIAM +source: macos_network_connection + +log_source: + preset: network_story + +default_log_source: + preset: network_story + +field_mapping: + DestinationPort: + - action_local_port + - action_remote_port + DestinationIp: + - action_local_ip + - action_remote_ip + SourcePort: + - action_local_port + - action_remote_port + SourceIp: + - action_local_ip + - action_remote_ip + dst_ip: + - action_local_ip + - action_remote_ip + dst_port: + - action_local_port + - action_remote_port + src_ip: + - action_local_ip + - action_remote_ip + src_port: + - action_local_port + - action_remote_port + Protocol: action_network_protocol + DestinationHostname: action_external_hostname + SourceHostname: agent_hostname + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/macos_process_creation.yml new file mode 100644 index 00000000..e02e77a4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/macos_process_creation.yml @@ -0,0 +1,29 @@ +platform: Palo Alto Cortex XSIAM +source: macos_process_creation + +log_source: + preset: xdr_process + +default_log_source: + preset: xdr_process + +field_mapping: + User: action_process_username + CommandLine: action_process_image_command_line + Image: action_process_image_path + LogonId: action_process_logon_id + Product: action_process_signature_product + Company: action_process_signature_vendor + IntegrityLevel: action_process_integrity_level + CurrentDirectory: action_process_cwd + ProcessId: action_process_os_pid + ParentProcessId: actor_process_os_pid + ParentCommandLine: actor_process_image_command_line + ParentImage: actor_process_image_path + ParentUser: actor_effective_username + ParentIntegrityLevel: actor_process_integrity_level + ParentLogonId: actor_process_logon_id + ParentProduct: actor_process_signature_product + ParentCompany: actor_process_signature_vendor + md5: action_process_image_md5 + sha256: action_process_image_sha256 \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/nginx_nginx.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/nginx_nginx.yml new file mode 100644 index 00000000..54072934 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/nginx_nginx.yml @@ -0,0 +1,16 @@ +platform: Palo Alto Cortex XSIAM +source: nginx_nginx + + +default_log_source: + datamodel: datamodel + dataset: nginx_nginx_raw + +field_mapping: + c-uri: xdm.network.http.url + c-useragent: User_agent + cs-method: xdm.network.http.method + cs-bytes: xdm.target.sent_bytes + c-uri-query: xdm.network.http.url + cs-referrer: xdm.event.description + sc-status: xdm.network.http.response_code \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/okta_okta.yml new file mode 100644 index 00000000..db2e2c47 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/okta_okta.yml @@ -0,0 +1,11 @@ +platform: Palo Alto Cortex XSIAM +source: okta_okta + + +default_log_source: + datamodel: datamodel + dataset: okta_okta_raw + +field_mapping: + eventType: xdm.event.type + eventtype: xdm.event.type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/proxy.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/proxy.yml new file mode 100644 index 00000000..846f872d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/proxy.yml @@ -0,0 +1,23 @@ +platform: Palo Alto Cortex XSIAM +source: proxy + +default_log_source: + datamodel: datamodel + +field_mapping: + c-uri: xdm.network.http.url + c-useragent: xdm.network.http.browser + cs-method: xdm.network.http.method + cs-bytes: xdm.target.sent_bytes + c-uri-query: xdm.network.http.url + cs-referrer: xdm.network.http.referrer + sc-status: xdm.network.http.response_code + cs-host: xdm.network.http.domain + cs-uri-query: xdm.network.http.url + cs-cookie-vars: xdm.network.http.http_header.value + c-uri-extension: xdm.network.http.url + cs-cookie: xdm.network.http.http_header.value + #cs-version: cs-version + r-dns: xdm.network.http.domain + post-body: xdm.network.http.http_header.value + url_category: xdm.network.http.url_category \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/slack_slack_raw.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/slack_slack_raw.yml new file mode 100644 index 00000000..6098e617 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/slack_slack_raw.yml @@ -0,0 +1,10 @@ +platform: Palo Alto Cortex XSIAM +source: slack_slack_raw + + +default_log_source: + datamodel: datamodel + dataset: slack_slack_raw + +field_mapping: + c-action: xdm.event.operation \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/webserver.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/webserver.yml new file mode 100644 index 00000000..b7791fc5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/webserver.yml @@ -0,0 +1,20 @@ +platform: Palo Alto Cortex XSIAM +source: webserver + +default_log_source: + datamodel: datamodel + +field_mapping: + c-uri: xdm.network.http.url + c-useragent: xdm.source.user_agent + cs-method: xdm.network.http.method + cs-bytes: xdm.target.sent_bytes + c-uri-query: xdm.network.http.url + cs-referrer: xdm.network.http.referrer + sc-status: xdm.network.http.response_code + cs-uri-stem: xdm.network.http.url + cs-uri-query: xdm.network.http.url + c-uri-path: xdm.network.http.url + cs-host: xdm.network.http.domain + uri_path: xdm.network.http.url + cs-uri: xdm.network.http.url diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_application.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_application.yml new file mode 100644 index 00000000..f215f241 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_application.yml @@ -0,0 +1,24 @@ +platform: Palo Alto Cortex XSIAM +source: windows_application + +default_log_source: + preset: xdr_event_log + +field_mapping: + EventID: action_evtlog_event_id + Provider_Name: provider_name + +raw_log_fields: + src_ip: regex + source: regex + additional_information: regex + EventData: regex + Channel: regex + statement: regex + Faulting application path: regex + object_name: regex + class_type: regex + action_id: regex + Data: regex + Message: regex + Level: regex diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_file_event.yml new file mode 100644 index 00000000..736f6215 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_file_event.yml @@ -0,0 +1,29 @@ +platform: Palo Alto Cortex XSIAM +source: windows_file_event + +log_source: + preset: xdr_file + +default_log_source: + preset: xdr_file + +field_mapping: + TargetFilename: action_file_name + SourceFilename: action_file_previous_file_name + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_image_load.yml new file mode 100644 index 00000000..daaffa63 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_image_load.yml @@ -0,0 +1,32 @@ +platform: Palo Alto Cortex XSIAM +source: windows_image_load + +log_source: + preset: xdr_image_load + +default_log_source: + preset: xdr_image_load + +field_mapping: + ImageLoaded: action_module_path + FileExtension: action_file_extension + md5: action_module_md5 + sha256: action_module_sha256 + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor + Signed: actor_process_signature_status #Signature status of the process: Signed = 1 SignedInvalid = 2 Unsigned = 3 FailedToObtain = 4 WeakHash = 5, where the MD5 is used as the hash algorithm. Unsupported = 6, which means the signature was not calculated. InvalidCVE2020_0601 = 7, which means the executable is malicious and is trying to exploit the windows vulnerability CVE2020-0601. Deleted = 8, which means that the file was deleted by the time the agent tried to calculate the signature. \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_network_connection.yml new file mode 100644 index 00000000..ba6ea04c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_network_connection.yml @@ -0,0 +1,58 @@ +platform: Palo Alto Cortex XSIAM +source: windows_network_connection + +log_source: + preset: network_story + +default_log_source: + preset: network_story + +field_mapping: + DestinationPort: + - action_local_port + - action_remote_port + DestinationIp: + - action_local_ip + - action_remote_ip + SourcePort: + - action_local_port + - action_remote_port + SourceIp: + - action_local_ip + - action_remote_ip + dst_ip: + - action_local_ip + - action_remote_ip + dst_port: + - action_local_port + - action_remote_port + src_ip: + - action_local_ip + - action_remote_ip + src_port: + - action_local_port + - action_remote_port + Protocol: action_network_protocol + DestinationHostname: action_external_hostname + SourceHostname: agent_hostname + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor + + +raw_log_fields: + Initiated: regex \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_pipe_created.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_pipe_created.yml new file mode 100644 index 00000000..0fae37fe --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_pipe_created.yml @@ -0,0 +1,12 @@ +platform: Palo Alto Cortex XSIAM +source: windows_pipe_created + +default_log_source: + preset: xdr_event_log + +field_mapping: + EventID: action_evtlog_event_id + +raw_log_fields: + PipeName: regex + Image: regex \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_powershell.yml new file mode 100644 index 00000000..100c75d3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_powershell.yml @@ -0,0 +1,20 @@ +platform: Palo Alto Cortex XSIAM +source: windows_powershell + + +default_log_source: + preset: xdr_event_log + +field_mapping: + EventID: action_evtlog_event_id + + +raw_log_fields: + CommandLine: regex + ScriptBlockText: regex + Payload: regex + HostApplication: regex + ContextInfo: regex + HostName: regex + EngineVersion: regex + Path: regex \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_process_access.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_process_access.yml new file mode 100644 index 00000000..f626eed5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_process_access.yml @@ -0,0 +1,20 @@ +platform: Palo Alto Cortex XSIAM +source: windows_process_access + +default_log_source: + preset: xdr_event_log + +field_mapping: + User: action_process_username + SourceUser: action_process_username + +raw_log_fields: + SourceProcessGUID: regex + SourceProcessId: regex + SourceThreadId: regex + SourceImage: regex + TargetProcessGUID: regex + TargerProcessId: regex + TargetImage: regex + GrantedAccess: regex + CallTrace: regex \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_process_creation.yml new file mode 100644 index 00000000..ec7f6cd2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_process_creation.yml @@ -0,0 +1,30 @@ +platform: Palo Alto Cortex XSIAM +source: windows_process_creation + +log_source: + preset: xdr_process + +default_log_source: + preset: xdr_process + +field_mapping: + User: action_process_username + CommandLine: action_process_image_command_line + Image: action_process_image_path + LogonId: action_process_logon_id + Product: action_process_signature_product + Company: action_process_signature_vendor + IntegrityLevel: action_process_integrity_level + CurrentDirectory: action_process_cwd + ProcessId: action_process_os_pid + ParentProcessId: actor_process_os_pid + ParentCommandLine: actor_process_image_command_line + ParentImage: actor_process_image_path + ParentUser: actor_effective_username + ParentIntegrityLevel: actor_process_integrity_level + ParentLogonId: actor_process_logon_id + ParentProduct: actor_process_signature_product + ParentCompany: actor_process_signature_vendor + md5: action_process_image_md5 + sha256: action_process_image_sha256 + OriginalFileName: actor_process_file_original_name \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_process_termination.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_process_termination.yml new file mode 100644 index 00000000..baf07e5b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_process_termination.yml @@ -0,0 +1,13 @@ +platform: Palo Alto Cortex XSIAM +source: windows_process_termination + +log_source: + preset: xdr_process + +default_log_source: + preset: xdr_process + +field_mapping: + Image: action_process_image_path + ProcessId: action_process_os_pid + ProcessGuid: ProcessGuid \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_registry_event.yml new file mode 100644 index 00000000..fc2a4b71 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_registry_event.yml @@ -0,0 +1,32 @@ +platform: Palo Alto Cortex XSIAM +source: windows_registry_event + +log_source: + preset: xdr_registry + +default_log_source: + preset: xdr_registry + +field_mapping: + Details: + - action_registry_value_name + - action_registry_data + TargetObject: action_registry_key_name + User: actor_effective_username + CommandLine: actor_process_image_command_line + Image: actor_process_image_path + LogonId: actor_process_logon_id + Product: actor_process_signature_product + Company: actor_process_signature_vendor + IntegrityLevel: actor_process_integrity_level + CurrentDirectory: actor_process_cwd + ProcessId: actor_process_os_id + ParentProcessId: causality_actor_process_os_id + ParentCommandLine: causality_actor_process_command_line + ParentImage: causality_actor_process_image_path + ParentUser: causality_actor_effective_username + ParentIntegrityLevel: causality_actor_process_integrity_level + ParentLogonId: causality_actor_process_logon_id + ParentProduct: causality_actor_process_signature_product + ParentCompany: causality_actor_process_signature_vendor + EventType: event_sub_type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_security.yml new file mode 100644 index 00000000..0c446f2a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_security.yml @@ -0,0 +1,152 @@ +platform: Palo Alto Cortex XSIAM +source: windows_security + +default_log_source: + preset: xdr_event_log + +field_mapping: + EventID: action_evtlog_event_id + Provider_Name: provider_name + SubjectAccountName: actor_effective_username + +raw_log_fields: + ParentImage: regex + AccessMask: regex + AccountName: regex + AllowedToDelegateTo: regex + AttributeLDAPDisplayName: regex + AuditPolicyChanges: regex + AuthenticationPackageName: regex + CallingProcessName: regex + Channel: regex + ComputerName: regex + EventType: regex + FailureReason: regex + FileName: regex + GrantedAccess: regex + Hashes: regex + HiveName: regex + IpAddress: regex + IpPort: regex + KeyLength: regex + LogonProcessName: regex + LogonType: regex + LinkName: regex + ProcessId: regex + PublishURLs: regex + ElevatedToken: regex + MemberName: regex + MemberSid: regex + NewProcessName: regex + ObjectClass: regex + ObjectName: regex + ObjectType: regex + ObjectValueName: regex + Path: regex + CommandLine: regex + OldUacValue: regex + CertIssuerName: regex + SubStatus: regex + DisplayName: regex + TaskContent: regex + ServiceSid: regex + CertThumbprint: regex + ClassName: regex + NotificationPackageName: regex + NewSd: regex + TestSigning: regex + TargetInfo: regex + ParentProcessId: regex + AccessList: regex + GroupMembership: regex + FilterName: regex + ChangeType: regex + LayerName: regex + ServiceAccount: regex + ClientProcessId: regex + AttributeValue: regex + SessionName: regex + TaskName: regex + ObjectDN: regex + TemplateContent: regex + NewTemplateContent: regex + SourcePort: regex + PasswordLastSet: regex + PrivilegeList: regex + DeviceDescription: regex + TargetServerName: regex + NewTargetUserName: regex + OperationType: regex + DestPort: regex + ServiceStartType: regex + OldTargetUserName: regex + UserPrincipalName: regex + Accesses: regex + DnsHostName: regex + DisableIntegrityChecks: regex + AuditSourceName: regex + Workstation: regex + DestAddress: regex + PreAuthType: regex + SecurityPackageName: regex + SubjectLogonId: regex + NewUacValue: regex + EnabledPrivilegeList: regex + RelativeTargetName: regex + CertSerialNumber: regex + SidHistory: regex + TargetLogonId: regex + KernelDebug: regex + CallerProcessName: regex + ProcessName: regex + Properties: regex + UserAccountControl: regex + RegistryValue: regex + SecurityID: regex + ServiceFileName: regex + SecurityDescriptor: regex + ServiceName: regex + ShareName: regex + NewValue: regex + Source: regex + Status: regex + SubjectDomainName: regex + SubjectUserName: regex + SubjectUserSid: regex + SourceAddr: regex + SourceAddress: regex + TargetName: regex + ServicePrincipalNames: regex + TargetDomainName: regex + TargetSid: regex + TargetUserName: regex + ObjectServer: regex + TargetUserSid: regex + TicketEncryptionType: regex + TicketOptions: regex + WorkstationName: regex + TransmittedServices: regex + AuthenticationAlgorithm: regex + LayerRTID: regex + BSSID: regex + BSSType: regex + CipherAlgorithm: regex + ConnectionId: regex + ConnectionMode: regex + InterfaceDescription: regex + InterfaceGuid: regex + OnexEnabled: regex + PHYType: regex + ProfileName: regex + SSID: regex + Domain: regex + ServiceType: regex + SourceName: regex + StartType: regex + UserID: regex + ParentProcessName: regex + ExceptionCode: regex + Service: regex + SamAccountName: regex + ImpersonationLevel: regex + PrimaryGroupId: regex \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_sysmon.yml new file mode 100644 index 00000000..8609ef23 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_sysmon.yml @@ -0,0 +1,61 @@ +platform: Palo Alto Cortex XSIAM +source: windows_sysmon + + +default_log_source: + preset: xdr_event_log + +field_mapping: + EventID: action_evtlog_event_id + OriginalFileName: actor_process_file_original_name + Description: action_evtlog_description + +raw_log_fields: + CommandLine: regex + Image: regex + ParentImage: regex + CallTrace: regex + Company: regex + CurrentDirectory: regex + DestinationHostname: regex + DestinationIp: regex + DestinationIsIpv6: regex + DestinationPort: regex + DestinationPortName: regex + Hashes: regex + Initiated: regex + IntegrityLevel: regex + ParentCommandLine: regex + Product: regex + Protocol: regex + RuleName: regex + SourceHostname: regex + SourceIp: regex + SourceIsIpv6: regex + SourcePort: regex + SourcePortName: regex + TargetFilename: regex + User: regex + Signed: regex + Signature: regex + SignatureStatus: regex + TargetObject: regex + Details: regex + QueryName: regex + QueryResults: regex + QueryStatus: regex + IsExecutable: regex + PipeName: regex + ImageLoaded: regex + ImagePath: regex + Imphash: regex + SourceImage: regex + StartModule: regex + TargetImage: regex + Device: regex + ProcessID: regex + FileVersion: regex + StartAddress: regex + StartFunction: regex + EventType: regex + GrantedAccess: regex \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_system.yml b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_system.yml new file mode 100644 index 00000000..5e602fa3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/palo_alto_cortex_xsiam/windows_system.yml @@ -0,0 +1,25 @@ +platform: Palo Alto Cortex XSIAM +source: windows_system + +default_log_source: + preset: xdr_event_log + +field_mapping: + EventID: action_evtlog_event_id + Provider_Name: provider_name + ImagePath: actor_process_image_path + +raw_log_fields: + AccountName: regex + ServiceName: regex + ServiceType: regex + StartType: regex + Origin: regex + HiveName: regex + Caption: regex + param1: regex + param2: regex + Channel: regex + DeviceName: regex + Message: regex + ComputerName: regex \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/qradar/aws_cloudtrail.yml new file mode 100644 index 00000000..cf2f4351 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/aws_cloudtrail.yml @@ -0,0 +1,35 @@ +platform: Qradar +source: aws_cloudtrail + + +log_source: + devicetype: [347] + +default_log_source: + devicetype: 347 + +field_mapping: + eventSource: eventSource + eventName: eventName + AdditionalEventData: AdditionalEventData + additionalEventData.MFAUsed: additionalEventData.MFAUsed + errorCode: errorCode + errorMessage: errorMessage + eventType: eventType + requestParameters: requestParameters + requestParameters.attribute: requestParameters.attribute + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP + requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort + requestParameters.userData: requestParameters.userData + responseElements: responseElements + responseElements.ConsoleLogin: responseElements.ConsoleLogin + responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword + responseElements.publiclyAccessible: responseElements.publiclyAccessible + status: status + terminatingRuleId: terminatingRuleId + userAgent: userAgent + userIdentity.arn: userIdentity.arn + userIdentity.principalId: userIdentity.principalId + userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type + userIdentity.type: userIdentity.type + userIdentity.userName: userIdentity.userName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/qradar/aws_eks.yml new file mode 100644 index 00000000..b1991bdc --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/aws_eks.yml @@ -0,0 +1,25 @@ +platform: Qradar +source: aws_eks + + +log_source: + devicetype: [382] + +default_log_source: + devicetype: 382 + +field_mapping: + annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy + aws_node_type: aws_node_type + objectRef.namespace: objectRef.namespace + objectRef.resource: objectRef.resource + objectRef.subresource: objectRef.subresource + requestObject.rules.resources: requestObject.rules.resources + requestObject.rules.verbs: requestObject.rules.verbs + requestObject.spec.containers.image: requestObject.spec.containers.image + requestURI: requestURI + stage: stage + user.groups: user.groups + user.username: user.username + verb: verb \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/qradar/azure_azureactivity.yml new file mode 100644 index 00000000..ad69505f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/azure_azureactivity.yml @@ -0,0 +1,22 @@ +platform: Qradar +source: azure_azureactivity + + +log_source: + devicetype: [413] + +default_log_source: + devicetype: 413 + +field_mapping: + ActivityStatus: ActivityStatus + ActivityStatusValue: ActivityStatusValue + ActivitySubstatusValue: ActivitySubstatusValue + Authorization: Authorization + Category: Category + CategoryValue: CategoryValue + OperationName: OperationName + OperationNameValue: OperationNameValue + ResourceId: ResourceId + ResourceProviderValue: ResourceProviderValue + Type: Type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/qradar/azure_azuread.yml new file mode 100644 index 00000000..cb2b3190 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/azure_azuread.yml @@ -0,0 +1,23 @@ +platform: Qradar +source: azure_azuread + + +log_source: + devicetype: [445] + +default_log_source: + devicetype: 445 + +field_mapping: + ActivityDisplayName: ActivityDisplayName + Category: Category + LoggedByService: LoggedByService + Result: Result + OperationName: OperationName + TargetResources: TargetResources + AADOperationType: AADOperationType + InitiatedBy: InitiatedBy + ResultReason: ResultReason + Status: Status + Status.errorCode: Status.errorCode + UserAgent: UserAgent \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/azure_m365.yml b/uncoder-core/app/translator/mappings/platforms/qradar/azure_m365.yml new file mode 100644 index 00000000..ed6426fa --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/azure_m365.yml @@ -0,0 +1,23 @@ +platform: Qradar +source: azure_m365 + + +log_source: + devicetype: [397] + +default_log_source: + devicetype: 397 + +field_mapping: + ClientInfoString: ClientInfoString + LogonError: LogonError + ModifiedProperties: ModifiedProperties + OfficeObjectId: OfficeObjectId + OfficeWorkload: OfficeWorkload + Operation: Operation + Parameters: + RecordType: RecordType + ResultStatus: ResultStatus + SourceFileExtension: SourceFileExtension + SourceFileName: SourceFileName + UserAgent: UserAgent \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/azure_signinlogs.yml b/uncoder-core/app/translator/mappings/platforms/qradar/azure_signinlogs.yml new file mode 100644 index 00000000..d2a0193d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/azure_signinlogs.yml @@ -0,0 +1,29 @@ +platform: Qradar +source: azure_signinlogs + + +log_source: + devicetype: [413] + +default_log_source: + devicetype: 413 + +field_mapping: + AppDisplayName: AppDisplayName + AppId: AppId + AuthenticationRequirement: AuthenticationRequirement + Category: Category + ConditionalAccessStatus: ConditionalAccessStatus + DeviceDetail: DeviceDetail + IsInteractive: IsInteractive + NetworkLocationDetails: NetworkLocationDetails + ResourceDisplayName: ResourceDisplayName + ResourceIdentity: ResourceIdentity + ResultDescription: ResultDescription + ResultType: ResultType + Status.errorCode: Status.errorCode + Status: Status + Status.failureReason: Status.failureReason + TokenIssuerType: TokenIssuerType + UserAgent: UserAgent + UserPrincipalName: UserPrincipalName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/default.yml b/uncoder-core/app/translator/mappings/platforms/qradar/default.yml new file mode 100644 index 00000000..69adb819 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/default.yml @@ -0,0 +1,106 @@ +platform: Qradar +source: default + + + +default_log_source: + devicetype: 12 + + +field_mapping: + icmp.type: IcmpType + icmp.code: IcmpCode + dst-port: + - DstPort + - DestinationPort + - remoteport + src-port: + - SourcePort + - localport + - sourcePort + src-ip: + - sourceip + - source_ip + - SourceIP + - sourceIP + - SrcHost + dst-ip: + - DestinationIP + - destinationip + - destination_ip + - destinationIP + - destinationaddress + - destination + - DstHost + User: + - userName + - EventUserName + - Alert Threat Cause Actor Name + - Username + - Security ID + CommandLine: Command + Protocol: + - IPProtocol + - protocol + Application: + - Application + - application + sourceNetwork: sourceNetwork + SourceHostName: + - HostCount-source + - identityHostName + - sourceAssetName + - HostCount-src + DestinationHostname: + - HostCount-destination + - Recipient Host + - DestinationHostName + - HostCount-dst + src-packets: + - PacketRatio-src + - src-packets + dst-packets: + - PacketRatio-dst + - dst-packets + src-bytes: src-bytes + dst-bytes: dst-bytes + ExternalSeverity: + - External Severity + - Observeit Severity + SourceMAC: + - SourceMAC + - MAC + - sourceMAC + DestinationMAC: DestinationMAC + SourceOS: + - SourceOS + - OS + DestinationOS: DestinationOS + TargetUserName: DestinationUserName + SourceUserName: SourceUserName + url_category: XForceCategoryByURL + EventSeverity: EventSeverity + Source: + - Source + - source + duration: duration + ThreatName: + - Threat Name + - Alert Blocked Threat Category + AnalyzerName: Analyzer Name + Classification: Classification + ResultCode: Alert Reason Code + Technique: Technique + Action: Action + Workstation: Machine Identifier + GroupMembership: Role Name + FileName: + - Filename + - File Name + - Encoded Filename + RegistryKey: + - Registry Key + - Target Object + RegistryValue: RegistryValue + ProcessPath: Process Path + hasIdentity: hasIdentity diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/dns.yml b/uncoder-core/app/translator/mappings/platforms/qradar/dns.yml new file mode 100644 index 00000000..d9aad78e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/dns.yml @@ -0,0 +1,16 @@ +platform: Qradar +source: dns + + +log_source: + devicetype: [185, 384] + +default_log_source: + devicetype: 185 + +field_mapping: + dns-query: URL + parent-domain: parent-domain + dns-answer: dns-answer + dns-record: URL + dns-record-type: DNSRecordType \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/firewall.yml b/uncoder-core/app/translator/mappings/platforms/qradar/firewall.yml new file mode 100644 index 00000000..e1313d6d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/firewall.yml @@ -0,0 +1,36 @@ +platform: Qradar +source: firewall + + +log_source: + devicetype: [4, 5, 6, 41, 49, 73, 82, 104, 120, 194, 206, 278, 296, 307, 331, 340, 455, 456] + +default_log_source: + devicetype: 4 + +field_mapping: + src-ip: + - sourceip + - sourceIP + - SourceIP + - SrcHost + - LocalHost + - Source + - NetworkView + - HostName + src-port: + - sourceport + - SrcPort + - LocalPort + dst-ip: + - destinationip + - DstHost + - RemoteHost + - Destination + dst-port: + - destinationport + - DstPort + - RemotePort + Protocol: IPProtocol + application: Application + Application: Application \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/gcp_gcp.audit.yml b/uncoder-core/app/translator/mappings/platforms/qradar/gcp_gcp.audit.yml new file mode 100644 index 00000000..cc1fc2c6 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/gcp_gcp.audit.yml @@ -0,0 +1,21 @@ +platform: Qradar +source: gcp_gcp.audit + + +log_source: + devicetype: [449] + +default_log_source: + devicetype: 449 + +field_mapping: + backupConfiguration.enabled: backupConfiguration.enabled + gcp.audit.method_name: gcp.audit.method_name + jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction + jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue + message: message + methodName: methodName + protoPayload.methodName: protoPayload.methodName + protoPayload.request.canIpForward: protoPayload.request.canIpForward + protoPayload.serviceName: protoPayload.serviceName + serviceName: serviceName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/linux_auditd.yml b/uncoder-core/app/translator/mappings/platforms/qradar/linux_auditd.yml new file mode 100644 index 00000000..db296a32 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/linux_auditd.yml @@ -0,0 +1,23 @@ +platform: Qradar +source: linux_auditd + + +log_source: + devicetype: [11] + +default_log_source: + devicetype: 11 + +field_mapping: + a0: Command + a1: Command + a2: Command + a3: Command + exe: Process Path + CommandLine: + - Process CommandLine + - Command + Image: Process Path + User: username + LogonId: Logon ID + ParentImage: Parent Process Path diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/qradar/linux_dns_query.yml new file mode 100644 index 00000000..e8805206 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/linux_dns_query.yml @@ -0,0 +1,17 @@ +platform: Qradar +source: linux_dns_query + + +log_source: + devicetype: [11] + category: [18081] + +default_log_source: + devicetype: 11 + category: 18081 + +field_mapping: + Image: Process Path + User: username + QueryName: URL + QueryResults: URL \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/qradar/linux_file_event.yml new file mode 100644 index 00000000..6ef2dcff --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/linux_file_event.yml @@ -0,0 +1,15 @@ +platform: Qradar +source: linux_file_event + + +log_source: + devicetype: [11] + category: [8028] + +default_log_source: + devicetype: 11 + category: 8028 + +field_mapping: + Image: Process Path + TargetFilename: Filename \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/qradar/linux_network_connection.yml new file mode 100644 index 00000000..7b1725ea --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/linux_network_connection.yml @@ -0,0 +1,21 @@ +platform: Qradar +source: linux_network_connection + + +log_source: + devicetype: [11] + category: [4012] + +default_log_source: + devicetype: 11 + category: 4012 + +field_mapping: + CommandLine: Command + Image: Process Path + DestinationHostname: Destination Hostname + DestinationIp: destinationip + DestinationPort: destinationport + SourceIp: sourceip + SourcePort: sourceport + ParentImage: Parent Process Path \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/qradar/linux_process_creation.yml new file mode 100644 index 00000000..67e3db21 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/linux_process_creation.yml @@ -0,0 +1,23 @@ +platform: Qradar +source: linux_process_creation + + +log_source: + devicetype: [11] + category: [8110] + +default_log_source: + devicetype: 11 + category: 8110 + +field_mapping: + CommandLine: + - Command + - ASACommand + - Command Arguments + Image: Process Path + ParentCommandLine: Parent Command + ParentImage: Parent Process Path + User: username + LogonId: Logon ID + EventID: ASASyslogCode \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/qradar/macos_dns_query.yml new file mode 100644 index 00000000..037bfc5a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/macos_dns_query.yml @@ -0,0 +1,17 @@ +platform: Qradar +source: macos_dns_query + + +log_source: + devicetype: [102] + category: [18081] + +default_log_source: + devicetype: 102 + category: 18081 + +field_mapping: + Image: Process Path + User: username + QueryName: URL + QueryResults: URL \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/macos_file_event.yml b/uncoder-core/app/translator/mappings/platforms/qradar/macos_file_event.yml new file mode 100644 index 00000000..62cca7e8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/macos_file_event.yml @@ -0,0 +1,15 @@ +platform: Qradar +source: macos_file_event + + +log_source: + devicetype: [102] + category: [8028] + +default_log_source: + devicetype: 102 + category: 8028 + +field_mapping: + Image: Process Path + TargetFilename: Filename \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/qradar/macos_network_connection.yml new file mode 100644 index 00000000..5fb908cd --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/macos_network_connection.yml @@ -0,0 +1,21 @@ +platform: Qradar +source: macos_network_connection + + +log_source: + devicetype: [102] + category: [4012] + +default_log_source: + devicetype: 102 + category: 4012 + +field_mapping: + CommandLine: Command + Image: Process Path + DestinationHostname: Destination Hostname + DestinationIp: destinationip + DestinationPort: destinationport + SourceIp: sourceip + SourcePort: sourceport + ParentImage: Parent Process Path \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/qradar/macos_process_creation.yml new file mode 100644 index 00000000..1e69aab2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/macos_process_creation.yml @@ -0,0 +1,17 @@ +platform: Qradar +source: macos_process_creation + + +log_source: + devicetype: [102] + category: [8110] + +default_log_source: + devicetype: 102 + category: 8110 + +field_mapping: + CommandLine: Command + Image: Process Path + ParentCommandLine: Parent Command + ParentImage: Parent Process Path \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/qradar/okta_okta.yml new file mode 100644 index 00000000..a6f4522c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/okta_okta.yml @@ -0,0 +1,33 @@ +platform: Qradar +source: okta_okta + + +log_source: + devicetype: [382] + +default_log_source: + devicetype: 382 + +field_mapping: + client.user.id: client.user.id + source.user.id: source.user.id + User: User + alternateId: alternateId + client.user.full_name: client.user.full_name + source.user.full_name: source.user.full_name + related.user: related.user + client.ip: client.ip + source.ip: source.ip + user_agent.original: user_agent.original + userAgent.os: userAgent.os + userAgent.browser: userAgent.browser + client.zone: client.zone + client.device: client.device + client.id: client.id + event.action: event.action + outcome.reason: outcome.reason + event.outcome: event.outcome + client.as.number: client.as.number + client.as.organization.name: client.as.organization.name + client.domain: client.domain + source.domain: source.domain \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/proxy.yml b/uncoder-core/app/translator/mappings/platforms/qradar/proxy.yml new file mode 100644 index 00000000..b179f8c0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/proxy.yml @@ -0,0 +1,45 @@ +platform: Qradar +source: proxy + + +log_source: + devicetype: [46, 103, 154, 191, 206, 200] + +default_log_source: + devicetype: 46 + +field_mapping: + c-uri: + - URL + - XForceCategoryByURL + c-useragent: User Agent + cs-method: + - HTTP Method + - Method + cs-bytes: Bytes Sent + #cs-cookie-vars: cs-cookie-vars + c-uri-extension: URL + c-uri-query: + - URL + - URL Path + - URL Query String + #cs-cookie: cs-cookie + cs-host: + - UrlHost + - URL Host + - URL Domain + - HTTP Host + cs-referrer: + - URL Referrer + - Referrer URL + cs-version: HTTP Version + r-dns: + - UrlHost + - URL Host + sc-status: + - HTTP Response Code + - Response Code + #post-body: post-body + url_category: + - XForceCategoryByURL + - Web Category \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/webserver.yml b/uncoder-core/app/translator/mappings/platforms/qradar/webserver.yml new file mode 100644 index 00000000..b43fbc8d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/webserver.yml @@ -0,0 +1,42 @@ +platform: Qradar +source: webserver + + +log_source: + devicetype: [10, 13, 239, 439] + +default_log_source: + devicetype: 10 + +field_mapping: + c-uri: + - URL + - XForceCategoryByURL + c-useragent: User Agent + cs-method: + - HTTP Method + - Method + cs-bytes: Bytes Sent + #cs-cookie-vars: cs-cookie-vars + c-uri-extension: URL + c-uri-query: + - URL + - URL Path + - URL Query String + #cs-cookie: cs-cookie + cs-host: + - UrlHost + - URL Host + - URL Domain + - HTTP Host + cs-referrer: + - URL Referrer + - Referrer URL + cs-version: HTTP Version + r-dns: + - UrlHost + - URL Host + sc-status: + - HTTP Response Code + - Response Code + #post-body: post-body \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_application.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_application.yml new file mode 100644 index 00000000..859a307f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_application.yml @@ -0,0 +1,28 @@ +platform: Qradar +source: windows_application + + +log_source: + devicetype: [12] + qideventcategory: [Application_Information, Application_Warning, Application_Error] + +default_log_source: + devicetype: 12 + qideventcategory: Application_Information + +field_mapping: + EventID: Event ID + Provider_Name: Provider_Name + Level: Level + src_ip: sourceip + source: source + additional_information: additional_information + EventData: EventData + Channel: Channel + statement: statement + Faulting application path: Faulting application path + object_name: Object Name + class_type: class_type + action_id: action_id + Data: Data + Message: Message \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_create_remote_thread.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_create_remote_thread.yml new file mode 100644 index 00000000..2682258b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_create_remote_thread.yml @@ -0,0 +1,22 @@ +platform: Qradar +source: windows_create_remote_thread + + +log_source: + devicetype: [12] + qid: [5001845] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + qid: 5001845 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + SourceImage: Source Process Path + TargetImage: Target Process Path + StartModule: Start Module + StartAddress: Start Address + StartFunction: Start Function + Image: Process Path + Hashes: File Hash diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_create_stream_hash.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_create_stream_hash.yml new file mode 100644 index 00000000..1490d023 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_create_stream_hash.yml @@ -0,0 +1,17 @@ +platform: Qradar +source: windows_create_stream_hash + + +log_source: + devicetype: [12] + qid: [5001834] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + qid: 5001834 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + Image: Process Path + Hashes: File Hash \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_dns_query.yml new file mode 100644 index 00000000..fa3ea5bd --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_dns_query.yml @@ -0,0 +1,19 @@ +platform: Qradar +source: windows_dns_query + + +log_source: + devicetype: [12] + category: [18081] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + category: 18081 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + Image: Process Path + User: username + QueryName: URL + QueryResults: URL \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_driver_load.yml new file mode 100644 index 00000000..ca4893ed --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_driver_load.yml @@ -0,0 +1,18 @@ +platform: Qradar +source: windows_driver_load + + +log_source: + devicetype: [12] + qid: [5001843] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + qid: 5001843 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + ImageLoaded: Process Path + ImagePath: ImagePath + Hashes: File Hash \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_file_event.yml new file mode 100644 index 00000000..10bd97f0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_file_event.yml @@ -0,0 +1,20 @@ +platform: Qradar +source: windows_file_event + + +log_source: + devicetype: [12] + category: [8028] + +default_log_source: + devicetype: 12 + category: 8028 + +field_mapping: + CreationUtcTime: endtime + FileName: Filename + Image: Process Path + OriginalFilename: OriginalFilename + PreviousCreationUtcTime: PreviousCreationUtcTime + TargetFilename: Filename + User: username \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_image_load.yml new file mode 100644 index 00000000..79d3bd66 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_image_load.yml @@ -0,0 +1,25 @@ +platform: Qradar +source: windows_image_load + + +log_source: + devicetype: [12] + category: [19247] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + category: 19247 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + Image: Process Path + ImageLoaded: + - Process Path + - LoadedImage + SignatureStatus: + - Signature Status + - SignatureStatus + OriginalFileName: OriginalFileName + Signed: Signed + FileExtension: File Extension \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_ldap_debug.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_ldap_debug.yml new file mode 100644 index 00000000..9f99a5ae --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_ldap_debug.yml @@ -0,0 +1,15 @@ +platform: Qradar +source: windows_ldap_debug + + +log_source: + devicetype: [12] + qideventcategory: [Microsoft-Windows-LDAP-Client/Debug] + +default_log_source: + devicetype: 12 + qideventcategory: Microsoft-Windows-LDAP-Client/Debug + +field_mapping: + EventID: Event ID + SearchFilter: SearchFilter \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_network_connection.yml new file mode 100644 index 00000000..b65b7571 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_network_connection.yml @@ -0,0 +1,22 @@ +platform: Qradar +source: windows_network_connection + + +log_source: + devicetype: [12] + category: [4012] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + category: 4012 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + Image: Process Path + DestinationHostname: Destination Hostname + DestinationIp: destinationip + DestinationPort: destinationport + SourceIp: sourceip + SourcePort: sourceport + Initiated: Initiated \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_ntlm.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_ntlm.yml new file mode 100644 index 00000000..0da6e26f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_ntlm.yml @@ -0,0 +1,17 @@ +platform: Qradar +source: windows_ntlm + + +log_source: + devicetype: [12] + qideventcategory: [Microsoft-Windows-NTLM/Operational] + +default_log_source: + devicetype: 12 + qideventcategory: Microsoft-Windows-NTLM/Operational + +field_mapping: + WorkstationName: WorkstationName + ProcessName: ProcessName + EventID: Event ID + TargetName: TargetName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_pipe_created.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_pipe_created.yml new file mode 100644 index 00000000..d2fae58a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_pipe_created.yml @@ -0,0 +1,15 @@ +platform: Qradar +source: windows_pipe_created + + +log_source: + devicetype: [12] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + PipeName: Pipe Name + Image: Process Path \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_powershell.yml new file mode 100644 index 00000000..c2f136d9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_powershell.yml @@ -0,0 +1,22 @@ +platform: Qradar +source: windows_powershell + + +log_source: + devicetype: [12] + qideventcategory: [Microsoft-Windows-PowerShell/Operational] + +default_log_source: + devicetype: 12 + qideventcategory: Microsoft-Windows-PowerShell/Operational + +field_mapping: + EventID: Event ID + CommandLine: Command + ScriptBlockText: ScriptBlockText + Payload: Payload + HostApplication: HostApplication + ContextInfo: ContextInfo + HostName: HostName + EngineVersion: EngineVersion + HostVersion: HostVersion \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_process_access.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_process_access.yml new file mode 100644 index 00000000..3b3bc478 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_process_access.yml @@ -0,0 +1,19 @@ +platform: Qradar +source: windows_process_access + + +log_source: + devicetype: [12] + qid: [5001829] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + qid: 5001829 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + TargetImage: Target Process Path + SourceImage: Source Process Path + CallTrace: CallTrace + GrantedAccess: GrantedAccess \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_process_creation.yml new file mode 100644 index 00000000..fcad6da1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_process_creation.yml @@ -0,0 +1,34 @@ +platform: Qradar +source: windows_process_creation + + +log_source: + devicetype: [12] + category: [8110] + +default_log_source: + devicetype: 12 + category: 8110 + +field_mapping: + CommandLine: + - Command + - Encoded Argument + - Command Arguments + CurrentDirectory: CurrentDirectory + Hashes: File Hash + Image: + - Process Path + - Process Name + - DGApplication + - ProcessName + IntegrityLevel: IntegrityLevel + ParentCommandLine: Parent Command + ParentImage: + - Parent Process Path + - ParentProcessName + ParentUser: ParentUser + Product: Product + User: + - username + - userName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_process_termination.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_process_termination.yml new file mode 100644 index 00000000..0109186c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_process_termination.yml @@ -0,0 +1,18 @@ +platform: Qradar +source: windows_process_termination + + +log_source: + devicetype: [12] + category: [8113] + +default_log_source: + devicetype: 12 + category: 8113 + +field_mapping: + Image: + - Process Path + - Terminated Process Name + ProcessId: ProcessId +# ProcessGuid: ProcessGuid \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_raw_access_thread.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_raw_access_thread.yml new file mode 100644 index 00000000..91eb2d9e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_raw_access_thread.yml @@ -0,0 +1,15 @@ +platform: Qradar +source: windows_raw_access_thread + + +log_source: + devicetype: [12] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + Device: Device + Image: Process Path \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_registry_event.yml new file mode 100644 index 00000000..01430904 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_registry_event.yml @@ -0,0 +1,20 @@ +platform: Qradar +source: windows_registry_event + + +log_source: + devicetype: [12] + category: [8012] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + category: 8012 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + TargetObject: Target Object + Image: Process Path + Details: Target Details + EventType: EventType + User: username \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_security.yml new file mode 100644 index 00000000..bb7ccef6 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_security.yml @@ -0,0 +1,193 @@ +platform: Qradar +source: windows_security + + +log_source: + devicetype: [12] + +default_log_source: + devicetype: 12 + +field_mapping: + EventID: + - Event ID + - EventID + - qidEventId + ParentImage: Parent Process Path + AccessMask: AccessMask + AccountName: Account Name + AllowedToDelegateTo: AllowedToDelegateTo + AttributeLDAPDisplayName: AttributeLDAPDisplayName + AuditPolicyChanges: AuditPolicyChanges + AuthenticationPackageName: AuthenticationPackageName + CallingProcessName: CallingProcessName + Channel: Channel + ComputerName: + - Machine Identifier + - Hostname + - identityNetBiosName + EventType: EventType + FailureReason: FailureReason + FileName: Filename + GrantedAccess: GrantedAccess + Hashes: File Hash + HiveName: HiveName + IpAddress: + - sourceIP + - SourceIP + - sourceip + - identityIP + IpPort: sourceport + KeyLength: KeyLength + LogonProcessName: LogonProcessName + LogonType: + - Logon Type + - Login Type + - MSLogonType + LinkName: LinkName + MemberName: MemberName + MemberSid: MemberSid + NewProcessName: + - Process Name + - New Process Name + ObjectClass: ObjectClass + ObjectName: + - Object Name + - objectname + - MSFileObjectName + - ObjectName_Filename + - ObjectName + ObjectType: + - Object Type + - ObjectType + ObjectValueName: ObjectValueName + Path: Path + CommandLine: + - Command + - Process Command Line + OldUacValue: OldUacValue + SubStatus: SubStatus + DisplayName: DisplayName + TaskContent: TaskContent + ServiceSid: ServiceSid + CertThumbprint: CertThumbprint + ClassName: ClassName + NotificationPackageName: NotificationPackageName + NewSd: NewSd + TestSigning: TestSigning + TargetInfo: TargetInfo + ClientProcessId: ClientProcessId + ParentProcessId: ParentProcessId + AccessList: AccessList + GroupMembership: + - GroupMembership + - GroupName + - Group Name + FilterName: FilterName + ChangeType: ChangeType + LayerName: LayerName + ServiceAccount: ServiceAccount + AttributeValue: AttributeValue + SessionName: SessionName + TaskName: + - TaskName + - Task Name + ObjectDN: ObjectDN + TemplateContent: TemplateContent + NewTemplateContent: NewTemplateContent + SourcePort: sourceport + PasswordLastSet: PasswordLastSet + PrivilegeList: PrivilegeList + DeviceDescription: DeviceDescription + TargetServerName: TargetServerName + NewTargetUserName: NewTargetUserName + OperationType: OperationType + DestPort: + - destinationport + - DstPort + ServiceStartType: ServiceStartType + OldTargetUserName: OldTargetUserName + UserPrincipalName: UserPrincipalName + Accesses: Accesses + DnsHostName: DnsHostName + DisableIntegrityChecks: DisableIntegrityChecks + AuditSourceName: AuditSourceName + Workstation: Machine Identifier + DestAddress: + - destinationip + - DestinationIP + - destinationaddress + PreAuthType: PreAuthType + SecurityPackageName: SecurityPackageName + SubjectLogonId: SubjectLogonId + NewUacValue: NewUacValue + EnabledPrivilegeList: EnabledPrivilegeList + RelativeTargetName: RelativeTargetName + CertSerialNumber: RelativeTargetName + SidHistory: SidHistory + TargetLogonId: TargetLogonId + KernelDebug: KernelDebug + CallerProcessName: CallerProcessName + ProcessName: Process Name + Properties: Properties + UserAccountControl: UserAccountControl + RegistryValue: Target Object + SecurityID: SecurityID + ServiceFileName: + - Service Filename + - ServiceFileName + - Service File Name + SecurityDescriptor: SecurityDescriptor + ServiceName: Service Name + ShareName: + - Share Name + - ShareName + NewValue: NewValue + Source: Source + Status: Status + SubjectAccountName: + - Subject Account Name + - SubjectAccountName + SubjectDomainName: SubjectDomainName + SubjectUserName: Target Username + SubjectUserSid: SubjectUserSid + SourceAddr: sourceip + SourceAddress: + - sourceip + - sourceaddress + TargetFilename: File Directory + TargetName: Target Username + ServicePrincipalNames: ServicePrincipalNames + TargetDomainName: TargetDomainName + TargetSid: TargetSid + TargetUserName: + - Target Username + - User + - userName + - Target User Name + ObjectServer: ObjectServer + TargetUserSid: TargetUserSid + TicketEncryptionType: TicketEncryptionType + TicketOptions: TicketOptions + WorkstationName: Machine Identifier + TransmittedServices: TransmittedServices + AuthenticationAlgorithm: AuthenticationAlgorithm + LayerRTID: LayerRTID + BSSID: BSSID + BSSType: BSSType + CipherAlgorithm: CipherAlgorithm + ConnectionId: ConnectionId + ConnectionMode: ConnectionMode + InterfaceDescription: InterfaceDescription + InterfaceGuid: InterfaceGuid + OnexEnabled: OnexEnabled + PHYType: PHYType + ProfileName: ProfileName + SSID: SSID + Domain: Domain + ServiceType: ServiceType + SourceName: SourceName + StartType: StartType + UserID: UserID + ParentProcessName: Parent Process Name + Service: Service \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_sysmon.yml new file mode 100644 index 00000000..a6bb7bc0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_sysmon.yml @@ -0,0 +1,63 @@ +platform: Qradar +source: windows_sysmon + + +log_source: + devicetype: [12] + qideventcategory: [Microsoft-Windows-Sysmon/Operational] + +default_log_source: + devicetype: 12 + qideventcategory: Microsoft-Windows-Sysmon/Operational + +field_mapping: + CommandLine: Command + Image: Process Path + ParentImage: Parent Process Path + EventID: Event ID + DestinationHostname: Destination Host Name + DestinationIp: destinationip + DestinationPort: destinationport + Hashes: File Hash + ParentCommandLine: Parent Command + RuleName: Rule Name + SourceHostname: Source Hostname + SourceIp: sourceip + SourcePort: sourceport + TargetFilename: Filename + User: username + SignatureStatus: Signature Status + TargetObject: Target Object + Details: Target Details + QueryName: URL + PipeName: Pipe Name + ImageLoaded: Process Path + Imphash: IMP Hash + SourceImage: Source Process Path + StartModule: Start Module + TargetImage: Target Process Path + Device: Device Name + ProcessID: Process ID + StartAddress: Start Address + StartFunction: Start Function + CallTrace: CallTrace + Company: Company + CurrentDirectory: CurrentDirectory + Description: Description + DestinationIsIpv6: DestinationIsIpv6 + DestinationPortName: DestinationPortName + Initiated: Initiated + IntegrityLevel: IntegrityLevel + Product: Product + Protocol: Protocol + SourceIsIpv6: SourceIsIpv6 + SourcePortName: SourcePortName + OriginalFileName: OriginalFileName + Signed: Signed + Signature: Signature + QueryResults: QueryResults + QueryStatus: QueryStatus + IsExecutable: IsExecutable + ImagePath: ImagePath + FileVersion: FileVersion + EventType: EventType diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_system.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_system.yml new file mode 100644 index 00000000..f18ca3e5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_system.yml @@ -0,0 +1,27 @@ +platform: Qradar +source: windows_system + + +log_source: + devicetype: [12] + qideventcategory: [System_Information, System_Warning, System_Error] + +default_log_source: + devicetype: 12 + qideventcategory: System_Information + +field_mapping: + EventID: Event ID + AccountName: AccountName + ImagePath: ImagePath + ServiceName: ServiceName + ServiceType: ServiceType + StartType: StartType + Provider_Name: Provider_Name + Origin: Origin + HiveName: HiveName + Caption: Caption + param1: param1 + param2: param2 + Channel: Channel + DeviceName: DeviceName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/qradar/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/qradar/windows_wmi_event.yml new file mode 100644 index 00000000..7790d7f9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/qradar/windows_wmi_event.yml @@ -0,0 +1,15 @@ +platform: Qradar +source: windows_wmi_event + + +log_source: + devicetype: [12] + qideventcategory: [Microsoft-Windows-WMI-Activity/Operational] + +default_log_source: + devicetype: 12 + qideventcategory: Microsoft-Windows-WMI-Activity/Operational + +field_mapping: + EventID: Event ID + Destination: Destination \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sentinel_one/default.yml b/uncoder-core/app/translator/mappings/platforms/sentinel_one/default.yml new file mode 100644 index 00000000..16f2f7e5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sentinel_one/default.yml @@ -0,0 +1,2 @@ +platform: Sentinel One Power Query +source: default diff --git a/uncoder-core/app/translator/mappings/platforms/sentinel_one/dns.yml b/uncoder-core/app/translator/mappings/platforms/sentinel_one/dns.yml new file mode 100644 index 00000000..d04d59e2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sentinel_one/dns.yml @@ -0,0 +1,12 @@ +platform: Sentinel One Power Query +source: dns + +field_mapping: + Image: src.process.image.path + CommandLine: src.process.cmdline + ParentImage: src.process.parent.image.path + ParentCommandLine: src.process.parent.cmdline + query: event.dns.request + answer: event.dns.response + QueryName: event.dns.request + record_type: event.dns.response \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sentinel_one/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/sentinel_one/linux_file_event.yml new file mode 100644 index 00000000..7fbf5c59 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sentinel_one/linux_file_event.yml @@ -0,0 +1,11 @@ +platform: Sentinel One Power Query +source: linux_file_event + +field_mapping: + Image: src.process.image.path + CommandLine: src.process.cmdline + ParentImage: src.process.parent.image.path + ParentCommandLine: src.process.parent.cmdline + TargetFilename: tgt.file.path + SourceFilename: tgt.file.oldPath + User: src.process.use \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_image_load.yml new file mode 100644 index 00000000..e6533f12 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_image_load.yml @@ -0,0 +1,9 @@ +platform: Sentinel One Power Query +source: windows_image_load + +field_mapping: + Image: Image + ImageLoaded: ImageLoaded + SignatureStatus: SignatureStatus + OriginalFileName: OriginalFileName + Signed: Signed \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_network_connection.yml new file mode 100644 index 00000000..f740d589 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_network_connection.yml @@ -0,0 +1,21 @@ +platform: Sentinel One Power Query +source: windows_network_connection + +field_mapping: + Image: src.process.image.path + CommandLine: src.process.cmdline + ParentImage: src.process.parent.image.path + ParentCommandLine: src.process.parent.cmdline + DestinationHostname: + - url.address + - event.dns.request + DestinationPort: dst.port.number + DestinationIp: dst.ip.address + User: src.process.user + SourceIp: src.ip.address + SourcePort: src.port.number + Protocol: NetProtocolName + dst_ip: dst.ip.address + src_ip: src.ip.address + dst_port: dst.port.number + src_port: src.port.number \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_pipe_created.yml b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_pipe_created.yml new file mode 100644 index 00000000..0f670481 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_pipe_created.yml @@ -0,0 +1,9 @@ +platform: Sentinel One Power Query +source: windows_pipe_created + +field_mapping: + PipeName: namedPipe.name + Image: src.process.image.path + CommandLine: src.process.cmdline + ParentImage: src.process.parent.image.path + ParentCommandLine: src.process.parent.cmdline \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_process_creation.yml new file mode 100644 index 00000000..f736fa46 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_process_creation.yml @@ -0,0 +1,21 @@ +platform: Sentinel One Power Query +source: windows_process_creation + +field_mapping: + ProcessId: tgt.process.pid + Image: tgt.process.image.path + Description: tgt.process.displayName + Publisher: tgt.process.publisher + Product: tgt.process.displayName + Company: tgt.process.publisher + CommandLine: tgt.process.cmdline + CurrentDirectory: tgt.process.image.path + User: tgt.process.user + TerminalSessionId: tgt.process.sessionid + IntegrityLevel: tgt.process.integrityLevel + md5: tgt.process.image.md5 + sha1: tgt.process.image.sha1 + sha256: tgt.process.image.sha256 + ParentProcessId: src.process.pid + ParentImage: src.process.image.path + ParentCommandLine: src.process.cmdline \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_registry_event.yml new file mode 100644 index 00000000..72f8db79 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sentinel_one/windows_registry_event.yml @@ -0,0 +1,10 @@ +platform: Sentinel One Power Query +source: windows_registry_event + +field_mapping: + Image: src.process.image.path + CommandLine: src.process.cmdline + ParentImage: src.process.parent.image.path + ParentCommandLine: src.process.parent.cmdline + TargetObject: registry.keyPath + Details: registry.value \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/apache_httpd.yml b/uncoder-core/app/translator/mappings/platforms/sigma/apache_httpd.yml new file mode 100644 index 00000000..9299fad4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/apache_httpd.yml @@ -0,0 +1,19 @@ +platform: Sigma +source: apache_httpd + +log_source: + product: [apache] + service: [httpd] + +default_log_source: + product: apache + service: httpd + +field_mapping: + c-uri: c-uri + c-useragent: c-useragent + cs-method: cs-method + cs-bytes: cs-bytes + c-uri-query: c-uri-query + cs-referrer: cs-referrer + sc-status: sc-status \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/apache_tomcat.yml b/uncoder-core/app/translator/mappings/platforms/sigma/apache_tomcat.yml new file mode 100644 index 00000000..d9dd8583 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/apache_tomcat.yml @@ -0,0 +1,19 @@ +platform: Sigma +source: apache_tomcat + +log_source: + product: [apache] + service: [tomcat] + +default_log_source: + product: apache + service: tomcat + +field_mapping: + c-uri: c-uri + c-useragent: c-useragent + cs-method: cs-method + cs-bytes: cs-bytes + c-uri-query: c-uri-query + cs-referrer: cs-referrer + sc-status: sc-status \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/sigma/aws_cloudtrail.yml new file mode 100644 index 00000000..7716ea12 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/aws_cloudtrail.yml @@ -0,0 +1,36 @@ +platform: Sigma +source: aws_cloudtrail + + +log_source: + product: [aws] + service: [cloudtrail] + +default_log_source: + product: aws + service: cloudtrail + +field_mapping: + eventSource: eventSource + eventName: eventName + AdditionalEventData.MFAUsed: additionalEventData.MFAUsed + errorCode: errorCode + errorMessage: errorMessage + eventType: eventType + requestParameters: requestParameters + requestParameters.attribute: requestParameters.attribute + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP + requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort + requestParameters.userData: requestParameters.userData + responseElements: responseElements + responseElements.ConsoleLogin: responseElements.ConsoleLogin + responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword + responseElements.publiclyAccessible: responseElements.publiclyAccessible + status: status + terminatingRuleId: terminatingRuleId + userAgent: userAgent + userIdentity.arn: userIdentity.arn + userIdentity.principalId: userIdentity.principalId + userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type + userIdentity.type: userIdentity.type + userIdentity.userName: userIdentity.userName diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/sigma/aws_eks.yml new file mode 100644 index 00000000..70412cdd --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/aws_eks.yml @@ -0,0 +1,27 @@ +platform: Sigma +source: aws_eks + + +log_source: + product: [aws] + service: [eks] + +default_log_source: + product: aws + service: eks + +field_mapping: + annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy + aws_node_type: aws_node_type + objectRef.namespace: objectRef.namespace + objectRef.resource: objectRef.resource + objectRef.subresource: objectRef.subresource + requestObject.rules.resources: requestObject.rules.resources + requestObject.rules.verbs: requestObject.rules.verbs + requestObject.spec.containers.image: requestObject.spec.containers.image + requestURI: requestURI + stage: stage + user.groups: user.groups + user.username: user.username + verb: verb diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/azure_AzureDiagnostics.yml b/uncoder-core/app/translator/mappings/platforms/sigma/azure_AzureDiagnostics.yml new file mode 100644 index 00000000..68710126 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/azure_AzureDiagnostics.yml @@ -0,0 +1,15 @@ +platform: Sigma +source: azure_AzureDiagnostics + + +log_source: + product: [azure] + service: [AzureDiagnostics] + +default_log_source: + product: azure + service: AzureDiagnostics + +field_mapping: + ResultDescription: ResultDescription + Category: Category diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/azure_BehaviorAnalytics.yml b/uncoder-core/app/translator/mappings/platforms/sigma/azure_BehaviorAnalytics.yml new file mode 100644 index 00000000..ae8e3f56 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/azure_BehaviorAnalytics.yml @@ -0,0 +1,22 @@ +platform: Sigma +source: azure_BehaviorAnalytics + + +log_source: + product: [azure] + service: [BehaviorAnalytics] + +default_log_source: + product: azure + service: BehaviorAnalytics + +field_mapping: + ActionType: ActionType + ActivityInsights: ActivityInsights + ActivityType: ActivityType + EventSource: EventSource + DevicesInsights: DevicesInsights + RiskDetail: RiskDetail + UsersInsights: UsersInsights + UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount + UsersInsights.IsNewAccount: UsersInsights.IsNewAccount diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/azure_aadnoninteractiveusersigninlogs.yml b/uncoder-core/app/translator/mappings/platforms/sigma/azure_aadnoninteractiveusersigninlogs.yml new file mode 100644 index 00000000..a0c4ba9d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/azure_aadnoninteractiveusersigninlogs.yml @@ -0,0 +1,15 @@ +platform: Sigma +source: azure_aadnoninteractiveusersigninlogs + + +log_source: + product: [azure] + service: [aadnoninteractiveusersigninlogs] + +default_log_source: + product: azure + service: aadnoninteractiveusersigninlogs + +field_mapping: + UserAgent: UserAgent + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/sigma/azure_azureactivity.yml new file mode 100644 index 00000000..7a17a916 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/azure_azureactivity.yml @@ -0,0 +1,24 @@ +platform: Sigma +source: azure_azureactivity + + +log_source: + product: [azure] + service: [azureactivity, activitylogs] + +default_log_source: + product: azure + service: azureactivity + +field_mapping: + ActivityStatus: ActivityStatus + ActivityStatusValue: ActivityStatusValue + ActivitySubstatusValue: ActivitySubstatusValue + Authorization: Authorization + Category: Category + CategoryValue: CategoryValue + OperationName: OperationName + OperationNameValue: OperationNameValue + ResourceId: ResourceId + ResourceProviderValue: ResourceProviderValue + Type: Type diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/sigma/azure_azuread.yml new file mode 100644 index 00000000..d46b9688 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/azure_azuread.yml @@ -0,0 +1,25 @@ +platform: Sigma +source: azure_azuread + + +log_source: + product: [azure] + service: [azuread, auditlogs] + +default_log_source: + product: azure + service: azuread + +field_mapping: + ActivityDisplayName: ActivityDisplayName + Category: Category + LoggedByService: LoggedByService + Result: Result + OperationName: OperationName + TargetResources: TargetResources + AADOperationType: AADOperationType + InitiatedBy: InitiatedBy + ResultReason: ResultReason + Status: Status + Status.errorCode: Status.errorCode + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/azure_m365.yml b/uncoder-core/app/translator/mappings/platforms/sigma/azure_m365.yml new file mode 100644 index 00000000..b9877a5b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/azure_m365.yml @@ -0,0 +1,25 @@ +platform: Sigma +source: azure_m365 + + +log_source: + product: [azure] + service: [m365, o365, office365] + +default_log_source: + product: azure + service: m365 + +field_mapping: + ClientInfoString: ClientInfoString + LogonError: LogonError + ModifiedProperties: ModifiedProperties + OfficeObjectId: OfficeObjectId + OfficeWorkload: OfficeWorkload + Operation: Operation + Parameters: Parameters + RecordType: RecordType + ResultStatus: ResultStatus + SourceFileExtension: SourceFileExtension + SourceFileName: SourceFileName + UserAgent: UserAgent diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/azure_signinlogs.yml b/uncoder-core/app/translator/mappings/platforms/sigma/azure_signinlogs.yml new file mode 100644 index 00000000..de0ab2a0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/azure_signinlogs.yml @@ -0,0 +1,31 @@ +platform: Sigma +source: azure_signinlogs + + +log_source: + product: [azure] + service: [signinlogs] + +default_log_source: + product: azure + service: signinlogs + +field_mapping: + AppDisplayName: AppDisplayName + AppId: AppId + AuthenticationRequirement: AuthenticationRequirement + Category: Category + ConditionalAccessStatus: ConditionalAccessStatus + DeviceDetail: DeviceDetail + IsInteractive: IsInteractive + NetworkLocationDetails: NetworkLocationDetails + ResourceDisplayName: ResourceDisplayName + ResourceIdentity: ResourceIdentity + ResultDescription: ResultDescription + ResultType: ResultType + Status.errorCode: Status.errorCode + Status: Status + Status.failureReason: Status.failureReason + TokenIssuerType: TokenIssuerType + UserAgent: UserAgent + UserPrincipalName: UserPrincipalName diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/default.yml b/uncoder-core/app/translator/mappings/platforms/sigma/default.yml new file mode 100644 index 00000000..f29d3617 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/default.yml @@ -0,0 +1,9 @@ +platform: Sigma +source: default + + +log_source: + product: windows + +default_log_source: + product: windows \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/dns.yml b/uncoder-core/app/translator/mappings/platforms/sigma/dns.yml new file mode 100644 index 00000000..dc36d5fd --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/dns.yml @@ -0,0 +1,15 @@ +platform: Sigma +source: dns + + +log_source: + category: [dns] + +default_log_source: + category: dns + +field_mapping: + dns-query: dns-query + parent-domain: parent-domain + dns-answer: dns-answer + dns-record: dns-record \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/firewall.yml b/uncoder-core/app/translator/mappings/platforms/sigma/firewall.yml new file mode 100644 index 00000000..87ab0cdb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/firewall.yml @@ -0,0 +1,15 @@ +platform: Sigma +source: firewall + + +log_source: + category: [firewall] + +default_log_source: + category: firewall + +field_mapping: + src-ip: src-ip + src-port: src-port + dst-ip: dst-ip + dst-port: dst-port \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/gcp_gcp.audit.yml b/uncoder-core/app/translator/mappings/platforms/sigma/gcp_gcp.audit.yml new file mode 100644 index 00000000..ef94b58d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/gcp_gcp.audit.yml @@ -0,0 +1,23 @@ +platform: Sigma +source: gcp_gcp.audit + + +log_source: + product: [gcp] + service: [gcp.audit] + +default_log_source: + product: gcp + service: gcp.audit + +field_mapping: + backupConfiguration.enabled: backupConfiguration.enabled + gcp.audit.method_name: gcp.audit.method_name + jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction + jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue + message: message + methodName: methodName + protoPayload.methodName: protoPayload.methodName + protoPayload.request.canIpForward: protoPayload.request.canIpForward + protoPayload.serviceName: protoPayload.serviceName + serviceName: serviceName diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/gcp_pubsub.yml b/uncoder-core/app/translator/mappings/platforms/sigma/gcp_pubsub.yml new file mode 100644 index 00000000..4756fa23 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/gcp_pubsub.yml @@ -0,0 +1,18 @@ +platform: Sigma +source: gcp_pubsub + + +log_source: + product: [gcp] + service: [pubsub] + +default_log_source: + product: gcp + service: pubsub + +field_mapping: + data.protoPayload.methodName: data.protoPayload.methodName + data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member + data.resource.type: data.resource.type diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/linux_auditd.yml b/uncoder-core/app/translator/mappings/platforms/sigma/linux_auditd.yml new file mode 100644 index 00000000..bd9f281d --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/linux_auditd.yml @@ -0,0 +1,18 @@ +platform: Sigma +source: linux_auditd + + +log_source: + product: [linux] + service: [auditd] + +default_log_source: + product: linux + service: auditd + +field_mapping: + a0: a0 + a1: a1 + a2: a2 + a3: a3 + exe: exe \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/sigma/linux_dns_query.yml new file mode 100644 index 00000000..41cc965a --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/linux_dns_query.yml @@ -0,0 +1,17 @@ +platform: Sigma +source: linux_dns_query + + +log_source: + product: [linux] + category: [dns_query] + +default_log_source: + product: linux + category: dns_query + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/sigma/linux_file_event.yml new file mode 100644 index 00000000..61231273 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/linux_file_event.yml @@ -0,0 +1,14 @@ +platform: Sigma +source: linux_file_event + +log_source: + product: [linux] + category: [file_event, file_access, file_create, file_delete, file_rename, file_change] + +default_log_source: + product: linux + category: file_event + +field_mapping: + Image: Image + TargetFilename: TargetFilename \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/sigma/linux_network_connection.yml new file mode 100644 index 00000000..cc51ccf2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/linux_network_connection.yml @@ -0,0 +1,20 @@ +platform: Sigma +source: linux_network_connection + + +log_source: + product: [linux] + category: [network_connection] + +default_log_source: + product: linux + category: network_connection + +field_mapping: + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + ParentImage: ParentImage \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/sigma/linux_process_creation.yml new file mode 100644 index 00000000..8d0df060 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/linux_process_creation.yml @@ -0,0 +1,17 @@ +platform: Sigma +source: linux_process_creation + + +log_source: + product: [linux] + category: [process_creation] + +default_log_source: + product: linux + category: process_creation + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/sigma/macos_dns_query.yml new file mode 100644 index 00000000..09b750d0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/macos_dns_query.yml @@ -0,0 +1,17 @@ +platform: Sigma +source: macos_dns_query + + +log_source: + product: [macos] + category: [dns_query] + +default_log_source: + product: macos + category: dns_query + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/sigma/macos_network_connection.yml new file mode 100644 index 00000000..b0d3ecf1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/macos_network_connection.yml @@ -0,0 +1,20 @@ +platform: Sigma +source: macos_network_connection + + +log_source: + product: [macos] + category: [network_connection] + +default_log_source: + product: macos + category: network_connection + +field_mapping: + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + ParentImage: ParentImage \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/sigma/macos_process_creation.yml new file mode 100644 index 00000000..f4caa0c9 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/macos_process_creation.yml @@ -0,0 +1,17 @@ +platform: Sigma +source: macos_process_creation + + +log_source: + product: [macos] + category: [process_creation] + +default_log_source: + product: macos + category: process_creation + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/nginx_nginx.yml b/uncoder-core/app/translator/mappings/platforms/sigma/nginx_nginx.yml new file mode 100644 index 00000000..5cca1b73 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/nginx_nginx.yml @@ -0,0 +1,19 @@ +platform: Sigma +source: nginx_nginx + +log_source: + product: [nginx] + service: [nginx] + +default_log_source: + product: nginx + service: nginx + +field_mapping: + c-uri: c-uri + c-useragent: c-useragent + cs-method: cs-method + cs-bytes: cs-bytes + c-uri-query: c-uri-query + cs-referrer: cs-referrer + sc-status: sc-status \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/sigma/okta_okta.yml new file mode 100644 index 00000000..c4bf3ee8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/okta_okta.yml @@ -0,0 +1,35 @@ +platform: Sigma +source: okta_okta + + +log_source: + product: [okta] + service: [okta] + +default_log_source: + product: okta + service: okta + +field_mapping: + client.user.id: client.user.id + source.user.id: source.user.id + User: User + alternateId: alternateId + client.user.full_name: client.user.full_name + source.user.full_name: source.user.full_name + related.user: related.user + client.ip: client.ip + source.ip: source.ip + user_agent.original: user_agent.original + userAgent.os: userAgent.os + userAgent.browser: userAgent.browser + client.zone: client.zone + client.device: client.device + client.id: client.id + event.action: event.action + outcome.reason: outcome.reason + event.outcome: event.outcome + client.as.number: client.as.number + client.as.organization.name: client.as.organization.name + client.domain: client.domain + source.domain: source.domain diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/proxy.yml b/uncoder-core/app/translator/mappings/platforms/sigma/proxy.yml new file mode 100644 index 00000000..152038e6 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/proxy.yml @@ -0,0 +1,25 @@ +platform: Sigma +source: proxy + + +log_source: + category: [proxy] + +default_log_source: + category: proxy + +field_mapping: + c-uri: c-uri + c-useragent: c-useragent + cs-method: cs-method + cs-bytes: cs-bytes + cs-cookie-vars: cs-cookie-vars + c-uri-extension: c-uri-extension + c-uri-query: c-uri-query + cs-cookie: cs-cookie + cs-host: cs-host + cs-referrer: cs-referrer + cs-version: cs-version + r-dns: r-dns + sc-status: sc-status + post-body: post-body \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/slack_slack.yml b/uncoder-core/app/translator/mappings/platforms/sigma/slack_slack.yml new file mode 100644 index 00000000..d06df1ed --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/slack_slack.yml @@ -0,0 +1,13 @@ +platform: Sigma +source: slack_slack + +log_source: + product: [slack] + service: [slack] + +default_log_source: + product: slack + service: slack + +field_mapping: + c-action: c-action \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/webserver.yml b/uncoder-core/app/translator/mappings/platforms/sigma/webserver.yml new file mode 100644 index 00000000..e58a99f4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/webserver.yml @@ -0,0 +1,25 @@ +platform: Sigma +source: webserver + + +log_source: + category: [webserver] + +default_log_source: + category: webserver + +field_mapping: + c-uri: c-uri + c-useragent: c-useragent + cs-method: cs-method + cs-bytes: cs-bytes + cs-cookie-vars: cs-cookie-vars + c-uri-extension: c-uri-extension + c-uri-query: c-uri-query + cs-cookie: cs-cookie + cs-host: cs-host + cs-referrer: cs-referrer + cs-version: cs-version + r-dns: r-dns + sc-status: sc-status + post-body: post-body \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_application.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_application.yml new file mode 100644 index 00000000..1f2364ed --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_application.yml @@ -0,0 +1,27 @@ +platform: Sigma +source: windows_application + +log_source: + product: [windows] + service: [application] + +default_log_source: + product: windows + service: application + +field_mapping: + EventID: EventID + src_ip: src_ip + source: source + additional_information: additional_information + EventData: EventData + Channel: Channel + statement: statement + Faulting application path: Faulting application path + object_name: object_name + class_type: class_type + action_id: action_id + Provider_Name: Provider_Name + Data: Data + Message: Message + Level: Level diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_bits_client.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_bits_client.yml new file mode 100644 index 00000000..43f71e2f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_bits_client.yml @@ -0,0 +1,17 @@ +platform: Sigma +source: windows_bits_client + + +log_source: + product: [windows] + category: [bits-client] + +default_log_source: + product: windows + category: bits-client + +field_mapping: + LocalName: LocalName + EventID: EventID + RemoteName: RemoteName + processPath: processPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_dns_query.yml new file mode 100644 index 00000000..78c6a067 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_dns_query.yml @@ -0,0 +1,17 @@ +platform: Sigma +source: windows_dns_query + + +log_source: + product: [windows] + category: [dns_query] + +default_log_source: + product: windows + category: dns_query + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_driver_load.yml new file mode 100644 index 00000000..3297e295 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_driver_load.yml @@ -0,0 +1,16 @@ +platform: Sigma +source: windows_driver_load + + +log_source: + product: [windows] + category: [driver_load] + +default_log_source: + product: macos + category: driver_load + +field_mapping: + ImageLoaded: ImageLoaded + ImagePath: ImagePath + Hashes: Hashes \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_image_load.yml new file mode 100644 index 00000000..cd654de5 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_image_load.yml @@ -0,0 +1,18 @@ +platform: Sigma +source: windows_image_load + + +log_source: + product: [windows] + category: [image_load] + +default_log_source: + product: windows + category: image_load + +field_mapping: + Image: Image + ImageLoaded: ImageLoaded + SignatureStatus: SignatureStatus + OriginalFileName: OriginalFileName + Signed: Signed \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_ldap_debug.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_ldap_debug.yml new file mode 100644 index 00000000..77d88409 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_ldap_debug.yml @@ -0,0 +1,15 @@ +platform: Sigma +source: windows_ldap_debug + + +log_source: + product: [windows] + service: [ldap_debug] + +default_log_source: + product: windows + service: ldap_debug + +field_mapping: + EventID: EventID + SearchFilter: SearchFilter \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_network_connection.yml new file mode 100644 index 00000000..cb36aab2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_network_connection.yml @@ -0,0 +1,20 @@ +platform: Sigma +source: windows_network_connection + + +log_source: + product: [windows] + category: [network_connection] + +default_log_source: + product: windows + category: network_connection + +field_mapping: + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + Initiated: Initiated \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_ntlm.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_ntlm.yml new file mode 100644 index 00000000..5f9ba0ac --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_ntlm.yml @@ -0,0 +1,17 @@ +platform: Sigma +source: windows_ntlm + + +log_source: + product: [windows] + service: [ntlm] + +default_log_source: + product: windows + service: ntlm + +field_mapping: + ProcessName: ProcessName + TargetName: TargetName + WorkstationName: WorkstationName + EventID: EventID \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_pipe_created.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_pipe_created.yml new file mode 100644 index 00000000..7934d1e2 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_pipe_created.yml @@ -0,0 +1,16 @@ +platform: Sigma +source: windows_pipe_created + + +log_source: + product: [windows] + category: [pipe_created] + +default_log_source: + product: windows + category: pipe_created + +field_mapping: + EventID: EventID + PipeName: PipeName + Image: Image \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_powershell.yml new file mode 100644 index 00000000..697a9379 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_powershell.yml @@ -0,0 +1,22 @@ +platform: Sigma +source: windows_powershell + + +log_source: + product: [windows] + service: [powershell] + category: [ps_classic_provider_start, ps_classic_script, ps_classic_start, ps_module, ps_script] + +default_log_source: + product: windows + service: powershell + +field_mapping: + EventID: EventID + CommandLine: CommandLine + ScriptBlockText: ScriptBlockText + Payload: Payload + HostApplication: HostApplication + ContextInfo: ContextInfo + HostName: HostName + EngineVersion: EngineVersion \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_process_access.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_process_access.yml new file mode 100644 index 00000000..3b6aeb2c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_process_access.yml @@ -0,0 +1,24 @@ + +platform: Sigma +source: windows_process_access + + +log_source: + product: [windows] + category: [process_access] + +default_log_source: + product: windows + category: process_access + +field_mapping: + SourceProcessGUID: SourceProcessGUID + SourceProcessId: SourceProcessId + SourceThreadId: SourceThreadId + SourceImage: SourceImage + TargetProcessGUID: TargetProcessGUID + TargerProcessId: TargerProcessId + TargetImage: TargetImage + GrantedAccess: GrantedAccess + CallTrace: CallTrace + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_process_creation.yml new file mode 100644 index 00000000..4b27937b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_process_creation.yml @@ -0,0 +1,23 @@ +platform: Sigma +source: windows_process_creation + + +log_source: + product: [windows] + category: [process_creation] + +default_log_source: + product: windows + category: process_creation + +field_mapping: + CommandLine: CommandLine + CurrentDirectory: CurrentDirectory + Hashes: Hashes + Image: Image + IntegrityLevel: IntegrityLevel + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage + ParentUser: ParentUser + Product: Product + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_registry_event.yml new file mode 100644 index 00000000..d44431b6 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_registry_event.yml @@ -0,0 +1,31 @@ +platform: Sigma +source: windows_registry_event + +log_source: + product: [windows] + category: [registry_event, registry_set, registry_delete, registry_add] + +default_log_source: + product: windows + category: registry_event + +field_mapping: + TargetObject: TargetObject + Image: Image + Details: Details + EventType: EventType + CommandLine: CommandLine + LogonId: LogonId + Product: Product + Company: Company + IntegrityLevel: IntegrityLevel + CurrentDirectory: CurrentDirectory + ProcessId: ProcessId + ParentProcessId: ParentProcessId + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage + ParentUser: ParentUser + ParentIntegrityLevel: ParentIntegrityLevel + ParentLogonId: ParentLogonId + ParentProduct: ParentProduct + ParentCompany: ParentCompany \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_security.yml new file mode 100644 index 00000000..ebb39a11 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_security.yml @@ -0,0 +1,147 @@ +platform: Sigma +source: windows_security + + +log_source: + product: [windows] + service: [security] + +default_log_source: + product: windows + service: security + +field_mapping: + EventID: EventID + ParentImage: ParentImage + AccessMask: AccessMask + AccountName: AccountName + AllowedToDelegateTo: AllowedToDelegateTo + AttributeLDAPDisplayName: AttributeLDAPDisplayName + AuditPolicyChanges: AuditPolicyChanges + AuthenticationPackageName: AuthenticationPackageName + CallingProcessName: CallingProcessName + Channel: Channel + ComputerName: ComputerName + EventType: EventType + FailureReason: FailureReason + FileName: FileName + GrantedAccess: GrantedAccess + Hashes: Hashes + HiveName: HiveName + IpAddress: IpAddress + IpPort: IpPort + KeyLength: KeyLength + LogonProcessName: LogonProcessName + LogonType: LogonType + LinkName: LinkName + MemberName: MemberName + MemberSid: MemberSid + NewProcessName: NewProcessName + ObjectClass: ObjectClass + ObjectType: ObjectType + ObjectValueName: ObjectValueName + Path: Path + CommandLine: CommandLine + OldUacValue: OldUacValue + CertIssuerName: CertIssuerName + SubStatus: SubStatus + DisplayName: DisplayName + TaskContent: TaskContent + ServiceSid: ServiceSid + CertThumbprint: CertThumbprint + ObjectName: ObjectName + ClassName: ClassName + NotificationPackageName: NotificationPackageName + NewSd: NewSd + TestSigning: TestSigning + TargetInfo: TargetInfo + ParentProcessId: ParentProcessId + AccessList: AccessList + GroupMembership: GroupMembership + FilterName: FilterName + ChangeType: ChangeType + LayerName: LayerName + ServiceAccount: ServiceAccount + ClientProcessId: ClientProcessId + AttributeValue: AttributeValue + SessionName: SessionName + TaskName: TaskName + ObjectDN: ObjectDN + TemplateContent: TemplateContent + NewTemplateContent: NewTemplateContent + SourcePort: SourcePort + PasswordLastSet: PasswordLastSet + PrivilegeList: PrivilegeList + DeviceDescription: DeviceDescription + TargetServerName: TargetServerName + NewTargetUserName: NewTargetUserName + OperationType: OperationType + DestPort: DestPort + ServiceStartType: ServiceStartType + OldTargetUserName: OldTargetUserName + UserPrincipalName: UserPrincipalName + Accesses: Accesses + DnsHostName: DnsHostName + DisableIntegrityChecks: DisableIntegrityChecks + AuditSourceName: AuditSourceName + Workstation: Workstation + DestAddress: DestAddress + PreAuthType: PreAuthType + SecurityPackageName: SecurityPackageName + SubjectLogonId: SubjectLogonId + NewUacValue: NewUacValue + EnabledPrivilegeList: EnabledPrivilegeList + RelativeTargetName: RelativeTargetName + CertSerialNumber: CertSerialNumber + SidHistory: SidHistory + TargetLogonId: TargetLogonId + KernelDebug: KernelDebug + CallerProcessName: CallerProcessName + Properties: Properties + UserAccountControl: UserAccountControl + RegistryValue: RegistryValue + SecurityID: SecurityID + ServiceFileName: ServiceFileName + SecurityDescriptor: SecurityDescriptor + ServiceName: ServiceName + ShareName: ShareName + NewValue: NewValue + Source: Source + Status: Status + SubjectDomainName: SubjectDomainName + SubjectUserName: SubjectUserName + SubjectUserSid: SubjectUserSid + SourceAddr: SourceAddr + SourceAddress: SourceAddress + TargetName: TargetName + ServicePrincipalNames: ServicePrincipalNames + TargetDomainName: TargetDomainName + TargetSid: TargetSid + TargetUserName: TargetUserName + ObjectServer: ObjectServer + TargetUserSid: TargetUserSid + TicketEncryptionType: TicketEncryptionType + TicketOptions: TicketOptions + WorkstationName: WorkstationName + TransmittedServices: TransmittedServices + AuthenticationAlgorithm: AuthenticationAlgorithm + LayerRTID: LayerRTID + BSSID: BSSID + BSSType: BSSType + CipherAlgorithm: CipherAlgorithm + ConnectionId: ConnectionId + ConnectionMode: ConnectionMode + InterfaceDescription: InterfaceDescription + InterfaceGuid: InterfaceGuid + OnexEnabled: OnexEnabled + PHYType: PHYType + ProfileName: ProfileName + SSID: SSID + Domain: Domain + ServiceType: ServiceType + SourceName: SourceName + StartType: StartType + UserID: UserID + ParentProcessName: ParentProcessName + Service: Service + ProcessName: ProcessName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_sysmon.yml new file mode 100644 index 00000000..6b7100cd --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_sysmon.yml @@ -0,0 +1,63 @@ +platform: Sigma +source: windows_sysmon + + +log_source: + product: [windows] + service: [sysmon] + +default_log_source: + product: windows + service: sysmon + +field_mapping: + CommandLine: CommandLine + Image: Image + ParentImage: ParentImage + EventID: EventID + CallTrace: CallTrace + Company: Company + CurrentDirectory: CurrentDirectory + Description: Description + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationIsIpv6: DestinationIsIpv6 + DestinationPort: DestinationPort + DestinationPortName: DestinationPortName + Hashes: Hashes + Initiated: Initiated + IntegrityLevel: IntegrityLevel + ParentCommandLine: ParentCommandLine + Product: Product + Protocol: Protocol + RuleName: RuleName + SourceHostname: SourceHostname + SourceIp: SourceIp + SourceIsIpv6: SourceIsIpv6 + SourcePort: SourcePort + SourcePortName: SourcePortName + TargetFilename: TargetFilename + User: User + OriginalFileName: OriginalFileName + Signed: Signed + Signature: Signature + SignatureStatus: SignatureStatus + TargetObject: TargetObject + Details: Details + QueryName: QueryName + QueryResults: QueryResults + QueryStatus: QueryStatus + IsExecutable: IsExecutable + PipeName: PipeName + ImageLoaded: ImageLoaded + ImagePath: ImagePath + Imphash: Imphash + SourceImage: SourceImage + StartModule: StartModule + TargetImage: TargetImage + Device: Device + ProcessID: ProcessID + FileVersion: FileVersion + StartAddress: StartAddress + StartFunction: StartFunction + EventType: EventType \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_system.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_system.yml new file mode 100644 index 00000000..9c0cf860 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_system.yml @@ -0,0 +1,27 @@ +platform: Sigma +source: windows_system + + +log_source: + product: [windows] + service: [system] + +default_log_source: + product: windows + service: system + +field_mapping: + EventID: EventID + AccountName: AccountName + ImagePath: ImagePath + ServiceName: ServiceName + ServiceType: ServiceType + StartType: StartType + Provider_Name: Provider_Name + Origin: Origin + HiveName: HiveName + Caption: Caption + param1: param1 + param2: param2 + Channel: Channel + DeviceName: DeviceName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/sigma/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/sigma/windows_wmi_event.yml new file mode 100644 index 00000000..40a4d512 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/sigma/windows_wmi_event.yml @@ -0,0 +1,15 @@ +platform: Sigma +source: windows_wmi_event + + +log_source: + product: [windows] + category: [wmi_event] + +default_log_source: + product: windows + category: wmi_event + +field_mapping: + Destination: Destination + EventID: EventID diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/aws_cloudtrail.yml b/uncoder-core/app/translator/mappings/platforms/splunk/aws_cloudtrail.yml new file mode 100644 index 00000000..96bb06b8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/aws_cloudtrail.yml @@ -0,0 +1,35 @@ +platform: Splunk +source: aws_cloudtrail + + +log_source: + sourcetype: [aws:cloudtrail] + +default_log_source: + sourcetype: aws:cloudtrail + +field_mapping: + eventSource: eventSource + eventName: eventName + AdditionalEventData: AdditionalEventData + additionalEventData.MFAUsed: additionalEventData.MFAUsed + errorCode: errorCode + errorMessage: errorMessage + eventType: eventType + requestParameters: requestParameters + requestParameters.attribute: requestParameters.attribute + requestParameters.ipPermissions.items.ipRanges.items.cidrIP: requestParameters.ipPermissions.items.ipRanges.items.cidrIP + requestParameters.ipPermissions.items.ipRanges.items.fromPort: requestParameters.ipPermissions.items.ipRanges.items.fromPort + requestParameters.userData: requestParameters.userData + responseElements: responseElements + responseElements.ConsoleLogin: responseElements.ConsoleLogin + responseElements.pendingModifiedValues.masterUserPassword: responseElements.pendingModifiedValues.masterUserPassword + responseElements.publiclyAccessible: responseElements.publiclyAccessible + status: status + terminatingRuleId: terminatingRuleId + userAgent: userAgent + userIdentity.arn: userIdentity.arn + userIdentity.principalId: userIdentity.principalId + userIdentity.sessionContext.sessionIssuer.type: userIdentity.sessionContext.sessionIssuer.type + userIdentity.type: userIdentity.type + userIdentity.userName: userIdentity.userName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/aws_eks.yml b/uncoder-core/app/translator/mappings/platforms/splunk/aws_eks.yml new file mode 100644 index 00000000..38e225d7 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/aws_eks.yml @@ -0,0 +1,25 @@ +platform: Splunk +source: aws_eks + + +log_source: + sourcetype: [aws:*] + +default_log_source: + sourcetype: aws:* + +field_mapping: + annotations.authorization.k8s.io\/decision: annotations.authorization.k8s.io\/decision + annotations.podsecuritypolicy.policy.k8s.io\/admit-policy: annotations.podsecuritypolicy.policy.k8s.io\/admit-policy + aws_node_type: aws_node_type + objectRef.namespace: objectRef.namespace + objectRef.resource: objectRef.resource + objectRef.subresource: objectRef.subresource + requestObject.rules.resources: requestObject.rules.resources + requestObject.rules.verbs: requestObject.rules.verbs + requestObject.spec.containers.image: requestObject.spec.containers.image + requestURI: requestURI + stage: stage + user.groups: user.groups + user.username: user.username + verb: verb \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/azure_AzureDiagnostics.yml b/uncoder-core/app/translator/mappings/platforms/splunk/azure_AzureDiagnostics.yml new file mode 100644 index 00000000..90fd75a1 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/azure_AzureDiagnostics.yml @@ -0,0 +1,13 @@ +platform: Splunk +source: azure_AzureDiagnostics + + +log_source: + sourcetype: [azure:*] + +default_log_source: + sourcetype: azure:* + +field_mapping: + ResultDescription: ResultDescription + Category: Category \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/azure_BehaviorAnalytics.yml b/uncoder-core/app/translator/mappings/platforms/splunk/azure_BehaviorAnalytics.yml new file mode 100644 index 00000000..e1f17620 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/azure_BehaviorAnalytics.yml @@ -0,0 +1,20 @@ +platform: Splunk +source: azure_BehaviorAnalytics + + +log_source: + sourcetype: [azure:*] + +default_log_source: + sourcetype: azure:* + +field_mapping: + ActionType: ActionType + ActivityInsights: ActivityInsights + ActivityType: ActivityType + EventSource: EventSource + DevicesInsights: DevicesInsights + RiskDetail: RiskDetail + UsersInsights: UsersInsights + UsersInsights.IsDormantAccount: UsersInsights.IsDormantAccount + UsersInsights.IsNewAccount: UsersInsights.IsNewAccount \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/azure_aadnoninteractiveusersigninlogs.yml b/uncoder-core/app/translator/mappings/platforms/splunk/azure_aadnoninteractiveusersigninlogs.yml new file mode 100644 index 00000000..ad6bb5eb --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/azure_aadnoninteractiveusersigninlogs.yml @@ -0,0 +1,13 @@ +platform: Splunk +source: azure_aadnoninteractiveusersigninlogs + + +log_source: + sourcetype: [azure:*] + +default_log_source: + sourcetype: azure:* + +field_mapping: + UserAgent: UserAgent + Type: Type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/azure_azureactivity.yml b/uncoder-core/app/translator/mappings/platforms/splunk/azure_azureactivity.yml new file mode 100644 index 00000000..337125f4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/azure_azureactivity.yml @@ -0,0 +1,22 @@ +platform: Splunk +source: azure_azureactivity + + +log_source: + sourcetype: [mscs:azure:*, azure:*] + +default_log_source: + sourcetype: mscs:azure:* + +field_mapping: + ActivityStatus: ActivityStatus + ActivityStatusValue: ActivityStatusValue + ActivitySubstatusValue: ActivitySubstatusValue + Authorization: Authorization + Category: Category + CategoryValue: CategoryValue + OperationName: OperationName + OperationNameValue: OperationNameValue + ResourceId: ResourceId + ResourceProviderValue: ResourceProviderValue + Type: Type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/azure_azuread.yml b/uncoder-core/app/translator/mappings/platforms/splunk/azure_azuread.yml new file mode 100644 index 00000000..69e3d195 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/azure_azuread.yml @@ -0,0 +1,23 @@ +platform: Splunk +source: azure_azuread + + +log_source: + sourcetype: [azure:aad:*] + +default_log_source: + sourcetype: azure:aad:* + +field_mapping: + ActivityDisplayName: ActivityDisplayName + Category: Category + LoggedByService: LoggedByService + Result: Result + OperationName: OperationName + TargetResources: TargetResources + AADOperationType: AADOperationType + InitiatedBy: InitiatedBy + ResultReason: ResultReason + Status: Status + Status.errorCode: Status.errorCode + UserAgent: UserAgent \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/azure_signinlogs.yml b/uncoder-core/app/translator/mappings/platforms/splunk/azure_signinlogs.yml new file mode 100644 index 00000000..4f669d89 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/azure_signinlogs.yml @@ -0,0 +1,29 @@ +platform: Splunk +source: azure_signinlogs + + +log_source: + sourcetype: [azure:aad:*] + +default_log_source: + sourcetype: azure:aad:* + +field_mapping: + AppDisplayName: AppDisplayName + AppId: AppId + AuthenticationRequirement: AuthenticationRequirement + Category: Category + ConditionalAccessStatus: ConditionalAccessStatus + DeviceDetail: DeviceDetail + IsInteractive: IsInteractive + NetworkLocationDetails: NetworkLocationDetails + ResourceDisplayName: ResourceDisplayName + ResourceIdentity: ResourceIdentity + ResultDescription: ResultDescription + ResultType: ResultType + Status.errorCode: Status.errorCode + Status: Status + Status.failureReason: Status.failureReason + TokenIssuerType: TokenIssuerType + UserAgent: UserAgent + UserPrincipalName: UserPrincipalName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/default.yml b/uncoder-core/app/translator/mappings/platforms/splunk/default.yml new file mode 100644 index 00000000..d0cbfc38 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/default.yml @@ -0,0 +1,9 @@ +platform: Splunk +source: default + + +log_source: + source: WinEventLog:* + +default_log_source: + source: WinEventLog:* diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/firewall.yml b/uncoder-core/app/translator/mappings/platforms/splunk/firewall.yml new file mode 100644 index 00000000..ed886d9c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/firewall.yml @@ -0,0 +1,17 @@ +platform: Splunk +source: firewall + + +log_source: + sourcetype: [fortigate_traffic] + index: [fortigate] + +default_log_source: + sourcetype: fortigate_traffic + index: fortigate + +field_mapping: + src-ip: src_ip + src-port: src_port + dst-ip: dst_ip + dst-port: dst_port diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/gcp_gcp.audit.yml b/uncoder-core/app/translator/mappings/platforms/splunk/gcp_gcp.audit.yml new file mode 100644 index 00000000..be54b882 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/gcp_gcp.audit.yml @@ -0,0 +1,21 @@ +platform: Splunk +source: gcp_gcp.audit + + +log_source: + sourcetype: [google:gcp:*] + +default_log_source: + index: google:gcp:* + +field_mapping: + backupConfiguration.enabled: backupConfiguration.enabled + gcp.audit.method_name: gcp.audit.method_name + jsonPayload.enforcedSecurityPolicy.configuredAction: jsonPayload.enforcedSecurityPolicy.configuredAction + jsonPayload.enforcedSecurityPolicy.matchedFieldValue: jsonPayload.enforcedSecurityPolicy.matchedFieldValue + message: message + methodName: methodName + protoPayload.methodName: protoPayload.methodName + protoPayload.request.canIpForward: protoPayload.request.canIpForward + protoPayload.serviceName: protoPayload.serviceName + serviceName: serviceName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/gcp_pubsub.yml b/uncoder-core/app/translator/mappings/platforms/splunk/gcp_pubsub.yml new file mode 100644 index 00000000..dbfd2736 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/gcp_pubsub.yml @@ -0,0 +1,16 @@ +platform: Splunk +source: gcp_pubsub + + +log_source: + sourcetype: [google:gcp:*] + +default_log_source: + index: google:gcp:* + +field_mapping: + data.protoPayload.methodName: data.protoPayload.methodName + data.protoPayload.request.function.timeout: data.protoPayload.request.function.timeout + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.action + data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member: data.protoPayload.serviceData.policyDelta.bindingDeltas{}.member + data.resource.type: data.resource.type \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/linux_auditd.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_auditd.yml new file mode 100644 index 00000000..afd115b0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/linux_auditd.yml @@ -0,0 +1,16 @@ +platform: Splunk +source: linux_auditd + + +log_source: + sourcetype: [linux:audit] + +default_log_source: + sourcetype: linux:audit + +field_mapping: + a0: a0 + a1: a1 + a2: a2 + a3: a3 + exe: exe \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/linux_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_dns_query.yml new file mode 100644 index 00000000..14ddc245 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/linux_dns_query.yml @@ -0,0 +1,15 @@ +platform: Splunk +source: linux_dns_query + + +log_source: + index: [linux] + +default_log_source: + index: linux + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_access.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_access.yml similarity index 78% rename from siem-converter/app/converter/mappings/platforms/splunk/linux_file_access.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/linux_file_access.yml index 0388f75e..7decb5cb 100644 --- a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_access.yml +++ b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_access.yml @@ -1,6 +1,6 @@ platform: Splunk source: linux_file_access -description: Text that describe current mapping + log_source: index: [linux] diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_change.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_change.yml similarity index 78% rename from siem-converter/app/converter/mappings/platforms/splunk/linux_file_change.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/linux_file_change.yml index 81ae50ce..62b51f76 100644 --- a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_change.yml +++ b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_change.yml @@ -1,6 +1,6 @@ platform: Splunk source: linux_file_change -description: Text that describe current mapping + log_source: index: [linux] diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_create.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_create.yml similarity index 78% rename from siem-converter/app/converter/mappings/platforms/splunk/linux_file_create.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/linux_file_create.yml index 52b38011..18c5b21c 100644 --- a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_create.yml +++ b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_create.yml @@ -1,6 +1,6 @@ platform: Splunk source: linux_file_create -description: Text that describe current mapping + log_source: index: [linux] diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_delete.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_delete.yml similarity index 78% rename from siem-converter/app/converter/mappings/platforms/splunk/linux_file_delete.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/linux_file_delete.yml index be3bbee4..785486b4 100644 --- a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_delete.yml +++ b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_delete.yml @@ -1,6 +1,6 @@ platform: Splunk source: linux_file_delete -description: Text that describe current mapping + log_source: index: [linux] diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_event.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_event.yml new file mode 100644 index 00000000..061c9a92 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_event.yml @@ -0,0 +1,13 @@ +platform: Splunk +source: linux_file_event + + +log_source: + index: [linux] + +default_log_source: + index: linux + +field_mapping: + Image: Image + TargetFilename: TargetFilename diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_rename.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_rename.yml similarity index 78% rename from siem-converter/app/converter/mappings/platforms/splunk/linux_file_rename.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/linux_file_rename.yml index 4502c461..67f7c145 100644 --- a/siem-converter/app/converter/mappings/platforms/splunk/linux_file_rename.yml +++ b/uncoder-core/app/translator/mappings/platforms/splunk/linux_file_rename.yml @@ -1,6 +1,6 @@ platform: Splunk source: linux_file_rename -description: Text that describe current mapping + log_source: index: [linux] diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_network_connection.yml similarity index 100% rename from siem-converter/app/converter/mappings/platforms/splunk/linux_network_connection.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/linux_network_connection.yml diff --git a/siem-converter/app/converter/mappings/platforms/splunk/linux_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/splunk/linux_process_creation.yml similarity index 100% rename from siem-converter/app/converter/mappings/platforms/splunk/linux_process_creation.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/linux_process_creation.yml diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/macos_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/splunk/macos_dns_query.yml new file mode 100644 index 00000000..6ad582c6 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/macos_dns_query.yml @@ -0,0 +1,15 @@ +platform: Splunk +source: macos_dns_query + + +log_source: + index: [macos] + +default_log_source: + index: macos + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_access.yml b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_access.yml similarity index 78% rename from siem-converter/app/converter/mappings/platforms/splunk/macos_file_access.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/macos_file_access.yml index 6c9cf807..305176a8 100644 --- a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_access.yml +++ b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_access.yml @@ -1,6 +1,6 @@ platform: Splunk source: macos_file_access -description: Text that describe current mapping + log_source: index: [macos] diff --git a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_change.yml b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_change.yml similarity index 78% rename from siem-converter/app/converter/mappings/platforms/splunk/macos_file_change.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/macos_file_change.yml index 4ff6ab13..c99c6464 100644 --- a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_change.yml +++ b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_change.yml @@ -1,6 +1,6 @@ platform: Splunk source: macos_file_change -description: Text that describe current mapping + log_source: index: [macos] diff --git a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_delete.yml b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_delete.yml similarity index 78% rename from siem-converter/app/converter/mappings/platforms/splunk/macos_file_delete.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/macos_file_delete.yml index 21316f24..25cdcf1a 100644 --- a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_delete.yml +++ b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_delete.yml @@ -1,6 +1,6 @@ platform: Splunk source: macos_file_delete -description: Text that describe current mapping + log_source: index: [macos] diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_event.yml b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_event.yml new file mode 100644 index 00000000..ef1bbda8 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_event.yml @@ -0,0 +1,13 @@ +platform: Splunk +source: macos_file_event + + +log_source: + index: [macos] + +default_log_source: + index: macos + +field_mapping: + Image: Image + TargetFilename: TargetFilename \ No newline at end of file diff --git a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_rename.yml b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_rename.yml similarity index 78% rename from siem-converter/app/converter/mappings/platforms/splunk/macos_file_rename.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/macos_file_rename.yml index 307cfcc4..25286894 100644 --- a/siem-converter/app/converter/mappings/platforms/splunk/macos_file_rename.yml +++ b/uncoder-core/app/translator/mappings/platforms/splunk/macos_file_rename.yml @@ -1,6 +1,6 @@ platform: Splunk source: macos_file_rename -description: Text that describe current mapping + log_source: index: [macos] diff --git a/siem-converter/app/converter/mappings/platforms/splunk/macos_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/splunk/macos_network_connection.yml similarity index 100% rename from siem-converter/app/converter/mappings/platforms/splunk/macos_network_connection.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/macos_network_connection.yml diff --git a/siem-converter/app/converter/mappings/platforms/splunk/macos_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/splunk/macos_process_creation.yml similarity index 100% rename from siem-converter/app/converter/mappings/platforms/splunk/macos_process_creation.yml rename to uncoder-core/app/translator/mappings/platforms/splunk/macos_process_creation.yml diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/okta_okta.yml b/uncoder-core/app/translator/mappings/platforms/splunk/okta_okta.yml new file mode 100644 index 00000000..3f55621f --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/okta_okta.yml @@ -0,0 +1,33 @@ +platform: Splunk +source: okta_okta + + +log_source: + sourcetype: [OktaIM2:*] + +default_log_source: + sourcetype: OktaIM2:* + +field_mapping: + client.user.id: client.user.id + source.user.id: source.user.id + User: User + alternateId: alternateId + client.user.full_name: client.user.full_name + source.user.full_name: source.user.full_name + related.user: related.user + client.ip: client.ip + source.ip: source.ip + user_agent.original: user_agent.original + userAgent.os: userAgent.os + userAgent.browser: userAgent.browser + client.zone: client.zone + client.device: client.device + client.id: client.id + event.action: event.action + outcome.reason: outcome.reason + event.outcome: event.outcome + client.as.number: client.as.number + client.as.organization.name: client.as.organization.name + client.domain: client.domain + source.domain: source.domain diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_bits_client.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_bits_client.yml new file mode 100644 index 00000000..babbd610 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_bits_client.yml @@ -0,0 +1,14 @@ +platform: Splunk +source: windows_bits_client + +log_source: + sourcetype: [XmlWinEventLog:Microsoft-Windows-Bits-Client/Operational] + +default_log_source: + sourcetype: XmlWinEventLog:Microsoft-Windows-Bits-Client/Operational + +field_mapping: + LocalName: LocalName + EventID: EventID + RemoteName: RemoteName + processPath: processPath \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_dns_query.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_dns_query.yml new file mode 100644 index 00000000..d8e40100 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_dns_query.yml @@ -0,0 +1,17 @@ +platform: Splunk +source: windows_dns_query + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + Image: Image + User: User + QueryName: QueryName + QueryResults: QueryResults \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_driver_load.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_driver_load.yml new file mode 100644 index 00000000..86b76510 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_driver_load.yml @@ -0,0 +1,16 @@ +platform: Splunk +source: windows_driver_load + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + ImageLoaded: ImageLoaded + ImagePath: ImagePath + Hashes: Hashes \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_access.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_access.yml new file mode 100644 index 00000000..48ab5786 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_access.yml @@ -0,0 +1,20 @@ +platform: Splunk +source: windows_file_access + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + CreationUtcTime: CreationUtcTime + FileName: FileName + Image: Image + OriginalFilename: OriginalFilename + PreviousCreationUtcTime: PreviousCreationUtcTime + TargetFilename: TargetFilename + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_change.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_change.yml new file mode 100644 index 00000000..f45393aa --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_change.yml @@ -0,0 +1,20 @@ +platform: Splunk +source: windows_file_change + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + CreationUtcTime: CreationUtcTime + FileName: FileName + Image: Image + OriginalFilename: OriginalFilename + PreviousCreationUtcTime: PreviousCreationUtcTime + TargetFilename: TargetFilename + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_create.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_create.yml new file mode 100644 index 00000000..485ea463 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_create.yml @@ -0,0 +1,20 @@ +platform: Splunk +source: windows_file_create + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + CreationUtcTime: CreationUtcTime + FileName: FileName + Image: Image + OriginalFilename: OriginalFilename + PreviousCreationUtcTime: PreviousCreationUtcTime + TargetFilename: TargetFilename + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_delete.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_delete.yml new file mode 100644 index 00000000..13660235 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_delete.yml @@ -0,0 +1,20 @@ +platform: Splunk +source: windows_file_delete + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + CreationUtcTime: CreationUtcTime + FileName: FileName + Image: Image + OriginalFilename: OriginalFilename + PreviousCreationUtcTime: PreviousCreationUtcTime + TargetFilename: TargetFilename + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_event.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_event.yml new file mode 100644 index 00000000..ed0855d3 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_event.yml @@ -0,0 +1,20 @@ +platform: Splunk +source: windows_file_event + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + CreationUtcTime: CreationUtcTime + FileName: FileName + Image: Image + OriginalFilename: OriginalFilename + PreviousCreationUtcTime: PreviousCreationUtcTime + TargetFilename: TargetFilename + User: User diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_rename.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_rename.yml new file mode 100644 index 00000000..dae50085 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_file_rename.yml @@ -0,0 +1,20 @@ +platform: Splunk +source: windows_file_rename + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + CreationUtcTime: CreationUtcTime + FileName: FileName + Image: Image + OriginalFilename: OriginalFilename + PreviousCreationUtcTime: PreviousCreationUtcTime + TargetFilename: TargetFilename + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_image_load.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_image_load.yml new file mode 100644 index 00000000..3cc22f55 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_image_load.yml @@ -0,0 +1,18 @@ +platform: Splunk +source: windows_image_load + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + Image: Image + ImageLoaded: ImageLoaded + SignatureStatus: SignatureStatus + OriginalFileName: OriginalFileName + Signed: Signed \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_ldap_debug.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_ldap_debug.yml new file mode 100644 index 00000000..f8241ba4 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_ldap_debug.yml @@ -0,0 +1,13 @@ +platform: Splunk +source: windows_ldap_debug + + +log_source: + sourcetype: [XmlWinEventLog:Microsoft-Windows-LDAP-Client/Debug] + +default_log_source: + sourcetype: XmlWinEventLog:Microsoft-Windows-LDAP-Client/Debug + +field_mapping: + EventID: EventID + SearchFilter: SearchFilter \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_network_connection.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_network_connection.yml new file mode 100644 index 00000000..7a92b32c --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_network_connection.yml @@ -0,0 +1,20 @@ +platform: Splunk +source: windows_network_connection + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + Image: Image + DestinationHostname: DestinationHostname + DestinationIp: DestinationIp + DestinationPort: DestinationPort + SourceIp: SourceIp + SourcePort: SourcePort + Initiated: Initiated \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_ntlm.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_ntlm.yml new file mode 100644 index 00000000..7902c0fe --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_ntlm.yml @@ -0,0 +1,15 @@ +platform: Splunk +source: windows_ntlm + + +log_source: + sourcetype: [XmlWinEventLog:Microsoft-Windows-NTLM/Operational] + +default_log_source: + sourcetype: XmlWinEventLog:Microsoft-Windows-NTLM/Operational + +field_mapping: + WorkstationName: WorkstationName + ProcessName: ProcessName + EventID: EventID + TargetName: TargetName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_powershell.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_powershell.yml new file mode 100644 index 00000000..db9be04b --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_powershell.yml @@ -0,0 +1,19 @@ +platform: Splunk +source: windows_powershell + + +log_source: + source: [WinEventLog:Microsoft-Windows-PowerShell/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-PowerShell/Operational + +field_mapping: + EventID: EventID + CommandLine: CommandLine + ScriptBlockText: ScriptBlockText + Payload: Payload + HostApplication: HostApplication + ContextInfo: ContextInfo + HostName: HostName + EngineVersion: EngineVersion \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_process_creation.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_process_creation.yml new file mode 100644 index 00000000..83a29f31 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_process_creation.yml @@ -0,0 +1,21 @@ +platform: Splunk +source: windows_process_creation + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational, WinEventLog:Security] + +default_log_source: + source: WinEventLog:Security + +field_mapping: + CommandLine: CommandLine + CurrentDirectory: CurrentDirectory + Hashes: Hashes + Image: Image + IntegrityLevel: IntegrityLevel + ParentCommandLine: ParentCommandLine + ParentImage: ParentImage + ParentUser: ParentUser + Product: Product + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_registry_event.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_registry_event.yml new file mode 100644 index 00000000..a2169567 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_registry_event.yml @@ -0,0 +1,18 @@ +platform: Splunk +source: windows_registry_event + + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + TargetObject: TargetObject + Image: Image + Details: Details + EventType: EventType + User: User \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_security.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_security.yml new file mode 100644 index 00000000..53156e2e --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_security.yml @@ -0,0 +1,148 @@ +platform: Splunk +source: windows_security + + +log_source: + source: [WinEventLog:Security] + +default_log_source: + source: WinEventLog:Security + +field_mapping: + EventID: EventID + ParentImage: ParentImage + AccessMask: AccessMask + AccountName: AccountName + AllowedToDelegateTo: AllowedToDelegateTo + AttributeLDAPDisplayName: AttributeLDAPDisplayName + AuditPolicyChanges: AuditPolicyChanges + AuthenticationPackageName: AuthenticationPackageName + CallingProcessName: CallingProcessName + Channel: Channel + ComputerName: ComputerName + EventType: EventType + FailureReason: FailureReason + FileName: FileName + GrantedAccess: GrantedAccess + Hashes: Hashes + HiveName: HiveName + IpAddress: IpAddress + IpPort: IpPort + KeyLength: KeyLength + LogonProcessName: LogonProcessName + LogonType: LogonType + LinkName: LinkName + MemberName: MemberName + MemberSid: MemberSid + NewProcessName: NewProcessName + ObjectClass: ObjectClass + ObjectName: ObjectName + ObjectType: ObjectType + ObjectValueName: ObjectValueName + Path: Path + CommandLine: CommandLine + OldUacValue: OldUacValue + CertIssuerName: CertIssuerName + SubStatus: SubStatus + DisplayName: DisplayName + TaskContent: TaskContent + ServiceSid: ServiceSid + CertThumbprint: CertThumbprint + ObjectName: ObjectName + ClassName: ClassName + NotificationPackageName: NotificationPackageName + NewSd: NewSd + TestSigning: TestSigning + TargetInfo: TargetInfo + ClientProcessId: ClientProcessId + ParentProcessId: ParentProcessId + AccessList: AccessList + GroupMembership: GroupMembership + FilterName: FilterName + ChangeType: ChangeType + LayerName: LayerName + ServiceAccount: ServiceAccount + ClientProcessId: ClientProcessId + AttributeValue: AttributeValue + SessionName: SessionName + TaskName: TaskName + ObjectDN: ObjectDN + TemplateContent: TemplateContent + NewTemplateContent: NewTemplateContent + SourcePort: SourcePort + PasswordLastSet: PasswordLastSet + PrivilegeList: PrivilegeList + DeviceDescription: DeviceDescription + TargetServerName: TargetServerName + NewTargetUserName: NewTargetUserName + OperationType: OperationType + DestPort: DestPort + ServiceStartType: ServiceStartType + OldTargetUserName: OldTargetUserName + UserPrincipalName: UserPrincipalName + Accesses: Accesses + DnsHostName: DnsHostName + DisableIntegrityChecks: DisableIntegrityChecks + AuditSourceName: AuditSourceName + Workstation: Workstation + DestAddress: DestAddress + PreAuthType: PreAuthType + SecurityPackageName: SecurityPackageName + SubjectLogonId: SubjectLogonId + NewUacValue: NewUacValue + EnabledPrivilegeList: EnabledPrivilegeList + RelativeTargetName: RelativeTargetName + CertSerialNumber: CertSerialNumber + SidHistory: SidHistory + TargetLogonId: TargetLogonId + KernelDebug: KernelDebug + CallerProcessName: CallerProcessName + ProcessName: ProcessName + Properties: Properties + UserAccountControl: UserAccountControl + RegistryValue: RegistryValue + SecurityID: SecurityID + ServiceFileName: ServiceFileName + SecurityDescriptor: SecurityDescriptor + ServiceName: ServiceName + ShareName: ShareName + NewValue: NewValue + Source: Source + Status: Status + SubjectDomainName: SubjectDomainName + SubjectUserName: SubjectUserName + SubjectUserSid: SubjectUserSid + SourceAddr: SourceAddr + SourceAddress: SourceAddress + TargetName: TargetName + ServicePrincipalNames: ServicePrincipalNames + TargetDomainName: TargetDomainName + TargetSid: TargetSid + TargetUserName: TargetUserName + ObjectServer: ObjectServer + TargetUserSid: TargetUserSid + TicketEncryptionType: TicketEncryptionType + TicketOptions: TicketOptions + WorkstationName: WorkstationName + TransmittedServices: TransmittedServices + AuthenticationAlgorithm: AuthenticationAlgorithm + LayerRTID: LayerRTID + BSSID: BSSID + BSSType: BSSType + CipherAlgorithm: CipherAlgorithm + ConnectionId: ConnectionId + ConnectionMode: ConnectionMode + InterfaceDescription: InterfaceDescription + InterfaceGuid: InterfaceGuid + OnexEnabled: OnexEnabled + PHYType: PHYType + ProfileName: ProfileName + SSID: SSID + Domain: Domain + ServiceType: ServiceType + SourceName: SourceName + StartType: StartType + UserID: UserID + ParentProcessName: ParentProcessName + Service: Service + ProcessName: ProcessName \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_sysmon.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_sysmon.yml new file mode 100644 index 00000000..89bf98e0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_sysmon.yml @@ -0,0 +1,63 @@ +platform: Splunk +source: windows_sysmon + +log_source: + source: [WinEventLog:Microsoft-Windows-Sysmon/Operational] + sourcetype: [XmlWinEventLog:Microsoft-Windows-Sysmon/Operational] + +default_log_source: + source: WinEventLog:Microsoft-Windows-Sysmon/Operational + sourcetype: XmlWinEventLog:Microsoft-Windows-Sysmon/Operational + +field_mapping: + CommandLine: CommandLine + ParentImage: ParentImage + EventID: EventCode + CallTrace: CallTrace + Company: Company + CurrentDirectory: CurrentDirectory + Description: Description + DestinationIp: DestinationIp + DestinationIsIpv6: DestinationIsIpv6 + DestinationPort: DestinationPort + DestinationPortName: DestinationPortName + Hashes: Hashes + Initiated: Initiated + IntegrityLevel: IntegrityLevel + ParentCommandLine: ParentCommandLine + Product: Product + Protocol: Protocol + RuleName: RuleName + SourceHostname: SourceHostname + SourceIp: SourceIp + SourceIsIpv6: SourceIsIpv6 + SourcePort: SourcePort + SourcePortName: SourcePortName + TargetFilename: TargetFilename + OriginalFileName: OriginalFileName + Signed: Signed + Signature: Signature + SignatureStatus: SignatureStatus + TargetObject: TargetObject + Details: Details + QueryName: QueryName + QueryResults: QueryResults + QueryStatus: QueryStatus + IsExecutable: IsExecutable + PipeName: PipeName + Imphash: Imphash + SourceImage: SourceImage + StartModule: StartModule + TargetImage: TargetImage + FileVersion: FileVersion + StartAddress: StartAddress + StartFunction: StartFunction + EventType: EventType + Image: process_name + ProcessID: process_guid + ImageLoaded: process + timestamp: _time + ImagePath: process_path + User: user_id + DestinationHostname: dest + Device: Computer \ No newline at end of file diff --git a/uncoder-core/app/translator/mappings/platforms/splunk/windows_wmi_event.yml b/uncoder-core/app/translator/mappings/platforms/splunk/windows_wmi_event.yml new file mode 100644 index 00000000..b1e415d0 --- /dev/null +++ b/uncoder-core/app/translator/mappings/platforms/splunk/windows_wmi_event.yml @@ -0,0 +1,13 @@ +platform: Splunk +source: windows_wmi_event + + +log_source: + sourcetype: [XmlWinEventLog:Microsoft-Windows-WMI-Activity/Operational] + +default_log_source: + sourcetype: XmlWinEventLog:Microsoft-Windows-WMI-Activity/Operational + +field_mapping: + Destination: Destination + EventID: EventID \ No newline at end of file diff --git a/siem-converter/app/converter/platforms/athena/parsers/__init__.py b/uncoder-core/app/translator/mappings/utils/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/athena/parsers/__init__.py rename to uncoder-core/app/translator/mappings/utils/__init__.py diff --git a/uncoder-core/app/translator/mappings/utils/load_from_files.py b/uncoder-core/app/translator/mappings/utils/load_from_files.py new file mode 100644 index 00000000..89ea2a94 --- /dev/null +++ b/uncoder-core/app/translator/mappings/utils/load_from_files.py @@ -0,0 +1,33 @@ +import os +from collections.abc import Generator + +import yaml + +from app.translator.const import APP_PATH + +COMMON_FIELD_MAPPING_FILE_NAME = "common.yml" +DEFAULT_FIELD_MAPPING_FILE_NAME = "default.yml" + + +class LoaderFileMappings: + base_mapping_filepath = os.path.join(APP_PATH, "mappings/platforms") + + @staticmethod + def load_mapping(mapping_file_path: str) -> dict: + with open(mapping_file_path) as yaml_obj: + try: + return yaml.safe_load(yaml_obj) + except yaml.YAMLError as err: + print(err) + return {} + + def load_platform_mappings(self, platform_dir: str) -> Generator[dict, None, None]: + platform_path = os.path.join(self.base_mapping_filepath, platform_dir) + for mapping_file in os.listdir(platform_path): + if mapping_file not in (COMMON_FIELD_MAPPING_FILE_NAME, DEFAULT_FIELD_MAPPING_FILE_NAME): + yield self.load_mapping(mapping_file_path=os.path.join(platform_path, mapping_file)) + yield self.load_mapping(mapping_file_path=os.path.join(platform_path, DEFAULT_FIELD_MAPPING_FILE_NAME)) + + def load_common_mapping(self, platform_dir: str) -> dict: + platform_path = os.path.join(self.base_mapping_filepath, platform_dir) + return self.load_mapping(mapping_file_path=os.path.join(platform_path, COMMON_FIELD_MAPPING_FILE_NAME)) diff --git a/uncoder-core/app/translator/platforms/__init__.py b/uncoder-core/app/translator/platforms/__init__.py new file mode 100644 index 00000000..f9c89733 --- /dev/null +++ b/uncoder-core/app/translator/platforms/__init__.py @@ -0,0 +1,14 @@ +import os + +from app.translator.tools.utils import execute_module +from const import PLATFORMS_PATH + + +def init_platforms() -> None: + for platform in [f for f in os.listdir(PLATFORMS_PATH) if os.path.isdir(os.path.join(PLATFORMS_PATH, f))]: + if not platform.startswith("__") and not platform.endswith("__"): + # Platforms __init__.py execution + execute_module(f"{PLATFORMS_PATH}/{platform}/__init__.py") + + +init_platforms() diff --git a/uncoder-core/app/translator/platforms/anomali/__init__.py b/uncoder-core/app/translator/platforms/anomali/__init__.py new file mode 100644 index 00000000..5cd64d01 --- /dev/null +++ b/uncoder-core/app/translator/platforms/anomali/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.anomali.renders.anomali import AnomaliQueryRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/anomali/const.py b/uncoder-core/app/translator/platforms/anomali/const.py new file mode 100644 index 00000000..3d14733d --- /dev/null +++ b/uncoder-core/app/translator/platforms/anomali/const.py @@ -0,0 +1,11 @@ +from app.translator.core.models.platform_details import PlatformDetails + +ANOMALI_QUERY_DETAILS = { + "platform_id": "anomali-aql-query", + "name": "Anomali Security Analytics Query", + "group_name": "Anomali Security Analytics", + "platform_name": "Query", + "group_id": "anomali", +} + +anomali_query_details = PlatformDetails(**ANOMALI_QUERY_DETAILS) diff --git a/uncoder-core/app/translator/platforms/anomali/mapping.py b/uncoder-core/app/translator/platforms/anomali/mapping.py new file mode 100644 index 00000000..aaecea36 --- /dev/null +++ b/uncoder-core/app/translator/platforms/anomali/mapping.py @@ -0,0 +1,18 @@ +from app.translator.core.mapping import BaseStrictLogSourcesPlatformMappings, LogSourceSignature +from app.translator.platforms.anomali.const import anomali_query_details + + +class AnomaliLogSourceSignature(LogSourceSignature): + def is_suitable(self) -> bool: + return True + + def __str__(self) -> str: + return "" + + +class AnomaliMappings(BaseStrictLogSourcesPlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> AnomaliLogSourceSignature: # noqa: ARG002 + return AnomaliLogSourceSignature() + + +anomali_query_mappings = AnomaliMappings(platform_dir="anomali", platform_details=anomali_query_details) diff --git a/siem-converter/app/converter/platforms/athena/renders/__init__.py b/uncoder-core/app/translator/platforms/anomali/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/athena/renders/__init__.py rename to uncoder-core/app/translator/platforms/anomali/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/anomali/renders/anomali.py b/uncoder-core/app/translator/platforms/anomali/renders/anomali.py new file mode 100644 index 00000000..6f9e89f6 --- /dev/null +++ b/uncoder-core/app/translator/platforms/anomali/renders/anomali.py @@ -0,0 +1,72 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render import PlatformQueryRender +from app.translator.managers import render_manager +from app.translator.platforms.anomali.const import anomali_query_details +from app.translator.platforms.anomali.mapping import AnomaliMappings, anomali_query_mappings +from app.translator.platforms.base.sql.renders.sql import SqlFieldValueRender + + +class AnomaliFieldValueRender(SqlFieldValueRender): + details: PlatformDetails = anomali_query_details + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + + value = f"'%{self._pre_process_value(field, value)}%'" + return f"{field} like {value}" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + + value = f"'%{self._pre_process_value(field, value)}'" + return f"{field} like {value}" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" + + value = f"'{self._pre_process_value(field, value)}%'" + return f"{field} like {value}" + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + return f'message contains "{self._pre_process_value(field, value)}"' + + +@render_manager.register +class AnomaliQueryRender(PlatformQueryRender): + details: PlatformDetails = anomali_query_details + mappings: AnomaliMappings = anomali_query_mappings + + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + comment_symbol = "--" + is_single_line_comment = True + + field_value_render = AnomaliFieldValueRender(or_token=or_token) + + @staticmethod + def _finalize_search_query(query: str) -> str: + return f"| where {query}" if query else "" diff --git a/uncoder-core/app/translator/platforms/arcsight/__init__.py b/uncoder-core/app/translator/platforms/arcsight/__init__.py new file mode 100644 index 00000000..f666494e --- /dev/null +++ b/uncoder-core/app/translator/platforms/arcsight/__init__.py @@ -0,0 +1,2 @@ +from app.translator.platforms.arcsight.renders.arcsight import ArcSightQueryRender # noqa: F401 +from app.translator.platforms.arcsight.renders.arcsight_cti import ArcsightKeyword # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/arcsight/const.py b/uncoder-core/app/translator/platforms/arcsight/const.py new file mode 100644 index 00000000..0f431d87 --- /dev/null +++ b/uncoder-core/app/translator/platforms/arcsight/const.py @@ -0,0 +1,26 @@ +from app.translator.core.models.platform_details import PlatformDetails + +ARCSIGHT_QUERY_DETAILS = { + "platform_id": "arcsight-query", + "name": "ArcSight Query", + "group_name": "ArcSight", + "group_id": "arcsight", + "platform_name": "Query", + "alt_platform_name": "CEF", +} + + +DEFAULT_ARCSIGHT_CTI_MAPPING = { + "SourceIP": "sourceAddress", + "DestinationIP": "destinationAddress", + "Domain": "destinationDnsDomain", + "URL": "requestUrl", + "HashMd5": "fileHash", + "HashSha1": "fileHash", + "HashSha256": "fileHash", + "HashSha512": "fileHash", + "Emails": "sender-address", + "Files": "winlog.event_data.TargetFilename", +} + +arcsight_query_details = PlatformDetails(**ARCSIGHT_QUERY_DETAILS) diff --git a/uncoder-core/app/translator/platforms/arcsight/escape_manager.py b/uncoder-core/app/translator/platforms/arcsight/escape_manager.py new file mode 100644 index 00000000..6478e2ff --- /dev/null +++ b/uncoder-core/app/translator/platforms/arcsight/escape_manager.py @@ -0,0 +1,14 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class ArcSightEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [EscapeDetails(pattern='(["\\()])', escape_symbols="\\\\\g<1>")] + } + + +arcsight_escape_manager = ArcSightEscapeManager() diff --git a/uncoder-core/app/translator/platforms/arcsight/mapping.py b/uncoder-core/app/translator/platforms/arcsight/mapping.py new file mode 100644 index 00000000..b5686f48 --- /dev/null +++ b/uncoder-core/app/translator/platforms/arcsight/mapping.py @@ -0,0 +1,18 @@ +from app.translator.core.mapping import BaseStrictLogSourcesPlatformMappings, LogSourceSignature +from app.translator.platforms.arcsight.const import arcsight_query_details + + +class ArcSightLogSourceSignature(LogSourceSignature): + def is_suitable(self) -> bool: + return True + + def __str__(self) -> str: + return "" + + +class ArcSightMappings(BaseStrictLogSourcesPlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> ArcSightLogSourceSignature: # noqa: ARG002 + return ArcSightLogSourceSignature() + + +arcsight_query_mappings = ArcSightMappings(platform_dir="arcsight", platform_details=arcsight_query_details) diff --git a/siem-converter/app/converter/platforms/base/__init__.py b/uncoder-core/app/translator/platforms/arcsight/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/base/__init__.py rename to uncoder-core/app/translator/platforms/arcsight/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/arcsight/renders/arcsight.py b/uncoder-core/app/translator/platforms/arcsight/renders/arcsight.py new file mode 100644 index 00000000..3bb65d38 --- /dev/null +++ b/uncoder-core/app/translator/platforms/arcsight/renders/arcsight.py @@ -0,0 +1,101 @@ +from typing import Optional, Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mapping import LogSourceSignature +from app.translator.core.mixins.tokens import ExtraConditionMixin +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.core.str_value_manager import StrValue, StrValueManager +from app.translator.managers import render_manager +from app.translator.platforms.arcsight.const import arcsight_query_details +from app.translator.platforms.arcsight.mapping import ArcSightMappings, arcsight_query_mappings +from app.translator.platforms.arcsight.str_value_manager import arcsight_str_value_manager + + +class ArcSightFieldValue(BaseFieldValueRender): + details: PlatformDetails = arcsight_query_details + str_value_manager: StrValueManager = arcsight_str_value_manager + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f'"{value}"' + + @staticmethod + def _wrap_int_value(value: int) -> str: + return f'"{value}"' + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.equal_modifier(field, val) for val in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True) + return f"{field} = {value}" + + def less_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f"{field} < {self._pre_process_value(field, value, wrap_str=True)}" + + def less_or_equal_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f"{field} <= {self._pre_process_value(field, value, wrap_str=True)}" + + def greater_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f"{field} > {self._pre_process_value(field, value, wrap_str=True)}" + + def greater_or_equal_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f"{field} > {self._pre_process_value(field, value, wrap_str=True)}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.not_equal_modifier(field, val) for val in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True) + return f"{field} != {value}" + + def is_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.is_none(field=field, value=v) for v in value)})" + return f"NOT _exists_:{field}" + + def is_not_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.is_not_none(field=field, value=v) for v in value)})" + return f"_exists_:{field}" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field, val) for val in value)})" + value = self._wrap_str_value(value) + return f"{field} CONTAINS {value}" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field, val) for val in value)})" + value = self._wrap_str_value(value) + return f"{field} ENDSWITH {value}" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field, val) for val in value)})" + value = self._wrap_str_value(value) + return f"{field} STARTSWITH {value}" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field, val) for val in value)})" + value = self._wrap_str_value(value) + return f"{field} CONTAINS {value}" + + +@render_manager.register +class ArcSightQueryRender(ExtraConditionMixin, PlatformQueryRender): + details: PlatformDetails = arcsight_query_details + mappings: ArcSightMappings = arcsight_query_mappings + + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + comment_symbol = "//" + + field_value_render = ArcSightFieldValue(or_token=or_token) + + def generate_prefix(self, log_source_signature: Optional[LogSourceSignature], functions_prefix: str = "") -> str: # noqa: ARG002 + return "" diff --git a/uncoder-core/app/translator/platforms/arcsight/renders/arcsight_cti.py b/uncoder-core/app/translator/platforms/arcsight/renders/arcsight_cti.py new file mode 100644 index 00000000..22b135cc --- /dev/null +++ b/uncoder-core/app/translator/platforms/arcsight/renders/arcsight_cti.py @@ -0,0 +1,18 @@ +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.arcsight.const import arcsight_query_details, DEFAULT_ARCSIGHT_CTI_MAPPING + + +@render_cti_manager.register +class ArcsightKeyword(RenderCTI): + details: PlatformDetails = arcsight_query_details + + default_mapping = DEFAULT_ARCSIGHT_CTI_MAPPING + field_value_template: str = "{key} = {value}" + or_operator: str = " OR " + group_or_operator: str = " OR " + or_group: str = "{or_group}" + result_join: str = "" + final_result_for_many: str = '({result}) AND type != 2 | rex field = flexString1 mode=sed "s//Sigma: None/g"\n' + final_result_for_one: str = '{result} AND type != 2 | rex field = flexString1 mode=sed "s//Sigma: None/g"\n' diff --git a/uncoder-core/app/translator/platforms/arcsight/str_value_manager.py b/uncoder-core/app/translator/platforms/arcsight/str_value_manager.py new file mode 100644 index 00000000..e9a98b2a --- /dev/null +++ b/uncoder-core/app/translator/platforms/arcsight/str_value_manager.py @@ -0,0 +1,27 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from app.translator.core.str_value_manager import StrValueManager +from app.translator.platforms.arcsight.escape_manager import ArcSightEscapeManager, arcsight_escape_manager + + +class ArcSightStrValueManager(StrValueManager): + escape_manager: ArcSightEscapeManager = arcsight_escape_manager + + +arcsight_str_value_manager = ArcSightStrValueManager() diff --git a/uncoder-core/app/translator/platforms/athena/__init__.py b/uncoder-core/app/translator/platforms/athena/__init__.py new file mode 100644 index 00000000..adb534b1 --- /dev/null +++ b/uncoder-core/app/translator/platforms/athena/__init__.py @@ -0,0 +1,3 @@ +from app.translator.platforms.athena.parsers.athena import AthenaQueryParser # noqa: F401 +from app.translator.platforms.athena.renders.athena import AthenaQueryRender # noqa: F401 +from app.translator.platforms.athena.renders.athena_cti import AthenaCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/athena/const.py b/uncoder-core/app/translator/platforms/athena/const.py new file mode 100644 index 00000000..ea10735d --- /dev/null +++ b/uncoder-core/app/translator/platforms/athena/const.py @@ -0,0 +1,26 @@ +from app.translator.core.models.platform_details import PlatformDetails + +ATHENA_QUERY_DETAILS = { + "platform_id": "athena-sql-query", + "name": "AWS Athena Query", + "group_name": "AWS Athena", + "platform_name": "Query", + "group_id": "athena", + "alt_platform_name": "OCSF", +} + +DEFAULT_ATHENA_CTI_MAPPING = { + "SourceIP": "src_endpoint", + "DestinationIP": "dst_endpoint", + "Domain": "dst_endpoint", + "URL": "http_request", + "HashMd5": "unmapped.file.hash.md5", + "HashSha1": "unmapped.file.hash.sha1", + "HashSha256": "unmapped.file.hash.sha256", + "HashSha512": "unmapped.file.hash.sha512", + "Email": "email", + "FileName": "file.name", +} + + +athena_query_details = PlatformDetails(**ATHENA_QUERY_DETAILS) diff --git a/uncoder-core/app/translator/platforms/athena/mapping.py b/uncoder-core/app/translator/platforms/athena/mapping.py new file mode 100644 index 00000000..3829d890 --- /dev/null +++ b/uncoder-core/app/translator/platforms/athena/mapping.py @@ -0,0 +1,26 @@ +from typing import Optional + +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature +from app.translator.platforms.athena.const import athena_query_details + + +class AthenaLogSourceSignature(LogSourceSignature): + def __init__(self, tables: Optional[list[str]], default_source: dict): + self.tables = set(tables or []) + self._default_source = default_source or {} + + def is_suitable(self, table: str) -> bool: + return table in self.tables + + def __str__(self) -> str: + return self._default_source.get("table", "") + + +class AthenaMappings(BasePlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> AthenaLogSourceSignature: + tables = mapping.get("log_source", {}).get("table") + default_log_source = mapping["default_log_source"] + return AthenaLogSourceSignature(tables=tables, default_source=default_log_source) + + +athena_query_mappings = AthenaMappings(platform_dir="athena", platform_details=athena_query_details) diff --git a/siem-converter/app/converter/platforms/base/lucene/__init__.py b/uncoder-core/app/translator/platforms/athena/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/base/lucene/__init__.py rename to uncoder-core/app/translator/platforms/athena/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/athena/parsers/athena.py b/uncoder-core/app/translator/platforms/athena/parsers/athena.py new file mode 100644 index 00000000..9e2bd555 --- /dev/null +++ b/uncoder-core/app/translator/platforms/athena/parsers/athena.py @@ -0,0 +1,31 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import parser_manager +from app.translator.platforms.athena.const import athena_query_details +from app.translator.platforms.athena.mapping import AthenaMappings, athena_query_mappings +from app.translator.platforms.base.sql.parsers.sql import SqlQueryParser + + +@parser_manager.register_supported_by_roota +class AthenaQueryParser(SqlQueryParser): + details: PlatformDetails = athena_query_details + mappings: AthenaMappings = athena_query_mappings + query_delimiter_pattern = r"\sFROM\s\S*\sWHERE\s" + table_pattern = r"\sFROM\s(?P
[a-zA-Z\.\-\*]+)\sWHERE\s" diff --git a/siem-converter/app/converter/platforms/base/lucene/parsers/__init__.py b/uncoder-core/app/translator/platforms/athena/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/base/lucene/parsers/__init__.py rename to uncoder-core/app/translator/platforms/athena/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/athena/renders/athena.py b/uncoder-core/app/translator/platforms/athena/renders/athena.py new file mode 100644 index 00000000..2b431af2 --- /dev/null +++ b/uncoder-core/app/translator/platforms/athena/renders/athena.py @@ -0,0 +1,44 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.athena.const import athena_query_details +from app.translator.platforms.athena.mapping import AthenaMappings, athena_query_mappings +from app.translator.platforms.base.sql.renders.sql import SqlFieldValueRender, SqlQueryRender + + +class AthenaFieldValueRender(SqlFieldValueRender): + details: PlatformDetails = athena_query_details + + +@render_manager.register +class AthenaQueryRender(SqlQueryRender): + details: PlatformDetails = athena_query_details + mappings: AthenaMappings = athena_query_mappings + + or_token = "OR" + + field_value_render = AthenaFieldValueRender(or_token=or_token) + comment_symbol = "--" + is_single_line_comment = True + + @staticmethod + def _finalize_search_query(query: str) -> str: + return f"WHERE {query}" if query else "" diff --git a/uncoder-core/app/translator/platforms/athena/renders/athena_cti.py b/uncoder-core/app/translator/platforms/athena/renders/athena_cti.py new file mode 100644 index 00000000..285b3e2e --- /dev/null +++ b/uncoder-core/app/translator/platforms/athena/renders/athena_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.athena.const import DEFAULT_ATHENA_CTI_MAPPING, athena_query_details + + +@render_cti_manager.register +class AthenaCTI(RenderCTI): + details: PlatformDetails = athena_query_details + + field_value_template: str = "{key} = '{value}'" + or_operator: str = " OR " + group_or_operator: str = " OR " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "SELECT * from eventlog where {result}\n" + final_result_for_one: str = "SELECT * from eventlog where {result}\n" + default_mapping = DEFAULT_ATHENA_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/athena/tokenizer.py b/uncoder-core/app/translator/platforms/athena/tokenizer.py new file mode 100644 index 00000000..aadf6405 --- /dev/null +++ b/uncoder-core/app/translator/platforms/athena/tokenizer.py @@ -0,0 +1,77 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import Any, ClassVar, Union + +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.models.field import FieldValue +from app.translator.core.models.identifier import Identifier +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.tools.utils import get_match_group + + +class AthenaTokenizer(QueryTokenizer): + single_value_operators_map: ClassVar[dict[str, str]] = { + "=": OperatorType.EQ, + "<=": OperatorType.LTE, + "<": OperatorType.LT, + ">=": OperatorType.GTE, + ">": OperatorType.GT, + "!=": OperatorType.NOT_EQ, + "<>": OperatorType.NOT_EQ, + "like": OperatorType.EQ, + } + multi_value_operators_map: ClassVar[dict[str, str]] = {"in": OperatorType.EQ} + + field_pattern = r'(?P"[a-zA-Z\._\-\s]+"|[a-zA-Z\._\-]+)' + num_value_pattern = rf"(?P<{ValueType.number_value}>\d+(?:\.\d+)*)\s*" + bool_value_pattern = rf"(?P<{ValueType.bool_value}>true|false)\s*" + single_quotes_value_pattern = ( + rf"""'(?P<{ValueType.single_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{{\}}\s]|'')*)'""" + ) + _value_pattern = rf"{num_value_pattern}|{bool_value_pattern}|{single_quotes_value_pattern}" + multi_value_pattern = rf"""\((?P<{ValueType.multi_value}>\d+(?:,\s*\d+)*|'(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{{\}}\s]|'')*'(?:,\s*'(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{{\}}\s]|'')*')*)\)""" # noqa: E501 + + wildcard_symbol = "%" + + @staticmethod + def should_process_value_wildcards(operator: str) -> bool: + return operator.lower() in ("like",) + + def get_operator_and_value(self, match: re.Match, operator: str = OperatorType.EQ) -> tuple[str, Any]: + if (num_value := get_match_group(match, group_name=ValueType.number_value)) is not None: + return operator, num_value + + if (bool_value := get_match_group(match, group_name=ValueType.bool_value)) is not None: + return operator, bool_value + + if (s_q_value := get_match_group(match, group_name=ValueType.single_quotes_value)) is not None: + return operator, s_q_value + + return super().get_operator_and_value(match, operator) + + @staticmethod + def create_field_value(field_name: str, operator: Identifier, value: Union[str, list]) -> FieldValue: + field_name = field_name.strip('"') + return FieldValue(source_name=field_name, operator=operator, value=value) + + def tokenize(self, query: str) -> list: + query = re.sub(r"\s*ESCAPE\s*'.'", "", query) # remove `ESCAPE 'escape_char'` in LIKE expr + return super().tokenize(query) diff --git a/siem-converter/app/converter/platforms/base/lucene/renders/__init__.py b/uncoder-core/app/translator/platforms/base/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/base/lucene/renders/__init__.py rename to uncoder-core/app/translator/platforms/base/__init__.py diff --git a/siem-converter/app/converter/platforms/base/spl/__init__.py b/uncoder-core/app/translator/platforms/base/aql/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/base/spl/__init__.py rename to uncoder-core/app/translator/platforms/base/aql/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/aql/const.py b/uncoder-core/app/translator/platforms/base/aql/const.py new file mode 100644 index 00000000..b2503bd4 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/const.py @@ -0,0 +1,10 @@ +from app.translator.core.custom_types.values import ValueType + +UTF8_PAYLOAD_PATTERN = r"UTF8\(payload\)" +NUM_VALUE_PATTERN = rf"(?P<{ValueType.number_value}>\d+(?:\.\d+)*)" +SINGLE_QUOTES_VALUE_PATTERN = rf"""'(?P<{ValueType.single_quotes_value}>(?:[:a-zA-Zа-яА-Я\*0-9=+%#\-\/\\|,;_<>`~".$&^@!?\(\)\{{\}}\[\]\s]|'')*)'""" # noqa: E501,RUF001 +TABLE_PATTERN = r"\s+FROM\s+[a-zA-Z.\-*]+" +TABLE_GROUP_PATTERN = r"\s+FROM\s+(?P
[a-zA-Z.\-*]+)" + +FIELD_NAME_PATTERN = rf"(?P<{ValueType.no_quotes_value}>[a-zA-Z0-9\._\-]+)" +DOUBLE_QUOTES_FIELD_NAME_PATTERN = rf'"(?P<{ValueType.double_quotes_value}>[a-zA-Z0-9\._\-\s]+)"' diff --git a/uncoder-core/app/translator/platforms/base/aql/escape_manager.py b/uncoder-core/app/translator/platforms/base/aql/escape_manager.py new file mode 100644 index 00000000..f0f232f2 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/escape_manager.py @@ -0,0 +1,18 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class AQLEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [EscapeDetails(pattern=r"(')", escape_symbols=r"'\1")], + ValueType.regex_value: [ + EscapeDetails(pattern=r"([$^*+()\[\]{}|.?\-\\])", escape_symbols=r"\\\1"), + EscapeDetails(pattern=r"(')", escape_symbols=r"'\1"), + ], + } + + +aql_escape_manager = AQLEscapeManager() diff --git a/uncoder-core/app/translator/platforms/base/aql/functions/__init__.py b/uncoder-core/app/translator/platforms/base/aql/functions/__init__.py new file mode 100644 index 00000000..813ec885 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/functions/__init__.py @@ -0,0 +1,147 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import os.path +import re +from typing import Optional, Union + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.core.exceptions.functions import InvalidFunctionSignature, NotSupportedFunctionException +from app.translator.core.functions import PlatformFunctions +from app.translator.core.models.functions.base import Function, ParsedFunctions +from app.translator.core.models.functions.sort import SortLimitFunction +from app.translator.core.models.query_tokens.field import Field +from app.translator.platforms.base.aql.const import TABLE_PATTERN +from app.translator.platforms.base.aql.functions.const import ( + AGGREGATION_FUNCTIONS_MAP, + AQLFunctionType, + func_aliases_ctx_var, +) +from app.translator.platforms.base.aql.functions.manager import AQLFunctionsManager, aql_functions_manager + + +class AQLFunctions(PlatformFunctions): + dir_path: str = os.path.abspath(os.path.dirname(__file__)) + + function_delimiter = "" + functions_pattern = r"\s(?P(group by|order by|last))" + manager: AQLFunctionsManager = aql_functions_manager + + def parse(self, query: str) -> tuple[str, ParsedFunctions]: + parsed = [] + not_supported = [] + invalid = [] + + query_prefix, query = re.split(TABLE_PATTERN, query, flags=re.IGNORECASE) + if not re.match(r"\s*SELECT\s+(?:UTF8\(payload\)|\*)", query_prefix, flags=re.IGNORECASE): + self._parse_function( + function_name=AQLFunctionType.fields, + function=query_prefix, + parsed=parsed, + not_supported=not_supported, + invalid=invalid, + ) + aliases = self._parse_aliases(parsed=parsed) + self.__set_aliases_ctx_var(value=list(aliases.keys())) + if search := re.search(self.functions_pattern, query, flags=re.IGNORECASE): + agg_functions = query[search.start() :] + query = query[: search.start()] + self._parse_function( + function_name=AQLFunctionType.aggregation_data_function, + function=agg_functions, + parsed=parsed, + not_supported=not_supported, + invalid=invalid, + ) + + if group_by_func := self.__filter_function_by_type(parsed, FunctionType.stats): + if table_func := self.__filter_function_by_type(parsed, FunctionType.table): + self.__group_by_post_processing(group_by_func, table_func) + else: + parsed = [func for func in parsed if func.name != FunctionType.stats] + not_supported.append(group_by_func.raw) + + parsed = self.__merge_sort_limit_functions(parsed) + self.__set_aliases_ctx_var(value=[]) + query = re.sub(r"[a-zA-Z0-9_\-\s]+WHERE", "", query, 1, flags=re.IGNORECASE) + return query, ParsedFunctions(functions=parsed, not_supported=not_supported, invalid=invalid, aliases=aliases) + + @staticmethod + def __set_aliases_ctx_var(value: list[str]) -> None: + func_aliases_ctx_var.set(value) + + @staticmethod + def __filter_function_by_type(functions: list[Function], function_type: str) -> Optional[Function]: + for func in functions: + if func.name == function_type: + return func + + @staticmethod + def __group_by_post_processing(group_by_func: Function, table_func: Function) -> None: + agg_functions = [] + for index, arg in enumerate(table_func.args): + if isinstance(arg, Function) and arg.name in AGGREGATION_FUNCTIONS_MAP.values(): + agg_functions.append(arg) + table_func.args[index] = arg.alias + + group_by_func.args = agg_functions + + @staticmethod + def __merge_sort_limit_functions(functions: list[Function]) -> list[Function]: + indices = [] + funcs = [] + for index, func in enumerate(functions): + if func.name == FunctionType.sort_limit: + func: SortLimitFunction + indices.append(index) + funcs.append(func) + + if len(funcs) == 2: # noqa: PLR2004 + funcs[1].args = funcs[1].args or funcs[0].args + funcs[1].limit = funcs[1].limit or funcs[0].limit + funcs[1].raw = f"{funcs[1].raw} {funcs[0].raw}" + functions.pop(indices[0]) + + return functions + + def _parse_function( + self, function: str, function_name: str, parsed: list[Function], not_supported: list[str], invalid: list[str] + ) -> None: + try: + function_parser = self.manager.get_hof_parser(function_name) + function_token = function_parser.parse(func_body=function, raw=function) + if isinstance(function_token, list): + parsed.extend(function_token) + else: + parsed.append(function_token) + except NotSupportedFunctionException: + not_supported.append(function) + except InvalidFunctionSignature: + invalid.append(function) + + @staticmethod + def _parse_aliases(parsed: list[Union[Field, Function]]) -> dict[str, Function]: + return { + arg.alias.name: arg + for function in parsed + for arg in function.args + if isinstance(arg, Function) and arg.alias + } + + +aql_functions = AQLFunctions() diff --git a/uncoder-core/app/translator/platforms/base/aql/functions/const.py b/uncoder-core/app/translator/platforms/base/aql/functions/const.py new file mode 100644 index 00000000..e9481cc0 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/functions/const.py @@ -0,0 +1,51 @@ +from contextvars import ContextVar + +from app.translator.core.custom_types.functions import FunctionType +from app.translator.tools.custom_enum import CustomEnum + + +class AQLFunctionType(CustomEnum): + lower: str = "LOWER" + upper: str = "UPPER" + min: str = "MIN" + max: str = "MAX" + sum: str = "SUM" + avg: str = "AVG" + count: str = "COUNT" + distinct_count: str = "DISTINCTCOUNT" + group_by: str = "GROUP BY" + last: str = "LAST" + fields: str = "SELECT" + limit: str = "LIMIT" + order_by: str = "ORDER BY" + + aggregation_data_function: str = "aggregation_data_function" + + +class AQLFunctionGroupType(CustomEnum): + agg = "agg" + str_conversion = "str_conversion" + + +class AQLSortOrderType(CustomEnum): + asc: str = "ASC" + desc: str = "DESC" + + +class AQLTimeFrameType(CustomEnum): + days = "DAYS" + hours = "HOURS" + minutes = "MINUTES" + + +func_aliases_ctx_var: ContextVar[list[str]] = ContextVar("func_aliases_ctx_var", default=[]) + + +AGGREGATION_FUNCTIONS_MAP = { + AQLFunctionType.avg: FunctionType.avg, + AQLFunctionType.count: FunctionType.count, + AQLFunctionType.distinct_count: FunctionType.distinct_count, + AQLFunctionType.max: FunctionType.max, + AQLFunctionType.min: FunctionType.min, + AQLFunctionType.sum: FunctionType.sum, +} diff --git a/uncoder-core/app/translator/platforms/base/aql/functions/manager.py b/uncoder-core/app/translator/platforms/base/aql/functions/manager.py new file mode 100644 index 00000000..4e112efd --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/functions/manager.py @@ -0,0 +1,8 @@ +from app.translator.core.functions import PlatformFunctionsManager + + +class AQLFunctionsManager(PlatformFunctionsManager): + ... + + +aql_functions_manager = AQLFunctionsManager() diff --git a/uncoder-core/app/translator/platforms/base/aql/log_source_map.py b/uncoder-core/app/translator/platforms/base/aql/log_source_map.py new file mode 100644 index 00000000..4a8daf66 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/log_source_map.py @@ -0,0 +1,68 @@ +from dataclasses import dataclass + + +@dataclass +class AQLLogSourceMap: + name: str + id_map: dict[str, int] + + +CATEGORYNAME_ID_MAP = { + "ACL Permit": 4012, + "Successful Registry Modification": 8012, + "File Created": 8028, + "Process Creation Success": 8110, + "DNS In Progress": 18081, + "Object Load Success": 19247, +} + +DEVICETYPE_ID_MAP = { + "Configurable Firewall Filter": 4, + "Juniper Networks Firewall and VPN": 5, + "Cisco PIX Firewall": 6, + "Apache HTTP Server": 10, + "Linux OS": 11, + "Microsoft Windows Security Event Log": 12, + "Microsoft IIS": 13, + "Cisco Adaptive Security Appliance (ASA)": 41, + "Squid Web Proxy": 46, + "F5 Networks BIG-IP LTM": 49, + "Fortinet FortiGate Security Gateway": 73, + "Symantec Gateway Security (SGS) Appliance": 82, + "Mac OS X": 102, + "Blue Coat SG Appliance": 103, + "Nortel Switched Firewall 6000": 104, + "Nortel Switched Firewall 5100": 120, + "Imperva SecureSphere": 154, + "ISC BIND": 185, + "Microsoft ISA": 191, + "Cisco ACE Firewall": 194, + "Risk Manager Default Question": 200, + "Palo Alto PA Series": 206, + "Oracle BEA WebLogic": 239, + "Barracuda Spam & Virus Firewall": 278, + "F5 Networks BIG-IP AFM": 296, + "Zscaler Nss": 331, + "Vormetric Data Security": 340, + "Amazon AWS CloudTrail": 347, + "Microsoft DNS Debug": 384, + "Microsoft Office 365": 397, + "Microsoft Azure Platform": 413, + "NGINX HTTP Server": 439, + "Microsoft Azure Active Directory": 445, + "Google Cloud Platform Firewall": 455, + "Amazon AWS Network Firewall": 456, +} + +QID_NAME_ID_MAP = { + "ProcessAccess": 5001829, + "FileCreateStreamHash": 5001834, + "Driver loaded": 5001843, + "CreateRemoteThread": 5001845, +} + +LOG_SOURCE_FUNCTIONS_MAP = { + r"CATEGORYNAME\(category\)": AQLLogSourceMap(name="category", id_map=CATEGORYNAME_ID_MAP), + r"LOGSOURCETYPENAME\(devicetype\)": AQLLogSourceMap(name="devicetype", id_map=DEVICETYPE_ID_MAP), + r"QIDNAME\(qid\)": AQLLogSourceMap(name="qid", id_map=QID_NAME_ID_MAP), +} diff --git a/uncoder-core/app/translator/platforms/base/aql/mapping.py b/uncoder-core/app/translator/platforms/base/aql/mapping.py new file mode 100644 index 00000000..55222a0a --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/mapping.py @@ -0,0 +1,63 @@ +from typing import Optional + +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature + + +class AQLLogSourceSignature(LogSourceSignature): + def __init__( + self, + device_types: Optional[list[int]], + categories: Optional[list[int]], + qids: Optional[list[int]], + qid_event_categories: Optional[list[int]], + default_source: dict, + ): + self.device_types = set(device_types or []) + self.categories = set(categories or []) + self.qids = set(qids or []) + self.qid_event_categories = set(qid_event_categories or []) + self._default_source = default_source or {} + + def is_suitable( + self, + devicetype: Optional[list[int]] = None, + category: Optional[list[int]] = None, + qid: Optional[list[int]] = None, + qideventcategory: Optional[list[int]] = None, + ) -> bool: + conditions = [ + set(devicetype).issubset(self.device_types) if devicetype else None, + set(category).issubset(self.categories) if category else None, + set(qid).issubset(self.qids) if qid else None, + set(qideventcategory).issubset(self.qid_event_categories) if qideventcategory else None, + ] + return self._check_conditions(conditions) + + def __str__(self) -> str: + return self._default_source.get("table", "events") + + @property + def extra_condition(self) -> str: + default_source = self._default_source + extra = [] + for key, value in default_source.items(): + if key != "table" and value: + _condition = f"{key}={value}" if isinstance(value, int) else f"{key}='{value}'" + extra.append(_condition) + return " AND ".join(extra) + + +class AQLMappings(BasePlatformMappings): + skip_load_default_mappings: bool = False + extend_default_mapping_with_all_fields: bool = True + + def prepare_log_source_signature(self, mapping: dict) -> AQLLogSourceSignature: + log_source = mapping.get("log_source", {}) + default_log_source = mapping["default_log_source"] + return AQLLogSourceSignature( + device_types=log_source.get("devicetype"), + categories=log_source.get("category"), + qids=log_source.get("qid"), + qid_event_categories=log_source.get("qideventcategory"), + default_source=default_log_source, + ) diff --git a/siem-converter/app/converter/platforms/base/spl/parsers/__init__.py b/uncoder-core/app/translator/platforms/base/aql/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/base/spl/parsers/__init__.py rename to uncoder-core/app/translator/platforms/base/aql/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/aql/parsers/aql.py b/uncoder-core/app/translator/platforms/base/aql/parsers/aql.py new file mode 100644 index 00000000..0dad8283 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/parsers/aql.py @@ -0,0 +1,127 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import Union + +from app.translator.core.exceptions.parser import TokenizerGeneralException +from app.translator.core.models.functions.base import ParsedFunctions +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser +from app.translator.platforms.base.aql.const import NUM_VALUE_PATTERN, SINGLE_QUOTES_VALUE_PATTERN, TABLE_GROUP_PATTERN +from app.translator.platforms.base.aql.functions import AQLFunctions, aql_functions +from app.translator.platforms.base.aql.log_source_map import LOG_SOURCE_FUNCTIONS_MAP +from app.translator.platforms.base.aql.tokenizer import AQLTokenizer +from app.translator.tools.utils import get_match_group + + +class AQLQueryParser(PlatformQueryParser): + tokenizer: AQLTokenizer = AQLTokenizer(aql_functions) + platform_functions: AQLFunctions = aql_functions + + log_source_functions = ("LOGSOURCENAME", "LOGSOURCEGROUPNAME") + log_source_function_pattern = r"\(?(?P___func_name___\([a-zA-Z]+\))(?:\s+like\s+|\s+ilike\s+|\s*=\s*)'(?P[%a-zA-Z\s]+)'\s*\)?\s+(?:and|or)?\s" # noqa: E501 + + log_source_key_types = ("devicetype", "qideventcategory", "category", "qid", *LOG_SOURCE_FUNCTIONS_MAP.keys()) + log_source_pattern = rf"___source_type___(?:\s+like\s+|\s+ilike\s+|\s*=\s*)(?:{SINGLE_QUOTES_VALUE_PATTERN}|{NUM_VALUE_PATTERN})(?:\s+(?:and|or)\s+|\s+)?" # noqa: E501 + num_value_pattern = r"[0-9]+" + multi_num_log_source_pattern = ( + rf"___source_type___\s+in\s+\((?P(?:{num_value_pattern}(?:\s*,\s*)?)+)\)(?:\s+(?:and|or)\s+|\s+)?" + ) + str_value_pattern = r"""'(?P(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{\}\s]|'')+)'""" + multi_str_log_source_pattern = ( + rf"""___source_type___\s+in\s+\((?P(?:{str_value_pattern}(?:\s*,\s*)?)+)\)(?:\s+(?:and|or)\s+|\s+)?""" + ) + + def __clean_query(self, query: str) -> str: + for func_name in self.log_source_functions: + pattern = self.log_source_function_pattern.replace("___func_name___", func_name) + while search := re.search(pattern, query, flags=re.IGNORECASE): + pos_start = search.start() + pos_end = search.end() + query = query[:pos_start] + query[pos_end:] + + return query + + @staticmethod + def __parse_multi_value_log_source(match: re.Match, query: str, pattern: str) -> tuple[str, list[str]]: + value = match.group("value") + pos_start = match.start() + pos_end = match.end() + query = query[:pos_start] + query[pos_end:] + return query, re.findall(pattern, value) + + @staticmethod + def __map_log_source_value(logsource_key: str, value: Union[str, int]) -> tuple[str, Union[int, str]]: + if log_source_map := LOG_SOURCE_FUNCTIONS_MAP.get(logsource_key): + return log_source_map.name, log_source_map.id_map.get(value, value) + return logsource_key, value + + @staticmethod + def __check_table(query: str) -> None: + if not re.search(TABLE_GROUP_PATTERN, query, flags=re.IGNORECASE): + raise TokenizerGeneralException + + def __parse_log_sources(self, query: str) -> tuple[dict[str, Union[list[str], list[int]]], str]: + log_sources = {} + + for log_source_key in self.log_source_key_types: + pattern = self.log_source_pattern.replace("___source_type___", log_source_key) + while search := re.search(pattern, query, flags=re.IGNORECASE): + num_value = get_match_group(search, group_name="num_value") + str_value = get_match_group(search, group_name="s_q_value") + value = num_value and int(num_value) or str_value + log_source_key, value = self.__map_log_source_value(log_source_key, value) + log_sources.setdefault(log_source_key, []).append(value) + pos_start = search.start() + pos_end = search.end() + query = query[:pos_start] + query[pos_end:] + + pattern = self.multi_num_log_source_pattern.replace("___source_type___", log_source_key) + if search := re.search(pattern, query, flags=re.IGNORECASE): + query, values = self.__parse_multi_value_log_source(search, query, self.num_value_pattern) + values = [int(v) for v in values] + log_sources.setdefault(log_source_key, []).extend(values) + + pattern = self.multi_str_log_source_pattern.replace("___source_type___", log_source_key) + if search := re.search(pattern, query, flags=re.IGNORECASE): + query, values = self.__parse_multi_value_log_source(search, query, self.str_value_pattern) + log_sources.setdefault(log_source_key, []).extend(values) + + return log_sources, query + + def _parse_query(self, text: str) -> tuple[str, dict[str, Union[list[str], list[int]]], ParsedFunctions]: + query = self.__clean_query(text) + self.__check_table(query) + query, functions = self.platform_functions.parse(query) + log_sources, query = self.__parse_log_sources(query) + return query, log_sources, functions + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + query, log_sources, functions = self._parse_query(raw_query_container.query) + query_tokens = self.get_query_tokens(query) + query_field_tokens, function_field_tokens, function_field_tokens_map = self.get_field_tokens( + query_tokens, functions.functions + ) + source_mappings = self.get_source_mappings(query_field_tokens + function_field_tokens, log_sources) + meta_info = raw_query_container.meta_info + meta_info.query_fields = query_field_tokens + meta_info.function_fields = function_field_tokens + meta_info.function_fields_map = function_field_tokens_map + meta_info.source_mapping_ids = [source_mapping.source_id for source_mapping in source_mappings] + return TokenizedQueryContainer(tokens=query_tokens, meta_info=meta_info, functions=functions) diff --git a/siem-converter/app/converter/platforms/base/spl/renders/__init__.py b/uncoder-core/app/translator/platforms/base/aql/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/base/spl/renders/__init__.py rename to uncoder-core/app/translator/platforms/base/aql/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/aql/renders/aql.py b/uncoder-core/app/translator/platforms/base/aql/renders/aql.py new file mode 100644 index 00000000..58fbc3ff --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/renders/aql.py @@ -0,0 +1,138 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.core.str_value_manager import StrValue +from app.translator.platforms.base.aql.mapping import AQLLogSourceSignature +from app.translator.platforms.base.aql.str_value_manager import aql_str_value_manager + + +class AQLFieldValueRender(BaseFieldValueRender): + str_value_manager = aql_str_value_manager + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f"'{value}'" + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.equal_modifier(field=field, value=v) for v in value])})" + if isinstance(value, StrValue) and value.has_spec_symbols: + return self.__render_i_like(field, value) + return f'"{field}"={self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}' + + def less_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f'"{field}"<{self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}' + + def less_or_equal_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f'"{field}"<={self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}' + + def greater_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f'"{field}">{self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}' + + def greater_or_equal_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f'"{field}">={self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}' + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + if isinstance(value, StrValue) and value.has_spec_symbols: + return self.__render_i_like(field, value, not_=True) + return f'"{field}"!={self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}' + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + return self.__render_i_like(field, value, startswith=True, endswith=True) + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + return self.__render_i_like(field, value, endswith=True) + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" + return self.__render_i_like(field, value, startswith=True) + + def __render_i_like( + self, + field: str, + value: DEFAULT_VALUE_TYPE, + startswith: bool = False, + endswith: bool = False, + not_: bool = False, + ) -> str: + prefix = "%" if endswith else "" + re_prefix = ".*" if endswith else "" + suffix = "%" if startswith else "" + re_suffix = ".*" if startswith else "" + if self.__has_special_symbols(value): + re_value = self._pre_process_value(field, value, value_type=ValueType.regex_value) + return self.__regex_modifier(field, f"{re_prefix}{re_value}{re_suffix}") + + value = self._pre_process_value(field, value, value_type=ValueType.value) + not_ = "NOT " if not_ else "" + return f"\"{field}\" {not_}ILIKE '{prefix}{value}{suffix}'" + + @staticmethod + def __has_special_symbols(value: DEFAULT_VALUE_TYPE) -> bool: + if any(char for char in str(value) if char in ("%", "_")): + return True + + return False + + @staticmethod + def __regex_modifier(field: str, value: str) -> str: + return f"\"{field}\" IMATCHES '{value}'" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + + if isinstance(value, StrValue): + value = self.str_value_manager.from_container_to_str(value, value_type=ValueType.regex_value) + return self.__regex_modifier(field, value) + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" + return self.__render_i_like("UTF8(payload)", value, startswith=True, endswith=True) + + +class AQLQueryRender(PlatformQueryRender): + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + def generate_prefix(self, log_source_signature: AQLLogSourceSignature, functions_prefix: str = "") -> str: # noqa: ARG002 + table = str(log_source_signature) + extra_condition = log_source_signature.extra_condition + return f"SELECT UTF8(payload) FROM {table} WHERE {extra_condition}" + + def wrap_with_comment(self, value: str) -> str: + return f"/* {value} */" + + @staticmethod + def _finalize_search_query(query: str) -> str: + return f"AND {query}" if query else "" diff --git a/uncoder-core/app/translator/platforms/base/aql/str_value_manager.py b/uncoder-core/app/translator/platforms/base/aql/str_value_manager.py new file mode 100644 index 00000000..2f13931b --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/str_value_manager.py @@ -0,0 +1,105 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy +from typing import ClassVar, Optional + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.str_value_manager import ( + CONTAINER_SPEC_SYMBOLS_MAP, + RE_STR_SPEC_SYMBOLS_MAP, + BaseSpecSymbol, + ReDigitalSymbol, + ReWhiteSpaceSymbol, + ReWordBoundarySymbol, + ReWordSymbol, + SingleSymbolWildCard, + StrValue, + StrValueManager, + UnboundLenWildCard, +) +from app.translator.platforms.base.aql.escape_manager import aql_escape_manager + +AQL_CONTAINER_SPEC_SYMBOLS_MAP = copy.copy(CONTAINER_SPEC_SYMBOLS_MAP) +AQL_CONTAINER_SPEC_SYMBOLS_MAP.update({SingleSymbolWildCard: "_", UnboundLenWildCard: "%"}) + + +class AQLStrValueManager(StrValueManager): + escape_manager = aql_escape_manager + container_spec_symbols_map: ClassVar[dict[type[BaseSpecSymbol], str]] = AQL_CONTAINER_SPEC_SYMBOLS_MAP + re_str_alpha_num_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = { + "b": ReWordBoundarySymbol, + "w": ReWordSymbol, + "d": ReDigitalSymbol, + "s": ReWhiteSpaceSymbol, + } + re_str_spec_symbols_map = RE_STR_SPEC_SYMBOLS_MAP + str_spec_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = { + "_": SingleSymbolWildCard, + "%": UnboundLenWildCard, + } + + def from_str_to_container( + self, + value: str, + value_type: str = ValueType.value, # noqa: ARG002 + escape_symbol: Optional[str] = None, # noqa: ARG002 + ) -> StrValue: + split = [] + prev_char = None + for char in value: + if char in self.str_spec_symbols_map: + split.append(self.str_spec_symbols_map[char]()) + else: + if char == "'": + if prev_char == "'": + split.append(char) + prev_char = char + continue + split.append(char) + + prev_char = char + + return StrValue(value, self._concat(split)) + + def from_re_str_to_container(self, value: str) -> StrValue: + value = value.replace("''", "'") + return super().from_re_str_to_container(value) + + def from_container_to_str(self, container: StrValue, value_type: str = ValueType.value) -> str: + result = "" + for el in container.split_value: + if isinstance(el, str): + result += self.escape_manager.escape(el, value_type) + elif isinstance(el, BaseSpecSymbol): + if value_type == ValueType.regex_value: + if isinstance(el, SingleSymbolWildCard): + result += "." + continue + if isinstance(el, UnboundLenWildCard): + result += ".*" + continue + + if pattern := self.container_spec_symbols_map.get(type(el)): + result += pattern + + return result + + +aql_str_value_manager = AQLStrValueManager() diff --git a/uncoder-core/app/translator/platforms/base/aql/tokenizer.py b/uncoder-core/app/translator/platforms/base/aql/tokenizer.py new file mode 100644 index 00000000..ff04be20 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/aql/tokenizer.py @@ -0,0 +1,146 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import ClassVar, Optional, Union + +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.functions import PlatformFunctions +from app.translator.core.models.functions.base import Function +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.function_value import FunctionValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.keyword import Keyword +from app.translator.core.str_value_manager import StrValue +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.platforms.base.aql.const import ( + DOUBLE_QUOTES_FIELD_NAME_PATTERN, + FIELD_NAME_PATTERN, + NUM_VALUE_PATTERN, + SINGLE_QUOTES_VALUE_PATTERN, + UTF8_PAYLOAD_PATTERN, +) +from app.translator.platforms.base.aql.functions.const import AQLFunctionGroupType +from app.translator.platforms.base.aql.str_value_manager import aql_str_value_manager +from app.translator.tools.utils import get_match_group + + +class AQLTokenizer(QueryTokenizer): + single_value_operators_map: ClassVar[dict[str, str]] = { + "=": OperatorType.EQ, + "<=": OperatorType.LTE, + "<": OperatorType.LT, + ">=": OperatorType.GTE, + ">": OperatorType.GT, + "!=": OperatorType.NOT_EQ, + "like": OperatorType.EQ, + "ilike": OperatorType.EQ, + "matches": OperatorType.REGEX, + "imatches": OperatorType.REGEX, + } + multi_value_operators_map: ClassVar[dict[str, str]] = {"in": OperatorType.EQ} + + field_pattern = r'(?P"[a-zA-Z\._\-\s]+"|[a-zA-Z\._\-]+)' + function_pattern = rf"(?P[a-zA-Z_]+)\((?:{FIELD_NAME_PATTERN}|{DOUBLE_QUOTES_FIELD_NAME_PATTERN})\)" + bool_value_pattern = rf"(?P<{ValueType.bool_value}>true|false)\s*" + _value_pattern = rf"{NUM_VALUE_PATTERN}|{bool_value_pattern}|{SINGLE_QUOTES_VALUE_PATTERN}" + multi_value_pattern = rf"""\((?P<{ValueType.multi_value}>[:a-zA-Z\"\*0-9=+%#\-_\/\\'\,.&^@!\(\s]*)\)""" + keyword_pattern = rf"{UTF8_PAYLOAD_PATTERN}\s+(?:like|LIKE|ilike|ILIKE)\s+{SINGLE_QUOTES_VALUE_PATTERN}" + + wildcard_symbol = "%" + str_value_manager = aql_str_value_manager + + def __init__(self, platform_functions: PlatformFunctions = None): + self.platform_functions = platform_functions + + @staticmethod + def should_process_value_wildcards(operator: Optional[str]) -> bool: + return operator and operator.lower() in ("like", "ilike") + + def get_operator_and_value( + self, match: re.Match, mapped_operator: str = OperatorType.EQ, operator: Optional[str] = None + ) -> tuple[str, StrValue]: + if (num_value := get_match_group(match, group_name=ValueType.number_value)) is not None: + return mapped_operator, StrValue(num_value, split_value=[num_value]) + + if (bool_value := get_match_group(match, group_name=ValueType.bool_value)) is not None: + return mapped_operator, StrValue(bool_value, split_value=[bool_value]) + + if (s_q_value := get_match_group(match, group_name=ValueType.single_quotes_value)) is not None: + if mapped_operator == OperatorType.REGEX: + return mapped_operator, self.str_value_manager.from_re_str_to_container(s_q_value) + + if self.should_process_value_wildcards(operator): + return mapped_operator, self.str_value_manager.from_str_to_container(s_q_value) + + return mapped_operator, self.str_value_manager.from_str_to_container(s_q_value) + + return super().get_operator_and_value(match, mapped_operator, operator) + + def escape_field_name(self, field_name: str) -> str: + return field_name.replace('"', r"\"").replace(" ", r"\ ").replace("(", "\(").replace(")", "\)") + + @staticmethod + def create_field_value(field_name: str, operator: Identifier, value: Union[str, list]) -> FieldValue: + field_name = field_name.strip('"') + return FieldValue(source_name=field_name, operator=operator, value=value) + + def search_keyword(self, query: str) -> tuple[Keyword, str]: + keyword_search = re.search(self.keyword_pattern, query) + _, value = self.get_operator_and_value(keyword_search) + keyword = Keyword(value=value.strip(self.wildcard_symbol)) + pos = keyword_search.end() + return keyword, query[pos:] + + def _search_function_value(self, function: Function, query: str) -> tuple[FunctionValue, str]: + operator = self.search_operator(query, function.raw) + if self.is_multi_value_flow(function.raw, operator, query): + query, grouped_values = self.search_multi_value(query=query, operator=operator, field_name=function.raw) + tokens = [ # always consists of 1 element + FunctionValue(function=function, operator=Identifier(token_type=op), value=values) + for op, values in grouped_values.items() + ] + return tokens[0], query + + query, operator, value = self.search_single_value(query=query, operator=operator, field_name=function.raw) + operator_token = Identifier(token_type=operator) + return FunctionValue(function=function, operator=operator_token, value=value), query + + def search_function_value(self, query: str) -> tuple[FunctionValue, str]: + str_conversion_func_parser = self.platform_functions.manager.get_parser(AQLFunctionGroupType.str_conversion) + if str_conversion_func_parser and (func_match := str_conversion_func_parser.get_func_match(query)): + function = str_conversion_func_parser.parse(func_match.name, func_match.match) + return self._search_function_value(function, query) + + return super().search_function_value(query) + + def _check_function_value_match(self, query: str, white_space_pattern: str = r"\s+") -> bool: + single_value_operator_group = rf"(?:{'|'.join(self.single_value_operators_map)})" + single_value_pattern = rf"""{self.function_pattern}\s*{single_value_operator_group}\s*{self.value_pattern}\s*""" + if re.match(single_value_pattern, query, re.IGNORECASE): + return True + + if self.multi_value_operators_map: + multi_value_operator_group = rf"(?:{'|'.join(self.multi_value_operators_map)})" + pattern = f"{self.function_pattern}{white_space_pattern}{multi_value_operator_group}{white_space_pattern}" + multi_value_pattern = rf"{pattern}{self.multi_value_pattern}" + if re.match(multi_value_pattern, query, re.IGNORECASE): + return True + + return False diff --git a/siem-converter/app/converter/platforms/carbonblack/__init__.py b/uncoder-core/app/translator/platforms/base/lucene/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/carbonblack/__init__.py rename to uncoder-core/app/translator/platforms/base/lucene/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/lucene/escape_manager.py b/uncoder-core/app/translator/platforms/base/lucene/escape_manager.py new file mode 100644 index 00000000..8c9f0164 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/lucene/escape_manager.py @@ -0,0 +1,17 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class LuceneEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [ + EscapeDetails(pattern=r'([_!@#$%^&*=+()\[\]{}|;:\'",.<>?/`~\-\s\\])', escape_symbols=r"\\\1") + ], + ValueType.ip: [EscapeDetails(pattern=r"([/])", escape_symbols=r"\\\1")], + } + + +lucene_escape_manager = LuceneEscapeManager() diff --git a/uncoder-core/app/translator/platforms/base/lucene/mapping.py b/uncoder-core/app/translator/platforms/base/lucene/mapping.py new file mode 100644 index 00000000..b367e2e6 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/lucene/mapping.py @@ -0,0 +1,22 @@ +from typing import Optional + +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature + + +class LuceneLogSourceSignature(LogSourceSignature): + def __init__(self, indices: Optional[list[str]], default_source: dict): + self.indices = set(indices or []) + self._default_source = default_source or {} + + def is_suitable(self, index: Optional[list[str]] = None, **kwargs) -> bool: # noqa: ARG002 + return self._check_conditions([set(index).issubset(self.indices) if index else None]) + + def __str__(self) -> str: + return self._default_source.get("index", "") + + +class LuceneMappings(BasePlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> LuceneLogSourceSignature: + indices = mapping.get("log_source", {}).get("index") + default_log_source = mapping.get("default_log_source", {}) + return LuceneLogSourceSignature(indices=indices, default_source=default_log_source) diff --git a/siem-converter/app/converter/platforms/carbonblack/mappings/__init__.py b/uncoder-core/app/translator/platforms/base/lucene/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/carbonblack/mappings/__init__.py rename to uncoder-core/app/translator/platforms/base/lucene/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/lucene/parsers/lucene.py b/uncoder-core/app/translator/platforms/base/lucene/parsers/lucene.py new file mode 100644 index 00000000..49f05c98 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/lucene/parsers/lucene.py @@ -0,0 +1,56 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re + +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser +from app.translator.platforms.base.lucene.tokenizer import LuceneTokenizer + + +class LuceneQueryParser(PlatformQueryParser): + tokenizer = LuceneTokenizer() + + log_source_pattern = r"___source_type___\s*(?:[:=])\s*(?:\"?(?P[%a-zA-Z_*:0-9\-/]+)\"|(?P[%a-zA-Z_*:0-9\-/]+))(?:\s+(?:and|or)\s+|\s+)?" # noqa: E501 + log_source_key_types = ("index", "event\.category") + + wrapped_with_comment_pattern = r"^\s*//.*(?:\n|$)" + + def _parse_query(self, query: str) -> tuple[str, dict[str, list[str]]]: + log_sources = {} + for source_type in self.log_source_key_types: + pattern = self.log_source_pattern.replace("___source_type___", source_type) + while search := re.search(pattern, query, flags=re.IGNORECASE): + group_dict = search.groupdict() + value = group_dict.get("d_q_value") or group_dict.get("value") + log_sources.setdefault(source_type, []).append(value) + pos_start = search.start() + pos_end = search.end() + query = query[:pos_start] + query[pos_end:] + + return query, log_sources + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + query, log_sources = self._parse_query(raw_query_container.query) + query_tokens = self.get_query_tokens(query) + query_field_tokens, _, _ = self.get_field_tokens(query_tokens) + source_mappings = self.get_source_mappings(query_field_tokens, log_sources) + meta_info = raw_query_container.meta_info + meta_info.query_fields = query_field_tokens + meta_info.source_mapping_ids = [source_mapping.source_id for source_mapping in source_mappings] + return TokenizedQueryContainer(tokens=query_tokens, meta_info=meta_info) diff --git a/siem-converter/app/converter/platforms/carbonblack/renders/__init__.py b/uncoder-core/app/translator/platforms/base/lucene/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/carbonblack/renders/__init__.py rename to uncoder-core/app/translator/platforms/base/lucene/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/lucene/renders/lucene.py b/uncoder-core/app/translator/platforms/base/lucene/renders/lucene.py new file mode 100644 index 00000000..f8511d82 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/lucene/renders/lucene.py @@ -0,0 +1,114 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Optional, Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.core.str_value_manager import StrValue +from app.translator.platforms.base.lucene.mapping import LuceneLogSourceSignature +from app.translator.platforms.base.lucene.str_value_manager import lucene_str_value_manager + + +class LuceneFieldValueRender(BaseFieldValueRender): + str_value_manager = lucene_str_value_manager + + @staticmethod + def _get_value_type(field_name: str, value: Union[int, str, StrValue], value_type: Optional[str] = None) -> str: # noqa: ARG004 + is_ip_field = field_name and (field_name.endswith(".ip") or field_name.endswith(".address")) + if is_ip_field and value_type != ValueType.regex_value: + return ValueType.ip + + return ValueType.value + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(self._pre_process_values_list(field, value)) + return f"{field}:({values})" + return f"{field}:{self._pre_process_value(field, value)}" + + def less_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f"{field}:<{self._pre_process_value(field, value)}" + + def less_or_equal_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f"{field}:[* TO {self._pre_process_value(field, value)}]" + + def greater_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f"{field}:>{self._pre_process_value(field, value)}" + + def greater_or_equal_modifier(self, field: str, value: Union[int, str, StrValue]) -> str: + return f"{field}:[{self._pre_process_value(field, value)} TO *]" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(self._pre_process_values_list(field, value)) + return f"NOT ({field} = ({values})" + return f"NOT ({field} = {self._pre_process_value(field, value)})" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(f"*{val}*" for val in self._pre_process_values_list(field, value)) + return f"{field}:({values})" + return f"{field}:*{self._pre_process_value(field, value)}*" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(f"*{val}" for val in self._pre_process_values_list(field, value)) + return f"{field}:({values})" + return f"{field}:*{self._pre_process_value(field, value)}" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(f"{val}*" for val in self._pre_process_values_list(field, value)) + return f"{field}:({values})" + return f"{field}:{self._pre_process_value(field, value)}*" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = [] + for val in value: + values.append( + f"/{self._pre_process_value(field, val, value_type=ValueType.regex_value)}/" + if isinstance(val, StrValue) + else f"/{val}/" + ) + return f"{field}:({self.or_token.join(values)})" + + if isinstance(value, StrValue): + return f"{field}:/{self._pre_process_value(field, value, value_type=ValueType.regex_value)}/" + + return f"{field}:/{value}/" + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.keywords(field=field, value=val) for val in value)})" + return f"*{self._pre_process_value(field, value)}*" + + +class LuceneQueryRender(PlatformQueryRender): + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + comment_symbol = "//" + is_single_line_comment = True + + def generate_prefix(self, log_source_signature: LuceneLogSourceSignature, functions_prefix: str = "") -> str: # noqa: ARG002 + return "" diff --git a/uncoder-core/app/translator/platforms/base/lucene/str_value_manager.py b/uncoder-core/app/translator/platforms/base/lucene/str_value_manager.py new file mode 100644 index 00000000..bad77b6f --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/lucene/str_value_manager.py @@ -0,0 +1,99 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import ClassVar, Optional + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.str_value_manager import ( + BaseSpecSymbol, + ReAnySymbol, + ReCaretSymbol, + ReCommaSymbol, + ReHyphenSymbol, + ReLeftCurlyBracket, + ReLeftParenthesis, + ReLeftSquareBracket, + ReOneOrMoreQuantifier, + ReOrOperator, + ReRightCurlyBracket, + ReRightParenthesis, + ReRightSquareBracket, + ReZeroOrMoreQuantifier, + ReZeroOrOneQuantifier, + SingleSymbolWildCard, + StrValue, + StrValueManager, + UnboundLenWildCard, +) +from app.translator.platforms.base.lucene.escape_manager import lucene_escape_manager + +RE_STR_SPEC_SYMBOLS_MAP = { + "?": ReZeroOrOneQuantifier, + "*": ReZeroOrMoreQuantifier, + "+": ReOneOrMoreQuantifier, + ".": ReAnySymbol, + "[": ReLeftSquareBracket, + "]": ReRightSquareBracket, + "(": ReLeftParenthesis, + ")": ReRightParenthesis, + "{": ReLeftCurlyBracket, + "}": ReRightCurlyBracket, + "|": ReOrOperator, + "^": ReCaretSymbol, + ",": ReCommaSymbol, + "-": ReHyphenSymbol, +} + + +class LuceneStrValueManager(StrValueManager): + escape_manager = lucene_escape_manager + str_spec_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = { + "?": SingleSymbolWildCard, + "*": UnboundLenWildCard, + } + re_str_spec_symbols_map = RE_STR_SPEC_SYMBOLS_MAP + + def from_str_to_container( + self, + value: str, + value_type: str = ValueType.value, # noqa: ARG002 + escape_symbol: Optional[str] = None, # noqa: ARG002 + ) -> StrValue: + split = [] + prev_char = None + for char in value: + if char == "\\": + if prev_char == "\\": + split.append(char) + prev_char = None + continue + elif char in self.str_spec_symbols_map: + if prev_char == "\\": + split.append(char) + else: + split.append(self.str_spec_symbols_map[char]()) + else: + split.append(char) + + prev_char = char + + return StrValue(value, self._concat(split)) + + +lucene_str_value_manager = LuceneStrValueManager() diff --git a/uncoder-core/app/translator/platforms/base/lucene/tokenizer.py b/uncoder-core/app/translator/platforms/base/lucene/tokenizer.py new file mode 100644 index 00000000..8be19ffe --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/lucene/tokenizer.py @@ -0,0 +1,145 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import ClassVar, Optional, Union + +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mixins.logic import ANDLogicOperatorMixin +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.keyword import Keyword +from app.translator.core.str_value_manager import StrValue +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.platforms.base.lucene.escape_manager import lucene_escape_manager +from app.translator.platforms.base.lucene.str_value_manager import lucene_str_value_manager +from app.translator.tools.utils import get_match_group + + +class LuceneTokenizer(QueryTokenizer, ANDLogicOperatorMixin): + single_value_operators_map: ClassVar[dict[str, str]] = { + ":>": OperatorType.GT, + ":<": OperatorType.LT, + ":": OperatorType.EQ, + "==": OperatorType.EQ, + } + multi_value_operators_map: ClassVar[dict[str, str]] = {":": OperatorType.EQ} + + field_pattern = r"(?P[a-zA-Z\.\-_]+)" + _num_value_pattern = r"\d+(?:\.\d+)?" + num_value_pattern = rf"(?P<{ValueType.number_value}>{_num_value_pattern})(?=\s+|\)|$)" + double_quotes_value_pattern = ( + rf'"(?P<{ValueType.double_quotes_value}>(?:[:a-zA-Z\*\?0-9=+%#№;\-_/,\'\.$&^@!\(\)\{{\}}\s]|\\\"|\\)*)"\s*' + ) + no_quotes_value_pattern = ( + rf"(?P<{ValueType.no_quotes_value}>(?:[a-zA-Z\*\?0-9=%#№!;_/,\'\.$@|]|\\[*?\"-_=%#№!;,\.$@/\s\\])+)\s*" + ) + re_value_pattern = rf"/(?P<{ValueType.regex_value}>(?:[:a-zA-Z\*\?0-9=+%#№;\\\-_\,\"\'\.$&^@!\(\)\{{\}}\[\]\s?<>|]|\\\/)+)/(?=\s+|\)|$)" # noqa: E501 + gte_value_pattern = rf"\[\s*(?P<{ValueType.greater_than_or_equal}>{_num_value_pattern})\s+TO\s+\*\s*\]" + lte_value_pattern = rf"\[\s*\*\s+TO\s+(?P<{ValueType.less_than_or_equal}>{_num_value_pattern})\s*\]" + range_value_pattern = rf"{gte_value_pattern}|{lte_value_pattern}" + _value_pattern = rf"{num_value_pattern}|{re_value_pattern}|{no_quotes_value_pattern}|{double_quotes_value_pattern}|{range_value_pattern}" # noqa: E501 + keyword_pattern = ( + rf"(?P<{ValueType.no_quotes_value}>(?:[a-zA-Z\*\?0-9=%#№;_/,\'\.$@]|\\[\"\(\)\[\]\{{\}}\:]|\\)+)(?=\s+|\)|$)" + ) + + multi_value_pattern = rf"""\((?P<{ValueType.multi_value}>[:a-zA-Z\"\*0-9=+%#№;\-_\/\\'\,.$&^@!\(\[\]\s|]+)\)""" + multi_value_check_pattern = r"___field___\s*___operator___\s*\(" + multi_value_delimiter_pattern = r"\s+(?:OR|or)\s+" + + escape_manager = lucene_escape_manager + + wildcard_symbol = "*" + + str_value_manager = lucene_str_value_manager + + @staticmethod + def create_field_value(field_name: str, operator: Identifier, value: Union[str, list]) -> FieldValue: + field_name = field_name.replace(".text", "") + field_name = field_name.replace(".keyword", "") + return FieldValue(source_name=field_name, operator=operator, value=value) + + @staticmethod + def clean_multi_value(value: str) -> str: + value = value.replace("\n", "").replace(" ", "") + value = value.strip('"') if value.startswith('"') and value.endswith('"') else value + return value.strip() + + def get_operator_and_value( # noqa: PLR0911 + self, match: re.Match, mapped_operator: str = OperatorType.EQ, operator: Optional[str] = None + ) -> tuple[str, StrValue]: + if (num_value := get_match_group(match, group_name=ValueType.number_value)) is not None: + return mapped_operator, StrValue(num_value, split_value=[num_value]) + + if (re_value := get_match_group(match, group_name=ValueType.regex_value)) is not None: + return OperatorType.REGEX, lucene_str_value_manager.from_re_str_to_container(re_value) + + if (n_q_value := get_match_group(match, group_name=ValueType.no_quotes_value)) is not None: + return mapped_operator, lucene_str_value_manager.from_str_to_container(n_q_value) + + if (d_q_value := get_match_group(match, group_name=ValueType.double_quotes_value)) is not None: + return mapped_operator, lucene_str_value_manager.from_str_to_container(d_q_value) + + if (gte_value := get_match_group(match, group_name=ValueType.greater_than_or_equal)) is not None: + return OperatorType.GTE, StrValue(gte_value, split_value=[gte_value]) + + if (lte_value := get_match_group(match, group_name=ValueType.less_than_or_equal)) is not None: + return OperatorType.LTE, StrValue(lte_value, split_value=[lte_value]) + + return super().get_operator_and_value(match, mapped_operator, operator) + + def group_values_by_operator(self, values: list[str], operator: str) -> dict[str, list[StrValue]]: + mapped_operator = self.map_operator(operator) + result = {} + for value in values: + if value.startswith("/") and value.endswith("/"): + value = lucene_str_value_manager.from_re_str_to_container(value.strip("/")) + result.setdefault(OperatorType.REGEX, []).append(value) + else: + value = lucene_str_value_manager.from_str_to_container(value) + operator, value = self.process_value_wildcards(value, mapped_operator) + result.setdefault(operator, []).append(value) + + return result + + def is_multi_value_flow(self, field_name: str, operator: str, query: str) -> bool: + check_pattern = self.multi_value_check_pattern + check_regex = check_pattern.replace("___field___", field_name).replace("___operator___", operator) + return bool(re.match(check_regex, query)) + + def search_keyword(self, query: str) -> tuple[Keyword, str]: + keyword_search = re.search(self.keyword_pattern, query) + _, value = self.get_operator_and_value(keyword_search) + _, value = self.process_value_wildcards(value) + keyword = Keyword(value=value) + pos = keyword_search.end() + return keyword, query[pos:] + + def _check_field_value_match(self, query: str, white_space_pattern: str = r"\s*") -> bool: + range_value_pattern = f"(?:{self.gte_value_pattern}|{self.lte_value_pattern})" + range_pattern = rf"{self.field_pattern}{white_space_pattern}:\s*{range_value_pattern}" + if re.match(range_pattern, query, re.IGNORECASE): + return True + + return super()._check_field_value_match(query, white_space_pattern=white_space_pattern) + + def tokenize(self, query: str) -> list[QUERY_TOKEN_TYPE]: + tokens = super().tokenize(query=query) + return self.add_and_token_if_missed(tokens=tokens) diff --git a/siem-converter/app/converter/platforms/chronicle/__init__.py b/uncoder-core/app/translator/platforms/base/spl/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/chronicle/__init__.py rename to uncoder-core/app/translator/platforms/base/spl/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/spl/const.py b/uncoder-core/app/translator/platforms/base/spl/const.py new file mode 100644 index 00000000..4faa6b34 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/spl/const.py @@ -0,0 +1,7 @@ +FIELD_PATTERN = r"(?P<___group_name___>[a-zA-Z0-9.\-_{}]+)" +DOUBLE_QUOTES_VALUE_PATTERN = r'"(?P<___group_name___>(?:[:a-zA-Z*0-9=+%#\-_/,;`?~‘○×\'.<>$&^@!\]\[(){}\s]|\\\"|\\)*)"' # noqa: RUF001 +SINGLE_QUOTES_VALUE_PATTERN = r"'(?P<___group_name___>(?:[:a-zA-Z*0-9=+%#\-_/,;`?~‘○×\".<>$&^@!\]\[(){}\s]|\\\'|\\)*)'" # noqa: RUF001 +NO_QUOTES_VALUES_PATTERN = ( + r"(?P<___group_name___>(?:[:a-zA-Z*0-9+%#\-_/,.$&^@!]|\\\s|\\=|\\!=|\\<|\\<=|\\>|\\>=|\\\\)+)" +) +NUM_VALUE_PATTERN = r"(?P<___group_name___>\d+(?:\.\d+)?)" diff --git a/uncoder-core/app/translator/platforms/base/spl/escape_manager.py b/uncoder-core/app/translator/platforms/base/spl/escape_manager.py new file mode 100644 index 00000000..9b7e0154 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/spl/escape_manager.py @@ -0,0 +1,12 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class SplEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = {ValueType.value: [EscapeDetails(pattern=r"([<>=\"'\|\\])")]} + + +spl_escape_manager = SplEscapeManager() diff --git a/uncoder-core/app/translator/platforms/base/spl/functions/__init__.py b/uncoder-core/app/translator/platforms/base/spl/functions/__init__.py new file mode 100644 index 00000000..9dc715f5 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/spl/functions/__init__.py @@ -0,0 +1,48 @@ +import os.path +import re + +from app.translator.core.exceptions.functions import InvalidFunctionSignature, NotSupportedFunctionException +from app.translator.core.functions import PlatformFunctions +from app.translator.core.models.functions.base import ParsedFunctions +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.platforms.base.spl.functions.const import SplFunctionType +from app.translator.platforms.base.spl.functions.manager import SplFunctionsManager, spl_functions_manager + + +class SplFunctions(PlatformFunctions): + dir_path: str = os.path.abspath(os.path.dirname(__file__)) + manager: SplFunctionsManager = spl_functions_manager + + @staticmethod + def prepare_query(query: str) -> str: + if query.startswith(SplFunctionType.search): + query = re.sub(SplFunctionType.search, "", query, 1) + return query + + def parse(self, query: str) -> tuple[str, ParsedFunctions]: + parsed = [] + not_supported = [] + invalid = [] + functions = query.split(self.function_delimiter) + result_query = self.prepare_query(functions[0]) + for func in functions[1:]: + func = func.strip() + split_func = func.split(" ") + func_name, func_body = split_func[0], " ".join(split_func[1:]) + try: + func_parser = self.manager.get_hof_parser(func_name) + parsed.append(func_parser.parse(func_body, func)) + except NotSupportedFunctionException: + not_supported.append(func) + except InvalidFunctionSignature: + invalid.append(func) + + return result_query, ParsedFunctions( + functions=parsed, + not_supported=[self.wrap_function_with_delimiter(func) for func in not_supported], + invalid=invalid, + ) + + @staticmethod + def parse_tstats_func(raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + raise NotSupportedFunctionException diff --git a/uncoder-core/app/translator/platforms/base/spl/functions/const.py b/uncoder-core/app/translator/platforms/base/spl/functions/const.py new file mode 100644 index 00000000..31329dec --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/spl/functions/const.py @@ -0,0 +1,31 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class SplFunctionType(CustomEnum): + avg = "avg" + count = "count" + distinct_count = "distinct_count" + earliest = "earliest" + eval = "eval" + fields = "fields" + latest = "latest" + max = "max" + min = "min" + rename = "rename" + search = "search" + sort = "sort" + stats = "stats" + sum = "sum" + table = "table" + values = "values" + where = "where" + + +class SplFunctionGroupType(CustomEnum): + agg = "agg" + time = "time" + + +class SplSortOrderType(CustomEnum): + asc = "+" + desc = "-" diff --git a/uncoder-core/app/translator/platforms/base/spl/functions/manager.py b/uncoder-core/app/translator/platforms/base/spl/functions/manager.py new file mode 100644 index 00000000..180ea1e2 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/spl/functions/manager.py @@ -0,0 +1,8 @@ +from app.translator.core.functions import PlatformFunctionsManager + + +class SplFunctionsManager(PlatformFunctionsManager): + ... + + +spl_functions_manager = SplFunctionsManager() diff --git a/siem-converter/app/converter/platforms/chronicle/mappings/__init__.py b/uncoder-core/app/translator/platforms/base/spl/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/chronicle/mappings/__init__.py rename to uncoder-core/app/translator/platforms/base/spl/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/spl/parsers/spl.py b/uncoder-core/app/translator/platforms/base/spl/parsers/spl.py new file mode 100644 index 00000000..f56af913 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/spl/parsers/spl.py @@ -0,0 +1,84 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re + +from app.translator.core.models.functions.base import ParsedFunctions +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser +from app.translator.platforms.base.spl.functions import SplFunctions +from app.translator.platforms.base.spl.tokenizer import SplTokenizer + +TSTATS_FUNC = "tstats" + + +class SplQueryParser(PlatformQueryParser): + log_source_pattern = r"^___source_type___\s*=\s*(?:\"(?P[%a-zA-Z_*:0-9\-/]+)\"|(?P[%a-zA-Z_*:0-9\-/]+))(?:\s+(?:and|or)\s+|\s+)?" # noqa: E501 + rule_name_pattern = r"`(?P(?:[:a-zA-Z*0-9=+%#\-_/,;`?~‘\'.<>$&^@!\]\[()\s])*)`" # noqa: RUF001 + log_source_key_types = ("index", "source", "sourcetype", "sourcecategory") + + platform_functions: SplFunctions = None + tokenizer = SplTokenizer() + + wrapped_with_comment_pattern = r"^\s*```(?:|\n|.)*```" + + def _parse_log_sources(self, query: str) -> tuple[dict[str, list[str]], str]: + log_sources = {} + for source_type in self.log_source_key_types: + log_sources.setdefault(source_type, []) + pattern = self.log_source_pattern.replace("___source_type___", source_type) + while search := re.search(pattern, query, flags=re.IGNORECASE): + group_dict = search.groupdict() + value = group_dict.get("d_q_value") or group_dict.get("value") + log_sources.setdefault(source_type, []).append(value) + pos_start = search.start() + pos_end = search.end() + query = query[:pos_start] + query[pos_end:] + query = query.replace("()", "") + + return log_sources, query + + def _parse_query(self, query: str) -> tuple[str, dict[str, list[str]], ParsedFunctions]: + if re.match(self.rule_name_pattern, query): + search = re.search(self.rule_name_pattern, query, flags=re.IGNORECASE) + query = query[: search.start()] + query[search.end() :] + query = query.strip() + log_sources, query = self._parse_log_sources(query) + query, functions = self.platform_functions.parse(query) + return query, log_sources, functions + + @staticmethod + def __is_tstats_query(query: str) -> bool: + return bool(re.match(r"\s*\|\s+tstats", query)) + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + if self.__is_tstats_query(raw_query_container.query): + return self.platform_functions.parse_tstats_func(raw_query_container) + + query, log_sources, functions = self._parse_query(raw_query_container.query) + query_tokens = self.get_query_tokens(query) + query_field_tokens, function_field_tokens, function_field_tokens_map = self.get_field_tokens( + query_tokens, functions.functions + ) + source_mappings = self.get_source_mappings(query_field_tokens + function_field_tokens, log_sources) + meta_info = raw_query_container.meta_info + meta_info.query_fields = query_field_tokens + meta_info.function_fields = function_field_tokens + meta_info.function_fields_map = function_field_tokens_map + meta_info.source_mapping_ids = [source_mapping.source_id for source_mapping in source_mappings] + return TokenizedQueryContainer(tokens=query_tokens, meta_info=meta_info, functions=functions) diff --git a/siem-converter/app/converter/platforms/chronicle/parsers/__init__.py b/uncoder-core/app/translator/platforms/base/spl/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/chronicle/parsers/__init__.py rename to uncoder-core/app/translator/platforms/base/spl/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/spl/renders/spl.py b/uncoder-core/app/translator/platforms/base/spl/renders/spl.py new file mode 100644 index 00000000..c3c36675 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/spl/renders/spl.py @@ -0,0 +1,98 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.core.str_value_manager import StrValue +from app.translator.platforms.base.spl.str_value_manager import spl_str_value_manager + + +class SplFieldValueRender(BaseFieldValueRender): + str_value_manager = spl_str_value_manager + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f'"{value}"' + + def _pre_process_value( + self, + field: str, + value: Union[bool, int, str, StrValue], + value_type: str = ValueType.value, + wrap_str: bool = False, + wrap_int: bool = False, # noqa: ARG002 + ) -> Union[int, str]: + value = super()._pre_process_value(field, value, value_type=value_type, wrap_str=wrap_str) + return self._wrap_str_value(str(value)) if not isinstance(value, str) else value + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.equal_modifier(field=field, value=v) for v in value])})" + return f"{field}={self._pre_process_value(field, value, wrap_str=True)}" + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + return f"{field}<{self._pre_process_value(field, value, wrap_str=True)}" + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f"{field}<={self._pre_process_value(field, value, wrap_str=True)}" + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + return f"{field}>{self._pre_process_value(field, value, wrap_str=True)}" + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f"{field}>={self._pre_process_value(field, value, wrap_str=True)}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + return f"{field}!={self._pre_process_value(field, value, wrap_str=True)}" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.contains_modifier(field=field, value=v) for v in value])})" + return f'{field}="*{self._pre_process_value(field, value)}*"' + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.endswith_modifier(field=field, value=v) for v in value])})" + return f'{field}="*{self._pre_process_value(field, value)}"' + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.startswith_modifier(field=field, value=v) for v in value])})" + return f'{field}="{self._pre_process_value(field, value)}*"' + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" + return f"{self._pre_process_value(field, value, wrap_str=True)}" + + +class SplQueryRender(PlatformQueryRender): + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + comment_symbol = "```" + + def wrap_with_comment(self, value: str) -> str: + return f"{self.comment_symbol} {value} {self.comment_symbol}" diff --git a/uncoder-core/app/translator/platforms/base/spl/str_value_manager.py b/uncoder-core/app/translator/platforms/base/spl/str_value_manager.py new file mode 100644 index 00000000..ef638d6c --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/spl/str_value_manager.py @@ -0,0 +1,61 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2023 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from typing import ClassVar, Optional + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.str_value_manager import BaseSpecSymbol, StrValue, StrValueManager, UnboundLenWildCard +from app.translator.platforms.base.spl.escape_manager import spl_escape_manager + + +class SplStrValueManager(StrValueManager): + escape_manager = spl_escape_manager + str_spec_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = {"*": UnboundLenWildCard} + + def from_str_to_container( + self, + value: str, + value_type: str = ValueType.value, # noqa: ARG002 + escape_symbol: Optional[str] = None, # noqa: ARG002 + ) -> StrValue: + split = [] + prev_char = None + for char in value: + if char == "\\": + if prev_char == "\\": + split.append("\\") + prev_char = None + continue + elif char in self.str_spec_symbols_map: + if prev_char == "\\": + split.append(char) + else: + split.append(self.str_spec_symbols_map[char]()) + elif char in ('"', "=", "|", "<", ">"): + split.append(char) + else: + if prev_char == "\\": + split.append(prev_char) + split.append(char) + + prev_char = char + + return StrValue(self.escape_manager.remove_escape(value), self._concat(split)) + + +spl_str_value_manager = SplStrValueManager() diff --git a/uncoder-core/app/translator/platforms/base/spl/tokenizer.py b/uncoder-core/app/translator/platforms/base/spl/tokenizer.py new file mode 100644 index 00000000..20133239 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/spl/tokenizer.py @@ -0,0 +1,81 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import Any, ClassVar, Optional + +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mixins.logic import ANDLogicOperatorMixin +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.platforms.base.spl.const import DOUBLE_QUOTES_VALUE_PATTERN as D_Q_V_PATTERN +from app.translator.platforms.base.spl.const import FIELD_PATTERN +from app.translator.platforms.base.spl.const import NO_QUOTES_VALUES_PATTERN as NO_Q_V_PATTERN +from app.translator.platforms.base.spl.const import NUM_VALUE_PATTERN as N_V_PATTERN +from app.translator.platforms.base.spl.const import SINGLE_QUOTES_VALUE_PATTERN as S_Q_V_PATTERN +from app.translator.platforms.base.spl.str_value_manager import spl_str_value_manager +from app.translator.tools.utils import get_match_group + + +class SplTokenizer(QueryTokenizer, ANDLogicOperatorMixin): + single_value_operators_map: ClassVar[dict[str, str]] = { + "=": OperatorType.EQ, + "<=": OperatorType.LTE, + "<": OperatorType.LT, + ">=": OperatorType.GTE, + ">": OperatorType.GT, + "!=": OperatorType.NOT_EQ, + } + multi_value_operators_map: ClassVar[dict[str, str]] = {"in": OperatorType.EQ} + + field_pattern = FIELD_PATTERN.replace("___group_name___", "field_name") + num_value_pattern = rf"{N_V_PATTERN.replace('___group_name___', ValueType.number_value)}(?=$|\s|\))" + double_quotes_value_pattern = rf"{D_Q_V_PATTERN.replace('___group_name___', ValueType.double_quotes_value)}\s*" + single_quotes_value_pattern = rf"{S_Q_V_PATTERN.replace('___group_name___', ValueType.single_quotes_value)}\s*" + no_quotes_value_pattern = rf"{NO_Q_V_PATTERN.replace('___group_name___', ValueType.no_quotes_value)}(?=$|\s|\))" + _value_pattern = ( + rf"{num_value_pattern}|{no_quotes_value_pattern}|{double_quotes_value_pattern}|{single_quotes_value_pattern}" + ) + multi_value_pattern = rf"""\((?P<{ValueType.multi_value}>[:a-zA-Z\"\*0-9=+%#\-_\/\\'\,;.$&^@!\{{\}}\(\s]+)\)""" + keyword_pattern = rf"{double_quotes_value_pattern}|{no_quotes_value_pattern}" + + wildcard_symbol = "*" + + str_value_manager = spl_str_value_manager + + def get_operator_and_value( + self, match: re.Match, mapped_operator: str = OperatorType.EQ, operator: Optional[str] = None + ) -> tuple[str, Any]: + if (num_value := get_match_group(match, group_name=ValueType.number_value)) is not None: + return mapped_operator, num_value + + if (no_q_value := get_match_group(match, group_name=ValueType.no_quotes_value)) is not None: + return mapped_operator, self.str_value_manager.from_str_to_container(no_q_value) + + if (d_q_value := get_match_group(match, group_name=ValueType.double_quotes_value)) is not None: + return mapped_operator, self.str_value_manager.from_str_to_container(d_q_value) + + if (s_q_value := get_match_group(match, group_name=ValueType.single_quotes_value)) is not None: + return mapped_operator, self.str_value_manager.from_str_to_container(s_q_value) + + return super().get_operator_and_value(match, mapped_operator, operator) + + def tokenize(self, query: str) -> list[QUERY_TOKEN_TYPE]: + tokens = super().tokenize(query=query) + return self.add_and_token_if_missed(tokens=tokens) diff --git a/siem-converter/app/converter/platforms/chronicle/renders/__init__.py b/uncoder-core/app/translator/platforms/base/sql/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/chronicle/renders/__init__.py rename to uncoder-core/app/translator/platforms/base/sql/__init__.py diff --git a/siem-converter/app/converter/platforms/crowdstrike/__init__.py b/uncoder-core/app/translator/platforms/base/sql/custom_types/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/crowdstrike/__init__.py rename to uncoder-core/app/translator/platforms/base/sql/custom_types/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/sql/custom_types/values.py b/uncoder-core/app/translator/platforms/base/sql/custom_types/values.py new file mode 100644 index 00000000..4edc6215 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/sql/custom_types/values.py @@ -0,0 +1,5 @@ +from app.translator.core.custom_types.values import ValueType + + +class SQLValueType(ValueType): + like_value = "like_value" diff --git a/uncoder-core/app/translator/platforms/base/sql/escape_manager.py b/uncoder-core/app/translator/platforms/base/sql/escape_manager.py new file mode 100644 index 00000000..e542d898 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/sql/escape_manager.py @@ -0,0 +1,19 @@ +from typing import ClassVar + +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails +from app.translator.platforms.base.sql.custom_types.values import SQLValueType + + +class SQLEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + SQLValueType.value: [EscapeDetails(pattern=r"(')", escape_symbols=r"'\1")], + SQLValueType.like_value: [EscapeDetails(pattern=r"(['%_\\])", escape_symbols=r"\\\1")], + SQLValueType.regex_value: [ + EscapeDetails(pattern=r"([$^*+()\[\]{}|.?\-\\])", escape_symbols=r"\\\1"), + EscapeDetails(pattern=r"(')", escape_symbols=r"'\1"), + ], + } + + +sql_escape_manager = SQLEscapeManager() diff --git a/siem-converter/app/converter/platforms/crowdstrike/mappings/__init__.py b/uncoder-core/app/translator/platforms/base/sql/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/crowdstrike/mappings/__init__.py rename to uncoder-core/app/translator/platforms/base/sql/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/sql/parsers/sql.py b/uncoder-core/app/translator/platforms/base/sql/parsers/sql.py new file mode 100644 index 00000000..01be3500 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/sql/parsers/sql.py @@ -0,0 +1,51 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re + +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser +from app.translator.platforms.base.sql.tokenizer import SqlTokenizer + + +class SqlQueryParser(PlatformQueryParser): + tokenizer = SqlTokenizer() + query_delimiter_pattern = r"\sFROM\s\S*\sWHERE\s" + table_pattern = r"\sFROM\s(?P
[a-zA-Z\.\-\*]+)\sWHERE\s" + + wrapped_with_comment_pattern = r"^\s*--.*(?:\n|$)" + + def _parse_query(self, query: str) -> tuple[str, dict[str, list[str]]]: + log_source = {"table": []} + if re.search(self.query_delimiter_pattern, query, flags=re.IGNORECASE): + table_search = re.search(self.table_pattern, query) + table = table_search.group("table") + log_source["table"] = [table] + return re.split(self.query_delimiter_pattern, query, flags=re.IGNORECASE)[1], log_source + + return query, log_source + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + query, log_sources = self._parse_query(raw_query_container.query) + query_tokens = self.get_query_tokens(query) + query_field_tokens, _, _ = self.get_field_tokens(query_tokens) + source_mappings = self.get_source_mappings(query_field_tokens, log_sources) + meta_info = raw_query_container.meta_info + meta_info.query_fields = query_field_tokens + meta_info.source_mapping_ids = [source_mapping.source_id for source_mapping in source_mappings] + return TokenizedQueryContainer(tokens=query_tokens, meta_info=meta_info) diff --git a/siem-converter/app/converter/platforms/crowdstrike/parsers/__init__.py b/uncoder-core/app/translator/platforms/base/sql/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/crowdstrike/parsers/__init__.py rename to uncoder-core/app/translator/platforms/base/sql/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/base/sql/renders/sql.py b/uncoder-core/app/translator/platforms/base/sql/renders/sql.py new file mode 100644 index 00000000..e7178922 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/sql/renders/sql.py @@ -0,0 +1,99 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mapping import LogSourceSignature +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.platforms.base.sql.custom_types.values import SQLValueType +from app.translator.platforms.base.sql.str_value_manager import sql_str_value_manager + + +class SqlFieldValueRender(BaseFieldValueRender): + str_value_manager = sql_str_value_manager + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f"'{value}'" + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.equal_modifier(field=field, value=v) for v in value])})" + return f"{field} = {self._pre_process_value(field, value, wrap_str=True)}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + return f"{field} != {self._pre_process_value(field, value, wrap_str=True)}" + + def less_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + return f"{field} < {self._pre_process_value(field, value, wrap_str=True)}" + + def less_or_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + return f"{field} <= {self._pre_process_value(field, value, wrap_str=True)}" + + def greater_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + return f"{field} > {self._pre_process_value(field, value, wrap_str=True)}" + + def greater_or_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + return f"{field} >= {self._pre_process_value(field, value, wrap_str=True)}" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + + value = f"'%{self._pre_process_value(field, value, value_type=SQLValueType.like_value)}%' escape '\\'" + return f"{field} like {value}" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + + value = f"'%{self._pre_process_value(field, value, value_type=SQLValueType.like_value)}' escape '\\'" + return f"{field} like {value}" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" + + value = f"'{self._pre_process_value(field, value, value_type=SQLValueType.like_value)}%' escape '\\'" + return f"{field} like {value}" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + regex_str = self._pre_process_value(field, value, value_type=ValueType.regex_value, wrap_str=True) + return f"regexp_like({field}, {regex_str})" + + +class SqlQueryRender(PlatformQueryRender): + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + comment_symbol = "--" + is_single_line_comment = True + + def generate_prefix(self, log_source_signature: LogSourceSignature, functions_prefix: str = "") -> str: # noqa: ARG002 + table = str(log_source_signature) if str(log_source_signature) else "eventlog" + return f"SELECT * FROM {table}" + + @staticmethod + def _finalize_search_query(query: str) -> str: + return f"WHERE {query}" if query else "" diff --git a/uncoder-core/app/translator/platforms/base/sql/str_value_manager.py b/uncoder-core/app/translator/platforms/base/sql/str_value_manager.py new file mode 100644 index 00000000..5f47b8be --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/sql/str_value_manager.py @@ -0,0 +1,112 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy +from typing import ClassVar, Optional + +from app.translator.core.str_value_manager import ( + CONTAINER_SPEC_SYMBOLS_MAP, + RE_STR_SPEC_SYMBOLS_MAP, + BaseSpecSymbol, + ReDigitalSymbol, + ReWhiteSpaceSymbol, + ReWordBoundarySymbol, + ReWordSymbol, + SingleSymbolWildCard, + StrValue, + StrValueManager, + UnboundLenWildCard, +) +from app.translator.platforms.base.sql.custom_types.values import SQLValueType +from app.translator.platforms.base.sql.escape_manager import sql_escape_manager + +SQL_CONTAINER_SPEC_SYMBOLS_MAP = copy.copy(CONTAINER_SPEC_SYMBOLS_MAP) +SQL_CONTAINER_SPEC_SYMBOLS_MAP.update({SingleSymbolWildCard: "_", UnboundLenWildCard: "%"}) + + +class SQLStrValueManager(StrValueManager): + escape_manager = sql_escape_manager + container_spec_symbols_map: ClassVar[dict[type[BaseSpecSymbol], str]] = SQL_CONTAINER_SPEC_SYMBOLS_MAP + re_str_alpha_num_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = { + "b": ReWordBoundarySymbol, + "w": ReWordSymbol, + "d": ReDigitalSymbol, + "s": ReWhiteSpaceSymbol, + } + re_str_spec_symbols_map = RE_STR_SPEC_SYMBOLS_MAP + str_spec_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = { + "_": SingleSymbolWildCard, + "%": UnboundLenWildCard, + } + + def from_str_to_container( + self, value: str, value_type: str = SQLValueType.value, escape_symbol: Optional[str] = None + ) -> StrValue: + split = [] + prev_char = None + for char in value: + if escape_symbol and char == escape_symbol: + if prev_char == escape_symbol: + split.append(char) + prev_char = None + continue + prev_char = char + continue + if not escape_symbol and char == "'": + if prev_char == "'": + split.append(char) + prev_char = None + continue + elif char in ("'", "_", "%") and value_type == SQLValueType.like_value: + if escape_symbol and prev_char == escape_symbol: + split.append(char) + elif char in self.str_spec_symbols_map: + split.append(self.str_spec_symbols_map[char]()) + else: + split.append(char) + + prev_char = char + + return StrValue(value, self._concat(split)) + + def from_re_str_to_container(self, value: str) -> StrValue: + value = value.replace("''", "'") + return super().from_re_str_to_container(value) + + def from_container_to_str(self, container: StrValue, value_type: str = SQLValueType.value) -> str: + result = "" + for el in container.split_value: + if isinstance(el, str): + result += self.escape_manager.escape(el, value_type) + elif isinstance(el, BaseSpecSymbol): + if value_type == SQLValueType.regex_value: + if isinstance(el, SingleSymbolWildCard): + result += "." + continue + if isinstance(el, UnboundLenWildCard): + result += ".*" + continue + + if pattern := self.container_spec_symbols_map.get(type(el)): + result += pattern + + return result + + +sql_str_value_manager = SQLStrValueManager() diff --git a/uncoder-core/app/translator/platforms/base/sql/tokenizer.py b/uncoder-core/app/translator/platforms/base/sql/tokenizer.py new file mode 100644 index 00000000..fe92c8f6 --- /dev/null +++ b/uncoder-core/app/translator/platforms/base/sql/tokenizer.py @@ -0,0 +1,103 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import Any, ClassVar, Optional, Union + +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.function_value import FunctionValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.models.query_tokens.keyword import Keyword +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.platforms.base.sql.custom_types.values import SQLValueType +from app.translator.platforms.base.sql.str_value_manager import sql_str_value_manager +from app.translator.tools.utils import get_match_group + +_ESCAPE_SYMBOL_GROUP_NAME = "escape_symbol" + + +class SqlTokenizer(QueryTokenizer): + single_value_operators_map: ClassVar[dict[str, str]] = { + "=": OperatorType.EQ, + "<=": OperatorType.LTE, + "<": OperatorType.LT, + ">=": OperatorType.GTE, + ">": OperatorType.GT, + "!=": OperatorType.NOT_EQ, + "<>": OperatorType.NOT_EQ, + "like": OperatorType.EQ, + } + multi_value_operators_map: ClassVar[dict[str, str]] = {"in": OperatorType.EQ} + + field_pattern = r'(?P"[a-zA-Z\._\-\s]+"|[a-zA-Z\._\-]+)' + num_value_pattern = rf"(?P<{ValueType.number_value}>\d+(?:\.\d+)*)\s*" + bool_value_pattern = rf"(?P<{ValueType.bool_value}>true|false)\s*" + single_quotes_value_pattern = rf"""'(?P<{ValueType.single_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-\/,_".$&^@!\(\)\{{\}}\s]|''|\\\'|\\\%|\\\_|\\\\|\\)*)'(?:\s+escape\s+'(?P<{_ESCAPE_SYMBOL_GROUP_NAME}>.)')?""" # noqa: E501 + _value_pattern = rf"{num_value_pattern}|{bool_value_pattern}|{single_quotes_value_pattern}" + multi_value_pattern = rf"""\((?P<{ValueType.multi_value}>\d+(?:,\s*\d+)*|'(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{{\}}\s]|'')*'(?:,\s*'(?:[:a-zA-Z\*0-9=+%#\-\/\\,_".$&^@!\(\)\{{\}}\s]|'')*')*)\)""" # noqa: E501 + re_field_value_pattern = rf"""regexp_like\({field_pattern},\s*'(?P<{ValueType.regex_value}>(?:[:a-zA-Z\*\?0-9=+%#№;\-_,"\.$&^@!\{{\}}\[\]\s?<>|]|\\\'|\\)+)'\)""" # noqa: E501 + + wildcard_symbol = "%" + + str_value_manager = sql_str_value_manager + + @staticmethod + def should_process_value_wildcards(operator: Optional[str]) -> bool: + return operator and operator.lower() in ("like",) + + def get_operator_and_value( + self, match: re.Match, mapped_operator: str = OperatorType.EQ, operator: Optional[str] = None + ) -> tuple[str, Any]: + if (num_value := get_match_group(match, group_name=ValueType.number_value)) is not None: + return mapped_operator, num_value + + if (bool_value := get_match_group(match, group_name=ValueType.bool_value)) is not None: + return mapped_operator, bool_value + + if (s_q_value := get_match_group(match, group_name=ValueType.single_quotes_value)) is not None: + escape_symbol = get_match_group(match, group_name=_ESCAPE_SYMBOL_GROUP_NAME) + should_process_value_wildcards = self.should_process_value_wildcards(operator) + value_type = SQLValueType.like_value if should_process_value_wildcards else SQLValueType.value + return mapped_operator, self.str_value_manager.from_str_to_container( + s_q_value, value_type=value_type, escape_symbol=escape_symbol + ) + + return super().get_operator_and_value(match, mapped_operator, operator) + + @staticmethod + def create_field_value(field_name: str, operator: Identifier, value: Union[str, list]) -> FieldValue: + field_name = field_name.strip('"') + return FieldValue(source_name=field_name, operator=operator, value=value) + + def _search_re_field_value(self, query: str) -> Optional[tuple[FieldValue, str]]: + if match := re.match(self.re_field_value_pattern, query, re.IGNORECASE): + group_dict = match.groupdict() + field_name = group_dict["field_name"] + value = self.str_value_manager.from_re_str_to_container(group_dict[ValueType.regex_value]) + operator = Identifier(token_type=OperatorType.REGEX) + return self.create_field_value(field_name, operator, value), query[match.end() :] + + def _get_next_token( + self, query: str + ) -> tuple[Union[FieldValue, FunctionValue, Keyword, Identifier, list[Union[FieldValue, Identifier]]], str]: + query = query.strip("\n").strip(" ").strip("\n") + if search_result := self._search_re_field_value(query): + return search_result + return super()._get_next_token(query) diff --git a/uncoder-core/app/translator/platforms/carbonblack/__init__.py b/uncoder-core/app/translator/platforms/carbonblack/__init__.py new file mode 100644 index 00000000..ebc8a99c --- /dev/null +++ b/uncoder-core/app/translator/platforms/carbonblack/__init__.py @@ -0,0 +1,2 @@ +from app.translator.platforms.carbonblack.renders.carbonblack import CarbonBlackQueryRender # noqa: F401 +from app.translator.platforms.carbonblack.renders.carbonblack_cti import CarbonBlackCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/carbonblack/const.py b/uncoder-core/app/translator/platforms/carbonblack/const.py new file mode 100644 index 00000000..4e7a6afe --- /dev/null +++ b/uncoder-core/app/translator/platforms/carbonblack/const.py @@ -0,0 +1,24 @@ +from app.translator.core.models.platform_details import PlatformDetails + +CARBON_BLACK_QUERY_DETAILS = { + "platform_id": "carbonblack", + "name": "Carbon Black Cloud", + "group_name": "VMware Carbon Black", + "group_id": "carbonblack-pack", + "platform_name": "Query (Cloud)", +} + + +DEFAULT_CARBONBLACK_CTI_MAPPING = { + "SourceIP": "netconn_local_ipv4", + "DestinationIP": "netconn_ipv4", + "Domain": "netconn_domain", + "URL": "netconn_domain", + "HashMd5": "hash", + "HashSha256": "hash", + "Files": "filemod_name", + "Emails": "process_username", +} + + +carbonblack_query_details = PlatformDetails(**CARBON_BLACK_QUERY_DETAILS) diff --git a/uncoder-core/app/translator/platforms/carbonblack/escape_manager.py b/uncoder-core/app/translator/platforms/carbonblack/escape_manager.py new file mode 100644 index 00000000..5fd8662c --- /dev/null +++ b/uncoder-core/app/translator/platforms/carbonblack/escape_manager.py @@ -0,0 +1,20 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class CarbonBlackEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [ + EscapeDetails( + pattern='([\s+\\-=&?!|(){}.\\[\\]^"~:/]|(?", + ) + ], + ValueType.regex_value: [EscapeDetails(pattern=r"([$^*+()\[\]{}|.?\-\\])", escape_symbols=r"\\\1")], + } + + +carbon_black_escape_manager = CarbonBlackEscapeManager() diff --git a/uncoder-core/app/translator/platforms/carbonblack/mapping.py b/uncoder-core/app/translator/platforms/carbonblack/mapping.py new file mode 100644 index 00000000..b31384b9 --- /dev/null +++ b/uncoder-core/app/translator/platforms/carbonblack/mapping.py @@ -0,0 +1,18 @@ +from app.translator.core.mapping import BaseStrictLogSourcesPlatformMappings, LogSourceSignature +from app.translator.platforms.carbonblack.const import carbonblack_query_details + + +class CarbonBlackLogSourceSignature(LogSourceSignature): + def is_suitable(self) -> bool: + return True + + def __str__(self) -> str: + return "" + + +class CarbonBlackMappings(BaseStrictLogSourcesPlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> CarbonBlackLogSourceSignature: + ... + + +carbonblack_query_mappings = CarbonBlackMappings(platform_dir="carbonblack", platform_details=carbonblack_query_details) diff --git a/siem-converter/app/converter/platforms/crowdstrike/renders/__init__.py b/uncoder-core/app/translator/platforms/carbonblack/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/crowdstrike/renders/__init__.py rename to uncoder-core/app/translator/platforms/carbonblack/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/carbonblack/renders/carbonblack.py b/uncoder-core/app/translator/platforms/carbonblack/renders/carbonblack.py new file mode 100644 index 00000000..df366c3e --- /dev/null +++ b/uncoder-core/app/translator/platforms/carbonblack/renders/carbonblack.py @@ -0,0 +1,103 @@ +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.managers import render_manager +from app.translator.platforms.carbonblack.const import carbonblack_query_details +from app.translator.platforms.carbonblack.mapping import CarbonBlackMappings, carbonblack_query_mappings +from app.translator.platforms.carbonblack.str_value_manager import ( + CarbonBlackStrValueManager, + carbon_black_str_value_manager, +) + + +class CarbonBlackFieldValueRender(BaseFieldValueRender): + details: PlatformDetails = carbonblack_query_details + str_value_manager: CarbonBlackStrValueManager = carbon_black_str_value_manager + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f'"{value}"' + + @staticmethod + def _wrap_int_value(value: int) -> str: + return f'"{value}"' + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.equal_modifier(field=field, value=v) for v in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{field}:{value}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = [ + self._pre_process_value(field, val, value_type=ValueType.value, wrap_str=True, wrap_int=True) + for val in value + ] + return f"(NOT {field}:({self.or_token.join(values)})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"(NOT {field}:{self.apply_value(value)})" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join( + [f"*{self._pre_process_value(field, val, value_type=ValueType.value)}*" for val in value] + ) + return f"{field}:({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value) + return f"{field}:*{value}*" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join( + [f"*{self._pre_process_value(field, val, value_type=ValueType.value)}" for val in value] + ) + return f"{field}:({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value) + return f"{field}:*{value}" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join( + [f"{self._pre_process_value(field, val, value_type=ValueType.value)}*" for val in value] + ) + return f"{field}:({values}" + value = self._pre_process_value(field, value, value_type=ValueType.value) + return f"{field}:{value}*" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.regex_value) + return f"{field}:/{value}/" + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.value) + return f"(*{value}*)" + + def is_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.is_none(field=field, value=v) for v in value)})" + return f"NOT _exists_:{field}" + + def is_not_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.is_not_none(field=field, value=v) for v in value)})" + return f"_exists_:{field}" + + +@render_manager.register +class CarbonBlackQueryRender(PlatformQueryRender): + details: PlatformDetails = carbonblack_query_details + mappings: CarbonBlackMappings = carbonblack_query_mappings + + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + comment_symbol = "//" + + field_value_render = CarbonBlackFieldValueRender(or_token=or_token) diff --git a/uncoder-core/app/translator/platforms/carbonblack/renders/carbonblack_cti.py b/uncoder-core/app/translator/platforms/carbonblack/renders/carbonblack_cti.py new file mode 100644 index 00000000..154ee0b5 --- /dev/null +++ b/uncoder-core/app/translator/platforms/carbonblack/renders/carbonblack_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.carbonblack.const import DEFAULT_CARBONBLACK_CTI_MAPPING, carbonblack_query_details + + +@render_cti_manager.register +class CarbonBlackCTI(RenderCTI): + details: PlatformDetails = carbonblack_query_details + + field_value_template: str = "{key}:{value}" + or_operator: str = " OR " + group_or_operator: str = " OR " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_CARBONBLACK_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/carbonblack/str_value_manager.py b/uncoder-core/app/translator/platforms/carbonblack/str_value_manager.py new file mode 100644 index 00000000..0f675093 --- /dev/null +++ b/uncoder-core/app/translator/platforms/carbonblack/str_value_manager.py @@ -0,0 +1,38 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from typing import ClassVar + +from app.translator.core.str_value_manager import ( + BaseSpecSymbol, + SingleSymbolWildCard, + StrValueManager, + UnboundLenWildCard, +) +from app.translator.platforms.carbonblack.escape_manager import CarbonBlackEscapeManager, carbon_black_escape_manager + + +class CarbonBlackStrValueManager(StrValueManager): + escape_manager: CarbonBlackEscapeManager = carbon_black_escape_manager + str_spec_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = { + "?": SingleSymbolWildCard, + "*": UnboundLenWildCard, + } + + +carbon_black_str_value_manager = CarbonBlackStrValueManager() diff --git a/uncoder-core/app/translator/platforms/chronicle/__init__.py b/uncoder-core/app/translator/platforms/chronicle/__init__.py new file mode 100644 index 00000000..700cd191 --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/__init__.py @@ -0,0 +1,5 @@ +from app.translator.platforms.chronicle.parsers.chronicle import ChronicleQueryParser # noqa: F401 +from app.translator.platforms.chronicle.parsers.chronicle_rule import ChronicleRuleParser # noqa: F401 +from app.translator.platforms.chronicle.renders.chronicle import ChronicleQueryRender # noqa: F401 +from app.translator.platforms.chronicle.renders.chronicle_cti import ChronicleQueryCTI # noqa: F401 +from app.translator.platforms.chronicle.renders.chronicle_rule import ChronicleSecurityRuleRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/chronicle/const.py b/uncoder-core/app/translator/platforms/chronicle/const.py new file mode 100644 index 00000000..5bb4363c --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/const.py @@ -0,0 +1,53 @@ +from app.translator.core.models.platform_details import PlatformDetails + +DEFAULT_CHRONICLE_SECURITY_RULE = """rule { + meta: + author = "" + description = "" + license = "" + version = "0.01" + rule_id = "" + status = "" + tags = "" + falsepositives = "" + severity = "" + created = "" + + events: + + + condition: + $e +}""" + +PLATFORM_DETAILS = {"group_id": "chronicle-pack", "group_name": "Google SecOps", "alt_platform_name": "UDM"} + +CHRONICLE_QUERY_DETAILS = { + "platform_id": "chronicle-yaral-query", + "name": "Google SecOps Query", + "platform_name": "Query (UDM)", + **PLATFORM_DETAILS, +} + +CHRONICLE_RULE_DETAILS = { + "platform_id": "chronicle-yaral-rule", + "name": "Google SecOps Rule", + "platform_name": "Rule (YARA-L)", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +DEFAULT_CHRONICLE_CTI_MAPPING = { + "DestinationIP": "target.ip", + "SourceIP": "principal.ip", + "HashSha256": "target.file.sha256", + "HashMd5": "target.file.md5", + "Emails": "network.email.from", + "Domain": "target.hostname", + "HashSha1": "target.file.sha1", + "Files": "target.file.full_path", + "URL": "target.url", +} + +chronicle_query_details = PlatformDetails(**CHRONICLE_QUERY_DETAILS) +chronicle_rule_details = PlatformDetails(**CHRONICLE_RULE_DETAILS) diff --git a/uncoder-core/app/translator/platforms/chronicle/escape_manager.py b/uncoder-core/app/translator/platforms/chronicle/escape_manager.py new file mode 100644 index 00000000..59d017ec --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/escape_manager.py @@ -0,0 +1,15 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class ChronicleEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [EscapeDetails(pattern='([\\\\|"])')], + ValueType.regex_value: [EscapeDetails(pattern='([\\\\|/(")\\[\\]{}.^$+<>!?])')], + } + + +chronicle_escape_manager = ChronicleEscapeManager() diff --git a/uncoder-core/app/translator/platforms/chronicle/mapping.py b/uncoder-core/app/translator/platforms/chronicle/mapping.py new file mode 100644 index 00000000..239f9692 --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/mapping.py @@ -0,0 +1,21 @@ +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature +from app.translator.platforms.chronicle.const import chronicle_query_details, chronicle_rule_details + + +class ChronicleLogSourceSignature(LogSourceSignature): + def is_suitable(self) -> bool: + return True + + def __str__(self) -> str: + return "" + + +class ChronicleMappings(BasePlatformMappings): + is_strict_mapping = True + + def prepare_log_source_signature(self, mapping: dict) -> ChronicleLogSourceSignature: + ... + + +chronicle_query_mappings = ChronicleMappings(platform_dir="chronicle", platform_details=chronicle_query_details) +chronicle_rule_mappings = ChronicleMappings(platform_dir="chronicle", platform_details=chronicle_rule_details) diff --git a/siem-converter/app/converter/platforms/elasticsearch/__init__.py b/uncoder-core/app/translator/platforms/chronicle/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/elasticsearch/__init__.py rename to uncoder-core/app/translator/platforms/chronicle/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/chronicle/parsers/chronicle.py b/uncoder-core/app/translator/platforms/chronicle/parsers/chronicle.py new file mode 100644 index 00000000..0cc1af82 --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/parsers/chronicle.py @@ -0,0 +1,43 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser +from app.translator.managers import parser_manager +from app.translator.platforms.chronicle.const import chronicle_query_details +from app.translator.platforms.chronicle.mapping import ChronicleMappings, chronicle_query_mappings +from app.translator.platforms.chronicle.tokenizer import ChronicleQueryTokenizer + + +@parser_manager.register_supported_by_roota +class ChronicleQueryParser(PlatformQueryParser): + mappings: ChronicleMappings = chronicle_query_mappings + tokenizer: ChronicleQueryTokenizer = ChronicleQueryTokenizer() + details: PlatformDetails = chronicle_query_details + + wrapped_with_comment_pattern = r"^\s*//.*(?:\n|$)" + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + query_tokens = self.get_query_tokens(raw_query_container.query) + query_field_tokens, _, _ = self.get_field_tokens(query_tokens) + source_mappings = self.get_source_mappings(query_field_tokens, {}) + meta_info = raw_query_container.meta_info + meta_info.query_fields = query_field_tokens + meta_info.source_mapping_ids = [source_mapping.source_id for source_mapping in source_mappings] + return TokenizedQueryContainer(tokens=query_tokens, meta_info=meta_info) diff --git a/uncoder-core/app/translator/platforms/chronicle/parsers/chronicle_rule.py b/uncoder-core/app/translator/platforms/chronicle/parsers/chronicle_rule.py new file mode 100644 index 00000000..1c923aaf --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/parsers/chronicle_rule.py @@ -0,0 +1,116 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re + +from app.translator.core.exceptions.parser import TokenizerGeneralException +from app.translator.core.mitre import MitreConfig, MitreInfoContainer +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, RawQueryContainer +from app.translator.managers import parser_manager +from app.translator.platforms.chronicle.const import chronicle_rule_details +from app.translator.platforms.chronicle.mapping import ChronicleMappings, chronicle_rule_mappings +from app.translator.platforms.chronicle.parsers.chronicle import ChronicleQueryParser +from app.translator.platforms.chronicle.tokenizer import ChronicleRuleTokenizer + + +@parser_manager.register +class ChronicleRuleParser(ChronicleQueryParser): + details: PlatformDetails = chronicle_rule_details + rule_name_pattern = r"rule\s+(?P[a-zA-Z0-9_]+)\s+{" + meta_info_pattern = r"meta:\n(?P[a-zA-Z0-9_\\\.*,>–<—~#$’`:;%+^\|?!@\s\"/=\-&'\(\)\[\]]+)\n\s+events:" # noqa: RUF001 + rule_pattern = r"events:\n\s*(?P[a-zA-Z\w0-9_%{}\|\.,!#^><:~\s\"\/=+?\-–&;$()`\*@\[\]'\\]+)\n\s+condition:" # noqa: RUF001 + event_name_pattern = r"condition:\n\s*(?P\$[a-zA-Z_0-9]+)\n" + mappings: ChronicleMappings = chronicle_rule_mappings + tokenizer = ChronicleRuleTokenizer() + mitre_config: MitreConfig = MitreConfig() + + def __parse_rule(self, rule: str) -> tuple[str, str, str]: + if (rule_name_search := re.search(self.rule_name_pattern, rule)) is None: + raise TokenizerGeneralException(error="Rule name couldn't be found in rule.") + rule_name = rule_name_search.group("rule_name") + + if (meta_info_search := re.search(self.meta_info_pattern, rule)) is None: + raise TokenizerGeneralException(error="Rule meta info couldn't be found in rule.") + meta_info = meta_info_search.group("meta_info") + + if (query_search := re.search(self.rule_pattern, rule)) is None: + raise TokenizerGeneralException(error="Query couldn't be found in rule.") + query = query_search.group("query") + + if (event_name_search := re.search(self.event_name_pattern, rule)) is None: + raise TokenizerGeneralException(error="Event name couldn't be found in rule.") + event_name = event_name_search.group("event_name") + + query = query.replace(f"{event_name}.", "") + return query.strip(" ").strip("\n"), rule_name, meta_info + + @staticmethod + def __prepare_title(name: str) -> str: + return " ".join(name.split("_")).title() + + @staticmethod + def __parse_meta_info(meta_info_str: str) -> dict: + parsed_meta_info = {} + + for info in meta_info_str.strip(" ").strip("\n").split("\n"): + key, value = info.split(" = ") + key = key.strip(" ") + if key in ("description", "license", "version", "sigma_id", "status", "severity", "created"): + parsed_meta_info[key] = value.strip(" ").strip('"') + elif key == "reference": + parsed_meta_info[key] = [value.strip(" ").strip('"')] + elif key in ("tags", "author"): + parsed_meta_info[key] = [i.strip(" ").strip('"') for i in value.split(",")] + + return parsed_meta_info + + def parse_mitre_attack_from_tags(self, tags: list) -> MitreInfoContainer: + parsed_techniques = [] + parsed_tactics = [] + for tag in set(tags): + tag = tag.lower() + if tag.startswith("attack."): + tag = tag[7::] + if tag[-1].isdigit(): + parsed_techniques.append(tag) + else: + parsed_tactics.append(tag) + return self.mitre_config.get_mitre_info(tactics=parsed_tactics, techniques=parsed_techniques) + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + query, rule_name, meta_info_str = self.__parse_rule(text) + parsed_meta_info = self.__parse_meta_info(meta_info_str) + + return RawQueryContainer( + query=query, + language=language, + meta_info=MetaInfoContainer( + id_=parsed_meta_info.get("sigma_id"), + title=self.__prepare_title(rule_name), + description=parsed_meta_info.get("description"), + author=parsed_meta_info.get("author"), + date=parsed_meta_info.get("created"), + license_=parsed_meta_info.get("license"), + severity=parsed_meta_info.get("severity"), + references=parsed_meta_info.get("reference"), + tags=parsed_meta_info.get("tags"), + status=parsed_meta_info.get("status"), + mitre_attack=self.parse_mitre_attack_from_tags(parsed_meta_info.get("tags") or []), + ), + ) diff --git a/siem-converter/app/converter/platforms/elasticsearch/mappings/__init__.py b/uncoder-core/app/translator/platforms/chronicle/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/elasticsearch/mappings/__init__.py rename to uncoder-core/app/translator/platforms/chronicle/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/chronicle/renders/chronicle.py b/uncoder-core/app/translator/platforms/chronicle/renders/chronicle.py new file mode 100644 index 00000000..50fd5cbf --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/renders/chronicle.py @@ -0,0 +1,108 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.managers import render_manager +from app.translator.platforms.chronicle.const import chronicle_query_details +from app.translator.platforms.chronicle.escape_manager import chronicle_escape_manager +from app.translator.platforms.chronicle.mapping import ChronicleMappings, chronicle_query_mappings + + +class ChronicleFieldValueRender(BaseFieldValueRender): + details: PlatformDetails = chronicle_query_details + escape_manager = chronicle_escape_manager + + def apply_value(self, value: Union[str, int], value_type: str = ValueType.value) -> Union[str, int]: + if isinstance(value, str): + if "*" in value: + return self.apply_asterisk_value(value) + value = self.clean_str_value(value) + return super().apply_value(value, value_type) + + def apply_asterisk_value(self, value: str) -> str: + value = value.replace(r"\\*", "*") + updated_value = super().apply_value(value, ValueType.regex_value) + return updated_value.replace(".*", "*").replace("*", ".*") + + @staticmethod + def clean_str_value(value: str) -> str: + if value.endswith("/"): + value = value.rstrip("/") + return value + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.equal_modifier(field=field, value=v) for v in value)})" + return f'{field} = "{self.apply_value(value)}" nocase' + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{field} < "{self.apply_value(value)}" nocase' + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{field} <= "{self.apply_value(value)}" nocase' + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{field} > "{self.apply_value(value)}" nocase' + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{field} >= "{self.apply_value(value)}" nocase' + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + return f'{field} != "{self.apply_value(value)}" nocase' + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + return f"{field} = /.*{self.apply_value(value)}.*/ nocase" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + return f"{field} = /.*{self.apply_value(value)}$/ nocase" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" + return f"{field} = /^{self.apply_value(value)}.*/ nocase" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + return f"{field} = /{self.apply_asterisk_value(value)}/ nocase" + + +@render_manager.register +class ChronicleQueryRender(PlatformQueryRender): + details: PlatformDetails = chronicle_query_details + mappings: ChronicleMappings = chronicle_query_mappings + + or_token = "or" + and_token = "and" + not_token = "not" + + field_value_render = ChronicleFieldValueRender(or_token=or_token) + comment_symbol = "//" + is_single_line_comment = True diff --git a/uncoder-core/app/translator/platforms/chronicle/renders/chronicle_cti.py b/uncoder-core/app/translator/platforms/chronicle/renders/chronicle_cti.py new file mode 100644 index 00000000..3d5d15ea --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/renders/chronicle_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.chronicle.const import DEFAULT_CHRONICLE_CTI_MAPPING, chronicle_query_details + + +@render_cti_manager.register +class ChronicleQueryCTI(RenderCTI): + details: PlatformDetails = chronicle_query_details + + field_value_template: str = '{key} = "{value}"' + or_operator: str = " or " + group_or_operator: str = " or " + or_group: str = "{or_group}" + result_join: str = "" + final_result_for_many: str = "{result}\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_CHRONICLE_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/chronicle/renders/chronicle_rule.py b/uncoder-core/app/translator/platforms/chronicle/renders/chronicle_rule.py new file mode 100644 index 00000000..fc9b0dcf --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/renders/chronicle_rule.py @@ -0,0 +1,132 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import re +from typing import Optional, Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer +from app.translator.managers import render_manager +from app.translator.platforms.chronicle.const import DEFAULT_CHRONICLE_SECURITY_RULE, chronicle_rule_details +from app.translator.platforms.chronicle.mapping import ChronicleMappings, chronicle_rule_mappings +from app.translator.platforms.chronicle.renders.chronicle import ChronicleFieldValueRender, ChronicleQueryRender + +_AUTOGENERATED_TEMPLATE = "Autogenerated Chronicle Security rule." + + +class ChronicleRuleFieldValueRender(ChronicleFieldValueRender): + details: PlatformDetails = chronicle_rule_details + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.equal_modifier(field=field, value=v) for v in value)})" + return f'{self.apply_field(field)} = "{self.apply_value(value)}"' + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{self.apply_field(field)} < "{self.apply_value(value)}"' + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{self.apply_field(field)} <= "{self.apply_value(value)}"' + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{self.apply_field(field)} > "{self.apply_value(value)}"' + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{self.apply_field(field)} >= "{self.apply_value(value)}"' + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + return f'{self.apply_field(field)} != "{self.apply_value(value)}"' + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + return f"re.regex({self.apply_field(field)}, `.*{self.apply_value(value)}.*`)" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + return f"re.regex({self.apply_field(field)}, `.*{self.apply_value(value)}`)" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" + return f"re.regex({self.apply_field(field)}, `{self.apply_value(value)}.*`)" + + @staticmethod + def apply_field(field: str) -> str: + return f"$e.{field}" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + return f"re.regex({self.apply_field(field)}, `{self.apply_asterisk_value(value)}`)" + + +@render_manager.register +class ChronicleSecurityRuleRender(ChronicleQueryRender): + details: PlatformDetails = chronicle_rule_details + mappings: ChronicleMappings = chronicle_rule_mappings + or_token = "or" + field_value_render = ChronicleRuleFieldValueRender(or_token=or_token) + + @staticmethod + def prepare_title(title: str) -> str: + if not title: + return title + new_title = re.sub(re.compile('[()*‘’:;+!,\\[\\].?`"-/]'), "", title.lower()) # noqa: RUF001 + new_title = re.sub(re.compile("\\s"), "_", new_title.lower()) + index = 0 + for i, title_char in enumerate(new_title): + if not title_char.isdigit(): + index = i + break + new_title = new_title[index:] + return new_title.strip("_") + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, # , + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = DEFAULT_CHRONICLE_SECURITY_RULE.replace("", query) + rule = rule.replace("", self.prepare_title(meta_info.title) or _AUTOGENERATED_TEMPLATE) + description = meta_info.description or _AUTOGENERATED_TEMPLATE + rule = rule.replace("", ", ".join(meta_info.author)) + rule = rule.replace("", description) + rule = rule.replace("", meta_info.license) + rule = rule.replace("", meta_info.id) + rule = rule.replace("", meta_info.severity) + rule = rule.replace("", meta_info.status) + rule = rule.replace("", ", ".join(meta_info.false_positives)) + rule = rule.replace("", ", ".join(meta_info.tags)) + rule = rule.replace("", str(meta_info.date)) + rule = self.wrap_with_unmapped_fields(rule, unmapped_fields) + return self.wrap_with_not_supported_functions(rule, not_supported_functions) diff --git a/uncoder-core/app/translator/platforms/chronicle/tokenizer.py b/uncoder-core/app/translator/platforms/chronicle/tokenizer.py new file mode 100644 index 00000000..a0943952 --- /dev/null +++ b/uncoder-core/app/translator/platforms/chronicle/tokenizer.py @@ -0,0 +1,119 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import Any, ClassVar, Optional + +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.platforms.chronicle.escape_manager import chronicle_escape_manager +from app.translator.tools.utils import get_match_group + + +class ChronicleQueryTokenizer(QueryTokenizer): + single_value_operators_map: ClassVar[dict[str, str]] = { + "=": OperatorType.EQ, + "<=": OperatorType.LTE, + "<": OperatorType.LT, + ">=": OperatorType.GTE, + ">": OperatorType.GT, + "!=": OperatorType.NOT_EQ, + } + + field_pattern = r"(?P[a-zA-Z0-9\._]+)" + num_value_pattern = rf"(?P<{ValueType.number_value}>\d+(?:\.\d+)*)\s*" + bool_value_pattern = rf"(?P<{ValueType.bool_value}>true|false)\s*" + double_quotes_value_pattern = rf'"(?P<{ValueType.double_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{{\}}\s]|\\\"|\\\\)*)"\s*(?:nocase)?' # noqa: E501 + re_value_pattern = ( + rf"/(?P<{ValueType.regex_value}>(?:\\\/|[:a-zA-Z\*0-9=+%#\\\-_\,\"\'\.$&^@!\(\)\{{\}}\s?])+)/\s*(?:nocase)?" + ) + _value_pattern = rf"{num_value_pattern}|{bool_value_pattern}|{double_quotes_value_pattern}|{re_value_pattern}" + escape_manager = chronicle_escape_manager + + wildcard_symbol = ".*" + + def get_operator_and_value( + self, match: re.Match, mapped_operator: str = OperatorType.EQ, operator: Optional[str] = None + ) -> tuple[str, Any]: + if (num_value := get_match_group(match, group_name=ValueType.number_value)) is not None: + return mapped_operator, num_value + + if (bool_value := get_match_group(match, group_name=ValueType.bool_value)) is not None: + return mapped_operator, bool_value + + if (d_q_value := get_match_group(match, group_name=ValueType.double_quotes_value)) is not None: + return mapped_operator, self.escape_manager.remove_escape(d_q_value) + + if (re_value := get_match_group(match, group_name=ValueType.regex_value)) is not None: + return OperatorType.REGEX, re_value + + return super().get_operator_and_value(match, mapped_operator, operator) + + def escape_field_name(self, field_name: str) -> str: + symbols_to_check = [".", "_", "$"] + for symbol in symbols_to_check: + field_name = field_name.replace(symbol, "\\" + symbol) + return field_name + + +class ChronicleRuleTokenizer(ChronicleQueryTokenizer): + field_pattern = r"(?P[$a-zA-Z0-9\._]+)" + regex_field_regex = r"re\.regex\((?P[$a-zA-Z\._]+)," + + double_quotes_value_pattern = ( + rf'"(?P<{ValueType.double_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{{\}}\s]|\\\"|\\\\)*)"' + ) + back_quotes_value_pattern = ( + rf"`(?P<{ValueType.back_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-_/,\'\"\\\.$&^@!\(\)\{{\}}\s])*)`" + ) + regex_value_pattern = rf"(?:{double_quotes_value_pattern}|{back_quotes_value_pattern})\s*\)\s*(?:nocase)?\s*" + regex_field_value_pattern = rf"{regex_field_regex}\s*{regex_value_pattern}" + + def search_field_value(self, query: str) -> tuple[FieldValue, str]: + if regex_field_value_search := re.match(self.regex_field_value_pattern, query): + field = regex_field_value_search.group("field") + operator, value = self.get_operator_and_value(regex_field_value_search, mapped_operator=OperatorType.REGEX) + operator, value = self.process_value_wildcards(value=value, operator=OperatorType.REGEX) + pos = regex_field_value_search.end() + query = query[pos:] + + operator_token = Identifier(token_type=operator) + field_value = self.create_field_value(field_name=field, operator=operator_token, value=value) + return field_value, query + + return super().search_field_value(query=query) + + def get_operator_and_value( + self, match: re.Match, mapped_operator: str = OperatorType.EQ, operator: Optional[str] = None + ) -> tuple[str, Any]: + if (d_q_value := get_match_group(match, group_name=ValueType.double_quotes_value)) is not None: + return mapped_operator, self.escape_manager.remove_escape(d_q_value) + + if (b_q_value := get_match_group(match, group_name=ValueType.back_quotes_value)) is not None: + return mapped_operator, self.escape_manager.remove_escape(b_q_value) + + return super().get_operator_and_value(match, mapped_operator, operator) + + def _check_field_value_match(self, query: str, white_space_pattern: str = r"\s+") -> bool: + if re.match(self.regex_field_value_pattern, query, re.IGNORECASE): + return True + + return super()._check_field_value_match(query, white_space_pattern) diff --git a/uncoder-core/app/translator/platforms/crowdstrike/__init__.py b/uncoder-core/app/translator/platforms/crowdstrike/__init__.py new file mode 100644 index 00000000..c7e9dfb5 --- /dev/null +++ b/uncoder-core/app/translator/platforms/crowdstrike/__init__.py @@ -0,0 +1,3 @@ +from app.translator.platforms.crowdstrike.parsers.crowdstrike import CrowdStrikeQueryParser # noqa: F401 +from app.translator.platforms.crowdstrike.renders.crowdstrike import CrowdStrikeQueryRender # noqa: F401 +from app.translator.platforms.crowdstrike.renders.crowdstrike_cti import CrowdStrikeCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/crowdstrike/const.py b/uncoder-core/app/translator/platforms/crowdstrike/const.py new file mode 100644 index 00000000..7a76084d --- /dev/null +++ b/uncoder-core/app/translator/platforms/crowdstrike/const.py @@ -0,0 +1,24 @@ +from app.translator.core.models.platform_details import PlatformDetails + +CROWDSTRIKE_QUERY_DETAILS = { + "platform_id": "crowdstrike-spl-query", + "name": "CrowdStrike Endpoint Security", + "platform_name": "Query (SPL)", + "group_id": "crowdstrike", + "group_name": "CrowdStrike Endpoint Security", +} + +DEFAULT_CROWDSTRIKE_CTI_MAPPING = { + "DestinationIP": "RemoteAddressIP4", + "SourceIP": "LocalAddressIP4", + "HashSha256": "SHA256HashData", + "HashMd5": "MD5HashData", + "Emails": "emails", + "Domain": "DomainName", + "HashSha1": "SHA1HashData", + "Files": "TargetFileName", + "URL": "HttpUrl", +} + + +crowdstrike_query_details = PlatformDetails(**CROWDSTRIKE_QUERY_DETAILS) diff --git a/uncoder-core/app/translator/platforms/crowdstrike/functions/__init__.py b/uncoder-core/app/translator/platforms/crowdstrike/functions/__init__.py new file mode 100644 index 00000000..f94e10bc --- /dev/null +++ b/uncoder-core/app/translator/platforms/crowdstrike/functions/__init__.py @@ -0,0 +1,8 @@ +from app.translator.platforms.base.spl.functions import SplFunctions + + +class CrowdStrikeFunctions(SplFunctions): + pass + + +crowd_strike_functions = CrowdStrikeFunctions() diff --git a/uncoder-core/app/translator/platforms/crowdstrike/mapping.py b/uncoder-core/app/translator/platforms/crowdstrike/mapping.py new file mode 100644 index 00000000..04918b23 --- /dev/null +++ b/uncoder-core/app/translator/platforms/crowdstrike/mapping.py @@ -0,0 +1,29 @@ +from typing import Optional + +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature +from app.translator.platforms.crowdstrike.const import crowdstrike_query_details + + +class CrowdStrikeLogSourceSignature(LogSourceSignature): + def __init__(self, event_simple_name: Optional[list[str]], default_source: dict): + self.event_simple_names = set(event_simple_name or []) + self._default_source = default_source or {} + + def is_suitable(self, event_simpleName: Optional[list[str]] = None) -> bool: # noqa: N803 + conditions = [set(event_simpleName).issubset(self.event_simple_names) if event_simpleName else None] + return self._check_conditions(conditions) + + def __str__(self) -> str: + return f"event_simpleName={self._default_source['event_simpleName']}" + + +class CrowdstrikeMappings(BasePlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> CrowdStrikeLogSourceSignature: + log_source = mapping.get("log_source", {}) + default_log_source = mapping["default_log_source"] + return CrowdStrikeLogSourceSignature( + event_simple_name=log_source.get("event_simpleName"), default_source=default_log_source + ) + + +crowdstrike_query_mappings = CrowdstrikeMappings(platform_dir="crowdstrike", platform_details=crowdstrike_query_details) diff --git a/siem-converter/app/converter/platforms/elasticsearch/parsers/__init__.py b/uncoder-core/app/translator/platforms/crowdstrike/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/elasticsearch/parsers/__init__.py rename to uncoder-core/app/translator/platforms/crowdstrike/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/crowdstrike/parsers/crowdstrike.py b/uncoder-core/app/translator/platforms/crowdstrike/parsers/crowdstrike.py new file mode 100644 index 00000000..08ec0b7f --- /dev/null +++ b/uncoder-core/app/translator/platforms/crowdstrike/parsers/crowdstrike.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import parser_manager +from app.translator.platforms.base.spl.parsers.spl import SplQueryParser +from app.translator.platforms.crowdstrike.const import crowdstrike_query_details +from app.translator.platforms.crowdstrike.functions import CrowdStrikeFunctions, crowd_strike_functions +from app.translator.platforms.crowdstrike.mapping import CrowdstrikeMappings, crowdstrike_query_mappings + + +@parser_manager.register_supported_by_roota +class CrowdStrikeQueryParser(SplQueryParser): + details: PlatformDetails = crowdstrike_query_details + + log_source_pattern = r"___source_type___\s*=\s*(?:\"(?P[%a-zA-Z_*:0-9\-/]+)\"|(?P[%a-zA-Z_*:0-9\-/]+))(?:\s+(?:and|or)\s+|\s+)?" # noqa: E501 + log_source_key_types = ("event_simpleName",) + + mappings: CrowdstrikeMappings = crowdstrike_query_mappings + platform_functions: CrowdStrikeFunctions = crowd_strike_functions + + wrapped_with_comment_pattern = r"^\s*`(?:|\n|.)*`" diff --git a/siem-converter/app/converter/platforms/elasticsearch/renders/__init__.py b/uncoder-core/app/translator/platforms/crowdstrike/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/elasticsearch/renders/__init__.py rename to uncoder-core/app/translator/platforms/crowdstrike/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/crowdstrike/renders/crowdstrike.py b/uncoder-core/app/translator/platforms/crowdstrike/renders/crowdstrike.py new file mode 100644 index 00000000..40911708 --- /dev/null +++ b/uncoder-core/app/translator/platforms/crowdstrike/renders/crowdstrike.py @@ -0,0 +1,44 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.base.spl.renders.spl import SplFieldValueRender, SplQueryRender +from app.translator.platforms.crowdstrike.const import crowdstrike_query_details +from app.translator.platforms.crowdstrike.functions import CrowdStrikeFunctions, crowd_strike_functions +from app.translator.platforms.crowdstrike.mapping import CrowdstrikeMappings, crowdstrike_query_mappings + + +class CrowdStrikeFieldValueRender(SplFieldValueRender): + details = crowdstrike_query_details + + +@render_manager.register +class CrowdStrikeQueryRender(SplQueryRender): + details: PlatformDetails = crowdstrike_query_details + mappings: CrowdstrikeMappings = crowdstrike_query_mappings + platform_functions: CrowdStrikeFunctions = None + + or_token = "OR" + field_value_render = CrowdStrikeFieldValueRender(or_token=or_token) + comment_symbol = "`" + + def init_platform_functions(self) -> None: + self.platform_functions = crowd_strike_functions + self.platform_functions.platform_query_render = self diff --git a/uncoder-core/app/translator/platforms/crowdstrike/renders/crowdstrike_cti.py b/uncoder-core/app/translator/platforms/crowdstrike/renders/crowdstrike_cti.py new file mode 100644 index 00000000..baabea37 --- /dev/null +++ b/uncoder-core/app/translator/platforms/crowdstrike/renders/crowdstrike_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.crowdstrike.const import DEFAULT_CROWDSTRIKE_CTI_MAPPING, crowdstrike_query_details + + +@render_cti_manager.register +class CrowdStrikeCTI(RenderCTI): + details: PlatformDetails = crowdstrike_query_details + + field_value_template: str = '{key}="{value}"' + or_operator: str = " OR " + group_or_operator: str = " " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_CROWDSTRIKE_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/elasticsearch/__init__.py b/uncoder-core/app/translator/platforms/elasticsearch/__init__.py new file mode 100644 index 00000000..f13f11f3 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/__init__.py @@ -0,0 +1,15 @@ +from app.translator.platforms.elasticsearch.parsers.detection_rule import ( + ElasticSearchRuleParser, # noqa: F401 + ElasticSearchRuleTOMLParser, # noqa: F401 +) +from app.translator.platforms.elasticsearch.parsers.elasticsearch import ElasticSearchQueryParser # noqa: F401 +from app.translator.platforms.elasticsearch.parsers.elasticsearch_eql import ElasticSearchEQLQueryParser # noqa: F401 +from app.translator.platforms.elasticsearch.renders.detection_rule import ElasticSearchRuleRender # noqa: F401 +from app.translator.platforms.elasticsearch.renders.elast_alert import ElastAlertRuleRender # noqa: F401 +from app.translator.platforms.elasticsearch.renders.elasticsearch import ElasticSearchQueryRender # noqa: F401 +from app.translator.platforms.elasticsearch.renders.elasticsearch_cti import ElasticsearchCTI # noqa: F401 +from app.translator.platforms.elasticsearch.renders.elasticsearch_eql import ElasticSearchEQLQueryRender # noqa: F401 +from app.translator.platforms.elasticsearch.renders.esql import ESQLQueryRender # noqa: F401 +from app.translator.platforms.elasticsearch.renders.esql_rule import ESQLRuleRender # noqa: F401 +from app.translator.platforms.elasticsearch.renders.kibana import KibanaRuleRender # noqa: F401 +from app.translator.platforms.elasticsearch.renders.xpack_watcher import XPackWatcherRuleRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/elasticsearch/const.py b/uncoder-core/app/translator/platforms/elasticsearch/const.py new file mode 100644 index 00000000..51402819 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/const.py @@ -0,0 +1,255 @@ +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.models.platform_details import PlatformDetails + +PLATFORM_DETAILS = {"group_id": "elk stack", "group_name": "Elastic Stack", "alt_platform_name": "ECS"} + +_ELASTIC_LUCENE_QUERY = "elastic-lucene-query" +_ELASTIC_LUCENE_RULE = "elastic-lucene-rule" +_ELASTIC_LUCENE_RULE_TOML = "elastic-lucene-rule-toml" +_ELASTIC_KIBANA_RULE = "elastic-kibana-rule" +_ELASTALERT_LUCENE_RULE = "elastalert-lucene-rule" +_ELASTIC_WATCHER_RULE = "elastic-watcher-rule" +_ELASTIC_ESQL_QUERY = "elastic-esql-query" +_ELASTIC_ESQL_RULE = "elastic-esql-rule" +_ELASTIC_EQL_QUERY = "elastic-eql-query" + +ELASTIC_QUERY_TYPES = { + _ELASTIC_LUCENE_QUERY, + _ELASTIC_LUCENE_RULE, + _ELASTIC_KIBANA_RULE, + _ELASTALERT_LUCENE_RULE, + _ELASTIC_WATCHER_RULE, + _ELASTIC_ESQL_QUERY, + _ELASTIC_ESQL_RULE, +} + +ELASTICSEARCH_LUCENE_QUERY_DETAILS = { + "platform_id": _ELASTIC_LUCENE_QUERY, + "name": "Elasticsearch Query", + "platform_name": "Query (Lucene)", + **PLATFORM_DETAILS, +} + +ELASTICSEARCH_ESQL_QUERY_DETAILS = { + "platform_id": _ELASTIC_ESQL_QUERY, + "name": "Elasticsearch ES|QL Query", + "platform_name": "Query (ES|QL)", + **PLATFORM_DETAILS, +} + +ELASTICSEARCH_ESQL_RULE_DETAILS = { + "platform_id": _ELASTIC_ESQL_RULE, + "name": "Elasticsearch ES|QL Rule", + "platform_name": "Rule (ES|QL)", + **PLATFORM_DETAILS, +} + +ELASTICSEARCH_RULE_DETAILS = { + "platform_id": _ELASTIC_LUCENE_RULE, + "name": "Elastic Rule", + "platform_name": "Detection Rule (Lucene)", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +ELASTICSEARCH_RULE_TOML_DETAILS = { + "platform_id": _ELASTIC_LUCENE_RULE_TOML, + "name": "Elastic Rule TOML", + "platform_name": "Detection Rule (Lucene) TOML", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +KIBANA_DETAILS = { + "platform_id": _ELASTIC_KIBANA_RULE, + "name": "Elastic Kibana Saved Search", + "platform_name": "Kibana SavedSearch (JSON)", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +ELASTALERT_DETAILS = { + "platform_id": _ELASTALERT_LUCENE_RULE, + "name": "ElastAlert", + "platform_name": "Alert (Lucene)", + "group_name": "ElastAlert", + "group_id": "elastalert", +} + +XPACK_WATCHER_DETAILS = { + "platform_id": _ELASTIC_WATCHER_RULE, + "name": "Elastic Watcher", + "platform_name": "Rule (Watcher)", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +ELASTICSEARCH_EQL_QUERY_DETAILS = { + "platform_id": _ELASTIC_EQL_QUERY, + "name": "Elasticsearch EQL Query", + "platform_name": "Query (EQL)", + **PLATFORM_DETAILS, +} + +elasticsearch_lucene_query_details = PlatformDetails(**ELASTICSEARCH_LUCENE_QUERY_DETAILS) +elasticsearch_esql_query_details = PlatformDetails(**ELASTICSEARCH_ESQL_QUERY_DETAILS) +elasticsearch_esql_rule_details = PlatformDetails(**ELASTICSEARCH_ESQL_RULE_DETAILS) +elasticsearch_rule_details = PlatformDetails(**ELASTICSEARCH_RULE_DETAILS) +elasticsearch_rule_toml_details = PlatformDetails(**ELASTICSEARCH_RULE_TOML_DETAILS) +elastalert_details = PlatformDetails(**ELASTALERT_DETAILS) +kibana_rule_details = PlatformDetails(**KIBANA_DETAILS) +xpack_watcher_details = PlatformDetails(**XPACK_WATCHER_DETAILS) +elastic_eql_query_details = PlatformDetails(**ELASTICSEARCH_EQL_QUERY_DETAILS) + +ELASTICSEARCH_DETECTION_RULE = { + "description": "Autogenerated ElasticSearch Detection Rule.", + "author": [], + "enabled": True, + "false_positives": [], + "filters": [], + "from": "now-360s", + "immutable": False, + "index": [], + "interval": "5m", + "rule_id": "", + "language": "lucene", + "output_index": ".siem-signals-default", + "max_signals": 100, + "risk_score": 65, + "name": "", + "query": "", + "meta": {"from": "1m"}, + "severity": SeverityType.high, + "tags": [], + "to": "now", + "type": "query", + "threat": [], + "version": 1, + "references": [], + "license": "", +} + +ELASTICSEARCH_ALERT = """alert: +- debug +description: +filter: +- query_string: + query: +index: winlogbeat-* +name: +priority: +realert: + minutes: 0 +type: any +""" + +KIBANA_RULE = { + "_id": "dcd74b95-3f36-4ed9-9598-0490951643aa-Malicious-PowerView-PowerShell-Commandlets", + "_type": "search", + "_source": { + "title": "Autogenerated Kibana rule", + "description": "Autogenerated Kibana rule.", + "hits": 0, + "columns": [], + "sort": ["@timestamp", "desc"], + "version": 1, + "kibanaSavedObjectMeta": {"searchSourceJSON": ""}, + }, +} + + +KIBANA_SEARCH_SOURCE_JSON = { + "index": "winlogbeat-*", + "filter": [], + "highlight": { + "pre_tags": ["@kibana-highlighted-field@"], + "post_tags": ["@/kibana-highlighted-field@"], + "fields": {"*": {}}, + "require_field_match": False, + "fragment_size": 2147483647, + }, + "query": {"query_string": {"query": "", "analyze_wildcard": True}}, +} + + +XPACK_WATCHER_RULE = { + "metadata": {"title": "", "description": "", "tags": [], "query": ""}, + "trigger": {"schedule": {"interval": "30m"}}, + "input": { + "search": { + "request": { + "body": { + "size": 0, + "query": { + "bool": { + "must": [{"query_string": {"query": "", "analyze_wildcard": True}}], + "filter": {"range": {"@timestamp": {"gte": "now-30m/m"}}}, + } + }, + }, + "indices": [], # put indices from mapping + } + } + }, + "condition": {"compare": {"ctx.payload.hits.total": {"not_eq": 0}}}, + "actions": { + "send_email": { + "throttle_period": "15m", + "email": { + "profile": "standard", + "from": "root@localhost", + "to": "root@localhost", + "subject": "", + "body": "Hits:\n{{#ctx.payload.hits.hits}}{{_source}}\n================================================================================\n{{/ctx.payload.hits.hits}}", # noqa: E501 + "attachments": {"data.json": {"data": {"format": "json"}}}, + }, + } + }, +} + +ESQL_RULE = { + "name": "", + "tags": [], + "interval": "5m", + "enabled": True, + "revision": 0, + "description": "", + "risk_score": 21, + "severity": "low", + "license": "", + "output_index": "", + "meta": {"from": "1m"}, + "author": [], + "false_positives": [], + "from": "now-360s", + "rule_id": "", + "max_signals": 100, + "risk_score_mapping": [], + "severity_mapping": [], + "threat": [], + "to": "now", + "references": [], + "version": 1, + "exceptions_list": [], + "immutable": False, + "related_integrations": [], + "required_fields": [], + "setup": "", + "type": "esql", + "language": "esql", + "query": "", + "actions": [], +} + +DEFAULT_ELASTICSEARCH_CTI_MAPPING = { + "DestinationIP": "destination.ip", + "SourceIP": "source.ip", + "HashSha512": "file.hash.sha512", + "HashSha256": "file.hash.sha256", + "HashMd5": "file.hash.md5", + "Emails": "email.from.address", + "Domain": "destination.domain", + "HashSha1": "file.hash.sha1", + "Files": "file.name", + "URL": "url.original", +} diff --git a/uncoder-core/app/translator/platforms/elasticsearch/escape_manager.py b/uncoder-core/app/translator/platforms/elasticsearch/escape_manager.py new file mode 100644 index 00000000..993fdcfa --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/escape_manager.py @@ -0,0 +1,31 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class ESQLQueryEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [ + EscapeDetails(pattern=r"\\", escape_symbols=r"\\\\\\\\"), + EscapeDetails(pattern=r'"', escape_symbols=r"\""), + EscapeDetails(pattern=r"'", escape_symbols=r"\'"), + ], + ValueType.regex_value: [ + EscapeDetails(pattern=r"\\", escape_symbols=r"\\\\\\\\"), + EscapeDetails(pattern=r'"', escape_symbols=r"\""), + EscapeDetails(pattern=r"'", escape_symbols=r"\'"), + ], + } + + +class EQLQueryEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [EscapeDetails(pattern=r"\\", escape_symbols=r"\\\\")], + ValueType.regex_value: [EscapeDetails(pattern=r"\\", escape_symbols=r"\\\\")], + } + + +esql_query_escape_manager = ESQLQueryEscapeManager() +eql_query_escape_manager = EQLQueryEscapeManager() diff --git a/uncoder-core/app/translator/platforms/elasticsearch/mapping.py b/uncoder-core/app/translator/platforms/elasticsearch/mapping.py new file mode 100644 index 00000000..32f0a0fa --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/mapping.py @@ -0,0 +1,31 @@ +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.elasticsearch.const import ( + elastalert_details, + elasticsearch_esql_query_details, + elastic_eql_query_details, + elasticsearch_lucene_query_details, + elasticsearch_rule_details, + kibana_rule_details, + xpack_watcher_details, +) + +DEFAULT_MAPPING_NAME = "default" + +elasticsearch_lucene_query_mappings = LuceneMappings( + platform_dir="elasticsearch", platform_details=elasticsearch_lucene_query_details +) +elasticsearch_rule_mappings = LuceneMappings(platform_dir="elasticsearch", platform_details=elasticsearch_rule_details) +elastalert_mappings = LuceneMappings(platform_dir="elasticsearch", platform_details=elastalert_details) +kibana_rule_mappings = LuceneMappings(platform_dir="elasticsearch", platform_details=kibana_rule_details) +xpack_watcher_mappings = LuceneMappings(platform_dir="elasticsearch", platform_details=xpack_watcher_details) +elastic_eql_query_mappings = LuceneMappings(platform_dir="elasticsearch", platform_details=elastic_eql_query_details) + + +class ElasticESQLMappings(LuceneMappings): + is_strict_mapping: bool = True + skip_load_default_mappings = True + + +esql_query_mappings = ElasticESQLMappings( + platform_dir="elasticsearch_esql", platform_details=elasticsearch_esql_query_details +) diff --git a/siem-converter/app/converter/platforms/fireeye_helix/__init__.py b/uncoder-core/app/translator/platforms/elasticsearch/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/fireeye_helix/__init__.py rename to uncoder-core/app/translator/platforms/elasticsearch/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/elasticsearch/parsers/detection_rule.py b/uncoder-core/app/translator/platforms/elasticsearch/parsers/detection_rule.py new file mode 100644 index 00000000..6d04b229 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/parsers/detection_rule.py @@ -0,0 +1,98 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" +from datetime import datetime + +from app.translator.core.mixins.rule import JsonRuleMixin, TOMLRuleMixin +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, RawMetaInfoContainer, RawQueryContainer +from app.translator.managers import parser_manager +from app.translator.platforms.elasticsearch.const import elasticsearch_rule_details, elasticsearch_rule_toml_details +from app.translator.platforms.elasticsearch.parsers.elasticsearch import ElasticSearchQueryParser +from app.translator.tools.utils import parse_rule_description_str + + +@parser_manager.register +class ElasticSearchRuleParser(ElasticSearchQueryParser, JsonRuleMixin): + details: PlatformDetails = elasticsearch_rule_details + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + rule = self.load_rule(text=text) + parsed_description = parse_rule_description_str(rule.get("description", "")) + + mitre_attack = self.mitre_config.get_mitre_info( + tactics=[threat_data["tactic"]["name"].replace(" ", "_").lower() for threat_data in rule.get("threat", [])], + techniques=[threat_data["technique"][0]["id"].lower() for threat_data in rule.get("threat", [])], + ) + + return RawQueryContainer( + query=rule["query"], + language=language, + meta_info=MetaInfoContainer( + id_=rule.get("rule_id"), + title=rule.get("name"), + description=parsed_description.get("description") or rule.get("description"), + references=rule.get("references", []), + author=parsed_description.get("author") or rule.get("author"), + severity=rule.get("severity"), + license_=parsed_description.get("license"), + tags=rule.get("tags"), + mitre_attack=mitre_attack, + ), + ) + + +@parser_manager.register +class ElasticSearchRuleTOMLParser(ElasticSearchQueryParser, TOMLRuleMixin): + details: PlatformDetails = elasticsearch_rule_toml_details + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + raw_rule = self.load_rule(text=text) + rule = raw_rule.get("rule") + metadata = raw_rule.get("metadata") + techniques = [] + for threat_data in rule.get("threat", []): + if threat_data.get("technique"): + techniques.append(threat_data["technique"][0]["id"].lower()) + mitre_attack = self.mitre_config.get_mitre_info( + tactics=[threat_data["tactic"]["name"].replace(" ", "_").lower() for threat_data in rule.get("threat", [])], + techniques=techniques, + ) + date = None + if metadata.get("creation_date"): + date = datetime.strptime(metadata.get("creation_date"), "%Y/%m/%d").strftime("%Y-%m-%d") + return RawQueryContainer( + query=rule["query"], + language=language, + meta_info=MetaInfoContainer( + id_=rule.get("rule_id"), + title=rule.get("name"), + description=rule.get("description"), + author=rule.get("author"), + date=date, + license_=rule.get("license"), + severity=rule.get("severity"), + references=rule.get("references"), + tags=rule.get("tags"), + mitre_attack=mitre_attack, + index=rule.get("index"), + language=rule.get("language"), + risk_score=rule.get("risk_score"), + type_=rule.get("type"), + raw_metainfo_container=RawMetaInfoContainer(from_=rule.get("from"), interval=rule.get("interval")), + ), + ) diff --git a/uncoder-core/app/translator/platforms/elasticsearch/parsers/elasticsearch.py b/uncoder-core/app/translator/platforms/elasticsearch/parsers/elasticsearch.py new file mode 100644 index 00000000..2f287a93 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/parsers/elasticsearch.py @@ -0,0 +1,30 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import parser_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.base.lucene.parsers.lucene import LuceneQueryParser +from app.translator.platforms.elasticsearch.const import elasticsearch_lucene_query_details +from app.translator.platforms.elasticsearch.mapping import elasticsearch_lucene_query_mappings + + +@parser_manager.register_supported_by_roota +class ElasticSearchQueryParser(LuceneQueryParser): + details: PlatformDetails = elasticsearch_lucene_query_details + mappings: LuceneMappings = elasticsearch_lucene_query_mappings diff --git a/uncoder-core/app/translator/platforms/elasticsearch/parsers/elasticsearch_eql.py b/uncoder-core/app/translator/platforms/elasticsearch/parsers/elasticsearch_eql.py new file mode 100644 index 00000000..377b1e08 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/parsers/elasticsearch_eql.py @@ -0,0 +1,37 @@ +import re + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser +from app.translator.managers import parser_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.elasticsearch.const import elastic_eql_query_details +from app.translator.platforms.elasticsearch.mapping import elastic_eql_query_mappings +from app.translator.platforms.elasticsearch.tokenizer import ElasticSearchEQLTokenizer + + +@parser_manager.register_supported_by_roota +class ElasticSearchEQLQueryParser(PlatformQueryParser): + details: PlatformDetails = elastic_eql_query_details + tokenizer = ElasticSearchEQLTokenizer() + mappings: LuceneMappings = elastic_eql_query_mappings + query_delimiter_pattern = r"\swhere\s" + + def _parse_query(self, query: str) -> tuple[str, dict[str, list[str]]]: + log_source = {"category": []} + if re.search(self.query_delimiter_pattern, query, flags=re.IGNORECASE): + sp_query = re.split(self.query_delimiter_pattern, query, flags=re.IGNORECASE) + if sp_query[0].lower() != "all": + log_source["category"].append(sp_query[0]) + return sp_query[1], log_source + return query, log_source + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + query, log_sources = self._parse_query(raw_query_container.query) + query_tokens = self.get_query_tokens(query) + query_field_tokens, _, _ = self.get_field_tokens(query_tokens) + source_mappings = self.get_source_mappings(query_field_tokens, log_sources) + meta_info = raw_query_container.meta_info + meta_info.query_fields = query_field_tokens + meta_info.source_mapping_ids = [source_mapping.source_id for source_mapping in source_mappings] + return TokenizedQueryContainer(tokens=query_tokens, meta_info=meta_info) diff --git a/siem-converter/app/converter/platforms/fireeye_helix/mappings/__init__.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/fireeye_helix/mappings/__init__.py rename to uncoder-core/app/translator/platforms/elasticsearch/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/elasticsearch/renders/detection_rule.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/detection_rule.py new file mode 100644 index 00000000..1142a26d --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/renders/detection_rule.py @@ -0,0 +1,120 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy +import json +from typing import Optional, Union + +from app.translator.core.mapping import SourceMapping +from app.translator.core.mitre import MitreConfig, MitreInfoContainer +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, MitreTechniqueContainer +from app.translator.managers import render_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.elasticsearch.const import ELASTICSEARCH_DETECTION_RULE, elasticsearch_rule_details +from app.translator.platforms.elasticsearch.mapping import elasticsearch_rule_mappings +from app.translator.platforms.elasticsearch.renders.elasticsearch import ( + ElasticSearchFieldValue, + ElasticSearchQueryRender, +) +from app.translator.tools.utils import get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated Elastic Rule" + + +class ElasticSearchRuleFieldValue(ElasticSearchFieldValue): + details: PlatformDetails = elasticsearch_rule_details + + +@render_manager.register +class ElasticSearchRuleRender(ElasticSearchQueryRender): + details: PlatformDetails = elasticsearch_rule_details + mappings: LuceneMappings = elasticsearch_rule_mappings + mitre: MitreConfig = MitreConfig() + + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + field_value_render = ElasticSearchRuleFieldValue(or_token=or_token) + + def __create_mitre_threat(self, mitre_attack: MitreInfoContainer) -> Union[list, list[dict]]: + if not mitre_attack.techniques: + return [] + threat = [] + + for tactic in mitre_attack.tactics: + tactic_render = {"id": tactic.external_id, "name": tactic.name, "reference": tactic.url} + sub_threat = {"tactic": tactic_render, "framework": "MITRE ATT&CK", "technique": []} + for technique in mitre_attack.techniques: + technique_id = technique.technique_id.lower() + if "." in technique_id: + technique_id = technique_id[: technique.technique_id.index(".")] + main_technique: Optional[MitreTechniqueContainer] = self.mitre.techniques.search(technique_id) + if main_technique and tactic.name in main_technique.tactic: + sub_threat["technique"].append( + { + "id": main_technique.technique_id, + "name": main_technique.name, + "reference": main_technique.url, + } + ) + if len(sub_threat["technique"]) > 0: + threat.append(sub_threat) + + return sorted(threat, key=lambda x: x["tactic"]["id"]) + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = copy.deepcopy(ELASTICSEARCH_DETECTION_RULE) + index = source_mapping.log_source_signature.default_source.get("index") if source_mapping else None + description_str = get_rule_description_str( + description=meta_info.description or rule["description"] or _AUTOGENERATED_TEMPLATE, + license_=meta_info.license, + ) + rule.update( + { + "query": query, + "description": description_str, + "name": meta_info.title or _AUTOGENERATED_TEMPLATE, + "rule_id": meta_info.id, + "author": meta_info.author, + "severity": meta_info.severity, + "references": meta_info.references, + "license": meta_info.license, + "tags": meta_info.tags, + "threat": self.__create_mitre_threat(meta_info.mitre_attack), + "false_positives": meta_info.false_positives, + "index": [index] if index else [], + } + ) + rule_str = json.dumps(rule, indent=4, sort_keys=False, ensure_ascii=False) + rule_str = self.wrap_with_unmapped_fields(rule_str, unmapped_fields) + return self.wrap_with_not_supported_functions(rule_str, not_supported_functions) diff --git a/uncoder-core/app/translator/platforms/elasticsearch/renders/elast_alert.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/elast_alert.py new file mode 100644 index 00000000..c6ea3a35 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/renders/elast_alert.py @@ -0,0 +1,86 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Optional + +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer +from app.translator.managers import render_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.elasticsearch.const import ELASTICSEARCH_ALERT, elastalert_details +from app.translator.platforms.elasticsearch.mapping import elastalert_mappings +from app.translator.platforms.elasticsearch.renders.elasticsearch import ( + ElasticSearchFieldValue, + ElasticSearchQueryRender, +) +from app.translator.tools.utils import get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated ElastAlert" +_SEVERITIES_MAP = {SeverityType.low: "4", SeverityType.medium: "3", SeverityType.high: "2", SeverityType.critical: "1"} + + +class ElasticAlertRuleFieldValue(ElasticSearchFieldValue): + details: PlatformDetails = elastalert_details + + +@render_manager.register +class ElastAlertRuleRender(ElasticSearchQueryRender): + details: PlatformDetails = elastalert_details + mappings: LuceneMappings = elastalert_mappings + + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + field_value_render = ElasticAlertRuleFieldValue(or_token=or_token) + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = ELASTICSEARCH_ALERT.replace("", query) + mitre_attack = [] + if meta_info and meta_info.mitre_attack: + mitre_attack.extend([technique.technique_id for technique in meta_info.mitre_attack.techniques]) + mitre_attack.extend([tactic.name for tactic in meta_info.mitre_attack.tactics]) + rule = rule.replace( + "", + get_rule_description_str( + author=meta_info.author, + description=meta_info.description or _AUTOGENERATED_TEMPLATE, + license_=meta_info.license, + rule_id=meta_info.id, + mitre_attack=mitre_attack, + ), + ) + rule = rule.replace("", meta_info.title or _AUTOGENERATED_TEMPLATE) + rule = rule.replace("", _SEVERITIES_MAP[meta_info.severity]) + rule = self.wrap_with_unmapped_fields(rule, unmapped_fields) + return self.wrap_with_not_supported_functions(rule, not_supported_functions) diff --git a/uncoder-core/app/translator/platforms/elasticsearch/renders/elasticsearch.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/elasticsearch.py new file mode 100644 index 00000000..817707ae --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/renders/elasticsearch.py @@ -0,0 +1,38 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.base.lucene.renders.lucene import LuceneFieldValueRender, LuceneQueryRender +from app.translator.platforms.elasticsearch.const import elasticsearch_lucene_query_details +from app.translator.platforms.elasticsearch.mapping import elasticsearch_lucene_query_mappings + + +class ElasticSearchFieldValue(LuceneFieldValueRender): + details: PlatformDetails = elasticsearch_lucene_query_details + + +@render_manager.register +class ElasticSearchQueryRender(LuceneQueryRender): + details: PlatformDetails = elasticsearch_lucene_query_details + mappings: LuceneMappings = elasticsearch_lucene_query_mappings + + or_token = "OR" + field_value_render = ElasticSearchFieldValue(or_token=or_token) diff --git a/uncoder-core/app/translator/platforms/elasticsearch/renders/elasticsearch_cti.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/elasticsearch_cti.py new file mode 100644 index 00000000..820b6d54 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/renders/elasticsearch_cti.py @@ -0,0 +1,40 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.elasticsearch.const import ( + DEFAULT_ELASTICSEARCH_CTI_MAPPING, + elasticsearch_lucene_query_details, +) + + +@render_cti_manager.register +class ElasticsearchCTI(RenderCTI): + details: PlatformDetails = elasticsearch_lucene_query_details + + field_value_template: str = '"{value}"' + or_operator: str = " OR " + group_or_operator: str = " OR " + or_group: str = "{processing_key}:({or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_ELASTICSEARCH_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/elasticsearch/renders/elasticsearch_eql.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/elasticsearch_eql.py new file mode 100644 index 00000000..530c404d --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/renders/elasticsearch_eql.py @@ -0,0 +1,135 @@ +from typing import Optional, Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.tokens import GroupType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mapping import LogSourceSignature +from app.translator.core.mixins.tokens import ExtraConditionMixin +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.core.str_value_manager import StrValueManager +from app.translator.managers import render_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.elasticsearch.const import elastic_eql_query_details +from app.translator.platforms.elasticsearch.mapping import elastic_eql_query_mappings +from app.translator.platforms.elasticsearch.str_value_manager import eql_str_value_manager + + +class ElasticSearchEQLFieldValue(BaseFieldValueRender): + details: PlatformDetails = elastic_eql_query_details + str_value_manager: StrValueManager = eql_str_value_manager + list_token = ", " + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f'"{value}"' + + @staticmethod + def _wrap_int_value(value: int) -> str: + return f'"{value}"' + + def apply_field(self, field: str) -> str: + if field.count("-") > 0 or field.count(" ") > 0 or field[0].isdigit(): + return f"`{field}`" + if field.endswith(".text"): + return field[:-5] + return field + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.list_token.join( + self._pre_process_value(field, v, value_type=ValueType.value, wrap_str=True, wrap_int=True) + for v in value + ) + return f"{self.apply_field(field)} : ({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{self.apply_field(field)} : {value}" + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{self.apply_field(field)} < {value}" + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{self.apply_field(field)} <= {value}" + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{self.apply_field(field)} > {value}" + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{self.apply_field(field)} >= {value}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.list_token.join( + self._pre_process_value(field, v, value_type=ValueType.value, wrap_str=True, wrap_int=True) + for v in value + ) + return f"{self.apply_field(field)} != ({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{self.apply_field(field)} != {value}" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.list_token.join( + f'"*{self._pre_process_value(field, v, value_type=ValueType.value)}*"' for v in value + ) + return f"{self.apply_field(field)} : ({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value) + return f'{self.apply_field(field)} : "*{value}*"' + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.list_token.join( + f'"*{self._pre_process_value(field, v, value_type=ValueType.value)}"' for v in value + ) + return f"{self.apply_field(field)} : ({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value) + return f'{self.apply_field(field)} : "*{value}"' + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.list_token.join( + f'"{self._pre_process_value(field, v, value_type=ValueType.value)}*"' for v in value + ) + return f"{self.apply_field(field)} : ({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value) + return f'{self.apply_field(field)} : "{value}*"' + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.regex_value, wrap_int=True) + return f'{self.apply_field(field)} regex~ "{value}.?"' + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" + return self._pre_process_value(field, value, wrap_str=True) + + def is_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + return f"{self.apply_field(field)} == null" + + def is_not_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + return f"{self.apply_field(field)} != null" + + +@render_manager.register +class ElasticSearchEQLQueryRender(ExtraConditionMixin, PlatformQueryRender): + details: PlatformDetails = elastic_eql_query_details + mappings: LuceneMappings = elastic_eql_query_mappings + or_token = "or" + and_token = "and" + not_token = "not" + comment_symbol = "//" + field_value_render = ElasticSearchEQLFieldValue(or_token=or_token) + + def generate_prefix(self, log_source_signature: Optional[LogSourceSignature], functions_prefix: str = "") -> str: # noqa: ARG002 + return "any where " + + def in_brackets(self, raw_list: list[QUERY_TOKEN_TYPE]) -> list[QUERY_TOKEN_TYPE]: + return [Identifier(token_type=GroupType.L_PAREN), *raw_list, Identifier(token_type=GroupType.R_PAREN)] diff --git a/uncoder-core/app/translator/platforms/elasticsearch/renders/esql.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/esql.py new file mode 100644 index 00000000..ebeba491 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/renders/esql.py @@ -0,0 +1,139 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from typing import Optional, Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.exceptions.render import UnsupportedRenderMethod +from app.translator.core.mapping import LogSourceSignature +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.managers import render_manager +from app.translator.platforms.elasticsearch.const import elasticsearch_esql_query_details +from app.translator.platforms.elasticsearch.mapping import ElasticESQLMappings, esql_query_mappings +from app.translator.platforms.elasticsearch.str_value_manager import ( + ESQLStrValueManager, + esql_str_value_manager +) + + +class ESQLFieldValueRender(BaseFieldValueRender): + details: PlatformDetails = elasticsearch_esql_query_details + str_value_manager: ESQLStrValueManager = esql_str_value_manager + + @staticmethod + def _make_case_insensitive(value: str) -> str: + container: list[str] = [] + for v in value: + if v.isalpha(): + container.append(f"[{v.upper()}{v.lower()}]") + else: + container.append(v) + return "".join(container) + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f'"{value}"' + + @staticmethod + def _wrap_int_value(value: int) -> str: + return f'"{value}"' + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.equal_modifier(field=field, value=v) for v in value])})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{field} == {value}" + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{field} < {value}" + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{field} <= {value}" + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{field} > {value}" + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{field} >= {value}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{field} != {value}" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + if field.endswith(".text"): + return self.regex_modifier(field=field, value=value) + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=False, wrap_int=True) + return f'{field} like "*{value}*"' + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if field.endswith(".text"): + return self.regex_modifier(field=field, value=value) + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"ends_with({field}, {value})" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if field.endswith(".text"): + return self.regex_modifier(field=field, value=value) + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"starts_with({field}, {value})" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.regex_value, wrap_str=False, wrap_int=True) + if isinstance(value, str): + value = self._make_case_insensitive(value) + return f'{field} rlike ".*{value}.*"' + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + raise UnsupportedRenderMethod(platform_name=self.details.name, method="Keywords") + + +@render_manager.register +class ESQLQueryRender(PlatformQueryRender): + details: PlatformDetails = elasticsearch_esql_query_details + mappings: ElasticESQLMappings = esql_query_mappings + comment_symbol = "//" + + or_token = "or" + and_token = "and" + not_token = "not" + field_value_render = ESQLFieldValueRender(or_token=or_token) + + def generate_prefix(self, log_source_signature: Optional[LogSourceSignature], functions_prefix: str = "") -> str: # noqa: ARG002 + table = str(log_source_signature) if str(log_source_signature) else "*" + return f"FROM {table} |" + + @staticmethod + def _finalize_search_query(query: str) -> str: + return f"WHERE {query}" if query else "" diff --git a/uncoder-core/app/translator/platforms/elasticsearch/renders/esql_rule.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/esql_rule.py new file mode 100644 index 00000000..6eebf0c4 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/renders/esql_rule.py @@ -0,0 +1,115 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +import copy +import json +from typing import Optional, Union + +from app.translator.core.mapping import LogSourceSignature, SourceMapping +from app.translator.core.mitre import MitreConfig +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, MitreInfoContainer +from app.translator.managers import render_manager +from app.translator.platforms.elasticsearch.const import ESQL_RULE, elasticsearch_esql_rule_details +from app.translator.platforms.elasticsearch.mapping import LuceneMappings, esql_query_mappings +from app.translator.platforms.elasticsearch.renders.esql import ESQLFieldValueRender, ESQLQueryRender + +_AUTOGENERATED_TEMPLATE = "Autogenerated ESQL Rule" + + +class ESQLRuleFieldValueRender(ESQLFieldValueRender): + details: PlatformDetails = elasticsearch_esql_rule_details + + +@render_manager.register +class ESQLRuleRender(ESQLQueryRender): + details: PlatformDetails = elasticsearch_esql_rule_details + mappings: LuceneMappings = esql_query_mappings + mitre: MitreConfig = MitreConfig() + + or_token = "or" + field_value_render = ESQLRuleFieldValueRender(or_token=or_token) + + def generate_prefix(self, log_source_signature: Optional[LogSourceSignature], functions_prefix: str = "") -> str: # noqa: ARG002 + table = str(log_source_signature) if str(log_source_signature) else "*" + return f"FROM {table} metadata _id, _version, _index |" + + def __create_mitre_threat(self, mitre_attack: MitreInfoContainer) -> Union[list, list[dict]]: + if not mitre_attack.techniques: + return [] + threat = [] + + for tactic in mitre_attack.tactics: + tactic_render = {"id": tactic.external_id, "name": tactic.name, "reference": tactic.url} + sub_threat = {"tactic": tactic_render, "framework": "MITRE ATT&CK", "technique": []} + for technique in mitre_attack.techniques: + technique_id = technique.technique_id.lower() + if "." in technique_id: + technique_id = technique_id[: technique.technique_id.index(".")] + main_technique = self.mitre.get_technique(technique_id) + if main_technique and tactic.name in main_technique.tactic: + sub_threat["technique"].append( + { + "id": main_technique.technique_id, + "name": main_technique.name, + "reference": main_technique.url, + } + ) + if len(sub_threat["technique"]) > 0: + threat.append(sub_threat) + + return sorted(threat, key=lambda x: x["tactic"]["id"]) + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = copy.deepcopy(ESQL_RULE) + rule.update( + { + "query": query, + "description": meta_info.description if meta_info else rule["description"] or _AUTOGENERATED_TEMPLATE, + "name": meta_info.title if meta_info else _AUTOGENERATED_TEMPLATE, + } + ) + if meta_info: + rule.update( + { + "rule_id": meta_info.id, + "author": [meta_info.author], + "severity": meta_info.severity, + "references": meta_info.references, + "license": meta_info.license, + "tags": meta_info.tags, + "threat": self.__create_mitre_threat(meta_info.mitre_attack), + "false_positives": meta_info.false_positives, + } + ) + rule_str = json.dumps(rule, indent=4, sort_keys=False, ensure_ascii=False) + if not_supported_functions: + rendered_not_supported = self.render_not_supported_functions(not_supported_functions) + return rule_str + rendered_not_supported + return rule_str diff --git a/uncoder-core/app/translator/platforms/elasticsearch/renders/kibana.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/kibana.py new file mode 100644 index 00000000..9985bc1b --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/renders/kibana.py @@ -0,0 +1,85 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy +import json +from typing import Optional + +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer +from app.translator.managers import render_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.elasticsearch.const import KIBANA_RULE, KIBANA_SEARCH_SOURCE_JSON, kibana_rule_details +from app.translator.platforms.elasticsearch.mapping import kibana_rule_mappings +from app.translator.platforms.elasticsearch.renders.elasticsearch import ( + ElasticSearchFieldValue, + ElasticSearchQueryRender, +) +from app.translator.tools.utils import get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated Elastic Kibana Saved Search" + + +class KibanaFieldValue(ElasticSearchFieldValue): + details: PlatformDetails = kibana_rule_details + + +@render_manager.register +class KibanaRuleRender(ElasticSearchQueryRender): + details: PlatformDetails = kibana_rule_details + mappings: LuceneMappings = kibana_rule_mappings + or_token = "OR" + field_value_render = KibanaFieldValue(or_token=or_token) + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + search_source = copy.deepcopy(KIBANA_SEARCH_SOURCE_JSON) + search_source["query"]["query_string"]["query"] = query + dumped_rule = json.dumps(search_source, sort_keys=False) + rule = copy.deepcopy(KIBANA_RULE) + rule["_source"]["kibanaSavedObjectMeta"]["searchSourceJSON"] = dumped_rule + rule["_source"]["title"] = meta_info.title or _AUTOGENERATED_TEMPLATE + descr = meta_info.description or rule["_source"]["description"] or _AUTOGENERATED_TEMPLATE + mitre_attack = [] + if meta_info and meta_info.mitre_attack: + mitre_attack.extend([technique.technique_id for technique in meta_info.mitre_attack.techniques]) + mitre_attack.extend([tactic.name for tactic in meta_info.mitre_attack.tactics]) + rule["_source"]["description"] = get_rule_description_str( + description=descr, + author=meta_info.author, + rule_id=meta_info.id, + license_=meta_info.license, + references=meta_info.references, + mitre_attack=mitre_attack, + ) + rule_str = json.dumps(rule, indent=4, sort_keys=False) + rule_str = self.wrap_with_unmapped_fields(rule_str, unmapped_fields) + return self.wrap_with_not_supported_functions(rule_str, not_supported_functions) diff --git a/uncoder-core/app/translator/platforms/elasticsearch/renders/xpack_watcher.py b/uncoder-core/app/translator/platforms/elasticsearch/renders/xpack_watcher.py new file mode 100644 index 00000000..abc02e84 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/renders/xpack_watcher.py @@ -0,0 +1,107 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy +import json +from typing import Optional + +from app.translator.core.mapping import SourceMapping +from app.translator.core.mitre import MitreConfig +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, MitreInfoContainer +from app.translator.managers import render_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.elasticsearch.const import XPACK_WATCHER_RULE, xpack_watcher_details +from app.translator.platforms.elasticsearch.mapping import xpack_watcher_mappings +from app.translator.platforms.elasticsearch.renders.elasticsearch import ( + ElasticSearchFieldValue, + ElasticSearchQueryRender, +) +from app.translator.tools.utils import get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated Elastic Watcher" + + +class XpackWatcherRuleFieldValue(ElasticSearchFieldValue): + details: PlatformDetails = xpack_watcher_details + + +@render_manager.register +class XPackWatcherRuleRender(ElasticSearchQueryRender): + details: PlatformDetails = xpack_watcher_details + mappings: LuceneMappings = xpack_watcher_mappings + or_token = "OR" + field_value_render = XpackWatcherRuleFieldValue(or_token=or_token) + mitre: MitreConfig = MitreConfig() + + def __create_mitre_threat(self, mitre_attack: MitreInfoContainer) -> dict: + result = {"tactics": [], "techniques": []} + + for tactic in mitre_attack.tactics: + result["tactics"].append({"external_id": tactic.external_id, "url": tactic.url, "tactic": tactic.name}) + for technique in mitre_attack.techniques: + result["techniques"].append( + { + "technique_id": technique.technique_id, + "technique": technique.name, + "url": technique.url, + "tactic": technique.tactic, + } + ) + + return result if result["tactics"] or result["techniques"] else {} + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = copy.deepcopy(XPACK_WATCHER_RULE) + mitre_attack = [] + if meta_info and meta_info.mitre_attack: + mitre_attack.extend([technique.technique_id for technique in meta_info.mitre_attack.techniques]) + mitre_attack.extend([tactic.name for tactic in meta_info.mitre_attack.tactics]) + rule["metadata"].update( + { + "query": query, + "title": meta_info.title or _AUTOGENERATED_TEMPLATE, + "description": get_rule_description_str( + description=meta_info.description or _AUTOGENERATED_TEMPLATE, + author=meta_info.author, + license_=meta_info.license, + mitre_attack=mitre_attack, + ), + "tags": self.__create_mitre_threat(mitre_attack=meta_info.mitre_attack), + } + ) + rule["input"]["search"]["request"]["body"]["query"]["bool"]["must"][0]["query_string"]["query"] = query + indices = source_mapping and [str(source_mapping.log_source_signature)] or [] + rule["input"]["search"]["request"]["indices"] = indices + rule["actions"]["send_email"]["email"]["subject"] = meta_info.title or _AUTOGENERATED_TEMPLATE + rule_str = json.dumps(rule, indent=4, sort_keys=False) + rule_str = self.wrap_with_unmapped_fields(rule_str, unmapped_fields) + return self.wrap_with_not_supported_functions(rule_str, not_supported_functions) diff --git a/uncoder-core/app/translator/platforms/elasticsearch/str_value_manager.py b/uncoder-core/app/translator/platforms/elasticsearch/str_value_manager.py new file mode 100644 index 00000000..3fcbafe9 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/str_value_manager.py @@ -0,0 +1,67 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from typing import ClassVar, Optional + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.str_value_manager import ( + BaseSpecSymbol, + ReDigitalSymbol, + ReWhiteSpaceSymbol, + ReWordSymbol, + SingleSymbolWildCard, + StrValue, + StrValueManager, + UnboundLenWildCard, +) +from app.translator.platforms.elasticsearch.escape_manager import ( + EQLQueryEscapeManager, + ESQLQueryEscapeManager, + eql_query_escape_manager, + esql_query_escape_manager, +) + + +class ESQLStrValueManager(StrValueManager): + escape_manager: ESQLQueryEscapeManager = esql_query_escape_manager + re_str_alpha_num_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = { + "w": ReWordSymbol, + "d": ReDigitalSymbol, + "s": ReWhiteSpaceSymbol, + } + + +class EQLStrValueManager(StrValueManager): + escape_manager: EQLQueryEscapeManager = eql_query_escape_manager + str_spec_symbols_map: ClassVar[dict[str, type[BaseSpecSymbol]]] = { + "?": SingleSymbolWildCard, + "*": UnboundLenWildCard, + } + + def from_str_to_container( + self, + value: str, + value_type: str = ValueType.value, # noqa: ARG002 + escape_symbol: Optional[str] = None, # noqa: ARG002 + ) -> StrValue: + split = [self.str_spec_symbols_map[char]() if char in self.str_spec_symbols_map else char for char in value] + return StrValue(value, self._concat(split)) + + +esql_str_value_manager = ESQLStrValueManager() +eql_str_value_manager = EQLStrValueManager() diff --git a/uncoder-core/app/translator/platforms/elasticsearch/tokenizer.py b/uncoder-core/app/translator/platforms/elasticsearch/tokenizer.py new file mode 100644 index 00000000..115144e8 --- /dev/null +++ b/uncoder-core/app/translator/platforms/elasticsearch/tokenizer.py @@ -0,0 +1,89 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" +import re +from typing import Any, ClassVar, Optional, Union + +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.platforms.base.lucene.tokenizer import LuceneTokenizer +from app.translator.platforms.elasticsearch.str_value_manager import eql_str_value_manager +from app.translator.tools.utils import get_match_group + + +class ElasticSearchTokenizer(LuceneTokenizer): + pass + + +class ElasticSearchEQLTokenizer(QueryTokenizer): + single_value_operators_map: ClassVar[dict[str, str]] = { + ":": OperatorType.EQ, + "==": OperatorType.EQ, + "<=": OperatorType.LTE, + "<": OperatorType.LT, + ">=": OperatorType.GTE, + ">": OperatorType.GT, + "!=": OperatorType.NOT_EQ, + "regex~": OperatorType.REGEX, + "regex": OperatorType.REGEX, + } + + multi_value_operators_map: ClassVar[dict[str, str]] = { + "in": OperatorType.EQ, + "in~": OperatorType.EQ, + ":": OperatorType.EQ, + } + wildcard_symbol = "*" + field_pattern = r"(?P[a-zA-Z\.\-_`]+)" + re_value_pattern = ( + rf'"(?P<{ValueType.regex_value}>(?:[:a-zA-Z*0-9=+%#\-_/,;`?~‘\'.<>$&^@!\]\[()\s]|\\\"|\\)*)\[\^[z|Z]\]\.\?"' # noqa: RUF001 + ) + double_quotes_value_pattern = ( + rf'"(?P<{ValueType.double_quotes_value}>(?:[:a-zA-Z*0-9=+%#\-_/,;`?~‘\'.<>$&^@!\]\[()\s]|\\\"|\\)*)"' # noqa: RUF001 + ) + _value_pattern = rf"{re_value_pattern}|{double_quotes_value_pattern}" + multi_value_pattern = rf"""\((?P<{ValueType.multi_value}>[:a-zA-Z\"\*0-9=+%#№;\-_\/\\'\,.$&^@!\(\[\]\s|]+)\)""" + multi_value_check_pattern = r"___field___\s*___operator___\s*\(" + keyword_pattern = ( + rf'"(?P<{ValueType.double_quotes_value}>(?:[:a-zA-Z*0-9=+%#\-_/,;`?~‘\'.<>$&^@!\]\[()\s]|\\\"|\\)*)"' # noqa: RUF001 + ) + + str_value_manager = eql_str_value_manager + + def get_operator_and_value( + self, match: re.Match, mapped_operator: str = OperatorType.EQ, operator: Optional[str] = None + ) -> tuple[str, Any]: + if (re_value := get_match_group(match, group_name=ValueType.regex_value)) is not None: + return OperatorType.REGEX, self.str_value_manager.from_re_str_to_container(re_value) + + if (d_q_value := get_match_group(match, group_name=ValueType.double_quotes_value)) is not None: + return mapped_operator, self.str_value_manager.from_str_to_container(d_q_value) + + return super().get_operator_and_value(match, mapped_operator, operator) + + def is_multi_value_flow(self, field_name: str, operator: str, query: str) -> bool: + check_pattern = self.multi_value_check_pattern + check_regex = check_pattern.replace("___field___", field_name).replace("___operator___", operator) + return bool(re.match(check_regex, query)) + + @staticmethod + def create_field_value(field_name: str, operator: Identifier, value: Union[str, list]) -> FieldValue: + field_name = field_name.replace("`", "") + return FieldValue(source_name=field_name, operator=operator, value=value) diff --git a/uncoder-core/app/translator/platforms/falco/__init__.py b/uncoder-core/app/translator/platforms/falco/__init__.py new file mode 100644 index 00000000..4e2ca546 --- /dev/null +++ b/uncoder-core/app/translator/platforms/falco/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.falco.renders.falco import FalcoRuleRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/falco/const.py b/uncoder-core/app/translator/platforms/falco/const.py new file mode 100644 index 00000000..1fec3aee --- /dev/null +++ b/uncoder-core/app/translator/platforms/falco/const.py @@ -0,0 +1,11 @@ +from app.translator.core.models.platform_details import PlatformDetails + +FALCO_RULE_DETAILS = { + "platform_id": "falco-yaml-rule", + "name": "Falco YAML Rule", + "platform_name": "Rule (YAML)", + "group_id": "falco", + "group_name": "Falco", +} + +falco_rule_details = PlatformDetails(**FALCO_RULE_DETAILS) diff --git a/uncoder-core/app/translator/platforms/falco/escape_manager.py b/uncoder-core/app/translator/platforms/falco/escape_manager.py new file mode 100644 index 00000000..76c61398 --- /dev/null +++ b/uncoder-core/app/translator/platforms/falco/escape_manager.py @@ -0,0 +1,17 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class FalcoRuleEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.regex_value: [ + EscapeDetails(pattern=r"([$^*+()\[\]{}|.?\-\\])", escape_symbols=r"\\\1"), + EscapeDetails(pattern=r"(')", escape_symbols=r"'\1"), + ] + } + + +falco_rule_escape_manager = FalcoRuleEscapeManager() diff --git a/uncoder-core/app/translator/platforms/falco/mapping.py b/uncoder-core/app/translator/platforms/falco/mapping.py new file mode 100644 index 00000000..3a325a66 --- /dev/null +++ b/uncoder-core/app/translator/platforms/falco/mapping.py @@ -0,0 +1,18 @@ +from app.translator.core.mapping import BaseStrictLogSourcesPlatformMappings, LogSourceSignature +from app.translator.platforms.falco.const import falco_rule_details + + +class FalcoRuleLogSourceSignature(LogSourceSignature): + def __str__(self) -> str: + return "" + + def is_suitable(self) -> bool: + return True + + +class FalcoRuleMappings(BaseStrictLogSourcesPlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> FalcoRuleLogSourceSignature: # noqa: ARG002 + return FalcoRuleLogSourceSignature() + + +falco_rule_mappings = FalcoRuleMappings(platform_dir="falco", platform_details=falco_rule_details) diff --git a/siem-converter/app/converter/platforms/fireeye_helix/renders/__init__.py b/uncoder-core/app/translator/platforms/falco/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/fireeye_helix/renders/__init__.py rename to uncoder-core/app/translator/platforms/falco/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/falco/renders/falco.py b/uncoder-core/app/translator/platforms/falco/renders/falco.py new file mode 100644 index 00000000..9ab54f06 --- /dev/null +++ b/uncoder-core/app/translator/platforms/falco/renders/falco.py @@ -0,0 +1,161 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from typing import ClassVar, Optional + +import yaml + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mapping import LogSourceSignature, SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer +from app.translator.core.models.query_tokens.field import Field +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.core.str_value_manager import StrValueManager +from app.translator.managers import render_manager +from app.translator.platforms.falco.const import falco_rule_details +from app.translator.platforms.falco.mapping import FalcoRuleMappings, falco_rule_mappings +from app.translator.platforms.falco.str_value_manager import falco_rule_str_value_manager + + +class FalcoRuleFieldValueRender(BaseFieldValueRender): + details = falco_rule_details + str_value_manager: StrValueManager = falco_rule_str_value_manager + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f'"{value}"' + + @staticmethod + def _wrap_int_value(value: str) -> str: + return f'"{value}"' + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.equal_modifier(field=field, value=v) for v in value])})" + return f"{field} = {self._pre_process_value(field, value, wrap_str=True)}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + return f"{field} != {self._pre_process_value(field, value, wrap_str=True)}" + + def less_modifier(self, field: str, value: int) -> str: + return f"{field} < {self._pre_process_value(field, value)}" + + def less_or_equal_modifier(self, field: str, value: int) -> str: + return f"{field} <= {self._pre_process_value(field, value)}" + + def greater_modifier(self, field: str, value: int) -> str: + return f"{field} > {self._pre_process_value(field, value)}" + + def greater_or_equal_modifier(self, field: str, value: int) -> str: + return f"{field} >= {self._pre_process_value(field, value)}" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.contains_modifier(field=field, value=v) for v in value])})" + value = self._pre_process_value(field, value, wrap_str=True, wrap_int=True) + return f"{field} contains {value}" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.endswith_modifier(field=field, value=v) for v in value])})" + value = self._pre_process_value(field, value, wrap_str=True, wrap_int=True) + return f"{field} endswith {value}" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.startswith_modifier(field=field, value=v) for v in value])})" + value = self._pre_process_value(field, value, wrap_str=True, wrap_int=True) + return f"{field} startswith {value}" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + regex_str = self._pre_process_value(field, value, value_type=ValueType.regex_value) + return f"{field} regex '{regex_str}'" + + def is_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.is_none(field=field, value=v) for v in value)})" + return f"{field} exists" + + +@render_manager.register +class FalcoRuleRender(PlatformQueryRender): + details: PlatformDetails = falco_rule_details + mappings: FalcoRuleMappings = falco_rule_mappings + + or_token = "or" + and_token = "and" + not_token = "not" + + comment_symbol = "//" + + field_value_render = FalcoRuleFieldValueRender(or_token=or_token) + + priority_map: ClassVar[dict[str, str]] = { + "unspecified": "NOTICE", + "info": "INFORMATIONAL", + "low": "WARNING", + "medium": "ERROR", + "high": "ERROR", + "critical": "CRITICAL", + } + + def generate_prefix(self, log_source_signature: Optional[LogSourceSignature], functions_prefix: str = "") -> str: # noqa: ARG002 + return "" + + def generate_output(self, fields: list[Field], unmapped_fields: list[str], source_mapping: SourceMapping) -> str: + extra_fields = [] + for field in fields: + if field.source_name in unmapped_fields: + extra_fields.append(field.source_name) + elif generic_field_name := field.get_generic_field_name(source_mapping.source_id): + extra_field = source_mapping.fields_mapping.get_platform_field_name(generic_field_name) + if extra_field: + extra_fields.append(extra_field) + extra_fields = [f"{field.replace('.', '_')}=%{field}" for field in extra_fields] + return f"shell in a container (container_name=%container.name {' '.join(extra_fields)})" + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = self._join_query_parts(prefix, query, functions) + rule = { + "rule": meta_info.title or "Falco Rule", + "condition": query, + "desc": meta_info.description or "Falco Rule", + "output": self.generate_output(meta_info.query_fields, unmapped_fields or [], source_mapping), + "priority": self.priority_map.get(meta_info.severity or "medium"), + } + rule_str = yaml.dump(rule, default_flow_style=False, sort_keys=False) + rule_str = self.wrap_with_meta_info(rule_str, meta_info) + rule_str = self.wrap_with_unmapped_fields(rule_str, unmapped_fields) + return self.wrap_with_not_supported_functions(rule_str, not_supported_functions) diff --git a/uncoder-core/app/translator/platforms/falco/str_value_manager.py b/uncoder-core/app/translator/platforms/falco/str_value_manager.py new file mode 100644 index 00000000..212c1e5d --- /dev/null +++ b/uncoder-core/app/translator/platforms/falco/str_value_manager.py @@ -0,0 +1,27 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from app.translator.core.str_value_manager import StrValueManager +from app.translator.platforms.falco.escape_manager import FalcoRuleEscapeManager, falco_rule_escape_manager + + +class FalcoRuleStrValueManager(StrValueManager): + escape_manager: FalcoRuleEscapeManager = falco_rule_escape_manager + + +falco_rule_str_value_manager = FalcoRuleStrValueManager() diff --git a/uncoder-core/app/translator/platforms/fireeye_helix/__init__.py b/uncoder-core/app/translator/platforms/fireeye_helix/__init__.py new file mode 100644 index 00000000..3b24c1b6 --- /dev/null +++ b/uncoder-core/app/translator/platforms/fireeye_helix/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.fireeye_helix.renders.fireeye_helix_cti import FireeyeHelixCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/fireeye_helix/const.py b/uncoder-core/app/translator/platforms/fireeye_helix/const.py new file mode 100644 index 00000000..b06e4d50 --- /dev/null +++ b/uncoder-core/app/translator/platforms/fireeye_helix/const.py @@ -0,0 +1,20 @@ +FIREEYE_HELIX_QUERY_DETAILS = { + "platform_id": "fireeye_helix", + "name": "FireEye Helix", + "group_name": "FireEye", + "group_id": "fireeye", + "platform_name": "Query", +} + +DEFAULT_FIREEYE_HELIX_CTI_MAPPING = { + "SourceIP": "~srcipv4", + "DestinationIP": "~dstipv4", + "Domain": "domain", + "URL": "url", + "HashMd5": "~hash", + "HashSha1": "~hash", + "HashSha256": "~hash", + "HashSha512": "~hash", + "Emails": "emails", + "Files": "filepath", +} diff --git a/siem-converter/app/converter/platforms/graylog/__init__.py b/uncoder-core/app/translator/platforms/fireeye_helix/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/graylog/__init__.py rename to uncoder-core/app/translator/platforms/fireeye_helix/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/fireeye_helix/renders/fireeye_helix_cti.py b/uncoder-core/app/translator/platforms/fireeye_helix/renders/fireeye_helix_cti.py new file mode 100644 index 00000000..51dba4e5 --- /dev/null +++ b/uncoder-core/app/translator/platforms/fireeye_helix/renders/fireeye_helix_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.fireeye_helix.const import DEFAULT_FIREEYE_HELIX_CTI_MAPPING, FIREEYE_HELIX_QUERY_DETAILS + + +@render_cti_manager.register +class FireeyeHelixCTI(RenderCTI): + details: PlatformDetails = PlatformDetails(**FIREEYE_HELIX_QUERY_DETAILS) + + field_value_template: str = "`{value}`" + or_operator: str = "," + group_or_operator: str = " " + or_group: str = "{processing_key}:[{or_group}]" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_FIREEYE_HELIX_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/forti_siem/__init__.py b/uncoder-core/app/translator/platforms/forti_siem/__init__.py new file mode 100644 index 00000000..11aaf8b6 --- /dev/null +++ b/uncoder-core/app/translator/platforms/forti_siem/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.forti_siem.renders.forti_siem_rule import FortiSiemRuleRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/forti_siem/const.py b/uncoder-core/app/translator/platforms/forti_siem/const.py new file mode 100644 index 00000000..157ea7e1 --- /dev/null +++ b/uncoder-core/app/translator/platforms/forti_siem/const.py @@ -0,0 +1,225 @@ +from dataclasses import dataclass + +from app.translator.core.models.platform_details import PlatformDetails + +FORTI_SIEM_RULE_DETAILS = { + "platform_id": "fortisiem-rule", + "name": "FortiSIEM Rule", + "platform_name": "Rule", + "group_id": "forti_siem", + "group_name": "FortiSIEM", +} + + +forti_siem_rule_details = PlatformDetails(**FORTI_SIEM_RULE_DETAILS) + + +FORTI_SIEM_RULE = """ +> + + + true + + Correlation + false + + + + + > + + + + + + + COUNT(*) >= 1 + + + + + + +""" + + +@dataclass +class SourceEventTypesContainer: + default_pattern: str + event_types_map: dict[int, list[str]] + + +SOURCES_EVENT_TYPES_CONTAINERS_MAP = { + "windows_sysmon": SourceEventTypesContainer( + default_pattern="Win-Sysmon-", + event_types_map={ + 1: ["Win-Sysmon-1-Create-Process"], + 2: ["Win-Sysmon-2-FileCreation-Time-Changed"], + 3: ["Win-Sysmon-3-Network-Connection-IPv4", "Win-Sysmon-3-Network-Connection-IPv6"], + 5: ["Win-Sysmon-5-Process-Terminated"], + 6: ["Win-Sysmon-6-Driver-Loaded"], + 7: ["Win-Sysmon-7-Image-Loaded"], + 8: ["Win-Sysmon-8-CreateRemoteThread"], + 9: ["Win-Sysmon-9-RawAccessRead"], + 10: ["Win-Sysmon-10-ProcessAccess"], + 11: ["Win-Sysmon-11-FileCreate"], + 12: [ + "Win-Sysmon-12-Registry-CreateKey", + "Win-Sysmon-12-Registry-DeleteKey", + "Win-Sysmon-12-Registry-CreateValue", + "Win-Sysmon-12-Registry-DeleteValue", + ], + 13: ["Win-Sysmon-13-Registry-SetValue"], + 14: ["Win-Sysmon-14-Registry-RenameKey", "Win-Sysmon-14-Registry-RenameValue"], + 15: ["Win-Sysmon-15-FileCreateStreamHash"], + 16: ["Win-Sysmon-16-Config-State-Changed"], + 17: ["Win-Sysmon-17-PipeCreated"], + 18: ["Win-Sysmon-18-PipeConnected"], + 19: ["Win-Sysmon-19-WMIEventFilterActivity"], + 20: ["Win-Sysmon-20-WMIEventConsumerActivity"], + 21: ["Win-Sysmon-21-WMIEventConsumerToFilterActivity"], + 22: ["Win-Sysmon-22-DNSQuery"], + }, + ), + "registry_event": SourceEventTypesContainer( + default_pattern="Win-Sysmon-", + event_types_map={ + 1: ["Win-Sysmon-1-Create-Process"], + 2: ["Win-Sysmon-2-FileCreation-Time-Changed"], + 3: ["Win-Sysmon-3-Network-Connection-IPv4", "Win-Sysmon-3-Network-Connection-IPv6"], + 5: ["Win-Sysmon-5-Process-Terminated"], + 6: ["Win-Sysmon-6-Driver-Loaded"], + 7: ["Win-Sysmon-7-Image-Loaded"], + 8: ["Win-Sysmon-8-CreateRemoteThread"], + 9: ["Win-Sysmon-9-RawAccessRead"], + 10: ["Win-Sysmon-10-ProcessAccess"], + 11: ["Win-Sysmon-11-FileCreate"], + 12: [ + "Win-Sysmon-12-Registry-CreateKey", + "Win-Sysmon-12-Registry-DeleteKey", + "Win-Sysmon-12-Registry-CreateValue", + "Win-Sysmon-12-Registry-DeleteValue", + ], + 13: ["Win-Sysmon-13-Registry-SetValue"], + 14: ["Win-Sysmon-14-Registry-RenameKey", "Win-Sysmon-14-Registry-RenameValue"], + 15: ["Win-Sysmon-15-FileCreateStreamHash"], + 16: ["Win-Sysmon-16-Config-State-Changed"], + 17: ["Win-Sysmon-17-PipeCreated"], + 18: ["Win-Sysmon-18-PipeConnected"], + 19: ["Win-Sysmon-19-WMIEventFilterActivity"], + 20: ["Win-Sysmon-20-WMIEventConsumerActivity"], + 21: ["Win-Sysmon-21-WMIEventConsumerToFilterActivity"], + 22: ["Win-Sysmon-22-DNSQuery"], + }, + ), + "windows_system": SourceEventTypesContainer( + default_pattern="Win-System-", + event_types_map={ + 104: ["Win-System-Microsoft-Windows-Eventlog-104"], + 7036: ["Win-System-Service-Control-Manager-7036-.*"], + 7045: ["Win-System-Service-Control-Manager-7045"], + }, + ), + "windows_powershell": SourceEventTypesContainer(default_pattern="Win-PowerShell-", event_types_map={}), + "windows_powershell_classic": SourceEventTypesContainer(default_pattern="Win-PowerShell-", event_types_map={}), + "windows_security": SourceEventTypesContainer( + default_pattern="Win-Security-", + event_types_map={ + 4768: ["Win-Security-4768-success", "Win-Security-4768-failure"], + 4769: ["Win-Security-4769-failure", "Win-Security-4769-success"], + }, + ), + "windows_app": SourceEventTypesContainer(default_pattern="Win-App-", event_types_map={}), + "windows_application": SourceEventTypesContainer(default_pattern="Win-App-", event_types_map={}), + "windows_wmi_event": SourceEventTypesContainer( + default_pattern="Win-Sysmon-", + event_types_map={ + 19: ["Win-Sysmon-19-WMI-Event-Filter-Activity"], + 20: ["Win-Sysmon-20-WMI-Event-Consumer-Activity"], + 21: ["Win-Sysmon-21-WMI-Event-ConsumerToFilter-Activity"], + }, + ), + "windows_appxdeployment": SourceEventTypesContainer( + default_pattern="Win-AppXDeployment-Server-", + event_types_map={ + 400: ["Win-AppXDeployment-Server-400"], + 401: ["Win-AppXDeployment-Server-401"], + 157: ["Win-AppXDeployment-Server-157"], + }, + ), + "windows_appxdeployment_server": SourceEventTypesContainer( + default_pattern="Win-AppXDeployment-Server-", + event_types_map={ + 400: ["Win-AppXDeployment-Server-400"], + 401: ["Win-AppXDeployment-Server-401"], + 157: ["Win-AppXDeployment-Server-157"], + }, + ), + "windows_appxpackaging_om": SourceEventTypesContainer( + default_pattern="Win-AppXDeployment-Server-", + event_types_map={ + 400: ["Win-AppXDeployment-Server-400"], + 401: ["Win-AppXDeployment-Server-401"], + 157: ["Win-AppXDeployment-Server-157"], + }, + ), + "windows_firewall_as": SourceEventTypesContainer( + default_pattern="Win-Firewall-AS-", + event_types_map={ + 2002: ["Win-Firewall-AS-2002"], + 2083: ["Win-Firewall-AS-2083"], + 2003: ["Win-Firewall-AS-2003"], + 2082: ["Win-Firewall-AS-2082"], + 2008: ["Win-Firewall-AS-2008"], + }, + ), + "windows_provider_name": SourceEventTypesContainer( + default_pattern="Win-PrintService-", + event_types_map={ + 216: ["Win-PrintService-Setup-Succeeded"], + 325: ["Win-PrintService-Remove-Printer-Failed"], + 322: ["Win-PrintService-Publish-Printer-Failed"], + 323: ["Win-PrintService-Publish-Printer-Failed"], + 326: ["Win-PrintService-Publish-Printer-Failed"], + 327: ["Win-PrintService-Publish-Printer-Failed"], + 328: ["Win-PrintService-Publish-Printer-Failed"], + 329: ["Win-PrintService-Publish-Printer-Failed"], + 331: ["Win-PrintService-Publish-Printer-Failed"], + 333: ["Win-PrintService-Publish-Printer-Failed"], + }, + ), + "windows_security_mitigations": SourceEventTypesContainer( + default_pattern="Win-Security-Mitigation-", + event_types_map={11: ["Win-Security-Mitigation-11"], 12: ["Win-Security-Mitigation-12"]}, + ), + "windows_dns_client": SourceEventTypesContainer( + default_pattern="Win-DNS-Client-", event_types_map={3008: ["Win-DNS-Client-3008"]} + ), + "windows_bits_client": SourceEventTypesContainer( + default_pattern="Win-App-Microsoft-Windows-Bits-Client-", + event_types_map={16403: ["Win-App-Microsoft-Windows-Bits-Client-16403"]}, + ), + "windows_diagnosis_scripted": SourceEventTypesContainer( + default_pattern="Win-Diagnostics-Scripted-", event_types_map={101: ["Win-Diagnostics-Scripted-101"]} + ), + "windows_msexchange_management": SourceEventTypesContainer( + default_pattern="Win-MSExchange-Management-", event_types_map={6: ["Win-MSExchange-Management-6"]} + ), + "windows_file_block": SourceEventTypesContainer( + default_pattern="Win-Sysmon-", event_types_map={27: ["Win-Sysmon-27-FileBlockExecutable"]} + ), + "windows_shell_core": SourceEventTypesContainer( + default_pattern="Win-Shell-Core-", event_types_map={28115: ["Win-Shell-Core-28115"]} + ), + "windows_codeintegrity_operational": SourceEventTypesContainer( + default_pattern="Win-CodeIntegrity-Operational-", + event_types_map={ + 3023: ["Win-CodeIntegrity-Operational-3023"], + 3033: ["Win-CodeIntegrity-Operational-3033"], + 3077: ["Win-CodeIntegrity-Operational-3077"], + }, + ), + "windows_openssh": SourceEventTypesContainer( + default_pattern="Win-OpenSSH-", event_types_map={4: ["Win-OpenSSH-4"]} + ), +} diff --git a/uncoder-core/app/translator/platforms/forti_siem/escape_manager.py b/uncoder-core/app/translator/platforms/forti_siem/escape_manager.py new file mode 100644 index 00000000..07820b25 --- /dev/null +++ b/uncoder-core/app/translator/platforms/forti_siem/escape_manager.py @@ -0,0 +1,14 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class FortiSiemEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.regex_value: [EscapeDetails(pattern=r'([*\\.()\[\]|{}^$+!?"])')] + } + + +forti_siem_escape_manager = FortiSiemEscapeManager() diff --git a/uncoder-core/app/translator/platforms/forti_siem/mapping.py b/uncoder-core/app/translator/platforms/forti_siem/mapping.py new file mode 100644 index 00000000..7fefa128 --- /dev/null +++ b/uncoder-core/app/translator/platforms/forti_siem/mapping.py @@ -0,0 +1,38 @@ +from typing import Optional + +from app.translator.core.mapping import BaseCommonPlatformMappings, LogSourceSignature +from app.translator.platforms.forti_siem.const import forti_siem_rule_details + + +class FortiSiemLogSourceSignature(LogSourceSignature): + def __init__(self, event_types: Optional[list[str]], default_source: dict): + self.event_types = set(event_types or []) + self._default_source = default_source or {} + + def is_suitable(self, event_type: Optional[list[str]] = None) -> bool: + return self._check_conditions([set(event_type).issubset(self.event_types) if event_type else None]) + + def __str__(self) -> str: + event_type = self._default_source.get("eventType", "") + if event_type: + if event_type.startswith(self.wildcard_symbol) and not event_type.endswith(self.wildcard_symbol): + return f'eventType REGEXP "{event_type.lstrip(self.wildcard_symbol)}$"' + if event_type.endswith(self.wildcard_symbol) and not event_type.startswith(self.wildcard_symbol): + return f'eventType REGEXP "^{event_type.rstrip(self.wildcard_symbol)}"' + + if self.wildcard_symbol in event_type: + return f'eventType REGEXP "{event_type}"' + + return f'eventType = "{event_type}"' + + return "" + + +class FortiSiemMappings(BaseCommonPlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> FortiSiemLogSourceSignature: + event_types = mapping.get("log_source", {}).get("eventType") + default_log_source = mapping["default_log_source"] + return FortiSiemLogSourceSignature(event_types=event_types, default_source=default_log_source) + + +forti_siem_rule_mappings = FortiSiemMappings(platform_dir="forti_siem", platform_details=forti_siem_rule_details) diff --git a/siem-converter/app/converter/platforms/graylog/mappings/__init__.py b/uncoder-core/app/translator/platforms/forti_siem/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/graylog/mappings/__init__.py rename to uncoder-core/app/translator/platforms/forti_siem/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/forti_siem/renders/forti_siem_rule.py b/uncoder-core/app/translator/platforms/forti_siem/renders/forti_siem_rule.py new file mode 100644 index 00000000..f9b3e942 --- /dev/null +++ b/uncoder-core/app/translator/platforms/forti_siem/renders/forti_siem_rule.py @@ -0,0 +1,387 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import re +from typing import Optional, Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.custom_types.tokens import GroupType, LogicalOperatorType, OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mapping import SourceMapping +from app.translator.core.mitre import MitreInfoContainer +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, TokenizedQueryContainer +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.core.str_value_manager import StrValue +from app.translator.managers import render_manager +from app.translator.platforms.forti_siem.const import ( + FORTI_SIEM_RULE, + SOURCES_EVENT_TYPES_CONTAINERS_MAP, + forti_siem_rule_details, +) +from app.translator.platforms.forti_siem.mapping import FortiSiemMappings, forti_siem_rule_mappings +from app.translator.platforms.forti_siem.str_value_manager import forti_siem_str_value_manager +from app.translator.tools.utils import concatenate_str, get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated FortiSIEM Rule" +_EVENT_TYPE_FIELD = "eventType" + +_SEVERITIES_MAP = { + SeverityType.informational: "1", + SeverityType.low: "3", + SeverityType.medium: "5", + SeverityType.high: "7", + SeverityType.critical: "9", +} + +_NOT_OPERATORS_MAP_SUBSTITUTES = { + OperatorType.EQ: OperatorType.NOT_EQ, + OperatorType.CONTAINS: OperatorType.NOT_CONTAINS, + OperatorType.STARTSWITH: OperatorType.NOT_STARTSWITH, + OperatorType.ENDSWITH: OperatorType.NOT_ENDSWITH, + OperatorType.REGEX: OperatorType.NOT_REGEX, +} + +_NOT_STR_FIELDS = [ + # int + "ruleId", + "procTrustLevel", + "destIpPort", + "eventAction", + "srcIpPort", + "winLogonType", + "msgLen", + "size", + # ip + "destIpAddr", + "srcIpAddr", +] + + +class FortiSiemFieldValueRender(BaseFieldValueRender): + details: PlatformDetails = forti_siem_rule_details + str_value_manager = forti_siem_str_value_manager + + and_token = " AND " + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.equal_modifier(field=field, value=v) for v in value])})" + + if isinstance(value, StrValue): + if value.has_spec_symbols: + return self.regex_modifier(field, value) + + value = forti_siem_str_value_manager.from_container_to_str(value) + return f"{field}={value}" if field in _NOT_STR_FIELDS else f'{field}="{value}"' + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.and_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + + if isinstance(value, StrValue): + if value.has_spec_symbols: + return self.not_regex_modifier(field, value) + + value = forti_siem_str_value_manager.from_container_to_str(value) + return f"{field}!={value}" if field in _NOT_STR_FIELDS else f'{field}!="{value}"' + + @staticmethod + def __prepare_regex_value(value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, StrValue): + value = forti_siem_str_value_manager.from_container_to_str(value, value_type=ValueType.regex_value) + + return value + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.contains_modifier(field=field, value=v) for v in value])})" + + value = self.__prepare_regex_value(value) + return f'{field} REGEXP "{value}"' + + def not_contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.and_token.join([self.not_contains_modifier(field=field, value=v) for v in value])})" + + value = self.__prepare_regex_value(value) + return f'{field} NOT REGEXP "{value}"' + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.endswith_modifier(field=field, value=v) for v in value])})" + + value = self.__prepare_regex_value(value) + return f'{field} REGEXP "{value}$"' + + def not_endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.and_token.join([self.not_endswith_modifier(field=field, value=v) for v in value])})" + + value = self.__prepare_regex_value(value) + return f'{field} NOT REGEXP "{value}$"' + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.startswith_modifier(field=field, value=v) for v in value])})" + + value = self.__prepare_regex_value(value) + return f'{field} REGEXP "^{value}"' + + def not_startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.and_token.join([self.not_startswith_modifier(field=field, value=v) for v in value])})" + + value = self.__prepare_regex_value(value) + return f'{field} NOT REGEXP "^{value}"' + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.regex_modifier(field=field, value=v) for v in value])})" + + value = self.__prepare_regex_value(value) + return f'{field} REGEXP "{value}"' + + def not_regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.and_token.join([self.not_regex_modifier(field=field, value=v) for v in value])})" + + value = self.__prepare_regex_value(value) + return f'{field} NOT REGEXP "{value}"' + + +@render_manager.register +class FortiSiemRuleRender(PlatformQueryRender): + details: PlatformDetails = forti_siem_rule_details + mappings: FortiSiemMappings = forti_siem_rule_mappings + + or_token = "OR" + and_token = "AND" + not_token = None + + group_token = "(%s)" + + field_value_render = FortiSiemFieldValueRender(or_token=or_token) + + @staticmethod + def __is_negated_token(prev_token: QUERY_TOKEN_TYPE) -> bool: + return isinstance(prev_token, Identifier) and prev_token.token_type == LogicalOperatorType.NOT + + @staticmethod + def __should_negate(is_negated_token: bool = False, negation_ctx: bool = False) -> bool: + if is_negated_token and negation_ctx: + return False + + return is_negated_token or negation_ctx + + @staticmethod + def __negate_token(token: QUERY_TOKEN_TYPE) -> None: + if isinstance(token, Identifier): + if token.token_type == LogicalOperatorType.AND: + token.token_type = LogicalOperatorType.OR + elif token.token_type == LogicalOperatorType.OR: + token.token_type = LogicalOperatorType.AND + elif isinstance(token, FieldValue): + token_type = token.operator.token_type + token.operator.token_type = _NOT_OPERATORS_MAP_SUBSTITUTES.get(token_type) or token_type + + def __replace_not_tokens(self, tokens: list[QUERY_TOKEN_TYPE]) -> list[QUERY_TOKEN_TYPE]: + not_token_indices = [] + negation_ctx_stack = [] + for index, token in enumerate(tokens[1:], start=1): + current_negation_ctx = negation_ctx_stack[-1] if negation_ctx_stack else False + prev_token = tokens[index - 1] + if is_negated_token := self.__is_negated_token(prev_token): + not_token_indices.append(index - 1) + + if isinstance(token, Identifier): + if token.token_type == GroupType.L_PAREN: + negation_ctx_stack.append(self.__should_negate(is_negated_token, current_negation_ctx)) + continue + if token.token_type == GroupType.R_PAREN: + if negation_ctx_stack: + negation_ctx_stack.pop() + continue + + if self.__should_negate(is_negated_token, current_negation_ctx): + self.__negate_token(token) + + for index in reversed(not_token_indices): + tokens.pop(index) + + return tokens + + def _generate_from_tokenized_query_container_by_source_mapping( + self, query_container: TokenizedQueryContainer, source_mapping: SourceMapping + ) -> str: + unmapped_fields = self.mappings.check_fields_mapping_existence( + query_container.meta_info.query_fields, + query_container.meta_info.function_fields_map, + self.platform_functions.manager.supported_render_names, + source_mapping, + ) + is_event_type_set = False + field_values = [token for token in query_container.tokens if isinstance(token, FieldValue)] + mapped_fields_set = set() + for field_value in field_values: + mapped_fields = self.mappings.map_field(field_value.field, source_mapping) + mapped_fields_set = mapped_fields_set.union(set(mapped_fields)) + if _EVENT_TYPE_FIELD in mapped_fields: + is_event_type_set = True + self.__update_event_type_values(field_value, source_mapping.source_id) + + tokens = self.__replace_not_tokens(query_container.tokens) + result = self.generate_query(tokens=tokens, source_mapping=source_mapping) + prefix = "" if is_event_type_set else self.generate_prefix(source_mapping.log_source_signature) + rendered_functions = self.generate_functions(query_container.functions.functions, source_mapping) + not_supported_functions = query_container.functions.not_supported + rendered_functions.not_supported + return self.finalize_query( + prefix=prefix, + query=result, + functions=rendered_functions.rendered, + not_supported_functions=not_supported_functions, + unmapped_fields=unmapped_fields, + meta_info=query_container.meta_info, + source_mapping=source_mapping, + fields=mapped_fields_set, + ) + + @staticmethod + def __update_event_type_values(field_value: FieldValue, source_id: str) -> None: + new_values = [] + for value in field_value.values: + if not str(value).isdigit(): + new_values.append(value) + + elif not (source_event_types_container := SOURCES_EVENT_TYPES_CONTAINERS_MAP.get(source_id, {})): + new_values.append(f"Win-.*-{value}[^\d]*") + field_value.operator = Identifier(token_type=OperatorType.REGEX) + + elif event_types := source_event_types_container.event_types_map.get(value, []): + new_values.extend(event_types) + + else: + new_values.append(f"{source_event_types_container.default_pattern}{value}-.*") + field_value.operator = Identifier(token_type=OperatorType.REGEX) + + field_value.values = new_values + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + fields: Optional[set[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = FORTI_SIEM_RULE.replace("", self.generate_rule_header(meta_info)) + title = meta_info.title or _AUTOGENERATED_TEMPLATE + rule = rule.replace("", self.generate_rule_name(title)) + rule = rule.replace("", self.generate_title(title)) + description = get_rule_description_str( + description=meta_info.description.replace("\n", " ") or _AUTOGENERATED_TEMPLATE, + rule_id=meta_info.id, + author=meta_info.author, + license_=meta_info.license, + references=meta_info.references, + ) + rule = rule.replace("", description) + rule = rule.replace("", self.generate_event_type(title, meta_info.severity)) + args_list = self.get_args_list(fields.copy()) + rule = rule.replace("", self.get_args_str(args_list)) + rule = rule.replace("", query) + rule = rule.replace("", ", ".join(args_list)) + rule = rule.replace("", self.get_attr_str(fields.copy())) + rule = self.wrap_with_unmapped_fields(rule, unmapped_fields) + return self.wrap_with_not_supported_functions(rule, not_supported_functions) + + @staticmethod + def get_attr_str(fields: set[str]) -> str: + fields.discard("hostName") + fields.discard("eventType") + fields.discard("phRecvTime") + fields = sorted(fields) + + if len(fields) == 0: + return "phRecvTime, rawEventMsg" + + return ", ".join(["phRecvTime", *fields, "rawEventMsg"]) + + @staticmethod + def get_args_str(fields: list[str]) -> str: + return ", ".join(f"{f} = Filter.{f}" for f in fields) + + @staticmethod + def get_args_list(fields: set[str]) -> list[str]: + fields.discard("eventType") + fields.add("hostName") + return sorted(fields) + + @staticmethod + def generate_event_type(title: str, severity: str) -> str: + title = title.replace(" ", "_") + return concatenate_str(f'eventType="PH_RULE_{title}"', f'severity="{_SEVERITIES_MAP[severity]}"') + + @staticmethod + def generate_title(title: str) -> str: + return re.sub(r"\s*[^a-zA-Z0-9 _-]+\s*", " ", title) + + @staticmethod + def generate_rule_name(title: str) -> str: + # rule name allows only a-zA-Z0-9 \/:.$- + rule_name = re.sub(r'\s*[^a-zA-Z0-9 /:.$_\'"-]+\s*', " ", title) + rule_name = re.sub("_", "-", rule_name) + return re.sub(r'[\'"()+,]*', "", rule_name) + + @staticmethod + def get_mitre_info(mitre_attack: Union[MitreInfoContainer, None]) -> tuple[str, str]: + if not mitre_attack: + return "", "" + tactics = set() + techniques = set() + for tactic in mitre_attack.tactics: + if tactic_name := tactic.name: + tactics.add(tactic_name) + + for tech in mitre_attack.techniques: + techniques.add(tech.technique_id) + tactics = tactics.union(set(tech.tactic)) + + return ", ".join(sorted(tactics)), ", ".join(sorted(techniques)) + + def generate_rule_header(self, meta_info: MetaInfoContainer) -> str: + header = 'group="PH_SYS_RULE_THREAT_HUNTING"' + tactics_str, techniques_str = self.get_mitre_info(meta_info.mitre_attack) + if tactics_str: + header = concatenate_str(header, f'subFunction="{tactics_str}"') + + if techniques_str: + header = concatenate_str(header, f'technique="{techniques_str}"') + + return concatenate_str(header, 'phIncidentCategory="Server" function="Security"') + + def wrap_with_comment(self, value: str) -> str: + return f"" diff --git a/uncoder-core/app/translator/platforms/forti_siem/str_value_manager.py b/uncoder-core/app/translator/platforms/forti_siem/str_value_manager.py new file mode 100644 index 00000000..1c6b34ed --- /dev/null +++ b/uncoder-core/app/translator/platforms/forti_siem/str_value_manager.py @@ -0,0 +1,39 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy + +from app.translator.core.str_value_manager import ( + CONTAINER_SPEC_SYMBOLS_MAP, + SingleSymbolWildCard, + StrValueManager, + UnboundLenWildCard, +) +from app.translator.platforms.forti_siem.escape_manager import forti_siem_escape_manager + +FORTI_CONTAINER_SPEC_SYMBOLS_MAP = copy.copy(CONTAINER_SPEC_SYMBOLS_MAP) +FORTI_CONTAINER_SPEC_SYMBOLS_MAP.update({SingleSymbolWildCard: ".?", UnboundLenWildCard: ".*"}) + + +class FortiSiemStrValueManager(StrValueManager): + escape_manager = forti_siem_escape_manager + container_spec_symbols_map = FORTI_CONTAINER_SPEC_SYMBOLS_MAP + + +forti_siem_str_value_manager = FortiSiemStrValueManager() diff --git a/uncoder-core/app/translator/platforms/graylog/__init__.py b/uncoder-core/app/translator/platforms/graylog/__init__.py new file mode 100644 index 00000000..9af149ad --- /dev/null +++ b/uncoder-core/app/translator/platforms/graylog/__init__.py @@ -0,0 +1,3 @@ +from app.translator.platforms.graylog.parsers.graylog import GraylogQueryParser # noqa: F401 +from app.translator.platforms.graylog.renders.graylog import GraylogQueryRender # noqa: F401 +from app.translator.platforms.graylog.renders.graylog_cti import GraylogCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/graylog/const.py b/uncoder-core/app/translator/platforms/graylog/const.py new file mode 100644 index 00000000..90270013 --- /dev/null +++ b/uncoder-core/app/translator/platforms/graylog/const.py @@ -0,0 +1,25 @@ +from app.translator.core.models.platform_details import PlatformDetails + +GRAYLOG_QUERY_DETAILS = { + "platform_id": "graylog-lucene-query", + "name": "Graylog", + "group_name": "Graylog", + "platform_name": "Query", + "group_id": "graylog", +} + +DEFAULT_GRAYLOG_CTI_MAPPING = { + "SourceIP": "source.ip", + "DestinationIP": "destination.ip", + "Domain": "destination.domain", + "URL": "url.original", + "HashMd5": "file.hash.md5", + "HashSha1": "file.hash.sha1", + "HashSha256": "file.hash.sha256", + "HashSha512": "file.hash.sha512", + "Emails": "emails", + "Files": "filePath", +} + + +graylog_query_details = PlatformDetails(**GRAYLOG_QUERY_DETAILS) diff --git a/uncoder-core/app/translator/platforms/graylog/mapping.py b/uncoder-core/app/translator/platforms/graylog/mapping.py new file mode 100644 index 00000000..42edc609 --- /dev/null +++ b/uncoder-core/app/translator/platforms/graylog/mapping.py @@ -0,0 +1,4 @@ +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.graylog.const import graylog_query_details + +graylog_query_mappings = LuceneMappings(platform_dir="graylog", platform_details=graylog_query_details) diff --git a/siem-converter/app/converter/platforms/graylog/renders/__init__.py b/uncoder-core/app/translator/platforms/graylog/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/graylog/renders/__init__.py rename to uncoder-core/app/translator/platforms/graylog/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/graylog/parsers/graylog.py b/uncoder-core/app/translator/platforms/graylog/parsers/graylog.py new file mode 100644 index 00000000..6252cd66 --- /dev/null +++ b/uncoder-core/app/translator/platforms/graylog/parsers/graylog.py @@ -0,0 +1,30 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import parser_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.base.lucene.parsers.lucene import LuceneQueryParser +from app.translator.platforms.graylog.const import graylog_query_details +from app.translator.platforms.graylog.mapping import graylog_query_mappings + + +@parser_manager.register_supported_by_roota +class GraylogQueryParser(LuceneQueryParser): + details: PlatformDetails = graylog_query_details + mappings: LuceneMappings = graylog_query_mappings diff --git a/siem-converter/app/converter/platforms/logpoint/__init__.py b/uncoder-core/app/translator/platforms/graylog/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/logpoint/__init__.py rename to uncoder-core/app/translator/platforms/graylog/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/graylog/renders/graylog.py b/uncoder-core/app/translator/platforms/graylog/renders/graylog.py new file mode 100644 index 00000000..77be5c30 --- /dev/null +++ b/uncoder-core/app/translator/platforms/graylog/renders/graylog.py @@ -0,0 +1,38 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.base.lucene.renders.lucene import LuceneFieldValueRender, LuceneQueryRender +from app.translator.platforms.graylog.const import graylog_query_details +from app.translator.platforms.graylog.mapping import graylog_query_mappings + + +class GraylogFieldValue(LuceneFieldValueRender): + details: PlatformDetails = graylog_query_details + + +@render_manager.register +class GraylogQueryRender(LuceneQueryRender): + details: PlatformDetails = graylog_query_details + mappings: LuceneMappings = graylog_query_mappings + + or_token = "OR" + field_value_render = GraylogFieldValue(or_token=or_token) diff --git a/uncoder-core/app/translator/platforms/graylog/renders/graylog_cti.py b/uncoder-core/app/translator/platforms/graylog/renders/graylog_cti.py new file mode 100644 index 00000000..ae8ee06a --- /dev/null +++ b/uncoder-core/app/translator/platforms/graylog/renders/graylog_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.graylog.const import DEFAULT_GRAYLOG_CTI_MAPPING, GRAYLOG_QUERY_DETAILS + + +@render_cti_manager.register +class GraylogCTI(RenderCTI): + details: PlatformDetails = PlatformDetails(**GRAYLOG_QUERY_DETAILS) + + field_value_template: str = '"{value}"' + or_operator: str = " " + group_or_operator: str = " OR " + or_group: str = "{processing_key}:({or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_GRAYLOG_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/hunters/__init__.py b/uncoder-core/app/translator/platforms/hunters/__init__.py new file mode 100644 index 00000000..7fcb753d --- /dev/null +++ b/uncoder-core/app/translator/platforms/hunters/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.hunters.renders.hunters import HuntersQueryRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/hunters/const.py b/uncoder-core/app/translator/platforms/hunters/const.py new file mode 100644 index 00000000..eb61c622 --- /dev/null +++ b/uncoder-core/app/translator/platforms/hunters/const.py @@ -0,0 +1,11 @@ +from app.translator.core.models.platform_details import PlatformDetails + +HUNTERS_QUERY_DETAILS = { + "platform_id": "hunters-sql-query", + "name": "Hunters Query", + "group_name": "Hunters", + "platform_name": "Query", + "group_id": "hunters", +} + +hunters_query_details = PlatformDetails(**HUNTERS_QUERY_DETAILS) diff --git a/uncoder-core/app/translator/platforms/hunters/mapping.py b/uncoder-core/app/translator/platforms/hunters/mapping.py new file mode 100644 index 00000000..73687ce7 --- /dev/null +++ b/uncoder-core/app/translator/platforms/hunters/mapping.py @@ -0,0 +1,22 @@ +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature +from app.translator.platforms.hunters.const import hunters_query_details + + +class HuntersLogSourceSignature(LogSourceSignature): + def __init__(self, default_source: dict): + self._default_source = default_source or {} + + def is_suitable(self) -> bool: + return True + + def __str__(self) -> str: + return self._default_source.get("table", "") + + +class HuntersMappings(BasePlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> HuntersLogSourceSignature: + default_log_source = mapping["default_log_source"] + return HuntersLogSourceSignature(default_source=default_log_source) + + +hunters_query_mappings = HuntersMappings(platform_dir="hunters", platform_details=hunters_query_details) diff --git a/siem-converter/app/converter/platforms/logpoint/mappings/__init__.py b/uncoder-core/app/translator/platforms/hunters/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/logpoint/mappings/__init__.py rename to uncoder-core/app/translator/platforms/hunters/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/hunters/renders/hunters.py b/uncoder-core/app/translator/platforms/hunters/renders/hunters.py new file mode 100644 index 00000000..4e977a16 --- /dev/null +++ b/uncoder-core/app/translator/platforms/hunters/renders/hunters.py @@ -0,0 +1,42 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.base.sql.renders.sql import SqlFieldValueRender, SqlQueryRender +from app.translator.platforms.hunters.const import hunters_query_details +from app.translator.platforms.hunters.mapping import HuntersMappings, hunters_query_mappings + + +class HuntersFieldValueRender(SqlFieldValueRender): + details: PlatformDetails = hunters_query_details + + +@render_manager.register +class HuntersQueryRender(SqlQueryRender): + details: PlatformDetails = hunters_query_details + mappings: HuntersMappings = hunters_query_mappings + + or_token = "OR" + + field_value_render = HuntersFieldValueRender(or_token=or_token) + + @staticmethod + def _finalize_search_query(query: str) -> str: + return f"WHERE {query}" if query else "" diff --git a/uncoder-core/app/translator/platforms/logpoint/__init__.py b/uncoder-core/app/translator/platforms/logpoint/__init__.py new file mode 100644 index 00000000..743bd5e2 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logpoint/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.logpoint.renders.logpoint_cti import LogpointCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/logpoint/const.py b/uncoder-core/app/translator/platforms/logpoint/const.py new file mode 100644 index 00000000..68685661 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logpoint/const.py @@ -0,0 +1,20 @@ +LOGPOINT_QUERY_DETAILS = { + "platform_id": "logpoint", + "name": "Logpoint", + "group_name": "Logpoint", + "platform_name": "Query", + "group_id": "logpoint", +} + +DEFAULT_LOGPOINT_CTI_MAPPING = { + "DestinationIP": "dst_ip", + "SourceIP": "src_ip", + "HashSha512": "hash", + "HashSha256": "hash", + "HashMd5": "hash", + "Emails": "emails", + "Domain": "host", + "HashSha1": "hash", + "Files": "files", + "URL": "url", +} diff --git a/siem-converter/app/converter/platforms/logpoint/renders/__init__.py b/uncoder-core/app/translator/platforms/logpoint/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/logpoint/renders/__init__.py rename to uncoder-core/app/translator/platforms/logpoint/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/logpoint/renders/logpoint_cti.py b/uncoder-core/app/translator/platforms/logpoint/renders/logpoint_cti.py new file mode 100644 index 00000000..1bf42fd5 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logpoint/renders/logpoint_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.logpoint.const import DEFAULT_LOGPOINT_CTI_MAPPING, LOGPOINT_QUERY_DETAILS + + +@render_cti_manager.register +class LogpointCTI(RenderCTI): + details: PlatformDetails = PlatformDetails(**LOGPOINT_QUERY_DETAILS) + + field_value_template: str = '"{value}"' + or_operator: str = ", " + group_or_operator: str = " " + or_group: str = "{processing_key} IN [{or_group}]" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_LOGPOINT_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/logrhythm_axon/__init__.py b/uncoder-core/app/translator/platforms/logrhythm_axon/__init__.py new file mode 100644 index 00000000..1f03d01a --- /dev/null +++ b/uncoder-core/app/translator/platforms/logrhythm_axon/__init__.py @@ -0,0 +1,2 @@ +from app.translator.platforms.logrhythm_axon.renders.logrhythm_axon_query import LogRhythmAxonQueryRender # noqa: F401 +from app.translator.platforms.logrhythm_axon.renders.logrhythm_axon_rule import LogRhythmAxonRuleRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/logrhythm_axon/const.py b/uncoder-core/app/translator/platforms/logrhythm_axon/const.py new file mode 100644 index 00000000..6eb29e0c --- /dev/null +++ b/uncoder-core/app/translator/platforms/logrhythm_axon/const.py @@ -0,0 +1,47 @@ +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.models.platform_details import PlatformDetails + +UNMAPPED_FIELD_DEFAULT_NAME = "general_information.raw_message" + +DEFAULT_LOGRHYTHM_AXON_RULE = { + "title": "Default LogRhythm Axon rule", + "version": 3, + "description": "Default LogRhythm Axon rule description.", + "observationPipeline": { + "pattern": { + "operations": [ + { + "touched": True, + "blockType": "LOG_OBSERVED", + "logObserved": {"filter": "query", "groupByFields": []}, + "operationType": "WHERE_PATTERN_OPERATION", + "isOutOfBoxRule": False, + "ruleElementKey": "rule_id", + } + ], + "afterMatchSkipStrategy": "SKIP_PAST_LAST_EVENT", + }, + "commonEvents": ["28de4ee0-ca58-40f5-9ac7-ca38edf7883a", "348a37e6-590e-4767-baae-a5c3951391ae"], + "metadataFields": {"threat.severity": SeverityType.medium}, + }, +} + +PLATFORM_DETAILS = {"group_id": "axon-ads", "group_name": "LogRhythm Axon"} + +LOGRHYTHM_AXON_QUERY_DETAILS = { + "platform_id": "axon-ads-query", + "name": "LogRhythm Axon Query", + "platform_name": "Query", + **PLATFORM_DETAILS, +} + +LOGRHYTHM_AXON_RULE_DETAILS = { + "platform_id": "axon-ads-rule", + "name": "LogRhythm Axon Rule", + "platform_name": "Rule", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +logrhythm_axon_query_details = PlatformDetails(**LOGRHYTHM_AXON_QUERY_DETAILS) +logrhythm_axon_rule_details = PlatformDetails(**LOGRHYTHM_AXON_RULE_DETAILS) diff --git a/uncoder-core/app/translator/platforms/logrhythm_axon/escape_manager.py b/uncoder-core/app/translator/platforms/logrhythm_axon/escape_manager.py new file mode 100644 index 00000000..ec576ea6 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logrhythm_axon/escape_manager.py @@ -0,0 +1,37 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class LogRhythmQueryEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [EscapeDetails(pattern=r"'", escape_symbols=r"''")], + ValueType.regex_value: [ + EscapeDetails(pattern=r"\\", escape_symbols=r"\\\\"), + EscapeDetails(pattern=r"\*", escape_symbols=r"\\*"), + EscapeDetails(pattern=r"\.", escape_symbols=r"\\."), + EscapeDetails(pattern=r"\^", escape_symbols=r"\\^"), + EscapeDetails(pattern=r"\$", escape_symbols=r"\\$"), + EscapeDetails(pattern=r"\|", escape_symbols=r"\\|"), + EscapeDetails(pattern=r"\?", escape_symbols=r"\\?"), + EscapeDetails(pattern=r"\+", escape_symbols=r"\\+"), + EscapeDetails(pattern=r"\(", escape_symbols=r"\\("), + EscapeDetails(pattern=r"\)", escape_symbols=r"\\)"), + EscapeDetails(pattern=r"\[", escape_symbols=r"\\["), + EscapeDetails(pattern=r"\]", escape_symbols=r"\\]"), + EscapeDetails(pattern=r"\{", escape_symbols=r"\\{"), + EscapeDetails(pattern=r"\}", escape_symbols=r"\\}"), + ], + } + + +class LogRhythmRuleEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [EscapeDetails(pattern=r"'", escape_symbols=r"''")] + } + + +logrhythm_query_escape_manager = LogRhythmQueryEscapeManager() +logrhythm_rule_escape_manager = LogRhythmRuleEscapeManager() diff --git a/uncoder-core/app/translator/platforms/logrhythm_axon/mapping.py b/uncoder-core/app/translator/platforms/logrhythm_axon/mapping.py new file mode 100644 index 00000000..dc70f44e --- /dev/null +++ b/uncoder-core/app/translator/platforms/logrhythm_axon/mapping.py @@ -0,0 +1,41 @@ +from typing import Optional + +from app.translator.core.mapping import DEFAULT_MAPPING_NAME, BasePlatformMappings, LogSourceSignature, SourceMapping +from app.translator.platforms.logrhythm_axon.const import logrhythm_axon_query_details, logrhythm_axon_rule_details + + +class LogRhythmAxonLogSourceSignature(LogSourceSignature): + def __init__(self, default_source: Optional[dict] = None): + self._default_source = default_source or {} + + def is_suitable(self) -> bool: + return True + + def __str__(self) -> str: + return "general_information.log_source.type_name" + + +class LogRhythmAxonMappings(BasePlatformMappings): + is_strict_mapping = True + + def prepare_mapping(self) -> dict[str, SourceMapping]: + source_mappings = {} + for mapping_dict in self._loader.load_platform_mappings(self._platform_dir): + log_source_signature = self.prepare_log_source_signature(mapping=mapping_dict) + fields_mapping = self.prepare_fields_mapping(field_mapping=mapping_dict.get("field_mapping", {})) + source_mappings[DEFAULT_MAPPING_NAME] = SourceMapping( + source_id=DEFAULT_MAPPING_NAME, log_source_signature=log_source_signature, fields_mapping=fields_mapping + ) + return source_mappings + + def prepare_log_source_signature(self, mapping: dict) -> LogRhythmAxonLogSourceSignature: + default_log_source = mapping.get("default_log_source") + return LogRhythmAxonLogSourceSignature(default_source=default_log_source) + + +logrhythm_axon_query_mappings = LogRhythmAxonMappings( + platform_dir="logrhythm_axon", platform_details=logrhythm_axon_query_details +) +logrhythm_axon_rule_mappings = LogRhythmAxonMappings( + platform_dir="logrhythm_axon", platform_details=logrhythm_axon_rule_details +) diff --git a/siem-converter/app/converter/platforms/logscale/__init__.py b/uncoder-core/app/translator/platforms/logrhythm_axon/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/logscale/__init__.py rename to uncoder-core/app/translator/platforms/logrhythm_axon/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/logrhythm_axon/renders/logrhythm_axon_query.py b/uncoder-core/app/translator/platforms/logrhythm_axon/renders/logrhythm_axon_query.py new file mode 100644 index 00000000..c9172b58 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logrhythm_axon/renders/logrhythm_axon_query.py @@ -0,0 +1,269 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.tokens import LogicalOperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.exceptions.core import StrictPlatformException +from app.translator.core.exceptions.render import BaseRenderException +from app.translator.core.mapping import LogSourceSignature, SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import TokenizedQueryContainer +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.managers import render_manager +from app.translator.platforms.logrhythm_axon.const import UNMAPPED_FIELD_DEFAULT_NAME, logrhythm_axon_query_details +from app.translator.platforms.logrhythm_axon.escape_manager import logrhythm_query_escape_manager +from app.translator.platforms.logrhythm_axon.mapping import LogRhythmAxonMappings, logrhythm_axon_query_mappings + + +class LogRhythmRegexRenderException(BaseRenderException): + ... + + +class LogRhythmAxonFieldValueRender(BaseFieldValueRender): + details: PlatformDetails = logrhythm_axon_query_details + escape_manager = logrhythm_query_escape_manager + + def __is_complex_regex(self, regex: str) -> bool: + regex_items = ("[", "]", "(", ")", "{", "}", "+", "?", "^", "$", "\\d", "\\w", "\\s", "-") + return any(v in regex for v in regex_items) + + def __is_contain_regex_items(self, value: str) -> bool: + regex_items = ("[", "]", "(", ")", "{", "}", "*", "+", "?", "^", "$", "|", ".", "\\d", "\\w", "\\s", "\\", "-") + return any(v in value for v in regex_items) + + def __regex_to_str_list(self, value: Union[int, str]) -> list[list[str]]: # noqa: PLR0912 + value_groups = [] + + stack = [] # [(element: str, escaped: bool)] + + for char in value: + if char == "\\": + if stack and stack[-1][0] == "\\" and stack[-1][1] is False: + stack.pop() + stack.append((char, True)) + else: + stack.append(("\\", False)) + elif char == "|": + if stack and stack[-1][0] == "\\" and stack[-1][1] is False: + stack.pop() + stack.append((char, True)) + elif stack: + value_groups.append("".join(element[0] for element in stack)) + stack = [] + else: + stack.append((char, False)) + if stack: + value_groups.append("".join(element[0] for element in stack if element[0] != "\\" or element[-1] is True)) + + joined_components = [] + for value_group in value_groups: + inner_joined_components = [] + not_joined_components = [] + for i in range(len(value_group)): + if value_group[i] == "*" and i > 0 and value_group[i - 1] != "\\": + inner_joined_components.append("".join(not_joined_components)) + not_joined_components = [] + else: + not_joined_components.append(value_group[i]) + if not_joined_components: + inner_joined_components.append("".join(not_joined_components)) + joined_components.append(inner_joined_components) + + return joined_components + + def __unmapped_regex_field_to_contains_string(self, field: str, value: str) -> str: + if self.__is_complex_regex(value): + raise LogRhythmRegexRenderException + values = self.__regex_to_str_list(value) + return ( + "(" + + self.or_token.join( + " AND ".join(f'{field} CONTAINS "{self.apply_value(value)}"' for value in value_list) + for value_list in values + ) + + ")" + ) + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if field == UNMAPPED_FIELD_DEFAULT_NAME: + return self.contains_modifier(field, value) + if isinstance(value, str): + return f'{field} = "{self.apply_value(value)}"' + if isinstance(value, list): + prepared_values = ", ".join(f"{self.apply_value(v)}" for v in value) + operator = "IN" if all(isinstance(v, str) for v in value) else "in" + return f"{field} {operator} [{prepared_values}]" + return f'{field} = "{self.apply_value(value)}"' + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + if field == UNMAPPED_FIELD_DEFAULT_NAME: + return self.contains_modifier(field, value) + if isinstance(value, int): + return f"{field} < {value}" + return f"{field} < '{self.apply_value(value)}'" + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + if field == UNMAPPED_FIELD_DEFAULT_NAME: + return self.contains_modifier(field, value) + if isinstance(value, int): + return f"{field} <= {value}" + return f"{field} <= {self.apply_value(value)}" + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + if field == UNMAPPED_FIELD_DEFAULT_NAME: + return self.contains_modifier(field, value) + if isinstance(value, int): + return f"{field} > {value}" + return f"{field} > {self.apply_value(value)}" + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + if field == UNMAPPED_FIELD_DEFAULT_NAME: + return self.contains_modifier(field, value) + if isinstance(value, int): + return f"{field} >= {value}" + return f"{field} >= {self.apply_value(value)}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if field == UNMAPPED_FIELD_DEFAULT_NAME: + return self.contains_modifier(field, value) + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + if isinstance(value, int): + return f"{field} != {value}" + return f"{field} != {self.apply_value(value)}" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + return f'{field} CONTAINS "{self.apply_value(value)}"' + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + if isinstance(value, str) and field == UNMAPPED_FIELD_DEFAULT_NAME: + return self.contains_modifier(field, value) + applied_value = self.apply_value(value, value_type=ValueType.regex_value) + return f'{field} matches ".*{applied_value}$"' + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" + if isinstance(value, str) and field == UNMAPPED_FIELD_DEFAULT_NAME: + return self.contains_modifier(field, value) + applied_value = self.apply_value(value, value_type=ValueType.regex_value) + return f'{field} matches "^{applied_value}.*"' + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if field == UNMAPPED_FIELD_DEFAULT_NAME and self.__is_contain_regex_items(value): + if isinstance(value, str): + return self.__unmapped_regex_field_to_contains_string(field, value) + if isinstance(value, list): + return self.or_token.join( + self.__unmapped_regex_field_to_contains_string(field=field, value=v) for v in value + ) + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + if isinstance(value, str) and field == UNMAPPED_FIELD_DEFAULT_NAME: + return self.contains_modifier(field, value) + return f'{field} matches "{value}"' + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + if isinstance(value, list): + rendered_keywords = [f'{UNMAPPED_FIELD_DEFAULT_NAME} CONTAINS "{v}"' for v in value] + return f"({self.or_token.join(rendered_keywords)})" + return f'{UNMAPPED_FIELD_DEFAULT_NAME} CONTAINS "{value}"' + + +@render_manager.register +class LogRhythmAxonQueryRender(PlatformQueryRender): + details: PlatformDetails = logrhythm_axon_query_details + + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + field_value_render = LogRhythmAxonFieldValueRender(or_token=or_token) + + mappings: LogRhythmAxonMappings = logrhythm_axon_query_mappings + comment_symbol = "//" + is_single_line_comment = True + + @staticmethod + def _finalize_search_query(query: str) -> str: + return f"AND {query}" if query else "" + + def generate_prefix(self, log_source_signature: LogSourceSignature, functions_prefix: str = "") -> str: # noqa: ARG002 + return str(log_source_signature) + + def apply_token(self, token: QUERY_TOKEN_TYPE, source_mapping: SourceMapping) -> str: + if isinstance(token, FieldValue) and token.field: + try: + mapped_fields = self.mappings.map_field(token.field, source_mapping) + except StrictPlatformException: + try: + return self.field_value_render.apply_field_value( + field=UNMAPPED_FIELD_DEFAULT_NAME, operator=token.operator, value=token.value + ) + except LogRhythmRegexRenderException as exc: + raise LogRhythmRegexRenderException( + f"Uncoder does not support complex regexp for unmapped field:" + f" {token.field.source_name} for LogRhythm Axon" + ) from exc + joined = self.logical_operators_map[LogicalOperatorType.OR].join( + [ + self.field_value_render.apply_field_value(field=field, operator=token.operator, value=token.value) + for field in mapped_fields + ] + ) + return self.group_token % joined if len(mapped_fields) > 1 else joined + + return super().apply_token(token, source_mapping) + + def _generate_from_tokenized_query_container_by_source_mapping( + self, query_container: TokenizedQueryContainer, source_mapping: SourceMapping + ) -> str: + unmapped_fields = self.mappings.check_fields_mapping_existence( + query_container.meta_info.query_fields, + query_container.meta_info.function_fields_map, + self.platform_functions.manager.supported_render_names, + source_mapping, + ) + prefix = self.generate_prefix(source_mapping.log_source_signature) + if "product" in query_container.meta_info.parsed_logsources: + prefix = f"{prefix} CONTAINS {query_container.meta_info.parsed_logsources['product'][0]}" + else: + prefix = f"{prefix} CONTAINS anything" + + result = self.generate_query(tokens=query_container.tokens, source_mapping=source_mapping) + rendered_functions = self.generate_functions(query_container.functions.functions, source_mapping) + not_supported_functions = query_container.functions.not_supported + rendered_functions.not_supported + return self.finalize_query( + prefix=prefix, + query=result, + functions=rendered_functions.rendered, + not_supported_functions=not_supported_functions, + unmapped_fields=unmapped_fields, + meta_info=query_container.meta_info, + source_mapping=source_mapping, + ) diff --git a/uncoder-core/app/translator/platforms/logrhythm_axon/renders/logrhythm_axon_rule.py b/uncoder-core/app/translator/platforms/logrhythm_axon/renders/logrhythm_axon_rule.py new file mode 100644 index 00000000..78a87c67 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logrhythm_axon/renders/logrhythm_axon_rule.py @@ -0,0 +1,101 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy +import json +from typing import Optional + +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer +from app.translator.managers import render_manager +from app.translator.platforms.logrhythm_axon.const import DEFAULT_LOGRHYTHM_AXON_RULE, logrhythm_axon_rule_details +from app.translator.platforms.logrhythm_axon.escape_manager import logrhythm_rule_escape_manager +from app.translator.platforms.logrhythm_axon.mapping import LogRhythmAxonMappings, logrhythm_axon_rule_mappings +from app.translator.platforms.logrhythm_axon.renders.logrhythm_axon_query import ( + LogRhythmAxonFieldValueRender, + LogRhythmAxonQueryRender, +) +from app.translator.tools.utils import get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated LogRhythm Axon Rule" +_SEVERITIES_MAP = { + SeverityType.critical: SeverityType.critical, + SeverityType.high: SeverityType.high, + SeverityType.medium: SeverityType.medium, + SeverityType.low: SeverityType.low, + SeverityType.informational: SeverityType.low, +} + + +class LogRhythmAxonRuleFieldValueRender(LogRhythmAxonFieldValueRender): + details: PlatformDetails = logrhythm_axon_rule_details + escape_manager = logrhythm_rule_escape_manager + + +@render_manager.register +class LogRhythmAxonRuleRender(LogRhythmAxonQueryRender): + details: PlatformDetails = logrhythm_axon_rule_details + mappings: LogRhythmAxonMappings = logrhythm_axon_rule_mappings + or_token = "or" + field_value_render = LogRhythmAxonRuleFieldValueRender(or_token=or_token) + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = copy.deepcopy(DEFAULT_LOGRHYTHM_AXON_RULE) + rule["observationPipeline"]["pattern"]["operations"][0]["logObserved"]["filter"] = query + rule["title"] = meta_info.title or _AUTOGENERATED_TEMPLATE + rule["description"] = get_rule_description_str( + description=meta_info.description or rule["description"] or _AUTOGENERATED_TEMPLATE, + author=meta_info.author, + license_=meta_info.license, + ) + rule["observationPipeline"]["pattern"]["operations"][0]["ruleElementKey"] = meta_info.id + rule["observationPipeline"]["metadataFields"]["threat.severity"] = _SEVERITIES_MAP.get( + meta_info.severity, SeverityType.medium + ) + if mitre_info := meta_info.mitre_attack: + if tactics := mitre_info.tactics: + rule["observationPipeline"]["metadataFields"]["threat.mitre_tactic"] = ", ".join( + f"{i.external_id}:{i.name}" for i in sorted(tactics, key=lambda x: x.external_id) + ) + if techniques := mitre_info.techniques: + rule["observationPipeline"]["metadataFields"]["threat.mitre_technique"] = ", ".join( + f"{i.technique_id}:{i.name}" for i in sorted(techniques, key=lambda x: x.technique_id) + ) + if meta_info.output_table_fields: + rule["observationPipeline"]["pattern"]["operations"][0]["logObserved"]["groupByFields"] = [ + self.mappings.map_field(field, source_mapping)[0] for field in meta_info.output_table_fields + ] + + json_rule = json.dumps(rule, indent=4, sort_keys=False) + json_rule = self.wrap_with_unmapped_fields(json_rule, unmapped_fields) + return self.wrap_with_not_supported_functions(json_rule, not_supported_functions) diff --git a/uncoder-core/app/translator/platforms/logscale/__init__.py b/uncoder-core/app/translator/platforms/logscale/__init__.py new file mode 100644 index 00000000..54f3f3db --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/__init__.py @@ -0,0 +1,5 @@ +from app.translator.platforms.logscale.parsers.logscale import LogScaleQueryParser # noqa: F401 +from app.translator.platforms.logscale.parsers.logscale_alert import LogScaleAlertParser # noqa: F401 +from app.translator.platforms.logscale.renders.logscale import LogScaleQueryRender # noqa: F401 +from app.translator.platforms.logscale.renders.logscale_alert import LogScaleAlertRender # noqa: F401 +from app.translator.platforms.logscale.renders.logscale_cti import LogScaleCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/logscale/const.py b/uncoder-core/app/translator/platforms/logscale/const.py new file mode 100644 index 00000000..efc05c46 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/const.py @@ -0,0 +1,43 @@ +from app.translator.core.models.platform_details import PlatformDetails + +DEFAULT_LOGSCALE_ALERT = { + "name": "", + "query": {"queryString": "", "isLive": True, "start": "1h"}, + "description": "", + "throttleTimeMillis": 60000, + "silenced": False, +} + +PLATFORM_DETAILS = {"group_id": "logscale-pack", "group_name": "Falcon LogScale"} + +LOGSCALE_QUERY_DETAILS = { + "platform_id": "logscale-lql-query", + "name": "Falcon LogScale Query", + "platform_name": "Query", + **PLATFORM_DETAILS, +} + +LOGSCALE_ALERT_DETAILS = { + "platform_id": "logscale-lql-rule", + "name": "Falcon LogScale Alert", + "platform_name": "Alert", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +DEFAULT_LOGSCALE_CTI_MAPPING = { + "DestinationIP": "dst_ip", + "SourceIP": "src_ip", + "HashSha512": "file.hash.sha512", + "HashSha256": "file.hash.sha256", + "HashMd5": "file.hash.md5", + "Emails": "email", + "Domain": "host", + "HashSha1": "file.hash.sha1", + "Files": "winlog.event_data.TargetFilename", + "URL": "url", +} + + +logscale_query_details = PlatformDetails(**LOGSCALE_QUERY_DETAILS) +logscale_alert_details = PlatformDetails(**LOGSCALE_ALERT_DETAILS) diff --git a/uncoder-core/app/translator/platforms/logscale/escape_manager.py b/uncoder-core/app/translator/platforms/logscale/escape_manager.py new file mode 100644 index 00000000..2609e639 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/escape_manager.py @@ -0,0 +1,14 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class LogscaleEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [EscapeDetails(pattern='(\\\\|/|\+|{|\[|\?|\*|"|\(|\))')] + } + + +logscale_escape_manager = LogscaleEscapeManager() diff --git a/uncoder-core/app/translator/platforms/logscale/functions/__init__.py b/uncoder-core/app/translator/platforms/logscale/functions/__init__.py new file mode 100644 index 00000000..2ee0344c --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/functions/__init__.py @@ -0,0 +1,47 @@ +import os.path +import re + +from app.translator.core.exceptions.functions import InvalidFunctionSignature, NotSupportedFunctionException +from app.translator.core.functions import PlatformFunctions +from app.translator.core.models.functions.base import ParsedFunctions +from app.translator.platforms.logscale.functions.const import LogScaleFunctionType +from app.translator.platforms.logscale.functions.manager import LogScaleFunctionsManager, log_scale_functions_manager + + +class LogScaleFunctions(PlatformFunctions): + dir_path: str = os.path.abspath(os.path.dirname(__file__)) + manager: LogScaleFunctionsManager = log_scale_functions_manager + + def parse(self, query: str) -> tuple[ParsedFunctions, str]: + parsed = [] + not_supported = [] + invalid = [] + functions = query.split(self.function_delimiter) + query_part = "" + for i, func in enumerate(functions): + if not (func_name_match := re.search(r"(?P[a-zA-Z:]+)\(", func)): + if i == 0: + query_part = func + continue + + func = f"{LogScaleFunctionType.search}({func})" + func_name_match = re.search(r"(?P[a-zA-Z:]+)\(", func) + func_name = func_name_match.group("func_name") + func = func.strip() + func_body = func[len(func_name) + 1 : len(func) - 1] + try: + func_parser = self.manager.get_hof_parser(func_name) + parsed.append(func_parser.parse(func_body, func)) + except NotSupportedFunctionException: + not_supported.append(func) + except InvalidFunctionSignature: + invalid.append(func) + + return ParsedFunctions( + not_supported=[self.wrap_function_with_delimiter(func) for func in not_supported], + functions=parsed, + invalid=invalid, + ), query_part + + +log_scale_functions = LogScaleFunctions() diff --git a/uncoder-core/app/translator/platforms/logscale/functions/const.py b/uncoder-core/app/translator/platforms/logscale/functions/const.py new file mode 100644 index 00000000..d9f1dee9 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/functions/const.py @@ -0,0 +1,13 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class LogScaleFunctionType(CustomEnum): + avg = "avg" + count = "count" + group_by = "groupBy" + max = "max" + min = "min" + search = "search" + sort = "sort" + sum = "sum" + table = "table" diff --git a/uncoder-core/app/translator/platforms/logscale/functions/manager.py b/uncoder-core/app/translator/platforms/logscale/functions/manager.py new file mode 100644 index 00000000..1211ee3d --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/functions/manager.py @@ -0,0 +1,8 @@ +from app.translator.core.functions import PlatformFunctionsManager + + +class LogScaleFunctionsManager(PlatformFunctionsManager): + ... + + +log_scale_functions_manager = LogScaleFunctionsManager() diff --git a/uncoder-core/app/translator/platforms/logscale/mapping.py b/uncoder-core/app/translator/platforms/logscale/mapping.py new file mode 100644 index 00000000..2ca91e99 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/mapping.py @@ -0,0 +1,25 @@ +from typing import Optional + +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature +from app.translator.platforms.logscale.const import logscale_alert_details, logscale_query_details + + +class LogScaleLogSourceSignature(LogSourceSignature): + def __init__(self, default_source: Optional[dict] = None): + self._default_source = default_source or {} + + def __str__(self) -> str: + return " ".join((f"{key}={value}" for key, value in self._default_source.items() if value)) + + def is_suitable(self) -> bool: + return True + + +class LogScaleMappings(BasePlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> LogScaleLogSourceSignature: + default_log_source = mapping.get("default_log_source") + return LogScaleLogSourceSignature(default_source=default_log_source) + + +logscale_query_mappings = LogScaleMappings(platform_dir="logscale", platform_details=logscale_query_details) +logscale_alert_mappings = LogScaleMappings(platform_dir="logscale", platform_details=logscale_alert_details) diff --git a/siem-converter/app/converter/platforms/logscale/mappings/__init__.py b/uncoder-core/app/translator/platforms/logscale/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/logscale/mappings/__init__.py rename to uncoder-core/app/translator/platforms/logscale/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/logscale/parsers/logscale.py b/uncoder-core/app/translator/platforms/logscale/parsers/logscale.py new file mode 100644 index 00000000..ddf2fcd1 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/parsers/logscale.py @@ -0,0 +1,55 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.functions.base import ParsedFunctions +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser +from app.translator.managers import parser_manager +from app.translator.platforms.logscale.const import logscale_query_details +from app.translator.platforms.logscale.functions import LogScaleFunctions, log_scale_functions +from app.translator.platforms.logscale.mapping import LogScaleMappings, logscale_query_mappings +from app.translator.platforms.logscale.tokenizer import LogScaleTokenizer + + +@parser_manager.register_supported_by_roota +class LogScaleQueryParser(PlatformQueryParser): + details: PlatformDetails = logscale_query_details + platform_functions: LogScaleFunctions = log_scale_functions + tokenizer = LogScaleTokenizer() + mappings: LogScaleMappings = logscale_query_mappings + + wrapped_with_comment_pattern = r"^\s*/\*(?:|\n|.)*\*/" + + def _parse_query(self, query: str) -> tuple[str, ParsedFunctions]: + functions, query = self.platform_functions.parse(query) + return query, functions + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + query, functions = self._parse_query(query=raw_query_container.query) + query_tokens = self.get_query_tokens(query) + query_field_tokens, function_field_tokens, function_field_tokens_map = self.get_field_tokens( + query_tokens, functions.functions + ) + source_mappings = self.get_source_mappings(query_field_tokens + function_field_tokens, {}) + meta_info = raw_query_container.meta_info + meta_info.query_fields = query_field_tokens + meta_info.function_fields = function_field_tokens + meta_info.function_fields_map = function_field_tokens_map + meta_info.source_mapping_ids = [source_mapping.source_id for source_mapping in source_mappings] + return TokenizedQueryContainer(tokens=query_tokens, meta_info=meta_info, functions=functions) diff --git a/uncoder-core/app/translator/platforms/logscale/parsers/logscale_alert.py b/uncoder-core/app/translator/platforms/logscale/parsers/logscale_alert.py new file mode 100644 index 00000000..3e89c093 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/parsers/logscale_alert.py @@ -0,0 +1,47 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.mixins.rule import JsonRuleMixin +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, RawQueryContainer +from app.translator.managers import parser_manager +from app.translator.platforms.logscale.const import logscale_alert_details +from app.translator.platforms.logscale.mapping import LogScaleMappings, logscale_alert_mappings +from app.translator.platforms.logscale.parsers.logscale import LogScaleQueryParser +from app.translator.tools.utils import parse_rule_description_str + + +@parser_manager.register +class LogScaleAlertParser(LogScaleQueryParser, JsonRuleMixin): + details: PlatformDetails = logscale_alert_details + mappings: LogScaleMappings = logscale_alert_mappings + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + rule = self.load_rule(text=text) + parsed_description = parse_rule_description_str(rule["description"]) + return RawQueryContainer( + query=rule["query"]["queryString"], + language=language, + meta_info=MetaInfoContainer( + id_=parsed_description.get("rule_id"), + author=parsed_description.get("author"), + references=parsed_description.get("references"), + title=rule.get("name"), + description=parsed_description.get("description") or rule.get("description"), + ), + ) diff --git a/siem-converter/app/converter/platforms/logscale/parsers/__init__.py b/uncoder-core/app/translator/platforms/logscale/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/logscale/parsers/__init__.py rename to uncoder-core/app/translator/platforms/logscale/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/logscale/renders/logscale.py b/uncoder-core/app/translator/platforms/logscale/renders/logscale.py new file mode 100644 index 00000000..1ca23243 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/renders/logscale.py @@ -0,0 +1,110 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.managers import render_manager +from app.translator.platforms.logscale.const import logscale_query_details +from app.translator.platforms.logscale.escape_manager import logscale_escape_manager +from app.translator.platforms.logscale.functions import LogScaleFunctions, log_scale_functions +from app.translator.platforms.logscale.mapping import LogScaleMappings, logscale_query_mappings + + +class LogScaleFieldValueRender(BaseFieldValueRender): + details: PlatformDetails = logscale_query_details + escape_manager = logscale_escape_manager + + def apply_field_name(self, field_name: str) -> str: + if not field_name.isalpha(): + return f'"{field_name}"' + return field_name + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.equal_modifier(field=field, value=v) for v in value)})" + if value == "": + return f'{self.apply_field_name(field_name=field)}=""' + if value == "*": + return f"{self.apply_field_name(field_name=field)}=/^/i" + return f"{self.apply_field_name(field_name=field)}=/{self.apply_value(value)}/i" + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{self.apply_field_name(field_name=field)}<"{self.apply_value(value)}"' + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{self.apply_field_name(field_name=field)}<="{self.apply_value(value)}"' + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{self.apply_field_name(field_name=field)}>"{self.apply_value(value)}"' + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{self.apply_field_name(field_name=field)}>="{self.apply_value(value)}"' + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + return f"{self.apply_field_name(field_name=field)}!=/{self.apply_value(value)}/i" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + return f"{self.apply_field_name(field_name=field)}=/{self.apply_value(value)}/i" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + return f"{self.apply_field_name(field_name=field)}=/{self.apply_value(value)}$/i" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" + return f"{self.apply_field_name(field_name=field)}=/^{self.apply_value(value)}/i" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + return f"{self.apply_field_name(field_name=field)}=/{self.apply_value(value)}/" + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" + return f"/{self.apply_value(value)}/i" + + +@render_manager.register +class LogScaleQueryRender(PlatformQueryRender): + details: PlatformDetails = logscale_query_details + mappings: LogScaleMappings = logscale_query_mappings + platform_functions: LogScaleFunctions = None + + or_token = "or" + and_token = "" + not_token = "not" + + field_value_render = LogScaleFieldValueRender(or_token=or_token) + + def init_platform_functions(self) -> None: + self.platform_functions = log_scale_functions + self.platform_functions.platform_query_render = self + + def wrap_with_comment(self, value: str) -> str: + return f"/* {value} */" diff --git a/uncoder-core/app/translator/platforms/logscale/renders/logscale_alert.py b/uncoder-core/app/translator/platforms/logscale/renders/logscale_alert.py new file mode 100644 index 00000000..0c184a44 --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/renders/logscale_alert.py @@ -0,0 +1,76 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy +import json +from typing import Optional + +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer +from app.translator.managers import render_manager +from app.translator.platforms.logscale.const import DEFAULT_LOGSCALE_ALERT, logscale_alert_details +from app.translator.platforms.logscale.mapping import LogScaleMappings, logscale_alert_mappings +from app.translator.platforms.logscale.renders.logscale import LogScaleFieldValueRender, LogScaleQueryRender +from app.translator.tools.utils import get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated Falcon LogScale Alert" + + +class LogScaleAlertFieldValueRender(LogScaleFieldValueRender): + details: PlatformDetails = logscale_alert_details + + +@render_manager.register +class LogScaleAlertRender(LogScaleQueryRender): + details: PlatformDetails = logscale_alert_details + mappings: LogScaleMappings = logscale_alert_mappings + or_token = "or" + field_value_render = LogScaleAlertFieldValueRender(or_token=or_token) + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = copy.deepcopy(DEFAULT_LOGSCALE_ALERT) + rule["query"]["queryString"] = query + rule["name"] = meta_info.title or _AUTOGENERATED_TEMPLATE + mitre_attack = [] + if meta_info.mitre_attack: + mitre_attack = sorted([f"ATTACK.{i.name}" for i in meta_info.mitre_attack.tactics]) + mitre_attack.extend(sorted([f"ATTACK.{i.technique_id}" for i in meta_info.mitre_attack.techniques])) + rule["description"] = get_rule_description_str( + description=meta_info.description or _AUTOGENERATED_TEMPLATE, + license_=meta_info.license, + author=meta_info.author, + mitre_attack=mitre_attack, + ) + + rule_str = json.dumps(rule, indent=4, sort_keys=False) + rule_str = self.wrap_with_unmapped_fields(rule_str, unmapped_fields) + return self.wrap_with_not_supported_functions(rule_str, not_supported_functions) diff --git a/uncoder-core/app/translator/platforms/logscale/renders/logscale_cti.py b/uncoder-core/app/translator/platforms/logscale/renders/logscale_cti.py new file mode 100644 index 00000000..cf2e45ad --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/renders/logscale_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.logscale.const import DEFAULT_LOGSCALE_CTI_MAPPING, logscale_query_details + + +@render_cti_manager.register +class LogScaleCTI(RenderCTI): + details: PlatformDetails = logscale_query_details + + field_value_template: str = '{key}="{value}"' + or_operator: str = " or " + group_or_operator: str = " or " + or_group: str = "{or_group}" + result_join: str = "" + final_result_for_many: str = '@stream="http" {result}\n' + final_result_for_one: str = '@stream="http" {result}\n' + default_mapping = DEFAULT_LOGSCALE_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/logscale/tokenizer.py b/uncoder-core/app/translator/platforms/logscale/tokenizer.py new file mode 100644 index 00000000..a96cd0ea --- /dev/null +++ b/uncoder-core/app/translator/platforms/logscale/tokenizer.py @@ -0,0 +1,76 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import Any, ClassVar, Optional + +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.tokens import LogicalOperatorType, OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mixins.logic import ANDLogicOperatorMixin +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.platforms.logscale.escape_manager import logscale_escape_manager +from app.translator.tools.utils import get_match_group + + +class LogScaleTokenizer(QueryTokenizer, ANDLogicOperatorMixin): + single_value_operators_map: ClassVar[dict[str, str]] = { + "=": OperatorType.EQ, + "<=": OperatorType.LTE, + "<": OperatorType.LT, + ">=": OperatorType.GTE, + ">": OperatorType.GT, + "!=": OperatorType.NOT_EQ, + } + + field_pattern = r"(?P[a-zA-Z\._\-]+)" + num_value_pattern = rf"(?P<{ValueType.number_value}>\d+(?:\.\d+)*)\s*" + double_quotes_value_pattern = ( + rf'"(?P<{ValueType.double_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{{\}}\s]|\\\"|\\)*)"\s*' + ) + re_value_pattern = rf"/(?P<{ValueType.regex_value}>[:a-zA-Z\*0-9=+%#\\\-_\,\"\'\.$&^@!\(\)\{{\}}\s?]+)/i?\s*" + _value_pattern = rf"""{num_value_pattern}|{re_value_pattern}|{double_quotes_value_pattern}""" + keyword_pattern = double_quotes_value_pattern + escape_manager = logscale_escape_manager + wildcard_symbol = "*" + + def get_operator_and_value( + self, match: re.Match, mapped_operator: str = OperatorType.EQ, operator: Optional[str] = None + ) -> tuple[str, Any]: + if (num_value := get_match_group(match, group_name=ValueType.number_value)) is not None: + return mapped_operator, num_value + + if (d_q_value := get_match_group(match, group_name=ValueType.double_quotes_value)) is not None: + return mapped_operator, self.escape_manager.remove_escape(d_q_value) + + if (re_value := get_match_group(match, group_name=ValueType.regex_value)) is not None: + return OperatorType.REGEX, re_value + + return super().get_operator_and_value(match, mapped_operator, operator) + + def _get_next_token(self, query: str) -> (list, str): + query = query.strip("\n").strip(" ").strip("\n") + if query.startswith("!"): + return Identifier(token_type=LogicalOperatorType.NOT), query[1:] + + return super()._get_next_token(query) + + def tokenize(self, query: str) -> list[QUERY_TOKEN_TYPE]: + tokens = super().tokenize(query=query) + return self.add_and_token_if_missed(tokens=tokens) diff --git a/uncoder-core/app/translator/platforms/microsoft/__init__.py b/uncoder-core/app/translator/platforms/microsoft/__init__.py new file mode 100644 index 00000000..45fa896b --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/__init__.py @@ -0,0 +1,10 @@ +from app.translator.platforms.microsoft.parsers.microsoft_defender import MicrosoftDefenderQueryParser # noqa: F401 +from app.translator.platforms.microsoft.parsers.microsoft_sentinel import MicrosoftSentinelQueryParser # noqa: F401 +from app.translator.platforms.microsoft.parsers.microsoft_sentinel_rule import ( + MicrosoftSentinelRuleParser, # noqa: F401 +) +from app.translator.platforms.microsoft.renders.microsoft_defender import MicrosoftDefenderQueryRender # noqa: F401 +from app.translator.platforms.microsoft.renders.microsoft_defender_cti import MicrosoftDefenderCTI # noqa: F401 +from app.translator.platforms.microsoft.renders.microsoft_sentinel import MicrosoftSentinelQueryRender # noqa: F401 +from app.translator.platforms.microsoft.renders.microsoft_sentinel_cti import MicrosoftSentinelCTI # noqa: F401 +from app.translator.platforms.microsoft.renders.microsoft_sentinel_rule import MicrosoftSentinelRuleRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/microsoft/const.py b/uncoder-core/app/translator/platforms/microsoft/const.py new file mode 100644 index 00000000..ca05bd7b --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/const.py @@ -0,0 +1,88 @@ +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.models.platform_details import PlatformDetails + +DEFAULT_MICROSOFT_SENTINEL_RULE = { + "displayName": "Autogenerated Microsoft Sentinel Rule", + "description": "Autogenerated Microsoft Sentinel Rule", + "severity": SeverityType.medium, + "enabled": True, + "query": "", + "queryFrequency": "PT30M", + "queryPeriod": "PT30M", + "triggerOperator": "GreaterThan", + "triggerThreshold": 0, + "suppressionDuration": "PT2H30M", + "suppressionEnabled": True, + "tactics": [], + "techniques": [], +} + +PLATFORM_DETAILS = {"group_id": "sentinel", "group_name": "Microsoft Sentinel"} + +_SENTINEL_KQL_QUERY = "sentinel-kql-query" +_SENTINEL_KQL_RULE = "sentinel-kql-rule" + +MICROSOFT_SENTINEL_QUERY_DETAILS = { + "platform_id": _SENTINEL_KQL_QUERY, + "name": "Microsoft Sentinel Query", + "platform_name": "Query (Kusto)", + **PLATFORM_DETAILS, +} + +MICROSOFT_SENTINEL_RULE_DETAILS = { + "platform_id": _SENTINEL_KQL_RULE, + "name": "Microsoft Sentinel Rule", + "platform_name": "Rule (Kusto)", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +MICROSOFT_SENTINEL_YAML_RULE_DETAILS = { + "platform_id": "sentinel-kql-yaml-rule", + "name": "Microsoft Sentinel YAML Rule", + "platform_name": "YAML Rule (Kusto)", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +MICROSOFT_DEFENDER_DETAILS = { + "platform_id": "mde-kql-query", + "group_name": "Microsoft Defender for Endpoint", + "name": "Microsoft Defender for Endpoint", + "platform_name": "Query (Kusto)", + "group_id": "microsoft-defender", +} + + +DEFAULT_MICROSOFT_DEFENDER_CTI_MAPPING = { + "DestinationIP": "RemoteIP", + "SourceIP": "LocalIP", + "HashSha256": "InitiatingProcessSHA256", + "HashMd5": "InitiatingProcessMD5", + "Emails": "SenderFromAddress", + "Domain": "RemoteUrl", + "HashSha1": "InitiatingProcessSHA1", + "Files": "FileName", + "URL": "RemoteUrl", +} + +DEFAULT_MICROSOFT_SENTINEL_CTI_MAPPING = { + "DestinationIP": "DestinationIp", + "SourceIP": "SourceIp", + "HashSha512": "FileHashSha512", + "HashSha256": "FileHashSha256", + "HashMd5": "FileHashMd5", + "Emails": "SenderFromAddress", + "Domain": "DestinationHostname", + "HashSha1": "FileHashSha1", + "Files": "TargetFileName", + "URL": "URL", +} + + +MICROSOFT_SENTINEL_QUERY_TYPES = {_SENTINEL_KQL_QUERY, _SENTINEL_KQL_RULE} + +microsoft_defender_query_details = PlatformDetails(**MICROSOFT_DEFENDER_DETAILS) +microsoft_sentinel_query_details = PlatformDetails(**MICROSOFT_SENTINEL_QUERY_DETAILS) +microsoft_sentinel_rule_details = PlatformDetails(**MICROSOFT_SENTINEL_RULE_DETAILS) +microsoft_sentinel_yaml_rule_details = PlatformDetails(**MICROSOFT_SENTINEL_YAML_RULE_DETAILS) diff --git a/siem-converter/app/converter/platforms/logscale/renders/__init__.py b/uncoder-core/app/translator/platforms/microsoft/custom_types/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/logscale/renders/__init__.py rename to uncoder-core/app/translator/platforms/microsoft/custom_types/__init__.py diff --git a/uncoder-core/app/translator/platforms/microsoft/custom_types/values.py b/uncoder-core/app/translator/platforms/microsoft/custom_types/values.py new file mode 100644 index 00000000..c51e6a63 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/custom_types/values.py @@ -0,0 +1,6 @@ +from app.translator.core.custom_types.values import ValueType + + +class MicrosoftValueType(ValueType): + verbatim_double_quotes_value = "v_d_q_value" + verbatim_single_quotes_value = "v_s_q_value" diff --git a/uncoder-core/app/translator/platforms/microsoft/escape_manager.py b/uncoder-core/app/translator/platforms/microsoft/escape_manager.py new file mode 100644 index 00000000..cd562cb9 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/escape_manager.py @@ -0,0 +1,12 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class MicrosoftEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = {ValueType.value: [EscapeDetails(pattern='(?:\\\\)?(")')]} + + +microsoft_escape_manager = MicrosoftEscapeManager() diff --git a/uncoder-core/app/translator/platforms/microsoft/functions/__init__.py b/uncoder-core/app/translator/platforms/microsoft/functions/__init__.py new file mode 100644 index 00000000..e0742815 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/functions/__init__.py @@ -0,0 +1,60 @@ +import os.path + +from app.translator.core.exceptions.functions import InvalidFunctionSignature, NotSupportedFunctionException +from app.translator.core.functions import PlatformFunctions +from app.translator.core.models.functions.base import ParsedFunctions +from app.translator.platforms.microsoft.functions.const import KQLFunctionType +from app.translator.platforms.microsoft.functions.manager import ( + MicrosoftFunctionsManager, + microsoft_defender_functions_manager, + microsoft_sentinel_functions_manager, +) + + +class MicrosoftFunctions(PlatformFunctions): + dir_path: str = os.path.abspath(os.path.dirname(__file__)) + + def parse(self, query: str) -> tuple[str, str, ParsedFunctions]: + parsed = [] + not_supported = [] + invalid = [] + split_query = query.split(self.function_delimiter) + table = split_query[0].strip() + query_parts = [] + for func in split_query[1:]: + func = func.strip() + split_func = func.split(" ") + func_name, func_body = split_func[0], " ".join(split_func[1:]) + if func_name == KQLFunctionType.where: + query_parts.append(func_body) + continue + + try: + func_parser = self.manager.get_hof_parser(func_name) + parsed.append(func_parser.parse(func_body, func)) + except NotSupportedFunctionException: + not_supported.append(func) + except InvalidFunctionSignature: + invalid.append(func) + result_query = " and ".join(f"({query_part})" for query_part in query_parts) + return ( + table, + result_query, + ParsedFunctions( + functions=parsed, + not_supported=[self.wrap_function_with_delimiter(func) for func in not_supported], + invalid=invalid, + ), + ) + + +class MicrosoftSentinelFunctions(MicrosoftFunctions): + manager: MicrosoftFunctionsManager = microsoft_sentinel_functions_manager + + +class MicrosoftDefenderFunctions(MicrosoftFunctions): + manager: MicrosoftFunctionsManager = microsoft_defender_functions_manager + + +microsoft_sentinel_functions = MicrosoftSentinelFunctions() +microsoft_defender_functions = MicrosoftDefenderFunctions() diff --git a/uncoder-core/app/translator/platforms/microsoft/functions/const.py b/uncoder-core/app/translator/platforms/microsoft/functions/const.py new file mode 100644 index 00000000..7e849bbf --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/functions/const.py @@ -0,0 +1,24 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class KQLFunctionType(CustomEnum): + avg = "avg" + count = "count" + distinct = "distinct" + distinct_count = "count_distinct" + extend = "extend" + max = "max" + min = "min" + project = "project" + project_rename = "project-rename" + search = "search" + sort = "sort" + sum = "sum" + summarize = "summarize" + top = "top" + where = "where" + + +class KQLSortOrderType(CustomEnum): + asc = "asc" + desc = "desc" diff --git a/uncoder-core/app/translator/platforms/microsoft/functions/manager.py b/uncoder-core/app/translator/platforms/microsoft/functions/manager.py new file mode 100644 index 00000000..93d486f8 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/functions/manager.py @@ -0,0 +1,9 @@ +from app.translator.core.functions import PlatformFunctionsManager + + +class MicrosoftFunctionsManager(PlatformFunctionsManager): + ... + + +microsoft_sentinel_functions_manager = MicrosoftFunctionsManager() +microsoft_defender_functions_manager = MicrosoftFunctionsManager() diff --git a/uncoder-core/app/translator/platforms/microsoft/mapping.py b/uncoder-core/app/translator/platforms/microsoft/mapping.py new file mode 100644 index 00000000..2ad307b6 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/mapping.py @@ -0,0 +1,53 @@ +from typing import Optional + +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature +from app.translator.platforms.microsoft.const import ( + microsoft_defender_query_details, + microsoft_sentinel_query_details, + microsoft_sentinel_rule_details, +) + + +class MicrosoftSentinelLogSourceSignature(LogSourceSignature): + def __init__(self, tables: Optional[list[str]], default_source: dict): + self.tables = set(tables or []) + self._default_source = default_source or {} + + def is_suitable(self, table: Optional[list[str]] = None) -> bool: + return self._check_conditions([set(table).issubset(self.tables) if table else None]) + + def __str__(self) -> str: + return self._default_source.get("table", "") + + +class MicrosoftSentinelMappings(BasePlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> MicrosoftSentinelLogSourceSignature: + tables = mapping.get("log_source", {}).get("table") + default_log_source = mapping["default_log_source"] + return MicrosoftSentinelLogSourceSignature(tables=tables, default_source=default_log_source) + + +microsoft_sentinel_query_mappings = MicrosoftSentinelMappings( + platform_dir="microsoft_sentinel", platform_details=microsoft_sentinel_query_details +) +microsoft_sentinel_rule_mappings = MicrosoftSentinelMappings( + platform_dir="microsoft_sentinel", platform_details=microsoft_sentinel_rule_details +) + + +class MicrosoftDefenderLogSourceSignature(MicrosoftSentinelLogSourceSignature): + pass + + +class MicrosoftDefenderMappings(MicrosoftSentinelMappings): + is_strict_mapping = True + + def prepare_log_source_signature(self, mapping: dict) -> MicrosoftDefenderLogSourceSignature: + tables = mapping.get("log_source", {}).get("table") + default_log_source = mapping["default_log_source"] + return MicrosoftDefenderLogSourceSignature(tables=tables, default_source=default_log_source) + + +microsoft_defender_query_mappings = MicrosoftDefenderMappings( + platform_dir="microsoft_defender", platform_details=microsoft_defender_query_details +) diff --git a/siem-converter/app/converter/platforms/microsoft/__init__.py b/uncoder-core/app/translator/platforms/microsoft/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/microsoft/__init__.py rename to uncoder-core/app/translator/platforms/microsoft/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/microsoft/parsers/microsoft_defender.py b/uncoder-core/app/translator/platforms/microsoft/parsers/microsoft_defender.py new file mode 100644 index 00000000..99fc551d --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/parsers/microsoft_defender.py @@ -0,0 +1,31 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import parser_manager +from app.translator.platforms.microsoft.const import microsoft_defender_query_details +from app.translator.platforms.microsoft.functions import MicrosoftFunctions, microsoft_defender_functions +from app.translator.platforms.microsoft.mapping import MicrosoftDefenderMappings, microsoft_defender_query_mappings +from app.translator.platforms.microsoft.parsers.microsoft_sentinel import MicrosoftSentinelQueryParser + + +@parser_manager.register_supported_by_roota +class MicrosoftDefenderQueryParser(MicrosoftSentinelQueryParser): + mappings: MicrosoftDefenderMappings = microsoft_defender_query_mappings + details: PlatformDetails = microsoft_defender_query_details + platform_functions: MicrosoftFunctions = microsoft_defender_functions diff --git a/uncoder-core/app/translator/platforms/microsoft/parsers/microsoft_sentinel.py b/uncoder-core/app/translator/platforms/microsoft/parsers/microsoft_sentinel.py new file mode 100644 index 00000000..e7392ea3 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/parsers/microsoft_sentinel.py @@ -0,0 +1,57 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + + +from app.translator.core.models.functions.base import ParsedFunctions +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser +from app.translator.managers import parser_manager +from app.translator.platforms.microsoft.const import microsoft_sentinel_query_details +from app.translator.platforms.microsoft.functions import MicrosoftFunctions, microsoft_sentinel_functions +from app.translator.platforms.microsoft.mapping import MicrosoftSentinelMappings, microsoft_sentinel_query_mappings +from app.translator.platforms.microsoft.tokenizer import MicrosoftSentinelTokenizer + + +@parser_manager.register_supported_by_roota +class MicrosoftSentinelQueryParser(PlatformQueryParser): + platform_functions: MicrosoftFunctions = microsoft_sentinel_functions + mappings: MicrosoftSentinelMappings = microsoft_sentinel_query_mappings + tokenizer = MicrosoftSentinelTokenizer() + details: PlatformDetails = microsoft_sentinel_query_details + + wrapped_with_comment_pattern = r"^\s*//.*(?:\n|$)" + + def _parse_query(self, query: str) -> tuple[str, dict[str, list[str]], ParsedFunctions]: + table, query, functions = self.platform_functions.parse(query) + log_sources = {"table": [table]} + return query, log_sources, functions + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + query, log_sources, functions = self._parse_query(query=raw_query_container.query) + query_tokens = self.get_query_tokens(query) + query_field_tokens, function_field_tokens, function_field_tokens_map = self.get_field_tokens( + query_tokens, functions.functions + ) + source_mappings = self.get_source_mappings(query_field_tokens + function_field_tokens, log_sources) + meta_info = raw_query_container.meta_info + meta_info.query_fields = query_field_tokens + meta_info.function_fields = function_field_tokens + meta_info.function_fields_map = function_field_tokens_map + meta_info.source_mapping_ids = [source_mapping.source_id for source_mapping in source_mappings] + return TokenizedQueryContainer(tokens=query_tokens, meta_info=meta_info, functions=functions) diff --git a/uncoder-core/app/translator/platforms/microsoft/parsers/microsoft_sentinel_rule.py b/uncoder-core/app/translator/platforms/microsoft/parsers/microsoft_sentinel_rule.py new file mode 100644 index 00000000..4a2cf6bf --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/parsers/microsoft_sentinel_rule.py @@ -0,0 +1,159 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from contextlib import suppress +from datetime import timedelta +from typing import Optional, Union + +import isodate +from isodate.isoerror import ISO8601Error + +from app.translator.core.mixins.rule import JsonRuleMixin, YamlRuleMixin +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import ( + MetaInfoContainer, + MitreInfoContainer, + RawMetaInfoContainer, + RawQueryContainer, +) +from app.translator.managers import parser_manager +from app.translator.platforms.microsoft.const import ( + microsoft_sentinel_rule_details, + microsoft_sentinel_yaml_rule_details, +) +from app.translator.platforms.microsoft.mapping import MicrosoftSentinelMappings, microsoft_sentinel_rule_mappings +from app.translator.platforms.microsoft.parsers.microsoft_sentinel import MicrosoftSentinelQueryParser +from app.translator.tools.utils import parse_rule_description_str + + +@parser_manager.register +class MicrosoftSentinelRuleParser(MicrosoftSentinelQueryParser, JsonRuleMixin): + details: PlatformDetails = microsoft_sentinel_rule_details + mappings: MicrosoftSentinelMappings = microsoft_sentinel_rule_mappings + + @staticmethod + def _parse_timeframe(raw_timeframe: Optional[str]) -> Optional[timedelta]: + with suppress(ISO8601Error): + return isodate.parse_duration(raw_timeframe) + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + rule = self.load_rule(text=text) + tags = [] + mitre_attack = self.mitre_config.get_mitre_info( + tactics=[tactic.lower() for tactic in rule.get("tactics", [])], + techniques=[technique.lower() for technique in rule.get("techniques", [])], + ) + + if mitre_attack: + for technique in mitre_attack.techniques: + tags.append(technique.technique_id.lower()) + for tactic in mitre_attack.tactics: + tags.append(tactic.name.lower().replace(" ", "_")) + parsed_description = parse_rule_description_str(rule.get("description", "")) + + return RawQueryContainer( + query=rule["query"], + language=language, + meta_info=MetaInfoContainer( + id_=parsed_description.get("rule_id"), + title=rule.get("displayName"), + description=parsed_description.get("description") or rule.get("description"), + timeframe=self._parse_timeframe(rule.get("queryFrequency", "")), + severity=rule.get("severity", "medium"), + mitre_attack=mitre_attack, + author=parsed_description.get("author") or [rule.get("author")], + license_=parsed_description.get("license"), + tags=tags, + references=parsed_description.get("references"), + ), + ) + + +@parser_manager.register +class MicrosoftSentinelYAMLRuleParser(YamlRuleMixin, MicrosoftSentinelRuleParser): + details: PlatformDetails = microsoft_sentinel_yaml_rule_details + mappings: MicrosoftSentinelMappings = microsoft_sentinel_rule_mappings + + def extract_tags(self, data: Union[dict, list, str]) -> list[str]: + tags = [] + if isinstance(data, dict): + for key, value in data.items(): + tags.extend(self.extract_tags(value)) + elif isinstance(data, list): + for item in data: + tags.extend(self.extract_tags(item)) + elif isinstance(data, str): + tags.append(data) + return tags + + def __get_tags_from_required_data_connectors(self, required_data_connectors: dict) -> list[str]: + return list(self.extract_tags(required_data_connectors)) + + def __get_tags_from_metadata(self, metadata: dict) -> list[str]: + fields_to_process = {} + for k, v in metadata.items(): + if k.lower() != "author": + fields_to_process[k] = v + + return list(self.extract_tags(fields_to_process)) + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + rule = self.load_rule(text=text) + tags = [] + mitre_attack: MitreInfoContainer = self.mitre_config.get_mitre_info( + tactics=[tactic.lower() for tactic in rule.get("tactics", [])], + techniques=[technique.lower() for technique in rule.get("relevantTechniques", [])], + ) + + if mitre_attack: + for technique in mitre_attack.techniques: + tags.append(technique.technique_id.lower()) + for tactic in mitre_attack.tactics: + tags.append(tactic.name.lower().replace(" ", "_")) + + tags.extend(self.__get_tags_from_required_data_connectors(rule.get("requiredDataConnectors", {}))) + tags.extend(self.__get_tags_from_metadata(rule.get("metadata", {}))) + + for tag in rule.get("tags", []): + if isinstance(tag, str): + tags.append(tag) + + timeframe = self._parse_timeframe(rule.get("queryFrequency", "")) + query_period = self._parse_timeframe(rule.get("queryPeriod", "")) + + return RawQueryContainer( + query=rule["query"], + language=language, + meta_info=MetaInfoContainer( + id_=rule.get("id"), + title=rule.get("name"), + description=rule.get("description"), + timeframe=timeframe, + query_period=query_period, + severity=rule.get("severity", "medium").lower(), + mitre_attack=mitre_attack, + author=rule.get("metadata", {}).get("author", {}).get("name", "").split(","), + tags=sorted(set(tags)), + raw_metainfo_container=RawMetaInfoContainer( + trigger_operator=rule.get("triggerOperator"), + trigger_threshold=rule.get("triggerThreshold"), + query_frequency=rule.get("queryFrequency"), + query_period=rule.get("queryPeriod"), + ), + ), + ) diff --git a/siem-converter/app/converter/platforms/microsoft/mappings/__init__.py b/uncoder-core/app/translator/platforms/microsoft/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/microsoft/mappings/__init__.py rename to uncoder-core/app/translator/platforms/microsoft/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_defender.py b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_defender.py new file mode 100644 index 00000000..69953044 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_defender.py @@ -0,0 +1,45 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.microsoft.const import microsoft_defender_query_details +from app.translator.platforms.microsoft.functions import MicrosoftFunctions, microsoft_defender_functions +from app.translator.platforms.microsoft.mapping import MicrosoftDefenderMappings, microsoft_defender_query_mappings +from app.translator.platforms.microsoft.renders.microsoft_sentinel import ( + MicrosoftSentinelFieldValueRender, + MicrosoftSentinelQueryRender, +) + + +class MicrosoftDefenderFieldValueRender(MicrosoftSentinelFieldValueRender): + details: PlatformDetails = microsoft_defender_query_details + + +@render_manager.register +class MicrosoftDefenderQueryRender(MicrosoftSentinelQueryRender): + mappings: MicrosoftDefenderMappings = microsoft_defender_query_mappings + details: PlatformDetails = microsoft_defender_query_details + platform_functions: MicrosoftFunctions = None + or_token = "or" + field_value_render = MicrosoftDefenderFieldValueRender(or_token=or_token) + + def init_platform_functions(self) -> None: + self.platform_functions = microsoft_defender_functions + self.platform_functions.platform_query_render = self diff --git a/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_defender_cti.py b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_defender_cti.py new file mode 100644 index 00000000..40726e4c --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_defender_cti.py @@ -0,0 +1,50 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import ClassVar + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.microsoft.const import ( + DEFAULT_MICROSOFT_DEFENDER_CTI_MAPPING, + microsoft_defender_query_details, +) + + +@render_cti_manager.register +class MicrosoftDefenderCTI(RenderCTI): + details: PlatformDetails = microsoft_defender_query_details + + field_value_templates_map: ClassVar[dict[str, str]] = { + "default": '{key} =~ "{value}"', + "url": '{key} has "{value}"', + } + or_operator: str = " or " + group_or_operator: str = " or " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "union * | where ({result})\n" + final_result_for_one: str = "union * | where {result}\n" + default_mapping = DEFAULT_MICROSOFT_DEFENDER_CTI_MAPPING + + def create_field_value(self, field: str, value: str, generic_field: str) -> str: + if field_value_template := self.field_value_templates_map.get(generic_field): + return field_value_template.format(key=field, value=value) + return self.field_value_templates_map["default"].format(key=field, value=value) diff --git a/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_sentinel.py b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_sentinel.py new file mode 100644 index 00000000..5176eb9b --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_sentinel.py @@ -0,0 +1,150 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.mapping import LogSourceSignature +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import RawQueryContainer +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.managers import render_manager +from app.translator.platforms.microsoft.const import microsoft_sentinel_query_details +from app.translator.platforms.microsoft.escape_manager import microsoft_escape_manager +from app.translator.platforms.microsoft.functions import MicrosoftFunctions, microsoft_sentinel_functions +from app.translator.platforms.microsoft.mapping import MicrosoftSentinelMappings, microsoft_sentinel_query_mappings + + +class MicrosoftSentinelFieldValueRender(BaseFieldValueRender): + details: PlatformDetails = microsoft_sentinel_query_details + escape_manager = microsoft_escape_manager + + @staticmethod + def __escape_value(value: Union[int, str]) -> Union[int, str]: + return value.replace("'", "''") if isinstance(value, str) else value + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, str): + return f"{field} =~ @'{self.__escape_value(value)}'" + if isinstance(value, list): + prepared_values = ", ".join(f"@'{self.__escape_value(v)}'" for v in value) + operator = "in~" if all(isinstance(v, str) for v in value) else "in" + return f"{field} {operator} ({prepared_values})" + return f"{field} == {self.apply_value(value)}" + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + if isinstance(value, int): + return f"{field} < {value}" + return f"{field} < '{self.apply_value(value)}'" + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + if isinstance(value, int): + return f"{field} <= {value}" + return f"{field} <= '{self.apply_value(value)}'" + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + if isinstance(value, int): + return f"{field} > {value}" + return f"{field} > '{self.apply_value(value)}'" + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + if isinstance(value, int): + return f"{field} >= {value}" + return f"{field} >= '{self.apply_value(value)}'" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + if isinstance(value, int): + return f"{field} !~ {value}" + return f"{field} !~ '{self.apply_value(value)}'" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + return f"{field} contains @'{self.__escape_value(value)}'" + + def not_contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.not_contains_modifier(field=field, value=v) for v in value)})" + return f"{field} !contains @'{self.__escape_value(value)}'" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + return f"{field} endswith @'{self.__escape_value(value)}'" + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value)})" + return f"{field} startswith @'{self.__escape_value(value)}'" + + def __regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + return f"{field} matches regex @'(?i){self.__escape_value(value)}'" + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.__regex_modifier(field=field, value=v) for v in value)})" + return f"({self.__regex_modifier(field=field, value=value)})" + + def not_regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"not ({self.or_token.join(self.__regex_modifier(field=field, value=v) for v in value)})" + return f"not ({self.__regex_modifier(field=field, value=value)})" + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.keywords(field=field, value=v) for v in value)})" + return f"* contains @'{self.__escape_value(value)}'" + + def is_none(self, field: str, value: Union[str, int]) -> str: # noqa: ARG002 + return f"isempty({self.apply_value(value)})" + + def is_not_none(self, field: str, value: Union[str, int]) -> str: # noqa: ARG002 + return f"isnotempty({self.apply_value(value)})" + + +@render_manager.register +class MicrosoftSentinelQueryRender(PlatformQueryRender): + details: PlatformDetails = microsoft_sentinel_query_details + platform_functions: MicrosoftFunctions = None + + or_token = "or" + and_token = "and" + not_token = "not" + + field_value_render = MicrosoftSentinelFieldValueRender(or_token=or_token) + + mappings: MicrosoftSentinelMappings = microsoft_sentinel_query_mappings + comment_symbol = "//" + is_single_line_comment = True + + def init_platform_functions(self) -> None: + self.platform_functions = microsoft_sentinel_functions + self.platform_functions.platform_query_render = self + + def generate_prefix(self, log_source_signature: LogSourceSignature, functions_prefix: str = "") -> str: # noqa: ARG002 + return str(log_source_signature) + + @staticmethod + def _finalize_search_query(query: str) -> str: + return f"| where {query}" if query else "" + + def generate_from_raw_query_container(self, query_container: RawQueryContainer) -> str: + return query_container.query diff --git a/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_sentinel_cti.py b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_sentinel_cti.py new file mode 100644 index 00000000..9ac314e8 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_sentinel_cti.py @@ -0,0 +1,40 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.microsoft.const import ( + DEFAULT_MICROSOFT_SENTINEL_CTI_MAPPING, + microsoft_sentinel_query_details, +) + + +@render_cti_manager.register +class MicrosoftSentinelCTI(RenderCTI): + details: PlatformDetails = microsoft_sentinel_query_details + + field_value_template: str = '@"{value}"' + or_operator: str = " or " + group_or_operator: str = " or " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "search ({result})\n" + final_result_for_one: str = "search {result}\n" + default_mapping = DEFAULT_MICROSOFT_SENTINEL_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_sentinel_rule.py b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_sentinel_rule.py new file mode 100644 index 00000000..3fa75c63 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/renders/microsoft_sentinel_rule.py @@ -0,0 +1,138 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy +import json +from typing import Optional + +import isodate + +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, MitreInfoContainer, RawQueryContainer +from app.translator.managers import render_manager +from app.translator.platforms.microsoft.const import DEFAULT_MICROSOFT_SENTINEL_RULE, microsoft_sentinel_rule_details +from app.translator.platforms.microsoft.mapping import MicrosoftSentinelMappings, microsoft_sentinel_rule_mappings +from app.translator.platforms.microsoft.renders.microsoft_sentinel import ( + MicrosoftSentinelFieldValueRender, + MicrosoftSentinelQueryRender, +) +from app.translator.tools.utils import get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated Microsoft Sentinel Rule" +_SEVERITIES_MAP = { + SeverityType.critical: SeverityType.high, + SeverityType.high: SeverityType.high, + SeverityType.medium: SeverityType.medium, + SeverityType.low: SeverityType.low, +} + + +class MicrosoftSentinelRuleFieldValueRender(MicrosoftSentinelFieldValueRender): + details: PlatformDetails = microsoft_sentinel_rule_details + + +@render_manager.register +class MicrosoftSentinelRuleRender(MicrosoftSentinelQueryRender): + details: PlatformDetails = microsoft_sentinel_rule_details + mappings: MicrosoftSentinelMappings = microsoft_sentinel_rule_mappings + or_token = "or" + field_value_render = MicrosoftSentinelRuleFieldValueRender(or_token=or_token) + + def __create_mitre_threat(self, mitre_attack: MitreInfoContainer) -> tuple[list, list]: + tactics = set() + techniques = [] + + for tactic in mitre_attack.tactics: + tactics.add(tactic.name) + + for technique in mitre_attack.techniques: + if technique.tactic: + for tactic in technique.tactic: + tactics.add(tactic) + techniques.append(technique.technique_id) + + return sorted(tactics), sorted(techniques) + + @staticmethod + def get_query_frequency(meta_info: MetaInfoContainer) -> Optional[str]: + if meta_info.timeframe: + return isodate.duration_isoformat(meta_info.timeframe) + if meta_info.raw_metainfo_container: + return meta_info.raw_metainfo_container.query_frequency + + @staticmethod + def get_query_period(meta_info: MetaInfoContainer) -> Optional[str]: + if meta_info.query_period: + return isodate.duration_isoformat(meta_info.query_period) + if meta_info.raw_metainfo_container: + return meta_info.raw_metainfo_container.query_period + + @staticmethod + def get_trigger_operator(meta_info: MetaInfoContainer) -> Optional[str]: + if meta_info.raw_metainfo_container: + return meta_info.raw_metainfo_container.trigger_operator + + @staticmethod + def get_trigger_threshold(meta_info: MetaInfoContainer) -> Optional[str]: + if meta_info.raw_metainfo_container: + return meta_info.raw_metainfo_container.trigger_threshold + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, + ) -> str: + if not kwargs.get("raw_query", False): + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = copy.deepcopy(DEFAULT_MICROSOFT_SENTINEL_RULE) + rule["query"] = query + rule["displayName"] = meta_info.title or _AUTOGENERATED_TEMPLATE + rule["description"] = get_rule_description_str( + description=meta_info.description or rule["description"] or _AUTOGENERATED_TEMPLATE, + author=meta_info.author, + license_=meta_info.license, + ) + rule["severity"] = _SEVERITIES_MAP.get(meta_info.severity, SeverityType.medium) + mitre_tactics, mitre_techniques = self.__create_mitre_threat(mitre_attack=meta_info.mitre_attack) + rule["tactics"] = mitre_tactics + rule["techniques"] = mitre_techniques + + if meta_info: + rule["queryFrequency"] = self.get_query_frequency(meta_info=meta_info) or rule["queryFrequency"] + rule["queryPeriod"] = self.get_query_period(meta_info=meta_info) or rule["queryPeriod"] + rule["triggerOperator"] = self.get_trigger_operator(meta_info=meta_info) or rule["triggerOperator"] + rule["triggerThreshold"] = self.get_trigger_threshold(meta_info=meta_info) or rule["triggerThreshold"] + + json_rule = json.dumps(rule, indent=4, sort_keys=False) + json_rule = self.wrap_with_unmapped_fields(json_rule, unmapped_fields) + return self.wrap_with_not_supported_functions(json_rule, not_supported_functions) + + def generate_from_raw_query_container(self, query_container: RawQueryContainer) -> str: + return self.finalize_query( + prefix="", query=query_container.query, functions="", meta_info=query_container.meta_info, raw_query=True + ) diff --git a/uncoder-core/app/translator/platforms/microsoft/tokenizer.py b/uncoder-core/app/translator/platforms/microsoft/tokenizer.py new file mode 100644 index 00000000..85cf3316 --- /dev/null +++ b/uncoder-core/app/translator/platforms/microsoft/tokenizer.py @@ -0,0 +1,93 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import Any, ClassVar, Optional + +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.mixins.operator import OperatorBasedMixin +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.platforms.microsoft.custom_types.values import MicrosoftValueType +from app.translator.platforms.microsoft.escape_manager import microsoft_escape_manager +from app.translator.tools.utils import get_match_group + + +class MicrosoftSentinelTokenizer(QueryTokenizer, OperatorBasedMixin): + single_value_operators_map: ClassVar[dict[str, str]] = { + "==": OperatorType.EQ, + "=~": OperatorType.EQ, + "<=": OperatorType.LTE, + "<": OperatorType.LT, + ">=": OperatorType.GTE, + ">": OperatorType.GT, + "!=": OperatorType.NOT_EQ, + "!~": OperatorType.NOT_EQ, + "contains": OperatorType.CONTAINS, + "startswith": OperatorType.STARTSWITH, + "endswith": OperatorType.ENDSWITH, + } + multi_value_operators_map: ClassVar[dict[str, str]] = {"in~": OperatorType.EQ, "in": OperatorType.EQ} + + field_pattern = r"(?P[a-zA-Z\.\-_]+)" + bool_value_pattern = rf"(?P<{MicrosoftValueType.bool_value}>true|false)\s*" + num_value_pattern = rf"(?P<{MicrosoftValueType.number_value}>\d+(?:\.\d+)*)\s*" + double_quotes_value_pattern = rf'"(?P<{MicrosoftValueType.double_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{{\}}\[\];<>?`~\s]|\\\"|\\\\)*)"\s*' # noqa: E501 + single_quotes_value_pattern = rf"'(?P<{MicrosoftValueType.single_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-_/,\"\.$&^@!\(\)\{{\}}\[\];<>?`~\s]|\\\'|\\\\)*)'\s*" # noqa: E501 + verbatim_double_quotes_value_pattern = rf'@"(?P<{MicrosoftValueType.verbatim_double_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-_/,\'\.$&^@!\(\)\{{\}}\[\];<>?`~\s\\]|"")*)"\s*' # noqa: E501 + verbatim_single_quotes_value_pattern = rf"@'(?P<{MicrosoftValueType.verbatim_single_quotes_value}>(?:[:a-zA-Z\*0-9=+%#\-_/,\"\.$&^@!\(\)\{{\}}\[\];<>?`~\s\\]|'')*)'\s*" # noqa: E501 + str_value_pattern = rf"""{double_quotes_value_pattern}|{single_quotes_value_pattern}|{verbatim_double_quotes_value_pattern}|{verbatim_single_quotes_value_pattern}""" # noqa: E501 + _value_pattern = rf"""{bool_value_pattern}|{num_value_pattern}|{str_value_pattern}""" + multi_value_pattern = ( + rf"""\((?P<{MicrosoftValueType.multi_value}>[:a-zA-Z\"\*0-9=+%#\-_\/\\'\,.&^@!\(\[\];<>?`~\s]+)\)""" + ) + keyword_pattern = rf"\*\s+contains\s+(?:{str_value_pattern})" + + escape_manager = microsoft_escape_manager + + def get_operator_and_value( # noqa: PLR0911 + self, match: re.Match, mapped_operator: str = OperatorType.EQ, operator: Optional[str] = None + ) -> tuple[str, Any]: + if (num_value := get_match_group(match, group_name=MicrosoftValueType.number_value)) is not None: + return mapped_operator, num_value + + if (bool_value := get_match_group(match, group_name=MicrosoftValueType.bool_value)) is not None: + return mapped_operator, bool_value + + if (d_q_value := get_match_group(match, group_name=MicrosoftValueType.double_quotes_value)) is not None: + return mapped_operator, self.escape_manager.remove_escape(d_q_value) + + if (s_q_value := get_match_group(match, group_name=MicrosoftValueType.single_quotes_value)) is not None: + return mapped_operator, self.escape_manager.remove_escape(s_q_value) + + group_name = MicrosoftValueType.verbatim_double_quotes_value + if (v_d_q_value := get_match_group(match, group_name=group_name)) is not None: + return mapped_operator, v_d_q_value + + group_name = MicrosoftValueType.verbatim_single_quotes_value + if (v_s_q_value := get_match_group(match, group_name=group_name)) is not None: + return mapped_operator, v_s_q_value + + return super().get_operator_and_value(match, mapped_operator, operator) + + def clean_multi_value(self, value: str) -> str: + value = value.strip(" ") + value = value.lstrip("@") + if value.startswith("'") and value.endswith("'") or value.startswith('"') and value.endswith('"'): + value = value[1:-1] + + return value diff --git a/uncoder-core/app/translator/platforms/opensearch/__init__.py b/uncoder-core/app/translator/platforms/opensearch/__init__.py new file mode 100644 index 00000000..d450a659 --- /dev/null +++ b/uncoder-core/app/translator/platforms/opensearch/__init__.py @@ -0,0 +1,4 @@ +from app.translator.platforms.opensearch.parsers.opensearch import OpenSearchQueryParser # noqa: F401 +from app.translator.platforms.opensearch.renders.opensearch import OpenSearchQueryRender # noqa: F401 +from app.translator.platforms.opensearch.renders.opensearch_cti import OpenSearchCTI # noqa: F401 +from app.translator.platforms.opensearch.renders.opensearch_rule import OpenSearchRuleRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/opensearch/const.py b/uncoder-core/app/translator/platforms/opensearch/const.py new file mode 100644 index 00000000..6522143c --- /dev/null +++ b/uncoder-core/app/translator/platforms/opensearch/const.py @@ -0,0 +1,69 @@ +from app.translator.core.models.platform_details import PlatformDetails + +PLATFORM_DETAILS = {"group_id": "opensearch", "group_name": "AWS OpenSearch", "alt_platform_name": "ECS"} + +OPENSEARCH_LUCENE_QUERY_DETAILS = { + "platform_id": "opensearch-lucene-query", + "name": "AWS OpenSearch Query", + "platform_name": "Query (Lucene)", + **PLATFORM_DETAILS, +} + +OPENSEARCH_RULE_DETAILS = { + "platform_id": "opensearch-lucene-rule", + "name": "AWS OpenSearch Rule", + "platform_name": "Rule (JSON)", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +opensearch_query_details = PlatformDetails(**OPENSEARCH_LUCENE_QUERY_DETAILS) +opensearch_rule_details = PlatformDetails(**OPENSEARCH_RULE_DETAILS) + + +OPENSEARCH_RULE = { + "type": "monitor", + "name": "Autogenerated OpenDistro rule", + "enabled": True, + "schedule": {"period": {"interval": 1, "unit": "MINUTES"}}, + "inputs": [ + { + "search": { + "indices": ["winlogbeat-*"], + "query": { + "size": 1000, + "query": {"bool": {"must": [{"query_string": {"query": "", "analyze_wildcard": True}}]}}, + }, + } + } + ], + "triggers": [ + { + "name": "Trigger: Malicious PowerView PowerShell Commandlets", + "severity": "4", + "condition": {"script": {"source": "ctx.results[0].hits.total.value > 0", "lang": "painless"}}, + "actions": [ + { + "name": "some alert", + "destination_id": "DESTINATION_ID", + "message_template": {"source": "", "lang": "mustache"}, + "throttle_enabled": False, + "subject_template": {"source": "", "lang": "mustache"}, + } + ], + } + ], +} + +DEFAULT_OPENSEARCH_CTI_MAPPING = { + "DestinationIP": "destination.ip", + "SourceIP": "source.ip", + "HashSha512": "file.hash.sha512", + "HashSha256": "file.hash.sha256", + "HashMd5": "file.hash.md5", + "Emails": "email.from.address", + "Domain": "destination.domain", + "HashSha1": "file.hash.sha1", + "Files": "file.name", + "URL": "url.original", +} diff --git a/uncoder-core/app/translator/platforms/opensearch/mapping.py b/uncoder-core/app/translator/platforms/opensearch/mapping.py new file mode 100644 index 00000000..ad0222ed --- /dev/null +++ b/uncoder-core/app/translator/platforms/opensearch/mapping.py @@ -0,0 +1,5 @@ +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.opensearch.const import opensearch_query_details, opensearch_rule_details + +opensearch_query_mappings = LuceneMappings(platform_dir="opensearch", platform_details=opensearch_query_details) +opensearch_rule_mappings = LuceneMappings(platform_dir="opensearch", platform_details=opensearch_rule_details) diff --git a/siem-converter/app/converter/platforms/microsoft/parsers/__init__.py b/uncoder-core/app/translator/platforms/opensearch/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/microsoft/parsers/__init__.py rename to uncoder-core/app/translator/platforms/opensearch/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/opensearch/parsers/opensearch.py b/uncoder-core/app/translator/platforms/opensearch/parsers/opensearch.py new file mode 100644 index 00000000..6a3a4444 --- /dev/null +++ b/uncoder-core/app/translator/platforms/opensearch/parsers/opensearch.py @@ -0,0 +1,30 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import parser_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.base.lucene.parsers.lucene import LuceneQueryParser +from app.translator.platforms.opensearch.const import opensearch_query_details +from app.translator.platforms.opensearch.mapping import opensearch_query_mappings + + +@parser_manager.register_supported_by_roota +class OpenSearchQueryParser(LuceneQueryParser): + details: PlatformDetails = opensearch_query_details + mappings: LuceneMappings = opensearch_query_mappings diff --git a/siem-converter/app/converter/platforms/microsoft/renders/__init__.py b/uncoder-core/app/translator/platforms/opensearch/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/microsoft/renders/__init__.py rename to uncoder-core/app/translator/platforms/opensearch/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/opensearch/renders/opensearch.py b/uncoder-core/app/translator/platforms/opensearch/renders/opensearch.py new file mode 100644 index 00000000..a1a3f1a6 --- /dev/null +++ b/uncoder-core/app/translator/platforms/opensearch/renders/opensearch.py @@ -0,0 +1,106 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.str_value_manager import StrValue +from app.translator.managers import render_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.base.lucene.renders.lucene import LuceneFieldValueRender, LuceneQueryRender +from app.translator.platforms.opensearch.const import opensearch_query_details +from app.translator.platforms.opensearch.mapping import opensearch_query_mappings + + +class OpenSearchFieldValueRender(LuceneFieldValueRender): + details: PlatformDetails = opensearch_query_details + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(f'"{val}"' for val in self._pre_process_values_list(field, value)) + return f"{field}:({values})" + return f'{field}:"{self._pre_process_value(field, value)}"' + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{field}:<"{self._pre_process_value(field, value)}"' + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{field}:[* TO "{self._pre_process_value(field, value)}"]' + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{field}:>"{self._pre_process_value(field, value)}"' + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f'{field}:["{self._pre_process_value(field, value)}" TO *]' + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(f'"{val}"' for val in self._pre_process_values_list(field, value)) + return f"NOT ({field} = ({values})" + return f'NOT ({field} = "{self._pre_process_value(field, value)}")' + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(f'"*{val}*"' for val in self._pre_process_values_list(field, value)) + return f"{field}:({values})" + return f'{field}:"*{self._pre_process_value(field, value)}*"' + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(f'"*{val}"' for val in self._pre_process_values_list(field, value)) + return f"{field}:({values})" + return f'{field}:"*{self._pre_process_value(field, value)}"' + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.or_token.join(f'"{val}*"' for val in self._pre_process_values_list(field, value)) + return f"{field}:({values})" + return f'{field}:"{self._pre_process_value(field, value)}*"' + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = [] + for val in value: + values.append( + f'"/{self._pre_process_value(field, val, value_type=ValueType.regex_value)}/"' + if isinstance(val, StrValue) + else f'"/{val}/"' + ) + return f"{field}:({self.or_token.join(values)})" + + if isinstance(value, StrValue): + return f'{field}:"/{self._pre_process_value(field, value, value_type=ValueType.regex_value)}/"' + + return f'{field}:"/{value}/"' + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.keywords(field=field, value=val) for val in value)})" + return f'"*{self._pre_process_value(field, value)}*"' + + +@render_manager.register +class OpenSearchQueryRender(LuceneQueryRender): + details: PlatformDetails = opensearch_query_details + mappings: LuceneMappings = opensearch_query_mappings + + or_token = "OR" + field_value_render = OpenSearchFieldValueRender(or_token=or_token) diff --git a/uncoder-core/app/translator/platforms/opensearch/renders/opensearch_cti.py b/uncoder-core/app/translator/platforms/opensearch/renders/opensearch_cti.py new file mode 100644 index 00000000..5991b487 --- /dev/null +++ b/uncoder-core/app/translator/platforms/opensearch/renders/opensearch_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.opensearch.const import DEFAULT_OPENSEARCH_CTI_MAPPING, opensearch_query_details + + +@render_cti_manager.register +class OpenSearchCTI(RenderCTI): + details: PlatformDetails = opensearch_query_details + + field_value_template: str = '"{value}"' + or_operator: str = " OR " + group_or_operator: str = " OR " + or_group: str = "{processing_key}:({or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_OPENSEARCH_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/opensearch/renders/opensearch_rule.py b/uncoder-core/app/translator/platforms/opensearch/renders/opensearch_rule.py new file mode 100644 index 00000000..13d1c6d6 --- /dev/null +++ b/uncoder-core/app/translator/platforms/opensearch/renders/opensearch_rule.py @@ -0,0 +1,96 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy +import json +from typing import Optional + +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, RawQueryContainer, TokenizedQueryContainer +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.managers import render_manager +from app.translator.platforms.base.lucene.mapping import LuceneMappings +from app.translator.platforms.opensearch.const import OPENSEARCH_RULE, opensearch_rule_details +from app.translator.platforms.opensearch.mapping import opensearch_rule_mappings +from app.translator.platforms.opensearch.renders.opensearch import OpenSearchFieldValueRender, OpenSearchQueryRender + +_AUTOGENERATED_TEMPLATE = "Autogenerated AWS OpenSearch Rule" +_SEVERITIES_MAP = {SeverityType.critical: "5", SeverityType.high: "4", SeverityType.medium: "3", SeverityType.low: "2"} + + +class OpenSearchRuleFieldValueRender(OpenSearchFieldValueRender): + details: PlatformDetails = opensearch_rule_details + + +@render_manager.register +class OpenSearchRuleRender(OpenSearchQueryRender): + details: PlatformDetails = opensearch_rule_details + mappings: LuceneMappings = opensearch_rule_mappings + + or_token = "OR" + and_token = "AND" + not_token = "NOT" + + field_value_render = OpenSearchRuleFieldValueRender(or_token=or_token) + + def __init__(self): + super().__init__() + self.fields = {} + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = copy.deepcopy(OPENSEARCH_RULE) + source = {"event.severity": _SEVERITIES_MAP[meta_info.severity], "message": "Alert"} + source.update(self.fields) + rule["name"] = meta_info.title or _AUTOGENERATED_TEMPLATE + rule["inputs"][0]["search"]["query"]["query"]["bool"]["must"][0]["query_string"]["query"] = query + rule["triggers"][0]["name"] = meta_info.title + rule["triggers"][0]["severity"] = _SEVERITIES_MAP[meta_info.severity] + rule["triggers"][0]["actions"][0]["message_template"]["source"] = str(source).replace(", ", ",\n") + rule_str = json.dumps(rule, indent=4, sort_keys=False) + rule_str = self.wrap_with_unmapped_fields(rule_str, unmapped_fields) + return self.wrap_with_not_supported_functions(rule_str, not_supported_functions) + + def apply_token(self, token: QUERY_TOKEN_TYPE, source_mapping: SourceMapping) -> str: + if isinstance(token, FieldValue) and token.field: + for field in self.mappings.map_field(token.field, source_mapping): + self.fields.update({field: f"{{ctx.results.0.hits.hits.0._source.{field}}}"}) + return super().apply_token(token, source_mapping) + + def generate( + self, raw_query_container: RawQueryContainer, tokenized_query_container: Optional[TokenizedQueryContainer] + ) -> str: + self.fields = {} + return super().generate( + raw_query_container=raw_query_container, tokenized_query_container=tokenized_query_container + ) diff --git a/uncoder-core/app/translator/platforms/opensearch/tokenizer.py b/uncoder-core/app/translator/platforms/opensearch/tokenizer.py new file mode 100644 index 00000000..36b109f7 --- /dev/null +++ b/uncoder-core/app/translator/platforms/opensearch/tokenizer.py @@ -0,0 +1,23 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.platforms.base.lucene.tokenizer import LuceneTokenizer + + +class OpenSearchTokenizer(LuceneTokenizer): + pass diff --git a/uncoder-core/app/translator/platforms/palo_alto/__init__.py b/uncoder-core/app/translator/platforms/palo_alto/__init__.py new file mode 100644 index 00000000..e0ed85a2 --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/__init__.py @@ -0,0 +1,2 @@ +from app.translator.platforms.palo_alto.renders.cortex_xdr import CortexXDRXQLQueryRender # noqa: F401 +from app.translator.platforms.palo_alto.renders.cortex_xsiam import CortexXSIAMXQLQueryRender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/palo_alto/const.py b/uncoder-core/app/translator/platforms/palo_alto/const.py new file mode 100644 index 00000000..12facc47 --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/const.py @@ -0,0 +1,36 @@ +from app.translator.core.custom_types.predefined_fields import IPLocationType, TimeType +from app.translator.core.models.platform_details import PlatformDetails + +PLATFORM_DETAILS = {} + +CORTEX_XSIAM_XQL_QUERY_DETAILS = { + "platform_id": "cortex-xql-query", + "name": "Palo Alto Cortex XSIAM Query", + "platform_name": "Query (XQL)", + "group_id": "cortex-xsiam", + "group_name": "Palo Alto Cortex XSIAM", +} + +CORTEX_XDR_XQL_QUERY_DETAILS = { + "platform_id": "cortex-xdr-xql-query", + "name": "Palo Alto Cortex XDR Query", + "platform_name": "Query (XQL)", + "group_id": "cortex-xdr", + "group_name": "Palo Alto Cortex XDR", +} + +cortex_xsiam_xql_query_details = PlatformDetails(**CORTEX_XSIAM_XQL_QUERY_DETAILS) +cortex_xdr_xql_query_details = PlatformDetails(**CORTEX_XDR_XQL_QUERY_DETAILS) + + +PREDEFINED_FIELDS_MAP = { + IPLocationType.asn: "loc_asn", + IPLocationType.asn_org: "loc_asn_org", + IPLocationType.city: "loc_city", + IPLocationType.continent: "loc_continent", + IPLocationType.country: "loc_country", + IPLocationType.lat_lon: "loc_latlon", + IPLocationType.region: "loc_region", + IPLocationType.timezone: "loc_timezone", + TimeType.timestamp: "_time", +} diff --git a/uncoder-core/app/translator/platforms/palo_alto/escape_manager.py b/uncoder-core/app/translator/platforms/palo_alto/escape_manager.py new file mode 100644 index 00000000..ecb8c68d --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/escape_manager.py @@ -0,0 +1,17 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class CortexXQLEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.regex_value: [ + EscapeDetails(pattern=r'([_!@#$%^&*=+()\[\]{}|;:\'",.<>?/`~\-\s\\])', escape_symbols=r"\\\1") + ], + ValueType.value: [EscapeDetails(pattern=r"([\*\"\\])", escape_symbols=r"\\\1")], + } + + +cortex_xql_escape_manager = CortexXQLEscapeManager() diff --git a/uncoder-core/app/translator/platforms/palo_alto/functions/__init__.py b/uncoder-core/app/translator/platforms/palo_alto/functions/__init__.py new file mode 100644 index 00000000..6bc3588c --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/functions/__init__.py @@ -0,0 +1,24 @@ +import os.path + +from app.translator.core.functions import PlatformFunctions +from app.translator.platforms.palo_alto.functions.manager import ( + CortexXQLFunctionsManager, + cortex_xdr_xql_functions_manager, + cortex_xsiam_xql_functions_manager, +) + + +class CortexXQLFunctions(PlatformFunctions): + dir_path: str = os.path.abspath(os.path.dirname(__file__)) + + +class CortexXSIAMXQLFunctions(CortexXQLFunctions): + manager: CortexXQLFunctionsManager = cortex_xsiam_xql_functions_manager + + +class CortexXDRXQLFunctions(CortexXQLFunctions): + manager: CortexXQLFunctionsManager = cortex_xdr_xql_functions_manager + + +cortex_xsiam_xql_functions = CortexXSIAMXQLFunctions() +cortex_xdr_xql_functions = CortexXDRXQLFunctions() diff --git a/uncoder-core/app/translator/platforms/palo_alto/functions/const.py b/uncoder-core/app/translator/platforms/palo_alto/functions/const.py new file mode 100644 index 00000000..91745fca --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/functions/const.py @@ -0,0 +1,49 @@ +from app.translator.tools.custom_enum import CustomEnum + + +class CortexXQLFunctionType(CustomEnum): + avg = "avg" + count = "count" + count_distinct = "count_distinct" + min = "min" + max = "max" + sum = "sum" + values = "values" + + divide = "divide" + multiply = "multiply" + + lower = "lowercase" + split = "split" + upper = "uppercase" + + array_length = "array_length" + extract_time = "extract_time" + incidr = "incidr" + + alter = "alter" + bin = "bin" + comp = "comp" + config = "config" + fields = "fields" + filter = "filter" + iploc = "iploc" + join = "join" + limit = "limit" + sort = "sort" + timeframe = "timeframe" + timestamp_diff = "timestamp_diff" + union = "union" + + +class CortexXQLSortOrderType(CustomEnum): + asc = "asc" + desc = "desc" + + +class CortexXQLTimeFrameType(CustomEnum): + years = "y" + months = "mo" + days = "d" + hours = "h" + minutes = "m" diff --git a/uncoder-core/app/translator/platforms/palo_alto/functions/manager.py b/uncoder-core/app/translator/platforms/palo_alto/functions/manager.py new file mode 100644 index 00000000..2970a010 --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/functions/manager.py @@ -0,0 +1,9 @@ +from app.translator.core.functions import PlatformFunctionsManager + + +class CortexXQLFunctionsManager(PlatformFunctionsManager): + ... + + +cortex_xsiam_xql_functions_manager = CortexXQLFunctionsManager() +cortex_xdr_xql_functions_manager = CortexXQLFunctionsManager() diff --git a/uncoder-core/app/translator/platforms/palo_alto/mapping.py b/uncoder-core/app/translator/platforms/palo_alto/mapping.py new file mode 100644 index 00000000..6bf2d111 --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/mapping.py @@ -0,0 +1,73 @@ +from typing import Optional, Union + +from app.translator.core.mapping import ( + BasePlatformMappings, + BaseStrictLogSourcesPlatformMappings, + FieldsMapping, + LogSourceSignature, + SourceMapping, +) +from app.translator.platforms.palo_alto.const import cortex_xdr_xql_query_details, cortex_xsiam_xql_query_details + + +class CortexXQLLogSourceSignature(LogSourceSignature): + def __init__(self, preset: Optional[list[str]], dataset: Optional[list[str]], default_source: dict): + self.preset = preset + self.dataset = dataset + self._default_source = default_source or {} + + def is_suitable(self, preset: Optional[list[str]] = None, dataset: Optional[list[str]] = None) -> bool: + conditions = [ + set(preset).issubset(self.preset) if preset else None, + set(dataset).issubset(self.dataset) if dataset else None, + ] + return self._check_conditions(conditions) + + @staticmethod + def __prepare_log_source_for_render(logsource: Union[str, list[str]], model: str = "datamodel") -> str: + if isinstance(logsource, list): + return f"{model} in ({', '.join(source for source in logsource)})" + return f"{model} = {logsource}" + + @property + def __data_model_scheme(self) -> str: + if data_model := self._default_source.get("datamodel"): + return f"{data_model} " + return "" + + def __str__(self) -> str: + if preset_data := self._default_source.get("preset"): + preset = self.__prepare_log_source_for_render(logsource=preset_data, model="preset") + return f"{self.__data_model_scheme}{preset}" + if dataset_data := self._default_source.get("dataset"): + dataset = self.__prepare_log_source_for_render(logsource=dataset_data, model="dataset") + return f"{self.__data_model_scheme}{dataset}" + return "datamodel dataset = *" + + +class CortexXQLLogSourceSignaturePreparer: + @staticmethod + def prepare_log_source_signature(mapping: dict) -> CortexXQLLogSourceSignature: + preset = mapping.get("log_source", {}).get("preset") + dataset = mapping.get("log_source", {}).get("dataset") + default_log_source = mapping["default_log_source"] + return CortexXQLLogSourceSignature(preset=preset, dataset=dataset, default_source=default_log_source) + + +class CortexXSIAMXQLMappings(CortexXQLLogSourceSignaturePreparer, BasePlatformMappings): + skip_load_default_mappings: bool = False + + def update_default_source_mapping(self, default_mapping: SourceMapping, fields_mapping: FieldsMapping) -> None: + ... + + +class CortexXDRXQLMappings(CortexXQLLogSourceSignaturePreparer, BaseStrictLogSourcesPlatformMappings): + ... + + +cortex_xsiam_xql_query_mappings = CortexXSIAMXQLMappings( + platform_dir="palo_alto_cortex_xsiam", platform_details=cortex_xsiam_xql_query_details +) +cortex_xdr_xql_query_mappings = CortexXDRXQLMappings( + platform_dir="palo_alto_cortex_xdr", platform_details=cortex_xdr_xql_query_details +) diff --git a/siem-converter/app/converter/platforms/microsoft/siem_functions/__init__.py b/uncoder-core/app/translator/platforms/palo_alto/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/microsoft/siem_functions/__init__.py rename to uncoder-core/app/translator/platforms/palo_alto/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/palo_alto/renders/base.py b/uncoder-core/app/translator/platforms/palo_alto/renders/base.py new file mode 100644 index 00000000..6983d0f3 --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/renders/base.py @@ -0,0 +1,205 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from typing import ClassVar, Optional, Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.const import QUERY_TOKEN_TYPE +from app.translator.core.context_vars import preset_log_source_str_ctx_var +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.render import BaseFieldFieldRender, BaseFieldValueRender, PlatformQueryRender +from app.translator.core.str_value_manager import StrValue +from app.translator.platforms.palo_alto.const import PREDEFINED_FIELDS_MAP +from app.translator.platforms.palo_alto.functions import CortexXQLFunctions +from app.translator.platforms.palo_alto.mapping import CortexXQLLogSourceSignature +from app.translator.platforms.palo_alto.str_value_manager import cortex_xql_str_value_manager + +SOURCE_MAPPING_TO_FIELD_VALUE_MAP = { + "windows_registry_event": { + "EventType": { + "SetValue": "REGISTRY_SET_VALUE", + "DeleteValue": "REGISTRY_DELETE_VALUE", + "CreateKey": "REGISTRY_CREATE_KEY", + } + } +} + + +class CortexXQLFieldValueRender(BaseFieldValueRender): + str_value_manager = cortex_xql_str_value_manager + + @staticmethod + def _get_value_type(field_name: str, value: Union[int, str, StrValue], value_type: Optional[str] = None) -> str: # noqa: ARG004 + if value_type: + return value_type + + if isinstance(value, StrValue) and value.has_spec_symbols: + return ValueType.regex_value + + return ValueType.value + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f'"{value}"' + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = ", ".join(f"{self._pre_process_value(field, v, ValueType.value, True)}" for v in value) + return f"{field} in ({values})" + + return f"{field} = {self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}" + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + return f"{field} < {self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}" + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f"{field} <= {self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}" + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + return f"{field} > {self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}" + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + return f"{field} >= {self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join([self.not_equal_modifier(field=field, value=v) for v in value])})" + return f"{field} != {self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}" + + def contains_modifier(self, field: str, value: Union[list, str]) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + if value.endswith("\\"): + return f'{field} ~= ".*{self._pre_process_value(field, value, value_type=ValueType.regex_value)}.*"' + return f"{field} contains {self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True)}" + + def not_contains_modifier(self, field: str, value: Union[list, str]) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + if value.endswith("\\"): + return f'{field} !~= ".*{self._pre_process_value(field, value, value_type=ValueType.regex_value)}.*"' + return f"{field} not contains {self._pre_process_value(field, value, ValueType.value, wrap_str=True)}" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.endswith_modifier(field=field, value=v) for v in value)})" + return f'{field} ~= ".*{self._pre_process_value(field, value, value_type=ValueType.regex_value)}"' + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + clause = self.or_token.join(self.startswith_modifier(field=field, value=v) for v in value) + return f"({clause})" + return f'{field} ~= "{self._pre_process_value(field, value, value_type=ValueType.regex_value)}.*"' + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + value = self._pre_process_value(field, value, value_type=ValueType.regex_value, wrap_str=True) + if value.endswith('\\\\"'): + value = value[:-1] + "]" + value[-1:] + value = value[:-4] + "[" + value[-4:] + return f"{field} ~= {value}" + + def not_regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.regex_modifier(field=field, value=v) for v in value)})" + return f"{field} !~= {self._pre_process_value(field ,value, value_type=ValueType.regex_value, wrap_str=True)}" + + def is_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.is_none(field=field, value=v) for v in value)})" + return f"{field} = null" + + def is_not_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.is_not_none(field=field, value=v) for v in value)})" + return f"{field} != null" + + def keywords(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + return f"({self.or_token.join(self.contains_modifier(field=field, value=v) for v in value)})" + if value.endswith("\\"): + return f'_raw_log ~= ".*{self._pre_process_value(field ,value, value_type=ValueType.regex_value)}.*"' + return f"_raw_log contains {self._pre_process_value(field ,value, value_type=ValueType.value, wrap_str=True)}" + + +class CortexXQLFieldFieldRender(BaseFieldFieldRender): + operators_map: ClassVar[dict[str, str]] = { + OperatorType.EQ: "=", + OperatorType.NOT_EQ: "!=", + OperatorType.LT: "<", + OperatorType.LTE: "<=", + OperatorType.GT: ">", + OperatorType.GTE: ">=", + } + + +class CortexXQLQueryRender(PlatformQueryRender): + predefined_fields_map = PREDEFINED_FIELDS_MAP + raw_log_field_patterns_map: ClassVar[dict[str, str]] = { + "regex": '| alter {field} = regextract(to_json_string(action_evtlog_data_fields)->{field}{{}}, "\\"(.*)\\"")', + "object": '| alter {field_name} = json_extract_scalar({field_object} , "$.{field_path}")', + "list": '| alter {field_name} = arraystring(json_extract_array({field_object} , "$.{field_path}")," ")', + } + platform_functions: CortexXQLFunctions = None + + or_token = "or" + and_token = "and" + not_token = "not" + query_parts_delimiter = "\n" + + field_field_render = CortexXQLFieldFieldRender() + comment_symbol = "//" + is_single_line_comment = False + + def process_raw_log_field(self, field: str, field_type: str) -> Optional[str]: + raw_log_field_pattern = self.raw_log_field_patterns_map.get(field_type) + if raw_log_field_pattern is None: + return + if field_type == "regex": + field = field.replace(".", r"\.") + return raw_log_field_pattern.format(field=field) + if field_type in ("object", "list") and "." in field: + field_object, field_path = field.split(".", 1) + field_name = field.replace(".", "_") + return raw_log_field_pattern.format(field_name=field_name, field_object=field_object, field_path=field_path) + + def generate_prefix(self, log_source_signature: CortexXQLLogSourceSignature, functions_prefix: str = "") -> str: + functions_prefix = f"{functions_prefix} | " if functions_prefix else "" + log_source_str = preset_log_source_str_ctx_var.get() or str(log_source_signature) + return f"{functions_prefix}{log_source_str}" + + def apply_token(self, token: QUERY_TOKEN_TYPE, source_mapping: SourceMapping) -> str: + if isinstance(token, FieldValue) and token.field: + field_name = token.field.source_name + if values_map := SOURCE_MAPPING_TO_FIELD_VALUE_MAP.get(source_mapping.source_id, {}).get(field_name): + values_to_update = [] + for token_value in token.values: + mapped_value: str = values_map.get(token_value, token_value) + values_to_update.append( + StrValue(value=mapped_value, split_value=mapped_value.split()) if mapped_value else token_value + ) + token.value = values_to_update + return super().apply_token(token=token, source_mapping=source_mapping) + + @staticmethod + def _finalize_search_query(query: str) -> str: + return f"| filter {query}" if query else "" diff --git a/uncoder-core/app/translator/platforms/palo_alto/renders/cortex_xdr.py b/uncoder-core/app/translator/platforms/palo_alto/renders/cortex_xdr.py new file mode 100644 index 00000000..fac4df3d --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/renders/cortex_xdr.py @@ -0,0 +1,41 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.palo_alto.const import cortex_xdr_xql_query_details +from app.translator.platforms.palo_alto.functions import cortex_xdr_xql_functions +from app.translator.platforms.palo_alto.mapping import CortexXDRXQLMappings, cortex_xdr_xql_query_mappings +from app.translator.platforms.palo_alto.renders.base import CortexXQLFieldValueRender, CortexXQLQueryRender + + +class CortexXDRXQLFieldValueRender(CortexXQLFieldValueRender): + details: PlatformDetails = cortex_xdr_xql_query_details + + +@render_manager.register +class CortexXDRXQLQueryRender(CortexXQLQueryRender): + details: PlatformDetails = cortex_xdr_xql_query_details + mappings: CortexXDRXQLMappings = cortex_xdr_xql_query_mappings + + field_value_render = CortexXDRXQLFieldValueRender(CortexXQLQueryRender.or_token) + + def init_platform_functions(self) -> None: + self.platform_functions = cortex_xdr_xql_functions + self.platform_functions.platform_query_render = self diff --git a/uncoder-core/app/translator/platforms/palo_alto/renders/cortex_xsiam.py b/uncoder-core/app/translator/platforms/palo_alto/renders/cortex_xsiam.py new file mode 100644 index 00000000..4b05b306 --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/renders/cortex_xsiam.py @@ -0,0 +1,41 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.palo_alto.const import cortex_xsiam_xql_query_details +from app.translator.platforms.palo_alto.functions import cortex_xsiam_xql_functions +from app.translator.platforms.palo_alto.mapping import CortexXSIAMXQLMappings, cortex_xsiam_xql_query_mappings +from app.translator.platforms.palo_alto.renders.base import CortexXQLFieldValueRender, CortexXQLQueryRender + + +class CortexXSIAMXQLFieldValueRender(CortexXQLFieldValueRender): + details: PlatformDetails = cortex_xsiam_xql_query_details + + +@render_manager.register +class CortexXSIAMXQLQueryRender(CortexXQLQueryRender): + details: PlatformDetails = cortex_xsiam_xql_query_details + mappings: CortexXSIAMXQLMappings = cortex_xsiam_xql_query_mappings + + field_value_render = CortexXSIAMXQLFieldValueRender(CortexXQLQueryRender.or_token) + + def init_platform_functions(self) -> None: + self.platform_functions = cortex_xsiam_xql_functions + self.platform_functions.platform_query_render = self diff --git a/uncoder-core/app/translator/platforms/palo_alto/str_value_manager.py b/uncoder-core/app/translator/platforms/palo_alto/str_value_manager.py new file mode 100644 index 00000000..e547f223 --- /dev/null +++ b/uncoder-core/app/translator/platforms/palo_alto/str_value_manager.py @@ -0,0 +1,56 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +import copy + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.str_value_manager import ( + CONTAINER_SPEC_SYMBOLS_MAP, + BaseSpecSymbol, + SingleSymbolWildCard, + StrValue, + StrValueManager, + UnboundLenWildCard, +) +from app.translator.platforms.palo_alto.escape_manager import cortex_xql_escape_manager + +CORTEX_XQL_CONTAINER_SPEC_SYMBOLS_MAP = copy.copy(CONTAINER_SPEC_SYMBOLS_MAP) +CORTEX_XQL_CONTAINER_SPEC_SYMBOLS_MAP.update({SingleSymbolWildCard: ".*", UnboundLenWildCard: ".*"}) + + +class CortexXQLStrValueManager(StrValueManager): + escape_manager = cortex_xql_escape_manager + container_spec_symbols_map = CORTEX_XQL_CONTAINER_SPEC_SYMBOLS_MAP + + def from_container_to_str(self, container: StrValue, value_type: str = ValueType.value) -> str: + result = "" + for el in container.split_value: + if isinstance(el, str): + result += self.escape_manager.escape(el, value_type) + elif isinstance(el, BaseSpecSymbol): + if value_type == ValueType.value: + if isinstance(el, (SingleSymbolWildCard, UnboundLenWildCard)): + result += "*" + elif pattern := self.container_spec_symbols_map.get(type(el)): + result += pattern + + return result + + +cortex_xql_str_value_manager = CortexXQLStrValueManager() diff --git a/uncoder-core/app/translator/platforms/qradar/__init__.py b/uncoder-core/app/translator/platforms/qradar/__init__.py new file mode 100644 index 00000000..09ed0612 --- /dev/null +++ b/uncoder-core/app/translator/platforms/qradar/__init__.py @@ -0,0 +1,3 @@ +from app.translator.platforms.qradar.parsers.qradar import QradarQueryParser # noqa: F401 +from app.translator.platforms.qradar.renders.qradar import QradarQueryRender # noqa: F401 +from app.translator.platforms.qradar.renders.qradar_cti import QRadarCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/qradar/const.py b/uncoder-core/app/translator/platforms/qradar/const.py new file mode 100644 index 00000000..ec16bd42 --- /dev/null +++ b/uncoder-core/app/translator/platforms/qradar/const.py @@ -0,0 +1,25 @@ +from app.translator.core.models.platform_details import PlatformDetails + +QRADAR_QUERY_DETAILS = { + "platform_id": "qradar-aql-query", + "name": "QRadar Query", + "platform_name": "Query (AQL)", + "group_id": "qradar", + "group_name": "QRadar", +} + +DEFAULT_QRADAR_CTI_MAPPING = { + "DestinationIP": "destinationip", + "SourceIP": "sourceip", + "HashSha512": "File Hash", + "HashSha256": "File Hash", + "HashMd5": "File Hash", + "Emails": "emails", + "Domain": "Hostname", + "HashSha1": "File Hash", + "Files": "Filename", + "URL": "URL", +} + + +qradar_query_details = PlatformDetails(**QRADAR_QUERY_DETAILS) diff --git a/uncoder-core/app/translator/platforms/qradar/mapping.py b/uncoder-core/app/translator/platforms/qradar/mapping.py new file mode 100644 index 00000000..e179e73b --- /dev/null +++ b/uncoder-core/app/translator/platforms/qradar/mapping.py @@ -0,0 +1,4 @@ +from app.translator.platforms.base.aql.mapping import AQLMappings +from app.translator.platforms.qradar.const import qradar_query_details + +qradar_query_mappings = AQLMappings(platform_dir="qradar", platform_details=qradar_query_details) diff --git a/siem-converter/app/converter/platforms/opensearch/__init__.py b/uncoder-core/app/translator/platforms/qradar/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/opensearch/__init__.py rename to uncoder-core/app/translator/platforms/qradar/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/qradar/parsers/qradar.py b/uncoder-core/app/translator/platforms/qradar/parsers/qradar.py new file mode 100644 index 00000000..ddb2f0cb --- /dev/null +++ b/uncoder-core/app/translator/platforms/qradar/parsers/qradar.py @@ -0,0 +1,32 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import parser_manager +from app.translator.platforms.base.aql.mapping import AQLMappings +from app.translator.platforms.base.aql.parsers.aql import AQLQueryParser +from app.translator.platforms.qradar.const import qradar_query_details +from app.translator.platforms.qradar.mapping import qradar_query_mappings + + +@parser_manager.register_supported_by_roota +class QradarQueryParser(AQLQueryParser): + details: PlatformDetails = qradar_query_details + mappings: AQLMappings = qradar_query_mappings + + wrapped_with_comment_pattern = r"^\s*/\*(?:|\n|.)*\*/" diff --git a/siem-converter/app/converter/platforms/opensearch/mappings/__init__.py b/uncoder-core/app/translator/platforms/qradar/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/opensearch/mappings/__init__.py rename to uncoder-core/app/translator/platforms/qradar/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/qradar/renders/qradar.py b/uncoder-core/app/translator/platforms/qradar/renders/qradar.py new file mode 100644 index 00000000..a6846dad --- /dev/null +++ b/uncoder-core/app/translator/platforms/qradar/renders/qradar.py @@ -0,0 +1,36 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.base.aql.mapping import AQLMappings +from app.translator.platforms.base.aql.renders.aql import AQLFieldValueRender, AQLQueryRender +from app.translator.platforms.qradar.const import qradar_query_details +from app.translator.platforms.qradar.mapping import qradar_query_mappings + + +class QradarFieldValueRender(AQLFieldValueRender): + details: PlatformDetails = qradar_query_details + + +@render_manager.register +class QradarQueryRender(AQLQueryRender): + details: PlatformDetails = qradar_query_details + mappings: AQLMappings = qradar_query_mappings + field_value_render = QradarFieldValueRender(or_token=AQLQueryRender.or_token) diff --git a/uncoder-core/app/translator/platforms/qradar/renders/qradar_cti.py b/uncoder-core/app/translator/platforms/qradar/renders/qradar_cti.py new file mode 100644 index 00000000..6159ba86 --- /dev/null +++ b/uncoder-core/app/translator/platforms/qradar/renders/qradar_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.qradar.const import DEFAULT_QRADAR_CTI_MAPPING, qradar_query_details + + +@render_cti_manager.register +class QRadarCTI(RenderCTI): + details: PlatformDetails = qradar_query_details + + field_value_template: str = "\"{key}\" = '{value}'" + or_operator: str = " or " + group_or_operator: str = " or " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "SELECT UTF8(payload) from events where {result}\n" + final_result_for_one: str = "SELECT UTF8(payload) from events where {result}\n" + default_mapping = DEFAULT_QRADAR_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/qualys/__init__.py b/uncoder-core/app/translator/platforms/qualys/__init__.py new file mode 100644 index 00000000..1e73d6b4 --- /dev/null +++ b/uncoder-core/app/translator/platforms/qualys/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.qualys.renders.qualys_cti import QualysCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/qualys/const.py b/uncoder-core/app/translator/platforms/qualys/const.py new file mode 100644 index 00000000..f7632710 --- /dev/null +++ b/uncoder-core/app/translator/platforms/qualys/const.py @@ -0,0 +1,20 @@ +QUALYS_QUERY_DETAILS = { + "platform_id": "qualys", + "name": "Qualys", + "platform_name": "IOC Query", + "group_name": "Qualys", + "group_id": "qualys", +} + +DEFAULT_QUALYS_CTI_MAPPING = { + "DestinationIP": "network.remote.address.ip", + "SourceIP": "network.local.address.ip", + "HashSha512": "file.hash.sha512", + "HashSha256": "file.hash.sha256", + "HashMd5": "file.hash.md5", + "Emails": "emails", + "Domain": "domain", + "HashSha1": "file.hash.sha1", + "Files": "file.name", + "URL": "url", +} diff --git a/siem-converter/app/converter/platforms/opensearch/parsers/__init__.py b/uncoder-core/app/translator/platforms/qualys/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/opensearch/parsers/__init__.py rename to uncoder-core/app/translator/platforms/qualys/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/qualys/renders/qualys_cti.py b/uncoder-core/app/translator/platforms/qualys/renders/qualys_cti.py new file mode 100644 index 00000000..3ccce6ba --- /dev/null +++ b/uncoder-core/app/translator/platforms/qualys/renders/qualys_cti.py @@ -0,0 +1,34 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.qualys.const import DEFAULT_QUALYS_CTI_MAPPING, QUALYS_QUERY_DETAILS + + +@render_cti_manager.register +class QualysCTI(RenderCTI): + details: PlatformDetails = PlatformDetails(**QUALYS_QUERY_DETAILS) + + field_value_template: str = "{key}:`{value}`" + or_operator: str = " or " + group_or_operator: str = " or " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_QUALYS_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/roota/__init__.py b/uncoder-core/app/translator/platforms/roota/__init__.py new file mode 100644 index 00000000..6e74f4cf --- /dev/null +++ b/uncoder-core/app/translator/platforms/roota/__init__.py @@ -0,0 +1,2 @@ +from app.translator.platforms.roota.parsers.roota import RootAParser # noqa: F401 +from app.translator.platforms.roota.renders.roota import RootARender # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/roota/const.py b/uncoder-core/app/translator/platforms/roota/const.py new file mode 100644 index 00000000..66e0d4ed --- /dev/null +++ b/uncoder-core/app/translator/platforms/roota/const.py @@ -0,0 +1,24 @@ +ROOTA_RULE_DETAILS = { + "name": "Roota", + "platform_id": "roota", + "platform_name": "Roota", + "group_name": "Roota", + "group_id": "roota", +} + + +ROOTA_RULE_TEMPLATE = { + "name": "", + "details": "", + "author": "", + "severity": "", + "type": "query", + "date": "", + "mitre-attack": [], + "detection": {"language": "", "body": ""}, + "logsource": {}, + "references": [], + "tags": [], + "license": "", + "uuid": "", +} diff --git a/siem-converter/app/converter/platforms/opensearch/renders/__init__.py b/uncoder-core/app/translator/platforms/roota/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/opensearch/renders/__init__.py rename to uncoder-core/app/translator/platforms/roota/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/roota/parsers/roota.py b/uncoder-core/app/translator/platforms/roota/parsers/roota.py new file mode 100644 index 00000000..972ff6a1 --- /dev/null +++ b/uncoder-core/app/translator/platforms/roota/parsers/roota.py @@ -0,0 +1,90 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from typing import ClassVar + +from app.translator.core.exceptions.core import RootARuleValidationException, UnsupportedRootAParser +from app.translator.core.mixins.rule import YamlRuleMixin +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser, QueryParser +from app.translator.managers import parser_manager +from app.translator.platforms.roota.const import ROOTA_RULE_DETAILS + + +@parser_manager.register_main +class RootAParser(QueryParser, YamlRuleMixin): + parser_manager = parser_manager + details: PlatformDetails = PlatformDetails(**ROOTA_RULE_DETAILS) + mandatory_fields: ClassVar[set[str]] = { + "name", + "details", + "author", + "severity", + "mitre-attack", + "detection", + "references", + "license", + } + + wrapped_with_comment_pattern = r"^\s*#.*(?:\n|$)" + + def __parse_meta_info(self, rule: dict) -> MetaInfoContainer: + mitre_attack = rule.get("mitre-attack") or [] + mitre_tags = [i.strip("") for i in mitre_attack.split(",")] if isinstance(mitre_attack, str) else mitre_attack + mitre_attack = self.parse_mitre_attack(mitre_tags) + rule_tags = rule.get("tags", []) + if isinstance(rule_tags, str): + rule_tags = [i.strip() for i in rule_tags.split(",")] + rule_tags += mitre_tags + + return MetaInfoContainer( + id_=rule.get("uuid"), + title=rule.get("name"), + description=rule.get("details"), + author=rule.get("author", "").split(", "), + date=rule.get("date"), + license_=rule.get("license"), + severity=rule.get("severity"), + references=rule.get("references"), + mitre_attack=mitre_attack, + tags=rule_tags, + ) + + def __get_parser_class(self, parser: str) -> PlatformQueryParser: + parser_class = self.parser_manager.get_supported_by_roota(parser) + if parser_class: + return parser_class + raise UnsupportedRootAParser(parser=parser) + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + rule = self.load_rule(text=text) + if missing_fields := self.mandatory_fields.difference(set(rule.keys())): + raise RootARuleValidationException(missing_fields=list(missing_fields)) + + detection = rule.get("detection", {}) + query = detection.get("body") + language = detection.get("language") + if not (query or language): + raise RootARuleValidationException(missing_fields=["detection"]) + + return RawQueryContainer(query=query, language=language, meta_info=self.__parse_meta_info(rule)) + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + parser = self.__get_parser_class(raw_query_container.language) + return parser.parse(raw_query_container) diff --git a/siem-converter/app/converter/platforms/qradar/__init__.py b/uncoder-core/app/translator/platforms/roota/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/qradar/__init__.py rename to uncoder-core/app/translator/platforms/roota/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/roota/renders/roota.py b/uncoder-core/app/translator/platforms/roota/renders/roota.py new file mode 100644 index 00000000..d52e0e97 --- /dev/null +++ b/uncoder-core/app/translator/platforms/roota/renders/roota.py @@ -0,0 +1,164 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" +import copy +import math +from datetime import timedelta +from typing import Optional + +import yaml + +from app.translator.core.context_vars import return_only_first_query_ctx_var, wrap_query_with_meta_info_ctx_var +from app.translator.core.exceptions.render import BaseRenderException +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.render import PlatformQueryRender, QueryRender +from app.translator.managers import RenderManager, render_manager +from app.translator.platforms.microsoft.const import MICROSOFT_SENTINEL_QUERY_DETAILS +from app.translator.platforms.microsoft.mapping import microsoft_sentinel_query_mappings +from app.translator.platforms.roota.const import ROOTA_RULE_DETAILS, ROOTA_RULE_TEMPLATE +from app.translator.platforms.sigma.const import SIGMA_RULE_DETAILS +from app.translator.platforms.sigma.mapping import sigma_rule_mappings + +_AUTOGENERATED_TEMPLATE = "Autogenerated Roota Rule" + + +class IndentedListDumper(yaml.Dumper): + def increase_indent(self, flow: bool = False, indentless: bool = False) -> None: # noqa: ARG002 + return super().increase_indent(flow, False) + + +@render_manager.register +class RootARender(PlatformQueryRender): + details: PlatformDetails = PlatformDetails(**ROOTA_RULE_DETAILS) + render_manager: RenderManager = render_manager + mappings = microsoft_sentinel_query_mappings + + @staticmethod + def __render_timeframe(timeframe: timedelta) -> str: + total_seconds = timeframe.total_seconds() + + week_ = 7 # days + day_ = 24 # hours + hour_ = 60 # minutes + minute_ = 60 # seconds + + if total_seconds >= week_ * day_ * hour_ * minute_: + timeframe_value = math.ceil(total_seconds / (week_ * day_ * hour_ * minute_)) + timeframe_unit = "w" + elif total_seconds >= day_ * hour_ * minute_: + timeframe_value = math.ceil(total_seconds / (day_ * hour_ * minute_)) + timeframe_unit = "d" + elif total_seconds >= hour_ * minute_: + timeframe_value = math.ceil(total_seconds / (hour_ * minute_)) + timeframe_unit = "h" + elif total_seconds >= minute_: + timeframe_value = math.ceil(total_seconds / minute_) + timeframe_unit = "m" + else: + timeframe_value = math.ceil(total_seconds) + timeframe_unit = "s" + return f"{timeframe_value}{timeframe_unit}" + + @staticmethod + def __normalize_log_source(log_source: dict) -> dict: + prepared_log_source = {} + for log_source_key, value in log_source.items(): + if isinstance(value, list): + value = value[0] + prepared_log_source[log_source_key] = value.lower() + return prepared_log_source + + def __get_data_for_roota_render( + self, raw_query_container: RawQueryContainer, tokenized_query_container: TokenizedQueryContainer + ) -> tuple: + if raw_query_container.language == SIGMA_RULE_DETAILS["platform_id"]: + rule_query_language = MICROSOFT_SENTINEL_QUERY_DETAILS["platform_id"] + prev_state_return_only_first_query_ctx_var = return_only_first_query_ctx_var.get() + prev_state_wrap_query_with_meta_info_ctx_var = wrap_query_with_meta_info_ctx_var.get() + return_only_first_query_ctx_var.set(True) + wrap_query_with_meta_info_ctx_var.set(False) + + render: QueryRender = render_manager.get(rule_query_language) + rule_query = render.generate( + raw_query_container=raw_query_container, tokenized_query_container=tokenized_query_container + ) + return_only_first_query_ctx_var.set(prev_state_return_only_first_query_ctx_var) + wrap_query_with_meta_info_ctx_var.set(prev_state_wrap_query_with_meta_info_ctx_var) + + return ( + rule_query, + rule_query_language, + self.__normalize_log_source(log_source=tokenized_query_container.meta_info.parsed_logsources), + ) + rule_query_language = raw_query_container.language.replace("rule", "query") + rule_query = raw_query_container.query + for source_mapping_id in tokenized_query_container.meta_info.source_mapping_ids: + if source_mapping_id == "default": + continue + if logsources := self.__get_logsources_by_source_mapping_id(source_mapping_id=source_mapping_id): + return rule_query, rule_query_language, self.__normalize_log_source(log_source=logsources) + return rule_query, rule_query_language, {} + + @staticmethod + def __get_logsources_by_source_mapping_id(source_mapping_id: str) -> Optional[dict]: + if source_mapping := sigma_rule_mappings.get_source_mapping(source_mapping_id): + return source_mapping.log_source_signature.log_sources + + def generate( + self, raw_query_container: RawQueryContainer, tokenized_query_container: Optional[TokenizedQueryContainer] + ) -> str: + if not tokenized_query_container or not tokenized_query_container.meta_info: + raise BaseRenderException("Meta info is required") + rule_query, rule_query_language, rule_logsources = self.__get_data_for_roota_render( + raw_query_container=raw_query_container, tokenized_query_container=tokenized_query_container + ) + + rule = copy.deepcopy(ROOTA_RULE_TEMPLATE) + rule["name"] = tokenized_query_container.meta_info.title or _AUTOGENERATED_TEMPLATE + rule["details"] = tokenized_query_container.meta_info.description or rule["details"] + rule["author"] = tokenized_query_container.meta_info.author_str or rule["author"] + rule["severity"] = tokenized_query_container.meta_info.severity or rule["severity"] + rule["date"] = tokenized_query_container.meta_info.date + rule["detection"]["language"] = rule_query_language + rule["detection"]["body"] = rule_query + rule["license"] = tokenized_query_container.meta_info.license + rule["uuid"] = tokenized_query_container.meta_info.id + rule["references"] = raw_query_container.meta_info.references or tokenized_query_container.meta_info.references + rule["tags"] = raw_query_container.meta_info.tags or tokenized_query_container.meta_info.tags + + if tokenized_query_container.meta_info.raw_mitre_attack: + rule["mitre-attack"] = tokenized_query_container.meta_info.raw_mitre_attack + elif tokenized_query_container.meta_info.mitre_attack: + techniques = [ + technique.technique_id.lower() + for technique in tokenized_query_container.meta_info.mitre_attack.techniques + ] + tactics = [ + tactic.name.lower().replace(" ", "-") + for tactic in tokenized_query_container.meta_info.mitre_attack.tactics + ] + rule["mitre-attack"] = techniques + tactics + + if tokenized_query_container.meta_info.timeframe: + rule["correlation"] = {} + rule["correlation"]["timeframe"] = self.__render_timeframe(tokenized_query_container.meta_info.timeframe) + + if rule_logsources: + rule["logsource"] = rule_logsources + + return yaml.dump(rule, Dumper=IndentedListDumper, default_flow_style=False, sort_keys=False, indent=4) diff --git a/uncoder-core/app/translator/platforms/rsa_netwitness/__init__.py b/uncoder-core/app/translator/platforms/rsa_netwitness/__init__.py new file mode 100644 index 00000000..1dbbd7ad --- /dev/null +++ b/uncoder-core/app/translator/platforms/rsa_netwitness/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.rsa_netwitness.renders.rsa_netwitness_cti import RSANetwitnessCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/rsa_netwitness/const.py b/uncoder-core/app/translator/platforms/rsa_netwitness/const.py new file mode 100644 index 00000000..fd3f95ad --- /dev/null +++ b/uncoder-core/app/translator/platforms/rsa_netwitness/const.py @@ -0,0 +1,20 @@ +RSA_NETWITNESS_QUERY_DETAILS = { + "platform_id": "rsa_netwitness", + "name": "RSA NetWitness", + "group_name": "RSA NetWitness", + "platform_name": "Query", + "group_id": "rsa_netwitness", +} + +DEFAULT_RSA_NETWITNESS_CTI_MAPPING = { + "DestinationIP": "ip.dst", + "SourceIP": "ip.src", + "HashSha512": "hash", + "HashSha256": "hash", + "HashMd5": "hash", + "Emails": "emails", + "Domain": "domain", + "HashSha1": "hash", + "Files": "files", + "URL": "web.page", +} diff --git a/siem-converter/app/converter/platforms/qradar/mappings/__init__.py b/uncoder-core/app/translator/platforms/rsa_netwitness/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/qradar/mappings/__init__.py rename to uncoder-core/app/translator/platforms/rsa_netwitness/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/rsa_netwitness/renders/rsa_netwitness_cti.py b/uncoder-core/app/translator/platforms/rsa_netwitness/renders/rsa_netwitness_cti.py new file mode 100644 index 00000000..fe40bb8c --- /dev/null +++ b/uncoder-core/app/translator/platforms/rsa_netwitness/renders/rsa_netwitness_cti.py @@ -0,0 +1,40 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.rsa_netwitness.const import ( + DEFAULT_RSA_NETWITNESS_CTI_MAPPING, + RSA_NETWITNESS_QUERY_DETAILS, +) + + +@render_cti_manager.register +class RSANetwitnessCTI(RenderCTI): + details: PlatformDetails = PlatformDetails(**RSA_NETWITNESS_QUERY_DETAILS) + + field_value_template: str = "'{value}'" + or_operator: str = ", " + group_or_operator: str = " or " + or_group: str = "({processing_key} = {or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_RSA_NETWITNESS_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/securonix/__init__.py b/uncoder-core/app/translator/platforms/securonix/__init__.py new file mode 100644 index 00000000..92f8b6af --- /dev/null +++ b/uncoder-core/app/translator/platforms/securonix/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.securonix.renders.securonix_cti import SecuronixCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/securonix/const.py b/uncoder-core/app/translator/platforms/securonix/const.py new file mode 100644 index 00000000..9e301819 --- /dev/null +++ b/uncoder-core/app/translator/platforms/securonix/const.py @@ -0,0 +1,20 @@ +SECURONIX_QUERY_DETAILS = { + "platform_id": "securonix", + "name": "Securonix", + "platform_name": "Query", + "group_name": "Securonix", + "group_id": "securonix", +} + +DEFAULT_SECURONIX_CTI_MAPPING = { + "DestinationIP": "@destinationaddress", + "SourceIP": "@sourceaddress", + "HashSha512": "@filehash", + "HashSha256": "@filehash", + "HashMd5": "@filehash", + "Emails": "emails", + "Domain": "@destinationhostname", + "HashSha1": "@filehash", + "Files": "@filename", + "URL": "@requesturl", +} diff --git a/siem-converter/app/converter/platforms/qradar/parsers/__init__.py b/uncoder-core/app/translator/platforms/securonix/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/qradar/parsers/__init__.py rename to uncoder-core/app/translator/platforms/securonix/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/securonix/renders/securonix_cti.py b/uncoder-core/app/translator/platforms/securonix/renders/securonix_cti.py new file mode 100644 index 00000000..28445d27 --- /dev/null +++ b/uncoder-core/app/translator/platforms/securonix/renders/securonix_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.securonix.const import DEFAULT_SECURONIX_CTI_MAPPING, SECURONIX_QUERY_DETAILS + + +@render_cti_manager.register +class SecuronixCTI(RenderCTI): + details: PlatformDetails = PlatformDetails(**SECURONIX_QUERY_DETAILS) + + field_value_template: str = '{key} CONTAINS "{value}"' + or_operator: str = " OR " + group_or_operator: str = " OR " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "index = archive AND {result}\n" + final_result_for_one: str = "index = archive AND {result}\n" + default_mapping = DEFAULT_SECURONIX_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/sentinel_one/__init__.py b/uncoder-core/app/translator/platforms/sentinel_one/__init__.py new file mode 100644 index 00000000..d73e2978 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sentinel_one/__init__.py @@ -0,0 +1,4 @@ +from app.translator.platforms.sentinel_one.renders.s1_cti import S1EventsCTI # noqa: F401 +from app.translator.platforms.sentinel_one.renders.sentinel_one_power_query import ( + SentinelOnePowerQueryRender, # noqa: F401 +) diff --git a/uncoder-core/app/translator/platforms/sentinel_one/const.py b/uncoder-core/app/translator/platforms/sentinel_one/const.py new file mode 100644 index 00000000..09dd07fe --- /dev/null +++ b/uncoder-core/app/translator/platforms/sentinel_one/const.py @@ -0,0 +1,34 @@ +from app.translator.core.models.platform_details import PlatformDetails + +PLATFORM_DETAILS = {"group_id": "sentinel-one", "group_name": "SentinelOne"} + +SENTINEL_ONE_EVENTS_QUERY_DETAILS = { + "platform_id": "s1-events", + "name": "SentinelOne Events Query", + "platform_name": "Query (Events)", + **PLATFORM_DETAILS, +} + +SENTINEL_ONE_POWER_QUERY_DETAILS = { + "platform_id": "sentinel-one-power-query", + "name": "SentinelOne Power Query", + "platform_name": "Power Query", + **PLATFORM_DETAILS, +} + +DEFAULT_S1EVENTS_CTI_MAPPING = { + "SourceIP": "SrcIP", + "DestinationIP": "DstIP", + "Domain": "DNS", + "URL": "Url", + "HashMd5": "Md5", + "HashSha1": "Sha1", + "HashSha256": "Sha256", + "HashSha512": "Sha512", + "Emails": "emails", + "Files": "TgtFilePath", +} + + +sentinel_one_events_query_details = PlatformDetails(**SENTINEL_ONE_EVENTS_QUERY_DETAILS) +sentinel_one_power_query_details = PlatformDetails(**SENTINEL_ONE_POWER_QUERY_DETAILS) diff --git a/uncoder-core/app/translator/platforms/sentinel_one/escape_manager.py b/uncoder-core/app/translator/platforms/sentinel_one/escape_manager.py new file mode 100644 index 00000000..dc1658e9 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sentinel_one/escape_manager.py @@ -0,0 +1,15 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class SentinelOnePowerQueryEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [EscapeDetails(pattern=r"\\", escape_symbols=r"\\\\")], + ValueType.regex_value: [EscapeDetails(pattern=r"([$^*+()\[\]{}|.?\-\\])", escape_symbols=r"\\\1")], + } + + +sentinel_one_power_query_escape_manager = SentinelOnePowerQueryEscapeManager() diff --git a/uncoder-core/app/translator/platforms/sentinel_one/mapping.py b/uncoder-core/app/translator/platforms/sentinel_one/mapping.py new file mode 100644 index 00000000..789990c2 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sentinel_one/mapping.py @@ -0,0 +1,20 @@ +from app.translator.core.mapping import BaseStrictLogSourcesPlatformMappings, LogSourceSignature +from app.translator.platforms.sentinel_one.const import sentinel_one_power_query_details + + +class SentinelOnePowerQueryLogSourceSignature(LogSourceSignature): + def is_suitable(self) -> bool: + return True + + def __str__(self) -> str: + return "" + + +class SentinelOnePowerQueryMappings(BaseStrictLogSourcesPlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> SentinelOnePowerQueryLogSourceSignature: + ... + + +sentinel_one_power_query_query_mappings = SentinelOnePowerQueryMappings( + platform_dir="sentinel_one", platform_details=sentinel_one_power_query_details +) diff --git a/siem-converter/app/converter/platforms/qradar/renders/__init__.py b/uncoder-core/app/translator/platforms/sentinel_one/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/qradar/renders/__init__.py rename to uncoder-core/app/translator/platforms/sentinel_one/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/sentinel_one/renders/s1_cti.py b/uncoder-core/app/translator/platforms/sentinel_one/renders/s1_cti.py new file mode 100644 index 00000000..a83702d9 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sentinel_one/renders/s1_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.sentinel_one.const import DEFAULT_S1EVENTS_CTI_MAPPING, sentinel_one_events_query_details + + +@render_cti_manager.register +class S1EventsCTI(RenderCTI): + details: PlatformDetails = sentinel_one_events_query_details + + field_value_template: str = '"{value}"' + or_operator: str = ", " + group_or_operator: str = " OR " + or_group: str = "{processing_key} in contains anycase ({or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_S1EVENTS_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/sentinel_one/renders/sentinel_one_power_query.py b/uncoder-core/app/translator/platforms/sentinel_one/renders/sentinel_one_power_query.py new file mode 100644 index 00000000..28752105 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sentinel_one/renders/sentinel_one_power_query.py @@ -0,0 +1,105 @@ +from typing import Union + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.values import ValueType +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render import BaseFieldValueRender, PlatformQueryRender +from app.translator.core.str_value_manager import StrValueManager +from app.translator.managers import render_manager +from app.translator.platforms.sentinel_one.const import sentinel_one_power_query_details +from app.translator.platforms.sentinel_one.mapping import ( + SentinelOnePowerQueryMappings, + sentinel_one_power_query_query_mappings, +) +from app.translator.platforms.sentinel_one.str_value_manager import sentinel_one_power_query_str_value_manager + + +class SentinelOnePowerQueryFieldValue(BaseFieldValueRender): + details: PlatformDetails = sentinel_one_power_query_details + str_value_manager: StrValueManager = sentinel_one_power_query_str_value_manager + list_token = ", " + + @staticmethod + def _wrap_str_value(value: str) -> str: + return f'"{value}"' + + def equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.list_token.join( + self._pre_process_value(field, v, value_type=ValueType.value, wrap_str=True) for v in value + ) + return f"{field} in ({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True) + return f"{field} = {value}" + + def less_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True) + return f"{field} < {value}" + + def less_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True) + return f"{field} <= {value}" + + def greater_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True) + return f"{field} > {value}" + + def greater_or_equal_modifier(self, field: str, value: Union[int, str]) -> str: + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True) + return f"{field} >= {value}" + + def not_equal_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.list_token.join( + self._pre_process_value(field, v, value_type=ValueType.value, wrap_str=True, wrap_int=True) + for v in value + ) + return f"{field} != ({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{field} != {value}" + + def contains_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.list_token.join( + self._pre_process_value(field, v, value_type=ValueType.value, wrap_str=True, wrap_int=True) + for v in value + ) + return f"{field} contains ({values})" + value = self._pre_process_value(field, value, value_type=ValueType.value, wrap_str=True, wrap_int=True) + return f"{field} contains {value}" + + def endswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + return self.contains_modifier(field, value) + + def startswith_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + return self.contains_modifier(field, value) + + def regex_modifier(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: + if isinstance(value, list): + values = self.list_token.join( + self.str_value_manager.escape_manager.escape( + self._pre_process_value(field, v, value_type=ValueType.regex_value, wrap_str=True, wrap_int=True) + ) + for v in value + ) + return f"{field} matches ({values})" + value = self._pre_process_value(field, value, value_type=ValueType.regex_value, wrap_str=True, wrap_int=True) + value = self.str_value_manager.escape_manager.escape(value) + return f"{field} matches {value}" + + def is_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + return f'not ({field} matches "\\.*")' + + def is_not_none(self, field: str, value: DEFAULT_VALUE_TYPE) -> str: # noqa: ARG002 + return f'{field} matches "\\.*"' + + +@render_manager.register +class SentinelOnePowerQueryRender(PlatformQueryRender): + details: PlatformDetails = sentinel_one_power_query_details + mappings: SentinelOnePowerQueryMappings = sentinel_one_power_query_query_mappings + or_token = "or" + and_token = "and" + not_token = "not" + comment_symbol = "//" + field_value_render = SentinelOnePowerQueryFieldValue(or_token=or_token) diff --git a/uncoder-core/app/translator/platforms/sentinel_one/str_value_manager.py b/uncoder-core/app/translator/platforms/sentinel_one/str_value_manager.py new file mode 100644 index 00000000..14ad93db --- /dev/null +++ b/uncoder-core/app/translator/platforms/sentinel_one/str_value_manager.py @@ -0,0 +1,30 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from app.translator.core.str_value_manager import StrValueManager +from app.translator.platforms.sentinel_one.escape_manager import ( + SentinelOnePowerQueryEscapeManager, + sentinel_one_power_query_escape_manager, +) + + +class SentinelOnePowerQueryStrValueManager(StrValueManager): + escape_manager: SentinelOnePowerQueryEscapeManager = sentinel_one_power_query_escape_manager + + +sentinel_one_power_query_str_value_manager = SentinelOnePowerQueryStrValueManager() diff --git a/uncoder-core/app/translator/platforms/sigma/__init__.py b/uncoder-core/app/translator/platforms/sigma/__init__.py new file mode 100644 index 00000000..b4c8f9cd --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/__init__.py @@ -0,0 +1,3 @@ +from app.translator.platforms.sigma.parsers.sigma import SigmaParser # noqa: F401 +from app.translator.platforms.sigma.renders.sigma import SigmaRender # noqa: F401 +from app.translator.platforms.sigma.renders.sigma_cti import SigmaRenderCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/sigma/const.py b/uncoder-core/app/translator/platforms/sigma/const.py new file mode 100644 index 00000000..02dc8ce1 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/const.py @@ -0,0 +1,23 @@ +from app.translator.core.models.platform_details import PlatformDetails + +SIGMA_RULE_DETAILS = { + "name": "Sigma", + "platform_id": "sigma", + "platform_name": "Sigma", + "group_name": "Sigma", + "group_id": "sigma", +} + +DEFAULT_SIGMA_CTI_MAPPING = { + "SourceIP": "dst_ip", + "DestinationIP": "dst_ip", + "Domain": "dest_domain", + "URL": "url", + "HashMd5": "Hashes", + "HashSha1": "Hashes", + "HashSha256": "Hashes", + "HashSha512": "Hashes", +} + + +sigma_rule_details = PlatformDetails(**SIGMA_RULE_DETAILS) diff --git a/uncoder-core/app/translator/platforms/sigma/escape_manager.py b/uncoder-core/app/translator/platforms/sigma/escape_manager.py new file mode 100644 index 00000000..26df6163 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/escape_manager.py @@ -0,0 +1,15 @@ +from typing import ClassVar + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.escape_manager import EscapeManager +from app.translator.core.models.escape_details import EscapeDetails + + +class SigmaEscapeManager(EscapeManager): + escape_map: ClassVar[dict[str, list[EscapeDetails]]] = { + ValueType.value: [EscapeDetails(pattern=r"([*?\\])", escape_symbols=r"\\\1")], + ValueType.regex_value: [EscapeDetails(pattern=r"([$^*+()\[\]{}|.?\-\\])", escape_symbols=r"\\\1")] + } + + +sigma_escape_manager = SigmaEscapeManager() diff --git a/uncoder-core/app/translator/platforms/sigma/mapping.py b/uncoder-core/app/translator/platforms/sigma/mapping.py new file mode 100644 index 00000000..6180c948 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/mapping.py @@ -0,0 +1,66 @@ +from typing import Optional, Union + +from app.translator.core.mapping import DEFAULT_MAPPING_NAME, BasePlatformMappings, LogSourceSignature, SourceMapping +from app.translator.platforms.sigma.const import sigma_rule_details + + +class SigmaLogSourceSignature(LogSourceSignature): + def __init__( + self, + product: Optional[list[str]], + category: Optional[list[str]], + service: Optional[list[str]], + default_source: dict = None, + ): + self.products = set(product or []) + self.categories = set(category or []) + self.services = set(service or []) + self._default_source = default_source or {} + + def is_suitable( + self, + service: Optional[list[str]] = None, + product: Optional[list[str]] = None, + category: Optional[list[str]] = None + ) -> bool: + product_match = set(product_.lower() for product_ in product or []).issubset(self.products) if product else False + category_match = set(category_.lower() for category_ in category or []).issubset(self.categories) if category else False + service_match = set(service_.lower() for service_ in service or [] or []).issubset(self.services) if service else False + if not product and not service: + return category_match + return product_match and service_match or product_match and category_match + + def __str__(self) -> str: + raise NotImplementedError + + @property + def log_sources(self) -> dict: + return self._default_source + + +class SigmaMappings(BasePlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> SigmaLogSourceSignature: + product = mapping.get("log_source", {}).get("product") + service = mapping.get("log_source", {}).get("service") + category = mapping.get("log_source", {}).get("category") + default_log_source = mapping["default_log_source"] + return SigmaLogSourceSignature( + product=product, service=service, category=category, default_source=default_log_source + ) + + def get_source_mappings_by_fields_and_log_sources( + self, field_names: list[str], log_sources: dict[str, list[Union[int, str]]] + ) -> list[SourceMapping]: + source_mappings = [] + for source_mapping in self._source_mappings.values(): + if source_mapping.source_id == DEFAULT_MAPPING_NAME: + continue + + log_source_signature: LogSourceSignature = source_mapping.log_source_signature + if log_source_signature and log_source_signature.is_suitable(**log_sources): + source_mappings.append(source_mapping) + + return source_mappings or [self._source_mappings[DEFAULT_MAPPING_NAME]] + + +sigma_rule_mappings = SigmaMappings(platform_dir="sigma", platform_details=sigma_rule_details) diff --git a/siem-converter/app/converter/platforms/qualys/__init__.py b/uncoder-core/app/translator/platforms/sigma/models/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/qualys/__init__.py rename to uncoder-core/app/translator/platforms/sigma/models/__init__.py diff --git a/uncoder-core/app/translator/platforms/sigma/models/compiler.py b/uncoder-core/app/translator/platforms/sigma/models/compiler.py new file mode 100644 index 00000000..c6092498 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/models/compiler.py @@ -0,0 +1,62 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from typing import Union + +from app.translator.core.custom_types.tokens import GroupType, LogicalOperatorType +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.keyword import Keyword +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.platforms.sigma.models.group import Group +from app.translator.platforms.sigma.models.operator import NOT, Operator + + +class DataStructureCompiler: + # pylint: disable=inconsistent-return-statements + def generate(self, tokens: list, group: Group = None) -> Union[Group, tuple[Group, list]]: # noqa: PLR0911 + if not tokens: + group.finalize() + return group + group = group if group else Group() + token = tokens[0] + if isinstance(token, (FieldValue, Keyword)): + group += token + return self.generate(tokens=tokens[1::], group=group) + if token.token_type in (LogicalOperatorType.AND, LogicalOperatorType.OR): + group.items = Operator(operator_type=token.token_type) + return self.generate(tokens=tokens[1::], group=group) + if token.token_type == LogicalOperatorType.NOT: + if isinstance(tokens[1], (FieldValue, Keyword)): + tokens.insert(2, Identifier(token_type=GroupType.R_PAREN)) + tokens.insert(1, Identifier(token_type=GroupType.L_PAREN)) + sub_group = Group() + sub_group.items = NOT() + not_sub_group, new_tokens = self.generate(tokens=tokens[1::], group=sub_group) + group += not_sub_group + return self.generate(new_tokens, group) + if token.token_type == GroupType.L_PAREN: + sub_group, new_tokens = self.generate(tokens=tokens[1::]) + if not sub_group.is_null: + group += sub_group + if isinstance(group.items, NOT): + group.finalize() + return group, new_tokens + return self.generate(new_tokens, group) + if token.token_type == GroupType.R_PAREN: + group.finalize() + return group, tokens[1::] diff --git a/siem-converter/app/converter/core/models/group.py b/uncoder-core/app/translator/platforms/sigma/models/group.py similarity index 90% rename from siem-converter/app/converter/core/models/group.py rename to uncoder-core/app/translator/platforms/sigma/models/group.py index d78eb8e8..9d12e145 100644 --- a/siem-converter/app/converter/core/models/group.py +++ b/uncoder-core/app/translator/platforms/sigma/models/group.py @@ -1,14 +1,10 @@ -from app.converter.core.models.operator import OR, AND, NOT -from app.converter.core.operator_types.tokens import GroupType +from app.translator.platforms.sigma.models.operator import AND, NOT, OR class Group: - token_type = GroupType.GROUP - parent_group = [] - sub_group = None - last_field = None - def __init__(self): + self.sub_group = None + self.last_field = None self._items = None def __add__(self, other): @@ -24,11 +20,11 @@ def __add__(self, other): self._items += other self.last_field = other return self - + @property - def _hash(self): + def _hash(self) -> int: return hash(str(self._items)) - + def __hash__(self): return hash(str(self._items)) diff --git a/uncoder-core/app/translator/platforms/sigma/models/modifiers.py b/uncoder-core/app/translator/platforms/sigma/models/modifiers.py new file mode 100644 index 00000000..fa98c8ce --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/models/modifiers.py @@ -0,0 +1,99 @@ +from typing import ClassVar, Optional, Union + +from app.translator.core.custom_types.tokens import GroupType, LogicalOperatorType, OperatorType +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.core.str_value_manager import StrValue +from app.translator.platforms.sigma.str_value_manager import sigma_str_value_manager + +_MULTY_MODIFIER_LEN = 2 + + +class ModifierManager: + and_token = Identifier(token_type=LogicalOperatorType.AND) + or_token = Identifier(token_type=LogicalOperatorType.OR) + + modifier_map: ClassVar[dict[str, str]] = {"re": OperatorType.REGEX} + + def __validate_modifiers(self, modifiers: Union[str, list]) -> bool: + if isinstance(modifiers, list) and len(modifiers) > _MULTY_MODIFIER_LEN: + raise + if isinstance(modifiers, list): + return all(self.__validate_modifiers(modifier) for modifier in modifiers) + return True + + def map_modifier(self, modifier: str) -> Identifier: + return Identifier(token_type=self.modifier_map.get(modifier, modifier)) + + def modifier_all(self, field_name: str, modifier: str, values: Union[str, list[str]]) -> Union[tuple, list]: + if (isinstance(values, list) and len(values) == 1) or isinstance(values, (str, int)): + operator = self.map_modifier(modifier=modifier) + values = self.convert_values_to_str_values(values, modifier) + return (FieldValue(source_name=field_name, operator=operator, value=values),) + + tokens = [] + for value in values: + tokens.extend(self.modifier_all(field_name=field_name, modifier=modifier, values=value)) + tokens.append(self.and_token) + return [Identifier(token_type=GroupType.L_PAREN), *tokens[:-1], Identifier(token_type=GroupType.R_PAREN)] + + @staticmethod + def __prepare_windash_value(value: str) -> Union[str, list[str]]: + if value.startswith(r"/"): + return [value, value.replace(r"/", "-", 1)] + if value.startswith(r"-"): + return [value, value.replace("-", r"/", 1)] + return value + + def modifier_windash( + self, field_name: str, modifier: Union[str, list], values: Union[str, list[str]] + ) -> Union[tuple, list]: + if isinstance(values, list): + tokens = [] + for value in values: + tokens.extend(self.modifier_windash(field_name=field_name, modifier=modifier, values=value)) + tokens.append(self.or_token) + return [Identifier(token_type=GroupType.L_PAREN), *tokens[:-1], Identifier(token_type=GroupType.R_PAREN)] + operator = self.map_modifier(modifier=modifier) + values = self.__prepare_windash_value(value=values) + values = self.convert_values_to_str_values(values, modifier) + return (FieldValue(source_name=field_name, operator=operator, value=values),) + + def apply_multi_modifier( + self, field_name: str, modifier: list, values: Union[str, list[str]] + ) -> Optional[Union[tuple, list]]: + if modifier[-1] == "all": + return self.modifier_all(field_name=field_name, modifier=modifier[0], values=values) + if modifier[-1] == "windash": + return self.modifier_windash(field_name=field_name, modifier=modifier[0], values=values) + + raise NotImplementedError + + def apply_modifier(self, field_name: str, modifier: list, values: Union[int, str, list[Union[int, str]]]) -> tuple: + modifier = modifier[0] + if modifier == "windash": + modifier = OperatorType.EQ + return self.modifier_windash(field_name=field_name, modifier=modifier, values=values) + operator = self.map_modifier(modifier=modifier) + values = self.convert_values_to_str_values(values, modifier) + return (FieldValue(source_name=field_name, operator=operator, value=values),) + + @staticmethod + def convert_values_to_str_values( + values: Union[int, str, list[Union[int, str]]], operator: str + ) -> Union[StrValue, list[StrValue]]: + if not isinstance(values, list): + values = [values] + + if operator == "re": + return [sigma_str_value_manager.from_re_str_to_container(str(value)) for value in values] + return [sigma_str_value_manager.from_str_to_container(str(value)) for value in values] + + def create_token(self, field_name: str, modifier: list, value: Union[str, list[str], int]) -> Union[tuple, list]: + if len(modifier) == _MULTY_MODIFIER_LEN: + return self.apply_multi_modifier(field_name=field_name, modifier=modifier, values=value) + return self.apply_modifier(field_name=field_name, modifier=modifier, values=value) + + def generate(self, field_name: str, modifier: list, value: Union[str, list[str], int]) -> Union[tuple, list]: + if self.__validate_modifiers(modifiers=modifier): + return self.create_token(field_name=field_name, modifier=modifier, value=value) diff --git a/uncoder-core/app/translator/platforms/sigma/models/operator.py b/uncoder-core/app/translator/platforms/sigma/models/operator.py new file mode 100644 index 00000000..b1610bed --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/models/operator.py @@ -0,0 +1,50 @@ +from abc import ABC + +from app.translator.core.custom_types.tokens import LogicalOperatorType + + +class BaseOperator(ABC): + token_type: str = None + + def __init__(self, items=None): + self.items = [] if items is None else items + + def __add__(self, other): + if isinstance(other, list): + self.items.extend(other) + else: + self.items.append(other) + return self + + def __repr__(self): + return f"{self.token_type}({self.items})" + + def remove_last_item(self): + self.items = self.items[:-1] + + +class AND(BaseOperator): + token_type = LogicalOperatorType.AND + + +class OR(BaseOperator): + token_type = LogicalOperatorType.OR + + +class NOT(BaseOperator): + token_type = LogicalOperatorType.NOT + + def __add__(self, other): + self.items = other + return self + + +class Operator: + def __new__(cls, operator_type, items=None): + if operator_type == LogicalOperatorType.AND: + return AND() + elif operator_type == LogicalOperatorType.OR: + return OR() + elif operator_type == LogicalOperatorType.NOT: + return NOT(items) + return None diff --git a/siem-converter/app/converter/platforms/qualys/mappings/__init__.py b/uncoder-core/app/translator/platforms/sigma/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/qualys/mappings/__init__.py rename to uncoder-core/app/translator/platforms/sigma/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/sigma/parsers/sigma.py b/uncoder-core/app/translator/platforms/sigma/parsers/sigma.py new file mode 100644 index 00000000..384b7a30 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/parsers/sigma.py @@ -0,0 +1,134 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from datetime import timedelta +from typing import Optional, Union + +from app.translator.core.exceptions.core import SigmaRuleValidationException +from app.translator.core.mixins.rule import YamlRuleMixin +from app.translator.core.models.query_tokens.field import Field +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import QueryParser +from app.translator.core.tokenizer import QueryTokenizer +from app.translator.managers import parser_manager +from app.translator.platforms.sigma.const import sigma_rule_details +from app.translator.platforms.sigma.mapping import SigmaMappings, sigma_rule_mappings +from app.translator.platforms.sigma.tokenizer import SigmaConditionTokenizer, SigmaTokenizer + + +@parser_manager.register_main +class SigmaParser(QueryParser, YamlRuleMixin): + details: PlatformDetails = sigma_rule_details + condition_tokenizer = SigmaConditionTokenizer() + tokenizer: SigmaTokenizer = SigmaTokenizer() + mappings: SigmaMappings = sigma_rule_mappings + mandatory_fields = {"title", "description", "logsource", "detection"} + + wrapped_with_comment_pattern = r"^\s*#.*(?:\n|$)" + + @staticmethod + def __parse_false_positives(false_positives: Union[str, list[str], None]) -> list: + if isinstance(false_positives, str): + return [i.strip() for i in false_positives.split(",")] + return false_positives + + @staticmethod + def __parse_timeframe(raw_timeframe: Optional[str] = None) -> Optional[timedelta]: + if raw_timeframe: + time_unit = raw_timeframe[-1].lower() + time_value = raw_timeframe[:-1] + + if time_value.isdigit(): + if time_unit == 's': + return timedelta(seconds=int(time_value)) + if time_unit == 'm': + return timedelta(minutes=int(time_value)) + if time_unit == 'h': + return timedelta(hours=int(time_value)) + if time_unit == 'd': + return timedelta(days=int(time_value)) + + def _get_meta_info( + self, + rule: dict, + source_mapping_ids: list[str], + parsed_logsources: dict, + fields_tokens: list[Field], + sigma_fields_tokens: Union[list[Field], None] = None, + ) -> MetaInfoContainer: + return MetaInfoContainer( + title=rule.get("title"), + id_=rule.get("id"), + description=rule.get("description"), + author=rule.get("author", '').split(', '), + date=rule.get("date"), + output_table_fields=sigma_fields_tokens, + query_fields=fields_tokens, + references=rule.get("references", []), + license_=rule.get("license"), + mitre_attack=self.parse_mitre_attack(rule.get("tags", [])), + severity=rule.get("level"), + status=rule.get("status"), + tags=sorted(set(rule.get("tags", []))), + false_positives=self.__parse_false_positives(rule.get("falsepositives")), + source_mapping_ids=source_mapping_ids, + parsed_logsources=parsed_logsources, + timeframe=self.__parse_timeframe(rule.get('detection', {}).get('timeframe')) + ) + + def __validate_rule(self, rule: dict): + if missing_fields := self.mandatory_fields.difference(set(rule.keys())): + raise SigmaRuleValidationException(missing_fields=list(missing_fields)) + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + return RawQueryContainer(query=text, language=language) + + def parse(self, raw_query_container: RawQueryContainer) -> TokenizedQueryContainer: + sigma_rule = self.load_rule(text=raw_query_container.query) + self.__validate_rule(rule=sigma_rule) + log_sources = { + key: [value] + for key, value in (sigma_rule.get("logsource", {})).items() + if key in ("product", "service", "category") + } + tokens = self.tokenizer.tokenize(detection=sigma_rule.get("detection")) + field_tokens = [token.field for token in QueryTokenizer.filter_tokens(tokens, FieldValue)] + field_names = [field.source_name for field in field_tokens] + source_mappings = self.mappings.get_source_mappings_by_fields_and_log_sources( + field_names=field_names, log_sources=log_sources + ) + QueryTokenizer.set_field_tokens_generic_names_map(field_tokens, source_mappings, self.mappings.default_mapping) + sigma_fields_tokens = None + if sigma_fields := sigma_rule.get("fields"): + sigma_fields_tokens = [Field(source_name=field) for field in sigma_fields] + QueryTokenizer.set_field_tokens_generic_names_map( + sigma_fields_tokens, source_mappings, self.mappings.default_mapping + ) + return TokenizedQueryContainer( + tokens=tokens, + meta_info=self._get_meta_info( + rule=sigma_rule, + source_mapping_ids=[source_mapping.source_id for source_mapping in source_mappings], + sigma_fields_tokens=sigma_fields_tokens, + parsed_logsources=log_sources, + fields_tokens=field_tokens + ), + ) diff --git a/siem-converter/app/converter/platforms/qualys/renders/__init__.py b/uncoder-core/app/translator/platforms/sigma/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/qualys/renders/__init__.py rename to uncoder-core/app/translator/platforms/sigma/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/sigma/renders/sigma.py b/uncoder-core/app/translator/platforms/sigma/renders/sigma.py new file mode 100644 index 00000000..e20b7682 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/renders/sigma.py @@ -0,0 +1,331 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from typing import Any, Optional, Union + +import yaml + +from app.translator.const import DEFAULT_VALUE_TYPE +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.custom_types.tokens import OperatorType +from app.translator.core.custom_types.values import ValueType +from app.translator.core.mapping import DEFAULT_MAPPING_NAME, SourceMapping +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.keyword import Keyword +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.render import QueryRender +from app.translator.core.str_value_manager import StrValue +from app.translator.managers import render_manager +from app.translator.platforms.sigma.const import sigma_rule_details +from app.translator.platforms.sigma.mapping import SigmaLogSourceSignature, SigmaMappings, sigma_rule_mappings +from app.translator.platforms.sigma.models.compiler import DataStructureCompiler +from app.translator.platforms.sigma.models.group import Group +from app.translator.platforms.sigma.models.operator import AND, NOT, OR +from app.translator.platforms.sigma.str_value_manager import sigma_str_value_manager +from app.translator.tools.utils import get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated Sigma Rule" + + +@render_manager.register +class SigmaRender(QueryRender): + selection_name = "selection" + selection_num = 0 + keyword_name = "keyword" + keyword_num = 0 + + comment_symbol = "#" + is_single_line_comment = True + + mappings: SigmaMappings = sigma_rule_mappings + details: PlatformDetails = sigma_rule_details + str_value_manager = sigma_str_value_manager + + @property + def selection(self): + return f"{self.selection_name}{self.selection_num}" + + @property + def keyword(self): + return f"{self.selection_name}{self.keyword_num}" + + def increase_selection(self) -> None: + self.selection_num += 1 + + def increase_keyword(self) -> None: + self.keyword_num += 1 + + def reset_counters(self) -> None: + self.keyword_num = 0 + self.selection_num = 0 + + def generate_data_structure(self, data: Any, source_mapping: SourceMapping): + if isinstance(data, OR): + return self.generate_or(data, source_mapping) + elif isinstance(data, AND): + return self.generate_and(data, source_mapping) + elif isinstance(data, NOT): + return self.generate_not(data, source_mapping) + elif isinstance(data, FieldValue): + return self.generate_field(data, source_mapping) + elif isinstance(data, Keyword): + return self.generate_keyword(data) + elif isinstance(data, Group): + return self.generate_group(data, source_mapping) + return None + + def generate_group(self, data: Any, source_mapping: SourceMapping): + sub_group = self.generate_data_structure(data.items, source_mapping) + if isinstance(sub_group, list): + return sub_group + if condition := sub_group.get("condition"): + sub_group["condition"] = ( + condition + if condition.count(self.selection_name) == 1 or condition.count(self.keyword_name) == 1 + else f"({condition})" + ) + return sub_group + + def generate_or(self, data: Any, source_mapping: SourceMapping): + result = {} + condition = "" + for i, item in enumerate(data.items): + updated_node = self.generate_data_structure(item, source_mapping) + if isinstance(updated_node, list): + if result and isinstance(result.get(self.keyword), list): + result[self.keyword].extend(updated_node) + else: + if self.keyword_num == self.selection_num: + self.increase_keyword() + self.increase_selection() + condition += f"{' or ' if condition else ''}{self.selection}" + result[self.selection] = updated_node + elif ( + result + and len(set(result.get(self.selection, [])).intersection(set(updated_node))) != 0 + and isinstance(data.items[i - 1], FieldValue) + and len(updated_node) == 1 + and self.selection not in updated_node + ): + field_name = list(updated_node.keys())[0] + if isinstance(result[self.selection][field_name], list): + result[self.selection][field_name].append(updated_node[field_name]) + else: + result[self.selection][field_name] = [result[self.selection][field_name], updated_node[field_name]] + elif not result and self.selection in updated_node: + result = updated_node + t_c = result.get("condition") + condition = ( + t_c + if not condition + else f"{condition} or {t_c if t_c.count(self.selection_name) == 1 else f'({t_c})'}" + ) + elif result and self.selection in updated_node: + result.update(updated_node) + t_c = result.get("condition") + condition += ( + f"{' or ' if condition else ''}{t_c if t_c.count(self.selection_name) == 1 else f'({t_c})'}" + ) + else: + self.increase_selection() + result[self.selection] = updated_node + condition += f"{' or ' if condition else ''}{self.selection}" + result.update({"condition": condition}) + self.increase_keyword() + return result + + def generate_and(self, data: Any, source_mapping: SourceMapping): + result = {} + condition = "" + for item in data.items: + updated_node = self.generate_data_structure(item, source_mapping) + if isinstance(updated_node, list): + self.increase_keyword() + self.increase_selection() + condition += f"{' and ' if condition else ''}{self.selection}" + result[self.selection] = updated_node + elif ( + result + and isinstance(updated_node, dict) + and isinstance(result.get(self.selection, []), dict) + and len(set(result.get(self.selection, [])).intersection(set(updated_node))) == 0 + and self.selection not in updated_node + ): + if isinstance(result[self.selection], list): + result[self.selection].append(updated_node) + else: + result[self.selection].update(updated_node) + elif not result and self.selection in updated_node: + result = updated_node + t_c = result.get("condition") + condition = t_c if not condition else f"{condition} and {t_c}" + elif result and self.selection in updated_node: + result.update(updated_node) + condition += f"{' and ' if condition else ''}{result.get('condition')}" + else: + self.increase_selection() + result[self.selection] = updated_node + condition += f"{' and ' if condition else ''}{self.selection}" + result.update({"condition": condition}) + self.increase_keyword() + return result + + def generate_not(self, data: Any, source_mapping: SourceMapping): + not_node = self.generate_data_structure(data.items, source_mapping) + if isinstance(not_node, list): + self.increase_selection() + not_node = {self.selection: not_node, "condition": f"not {self.selection}"} + elif not not_node.get("condition"): + self.increase_selection() + if list(not_node.keys())[0] == self.keyword_name: + not_node = list(not_node.values()) + not_node = {self.selection: not_node, "condition": f"not {self.selection}"} + elif condition := not_node.get("condition"): + not_node["condition"] = f"not {condition}" + return not_node + + def generate_field(self, data: FieldValue, source_mapping: SourceMapping): + field_name = self.mappings.map_field(data.field, source_mapping)[0] + if data.operator.token_type not in ( + OperatorType.EQ, + OperatorType.LT, + OperatorType.LTE, + OperatorType.GT, + OperatorType.GTE, + OperatorType.NOT_EQ, + ): + field_name = f"{field_name}|{data.operator.token_type}" + + value_type_map = { + OperatorType.REGEX: ValueType.regex_value + } + value_type = value_type_map.get(data.operator.token_type, ValueType.value) + values = self.__pre_process_values(data.values, value_type) + if len(values) == 1: + return {field_name: values[0]} + elif len(values) == 0: + return {field_name: ""} + return {field_name: values} + + def __pre_process_values( + self, + values: DEFAULT_VALUE_TYPE, + value_type: str = ValueType.value + ) -> list[Union[int, str]]: + processed = [] + for v in values: + if isinstance(v, StrValue): + processed.append(self.str_value_manager.from_container_to_str(v, value_type=value_type)) + elif isinstance(v, str): + processed.append(v) + else: + processed.append(v) + + return processed + + def generate_keyword(self, data: Keyword): + return self.__pre_process_values(data.values) + + def __base_detection(self, data: dict): + self.increase_selection() + return {self.selection: data, "condition": self.selection} + + @staticmethod + def generate_aggregation(detection: dict, agg_data): + if timeframe := agg_data.get("timeframe"): + detection.update({"timeframe": timeframe}) + if agg_func := agg_data.get("agg_func"): + agg_condition = f" | {agg_func}" + if agg_field := agg_data.get("agg_field"): + agg_condition += f"({agg_field})" + else: + agg_condition += "()" + + if group_by := agg_data.get("group_by"): + agg_condition += f" by {', '.join(group_by)}" + + if agg_value := agg_data.get("agg_value"): + agg_condition += f" {agg_value}" + + detection["condition"] += agg_condition + return detection + + def generate_detection(self, data: Any, source_mapping: SourceMapping) -> dict: + detection = self.generate_data_structure(data, source_mapping) + if self.selection not in detection: + detection = self.__base_detection(detection) + condition = detection.pop("condition") + if condition.startswith("(") and condition.endswith(")") and "(" not in condition[1:-1]: + condition = condition.rstrip(")").lstrip("(") + detection.update({"condition": condition}) + self.reset_counters() + + return detection + + def __get_source_mapping(self, source_mapping_ids: list[str]) -> SourceMapping: + for source_mapping_id in source_mapping_ids: + if source_mapping := self.mappings.get_source_mapping(source_mapping_id): + return source_mapping + + return self.mappings.get_source_mapping(DEFAULT_MAPPING_NAME) + + def generate_from_raw_query_container(self, query_container: RawQueryContainer) -> str: + raise NotImplementedError + + def generate_from_tokenized_query_container(self, query_container: TokenizedQueryContainer) -> str: + self.reset_counters() + + meta_info = query_container.meta_info + source_mapping = self.__get_source_mapping(meta_info.source_mapping_ids) + log_source_signature: SigmaLogSourceSignature = source_mapping.log_source_signature + prepared_data_structure = DataStructureCompiler().generate(tokens=query_container.tokens) + rendered_functions = self.platform_functions.render(query_container.functions.functions, source_mapping) + not_supported_functions = query_container.functions.not_supported + rendered_functions.not_supported + + description_str = get_rule_description_str( + description=meta_info.description or _AUTOGENERATED_TEMPLATE, + license_=meta_info.license + ) + + rule = { + "title": meta_info.title or _AUTOGENERATED_TEMPLATE, + "id": meta_info.id, + "description": description_str, + "status": "experimental", + "author": query_container.meta_info.author_str, + "references": meta_info.references, + "tags": meta_info.tags, + "logsource": log_source_signature.log_sources, + "fields": meta_info.output_table_fields or [], + "detection": self.generate_detection(prepared_data_structure, source_mapping=source_mapping), + "level": meta_info.severity or SeverityType.low, + "falsepositives": "", + } + rule = yaml.dump(rule, default_flow_style=False, sort_keys=False) + if not_supported_functions: + rendered_not_supported = self.render_not_supported_functions(not_supported_functions) + return rule + rendered_not_supported + return rule + + def generate(self, raw_query_container: RawQueryContainer, tokenized_query_container: Optional[TokenizedQueryContainer]) -> str: + if tokenized_query_container: + return self.generate_from_tokenized_query_container(tokenized_query_container) + + return self.generate_from_raw_query_container(raw_query_container) + diff --git a/uncoder-core/app/translator/platforms/sigma/renders/sigma_cti.py b/uncoder-core/app/translator/platforms/sigma/renders/sigma_cti.py new file mode 100644 index 00000000..680965f1 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/renders/sigma_cti.py @@ -0,0 +1,43 @@ +import uuid +import yaml + +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.models.iocs import IocsChunkValue +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.sigma.const import sigma_rule_details, DEFAULT_SIGMA_CTI_MAPPING + + +@render_cti_manager.register +class SigmaRenderCTI(RenderCTI): + details: PlatformDetails = sigma_rule_details + default_mapping = DEFAULT_SIGMA_CTI_MAPPING + + def render(self, data: list[list[IocsChunkValue]]) -> list[str]: + final_result = [] + for iocs_chunk in data: + data_values = self.collect_sigma_data_values(iocs_chunk) + rule = { + "title": "Sigma automatically generated based on IOCs", + "id": uuid.uuid4().__str__(), + "description": "Detects suspicious activity based on IOCs.", + "status": "experimental", + "author": "SOC Prime", + "logsource": {"product": "windows"}, + "fields": list(data_values.keys()), + "detection": {"selection": data_values, "condition": "selection"}, + "level": SeverityType.low, + "falsepositives": "", + } + final_result.append(yaml.dump(rule, default_flow_style=False, sort_keys=False)) + return final_result + + def collect_sigma_data_values(self, chunk: list[IocsChunkValue]) -> dict: + raw_data_values = {} + for value in chunk: + if value.platform_field in raw_data_values.keys(): + raw_data_values[value.platform_field].append(value.value) + else: + raw_data_values[value.platform_field] = [value.value] + return raw_data_values diff --git a/uncoder-core/app/translator/platforms/sigma/str_value_manager.py b/uncoder-core/app/translator/platforms/sigma/str_value_manager.py new file mode 100644 index 00000000..6d3abe56 --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/str_value_manager.py @@ -0,0 +1,79 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" +from typing import Optional + +from app.translator.core.custom_types.values import ValueType +from app.translator.core.str_value_manager import ( + RE_STR_SPEC_SYMBOLS_MAP, + ReDigitalSymbol, + ReWhiteSpaceSymbol, + ReWordBoundarySymbol, + ReWordSymbol, + SingleSymbolWildCard, + StrValue, + StrValueManager, + UnboundLenWildCard +) +from app.translator.platforms.sigma.escape_manager import sigma_escape_manager + + +class SigmaStrValueManager(StrValueManager): + escape_manager = sigma_escape_manager + str_spec_symbols_map = {"?": SingleSymbolWildCard, "*": UnboundLenWildCard} + re_str_alpha_num_symbols_map = { + "b": ReWordBoundarySymbol, + "w": ReWordSymbol, + "d": ReDigitalSymbol, + "s": ReWhiteSpaceSymbol + } + re_str_spec_symbols_map = RE_STR_SPEC_SYMBOLS_MAP + + def from_str_to_container( + self, + value: str, + value_type: str = ValueType.value, # noqa: ARG002 + escape_symbol: Optional[str] = None # noqa: ARG002 + ) -> StrValue: + split = [] + prev_char = None + for char in value: + if char == "\\": + if prev_char == "\\": + split.append(char) + prev_char = None + continue + elif char in self.str_spec_symbols_map: + if prev_char == "\\": + split.append(char) + else: + split.append(self.str_spec_symbols_map[char]()) + else: + if prev_char == "\\": + split.append(prev_char) + split.append(char) + + prev_char = char + + if prev_char == "\\": + split.append(prev_char) + + return StrValue(value, self._concat(split)) + + +sigma_str_value_manager = SigmaStrValueManager() diff --git a/uncoder-core/app/translator/platforms/sigma/tokenizer.py b/uncoder-core/app/translator/platforms/sigma/tokenizer.py new file mode 100644 index 00000000..faa0970a --- /dev/null +++ b/uncoder-core/app/translator/platforms/sigma/tokenizer.py @@ -0,0 +1,185 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re +from typing import Union + +from app.translator.core.custom_types.tokens import GroupType, LogicalOperatorType +from app.translator.core.exceptions.parser import TokenizerGeneralException +from app.translator.core.models.query_tokens.field_value import FieldValue +from app.translator.core.models.query_tokens.keyword import Keyword +from app.translator.core.models.query_tokens.identifier import Identifier +from app.translator.platforms.sigma.models.modifiers import ModifierManager + + +class Selection: + token_type = "selection" + + def __init__(self, name): + self.name = name + + def __repr__(self): + return f"Selection({self.name})" + + +class SigmaTokenizer: + modifier_manager = ModifierManager() + + def __init__(self): + self.supported_selection_types = {dict: self.__parse_and_selection, list: self.__parse_or_selection} + + def __parse_field(self, field_name: str, values: Union[int, str, list[Union[int, str]]]) -> Union[list, FieldValue]: + field_name, *modifier = field_name.split("|") if "|" in field_name else (field_name, "=") + return self.modifier_manager.generate(field_name=field_name, modifier=modifier, value=values) + + @staticmethod + def __parse_keyword(values): + return (Keyword(value=values),) + + def __parse_and_selection(self, selection: dict): + result = [] + for field, values in selection.items(): + token = self.__parse_field(field_name=field, values=values) + result.extend((*token, Identifier(token_type=LogicalOperatorType.AND))) + return [Identifier(token_type=GroupType.L_PAREN), *result[:-1], Identifier(token_type=GroupType.R_PAREN)] + + def __parse_or_selection(self, selection: list): + result = [] + for field_values in selection: + field, values = next(iter(field_values.items())) + token = self.__parse_field(field_name=field, values=values) + result.extend((*token, Identifier(token_type=LogicalOperatorType.OR))) + return [Identifier(token_type=GroupType.L_PAREN), *result[:-1], Identifier(token_type=GroupType.R_PAREN)] + + def __get_one_field_tokenized(self, selection: Union[dict, list]): + if isinstance(selection, dict): + field, value = next(iter(selection.items())) + return self.__parse_field(field_name=field, values=value) + elif isinstance(selection, list) and all(isinstance(i, dict) for i in selection): + field, value = next(iter(selection[0].items())) + return self.__parse_field(field_name=field, values=value) + else: + return self.__parse_keyword(values=selection) + + def __parse_selection(self, selection: Union[dict, list]): + if len(selection) == 1 and type(selection) in self.supported_selection_types: + return self.__get_one_field_tokenized(selection=selection) + if isinstance(selection, list) and all(isinstance(i, str) for i in selection): + return self.__parse_keyword(selection) + if selection_method := self.supported_selection_types.get(type(selection)): + return selection_method(selection=selection) + raise Exception("Unsupported sigma type") + + def tokenize(self, detection: dict) -> list: + condition = detection.pop("condition") + condition_tokens = SigmaConditionTokenizer().tokenize(condition=condition, detection=detection) + tokenized = [] + for token in condition_tokens: + if isinstance(token, Selection): + selection_values = detection.get(token.name) + selection = self.__parse_selection(selection=selection_values) + tokenized.extend(selection) + else: + tokenized.append(token) + return tokenized + + +class SigmaConditionTokenizer: + logical_operator_pattern = r"\s?(?Pand|or|not)\s?" + selection_pattern = r"(?P[$a-zA-Z\._\-0-9]+)" + multi_selection_pattern = r"(?P[$a-zA-Z\._\-0-9]+)(?:\*){1}" + one_of_selection = "1 of " + all_of_selection = "all of " + + def __init__(self): + self.detection = {} + + def __get_selection(self, condition: str) -> tuple[Selection, str]: + selection_search = re.search(self.selection_pattern, condition) + if selection_search is None: + raise TokenizerGeneralException(error=f"Selection couldn't be found in condition: {condition}") + selection = selection_search.group("selection_name") + pos = selection_search.end() + return Selection(name=selection), condition[pos:] + + def __get_group(self, condition: str, operator: Identifier) -> tuple[list, str]: + selection_pattern_search = re.search(self.multi_selection_pattern, condition) + if selection_pattern_search is None: + raise TokenizerGeneralException(error=f"Selection couldn't be found in condition: {condition}") + selection_pattern = selection_pattern_search.group("selection_pattern") + pos = selection_pattern_search.end() + tokens = [] + for selection in self.detection: + if selection.startswith(selection_pattern): + tokens.append(Selection(name=selection)) + tokens.append(operator) + return [ + Identifier(token_type=GroupType.L_PAREN), + *tokens[:-1], + Identifier(token_type=GroupType.R_PAREN), + ], condition[pos:] + + @staticmethod + def get_missed_parentheses(tokens: list[Union[Selection, Identifier]]) -> list[int]: + missed_indices = [] + for index in range(len(tokens) - 1): + token = tokens[index] + next_token = tokens[index + 1] + if token.token_type == LogicalOperatorType.NOT and next_token.token_type != GroupType.L_PAREN: + missed_indices.append(index + 1) + return missed_indices + + def __add_parentheses_after_and_not( + self, tokens: list[Union[Selection, Identifier]] + ) -> list[Union[Selection, Identifier]]: + indices = self.get_missed_parentheses(tokens=tokens) + for index in reversed(indices): + tokens.insert(index + 1, Identifier(token_type=GroupType.R_PAREN)) + tokens.insert(index, Identifier(token_type=GroupType.L_PAREN)) + return tokens + + def __get_identifier( + self, condition: str + ) -> Union[tuple[Identifier, str], tuple[list, str], tuple[Selection, str]]: + condition = condition.strip("\n").strip(" ").strip("\n") + if condition.startswith(GroupType.L_PAREN): + return Identifier(token_type=GroupType.L_PAREN), condition[1:] + if condition.startswith(GroupType.R_PAREN): + return Identifier(token_type=GroupType.R_PAREN), condition[1:] + if logical_operator_search := re.match(self.logical_operator_pattern, condition, re.IGNORECASE): + logical_operator = logical_operator_search.group("logical_operator") + pos = logical_operator_search.end() + return Identifier(token_type=logical_operator.lower()), condition[pos:] + if condition.startswith(self.one_of_selection): + condition = re.sub(self.one_of_selection, "", condition, 1) + return self.__get_group(condition=condition, operator=Identifier(token_type=LogicalOperatorType.OR)) + if condition.startswith(self.all_of_selection): + condition = re.sub(self.all_of_selection, "", condition, 1) + return self.__get_group(condition=condition, operator=Identifier(token_type=LogicalOperatorType.AND)) + return self.__get_selection(condition) + + def tokenize(self, condition: str, detection: dict) -> list: + self.detection = detection + tokenized = [] + while condition: + identifier, condition = self.__get_identifier(condition=condition) + if isinstance(identifier, list): + tokenized.extend(identifier) + else: + tokenized.append(identifier) + return self.__add_parentheses_after_and_not(tokens=tokenized) diff --git a/uncoder-core/app/translator/platforms/snowflake/__init__.py b/uncoder-core/app/translator/platforms/snowflake/__init__.py new file mode 100644 index 00000000..c0eae983 --- /dev/null +++ b/uncoder-core/app/translator/platforms/snowflake/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.snowflake.renders.snowflake_cti import SnowflakeCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/snowflake/const.py b/uncoder-core/app/translator/platforms/snowflake/const.py new file mode 100644 index 00000000..4f9e390b --- /dev/null +++ b/uncoder-core/app/translator/platforms/snowflake/const.py @@ -0,0 +1,20 @@ +SNOWFLAKE_QUERY_DETAILS = { + "platform_id": "snowflake", + "name": "Snowflake Query", + "group_name": "Snowflake", + "group_id": "snowflake-pack", + "platform_name": "Query (SQL)", +} + +DEFAULT_SNOWFLAKE_CTI_MAPPING = { + "SourceIP": "source.ip", + "DestinationIP": "destination.ip", + "Domain": "destination.domain", + "URL": "url.original", + "HashMd5": "file.hash.md5", + "HashSha1": "file.hash.sha1", + "HashSha256": "file.hash.sha256", + "HashSha512": "file.hash.sha512", + "Files": "file.path", + "Emails": "user.name", +} diff --git a/siem-converter/app/converter/platforms/roota/__init__.py b/uncoder-core/app/translator/platforms/snowflake/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/roota/__init__.py rename to uncoder-core/app/translator/platforms/snowflake/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/snowflake/renders/snowflake_cti.py b/uncoder-core/app/translator/platforms/snowflake/renders/snowflake_cti.py new file mode 100644 index 00000000..125a7c8a --- /dev/null +++ b/uncoder-core/app/translator/platforms/snowflake/renders/snowflake_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.snowflake.const import DEFAULT_SNOWFLAKE_CTI_MAPPING, SNOWFLAKE_QUERY_DETAILS + + +@render_cti_manager.register +class SnowflakeCTI(RenderCTI): + details: PlatformDetails = PlatformDetails(**SNOWFLAKE_QUERY_DETAILS) + + field_value_template: str = "{key} = '{value}'" + or_operator: str = " OR " + group_or_operator: str = " OR " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "SELECT * FROM table WHERE {result}\n" + final_result_for_one: str = "SELECT * FROM table WHERE {result}\n" + default_mapping = DEFAULT_SNOWFLAKE_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/splunk/__init__.py b/uncoder-core/app/translator/platforms/splunk/__init__.py new file mode 100644 index 00000000..21b1049b --- /dev/null +++ b/uncoder-core/app/translator/platforms/splunk/__init__.py @@ -0,0 +1,5 @@ +from app.translator.platforms.splunk.parsers.splunk import SplunkQueryParser # noqa: F401 +from app.translator.platforms.splunk.parsers.splunk_alert import SplunkAlertParser, SplunkAlertYMLParser # noqa: F401 +from app.translator.platforms.splunk.renders.splunk import SplunkQueryRender # noqa: F401 +from app.translator.platforms.splunk.renders.splunk_alert import SplunkAlertRender # noqa: F401 +from app.translator.platforms.splunk.renders.splunk_cti import SplunkCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/splunk/const.py b/uncoder-core/app/translator/platforms/splunk/const.py new file mode 100644 index 00000000..a81a2bb8 --- /dev/null +++ b/uncoder-core/app/translator/platforms/splunk/const.py @@ -0,0 +1,69 @@ +from app.translator.core.models.platform_details import PlatformDetails + +DEFAULT_SPLUNK_ALERT = """ +[] +alert.severity = +description = +cron_schedule = 0 * * * * +disabled = 1 +is_scheduled = 1 +is_visible = 1 +dispatch.earliest_time = -60m@m +dispatch.latest_time = now +search = +alert.suppress = 0 +alert.track = 1 +actions = risk,notable +action.risk = 1 +action.risk.param._risk_object_type = user +action.risk.param._risk_score = 75 +action.correlationsearch = 0 +action.correlationsearch.enabled = 1 +action.notable.param.rule_title = +action.notable.param.rule_description = +action.correlationsearch.label = + +""" + +PLATFORM_DETAILS = {"group_id": "splunk-pack", "group_name": "Splunk"} + +SPLUNK_QUERY_DETAILS = { + "platform_id": "splunk-spl-query", + "name": "Splunk Query", + "platform_name": "Query (SPL)", + **PLATFORM_DETAILS, +} + +SPLUNK_ALERT_DETAILS = { + "platform_id": "splunk-spl-rule", + "name": "Splunk Alert", + "platform_name": "Alert (SPL)", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +SPLUNK_ALERT_YML_DETAILS = { + "platform_id": "splunk-spl-rule-yml", + "name": "Splunk Alert YML", + "platform_name": "Alert (SPL) YML", + "first_choice": 0, + **PLATFORM_DETAILS, +} + +DEFAULT_SPLUNK_CTI_MAPPING = { + "DestinationIP": "dest_ip", + "SourceIP": "src_ip", + "HashSha512": "file_hash", + "HashSha256": "file_hash", + "HashMd5": "file_hash", + "Emails": "All_Email.src_user", + "Domain": "dest_host", + "HashSha1": "file_hash", + "Files": "file_path", + "URL": "url", +} + + +splunk_query_details = PlatformDetails(**SPLUNK_QUERY_DETAILS) +splunk_alert_details = PlatformDetails(**SPLUNK_ALERT_DETAILS) +splunk_alert_yml_details = PlatformDetails(**SPLUNK_ALERT_YML_DETAILS) diff --git a/uncoder-core/app/translator/platforms/splunk/functions/__init__.py b/uncoder-core/app/translator/platforms/splunk/functions/__init__.py new file mode 100644 index 00000000..ba3702fe --- /dev/null +++ b/uncoder-core/app/translator/platforms/splunk/functions/__init__.py @@ -0,0 +1,8 @@ +from app.translator.platforms.base.spl.functions import SplFunctions + + +class SplunkFunctions(SplFunctions): + pass + + +splunk_functions = SplunkFunctions() diff --git a/uncoder-core/app/translator/platforms/splunk/mapping.py b/uncoder-core/app/translator/platforms/splunk/mapping.py new file mode 100644 index 00000000..b5750532 --- /dev/null +++ b/uncoder-core/app/translator/platforms/splunk/mapping.py @@ -0,0 +1,55 @@ +from typing import Optional + +from app.translator.core.mapping import BasePlatformMappings, LogSourceSignature +from app.translator.platforms.splunk.const import splunk_alert_details, splunk_query_details + + +class SplunkLogSourceSignature(LogSourceSignature): + def __init__( + self, + sources: Optional[list[str]], + source_types: Optional[list[str]], + source_categories: Optional[list[str]], + indices: Optional[list[str]], + default_source: Optional[dict] = None, + ): + self.sources = set(sources or []) + self.source_types = set(source_types or []) + self.source_categories = set(source_categories or []) + self.indices = set(indices or []) + self._default_source = default_source or {} + + def is_suitable( + self, + source: Optional[list[str]] = None, + sourcetype: Optional[list[str]] = None, + sourcecategory: Optional[list[str]] = None, + index: Optional[list[str]] = None, + ) -> bool: + conditions = [ + set(source).issubset(self.sources) if source else None, + set(sourcetype).issubset(self.source_types) if sourcetype else None, + set(sourcecategory).issubset(self.source_categories) if sourcecategory else None, + set(index).issubset(self.indices) if index else None, + ] + return self._check_conditions(conditions) + + def __str__(self) -> str: + return " AND ".join((f"{key}={value}" for key, value in self._default_source.items() if value)) + + +class SplunkMappings(BasePlatformMappings): + def prepare_log_source_signature(self, mapping: dict) -> SplunkLogSourceSignature: + log_source = mapping.get("log_source", {}) + default_log_source = mapping["default_log_source"] + return SplunkLogSourceSignature( + sources=log_source.get("source"), + source_types=log_source.get("sourcetype"), + source_categories=log_source.get("sourcecategory"), + indices=log_source.get("index"), + default_source=default_log_source, + ) + + +splunk_query_mappings = SplunkMappings(platform_dir="splunk", platform_details=splunk_query_details) +splunk_alert_mappings = SplunkMappings(platform_dir="splunk", platform_details=splunk_alert_details) diff --git a/siem-converter/app/converter/platforms/roota/parsers/__init__.py b/uncoder-core/app/translator/platforms/splunk/parsers/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/roota/parsers/__init__.py rename to uncoder-core/app/translator/platforms/splunk/parsers/__init__.py diff --git a/uncoder-core/app/translator/platforms/splunk/parsers/splunk.py b/uncoder-core/app/translator/platforms/splunk/parsers/splunk.py new file mode 100644 index 00000000..2370717a --- /dev/null +++ b/uncoder-core/app/translator/platforms/splunk/parsers/splunk.py @@ -0,0 +1,34 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import parser_manager +from app.translator.platforms.base.spl.parsers.spl import SplQueryParser +from app.translator.platforms.splunk.const import splunk_query_details +from app.translator.platforms.splunk.functions import SplunkFunctions, splunk_functions +from app.translator.platforms.splunk.mapping import SplunkMappings, splunk_query_mappings + + +@parser_manager.register_supported_by_roota +class SplunkQueryParser(SplQueryParser): + details: PlatformDetails = splunk_query_details + mappings: SplunkMappings = splunk_query_mappings + platform_functions: SplunkFunctions = splunk_functions + + log_source_pattern = r"^___source_type___\s*=\s*(?:\"(?P[%a-zA-Z_*:0-9\-/]+)\"|(?P[%a-zA-Z_*:0-9\-/]+))(?:\s+(?:and|or)\s+|\s+)?" # noqa: E501 + log_source_key_types = ("index", "source", "sourcetype", "sourcecategory") diff --git a/uncoder-core/app/translator/platforms/splunk/parsers/splunk_alert.py b/uncoder-core/app/translator/platforms/splunk/parsers/splunk_alert.py new file mode 100644 index 00000000..602191b3 --- /dev/null +++ b/uncoder-core/app/translator/platforms/splunk/parsers/splunk_alert.py @@ -0,0 +1,118 @@ +""" +Uncoder IO Commercial Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +This file is part of the Uncoder IO Commercial Edition ("CE") and is +licensed under the Uncoder IO Non-Commercial License (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://github.com/UncoderIO/UncoderIO/blob/main/LICENSE + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +----------------------------------------------------------------- +""" + +import re + +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.mitre import MitreConfig +from app.translator.core.mixins.rule import YamlRuleMixin +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, MitreInfoContainer, RawQueryContainer +from app.translator.managers import parser_manager +from app.translator.platforms.splunk.const import splunk_alert_details, splunk_alert_yml_details +from app.translator.platforms.splunk.mapping import SplunkMappings, splunk_alert_mappings +from app.translator.platforms.splunk.parsers.splunk import SplunkQueryParser + + +@parser_manager.register +class SplunkAlertParser(SplunkQueryParser): + details: PlatformDetails = splunk_alert_details + mappings: SplunkMappings = splunk_alert_mappings + mitre_config: MitreConfig = MitreConfig() + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + rule_id: str = "" + rule_name: str = "" + severity: str = "" + mitre_attack_container: MitreInfoContainer = None + if severity_match := re.search(r"alert\.severity\s*=\s*(\d+)", text): + level_map = { + "1": SeverityType.low, + "2": SeverityType.medium, + "3": SeverityType.high, + "4": SeverityType.critical, + } + severity = level_map.get(str(severity_match.group(1)), "low") + + if mitre_attack_match := re.search(r'"mitre_attack":\s*\["([^"]+)"\]', text): + raw_mitre_attack = [attack.strip().strip("'") for attack in mitre_attack_match.group(1).split(",")] + mitre_attack_container = self.mitre_config.get_mitre_info( + tactics=[i.lower() for i in raw_mitre_attack if not i[-1].isdigit()], + techniques=[i.lower() for i in raw_mitre_attack if i[-1].isdigit()], + ) + + if rule_id_match := re.search(r"Rule ID:\s*([\w-]+)", text): + rule_id = rule_id_match.group(1) + if rule_name_match := re.search(r"action\.notable\.param\.rule_title\s*=\s*(.*)", text): + rule_name = rule_name_match.group(1) + + query = re.search(r"search\s*=\s*(?P.+)", text).group("query") + description = re.search(r"description\s*=\s*(?P.+)", text).group("description") + return RawQueryContainer( + query=query, + language=language, + meta_info=MetaInfoContainer( + id_=rule_id, + title=rule_name, + description=description, + severity=severity, + mitre_attack=mitre_attack_container, + ), + ) + + +@parser_manager.register +class SplunkAlertYMLParser(SplunkQueryParser, YamlRuleMixin): + details: PlatformDetails = splunk_alert_yml_details + mappings: SplunkMappings = splunk_alert_mappings + mitre_config: MitreConfig = MitreConfig() + + def parse_raw_query(self, text: str, language: str) -> RawQueryContainer: + rule = self.load_rule(text) + mitre_attack_container = self.mitre_config.get_mitre_info( + techniques=rule.get("tags", {}).get("mitre_attack_id", []) + ) + description = rule.get("description", "") + if rule.get("how_to_implement", ""): + description = f'{description} {rule.get("how_to_implement", "")}' + tags = rule.get("tags", {}).get("analytic_story", []) + if rule.get("type"): + tags.append(rule.get("type")) + false_positives = None + if rule.get("known_false_positives"): + false_positives = ( + rule["known_false_positives"] + if isinstance(rule["known_false_positives"], list) + else [rule["known_false_positives"]] + ) + return RawQueryContainer( + query=rule.get("search"), + language=language, + meta_info=MetaInfoContainer( + id_=rule.get("id"), + title=rule.get("name"), + date=rule.get("date"), + author=rule.get("author").split(", "), + status=rule.get("status"), + description=description, + false_positives=false_positives, + references=rule.get("references"), + mitre_attack=mitre_attack_container, + tags=tags, + ), + ) diff --git a/siem-converter/app/converter/platforms/rsa_netwitness/__init__.py b/uncoder-core/app/translator/platforms/splunk/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/rsa_netwitness/__init__.py rename to uncoder-core/app/translator/platforms/splunk/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/splunk/renders/splunk.py b/uncoder-core/app/translator/platforms/splunk/renders/splunk.py new file mode 100644 index 00000000..7a50d3d1 --- /dev/null +++ b/uncoder-core/app/translator/platforms/splunk/renders/splunk.py @@ -0,0 +1,44 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.managers import render_manager +from app.translator.platforms.base.spl.renders.spl import SplFieldValueRender, SplQueryRender +from app.translator.platforms.splunk.const import splunk_query_details +from app.translator.platforms.splunk.functions import SplunkFunctions, splunk_functions +from app.translator.platforms.splunk.mapping import SplunkMappings, splunk_query_mappings + + +class SplunkFieldValueRender(SplFieldValueRender): + details: PlatformDetails = splunk_query_details + + +@render_manager.register +class SplunkQueryRender(SplQueryRender): + details: PlatformDetails = splunk_query_details + mappings: SplunkMappings = splunk_query_mappings + platform_functions: SplunkFunctions = None + + or_token = "OR" + + field_value_render = SplunkFieldValueRender(or_token=or_token) + + def init_platform_functions(self) -> None: + self.platform_functions = splunk_functions + self.platform_functions.platform_query_render = self diff --git a/uncoder-core/app/translator/platforms/splunk/renders/splunk_alert.py b/uncoder-core/app/translator/platforms/splunk/renders/splunk_alert.py new file mode 100644 index 00000000..d1b16877 --- /dev/null +++ b/uncoder-core/app/translator/platforms/splunk/renders/splunk_alert.py @@ -0,0 +1,87 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from typing import Optional + +from app.translator.core.custom_types.meta_info import SeverityType +from app.translator.core.mapping import SourceMapping +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.models.query_container import MetaInfoContainer, MitreInfoContainer +from app.translator.managers import render_manager +from app.translator.platforms.splunk.const import DEFAULT_SPLUNK_ALERT, splunk_alert_details +from app.translator.platforms.splunk.mapping import SplunkMappings, splunk_alert_mappings +from app.translator.platforms.splunk.renders.splunk import SplunkFieldValueRender, SplunkQueryRender +from app.translator.tools.utils import get_rule_description_str + +_AUTOGENERATED_TEMPLATE = "Autogenerated Splunk Alert" +_SEVERITIES_MAP = {SeverityType.critical: "4", SeverityType.high: "3", SeverityType.medium: "2", SeverityType.low: "1"} + + +class SplunkAlertFieldValueRender(SplunkFieldValueRender): + details: PlatformDetails = splunk_alert_details + + +@render_manager.register +class SplunkAlertRender(SplunkQueryRender): + details: PlatformDetails = splunk_alert_details + mappings: SplunkMappings = splunk_alert_mappings + + or_token = "OR" + field_value_render = SplunkAlertFieldValueRender(or_token=or_token) + + @staticmethod + def __create_mitre_threat(mitre_attack: MitreInfoContainer) -> dict: + mitre_attack_render = {"mitre_attack": []} + + for technique in mitre_attack.techniques: + mitre_attack_render["mitre_attack"].append(technique.technique_id) + for tactic in mitre_attack.tactics: + mitre_attack_render["mitre_attack"].append(tactic.name) + mitre_attack_render["mitre_attack"].sort() + return mitre_attack_render + + def finalize_query( + self, + prefix: str, + query: str, + functions: str, + meta_info: Optional[MetaInfoContainer] = None, + source_mapping: Optional[SourceMapping] = None, # noqa: ARG002 + not_supported_functions: Optional[list] = None, + unmapped_fields: Optional[list[str]] = None, + *args, # noqa: ARG002 + **kwargs, # noqa: ARG002 + ) -> str: + query = super().finalize_query(prefix=prefix, query=query, functions=functions) + rule = DEFAULT_SPLUNK_ALERT.replace("", query) + rule = rule.replace("", meta_info.title or _AUTOGENERATED_TEMPLATE) + rule = rule.replace("", _SEVERITIES_MAP.get(meta_info.severity, "1")) + rule_description = get_rule_description_str( + author=meta_info.author, + description=meta_info.description or _AUTOGENERATED_TEMPLATE, + license_=meta_info.license, + rule_id=meta_info.id, + ) + rule = rule.replace("", rule_description) + mitre_techniques = self.__create_mitre_threat(mitre_attack=meta_info.mitre_attack) + if mitre_techniques: + mitre_str = f"action.correlationsearch.annotations = {mitre_techniques})" + rule = rule.replace("", mitre_str) + rule = self.wrap_with_unmapped_fields(rule, unmapped_fields) + return self.wrap_with_not_supported_functions(rule, not_supported_functions) diff --git a/uncoder-core/app/translator/platforms/splunk/renders/splunk_cti.py b/uncoder-core/app/translator/platforms/splunk/renders/splunk_cti.py new file mode 100644 index 00000000..60d26cea --- /dev/null +++ b/uncoder-core/app/translator/platforms/splunk/renders/splunk_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.splunk.const import DEFAULT_SPLUNK_CTI_MAPPING, splunk_query_details + + +@render_cti_manager.register +class SplunkCTI(RenderCTI): + details: PlatformDetails = splunk_query_details + + field_value_template: str = '{key}="{value}"' + or_operator: str = " OR " + group_or_operator: str = " OR " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_SPLUNK_CTI_MAPPING diff --git a/uncoder-core/app/translator/platforms/sumo_logic/__init__.py b/uncoder-core/app/translator/platforms/sumo_logic/__init__.py new file mode 100644 index 00000000..5d25614c --- /dev/null +++ b/uncoder-core/app/translator/platforms/sumo_logic/__init__.py @@ -0,0 +1 @@ +from app.translator.platforms.sumo_logic.renders.sumologic_cti import SumologicCTI # noqa: F401 diff --git a/uncoder-core/app/translator/platforms/sumo_logic/const.py b/uncoder-core/app/translator/platforms/sumo_logic/const.py new file mode 100644 index 00000000..2fa1019e --- /dev/null +++ b/uncoder-core/app/translator/platforms/sumo_logic/const.py @@ -0,0 +1,21 @@ +SUMO_LOGIC_QUERY_DETAILS = { + "platform_id": "sumologic", + "name": "Sumo Logic Query", + "group_name": "Sumo Logic", + "platform_name": "Query", + "first_choice": 0, + "group_id": "sumologic", +} + +DEFAULT_SUMOLOGIC_CTI_MAPPING = { + "SourceIP": "src_ip", + "DestinationIP": "dst_ip", + "Domain": "host", + "URL": "url", + "HashMd5": "fileHash", + "HashSha1": "fileHash", + "HashSha256": "fileHash", + "HashSha512": "fileHash", + "Emails": "flattened_destinations", + "Files": "files", +} diff --git a/siem-converter/app/converter/platforms/rsa_netwitness/mappings/__init__.py b/uncoder-core/app/translator/platforms/sumo_logic/renders/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/rsa_netwitness/mappings/__init__.py rename to uncoder-core/app/translator/platforms/sumo_logic/renders/__init__.py diff --git a/uncoder-core/app/translator/platforms/sumo_logic/renders/sumologic_cti.py b/uncoder-core/app/translator/platforms/sumo_logic/renders/sumologic_cti.py new file mode 100644 index 00000000..f268265e --- /dev/null +++ b/uncoder-core/app/translator/platforms/sumo_logic/renders/sumologic_cti.py @@ -0,0 +1,37 @@ +""" +Uncoder IO Community Edition License +----------------------------------------------------------------- +Copyright (c) 2024 SOC Prime, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +----------------------------------------------------------------- +""" + +from app.translator.core.models.platform_details import PlatformDetails +from app.translator.core.render_cti import RenderCTI +from app.translator.managers import render_cti_manager +from app.translator.platforms.sumo_logic.const import SUMO_LOGIC_QUERY_DETAILS, DEFAULT_SUMOLOGIC_CTI_MAPPING + + +@render_cti_manager.register +class SumologicCTI(RenderCTI): + details: PlatformDetails = PlatformDetails(**SUMO_LOGIC_QUERY_DETAILS) + + field_value_template: str = '{key} = "{value}"' + or_operator: str = " OR " + group_or_operator: str = " OR " + or_group: str = "({or_group})" + result_join: str = "" + final_result_for_many: str = "({result})\n" + final_result_for_one: str = "{result}\n" + default_mapping = DEFAULT_SUMOLOGIC_CTI_MAPPING diff --git a/siem-converter/app/converter/platforms/rsa_netwitness/renders/__init__.py b/uncoder-core/app/translator/tools/__init__.py similarity index 100% rename from siem-converter/app/converter/platforms/rsa_netwitness/renders/__init__.py rename to uncoder-core/app/translator/tools/__init__.py diff --git a/uncoder-core/app/translator/tools/const.py b/uncoder-core/app/translator/tools/const.py new file mode 100644 index 00000000..caee962e --- /dev/null +++ b/uncoder-core/app/translator/tools/const.py @@ -0,0 +1,18 @@ +from typing import Literal + +IP_IOC_REGEXP_PATTERN = r"(?:^|[ \/\[(\"',;>|])((?:25[0-5]|2[0-4]\d|[0-1]?\d{1,2})(?:\.(?:25[0-5]|2[0-4]\d|[0-1]?\d{1,2})){3})(?=[\s)\]\"',;:\/?\n<|]|$)" # noqa: E501 +DOMAIN_IOC_REGEXP_PATTERN = r"(?:^|[\s\/\[\]@(\"',;{>|])(?:(?:http[s]?|ftp):\/\/?)?([^:\\\/\s({\[\]@\"'`,]+\.[a-zA-Z]+)(?:(?:(?:[/|:]\w+)*\/)(?:[\w\-.]+[^#?\s]+)?(?:[\w/\-&?=%.#]+(?:\(\))?)?)?(?=[\s)\]\"',;<|]|$)" # noqa: E501 +URL_IOC_REGEXP_PATTERN = r"(?:^|[\s\/\[\]@(\"',;{>|])((?:(?:http[s]?|ftp):\/\/?)+(?:[^:\\\/\s({\[\]@\"'`,]+\.[a-zA-Z0-9]+)(?:(?:(?:[/|:]\w+)*\/)(?:[\w\-.]+[^#?\s<']+)?(?:[\w/\-&?=%.#]+(?:\(\))?)?)?)(?=[\s)\]\"',;<|]|$)" # noqa: E501 + +IOCType = Literal["ip", "domain", "url", "hash"] +HashType = Literal["md5", "sha1", "sha256", "sha512"] +IocParsingRule = Literal["replace_dots", "remove_private_and_reserved_ips", "replace_hxxp"] + +HASH_MAP = {"md5": "HashMd5", "sha1": "HashSha1", "sha256": "HashSha256", "sha512": "HashSha512"} + +hash_regexes = { + "md5": r"(?:^|[\s\/\[(\"',;{>|])([A-Fa-f0-9]{32})(?=[\s)\]\"',;\n<|]|$)", + "sha1": r"(?:^|[\s\/\[(\"',;{>|])([A-Fa-f0-9]{40})(?=[\s)\]\"',;\n<|]|$)", + "sha256": r"(?:^|[\s\/\[(\"',;{>|])([A-Fa-f0-9]{64})(?=[\s)\]\"',;\n<|]|$)", + "sha512": r"(?:^|[\s\/\[(\"',;{>|])([A-Fa-f0-9]{128})(?=[\s)\]\"',;\n<|]|$)", +} diff --git a/uncoder-core/app/translator/tools/custom_enum.py b/uncoder-core/app/translator/tools/custom_enum.py new file mode 100644 index 00000000..6d4b1b7e --- /dev/null +++ b/uncoder-core/app/translator/tools/custom_enum.py @@ -0,0 +1,29 @@ +class CustomEnumMeta(type): + def __new__(cls, name: str, bases: tuple, attrs: dict): + _attrs, _enum = {}, {} + for base in bases: + if isinstance(base, cls): + _enum.update(getattr(base, "_enum", {})) + + for key, value in attrs.items(): + if key[:2] == key[-2:] == "__" or hasattr(value, "__get__"): + _attrs[key] = value + else: + _enum[key] = value + + _attrs["_enum"] = _enum + + return super().__new__(cls, name, bases, _attrs) + + def __contains__(cls, name: str): + return name in cls._enum.values() + + def __getattr__(cls, name: str): + try: + return cls._enum[name] + except KeyError as e: + raise AttributeError(name) from e + + +class CustomEnum(metaclass=CustomEnumMeta): + pass diff --git a/uncoder-core/app/translator/tools/decorators.py b/uncoder-core/app/translator/tools/decorators.py new file mode 100644 index 00000000..59e8f94c --- /dev/null +++ b/uncoder-core/app/translator/tools/decorators.py @@ -0,0 +1,28 @@ +from collections.abc import Callable +from typing import Any + +from app.translator.core.exceptions.core import BasePlatformException +from app.translator.core.exceptions.iocs import BaseIOCsException +from app.translator.core.exceptions.parser import BaseParserException +from app.translator.core.exceptions.render import BaseRenderException + + +def handle_translation_exceptions(func: Callable[..., ...]) -> Callable[..., tuple[bool, Any]]: + def exception_handler(*args, **kwargs) -> tuple[bool, Any]: + try: + result = func(*args, **kwargs) + except (BaseParserException, BasePlatformException, BaseRenderException, BaseIOCsException) as err: + print(f"Unexpected error. {err!s}") + return False, str(err) + except Exception as err: + print(f"Unexpected error. {err!s}") + return False, "Unexpected error. To resolve it, please, contact us." + else: + if result: + print("Translated successfully.") + return True, result + + print("Unexpected error.") + return False, "Unexpected error. To resolve it, please, contact us." + + return exception_handler diff --git a/uncoder-core/app/translator/tools/singleton_meta.py b/uncoder-core/app/translator/tools/singleton_meta.py new file mode 100644 index 00000000..83dcd641 --- /dev/null +++ b/uncoder-core/app/translator/tools/singleton_meta.py @@ -0,0 +1,11 @@ +from typing import ClassVar + + +class SingletonMeta(type): + _instances: ClassVar[dict] = {} + + def __call__(cls, *args, **kwargs): + if cls not in cls._instances: + instance = super().__call__(*args, **kwargs) + cls._instances[cls] = instance + return cls._instances[cls] diff --git a/uncoder-core/app/translator/tools/utils.py b/uncoder-core/app/translator/tools/utils.py new file mode 100644 index 00000000..1c69efc7 --- /dev/null +++ b/uncoder-core/app/translator/tools/utils.py @@ -0,0 +1,102 @@ +import importlib.util +import re +from contextlib import suppress +from typing import Optional + + +def execute_module(path: str) -> None: + with suppress(FileNotFoundError): + spec = importlib.util.spec_from_file_location("__init__", path) + init_module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(init_module) + + +def execute_module(path: str) -> None: + with suppress(FileNotFoundError): + spec = importlib.util.spec_from_file_location("__init__", path) + init_module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(init_module) + + +def get_match_group(match: re.Match, group_name: str) -> Optional[str]: + try: + return match.group(group_name) + except IndexError: + return + + +def concatenate_str(str1: str, str2: str) -> str: + return str1 + " " + str2 if str1 else str2 + + +def get_mitre_attack_str(mitre_attack: list) -> str: + return f"MITRE ATT&CK: {', '.join(mitre_attack).upper()}." + + +def get_author_str(author: list[str]) -> str: + return f"Author: {', '.join(author)}." + + +def get_license_str(license_: str) -> str: + license_str = f"License: {license_}" + if not license_str.endswith("."): + license_str += "." + return license_str + + +def get_description_str(description: str) -> str: + if description != "" and not description.endswith("."): + description += "." + return description + + +def get_rule_id_str(rule_id: str) -> str: + return f"Rule ID: {rule_id}." + + +def get_references_str(references: list[str]) -> str: + return f"References: {', '.join(references)}." + + +def get_rule_description_str( + description: str, + author: Optional[list[str]] = None, + rule_id: Optional[str] = None, + license_: Optional[str] = None, + mitre_attack: Optional[list[str]] = None, + references: Optional[list[str]] = None, +) -> str: + rule_description = get_description_str(description) + if author: + rule_description = concatenate_str(rule_description, get_author_str(author)) + if rule_id: + rule_description = concatenate_str(rule_description, get_rule_id_str(rule_id)) + if license_: + rule_description = concatenate_str(rule_description, get_license_str(license_)) + if mitre_attack: + rule_description = concatenate_str(rule_description, get_mitre_attack_str(mitre_attack)) + if references: + rule_description = concatenate_str(rule_description, get_references_str(references)) + return rule_description + + +def parse_rule_description_str(description: str) -> dict: + parsed = {} + keys_map = { + "references": "Reference", + "mitre_attack": "MITRE ATT&CK", + "license": "License", + "rule_id": "Rule ID", + "author": "Author", + } + pattern = r"___name___:\s*(?P.+)\." + for key, name in keys_map.items(): + if search := re.search(pattern.replace("___name___", name), description): + if key in ("author", "references"): + parsed[key] = [value.strip() for value in search.group("value").split(",")] + else: + parsed[key] = search.group("value") + description = description[: search.start()] + + parsed["description"] = description.strip() + return parsed diff --git a/uncoder-core/app/translator/translator.py b/uncoder-core/app/translator/translator.py new file mode 100644 index 00000000..746ad3bb --- /dev/null +++ b/uncoder-core/app/translator/translator.py @@ -0,0 +1,145 @@ +import logging +from collections import Counter +from typing import Optional, Union + +from app.translator.core.exceptions.core import UnsupportedPlatform +from app.translator.core.models.query_container import RawQueryContainer, TokenizedQueryContainer +from app.translator.core.parser import PlatformQueryParser, QueryParser +from app.translator.core.render import QueryRender +from app.translator.managers import ParserManager, RenderManager, parser_manager, render_manager +from app.translator.platforms.elasticsearch.const import ELASTIC_QUERY_TYPES +from app.translator.platforms.microsoft.const import MICROSOFT_SENTINEL_QUERY_TYPES +from app.translator.platforms.roota.parsers.roota import RootAParser +from app.translator.platforms.sigma.mapping import sigma_rule_mappings +from app.translator.tools.decorators import handle_translation_exceptions + + +class Translator: + render_manager: RenderManager = render_manager + parser_manager: ParserManager = parser_manager + + def __init__(self): + self.logger = logging.getLogger("translator") + + def __get_parser(self, source: str) -> QueryParser: + parser = self.parser_manager.get(source) + if not parser: + raise UnsupportedPlatform(platform=source, is_parser=True) + + return parser + + def __get_render(self, target: str) -> QueryRender: + if not (render := self.render_manager.get(target)): + raise UnsupportedPlatform(platform=target) + + return render + + @staticmethod + def __is_one_vendor_translation(source: str, target: str) -> bool: + vendors_query_types = [ELASTIC_QUERY_TYPES, MICROSOFT_SENTINEL_QUERY_TYPES] + for vendor_query_types in vendors_query_types: + if source in vendor_query_types and target in vendor_query_types: + return True + + return False + + def parse_raw_query( + self, text: str, source: str + ) -> tuple[Union[PlatformQueryParser, RootAParser], RawQueryContainer]: + parser = self.__get_parser(source) + text = parser.remove_comments(text) + return parser, parser.parse_raw_query(text, language=source) + + def parse_meta_info(self, text: str, source: str) -> Union[dict, RawQueryContainer]: + parser, raw_query_container = self.parse_raw_query(text=text, source=source) + source_mappings = parser.get_source_mapping_ids_by_logsources(raw_query_container.query) + log_sources = {"product": Counter(), "service": Counter(), "category": Counter()} + sigma_source_mappings = sigma_rule_mappings.get_source_mappings_by_ids( + [source_mapping.source_id for source_mapping in source_mappings], return_default=False + ) + for sigma_source_mapping in sigma_source_mappings: + if product := sigma_source_mapping.log_source_signature.log_sources.get("product"): + log_sources["product"][product] += 1 + if service := sigma_source_mapping.log_source_signature.log_sources.get("service"): + log_sources["service"][service] += 1 + if category := sigma_source_mapping.log_source_signature.log_sources.get("category"): + log_sources["category"][category] += 1 + return log_sources, raw_query_container + + @handle_translation_exceptions + def __parse_incoming_data( + self, text: str, source: str, target: Optional[str] = None + ) -> tuple[RawQueryContainer, Optional[TokenizedQueryContainer]]: + parser, raw_query_container = self.parse_raw_query(text=text, source=source) + tokenized_query_container = None + if not (target and self.__is_one_vendor_translation(raw_query_container.language, target)): + tokenized_query_container = parser.parse(raw_query_container) + + return raw_query_container, tokenized_query_container + + @handle_translation_exceptions + def __render_translation( + self, + raw_query_container: RawQueryContainer, + tokenized_query_container: Optional[TokenizedQueryContainer], + target: str, + ) -> str: + render = self.__get_render(target) + return render.generate( + raw_query_container=raw_query_container, tokenized_query_container=tokenized_query_container + ) + + def __translate_one(self, text: str, source: str, target: str) -> (bool, str): + status, parsed_data = self.__parse_incoming_data(text=text, source=source, target=target) + if not status: + return status, parsed_data + + raw_query_container, tokenized_query_container = parsed_data + return self.__render_translation( + raw_query_container=raw_query_container, tokenized_query_container=tokenized_query_container, target=target + ) + + def __translate_all(self, text: str, source: str) -> list[dict]: + status, parsed_data = self.__parse_incoming_data(text=text, source=source) + if not status: + return [{"status": status, "result": parsed_data, "platform_id": source}] + + raw_query_container, tokenized_query_container = parsed_data + result = [] + for target in self.render_manager.all_platforms(): + if target == source: + continue + + if raw_query_container and self.__is_one_vendor_translation(raw_query_container.language, target): + status, data = self.__render_translation( + raw_query_container=raw_query_container, tokenized_query_container=None, target=target + ) + else: + status, data = self.__render_translation( + raw_query_container=raw_query_container, + tokenized_query_container=tokenized_query_container, + target=target, + ) + result.append({"status": status, "result": data, "platform_id": target}) + + return result + + def translate_one(self, text: str, source: str, target: str) -> (bool, str): + if source == target: + return True, text + return self.__translate_one(text=text, source=source, target=target) + + def translate_all(self, text: str, source: str) -> list[dict]: + return self.__translate_all(text=text, source=source) + + def get_all_platforms(self) -> tuple: + return self.get_renders(), self.get_parsers() + + def get_parsers(self) -> list: + return self.parser_manager.get_platforms_details + + def get_renders(self) -> list: + return self.render_manager.get_platforms_details + + +app_translator = Translator() diff --git a/uncoder-core/const.py b/uncoder-core/const.py new file mode 100644 index 00000000..22a7c66a --- /dev/null +++ b/uncoder-core/const.py @@ -0,0 +1,5 @@ +import os + +ROOT_PROJECT_PATH = os.path.abspath(os.path.dirname(__file__)) + +PLATFORMS_PATH = ROOT_PROJECT_PATH + "/app/translator/platforms" diff --git a/uncoder-core/pyproject.toml b/uncoder-core/pyproject.toml new file mode 100644 index 00000000..16b85a26 --- /dev/null +++ b/uncoder-core/pyproject.toml @@ -0,0 +1,29 @@ +[tool.ruff] +line-length = 120 +target-version = "py39" + +select = ["A", "ANN", "ARG", "C4", "E", "F", "ICN", "N", "PL", "Q", "RET", "RSE", "RUF", "SIM", "TID", "TRY", "UP", "W"] +ignore = [ + "ANN002", "ANN003", "ANN101", "ANN102", "ANN204", + "N818", + "PLR0913", + "PLW2901", + "RET502", "RET503", + "TRY002", "TRY003", + "UP007", + "W605" +] + +exclude = [ + "app/translator/platforms/arcsight/", + "app/translator/platforms/datadog/", + "app/translator/platforms/devo/", + "app/translator/platforms/sigma", + "app/translator/platforms/sumo_logic" +] + +[tool.ruff.format] +skip-magic-trailing-comma = true + +[tool.ruff.lint.flake8-builtins] +builtins-ignorelist = ["id", "max", "min", "sum"] diff --git a/uncoder-core/requirements.txt b/uncoder-core/requirements.txt new file mode 100644 index 00000000..90c4901e --- /dev/null +++ b/uncoder-core/requirements.txt @@ -0,0 +1,10 @@ +fastapi>=0.109.1 +uvicorn[standard]>=0.21.0,<0.22.0 +pydantic~=1.10.13 +PyYAML~=6.0.1 +colorama~=0.4.6 +ruff==0.1.13 +ujson==5.9.0 +xmltodict~=0.13.0 +isodate==0.6.1 +toml==0.10.2 diff --git a/siem-converter/server.py b/uncoder-core/server.py similarity index 79% rename from siem-converter/server.py rename to uncoder-core/server.py index 3468db08..469fe0b3 100644 --- a/siem-converter/server.py +++ b/uncoder-core/server.py @@ -4,20 +4,14 @@ from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware - -from app.routers.translate import st_router -from app.routers.ioc_translate import iocs_router from app.routers.assistance import assistance_router, lifespan - +from app.routers.ioc_translate import iocs_router +from app.routers.translate import st_router app = FastAPI(title="Siem Converter API", version="0.1.0", lifespan=lifespan) app.add_middleware( - CORSMiddleware, - allow_origins=["*"], - allow_credentials=True, - allow_methods=["*"], - allow_headers=["*"], + CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"] ) app.include_router(st_router) @@ -26,7 +20,7 @@ if __name__ == "__main__": - host = os.environ.get("HOST", "127.0.0.1") + host = os.environ.get("HOST", "0.0.0.0") port = os.environ.get("PORT", "8000") if not port.isnumeric(): raise Exception("Port should be a number!") diff --git a/uncoder-core/settings.py b/uncoder-core/settings.py new file mode 100644 index 00000000..bf3ff3f3 --- /dev/null +++ b/uncoder-core/settings.py @@ -0,0 +1,3 @@ +import os + +INIT_FUNCTIONS = os.getenv("INIT_FUNCTIONS", "0").lower() in ("true", "1", "t") diff --git a/uncoder-os/.env.common b/uncoder-os/.env.common new file mode 100644 index 00000000..73739ab8 --- /dev/null +++ b/uncoder-os/.env.common @@ -0,0 +1 @@ +MODULE_VERSION=1.2.0 diff --git a/uncoder-os/.gitignore b/uncoder-os/.gitignore index 71036779..03988423 100644 --- a/uncoder-os/.gitignore +++ b/uncoder-os/.gitignore @@ -20,3 +20,4 @@ yarn-error.log* # .env /.idea +.editorconfig diff --git a/uncoder-os/Dockerfile b/uncoder-os/Dockerfile index 9aef6828..87814f92 100644 --- a/uncoder-os/Dockerfile +++ b/uncoder-os/Dockerfile @@ -1,4 +1,4 @@ -FROM node:20.8.1-alpine3.18 +FROM node:21-alpine3.20 WORKDIR /app ENV PATH /app/node_modules/.bin:$PATH COPY package.json tsconfig.json webpack.config.js .env ./ diff --git a/uncoder-os/configs/webpack.common.js b/uncoder-os/configs/webpack.common.js index f1a2073c..3701f825 100644 --- a/uncoder-os/configs/webpack.common.js +++ b/uncoder-os/configs/webpack.common.js @@ -19,7 +19,25 @@ const rules = [ onlyCompileBundledFiles: true, }, }], - exclude: /node_modules/, + exclude: [/node_modules/, /workers/], + }, + { + test: /\.worker\.ts$/, + use: [ + { + loader: 'worker-loader', + options: { + inline: 'no-fallback', + }, + }, + 'babel-loader', + { + loader: 'ts-loader', + options: { + onlyCompileBundledFiles: true, + }, + }], + exclude: [/node_modules/], }, { test: /\.svg$/, diff --git a/uncoder-os/configs/webpack.compile.js b/uncoder-os/configs/webpack.compile.js index ba139f83..774852a9 100644 --- a/uncoder-os/configs/webpack.compile.js +++ b/uncoder-os/configs/webpack.compile.js @@ -12,7 +12,7 @@ const plugins = [ name: 'uncoder', filename: 'remoteEntry.js', exposes: { - './UncoderEditor': './src/pages/UncoderEditor/bootstrap', + './UncoderEditor': './src/pages/UncoderEditor/bootstrap.tsx', }, shared: packageJson.dependencies, }), diff --git a/uncoder-os/package.json b/uncoder-os/package.json index b9fa8cf7..d5b02123 100644 --- a/uncoder-os/package.json +++ b/uncoder-os/package.json @@ -16,53 +16,59 @@ "react": "^18", "react-ace": "^10.1.0", "react-dom": "^18", + "react-error-boundary": "^4.0.11", "react-redux": "^8.1.3", + "react-tooltip": "^5.24.0", "redux": "^4.2.1", "simplebar-react": "^3.2.4" }, "devDependencies": { - "@babel/core": "^7.23.2", - "@babel/eslint-parser": "^7.22.15", - "@babel/preset-env": "^7.23.2", - "@babel/preset-react": "^7.22.15", - "@babel/preset-typescript": "^7.23.2", - "@html-eslint/eslint-plugin": "^0.19.1", - "@html-eslint/parser": "^0.19.1", + "@babel/core": "^7.25.2", + "@babel/eslint-parser": "^7.25.1", + "@babel/preset-env": "^7.25.4", + "@babel/preset-react": "^7.24.7", + "@babel/preset-typescript": "^7.24.7", + "@html-eslint/eslint-plugin": "^0.27.0", + "@html-eslint/parser": "^0.27.0", "@svgr/webpack": "^8.1.0", "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", - "@typescript-eslint/eslint-plugin": "^6.8.0", - "@typescript-eslint/parser": "^6.8.0", - "babel-loader": "^9.1.3", - "copy-webpack-plugin": "^11.0.0", + "@typescript-eslint/eslint-plugin": "^8.7.0", + "@typescript-eslint/parser": "^8.7.0", + "babel-loader": "^9.2.1", + "copy-webpack-plugin": "^12.0.2", "cross-env": "^7.0.3", - "css-loader": "^6.8.1", - "dotenv": "^16.3.1", - "dotenv-webpack": "^8.0.1", - "eslint": "^8.51.0", + "css-loader": "^7.1.2", + "dotenv": "^16.4.5", + "dotenv-webpack": "^8.1.0", + "eslint": "^9.11.1", "eslint-config-airbnb": "^19.0.4", - "eslint-plugin-html": "^7.1.0", - "eslint-plugin-import": "^2.28.1", - "eslint-plugin-jsonc": "^2.9.0", - "eslint-plugin-local-rules": "^2.0.0", - "eslint-plugin-react-hooks": "^4.6.0", + "eslint-plugin-html": "^8.1.2", + "eslint-plugin-import": "^2.30.0", + "eslint-plugin-jsonc": "^2.16.0", + "eslint-plugin-local-rules": "^3.0.2", + "eslint-plugin-react-hooks": "^4.6.2", "file-loader": "^6.2.0", - "html-webpack-plugin": "^5.5.3", - "mini-css-extract-plugin": "^2.7.6", - "node-polyfill-webpack-plugin": "^2.0.1", - "postcss": "^8.4.31", - "postcss-inline-base64": "^7.3.1", - "postcss-loader": "^7.3.3", - "sass": "^1.69.4", - "sass-loader": "^13.3.2", - "source-map-loader": "^4.0.1", - "style-loader": "^3.3.3", - "ts-loader": "^9.5.0", - "typescript": "^5.2.2", - "webpack": "^5.89.0", - "webpack-cli": "^5.1.4", - "webpack-dev-server": "^4.15.1", - "webpack-merge": "^5.10.0" + "html-webpack-plugin": "^5.6.3", + "mini-css-extract-plugin": "^2.9.2", + "node-polyfill-webpack-plugin": "^4.0.0", + "postcss": "8.4.47", + "postcss-inline-base64": "7.3.1", + "postcss-loader": "8.1.1", + "sass": "1.77.6", + "sass-loader": "16.0.2", + "source-map-loader": "^5.0.0", + "style-loader": "^4.0.0", + "ts-loader": "^9.5.1", + "typescript": "^5.6.3", + "webpack": "5.96.1", + "webpack-cli": "5.1.4", + "webpack-dev-server": "^5.1.0", + "webpack-merge": "^6.0.1", + "worker-loader": "^3.0.8" + }, + "resolutions": { + "@types/mime": "3.0.4" } } diff --git a/uncoder-os/src/App.tsx b/uncoder-os/src/App.tsx new file mode 100644 index 00000000..1017aa71 --- /dev/null +++ b/uncoder-os/src/App.tsx @@ -0,0 +1,11 @@ +import React, { FC, Suspense } from 'react'; +import { MainPage } from './pages/MainPage'; +import { Spinner } from './components/Spinner'; + +const App: FC = () => ( + }> + + +); + +export default App; diff --git a/uncoder-os/src/assets/sass/fonts/index.sass b/uncoder-os/src/assets/sass/fonts/index.sass index 588d116b..674f1fd6 100644 --- a/uncoder-os/src/assets/sass/fonts/index.sass +++ b/uncoder-os/src/assets/sass/fonts/index.sass @@ -1,2 +1,2 @@ -@import ./inter -@import ./jetBrains +@use "./inter" as * +@use "./jetBrains" as * diff --git a/uncoder-os/src/assets/sass/helpers/common.sass b/uncoder-os/src/assets/sass/helpers/common.sass index dfe9a28d..80df6c4a 100644 --- a/uncoder-os/src/assets/sass/helpers/common.sass +++ b/uncoder-os/src/assets/sass/helpers/common.sass @@ -1,25 +1,29 @@ +@use "variables" as variables +@use "mixins" as mixins +@use "helpers" as helpers + html, body - background-color: $backgroundDarkBlue + background-color: variables.$backgroundDarkBlue body min-width: 375px - font-family: $inter + font-family: variables.$inter font-size: 14px font-weight: 400 - color: $textDefault - +scrollbars + color: variables.$textDefault + +mixins.scrollbars .inner margin: 0 auto padding: 0 32px width: 100% max-width: 1920px - +md + +helpers.md padding: 0 12px &--xl max-width: 1650px - +xlMin + +helpers.xlMin max-width: 1920px &--md max-width: 1110px @@ -31,7 +35,7 @@ body .simplebar-track .simplebar-scrollbar &::before - background-color: $backgroundSilverBlue + background-color: variables.$backgroundSilverBlue .three-dots max-width: 100% @@ -65,7 +69,7 @@ body left: 0 width: 100% height: 1px - background-color: $backgroundWhite + background-color: variables.$backgroundWhite transform: scaleX(0) transition: transform .15s &:hover @@ -77,15 +81,15 @@ body &--white &, &:hover - color: $textDefault + color: variables.$textDefault &::after - background-color: $backgroundWhite + background-color: variables.$backgroundWhite &--green &, &:hover - color: $textSuccess + color: variables.$textSuccess &::after - background-color: $backgroundSuccess + background-color: variables.$backgroundSuccess &--underline &::after transform: scaleX(1) diff --git a/uncoder-os/src/assets/sass/helpers/index.sass b/uncoder-os/src/assets/sass/helpers/index.sass index ce16580b..b877d0dc 100644 --- a/uncoder-os/src/assets/sass/helpers/index.sass +++ b/uncoder-os/src/assets/sass/helpers/index.sass @@ -1,6 +1,6 @@ -@import mixins -@import reset -@import helpers -@import margin -@import variables -@import common +@use "mixins" as * +@use "reset" as * +@use "helpers" as * +@use "margin" as * +@use "variables" as * +@use "common" as * diff --git a/uncoder-os/src/assets/sass/helpers/mixins.sass b/uncoder-os/src/assets/sass/helpers/mixins.sass index 8507f084..74186d03 100644 --- a/uncoder-os/src/assets/sass/helpers/mixins.sass +++ b/uncoder-os/src/assets/sass/helpers/mixins.sass @@ -1,4 +1,8 @@ -=scrollbars($size: 6px, $foreground-color: $backgroundSilverBlue, $background-color: transparent) +@use 'variables' as variables + +=scrollbars($size: 6px, $foreground-color: variables.$backgroundSilverBlue, $background-color: transparent) + // Standard version (Firefox only for now) + scrollbar-color: $foreground-color $background-color // For Chrome & Safari &::-webkit-scrollbar width: $size @@ -8,6 +12,3 @@ border-radius: 10px &::-webkit-scrollbar-track background: $background-color - - // Standard version (Firefox only for now) - scrollbar-color: $foreground-color $background-color diff --git a/uncoder-os/src/assets/sass/index.sass b/uncoder-os/src/assets/sass/index.sass index f9eeea05..f5e478bf 100644 --- a/uncoder-os/src/assets/sass/index.sass +++ b/uncoder-os/src/assets/sass/index.sass @@ -1,2 +1,2 @@ -@import helpers/index -@import fonts/index +@use "helpers/index" as * +@use "fonts/index" as * diff --git a/uncoder-os/src/assets/svg/GridIcon.svg b/uncoder-os/src/assets/svg/GridIcon.svg index d6c78d63..1195f6c9 100644 --- a/uncoder-os/src/assets/svg/GridIcon.svg +++ b/uncoder-os/src/assets/svg/GridIcon.svg @@ -1 +1,54 @@ - \ No newline at end of file + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/uncoder-os/src/assets/svg/LittleDownIcon.svg b/uncoder-os/src/assets/svg/LittleDownIcon.svg new file mode 100644 index 00000000..eb0241c0 --- /dev/null +++ b/uncoder-os/src/assets/svg/LittleDownIcon.svg @@ -0,0 +1,27 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/uncoder-os/src/components/AdditionalButtons/AdditionalButton/AdditionalButton.sass b/uncoder-os/src/components/AdditionalButtons/AdditionalButton/AdditionalButton.sass index 45f6a66b..2af1584d 100644 --- a/uncoder-os/src/components/AdditionalButtons/AdditionalButton/AdditionalButton.sass +++ b/uncoder-os/src/components/AdditionalButtons/AdditionalButton/AdditionalButton.sass @@ -1,4 +1,4 @@ -@import ../../../assets/sass/helpers/variables +@use "../../../assets/sass/helpers/variables" as * .additional-button text-align: center diff --git a/uncoder-os/src/components/Banner/Banner.sass b/uncoder-os/src/components/Banner/Banner.sass index 3808c40d..60ccb5c6 100644 --- a/uncoder-os/src/components/Banner/Banner.sass +++ b/uncoder-os/src/components/Banner/Banner.sass @@ -1,5 +1,5 @@ -@import ../../assets/sass/helpers/variables -@import ../../assets/sass/helpers/helpers +@use "../../assets/sass/helpers/variables" as * +@use "../../assets/sass/helpers/helpers" as * .banner-grid padding: 16px 24px diff --git a/uncoder-os/src/components/Buttons/Button/Button.sass b/uncoder-os/src/components/Buttons/Button/Button.sass index 46e6ef1d..accf0359 100644 --- a/uncoder-os/src/components/Buttons/Button/Button.sass +++ b/uncoder-os/src/components/Buttons/Button/Button.sass @@ -1,4 +1,4 @@ -@import ../../../assets/sass/helpers/variables +@use "../../../assets/sass/helpers/variables" as * .button padding: 0 10px @@ -14,8 +14,11 @@ font-weight: 400 font-size: 14px color: $textDarkBlue + text-decoration: none transition: background-color .1s cursor: pointer + &:hover + text-decoration: none &:disabled, &.disabled &, @@ -58,3 +61,5 @@ font-size: 16px &.button--icon min-width: 30px + &--inline + display: inline-flex diff --git a/uncoder-os/src/components/Buttons/Button/ButtonLink.tsx b/uncoder-os/src/components/Buttons/Button/ButtonLink.tsx new file mode 100644 index 00000000..4e32db87 --- /dev/null +++ b/uncoder-os/src/components/Buttons/Button/ButtonLink.tsx @@ -0,0 +1,17 @@ +import React, { AnchorHTMLAttributes, FC } from 'react'; + +import './Button.sass'; + +type ButtonLinkPropsType = AnchorHTMLAttributes & { + classes?: string; +}; + +export const ButtonLink: FC = ({ + children, + classes = '', + ...otherProps +}) => ( + + {children} + +); diff --git a/uncoder-os/src/components/Buttons/Button/index.ts b/uncoder-os/src/components/Buttons/Button/index.ts index fe9c53c5..d2896d97 100644 --- a/uncoder-os/src/components/Buttons/Button/index.ts +++ b/uncoder-os/src/components/Buttons/Button/index.ts @@ -1 +1,2 @@ export { Button } from './Button'; +export { ButtonLink } from './ButtonLink'; diff --git a/uncoder-os/src/components/Buttons/TranslateButton/TranslateButton.sass b/uncoder-os/src/components/Buttons/TranslateButton/TranslateButton.sass new file mode 100644 index 00000000..412f12d8 --- /dev/null +++ b/uncoder-os/src/components/Buttons/TranslateButton/TranslateButton.sass @@ -0,0 +1,8 @@ +@use "../../../assets/sass/helpers/variables" as * + +.button--upper + &:disabled, &:disabled:hover + border-color: $borderDisabled + background-color: $iconDisabled + box-shadow: none + z-index: 1 diff --git a/uncoder-os/src/components/Buttons/TranslateButton/TranslateButton.tsx b/uncoder-os/src/components/Buttons/TranslateButton/TranslateButton.tsx index 47d53f79..97c5c1ce 100644 --- a/uncoder-os/src/components/Buttons/TranslateButton/TranslateButton.tsx +++ b/uncoder-os/src/components/Buttons/TranslateButton/TranslateButton.tsx @@ -1,9 +1,26 @@ import React, { FC } from 'react'; import { Button } from '../Button'; +import { Tooltip } from '../../Tooltip'; import { useTranslateButton } from './useTranslateButton'; +import './TranslateButton.sass'; + export const TranslateButton: FC = () => { - const { onClickHandler } = useTranslateButton(); + const { isActive, disabledMessage, onClickHandler } = useTranslateButton(); + + if (!isActive) { + return ( + +