1

Malware Detection and Analysis: Challenges and Research

Opportunities
Zahid Akhtar
Department of Network and Computer Security,
State University of New York Polytechnic Institute, USA.
Email: akhtarz@sunypoly.edu

Malwares are continuously growing in sophistication and hobbyists and cyber-offenders trying to show their ability
arXiv:2101.08429v1 [cs.CR] 21 Jan 2021

numbers. Over the last decade, remarkable progress has been by causing havoc and to steal information potentially for
achieved in anti-malware mechanisms. However, several pressing monetary gains, respectively. They are popularly known as
issues (e.g., unknown malware samples detection) still need to
be addressed adequately. This article first presents a concise hackers, black hats and crackers, and could be external/internal
overview of malware along with anti-malware and then sum- menace, industrial spies or foreign governments. Malwares
marizes various research challenges. This is a theoretical and can be used to change or erase data from victim computers,
perspective article that is hoped to complement earlier articles to collect confidential information, or to hijack systems in
and works. order to attack other devices, send spams, host and share illicit
contents, bring down servers, penetrate networks, and cripple
critical infrastructures.
I. I NTRODUCTION
Consequently, a broad range of tools and schemes have
Use of personal computers and mobile devices coupled been devised to detect and mitigate malware attacks [1]. Anti-
with internet has now become integral part of everyday life. malware systems thwart malwares by determining whether
This ubiquity of high interconnectivity has prompted many given program has malign intent or not [4]. Despite great
serious privacy and security menaces as well as different advancement of malware defense techniques and their inces-
other malicious activities. For instance, 117 million LinkedIn sant evolution, badwares still can bypass the anti-malware
user’s email and password were made publicly available by solutions owing to mainly sophisticated packers and weakest
hackers in 2016. In 2017, Uber revealed that its server was link, i.e., humans. Namely, most anti-malware methods do not
attacked and 57 million drivers and riders data were stolen. exhibit low enough error rates. Additionally, their performance
While, in 2018 almost 50 million Facebook accounts were particularly drops when they face unknown malwares. While,
compromised due to security breach. Similarly, cyberattacks daily 360,000 novel malware samples hit the scene [4]. As
on Norway’s ‘Health South East RHF’ healthcare organization anti-malware becomes more avant-garde so as malwares in
in 2018 exposed health record of more than half of country’s the wild, thereby escalating the arms race between malware
population. Moreover, it is estimated that on an average guardians and writers. The quests for scalable and robust
every 10 second a new malicious code specimen is released automated malware detection frameworks still have to go
to attack mobile devices [1]. A surge of cyberattacks with a long way. This article presents an overview of malwares
increasing number and sophistication can be seen with each and their defenses formulated in recent years, and highlights
passing year, which is impacting governments, enterprises and challenges, open issues and research opportunities for re-
individual alike and causing severe reputation, financial and searchers and engineers. It is a perspective and academic
social damages. For example, malicious cyber activities in article, which is aimed at complementing existing studies and
2016 cost U.S. economy alone up to 109 billion USD [2]. prompt interdisciplinary research.
Different types of cyberattacks are presently being per-
formed by cybercriminals, e.g., man-in-the-middle, malware II. M ALWARE C ATEGORIES
or birthday attack. In particular, malware attacks have ad- Malwares, as depicted in Fig. 1, can be divided into various
vanced as one of the main formidable issues in cybersecurity types depending on how they infect, propagate or exploit
domain as well as primary tool utilized by cybercriminals [3]. the target system as follows [3]. Please note that some of
Malware is a short form of malicious software. In French the malware types/tools/names fall in gray area of features
language, ‘mal’ means ‘bad’. Malware is a catch-all term intended for begin purposes as well, e.g., cookie, Wireshark,
widely employed to denote various kinds of unwanted harmful etc.
software programs with malicious motives [4]. When malware
is executed on a system or computing device it attempts
to breach the system/device’s security policies regarding in- A. Virus
tegrity, confidentiality and availability of data. Other names for A piece of code that duplicates, reproduces or propagates
malware are badware, malicious code, malicious executable itself across files, programs and machines if they are network-
and malicious program. Malwares are developed or used by connected. Viruses cannot execute independently, therefore
 2

they are mainly appended to ‘host’ programs/files (e.g., ex- 4) Cookie

ecutable files, master boot record). When executed by ‘host’ Cookies are plain text files with information of user’s web
can corrupt or destroy files, programs, computer’s functioning browsing sessions. They are stored on user’s computer/device
and shared network that may result in denial of service and for future use. Although cookies seemingly are not detri-
system’s performance degradation. Examples of viruses are mental, they can be menace if exploited by some spyware.
Melissa virus and Creeper virus. Likewise, tracking cookies can be utilized by hackers to gain
user’s personal details.
B. Worm 5) Riskware
Unlike virus, worm does not need ‘host’ but can run Riskware (aka grayware) is a genuine program when uti-
independently. Worms are self-replicating and self-propagating lized by the attacker can cause damage to system security
via a network or storage devices. Worms exploit operation or data by deletion, duplication, or modification. Authentic
system vulnerabilities, but do not corrupt user or system files. programs for riskware could be IRC client, file downloaders,
They consume computing and network resources by residing etc.
in main memory while replicating and spreading themselves 6) Sniffer
causing DoS and SPD. Examples are MyDoom and SQL It is a malicious program that observes and captures the
Slammer. network traffics. It analyzes various fields of packets and
gather data for preparation of malware attacks. Sniffers can
C. Trojan be ‘Ethereal’ (i.e., legitimate used for troubleshooting) and
Trojan surfaces as benign program but performs malevolent ‘BUTTSniffer’ (i.e., illegitimate for malign purposes). Exam-
activities in the backend without user’s knowledge. Trojans ples of sniffers are Wireshark and Aircrack-ng.
usually do not infect files or replicate themselves, rather create
backdoors for unauthorized system access to delete files, install E. Ransomware
programs or extricate private data (e.g., passwords). Examples Ransomwares are covertly installed on a victim computer to
are Zeus and Zitmo. execute a cryptovirology attack. This malware type encrypts
the data or locks down the system, thereby restricting user
D. Spyware access till ransom is paid. Specifically, ransomwares can be
Spyware spies on users without their knowledge or consent. classified in two main groups, viz. locker ransomwares that
It is generally used to surveil user activities, gather keystrokes decline access to the system/device functionalities, and crypto
or harvest sensitive information (e.g., login credentials). Ex- ransomware that avert access to files/data. Ransomwares ex-
amples of spyware are LogKext and GPSSpy. Following are amples are FakeDefender and TorrentLocker.
popular spyware sub-categories:
1) Adware F. Scareware
Adware dispenses either spiteful code/content or ads to
infected users via web browser, mobile app or PC’s desktop Scareware deludes people into buying/downloading inessen-
to generate attacker’s revenue. Another name of this malware tial and potentially perilous security software, opening attach-
is malvertising, as it may use reputed companies/banners to ments or visiting a malevolent website. It mostly attempts to
distribute malicious codes. It can be considered as a subcat- frighten users (e.g., by displaying false warning messages),
egory of spyware, but unlikely leading to a big harm until and when installed it collects stored information from victim
coupled with other spywares. Examples are AllSearchApp and system, which maybe sold to cybercriminals. Examples are
Plankton. Mac Defender and Registry Cleaner XP.
Pornware is also seen as a subclass of adware, when
installed maliciously without user’s knowledge to display G. Diallerware
pornographic materials. Diallerware send premium-rate SMS/multimedia messages
2) Keylogger without mobile user’s knowledge, thereby causing mone-
This malware is also called keystroke logger, password tary sums to user. The premium-rate SMS/multimedia mes-
grabbers, sniffer or information-stealing malware, which is sages/calls provide value-added services, which can be abused
employed by attackers to record each keystroke to steal sensi- by attackers. Attackers lure mobile owners to sign up for the
tive data (e.g., passwords, credit card numbers). Keylogger premium services managed by themselves, e.g., HippoSMS.
is generally transferred to a system when spiteful-software Diallerware blocks the messages from service providers to
is installed or -site is visited. Examples are SpyEye and users to avoid user’s awareness of unwanted additional
Formgrabber. charges.
3) Trackware
Trackware is unwanted software that tracks and collects
user activities and habits then share data with a third party. H. Bot
Though trackwares harm user’s privacy, they do not harvest A bot (abbreviated from robot) is a malicious program,
confidential or personally identifiable information. Examples which enables attacker (aka botmaster or bot herder) to re-
are Trackware. Rewardnet and Win32/WebHancer.A. motely control infected machine without user’s knowledge via
 3

Malware

Browser
Virus Worm Trojan Spyware Ransomware Scareware Diallerware Bot Rootkit Backdoor Downloader
Hijackers

Adware Keylogger Trackware Cookie Riskware Sniffer Spamware Reverse Shell Bootkit

Fig. 1: General taxonomy of malware.

a command and control (C&C) channel from a system called K. Browser Hijackers
C&C server. A cluster of bots controlled by a sole server It is an undesired software that alters settings of web
is known as botnet. Botnets can be employed to organize browser without user’s consent either to inject ads in the
DDoS attacks, phishing fraud, sending spams, etc. Well-known browser or replace home/error page and search engine. Some
examples are Sdbot and Agobot. of them may access sensitive data with spyware. Examples are
1) Spamware CoolWebSearch and RocketTab.
Spamware (aka spam sending malware or spambot) is ma-
licious software designed to search and compile list of email L. Downloader
addresses as well as sending large number of spam emails. It
is an element of a botnet functioning as a distributed spam- It is a malicious program that downloads and installs/runs
sending network. Spamware can use infected user’s email new versions of malwares from internet on compromised
ID or IP address to send emails, which may consume great computers. Downloader is usually embedded in websites and
amount of bandwidth and slow down the system. Examples software. Examples are Trojan-Downloader:W32/JQCN and
are Trik Spam and Necurs botnet. Trojan-Downloader:OSX/Jahlev.A.
2) Reverse Shell
A reverse shell is an unauthorized program (malware) III. M ALWARE C ONCEALMENT T ECHNIQUES
that provides access of undermined computer to the attacker. To evade anti-malwares, malware writers have applied fol-
Reverse shell enables attacker to run and type command on lowing different malware camouflage approaches [3]:
host as the attacker is local. Examples are Netcat and JSP web
shell. A. Encryption
Encrypted malware by this method consists of encryption
I. Rootkit
and decryption algorithms, keys and malicious codes. Each
A rootkit is a stealthy software that is devised to conceal time attacker employs new encryption algorithm and key to
specific programs/processes and enabling privileged access generate novel malware version. Since decryption algorithm
to computer/data. Rootkit allows the attacker accessing and remains same, there is a higher probability to be detected.
controlling the system remotely without being detected, as it The main target of this procedure is to avoid static analysis
normally runs with root privileges and subverts system logs and delaying investigation process. CASCADE was reported
and security software. Examples are NTRootkit and Stuxnet. as the first encrypted malware in 1987.
1) Bootkit
Bootkit is an advanced form of rootkits that infects master B. Packing
boot record or volume boot record. Since it resides in boot
sector, it is difficult to be detected by security software, and Packing mechanism is utilized to compress/encrypt malware
also stays active after system reboot. Well-known examples executable file. To detect malwares with packing technique,
are BOOTRASH and FinFisher. reverse engineering methods or correct unpacking algorithm
is needed, which sometime is hard as it requires knowledge
J. Backdoor of true packing/compression algorithm. UPX and Upack are
Backdoor is a malware that installs by itself and creates examples of packing.
secret entrance for attackers to bypass system’s authentication
procedures and to access and perform illegitimate activities. C. Obfuscation
Backdoors are never utilized alone but as foregoing malware This technique obscures program’s principal logic to stop
attacks of other kinds, as they do not harm but furnish wider others gaining associated knowledge of the code. Malwares
attack surfaces. A notable backdoor tool is Remote Access with obfuscation and their deleterious functionality stay un-
Terminal/Trojan (RAT). Other examples are Basebridge and intelligible till activated. Quintessential obfuscation strategies
Olyx. are inessential jumps and including garbage commands.
 4

Malware
Decision
Input Sample Feature Extraction Feature Selection Classifier/Clustering
(Explanation: Malware Analysis)
Benign
(Cleanware)

x1 y1 y2 Predicted Class

Probability Density
P N
… …
xk yk

Actual Class
P TP FN
xk+1 yk+1
--- --- N FP TN
xd yk (k<d) Y Malware Families

y1

Fig. 2: A generic malware detection and analysis system. First, input sample is provided to feature extraction module that yields
feature representation vector. A feature reduction/selection process is carried out on feature representation vector to obtain
fixed dimensionality regardless of length of input sample for enhanced performance. A classification/clustering technique
is trained on available set of malware and benign samples. During detection/analysis, unseen sample is reported by the
classification/clustering techniques as malware or not. Further analysis is also sometimes performed, e.g., describing suspicious
(or benign) characteristics present in the sample.

D. Polymorphism malware analysis such as malware variants detection (i.e.,

Polymorphic malware is designed to alter its appearance recognizing variant and families), malware category detection
every time it is executed while retaining original code entirely. (i.e., categorizing based on malwares’ prominent behaviors
Compared to encryption technique, boundless number of en- and objectives), malware similarity and novelty detection
cryption algorithms can be utilized by a polymorphic malware (i.e., acquiring knowledge about unknown sample by specific
such that in each implementation a decryption code’s portion is similarities and differences against known ones), malware de-
mutated. Transformation engine is generally kept in encrypted velopment detection (i.e., finding out if the malware writer has
malware. When any mutation occurs, a random encryption previously submitted it to online defense tools), and malware
algorithm is produced to re-encrypt the engine and malware attribution (i.e., identifying its programming language, from
with new decryption key. Different inimical actions can be where its launched and actor/group involved).
embedded under encryption operations. Since original code
remains intact, polymorphic malwares comparatively become
easy to be detected. First known polymorphic virus developed A. Malware Analysis
in 1990 is 1260.
In general, malware analysis is deployed both for detect-
E. Metamorphism ing/classification and other investigations (e.g., understand-
ing the working to devise novel identification schemes) of
Metamorphism malware (aka body-polymorphic malware) malware. Different features such as strings (i.e., frequency
mutates its malevolent codes in each execution to create of code fragments, names, etc.), byte sequences (i.e., char-
novel instance that has no similitude with native codes, but acterization of byte-level contents), opcodes (i.e., identifica-
functionality yet remains the same. There are two categories tion of machine/assembly-level operations), system/APIs calls
of metamorphic malwares. Open-world malware that mutates (i.e., analyses of execution traces/disassembly code or char-
by communicating with other sites over net, e.g., Conficker acterization of APIs’ executed actions), call graphs and data
worm. Open-world malware that reprograms itself without dependent (i.e., analyses of data being exchanged between
external communication by mutating binary code (i.e., binary process calls), control flow graphs (i.e., behavior relation-
transformer) or employing pseudocode representation, e.g., ships of data flow between system resources), multilayer
Win32/Apparition virus. dependency chains (i.e., characterization of sub-behaviors to
capture interactions among samples and system levels), causal
IV. M ALWARE D ETECTION AND A NALYSIS S YSTEM dependency graphs (i.e., tracking persistent state changes on
As Fig. 2 shows, a generic malware detection system target system), influence graphs (i.e., encoding of downloads
consists of four main modules: feature extraction, feature by malware), memory accesses (i.e., analyses of memory
selection, classification/clustering, and decision. The raw data during malware executions), file system (i.e., frequency of
sample is input to feature extraction module, which extricates created/deleted/modified files), system registry (i.e., count of
salient attributes as a feature set. Next, feature selection is queried/deleted/modified registry keys), CPU registers (i.e.,
performed to tackle the curse of dimensionality, to reduce the frequency of registers usages/changes), function length (i.e.,
computational complexity, and to increase performance of the number of bytes in a function), exceptions (i.e., exceptions
system by quantifying feature correlations. The resultant fea- prompted during malware execution) and network traffic (i.e.,
ture vector is given to a classifier/clustering scheme. Finally, analyses of incoming and outgoing packets, visited addresses,
decision module is employed either to acquire the final binary etc.) are being used for malware analysis. Malware analysis
decision: malware or benign (cleanware), or/and for additional can be conducted in following three ways:
 5

1) Static Analysis et al. [17] presented a deep learning strategy comprised

It is also called signature-based, code analysis, white- of AutoEncoder, multilayer restricted Boltzmann machines
box or misuse detection approach. Methods in this category and associative memory. The framework detects malware in
generally review statically the code-structure for traits of embedded systems via Windows API calls extricated from
infections using a pre-defined list of known assails’ signatures portable executable files.
without executing the sample [4]. However, advanced static Though dynamic analysis techniques are independent of
analysis techniques may run the sample by deploying reverse malware source-code and can detect unknown and zero-day
engineering, i.e., obtaining binary and assembly codes using malware instances, they require more resources (e.g., memory,
decompiler, disassembler and debugger. CPU time and disk space) and have high computational cost
Hellal et al. [5] presented a call code graph mining based and false positive rates.
static analysis scheme, called minimal contrast frequent sub- 3) Hybrid Analysis
graph miner algorithm, to distinguish variants of malware It is also called gray-box approach. Neither static- nor
in Windows environment. Schultz et al. [6] used features dynamic-analysis methods are unable to provide perfect anti-
like list of DLLs functions, system calls and hex-dump to malware solutions. Thus, hybrid-analysis approaches, which
detect novel and unseen malicious executables. Martin et al. combine benefits of both static and dynamic analyses, is more
[7] designed a malware detection method that uses third- desirable. For instance, Santos et al. [18] designed a hybrid
party API calls in Java files and multi-objective optimiza- method that integrates static (i.e., opcodes frequency) and
tion classification. While, Narayanan et al. [8] developed a dynamic (i.e., executable’s execution trace data) features with
mutli-view (MKLDROID) framework utilizing a graph kernel multitude of classifiers. Authors in [19] proposed a hybrid
with multiple kernel learning to determine sets of semantic technique that collects system calling runtime data and then
structures and contextual information from Android apps for utilizes a static scheme for mobile malware detection. While,
malware/malicious code localization. Yerima and Sezer [9] Dali et al. [20] developed a method that uses FlowDroid
proposed Android malware detection that analyzes permissions static analysis tool and sensitive sources data flows with deep
and intents from the apps via multilevel classifier rank fusion learning-based classifier.
architecture. Recenlty, Cakir et al. [10] designed a shallow
deep learning based method that employed word2vec features
via opcodes and a Gradient boosting classifier. B. Feature selection
Though static analysis techniques are capable of fast recog-
nizing malwares in versatile applications and pose no risk of The performance of malware detection depends on choice
infection while analyzing malwares, they need huge number of feature representation and length. The feature selec-
of pre-defined signature dataset. Moreover, they suffer from tion/dimensionality reduction is conducted to attain a set
runtime overhead and cannot discriminate variations of known- of more discriminative features for enhanced performance.
or obscure-malwares and zero-day intrusions. Various anti-malwares have been presented using filter, wrap-
2) Dynamic Analysis per and embedding based feature selection algorithms such
It is also called behavior-based, behavioral analysis, as distributed-, hierarchical-, correlation-, low-rank matrix
anomaly-based, specification-based or black-box approach. approximation-, forward-, backward-, local sensitive hashing-
Methods in this category assess samples via their activities , max relevance, adaptive feature scaling-, spectral graph
by executing them in a confined/simulated environment, e.g., theory-, F1-score, F2-score, mean decrease impurity-, docu-
sandboxed, simulator, debugger, virtual machine or emulator. ment frequency-, information gain-, information gain ratio-,
Miao et al. [11] proposed a bi-layer behavior abstraction principal component analysis- and latent dirichlet allocation
technique via semantic examination of dynamic API sequences [3].
in Windows environment. Lower- and higher-layer behaviors
were captured using data dependence of APIs and complex
C. Classification/Clustering
good interpretability of lower abstractions, respectively. In
[12], authors developed a graph-based model harnessing re- To identify if a given sample is malicious or/and to de-
lations (i.e., dependency graphs) among system-calls’ groups termine malware family, various binary and multiclass clas-
for smartphone malicious software detection, but the model sification techniques such as Multilayer Perceptron, Support
requires high time consumption. Authors in [13] presented Vector Machines, Naı̈ve Bayes, Decision Tree, Rule-based,
a compression-based feature mining on system/API calls’ Random Forests, Multiple Kernel Learning, K-Nearest Neigh-
quantitative information flow graphs to detect Windows mal- bors, Logistic Regression, Ensemble, Multi-Objective Evolu-
ware. Mao et al. [14] designed a security dependence net- tionary by Genetic Algorithm, Deep Belief Networks have
work from access behaviors to evaluate importance of system been employed [4].
resources (e.g., files, registry, and processes) and malware Hierarchical-, K-means-, meanShift-, K-medoid
detection. While, Egele et al. [15] presented a dynamic blan- partitional-, density-based spatial-, prototype-, self-organizing
ket execution function that employs high-level API-relevant maps-, single-linkage- and locality sensitive hashing-based
semantic features. Enck et al. [16] presented TaintDroid for clustering techniques have been utilized to categorize malware
dynamic taint examination to trace leakage of sensitive data samples exhibiting identical behaviors into different groups
(e.g., microphone, GPS and camera) in third-party apps. Ye or to generate signatures for detection [3].
 6

D. Evaluation Metrics more complex and residual deep learning, dictionary learning
Performance of malware detection methods is generally and data mining should be explored for feature segmenta-
evaluated by False Positive Rate = FP/(FP + TN), True Positive tion/representation learning/selection/classification and deter-
Rate = TP/(TP + FN), specificity = TN/(TN + FP), precision mining temporal relationships within and between malware
= TP/(TP + FP), accuracy = (TP + TN)/(TP + TN + FP sections.
+ FN), where TP, FP, TN and FN are true positives, false
positives, true negatives and false negatives, respectively. Mal- C. Mobile device malwares
ware samples are commonly considered as positive instances. Smart-devices connected to internet is growing exponen-
Moreover, Matthews correlation coefficient, F-score, Kappa tially, so as malwares (especially via third party apps) against
statistic, confusion matrix, receiver operating characteristic them. Insubstantial studies have been conducted on mobile
and under the curve measures have been used. While, for device malwares. Moreover, most existing anti-malware tech-
clustering-based algorithms Macro-F1 and Micro-F1 metrics, niques are not real-time and unsuited for mobile devices
respectively, for accentuating the performance on rare and because of high computational cost and/or features complexity
common categories [3], [4]. used for analysis. Thus, real-time lightweight mobile anti-
malwares via Bayesian classification is an interesting re-
V. R ESEARCH CHALLENGES AND OPPORTUNITIES search direction to be explored. Multiple information from
The ever-growing demand of minimized failure rates of anti- in-built sensors (e.g., accelerometer) may enhance mobile
malware solutions have opened up exigent research opportu- anti-malware performance. Mobile hardware malware detec-
nities and challenges to be resolved yet. tion and removal is another issue that needs serious explo-
ration. Sooner mobile anti-malware-inspired techniques will
substantially impact smart-devices design. Anyway, smart-
A. Issues in existing anti-malware methods
device malwares should be tackled both by preventive and
Malwares are still exponentially evolving in sophistication, effective countermeasures. App developers should assure that
and more difficult plights lie ahead. Most prior static and dy- their apps are abiding security and privacy policies. App stores
namic or hybrid methods do not work for novel/unknown/zero- administrators should vet and remove dubious apps. Users
day signatures and require virtual environment plus are time should use superior anti-malwares and install trusted apps.
consuming, respectively. Nonetheless, virtual environments On the whole, wearable and mobile devices malware and
are becoming less effective as malware writers are usually anti-malware are a new research field in cybersecurity with
one step ahead by implementing high-level new techniques pressing problems worth researching like malware affecting
to conceal malicious features. Though efforts are afoot to device’s master boot record or stealthily exploiting device to
design multi-level and parallel processing system, existing mine cryptocurrency, and how a malware performing well on
anti-malware methods/tools all in all are not adequate or benchmark data will be better under real-world environments.
potent for higher levels of concealments. Current anti-malware
systems also face challenges like scalability, lack of truly real-
world representative datasets, irreproducibility of published D. Large-scale benchmark databases
results, low generalization and detection disagreement among Advancement in malware research deeply depends on the
them for the same samples. There is a need of improved public availability of comprehensive benchmark datasets incor-
and comprehensive malware countermeasures, which could be porating accurate labels and contexts. Most existing databases
developed by utilizing recent advanced-machine/deep learn- suffers from limitations like small size, missing informa-
ing, -data mining and -adaptive schemes. Also, approaches tion/features, imbalanced-classes, and not publicly available.
embodying anomaly analysis with behavioral data should be Lack of adequate large-scale public datasets has stymied
designed to investigate what the malware is doing rather than research on malware. Benchmark public datasets will assist
how it is doing. This may result in minimized error and false to compare independent anti-malware schemes, determine
alarm rates. inter and intra relationships between security infringement
phenomena and unify malware findings to draw determined
B. Advanced machine learning (AML) techniques for anti- conclusions with reference to statistical significance. Neverthe-
malware less, collecting large-scale heterogenous annotated databases is
challenging and time- and resource-consuming due to malware
Quintessential anti-malwares often depend on non-linear attributes, forms and behaviors diversity. Crowdsourcing may
adversary explicit models and expert domain knowledge, help accumulating different annotated large-scale databsets.
thereby making them prone to overfitting and lower overall
reliability. Conversely, AML techniques attempt to imitate at-
tackers with various content, contexts, etc. rather than explicit E. Graph-based malware analysis
models/systems/attacks. Few preliminary studies on shallow Malwares with concealments are dominant nowadays and
AML usage for anti-malware has been conducted, but still a effectual in evading conventional anti-malwares that largely
lot of efforts to be done regarding AML anti-malware. For im- disregard learning and identifying the underlying relationships
proved accuracy, flexibility and scalability on wide range and between samples and variants, and contextual information.
unknown samples, AML paradigms like open set recognition, Graph-based relationship representations and features (e.g.,
 7

data- and control-flow graphs, call graphs, data-, program-, network. IoT cyber-security is relatively new research realm
and control-dependency graphs) offer interesting possibility and quite challenging owing to heterogeneous networks with
even when malware code is altered as it helps in tracking multisource data and several categories of nodes. To this
malware genealogy in different settings. Devising graph-based end, different routes (e.g., predictive and blockchain) could
anti-malwares yet have issues from data heterogeneity, noisy be effective. Predictive security is attaining cyber resiliency
and incomplete labels, and computational cost during real- by devising models that predict future attacks and prevent in
time detection. Up to some extend such challenges may advance. As there is a strong correlation between security
be addressed in decentralized fashion. Furthermore, use of infractions and human blunders, predictive models should
multiple directed and undirected graphs, multi-view spectral consider computer networking, social sciences, descriptive
clustering, heterogeneous networks, multiple graph kernel theory, uncertain behavior theory and psychology from at-
learning, dynamic graph mining and deep graph convolution tackers, users and administrators’ perspectives at different
kernels to capture contextual and structural information could granularity levels. Blockchain can be utilized for self-healing
be fruitful area of research. of compromised devices/systems. Models could be devised
that exploit e.g., redundancy to heal corrupted codes/software
F. Bio-inspired anti-malware by good codes replacements, since in blockchain one can trace
and roll back the firmware versions. However, such models
Several limitations of traditional anti-malwares could be should also be capable to handle resource, energy and com-
suppressed by bio-inspired (e.g., biological immune sys- munication constraints, which may be achieved by lightweight
tem, biological evolution, genetic algorithms and swarm in- machine/transfer/reinforcement learning based access control
telligence) techniques. Comparatively these techniques are protocols.
lightweight, highly scalable and less resource-constrained.
Adaptive bio-inspired techniques that is used both for intelli-
I. Deception and moving target anti-malware techniques
gent concealment-invariant feature extraction and classification
can dramatically enhance accuracy in the wild. Bio-inspired Deception techniques (e.g., honeypot) are being used to
methods that define particular objective functions to discrim- detect and prevent malwares, which lures adversaries to strike
inate a system under attack from a malfunctioning or failing in order to mislead them with false information. There are
may also help strengthening the security. Combination of bio- two kinds of honeypots, i.e., client and server. Honeypot helps
inspired algorithms with deep neural networks is one of the to reduce false positives and prevent DDoS attacks. Com-
most promising direction, however has been explored less in plex attacks/tools (e.g., polymorphic malware) is increasing
anti-malwares. to identify honeypots or to alter their behaviors to deceive
honeypots themselves. Also, honeypot can be exploited by
attackers to undermine other sensitive parts of frameworks.
G. Defense-in-depth anti-malware
More complicated honeypot and honey nets (i.e., bunch of
Anti-malware strategy that is composed of multiple defense honeypots) schemes (e.g., shadow honeypots) should be de-
levels/lines rather than single is called defense-in-depth. Such vised as compromised honeypot will put security of whole
strong defensive mechanism is expected to be more robust organization in danger.
as it doesn’t depend on one defense technique and if one Moving target techniques (aka dynamic platform methods-
is breached the others aren’t. Each machine/cyber-system DPMs) dynamically randomizes system components to sup-
architecture can be divided in various levels of depth, e.g., in press successful attacks’ likelihood and shorten attack lifetime.
a power grid system, the meters, communication frameworks, Though adversary must undermine all platforms not one to
and smaller components, respectively, could be envisaged as evade DPMs, DPMs require complicated application state
lowest, intermediate and highest level. Another solution is ac- synchronization among varying platforms, and expand the sys-
tive or adaptive malware defense. Active defense has received tem’s attack surface. Much less efforts have been dedicated to
little attention due to inherent complexity, where developer developing well-articulated attack models and how to upgrade
anticipates attack scenarios at different level and accordingly deception elements and strategy to confront dynamic changes
devises malware countermeasures. In adaptive defense, the in attack behaviors. Future research should concentrate on
system is persistently updated by retraining/appending novel devising unorthodox methodologies, performing real-world
features or dynamically adjusted corresponding to reshaping analyses to compute and compare effectiveness of deception
environments. Adaptive defenses would require fast, auto- and DPMs techniques, and studying if DPMs conflict or can
mated and computationally effective and could use unsuper- co-exist with other anti-malwares.
vised learning and concept drift.
J. Decentralized anti-malware
H. Internet of things (IoT) attacks Data sharing and trust management hinder current anti-
IoT are progressively being used in different domains rang- malwares advancement, which can be resolved by decentral-
ing from smart-cities to smart- and military-grids. Despite ized malware detectors using blockchain technology. But it
finest security endeavors, IoT devices/systems can also be has received little attention till now. For intersection of anti-
compromised by innovative cyber-attacks. Security of IoT malware and blockchain technology, future directions will
technology is more crucial as it is always connected to a include exploring overhead traffic handling, quality and sparse
 8

malware signatures, building accurate dynamic normal nature of synchronous parallel processing (e.g., Spark) and to develop
of traffics, reducing massive false alerts, energy and cost, body of knowledge on pros and cons of big data anti-malware
blockchain latency, case-by-case scenario investigation, and tools to assist practitioners.
more proof-of-concept implementations.
N. Malware analysis visualization systems
K. Botnet countermeasures Existing methods to analyze malwares are time-consuming
Thwarting botnets has become key area. Several botnet de- for malware analysts. Highly interactive visual analysis would
tection and defense architectures have been proposed. Various aid researchers and practitioners to forensically investigate,
issues surround botnet countermeasure study, e.g., difficulties summarize, classify and compare malwares more easily. Most
in testing devised botnet defenses in real scenarios/data. Be- prior techniques are very limited with regard to interactivity,
sides, lack of widely acknowledged benchmark or standard mapping temporal dimensions, scalability and representation
methodology to quantitative evaluate or compare bot defenses space (e.g., they are superficially 2D rather than 3D). The
presumably due to privacy and data sharing concerns. Botnets, field of developing malware visualization systems covering
including IoT bot and socialbot, will continue to rise until consequential rang of malware types and environments is vital
effective means both technical and non-technical are taken. and emerging. Encyclopedic visualization systems will lead
Technical factors include passive internet service providers analysts/researchers to ascertain novel research domains in the
and unassertive software. Non-technical factors include estab- years to come.
lishing distributed global environment, local and multinational
legal issues and poor user awareness. O. Multimodal anti-malwares
Multimodal anti-malwares, which consolidate evidences
L. Privacy preservation from different kinds of features/sources (e.g., string, permis-
Malwares that steal sensitive information has received much sion, elements converted to image matrices) can overcome nu-
attention. However, preserving user privacy in malware analy- merous constraints in frameworks that consider only one/fewer
sis (especially at the cloud or third party server) and malware features. Multimodal frameworks are more flexible and can
data sharing is yet an open and seldom touched concern. significantly enhance the accuracy of unimodal ones in the
Establishing privacy and regaining trust in commercial anti- wild. Multimodal may include multiple sensors, algorithms
malwares would become difficult if user’s privacy/data is and instances, and information can be fused at feature, score
compromised once. Majority of prior anti-malwares overlook or decision level. There is ample room to develop novel fusion
the privacy and security of user, data and network. Thus, architectures. Moreover, multimodal frameworks are expected
reasonably little has been worked on privacy protection frame- to be intrinsically more robust to concealments, but no study
works to respect public and law opinions. Privacy preservation investigated how robust are they to concealments.
mechanisms that do not influence the detection performance is
practically worthy of contemplation. Formulating lightweight P. Clustering for malware analysis
detection and privacy protection systems usable on mobile Previous works have shown that clustering could be a useful
devices to balance security, efficacy, privacy and power con- tool to effectively classify unknown malwares for improved
sumption demands special considerations. More innovative generalization, to underline unseen family’s behaviors for
privacy preservation approaches (e.g., allowing user to sta- thorough analysis that may help more robust anti-malware
bilize privacy, convenience and security levels) in malware schemes, and to label huge volumes of malwares in fast and
analysis has been highlighted by many experts as an essential automatic fashion that has become major challenge. Future
future research to be carried out. goal should be further improving accuracy of clustering-based
malware analysis using cluster quality measurements, contex-
M. Big data malware analysis tual/metadata information, and boosted genetic algorithms, etc.
Attentions should also be given to rectify security issues, e.g.,
The demand for big data malware analysis frameworks
poisoning and obfuscation attacks against targeted clusters.
is steadily expanding. Practitioners are working to resolve
big data malware challenges such as volume (e.g., collect-
ing, cleaning and compressing data/labels), velocity (e.g., Q. Hardware-based Solutions
real-time online training, learning, processing or streaming Hardware-based detectors are recently getting momentum
big data), variety (e.g., heterogeneous multi-view data learn- against proliferation of malware. Such detection mechanisms
ing/embedding), veracity (e.g., leaning with contradicting and utilize low-level architectural features, which are obtained by
unreliable data), and value (explainable ML based malware redesigning the micro-architecture of computer processors,
analysis). Another promising future research direction is de- e.g., CPUs with special registers providing hardware and
vising large-scale feature selection techniques, which are software anomaly events. Nevertheless, research in this domain
less-dependent on feature engineering, via distributed feature and trustworthy systems (i.e., inherently secure and reliable
selection, low-rank matrix approximation, adaptive feature against human errors and hostile parties) is yet in its initial
scaling, spectral graph theory, and fuzzy and neuro-fuzzy genesis and has to go a long way. Furthermore, there is dearth
clustering. Rigorous efforts need to be made to investigate use of studies on efficacy of anti-malwares combining hardware-
 9

and software-based techniques that have exceptional potential current. Some training camps/workshops are being held by
to uncover extra elaborate malwares. Likewise, smart devices’ companies/organizations also for general public, but they are
sensors (e.g., GPS and ambient light sensors) data could also exceptionally expensive. More on-line free-to-access training
be used as additional feature vector to profile malware. courses will surely diminish malware damages.

R. Malware adversarial learning U. Interdisciplinary research

Machine-learning (ML) recently has been used to achieve To advance state-of-the-art malware analysis, the research
effective malware defenses, however they are not designed for and industrial communities need to support and promote
situations where an adversary is actively trying to impact out- interdisciplinary fundamental science research and develop-
comes. Specially, deep learning-based countermeasures lack ment (including contributions from machine learning, human
robustness against adversarial examples (i.e., sample crafted psychology, computer engineering, etc.) to accomplish de-
from genuine samples with careful minor perturbations). At- pendable, natural, and generalized anti-malware techniques.
tackers can also inject poisoning samples in (online/adaptive)
training database with the aim to remarkably decrease ML-
malware countermeasure’s accuracy at testing phase. A com- VI. C ONCLUSION
prehensive analysis for each malware considering attacker’s Malwares, including in mobile and smart devices, have
capability and what features to what extend should be modified become more sophisticated and greater in frequency during
to avoid detection has not been done yet. It is still difficult recent years. Although there exist many defense tools and
task to design ML anti-malwares that are robust in adversarial mechanisms, malware detection and analysis are still challeng-
setting. Researchers should explore malware adversarial ML ing tasks, since malware developers continuously conceal the
in identifying probable countermeasures’ vulnerabilities both information in attacks or evolve cyber-attacks to circumvent
at train and test stages, devising homologous assails and their newer security techniques, plus some prior methods face low
impacts and developing techniques to enhance robustness of generalization to unknown malwares and scalability issues.
ML-based anti-malwares. It is hoped that this academic and perspective article will
stimulate focused interdisciplinary research and development
S. Performance evaluation framework in anti-malware towards aggrandizing its full potential in
different cyberspace applications.
Malware analysis accuracy/performance, which is used to
evaluate, compare, or configure anti-malwares, in general
lacks standardization. A unified and comprehensive evaluation R EFERENCES
framework should be developed to rank present and future [1] P. Faruki, A. Bharmal, V. Laxmi, V. Ganmoor, M. S. Gaur, M. Conti, M.
methods, that incorporates static and dynamic techniques, Rajarajan, “Android Security: A Survey of Issues, Malware Penetration,
adversary’s goal, knowledge and capability, attack strategies and Defenses”, IEEE Communications Surveys & Tutorials, 17-2:998-
1022, 2015.
at train and test phase, evaluation metrics (i.e., security- and [2] “The Cost of Malicious Cyber Activity to the U.S. Economy”, Council
privacy-relevant error rates as most current methods do not of Economic Advisors, February 2018, https://www.whitehouse.gov/wp-
cover all aspects), and common parlance to elucidate anti- content/uploads/2018/02/The-Cost-of-Malicious-Cyber-Activity-to-the-
U.S.- Economy.pdf (Last accessed: 2/12/2018).
malware performances. Any such framework with common [3] A. Souri, R. Hosseini, “A state-of-the-art survey of malware detection
criteria and open online platform for evaluating resilient, mal- approaches using data mining techniques”, Human-centric Computing and
ware sophistication, decision making, policies, experimental Information Sciences, 8:1-22, 2018.
[4] D. Ucci, L. Aniello, R. Baldoni, “Survey on the usage of machine learning
setups, big databases, and open source codes will surely help techniques for malware analysis”, arXiv preprint arXiv:1710.08189, pp.
both in reporting baseline performances without giving a false 1-67, 2017.
sense of progress and encouraging reproducible research on [5] A. Hellal, L.B. Romdhane, “Minimal contrast frequent pattern mining for
malware detection”, Computer Security, 62:19-32, 2016.
scalability and challenges in real-world scenarios. [6] M.G. Schultz, E. Eskin, F. Zadok, “Data mining methods for detection of
new malicious executables”, Proceedings of IEEE Symposium on Security
and Privacy, pp. 1-12, 2001.
T. Malware education [7] A. Martin, H.D. Menendez, D. Camacho, “MOCDroid: multi-objective
evolutionary classifier for Android malware detection”, Soft Computing,
Most malwares succeed contemplating humans as weakest 21:7405-7415, 2016.
link. Additionally, there is growing demand for cybersecurity [8] A. Narayanan, M. Chandramohan, L. Chen, Y. Liu, “A multi-view
workforces, therefore it is imperative to educate people about context-aware approach to Android malware detection and malicious code
malware safety. In academic institutions, malware analysis localization”, 2017:1-53, Empir Software Eng., 2017.
[9] S.Y. Yerima and S. Sezer, “DroidFusion: A novel multilevel classifier
and related courses should be taught both at undergraduate fusion approach for Android malware detection”, IEEE Transactions on
and graduate levels. Nonetheless, relatively very limited col- Cybernetics, 10:1-14, 2018.
leges/universities offer malware courses, which maybe because [10] B. Cakir and E. Dogdu, “Malware classification using deep learning
methods”, Proceedings of ACMSE Conference, pp. 1-10, 2018.
of the shortage of agreement on fundamental topics among [11] Q. Miao, J. Liu, Y. Cao, J. Song, “Malware detection using bilayer
institutions, book and training providers, and ethical sensitivity behavior abstraction and improved one-class support vector machines”,
of educating/creating white-hats. Moreover, most academic International Journal of Information Security, 15:361-379, 2016.
[12] S.D. Nikolopoulos, I. Polenakis, “A graph-based model for malware de-
courses being offered are practitioner-oriented but not science- tection and classification using system-call groups”, Journal of Computer
/research-oriented and heavily rely on text books that are not Virology Hacking Techniques, 13:29-46, 2016.
 10

[13] T. Wuechner, A. Cislak, M. Ochoa, A. Pretschner, “Leveraging

compression-based graph mining for behavior based malware detection”,
IEEE Transactions Dependable Secure Computing, 16:1-14, 2017.
[14] W. Mao, Z. Cai, D. Towsley, Q. Feng, X. Guan, “Security importance as-
sessment for system objects and malware detection”, Computer Security,
68:47-68, 2017.
[15] M. Egele, M. Woo, P. Chapman, D. Brumley, “Blanket execution: Dy-
namic similarity testing for program binaries and components”, USENIX
Security Symposium, pp. 303-317, 2014.
[16] W. Enck, P. Gilbert, B.-G. Chun, L.P. Cox, J. Jung, P. McDaniel, A.N.
Sheth, “TaintDroid: An information-flow tracking system for realtime
privacy monitoring on smartphones”, ACM Transactions Computing
System, 32(2):1-8, 2014.
[17] Y. Ye, L. Chen, S. Hou, W. Hardy, X. Li, “DeepAM: a heterogeneous
deep learning framework for intelligent malware detection”, Knowledge
Information System, 54:265-285, 2017.
[18] I. Santos, J. Devesa, F. Brezo, J. Nieves, P.G. Bringas, “OPEM: A
static-dynamic approach for machine-learning-based malware detection”,
International Joint Conference CISIS’12-ICEUTE’12-SOCO’12, pp. 271-
280, 2012.
[19] Fei Tong and Zheng Yan, “A hybrid approach of mobile malware
detection in Android”, Journal of Parallel Distributed Computing, 103:22-
31, 2017.
[20] Z. Dali, J. Hao, Y. Ying, D. Wu, C. Weiyi, “DeepFlow: deep learning-
based malware detection by mining Android application for abnormal
usage of sensitive data”, IEEE symposium on computers and communi-
cations, pp. 438-443, 2017.