skip to main content
10.1145/1879141.1879202acmconferencesArticle/Chapter ViewAbstractPublication PagesimcConference Proceedingsconference-collections
research-article

What happened in my network: mining network events from router syslogs

Published: 01 November 2010 Publication History

Abstract

Router syslogs are messages that a router logs to describe a wide range of events observed by it. They are considered one of the most valuable data sources for monitoring network health and for trou- bleshooting network faults and performance anomalies. However, router syslog messages are essentially free-form text with only a minimal structure, and their formats vary among different vendors and router OSes. Furthermore, since router syslogs are aimed for tracking and debugging router software/hardware problems, they are often too low-level from network service management perspectives. Due to their sheer volume (e.g., millions per day in a large ISP network), detailed router syslog messages are typically examined only when required by an on-going troubleshooting investigation or when given a narrow time range and a specific router under suspicion. Automated systems based on router syslogs on the other hand tend to focus on a subset of the mission critical messages (e.g., relating to network fault) to avoid dealing with the full diversity and complexity of syslog messages. In this project, we design a Sys-logDigest system that can automatically transform and compress such low-level minimally-structured syslog messages into meaningful and prioritized high-level network events, using powerful data mining techniques tailored to our problem domain. These events are three orders of magnitude fewer in number and have much better usability than raw syslog messages. We demonstrate that they provide critical input to network troubleshooting, and net- work health monitoring and visualization.

References

[1]
Emc lonix website. http://www.emc.com/ products/family/ionix-family.htm.
[2]
Ibm netcool website.http://www-01.ibm.com/ software/tivoli/welcome/netcool.
[3]
Rakesh Agrawal, Tomasz Imielinski, and Arun Swami. Mining association rules between sets of items in large databases. In Proc. ACM SIGMOD, 1993.
[4]
Daniela Brauckhoff, Xenofontas Dimitropoulos, Arno Wagner, and Kavè Salamatian. Anomaly extraction in backbone networks using association rules. In Proc. ACM IMC, 2009.
[5]
Peter J. Brockwell and Richard A. Davis. Introduction to Time Series and Forecasting. Springer, 2002.
[6]
R. Gerhards and Adiscon GmbH. The Syslog Protocol. In IETF RFC, 2009.
[7]
Yiyi Huang, Nick Feamster, Anukool Lakhina, and Jim (Jun) Xu. Diagnosing network disruptions with network-wide analysis. SIGMETRICS Perform. Eval. Rev., 35(1):61--72, 2007.
[8]
Charles R. Kalmaneka, Zihui Ge, Seungjoon Lee, Carsten Lund, Dan Pei, Joseph Seidel, Jacobus Van der Merwe, and Jennifer Yates. Darkstar: Using exploratory data mining to raise the bar on network reliability and performance. In Proc. the 7th international workshop on Design of Reliable Communication Networks (DRCN), October 2009.
[9]
Srikanth Kandula, Ranveer Chandra, and Dina Katabi. WhatâA -- Z -- s going on? learning communication rules in edge networks. In Proc. ACM SIGCOMM, 2008.
[10]
Srikanth Kandula, Ratul Mahajan, Patrick Verkaik, Sharad Agarwal, Jitendra Padhye, and Paramvir Bahl. Detailed diagnosis in enterprise networks. In Proc. ACM SIGCOMM, 2009.
[11]
R. R. Kompella, J. Yates, A. Greenberg, and A. C. Snoeren. Detection and localization of network blackholes. In Proc. INFOCOM, 2007.
[12]
Anukool Lakhina, Mark Crovella, and Christophe Diot. Mining anomalies using traffic feature distributions. In SIGCOMM'05: Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications, pages 217--228, New York, NY, USA, 2005. ACM.
[13]
A. Mahimkar, Z. Ge, A. Shaikh, J. Wang J. Yates, Y. Zhang, and Q. Zhao. Towards automated performance diagnosis in a large iptv network. In Proc. ACM SIGCOMM, 2009.
[14]
A. Mahimkar, J. Yates, Y. Zhang, A. Shaikh, J. Wang, Z. Ge, and C. T. Ee. Troubleshooting chronic conditions in large ip networks. In Proc. ACM CoNEXT, 2008.
[15]
Ajay Mahimkar, Han Hee Song, Zihui Ge, Aman Shaikh, Jia Wang, Jennifer Yates, Yin Zhang, and Joanne Emmons. Detecting the performance impact of upgrades in large operational networks. In Proc. ACM SIGCOMM, 2010.
[16]
Dave Plonka. Flowscan: A network traffic flow reporting and visualization tool. In Proc. USENIX System Admin. Conf., 2000.
[17]
M. Tariq, A. Zeitoun, V. Valancius, N. Feamster, and M. Ammar. Answering what-if deployment and configuration questions with wise. In Proc. SIGCOM, 2008.
[18]
Yinglian Xie, Fang Yu, Kannan Achan, Rina Panigrahy, Geoff Hulten, and Ivan Osipkov. Spamming botnets: Signatures and characteristics. In Proc. ACM SIGCOMM, 2008.
[19]
Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I. Jordan. Detecting large-scale system problems by mining console logs. In SOSP '09: Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, pages 117--132, New York, NY, USA, 2009. ACM.
[20]
Kenji Yamanishi and Yuko Maruyama. Dynamic syslog mining for network failure monitoring. In Proc. ACM KDD, August 2005.
[21]
Nong Ye, S. Vilbert, and Qiang Chen. Computer intrusion detection through EWMA for autocorrelated and uncorrelated data. In IEEE transactions on reliability, October 2003.

Cited By

View all
  • (2023)Deep Reinforcement Learning Based Command Control System for Automating Fault Diagnosis2023 19th International Conference on Network and Service Management (CNSM)10.23919/CNSM59352.2023.10327867(1-5)Online publication date: 30-Oct-2023
  • (2023)LPV: A Log Parsing Framework Based on VectorizationIEEE Transactions on Network and Service Management10.1109/TNSM.2023.324812420:3(2711-2725)Online publication date: Sep-2023
  • (2023)System Log Parsing: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3222417(1-20)Online publication date: 2023
  • Show More Cited By

Index Terms

  1. What happened in my network: mining network events from router syslogs

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    IMC '10: Proceedings of the 10th ACM SIGCOMM conference on Internet measurement
    November 2010
    496 pages
    ISBN:9781450304832
    DOI:10.1145/1879141
    • Program Chair:
    • Mark Allman
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    In-Cooperation

    • USENIX Assoc: USENIX Assoc

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 01 November 2010

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. syslog
    2. troubleshooting

    Qualifiers

    • Research-article

    Conference

    IMC '10
    IMC '10: Internet Measurement Conference
    November 1 - 30, 2010
    Melbourne, Australia

    Acceptance Rates

    Overall Acceptance Rate 277 of 1,083 submissions, 26%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)23
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Deep Reinforcement Learning Based Command Control System for Automating Fault Diagnosis2023 19th International Conference on Network and Service Management (CNSM)10.23919/CNSM59352.2023.10327867(1-5)Online publication date: 30-Oct-2023
    • (2023)LPV: A Log Parsing Framework Based on VectorizationIEEE Transactions on Network and Service Management10.1109/TNSM.2023.324812420:3(2711-2725)Online publication date: Sep-2023
    • (2023)System Log Parsing: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3222417(1-20)Online publication date: 2023
    • (2022)BigBen: Telemetry Processing for Internet-Wide Event MonitoringIEEE Transactions on Network and Service Management10.1109/TNSM.2022.318459319:3(2625-2638)Online publication date: Sep-2022
    • (2021)Network Problem Diagnostics using Typographic Error Correction2021 17th International Conference on Network and Service Management (CNSM)10.23919/CNSM52442.2021.9615525(482-490)Online publication date: 25-Oct-2021
    • (2021)An empirical investigation of practical log anomaly detection for online service systemsProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3473933(1404-1415)Online publication date: 20-Aug-2021
    • (2021)Design of Interactive Visualizations for Next-Generation Ultra-Large Communication NetworksIEEE Access10.1109/ACCESS.2021.3057803(1-1)Online publication date: 2021
    • (2021)amulog: A general log analysis framework for comparison and combination of diverse template generation methods*International Journal of Network Management10.1002/nem.219532:4Online publication date: 19-Dec-2021
    • (2020)Efficient and Robust Syslog Parsing for Network Devices in Datacenter NetworksIEEE Access10.1109/ACCESS.2020.29726918(30245-30261)Online publication date: 2020
    • (2019)Anomalies Detection and Proactive Defence of Routers Based on Multiple Information LearningEntropy10.3390/e2108073421:8(734)Online publication date: 26-Jul-2019
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media

    pFad - Phonifier reborn

    Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

    Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


    Alternative Proxies:

    Alternative Proxy

    pFad Proxy

    pFad v3 Proxy

    pFad v4 Proxy