Introduction to Logging

with the ELK Stack

Amy Ghate
Solutions Architect

1
 +
=
+ ObservaBLT
Observability

2
 Elastic Approach to Observability

Dev & Ops Teams

Log Data Metrics Data APM Data Uptime Data

Web Logs Container Metrics Real User Monitoring Uptime

App Logs Host Metrics Txn Perf Monitoring Response Time
Database Logs Database Metics Distributed Tracing
Container Logs Network Metrics
Storage Metrics

3
 Agenda
Things we're going to cover

1 Challenges with log analytics

2 Sending logs to Elasticsearch

3 Beyond logging: Observability

4 Leveraging Elastic security

4
 Agenda
Challenges with log analytics

1 Challenges with log analytics

2 Sending logs to Elasticsearch

3 Beyond logging: Observability

4 Leveraging Elastic security

5
 Logs for one host or app
This is fairly straightforward
$ > tail -f /var/log/messages

Dec 10 14:05:30 justa-build kernel: type=1326 audit(1575986730.517:383998660): auid=4294967295

uid=0 gid=0 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 pid=17069 comm="node"
sig=0 arch=c000003e syscall=324 compat=0 ip=0x7efe9c254889 code=0x50000
Dec 10 14:05:30 justa-build kernel: type=1326 audit(1575986730.551:383998661): auid=4294967295
uid=0 gid=0 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 pid=17069 comm="node"
sig=0 arch=c000003e syscall=332 compat=0 ip=0x7efe9c269171 code=0x50000
Dec 10 14:05:33 justa-build kernel: type=1326 audit(1575986733.110:383998662): auid=4294967295
uid=0 gid=0 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 pid=17179 comm="node"
sig=0 arch=c000003e syscall=324 compat=0 ip=0x7fee1cf0f889 code=0x50000
Dec 10 14:05:33 justa-build kernel: type=1326 audit(1575986733.150:383998663): auid=4294967295
uid=0 gid=0 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 pid=17179 comm="node"
sig=0 arch=c000003e syscall=332 compat=0 ip=0x7fee1cf24171 code=0x50000
Dec 10 14:05:35 justa-build kernel: type=1326 audit(1575986735.155:383998664): auid=4294967295
uid=0 gid=0 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 pid=17367 comm="node"
sig=0 arch=c000003e syscall=324 compat=0 ip=0x7ffb3b7bf889 code=0x50000
Dec 10 14:05:35 justa-build kernel: type=1326 audit(1575986735.194:383998665): auid=4294967295
uid=0
6 gid=0 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 pid=17367 comm="node"
 Interacting with logs
Built-in tools for log viewing
• grep
• tail
• cat / less / more / type
• sed / awk / perl
• vim / notepad / event viewer
• clever combinations of the above

7
8
 Immediate needs for log analytics
What's missing from the previous desktop

• Easy setup for a variety of sources

• Correlating and cross referencing
• Searching, filtering, and highlighting
• Visualize
• Anomaly detection and alerting
• Flexible retention

9
 Agenda
Things we're going to cover

1 Challenges with log analytics

2 Sending logs to Elasticsearch

3 Beyond logging: Observability

4 Leveraging Elastic security

10
 We're running in Elastic Cloud
Works the same in the cloud or running the default distribution

11
 Click on the Logging Button
Works the same in the cloud or running the default distribution

12
 Many choices
We're going to ingest the System logs

13
 Detailed instructions
Context-aware instructions for cloud or on-prem installs

14
 Getting Started
Cloud or on-prem installs

• Download and install Filebeat

• Edit the configuration
• Enable and configure the system
module
• Start Filebeat
• Check out the dashboard!

15
 Steps
Download and install Filebeat
$ >curl -LO --silent \
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.0-darwin-x86_64.tar.gz

$ >tar xzvf filebeat-7.5.0-darwin-x86_64.tar.gz

$ >cd filebeat-7.5.0-darwin-x86_64
$ >ls -1
LICENSE.txt
NOTICE.txt
README.md
fields.yml
filebeat*
filebeat.reference.yml
filebeat.yml
kibana/
module/
modules.d/

16
 Steps
Edit the configuration

• Download and install Filebeat

• Edit the configuration
• Enable and configure the system
module
• Start Filebeat
• Check out the dashboard!

17
 Configuration
Cloud aware - using superuser

18
 Edit the configuration
Copy the snippet, paste in the password

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.

cloud.id: "Sandbox:dXMtY2VudHJ..."
cloud.auth: "elastic:long-random-password" # because we are using Elastic Cloud

output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"] ← If we were not using Elastic Cloud
#username: "elastic" ←
#password: "long-random-password" ←

-UU-:----F1 filebeat.yml (YAML) .

19
 Steps
Set up the system module

• Download and install Filebeat

• Edit the configuration
• Enable and configure the system
module
• Start Filebeat
• Check out the dashboard!

20
 Enable the system module
Again, just copy and paste the snippet

$ >./filebeat modules enable system

21
 Enable the system module
Again, just copy and paste the snippet

$ >./filebeat modules enable system

Enabled system

22
 Enable the system module
Check your work

$ >./filebeat modules enable system

Enabled system

# Can also verify

23
 Enable the system module
Check your work

$ >./filebeat modules enable system

Enabled system

# Can also verify

$ >./filebeat modules list

24
 Enable the system module
All good

$ >./filebeat modules enable system

Enabled system

# Can also verify

$ >./filebeat modules list

Enabled:
system

Disabled:
apache
auditd
aws
azure
(...)

25
 Steps
Start Filebeat

• Download and install Filebeat

• Edit the configuration
• Enable and configure the system
module
• Start Filebeat
• Check out the dashboard!

26
 And start it up!
Startup steps
$ >

27
 First run the setup process
Setup preps dashboards and indices
$ >./filebeat setup

28
 First run the setup process
Setup preps dashboards and indices
$ >./filebeat setup
Index setup finished.

29
 First run the setup process
Setup preps dashboards and indices
$ >./filebeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)

30
 First run the setup process
Setup preps dashboards and indices
$ >./filebeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded machine learning job configurations
Loaded Ingest pipelines

31
 Finally, start it!
-e tells it to send messages to console
$ >./filebeat -e

32
 Finally, start it!
-e tells it to send messages to console
$ >./filebeat -e

2019-12-09T18:02:42.500Z INFO instance/beat.go:610Home path:

[/home/user/logs-demo/filebeat-7.5.0-linux-x86_64] Config path:
[/home/user/logs-demo/filebeat-7.5.0-linux-x86_64] Data path:
[/home/user/logs-demo/filebeat-7.5.0-linux-x86_64/data] Logs path:
[/home/user/logs-demo/filebeat-7.5.0-linux-x86_64/logs]
2019-12-09T18:02:42.501Z INFO instance/beat.go:618Beat ID: 04e276d0-79bd-40e3-9c83-3cdc4a64f791
2019-12-09T18:02:42.513Z INFO add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata:
hosting provider type detected as gcp,
metadata={"availability_zone":"us-east1-b","instance":{"id":"8271592631829869565","name":"user-smi
th-build"},"machine":{"type":"n1-standard-8"},"project":{"id":"elastic-product-marketing"},"provid
er":"gcp"}
2019-12-09T18:02:42.564Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully
installed
(...)

33
 Essential needs for log analytics
Recall the earlier list

• Easy setup for a variety of sources

• Correlating and cross referencing
• Searching, filtering, and highlighting
• Visualize
• Anomaly detection and alerting
• Flexible retention

34
 Needs for log analytics
Easy setup for variety of log sources

35
 Needs for log analytics
Correlating and cross referencing

36
 Needs for log analytics
Searching, filtering, and highlighting

37
 Needs for log analytics
Visualize

38
 Needs for log analytics
Visualize

39
 Needs for log analytics
Visualize

40
 Anomaly detection and alerting
Can't stare at the screen all day

41
 Needs for log analytics
Flexible retention

42
 Needs for log analytics
Anomaly detection and alerting

43
 Essential needs for log analytics
From the earlier list

✓ Easy setup for a variety of sources

✓ Correlating and cross referencing
✓ Searching, filtering, and highlighting
✓ Visualize
✓ Anomaly detection and alerting
✓ Flexible retention

44
 Agenda
Beyond logging: Observability

1 Challenges with log analytics

2 Sending logs to Elasticsearch

3 Beyond logging: Observability

4 Leveraging Elastic security

45
 You can add metrics in the same manner
Select your integration

46
 Many integrations
For example, system metrics

47
 Metrics
Visualizing metrics

t e m
s
S board
y
as h
D

48
 Metrics
Visualizing metrics

t e m
s
S board
y
as h
D

49
 Metrics
Exploring metrics

i cs
e t r
M lorer
E x p

50
 Metrics
Inventory view with multiple perspectives

51
 Integrated Experience
Observability with one datastore

h ow n
S atio
e g r n
t
in igat i o
nav

52
 Setting up APM
Instructions in Kibana

53
 Application Performance Monitoring
Distributed Tracing

i ew
a l lV
er f
a t
W

54
 Uptime Monitoring
Service availability

t a
Jus shot
e en
scr

55
 Uptime Monitoring
Service availability

t a
Jus shot
e en
scr

56
 Uptime Monitoring
Integrated experience

t a
Jus shot
e en
scr

57
 Integrated Experience
Observability with one datastore

0 2 0
w 2 d
h o oa r
S hb
das

58
 Integrated Experience
Observability with one datastore

0 2 0
w 2 d
h o oa r
S hb
das

59
 Integrated Experience
Observability with one datastore

0 2 0
w 2
o
S anvas
h
C

60
 Agenda
Securing your Beats

1 Challenges with log analytics

2 Sending logs to Elasticsearch

3 Beyond logging: Observability

4 Leveraging Elastic security

61
 Recall the Filebeat steps
Use parameterized credentials

• Download and install Filebeat

• Edit the configuration
• Enable and configure the system
module
• Start Filebeat

62
 beats_writer Role
Required permissions

• Cluster Permissions:
‒ monitor
‒ read_ilm
‒ manage_index_templates
‒ manage_pipeline

• Index Privileges (*beat-*)

‒ create_index
‒ index
‒ view_index_metadata

https://www.elastic.co/guide/en/beats/filebeat/current/feature-roles.html
63
 Corresponding User
Tying roles to users

• Give the user the corresponding

roles
• Create a secure password
• beats-writer gets the writer
role we created, plus the shipped
beats_system role

https://www.elastic.co/guide/en/beats/filebeat/current/feature-roles.html
64
 Set up the keystore
Hiding credentials for beats-writer
$ >./filebeat keystore
Manage secrets keystore
• Command: filebeat keystore
Usage:
filebeat keystore [command] • Create the keystore
Available Commands: • filebeat keystore add:
add Add secret
create Create keystore ‒ BEATS_WRITER_USER
list List keystore
remove Remove secret ‒ BEATS_WRITER_PASSWORD
• Access keys via ${KEY_NAME}

65
 Previous Configuration
Had the user & password hardcoded
File Edit Options Buffers Tools Help

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.

cloud.id: "Sandbox:dXMtY2VudHJ..."
cloud.auth: "elastic:long-random-password" # because we are using Elastic Cloud

-UU-:----F1 filebeat.yml (YAML) .

66
 Parameterize the user
Had the user & password hardcoded
File Edit Options Buffers Tools Help

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.

cloud.id: "Sandbox:dXMtY2VudHJ..."
cloud.auth: "${BEATS_WRITER_USER}:long-random-password" # because we are using Elastic Cloud

-UU-:----F1 filebeat.yml (YAML) .

67
 And the password
No more plain text!
File Edit Options Buffers Tools Help

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.

cloud.id: "Sandbox:dXMtY2VudHJ..."
cloud.auth: "${BEATS_WRITER_USER}:${BEATS_WRITER_PASSWORD}" # because we are using Elastic Cloud

-UU-:----F1 filebeat.yml (YAML) .

68
 Starts the same way
Automatically picks up the keystore
$ >./filebeat -e

69
 Finally, start it!
assumes that you've run setup
$ >./filebeat -e

2019-12-09T18:02:42.500Z INFO instance/beat.go:610Home path:

[/home/user/logs-demo/filebeat-7.5.0-linux-x86_64] Config path:
[/home/user/logs-demo/filebeat-7.5.0-linux-x86_64] Data path:
[/home/user/logs-demo/filebeat-7.5.0-linux-x86_64/data] Logs path:
[/home/user/logs-demo/filebeat-7.5.0-linux-x86_64/logs]
2019-12-09T18:02:42.501Z INFO instance/beat.go:618Beat ID: 04e276d0-79bd-40e3-9c83-3cdc4a64f791
2019-12-09T18:02:42.513Z INFO add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata:
hosting provider type detected as gcp,
metadata={"availability_zone":"us-east1-b","instance":{"id":"8271592631829869565","name":"user-smi
th-build"},"machine":{"type":"n1-standard-8"},"project":{"id":"elastic-product-marketing"},"provid
er":"gcp"}
2019-12-09T18:02:42.564Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully
installed
(...)

70
 Continuing your Journey
Where to find more information

• Spin up a cluster
‒ Hosted: cloud.elastic.co
‒ Self managed - elastic.co/downloads

• Explore live examples @ elastic.co/demos

• Watch webinars @ elastic.co/videos
• Chat with us @ Forums : https://discuss.elastic.co/
• Go deeper with documentation @ elastic.co/guide
• Sign up for training @ elastic.co/training
• Attend a local meetup or Elastic{ON}
71
 Q&A

Thank you!

72