Kubernetes Addons
Kubernetes Addons
- Most of the Kubernetes cluster components like api-server, kube-scheduler, Etcd, kube
proxy, etc,. run as containers.
- However, the kubelet component runs as a native systemd service.
- We can find the kubernetes pod logs in the following directories of every worker node.
o /var/log/containers: All the container logs are present in a single location.
o /var/log/pods/: Under this location, the container logs are organized in separate pod
folders
/var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/. Each pod
folder contains the individual container folder and its respective log file.
- If underlying container engine is docker then, we will also find logs in
/var/lib/docker/containers folder.
Types of logs:
- Application logs: Logs from user deployed applications. Application logs help in understanding
what is happening inside the application.
- Kubernetes Cluster components: Logs from api-server, kube-scheduler, etcd, kube-proxy, etc.
These logs help you troubleshoot Kubernetes cluster issues.
- Kubernetes Audit logs: All logs related to API activity recorded by the API server. Primarily used
for investigating suspicious API activity.
Logging Patterns:
- Node Level Logging Agent: In this method, a node-level login agent (Eg: Fluentd) reads the log
file created using container STDOUT and STDERR streams and then sends it to a logging backend
like Elasticsearch. This is a commonly used logging pattern and works pretty well without any
overhead.
- Streaming sidecar container: This streaming sidecar method is useful when the application
cannot write logs to the STDOUT and STDERR streams directly. So, the application container
writes all the logs to a file within the container. Then a sidecar container reads from that log file
and streams it to STDOUT and STDERR. Rest is the same as the first method.
1
- Sidecar Logging Agent: In this method, the logs don’t get streamed to STDOUT and STDERR.
Instead, a sidecar container with a logging agent would be running along with the application
container. Then, the logging agent would directly stream the logs to the logging backend.
Elastic Search:
2
Monitoring
https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/
http://10.0.0.219:30200 – kibana
http://10.0.0.27:30300 – Prometheus
http://10.0.0.27:32000 – Grafana