Currently Thumbor logs can be found either via systemd's journal or under /srv/log/thumbor. We should figure out how to stream logs to logstash and view them in kibana.
Description
Description
| Status | Subtype | Assigned | Task | |
|---|---|---|---|---|
| · · · | ||||
| Resolved | hnowlan | T216815 Upgrade Thumbor to Buster | ||
| Resolved | fgiunchedi | T212946 Stream Thumbor logs to logstash | ||
| · · · |
Event Timeline
Comment Actions
Sure! In terms of figuring out how, since the logs are already present in syslog this should be a pretty straightforward case.
We can approach it by first including ::profile::rsyslog::kafka_shipper in the appropriate thumbor role. This will plumb the hosts with rsyslog kafka dependencies and the rsyslog lookup/output configs for shipping to logstash.
Then, we can update the rsyslog lookup table at modules/profile/files/rsyslog/lookup_table_output.json to direct syslogs with thumbor program names to both kafka (logstash) and local outputs with config lines to the effect of:
{"index" : "thumbor@8801", "value" : "kafka local" },
Since thumbor hosts have something like 40 thumbor instances, each with unique program names, it will need 40 lines in this file. This isn't ideal, but afaik lookup tables are not currently able to match by "startswith" in the same way that the local rsyslog omfile rule is configured. Ran into a similar case with prometheus. Maybe there's a way to work around it, but listing them down would be good enough for now.
@jijiki did you want to take a stab at a patch for this? If not no worries, I'd be happy to do it.
Comment Actions
@fgiunchedi @herron I agree, it is a good opportunity to move it to the new infrastructure:D. Since we will be upgrading Thumbor servers to Stretch soon, we could first do the upgrade along with T212941 and then go ahead and do T212946 and T187765
@herron should I ping you when we get to the logging part? Depending on availability and needs, we can figure out how we will implement it
Comment Actions
Reviving this as part of this Q's OKRs to move services off logstash non-kafka inputs, I'll followup with patches to move to the localhost-udp compatibility endpoint! (See T242609)
Comment Actions
This is complete (i.e. T242609: Move thumbor to the logging pipeline), resolving. Feel free to reopen though!