Monitoring and Logging are both essential, together. They are like two sides of the same coin: health-check.
Initially, the dev community had monolithic environments so checking faults was an easy task. Even today, one can check the output(s) of an application with Kubernetes CLI but things get complicated when there are more than a few containers involved for an application/service.
The need of an hour is to get the logs from the VMs deployed on cloud, from Kubernetes and of course, from the deployed applications.
A perfect illustration of deploying the above use-case is done here; well-explained following architecture of fluent-d and fluent -bit.
There is a video I would recommend which teaches how to setup fluentd for Kubernetes, although it doesn’t cover up security.
Although, one thing that I have realized through multiple internships is that when health-checks are concerned, it is always better to buy rather than to build. LogDNA costs you a meagre amount and saves a lot of dev effort when you need logs for ES on Kubernetes.
Logstash and its web interface tool Kibana with ES are termed as ESK and they are used for monitoring KS as well. This tutorial explains why and how one should use(setup) them using falco and fluentd.
The best way to do this, in a generic sense, is to use Heapster(inject it). This web-page gives a brief description to get started on “Why Heapster?”. One needs to measure ES performance, keep an eye on all the logs in a minimal effort. Heapster with Kibana (for visualization) is the answer.
By this article, I wanted to convey that even though being a Data Engineer, I am doing the infra setup for logging; main motive being, I know the loose points of my infra and my data services(microservices). The aforementioned resources were referred by me while developing Event Sourcing mechanism at SocialCops. We are yet to finalize the stack for health-checkups but we know our workaround since all sort of built and available tools are already tried.