My take on Logging Levels

A long time ago, while developing a realtime server for Verint, the system's architect devised a great document called 'logging policy'.

It explained how a log message should look like, which information they should contain, and elaborated the rules for each log message level: For example: "There is no such thing as a 'Good' warning. Do NOT put 'Server is now up' messages in warning level".

Back then, the common practice ranged more than five levels, from critical to verbose.

At least at the beginning of development, I found that, though it's nice to do so, right now the most effective number of logging levels is 2. Debug mode and production mode.
Production mode should contain high level log events (Flow events, processing metrics) and errors, debug - the rest.

Logging is one of the strongest maintainability tools for your server. However, keep it simple, or you'll drown in misleading information.

DO NOT UPGRADE!

Too often, when an application fails on memory or performs poorly, the immediate solution would be upgrading the machine.
Add more cores, more memory - And think that this would clear the problem from under the rug.

This strategy actually causes even more damage - As you will have to deal with the real issue (YOUR CODE...) later, when the system has more data, more angry clients, and more code that breaks.

If your application consumes too much CPU, you should profile it and solve the problem.
If your application leaks memory, you should find out where.

Taking the lazy approach would cause you credibility issues once you will really need that upgrade.

Performance - Where it counts.

A good report does magic with large data.
It is supposed to take millions of records, and summarise them into a visual entity that transforms into something meaningful for the user.

Recently, in one of my sessions, a client wanted me to focus on improving performance for a specific report in his system.
Report was taking 5 minutes to generate, clogged the system's database, and was comprised of one of the largest SQL statements I've seen in my life.
Instead of starting to spend hours and hours trying to optimise and dig deep, something that would cost my client a lot of money, I asked him a simple question:

- Who produces this report, and how often does it happen?

Turns out it was a weekly (Even monthly in some customers) report, printed and handed out to a manager.
Assigning a periodic process running on a backup database, and automatically sending the result via mail to the manager saved a lot of money and headache to my client.

Performance is important, but you should always focus on improving it where it makes a real difference.

Do it where it counts!