Log²: a cost-aware logging mechanism for performance diagnosis

Logging has been a common practice for monitoring and diagnosing performance issues. However, logging comes at a cost, especially for large-scale online service systems. First, the overhead incurred by intensive logging is non-negligible. Second, it is costly to diagnose a performance issue if there are a tremendous amount of redundant logs. Therefore, we believe that it is important to limit the overhead incurred by logging, without sacrificing the logging effectiveness. In this paper we propose <i>Log</i>², a cost-aware logging mechanism. Given a “budget” (defined as the maximum volume of logs allowed to be output in a time interval), <i>Log</i>² makes the “whether to log” decision through a two-phase filtering mechanism. In the first phase, a large number of irrelevant logs are discarded efficiently. In the second phase, useful logs are cached and output while complying with logging budget. In this way, <i>Log</i>² keeps the useful logs and discards the less useful ones. We have implemented <i>Log</i>² and evaluated it on an open source system as well as a real-world online service system from Microsoft. The experimental results show that <i>Log</i>² can control logging overhead while preserving logging effectiveness.