Log Management and PHP Debugging

Jump to Main Article

Best Practices for Logging in a LAMP stack application

1. Time Synchronization – Time is critical for IT data analysis. Knowing when something happened is critical; a slight time skew can give a very different perspective on a series of events than what you’d expect. Therefore, any custom logs should include a well formatted time string. Also, it’s critical to use NTP to have a common view of time across all components of the application.
2. Common Transaction IDs – As data moves between different modules of the application, transferring a constant unique transaction ID makes it easier to track the flow of data and operations through a multi-tier application. In the absence of one common ID, it is useful to have some external mechanism that can link together locally unique transaction IDs.
3. Signpost – Analyzing the log data is only as useful as the relevance of the content to analyze. Thus, it is important to not only log data that is associated with errors, but also to signpost about progress through the application. While it does not make sense to overwhelm the system and log every operation, it is critical to log key transitions between modules or components.

Imagine a large distributed application built of multiple disparate software components. Further, imagine that many of those components are black boxes into which you have little visibility, either because they come from an external source, or, even worse, because they come from another division in the same company. Now, imagine that your software is written in a language that cannot detect basic errors during compile and linking time… because there is no compilation or linking. Finally, imagine that you’ve frantically written that software without thought of debugging, security, or performance management because you need to get something done quickly. If that sounds like a terrifying nightmare, check out your web-based application. No, you’re not dreaming. You’ve simply got a large distributed web application built with PHP, likely using the LAMP (Linux, Apache, MySQL, and PHP) stack. You’re not alone, and there are tools to help you manage debugging, security, and performance.

Successfully deploying a LAMP-stack application presents multiple challenges. In any application, errors and anomalies tend to emerge at the “glue” or “interface” layers. LAMP begins with distinct OS (Linux), Web server (Apache), and Database (MySQL) components even before a developer adds his own PHP modules. While each of the base components is robust, problems can arise between them. For example, if MySQL is consuming excessive resources, that may trigger anomalous behavior by Linux or Apache. Thus, one needs to be able to not only look at each part of the stack individually, but also the interactions and interdependencies among them to truly understand the environment. Furthermore, any complex application will span across multiple systems. Failures or errors on any given system will affect other parts of the application. Therefore, the interactions will span across multiple physical and virtual systems. Finally, most applications will require multiple custom software components to work together. If any one part of the application flow breaks down, the entire application will likely fail. Unfortunately, it can be extremely difficult to track where any specific transaction failed, slowed down, or returned erroneous results. The result is almost always finger pointing between the developers who wrote the modules. One needs a consolidated, centralized picture to gain insight into LAMP-based applications’ errors, security holes, and performance challenges.

The PHP component of LAMP-stack applications creates additional debug, security, and performance challenges. Like any programming language, PHP’s strengths are also its weaknesses. PHP is ideal for rapidly creating and modifying applications. Unfortunately, as the application matures, PHP is extremely difficult to maintain and debug. For example, since PHP is a dynamic language, there are no facilities to detect compile-time and linker-time errors (read: basic programming errors) prior to execution. Instead, those errors are found by somebody running the application and executing those lines of code. This challenge is exacerbated by the rapidity of development and the frequency of using code conversion tools (e.g. Flash to PHP) which conduct even fewer checks for errors and bugs. Thus, the traditional baseline error-catching tools are no longer available to the PHP application developer. As a result, a PHP application must be treated differently than traditional enterprise applications:

• Your users are often your QA department, so you need to aggressively monitor their results to rapidly identify bugs for resolution
• Since your users are your QA, you are unlikely to get them to tell you what they were doing, so you need to find other ways to get the data necessary for debugging
• Since your language doesn’t have automatic input validation, you need to worry about security breaches from users entering inappropriate values into input fields (i.e. you ask for a “birthdate” and I try to enter the Gettysburg Address and overwhelm the system)

When you face such massive challenges in delivering a LAMP-stack distributed application, it’s easy to imagine that the approach might be – “I’m going to hope nothing goes wrong.” Fortunately, LogLogic delivers a better approach. By leveraging the IT data that the LAMP-stack application creates, a company can truly gain 360 Insight into their LAMP-stack application. LAMP-stack applications generate enough log data to dramatically reduce the time-to-resolution for critical errors, identify security and performance exposures, and monitor the overall health of the application infrastructure.

LogLogic reduces the time it takes to resolve critical errors. First, it’s important to triage and understand what errors are critical. Each component of the LAMP-stack will generate an error message when something goes wrong. While you can gather an ocean of data from the application, LogLogic makes it easy to identify the messages that identify errors. Furthermore, you can then categorize the type of errors, so you can see which errors are being encountered most frequently, the errors that the largest number of users are hitting, or the most severe bugs. Second, once you’ve prioritized the bugs, then you must resolve the bugs quickly and efficiently. LogLogic delivers what every developer and IT person wants – information. What were the steps that led to the error? What else was happening in the environment at the time of the error? All of that information can be found in the logs. To improve the efficiency of that experience, there are some simple “logging best practices” to follow (See inset). Finally, LogLogic helps validate that the new code change works as expected. There are always 2 concerns when fixing a bug – “Did I fix the bug?” and “Did I break something else?” With LogLogic, development and QA can watch the logs and see that the tests have executed the repaired code and that everything is behaving as expected. Furthermore, when running a battery of regression tests, LogLogic can show that the general results of the tests match prior results (i.e. no regressions). From triage to investigation to validation, LogLogic’s 360 Insight reduces the “time to fix” for resolving errors in PHP LAMP stack applications.

LogLogic can help detect and locate performance and security exposures in a PHP-based LAMP stack application. LogLogic offers three levels of solutions for performance and security emergencies. First, with a LAMP-stack application, the components are vocal about running low on resources (low on CPU, out of memory, etc.). Many companies adopt simple standards – “If I get more than ‘x’ ‘out of memory’ conditions in a 5 minute period, throw an alert.” Others work more from a baseline – “If I get x% more ‘CPU above 90%’ messages than normal, throw an alert.” Still others work on the basis of – “I’ve never seen this error condition before, throw an alert.” The same approach can work with security breaches. Usually a compromised component or system will either generate significantly more or significantly fewer log messages than normal. Companies will configure an alert to be sent in either case. Second, if there are particular security attacks that are of concern, customers can configure simple and advanced (correlated) alerts. In general, these alerts will detect standard internal and external security threats against your infrastructure and application. While general trending can help detect broad breaches that are often associated with PHP, it is important to detect the more traditional attacks, as well. Third, once LogLogic has helped detect a security or performance anomaly, users can quickly drill down to locate the area from which the threat is coming. LogLogic can identify the server, module, or infrastructure that has anomalous behavior. After all, if an application is under attack or overloaded, taking quick action is the #1 priority. After the emergency has abated, forensic analysis and error resolution can proceed.

 LogLogic can also monitor the overall health of the application infrastructure. The largest challenge in monitoring a distributed LAMP stack application is knowing what to monitor. With rapid deployment and scaling, the application can span across a variety of infrastructures. As more companies leverage cloud resources to help scale their applications, the challenge has become more significant. With LogLogic, tracking the components can be significantly easier. With each new server (virtual or physical), application, or module, the administrator directs the logs to the central log server. LogLogic automatically identifies the type of component (e.g. Linux server, firewall, application X), so that the central administrators have an accurate view of their application infrastructure. Once a company knows where everything is, they can begin to understand the dependencies between application components, underlying infrastructure, and locations. If the application developers follow the “logging best practices,” the persistent IDs that generate signposts through the application flow can highlight both the software and hardware flow of a complex application. Finally, LogLogic makes it easy to understand trends in the application. Perhaps certain components have had elevated numbers of bugs or issues which may highlight an area of code that needs additional oversight. This can also apply to parts of the infrastructure (especially interesting when dealing with cloud providers). By retaining all of the historical information about what has happened on the system, understanding the relationship between components, and knowing where everything is, companies can gain significantly more insight into their LAMP stack application than ever before.

LAMP stack applications come fraught with challenges. With multiple independent components, rapidly written PHP modules, and distributed infrastructure, it may seem that a company’s only hope is to hope that nothing goes wrong. LogLogic delivers 360 Insight for LAMP stack applications. From detecting attacks, errors, and performance challenges to resolving them, to understanding the broader picture of the application deployment, LogLogic brings order to the chaos of web development.