Year after year Apache continues to be the most popular and widely used web server worldwide. However, although it is well proven in its industry, Apache is not free of problems. In this article, I will be outlining some tips which can be used when analyzing and remediating the most common issues encountered in Apache.
Verify your Apache HTTP Server configuration
Apache HTTP Server issues may also be a result of a misconfigured Apache httpd.conf configuration file. Going over the whole configuration file searching for typos may be a cumbersome task, but thankfully Apache provides a way to scan your httpd.conf file for any syntax errors. This can be done by using the configtest tool from the apachectl program. For Apache on Unix systems you would need to execute apachectl –configtest from the command line and for Apache on Windows systems you would need to first navigate to the Apache ‘bin’ directory from the command line and then execute httpd.exe –t.
Figure 1 – Apache HTTP Server found a syntax error on the ServerTokens directive
In the above example this tool found a syntax error on line 560 of the httpd.conf file. It appears that the ServerTokens directive was misspelled with an extra ‘t’ present at the beginning of the ‘Tokens’ part. Other issues may remain after this is solved so it’s recommended that you rerun the configtest tool after correcting the issue and restarting Apache.
Use the latest version of Apache HTTP Server
Old versions of Apache HTTP Server may contain bugs which could be causing the issues you are experiencing. Furthermore, using an outdated version of Apache may also lead to security vulnerabilities. A list of vulnerabilities and bugs in previous versions of Apache can be found here and the latest version of Apache can be downloaded from here.
Apache HTTP Server logs
First and foremost, the Apache HTTP Server error log should be analysed as this provides detailed information about any errors that have occurred on your web server. By default errors are logged in the error_log file located in the logs directory inside the Apache root installation. Logging levels can also be adjusted from the Apache httpd.conf configuration file in order to specify which type of errors are recorded. Information on the eight different log levels can be found here. Adjusting the log level to a higher one may present you with more information on the issue, but will also make it more difficult to find what you are after. Apart from error logs, Apache also provides access logs, which record all requests processed by the server. These logs may also give additional explanations of what could have caused the issue and can also complement information found in the error logs.
Use the mod_log_forensic module
The mod_log_forensic module is used to provide forensic logging of client requests. This includes logging requests before and after they have been processed, where the same requests are referenced with the same ID. Therefore, any issues caused by specific requests can be easily identified. This can help analyse which requests may be causing your web server to stall or crash. To enable this module, you need to setup the following lines in your Apache httpd.conf configuration file:
LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule unique_id_module modules/mod_unique_id.so
ForensicLog logs/forensic_log
Also, the check_forensic Bash script can be used in combination with the mod_log_forensic module to list any incomplete requests found in the forensic log. An example of how the check_forensic tool can be used is included below:
check_forensic <log_file>
Use the mod_whatkilledus module
When things go really bad, and Apache server crashes, The mod_whatkilledus module can be used to log detailed technical information about the crash together with the original client request which caused it. Additionally, if the mod_backtrace module is enabled, a backtrace showing the point of failure would be included, which is useful to annotate error logs with backtraces after certain conditions are met. For Unix systems, these modules will only function if the --enable-exception-hook argument is enabled on the httpd build. On the other hand, there are no specific requirements for Windows systems. Setup and configuration instructions for mod_whatkilledus and mod_backtrace can be found from here.
An example of a crash log in the mod_whatkilledus logs
Check third-party modules
It’s possible that third-party modules could be causing the issues that you are encountering with your Apache HTTP Server install. Therefore, you should disable all third party modules and check if the issue can still be reproduced. If disabling these modules solved the issue, you can then try to re-enable each module one by one in order to identify by elimination which of the modules was causing the issue.
Run Apache HTTP Server as a single process and use debugging tools
A typical Apache HTTP Server installation runs with several processes. However, to simplify troubleshooting it’s best to run Apache as a single process. This can be done by using the X option when starting Apache. The example below will start Apache in a single process mode, where Apache will not fork any new children or detach from the terminal.
$ httpd –X
Following this all traffic and communications will pass through one process, allowing troubleshooting procedures to be used on a single process instead of multiple processes. You would then be able to run the Apache httpd under a debugger, obtain a backtrace of a crash, and also force the server to perform a core dump. The official Apache developer resources found here explain how to do all this in more detail for Apache servers running on both Windows and Unix systems.
Issues with script execution
Dynamic content is generally served through scripts which are executed by Apache HTTP Server through the mod_cgi module. This module includes its own logging mechanism for recording any errors encountered during the execution of these scripts. After enabling the ScriptLog directive, mod_cgi will record the output of any scripts that did not execute as intended, including the server response code, the request that was received, and any response that was issued to the client. To enable this, you need to modify the httpd.conf configuration file and specify the ScriptLog directive and the location where the logs will be saved, as indicated below.
ScriptLog logs/cgi_log
Apache community resources
Apache is an open-source project with large community support and thus there are a number of resources that one can use for help on Apache problems. Most commonly, issues would already have been documented across various forums or websites and therefore a simple Google search may lead to a few answers or suggestions. Other resources that you can refer to include:
- The official Apache HTTP Server documentation available here.
- Official Apache HTTP Server user support and discussions described here.
- stackoverflow questions tagged with the Apache keyword available here.
Test your web applications (and their corresponding web server configurations) by running an automated scan for security vulnerabilities with an Apache Security Scanner like Acunetix
