PHP Log Searching Tutorial

This PHP tutorial is for intermediate users and will show you how to use directories for parsing files located there. It’s useful if you want a log searching for your server. You can make a GUI, add a cronjob for it and you’re done. When a string is found it sends an e-mail to address provided using SwiftMailer. I will make a SwiftMailer tutorial at a later time.

Here goes the PHP code:

<?php function GetMail() { //make an SQL statement for getting the e-mail from the database. return $to; } function GetNeedle() { //make an SQL statement for getting the keyword for the database return $needle; } $to = GetMail(); $needle = GetNeedle(); $dirpath = '/var/www/'; //value not found in db $logpath = '/var/www/test/search.log'; //needs write permisions for vmin log file explained in readme.txt $dir = opendir($dirpath); $log = file($logpath); //search.log will be in an array $log_write = fopen($logpath,"w"); //we also need to write our current search in vmin.log $from = "andreicorb@gmail.com"; //data for swiftmailer. Here I am using smtp. $passwd = "password"; function SendMail($to, $from, $needle, $file, $i, $passwd) { //Create the Transport $transport = Swift_SmtpTransport::newInstance('smtp.gmail.com') ->setPort(465)<br /> ->setEncryption(‘ssl’)<br /> ->setUsername(“$from”)<br /> ->setPassword(“$passwd”)<br /> ;</p> <p> //Create the Mailer using your created Transport<br /> $mailer = Swift_Mailer::newInstance($transport);</p> <p> //Create a message<br /> $message = Swift_Message::newInstance(“$file”)<br /> ->setFrom(array(“$from” => “Search Log Alert “))<br /> ->setTo(array(“$to” => ‘Log Searcher’))<br /> ->setBody(“$needle found at line $i in file $file”)<br /> ;<br /> $result = $mailer->send($message);</p> <p>}</p> <p>function CheckNeedle($log, $log_write, $needle) {<br /> $needle_changed = 0;<br /> if (trim($log[0]) !== $needle) $needle_changed = 1;<br /> fwrite($log_write,”$needle\n”);<br /> return $needle_changed;<br /> }</p> <p>$needle_changed = CheckNeedle($log, $log_write, $needle);<br /> //read the nr. of lines parsed in last search . 0 if new path or files are found.<br /> function GetLines($needle_changed, $log, $file) {<br /> $last_line = 0;<br /> if ($needle_changed == 0) {<br /> for($i = 0;$i < count($log); $i++){ if (strcmp($file,trim($log[$i])) == 0) {$last_line = (int)trim($log[$i+1]);} } } return $last_line; }

I will explain the idea in a few words: Open the specified directory, search each file, if the string is found then send the e-mail. Also and the end of each search remeber the last line searched because the log file can be very big (100 gb) so we don’t want to search from the the first line of each file again.
Function CheckNeedle checks if string changed from the previous search. So if needle changed then we need to reset our search to 0.
Function GetLines reads the nr. of last line for each file.

Let’s go on:

function SearchNeedle($last_line, $file, $needle, $to, $from, $log_write) {<br /> global $passwd;<br /> $handle = @fopen($file, “r”);<br /> if ($handle) {<br /> $i = 0;<br /> while (($buffer = fgets($handle, 4096)) !== false) {<br /> while ($i < $last_line-1) { $data = fgets($handle,4096); ++$i; } $i++; $pos = strpos($buffer,$needle); if($pos === false) {} else { echo " $needle found at line $i in file $file"; SendMail($to, $from, $needle, $file, $i, $passwd); } } } fwrite($log_write,"$file\n$i\n"); if (!feof($handle)) { echo "Unexpected error"; } fclose($handle); }

Function SearchNeedle goes on each line of the file, starting from the line previously searched and checks if needle is found and sends the e-mail to address provided.

// read given dir and start the searching<br /> function Start($dir, $log, $needle, $needle_changed, $log_write, $to, $from) {</p> <p> while (false !== ($file = readdir($dir)) ) {<br /> if ($file != “.” && $file != “..” && $file != “search.log”) {<br /> $last_line = GetLines($needle_changed, $log, $file);<br /> SearchNeedle($last_line, $file, $needle, $to, $from, $log_write);<br /> }<br /> }<br /> fclose($log_write);<br /> closedir($dir);<br /> }<br /> }</p> <p>Start($dir, $log, $needle, $needle_changed, $log_write, $to, $from);<br /> ?>

Finally I made a Start procedure to open the directory and start SearchNeedle.

As this is for intermediate users I did not explain line by line, instruction by instruction. But I will provide some links for each PHP function used in this tutorial.

strcmp: http://php.net/manual/en/function.strcmp.php
readdir: http://php.net/manual/en/function.readdir.php
strpos: http://php.net/manual/en/function.strpos.php
trim: http://php.net/manual/en/function.trim.php

The script was tested on 2-10 gb files and it’s working good. Also don’t forget to include swiftmailer in your script.
You can download it from here: http://swiftmailer.org/download.
You can use require_once directive to do that.