Web Administraticn Scripts - Redux

Dave explains how to add contextual memory to a script so it  can keep track of trends in its environment. Then he explores  why that's harder if the script stops and starts over time.   
It's been months,
 and I'm still  dealing with a DDOS (distributed  denial of service) attack on my server-  an attack that I can see is coming from  China, but there's not really much I  can do about it other than try to tweak  firewall settings and so on.  ln my last article, I started  describing analyzing Apache log  files and system settings to try to  create some scripts that can watch  for DDQS attacks and flag them  before that inevitable e-mail from a  customer about the site being down  or incredibly slow.  Honestly, it's pretty disheartening  dealing with these anonymous attacks,  but that's another column. If you've had  experience with this sort of long-term  problem, I'd be interested in hearing
about how you finally resolved it.  The first script I worked on simply  tracks how many processes are  running on the server. |t's just a small  step from monitoring the output  of the uptime program, which, of  course, could be done as easily as:       

oad=“$(uptime | cut -d\ -f14 | cut -d. -f1)"
1f [ $1oad -gt 4 ] : then
echo "Uptime greater than threshold. what's going on?
uptime

That's marginally interesting, but  let's go back to analyzing the number  of processes running, as that's a  better indication that there might be  a DDOS happening. Last time, I used  a loop of ps ; sleep and kept track       

Once you've run this script for a few hours,  you'II have a pretty good idea of typical traffic  load on your server, which is critical for you to  be able to detect out-of-pattern variations.

of the min/max values. After running  for a while, the script's output was  something like this:  Current ps count=90: m1'n=76 ma×=150. tal1y=70 and average=107  lnterpret this as after running for  90 cycles, the minimum number of  Apache (httpd) processes running at  any given time was 76, max Was 150  and average was 107.  Once you've run this script for a few  hours, you'll have a pretty good idea  of typical traffic load on your server,  which is critical for you to be able to  detect out-of-pattern variations. lf  my average Apache process count is  107 and the server has 917 processes,  there's a problem. On the other hand,  if the average load is 325 processes,  600 isn't too far out of band and  could be representative of a spike in  traffic rather than the beginnings of  an actual attack.  lwrapped up last month's article  by showing a script suitable for  running from cron that would look  for abnormal traffic spikes. But it had  some problems, as l highlighted at the  end of the column, my nerd version  of a cliffhanger-no murder unsolved,  no car in the driveway, no police  sirens, just some code that needs  improvement. Hey, work with me here!  The core problem with the script  was really a lack of context. That is,  it's not the very first time that it sees  abnormally high process counts that  lwant to be notified, but the third or  fourth time in a row. And then once  it has notified me, I want the script  to be smart enough not to notify me  again unless things settle down and  then jump up again.  Context.  With some scripts, this sort of thing  can be quite tricky, requiring temporary  files and semaphores to ensure that  the script doesn't step on another  version of itself reading/Writing the  file simultaneously. lt's doable, but you  really have to think about worst-case  situations, temporarily blocked l/O  channels and so on.

But, what happens when there's then an  iteration where the count isn't greater than  the max threshold?

Fortunately, that's not the situation  here. In fact, you can accomplish the  addition of context by adding a couple  state variables. Let's start by pulling  the monitoring script back in:  #!/bin/sh  # DDOS — keep an eye on process count to  # detect a blossoming DDOS attack  pattern="httpd‘  max=200 # avoid false positives  admin="d1taylor@gmail.com"  count="$(ps -C $pattern | wc -l)“  if [ $count -gt $max ] ; then  echo "Warning: DDOS in process? Current httpd  hcount = $count" | sendmail  $admin  fi  exit 0  Let's use a "repeated" variable  and set it to send a notification after  the fourth occurrence of too many  processes. That's just a matter of  changing the conditional statement:  if [ Scount -gt $max ] ; then  repeated=$(( $repeated + 1 ))  if [ $repeated -eq 4 ] ; then  echo "warning: DDOS in process? Current httpd count  = $count" | sendmail $admin  fi  fi  Not too difficult. But, what happens  when there's then an iteration where  the count isn't greater than the max  threshold? That's also easily handled  I  I  if you re willing to keep redundantly  setting repeated to zero. The outer  "fl" simply changes to:  else  repeated=0  fi  These additions produce both of  the desired updates actually, because  repeated ensures that it won't notify  of a problem until it's happened  a few times, and the conditional  $repeated -eq 4 rather than, say,  $repeated —gt 4, also means that  if it's slammed for 15 iterations, you'll  still see only one e-mail message.  Finally, visualize a sequence where  you get a spike for a while, then it  quiets down. Then another spike       

hits for a few seconds, then it quiets  down for an hour or two, and then  it spikes again. That scenario will  work as desired too, sending e-mail  twice, once for each sustained spike,  but not for the one-iteration spike in  the middle (because in that instance,  repeated never gets beyond 1).  If you've been looking closely at  the script, you'll have noticed that  what appears at first glance to be an  innocuous "echo" statement is in fact  producing an e-mail message to the  administrator. It's the second half of the  statement that's doing the heavy lifting:  echo "Warning: DDOS in process? Current httpd count  = $count" I sendmail $adm1'n  I'm old-school, so I go straight to  the |\/ITA (mail transport agent), but  in reality, a better way to do this,  an approach that would have the  e-mail include a subject line, is to do  something more like this:  echo | mail -s "DDOS Warning" $admin  Because sendmail is designed to be  used behind the scenes, it lacks the  refinement of things like a -s flag to  specify the subject of the message.  In fact, you can make this even a  bit more elegant by turning the entire  command into a variable:  sendmsg="mal'1 -s 'DDOS Warning‘ $admfn"  Then you easily can tack it onto the  end of the echo statement:  echo I $sendmsg  What's the biggest problem with  this approach? That if there are any  errors, the pipe is only for stdout, not  stderr, and those errors will be lost.  If you were going to run the  script every few minutes from  cron, you could use its capability  of e-mailing stdout + stderr, but  that would involve a more complex  contextual tracking solution (as  discussed earlier).  Well, I'm off to install this script  on my server and see how it does for  a while. And you? Your assignment,  dear reader, is to shoot in some  proposed topics for me to explore in  my column. What scripts would you  like to see created, doing what?!