Web Apps Since 2004.

Is your website boring?  Kick it up a notch.  Encodable apps easily drop into your website to make it modern, fun, and interactive, with features like file uploads, user accounts, paid subscriptions, protected pages, live chat, visitor logging, mailing lists, and more.
All apps include:
• Money-back guarantee
• No monthly fees
• Free tech support
• Easy setup (we can even do it for you!)

Count IP Addresses in Access Log File: BASH One-Liner

# Filed on Dec 17, 2008 by AnthonyDiSante 5 replies

Recently my server was nearly overloaded by a web spider that was severely stupid and/or malfunctioning.  It was making multiple requests every second for totally nonsensical URLs.

When I first noticed my server slowing down, I checked my Apache access.log file.  Normally, when you want to learn about the real people viewing real pages on your site, VisitorLog is the app for that; but when you’re dealing with a crazy web spider, you have to go right to the Apache logs.

Since encodable.com normally gets about 1000 visitors per day anyway, a visual inspection of the logfile did not make it immediately obvious which IP address was making the most requests.  There are lots of hits from my own IP, for example, but not enough to slow the server down.

One quick way to see which IP addresses are most active is to sort by them:

cat access.log |cut -d ' ' -f 1 |sort

The cut command there simply throws away all the output except for the first field on each line, which is the IP address.  Then we sort them.  We can then scroll up through the terminal window and get a quick-and-dirty visual indication of which IP is most prevalent.

But in my case, I had quite a few IPs with several hundred hits, and that’s not enough to cause a problem.  I needed to see which ones were in the thousands, but scrolling up through the terminal output it’s not especially easy to see the difference between say 500 lines and 1000 lines.

I needed an actual count of the number of times each IP address appeared the access log.  I came up with the following BASH one-liner to do it (split onto multiple lines here only for readability):

FILE=/path/to/access.log;
 for ip in `cat $FILE |cut -d ' ' -f 1 |sort |uniq`;
 do { COUNT=`grep ^$ip $FILE |wc -l`;
 if [[ "$COUNT" -gt "500" ]]; then echo "$COUNT:   $ip";
 fi }; done

First it creates a for-loop based on the output of the uniq command, so each iteration of the loop is for a different unique IP from the log.  It then greps the log for that IP and uses "wc -l" to count the lines in the output.  Finally, if the count is greater than 500, it displays the count and the IP, like so:

6975:   124.115.3.33
5648:   124.115.5.169
1514:   66.219.73.236
1451:   74.204.11.20

As you can see, the stupid spider was coming from the 124.115.* IP range.

UPDATE: even easier: the uniq command has a -c argument that does most of this work automatically.  It counts the occurrences of each unique line.  Then a quick sort -n and a tail shows the big ones.  Also, I tend to use "cut" as above, but one of the Dreamhost guys reminded me that awk may be a little more straightforward:

cat /path/to/access.log |awk '{print $1}' |sort
 |uniq -c |sort -n |tail

Comments:

01. Apr 1, 2009 at 09:12am by Dodger:

perl -e ’$ip{(split)[0]}++ while <>; print map "$_ : $ip{$_}\n", sort {$ip{$b} <=> $ip{$a}} keys %ip’ /var/log/apache2/access_log

02. Apr 2, 2009 at 09:11am by AnthonyDiSante:

Close... swap the output columns and pipe to head and that would do it.  Of course, that’s much more unwieldy than the awk/sort/uniq method.

04. Jul 31, 2012 at 06:31pm by ahmet:

i can not use | vertical bar character on the ssh , why?

05. Aug 1, 2012 at 02:47am by AnthonyDiSante:

I’m not sure.  Maybe this will help?

06. Apr 24, 2015 at 12:38am by Shannon:

This was a big help, thanks for the tutorial

Reply to this message here:

Your name
Email (why?)
Website (if you have one)
Subject
search posts:

home | archives ]

Shopping Cart

Your cart is empty.

Client Quotes

The work, the thought and the organization you put into this app is incredible.
– Bruce C.
Thank you VERY much for all of your help.  You've really impressed me.  We have support agreements for other software that costs thousands of dollars / year (just for the support), and most of them aren't as helpful as you have been.
– Keith Y.
FileChucker is helping drive the backend of several high profile entertainment sites for people like Shania Twain and Dolly Parton.  We're also using it to drive backend file uploads for a multi-billion dollar banking institution.  It's a great product.  We've tried other "chucking" upload solutions with progress bars using flash and php, but nothing works as reliably as FileChucker.
– Michael W.
Thanks again for a great product and great support - beyond expectations.
– Greg S.
You've done a wonderful job with FileChucker and UserBase, and they have made a big difference to how our website runs.
– Nicholas H.
Why didn't I just do this from the get-go?  So much easier.  Thanks for your work.  FileChucker makes my work easier.
– Dominic M.
Our members think your software is fantastic...  I would recommend your software and your company to anyone.  Thanks for all your help.  It has been a pleasure dealing with you.
– Tommy A.
Do you know how rare it is to have a "canned" shopping cart that can easily do complex pricing options on a single item?  Basically, they don't exist!  I have looked.  Everywhere!  And the few that might even come close to CornerStore's functionality cost a fortune!
– Tashina P.
I looked all over trying to find a simple cgi script.  I found that FileChucker was by far the best.  If you have issues with your hosting service's php.ini max upload size then this is the way to go.  Looking forward to future enhancements.
– Bob C.
FileChucker is a great drop-in solution for file uploads, and worth every penny of its very reasonable cost.  Encodable's support is excellent to boot.
– Loren A.
I just want to say you guys really stand alone in that you have a quality product and you provide genuine customer service.  It's sad but those qualities are seldom found separately, much less together.  Thanks again for your time and help.
– Alex S.
I just wanted to say that yours is the first product that I've tested so far that hasn't failed on handling uploads.  This is going to work for a print company, so they are handling nothing but large files and all the other solutions I've tried so far have not been reliable.  So far yours has been 100% successful in my tests.
– Kevin H.
Just one word: Fantastic.  10-minute job to plug FileChucker into my app, and it now works a treat.  It's through the hard work by people like yourselves that make my job so much easier.  Congratulations on an outstanding product... Many many thanks.
– Sean F.
FileChucker is working great...  Clients love it.  Vendors love it.  We love it.
– Gerry W.
The amount of customization in the program is incredible.  I was able to integrate it into my existing page layout relatively simply.  I was also able to easily customize the look/feel to match the current site.
– Jason M.
Nice script, it's saving the day on our project.
– Aaron W.
I just installed the demo of your product and got it up and running in no time.  I searched high and low for a decent login script and thank God I found yours.
– Adrian F.