Results 1 to 6 of 6

Thread: How to limit server access?

  1. #1
    Join Date
    Jun 2006
    Location
    Austrian in Taiwan
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    How to limit server access?

    I got a problem with a search engine crawler. It is nice when they come, but now they kill my site: Bing e.g., found my blog and crawles back to day 1, - not with 1, but with xx instances. Server load I could see was 96 !!!!

    I want to limit access to the server so that the load is in a "normal range"

    Is there a way to limit search engines to one instance only?
    Is there another way to limit acces when it is geting critical?

    The wordpress blog uses a database and this one is at his moment still on the same cloud instance. I hope that I can get this out into another instance soon.
    I am from the government, I am here to help you! Just ask!
    And don't expect more to be happen than with any other government!
    ---
    Ubuntu 15.10 on AMD64

  2. #2
    Join Date
    Jun 2007
    Location
    Porirua, New Zealand
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: How to limit server access?

    The robots.txt and .htaccess files might be good places to start.
    Forum DOs and DON'Ts
    Never assume that information you find using a search engine is up-to-date.
    Please use CODE tags.
    A low-volume blog

  3. #3
    Join Date
    Jun 2006
    Location
    Austrian in Taiwan
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: How to limit server access?

    I don't see how I can limit with that, just how to block, ... that is different.

    It does not bother that the robots are coming, just too many at once is not good.


    And what if it is just a great article that 1000s of people want to see? Maybe I limit in apache2.conf:

    <IfModule mpm_prefork_module>
    StartServers 5
    MinSpareServers 5
    MaxSpareServers 10
    MaxClients 150
    MaxRequestsPerChild 0
    </IfModule>

    change to MaxClients to 10

    What is your opinion?
    I am from the government, I am here to help you! Just ask!
    And don't expect more to be happen than with any other government!
    ---
    Ubuntu 15.10 on AMD64

  4. #4
    Join Date
    Oct 2005
    Location
    Al Ain
    Beans
    8,537

    Re: How to limit server access?

    Howdy,

    Iptables has a limit function. You can experiment and change the below somewhat.

    Code:
    # General new connection rate limiting for DOS and Brute Force protection
    iptables -I INPUT -p TCP -m state --state NEW -m limit --limit 30/minute --limit-burst 5 -j ACCEPT

  5. #5
    Join Date
    Nov 2008
    Location
    S.H.I.E.L.D. 6-1-6
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: How to limit server access?

    Quote Originally Posted by ELMIT View Post
    I don't see how I can limit with that, just how to block, ... that is different.

    It does not bother that the robots are coming, just too many at once is not good.


    And what if it is just a great article that 1000s of people want to see? Maybe I limit in apache2.conf:

    <IfModule mpm_prefork_module>
    StartServers 5
    MinSpareServers 5
    MaxSpareServers 10
    MaxClients 150
    MaxRequestsPerChild 0
    </IfModule>

    change to MaxClients to 10

    What is your opinion?
    You can.

    you are looking for the Crawl-Delay function
    Code:
    User-agent: Google
    Crawl-Delay: 120
    That only works for Yahoo - you will have to use Google webmaster tools to accomplish the same for google. http://support.google.com/webmasters...n&answer=48620

    Btw, is this the blog you mentioned in the other thread?

    If it is, there are actually some managed providers that specialize in Wordpress Hosting, and you may find hosting at those places a bit easier. Some of the more popular ones today are Page.ly, Zippykid, and WPEngine.
    Last edited by sandyd; October 14th, 2012 at 03:31 PM.
    Don't waste your energy trying to change opinions ... Do your thing, and don't care if they like it.

  6. #6
    Join Date
    Jun 2011
    Beans
    357

    Re: How to limit server access?

    You can use the ufw firewall tool to easily limit mass connection attempts like this. A simple command line

    sudo ufw limit 80/tcp

    should do the trick, assuming your web server is running on port 80. I believe that, by default, the client is limited to 6 connections within 30 seconds, which should be enough for normal users, but it will slow down web crawlers.

    See the "man ufw" page for more details.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •