No announcement yet.

Is it possible to use modesecurity to exclude bot agents?


Unconfigured Ad Widget

  • Filter
  • Time
  • Show
Clear All
new posts

  • Is it possible to use modesecurity to exclude bot agents?


    The currently used rule is Atomic Advanced.

    Is using a custom rule to exclude bot agents possible?

    SecRule REQUEST_HEADERS:User-Agent "@rx (?:AhrefsBot)" "msg:'AhrefsBot Spiderbot blocked',phase:1,log,id:7777771,t:none,block,statu s:403"

    if ($http_user_agent ~ (MJ12bot|LieBaoFast|UCBrowser|MQQBrowser|Mb2345Bro wser|gumgum-bot|postmanruntime|ag_dm_spider|scrapy|chimebot|tr endkite-akashic-crawler|ZoominfoBot|Sogou|ALittle|Keybot|Buck|curl |webprosbot|RestSharp|Snap|SemrushBot|AhrefsBot|Da taForSeoBot)) {
    return 403;

  • #2
    Hello Rachel,

    I can guide you through some steps to exclude bot agents using ModSecurityAnd I assume you have already installed it. If you haven't, then follow me.

    We already have an included directory for ModSecurity rules in the Apache configuration setup.

    Configure ModSecurity to Block bot Agents

    Include /etc/httpd/conf/modsecurity.d/rules/*.conf

    Now, we need to create a new .conf to start detecting agents.

    nano /etc/httpd/conf/modsecurity.d/rules/block_user_agents.conf

    We will add the following and save the file.

    SecRule REQUEST_HEADERS:User-Agent "@pmFromFile badbots.txt" "id:350001,rev:1,severity:2,log,msg:'BAD BOT - Detected and Blocked. '"

    We will create the list of User Agents to be detected and blocked:

    nano /etc/httpd/conf/modsecurity.d/rules/badbots.txt
    Then you can insert the list of the user agents if you want to allow them. like,


    Now, We need to configure Fail2ban

    First, you will need to install Fail2ban

    yum -y install fail2ban

    Now you need to create a new jail file,

    nano /etc/fail2ban/jail.local

    local jail allows the main fail2ban configuration to be updated with the new updates

    enabled = true
    filter = apache-useragent
    logpath = /var/log/httpd/error_log
    action = iptables-multiport[name=apache-badbots, port="http,https", protocol=tcp]
    postback[name=BADBOT, port="http,https", protocol=tcp]
    maxretry = 2
    bantime = 172800
    ignoreip =

    Update ignoreip with any local IPs or any others you want to allow in regardless of the UserAgent. This allows each IP to access twice with the UserAgent indicated in the list, after that it will be banned.

    You will then want to create the failregex pattern

    nano /etc/fail2ban/filter.d/apache-useragent.conf

    And you need to add the following:

    # Fail2Ban configuration file
    # [Definition] # Option: failregex
    # Notes.: Regexp to catch known spambots and software alike. Please verify
    # that it is your intent to block IPs which were driven by
    # abovementioned bots.
    # Values: TEXT
    # failregex = [[]client <HOST>[]] ModSecurity: Access denied with code 406 .* [[]msg "BAD BOT - Detected and Blocked. "[]] .*$ # Option: ignoreregex
    # Notes.: regex to ignore. If this regex matches, the line is ignored.
    # Values: TEXT

    Go ahead and restart fail2ban and apache.

    service httpd restart service fail2ban restart

    Hope you will find this helpful.