How to restrict access using .htaccess and robots.txt

→ Are you a new visitor? Please visit the page guidance for new visitors ←

How to restrict access using .htaccess and robots.txt

There will come a time when you want to restrict access to a certain directory and you don’t want to trouble with password protection and creating users and so on. It is quite simple to manipulate the access in a directory by using a simple .htaccess file. Almost all hosting providers allow this and it’s best to make use of it when needed.

  • Deny Directory Listing

I’ll start by showing you how to disable the directory listing, this will only be triggered in case that folder does not have an index file (index.php, index.html, index.htm etc.). You can create a .htaccess file with the following line in the directory where you want this disabled:

What you need to know however is that the subdirectory from the location where your .htaccess will contain this line will all be having directory listing disabled. So in case you want one of the subdirectories to show the directory listing you can use the following line in a .htaccess file:

Just make sure that this second .htaccess file is only present inside the folder where you want to allow directory listing.

You will see a 403 error, forbidden message whenever Directory listing is not allowed.

  •  Deny all access to a specific location using .htaccess

This is again pretty easy doing it using a .htaccess file in the directory where all access should be restricted. This again is will affect all files and subdirectories. The code that you need to use is:

  •  Allow access for a specific IP using .htaccess

But what would you do if you want to have access but only from your IP address ? There are times when you need to block access to resolve a specific issue. Allowing access only to your IP address is what you should look for in such cases. To do this you can use the following code in the same .htaccess file:

It’s important to use the correct IP address in the allow statement. You can find your IP address here:

In the above example it will restrict all access to your specific location, but it will allow access to IP:

In the same way you can block access to a network class or specific IPs using a similar code like:

Or for example you only want to allow access to your local area connection:

And that’s about it for restricting access using .htaccess. You can also use something similar to block robots, but using the robots.txt file instead:

This will prevent GoogleBot to not access your /restricted-folder/.

Some spiders that you will probably need to know when using robots:

  • GoogleBot – Google
  • Slurp – Yahoo
  • MSNBot – Bing

And to allow just certain spiders to crawl your site, for example only Google, you can use:

If you have problems, comment bellow and I’ll try to help.

Request an article ←