One of the limitations of force.com sites is the lockout feature which will not work out of the box on portals or force.com sites. So We need to create our own custom solution to implement this feature on force.com site. So here is a quick solution for users who want to build this feature on their force.com sites.

1. On the login method where you implement the site.login method, create a counter variable which can be a property in the controller.
2. If the login fails where site.login method would return null, increment the counter by 1. If the counter reaches the maximum limit, call Site.forgotpassword method which would reset the user’s password with a temporary email to the user.
3. User would login with temporary password and change the password.
4. You can create a custom field in user object called loginFailedctr to track users who had been locked out for future reporting.
So using the above techniques, you can easily create a custom lockout feature in salesforce automatically.

loggedInPage = Site.login(new password,confirm password,landingUrl);
if(loggedInPage == null)
{
loginFailedCtr++;

//If the login attempts is greater than 3, reset the password.
if(loginFailedCtr > 3)
{
Site.forgotPassword(viewstatecontroller.userID_str);
//Update counter for failed attempts
acsUserDataUpd.LoginFailedCtr__c = viewstateController.loginFailedCtr;
Helper_DMLOperations.upsertRecord(acsUserDataUpd);
//Redirect to accountlockout page.
loggedInPage= new PageReference('/apex/AcctLock');
}