Lachlan's Blogzone: c#

SEO sites like http://www.seomoz.org/rank-checker are great if you want to check out how your page ranks in Google for a given set of keywords, but you can only perform 5 searches per day using the free account :(.
There are others out there that will let you perform lots of searches, but restrict you to the first 100 results.

Apparently Google limits a given user to only so many (1000?) searches per day before having to use a CAPTCHA to perform subsequent searches, which I guess is understandable.

But what if you want to do more than 5 but less than 1000? And you want to scan the first 1000 results, because your site is really crap and you want to see if it's getting better? Luckily ,it is trivial to replicate the behaviour of SEOMOZ using simple screen scraping techniques. If you use the code below and install it on your localhost IIS you will be able to perform (practically) unlimited SEO searches. With the caveat that Google may at their whim change their HTML of course and hence break it!

Use at your own risk would be my advice.

You will need the latest .Net Framework 3.5 to run this. Create a new web app project in Visual Studio Pro 2008 or Web Dev Express and make a new .aspx page. Copy and paste the code below and voila, it should all work! (Oh yeah, you'll need a couple of images too if you want the nice AJAX effect - I suggest http://www.ajaxload.info/) If you are wondering it's Open Source so feel free to make it work better for you!

(Copy all to new .aspx page)

<%@ Import Namespace="System.IO" %>

<%@ Import Namespace="System.Net" %>

<%@ Page Language="vb" AutoEventWireup="false" EnableViewState="false" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<script runat="server">

   Private _baseURL As String = "http://www.google.co.nz/search?num=100"



    Protected Sub btnGo_Click(ByVal sender As Object, ByVal e As EventArgs) Handles btnGo.Click

        Dim baseurl As String = _baseURL & "&q=" & Me.txtKeywords.Text.Trim

        Dim numTries As Int32 = Math.Min(CInt(Me.cbxResults.Text) \ 100, 10)

        For iTry As Int32 = 1 To numTries

            Dim url As String = baseurl

            If iTry > 1 Then

                url &= "&start=" & ((iTry - 1) * 100).ToString

            End If

            Dim results As String = Me.readHTMLPage(url)

            If results.ToLower.Contains(Me.txtDomain.Text.Trim.ToLower) Then

                Me.showResults(results, (iTry - 1) * 100, url)

                Return

            End If

        Next

        ' If got here then not found

        Me.lblResult.Text = "Not found..."

        Me.lblResult.Visible = True

    End Sub

    Private Sub showResults(ByVal someHTML As String, ByVal addResults As Int32, ByVal googleURL As String)

        Dim googleResults As New List(Of String)(someHTML.Split(New String() {"<div class=g>", "<div class=g "}, StringSplitOptions.RemoveEmptyEntries))

        If googleResults.Count > 0 Then

            googleResults.RemoveAt(0) ' Remove the header rubbish from google

        End If

        Dim resultNum As Int32 = 1

        For Each aResult As String In googleResults

            If aResult.ToLower.Contains(Me.txtDomain.Text.Trim.ToLower) Then

                Me.lblResult.Text = "Result found at position " & (addResults + resultNum).ToString()

                Me.lblResult.Visible = True

                Dim link As New HyperLink()

                link.NavigateUrl = googleURL

                link.Text = googleURL

                Me.panelResults.Controls.Add(link)

                Me.panelResults.Controls.Add(New LiteralControl("<br /><hr />"))

                ' Show the resulting text

                If aResult.ToLower.Contains("<br clear=""all""/>") Then

                    ' Remove below that

                    aResult = Split(aResult, "<br clear=""all""/>", , CompareMethod.Text)(0)

                End If

                aResult = Replace(aResult, "onmousedown", "onmousedownX") ' Disable on mousedown

                Dim lit As New LiteralControl(aResult)

                Me.panelResults.Controls.Add(lit)

                Return

            End If

            resultNum += 1

        Next

    End Sub

    Private Function readHTMLPage(ByVal url As String) As String

        Dim result As String

        'Threading.Thread.Sleep(3000)

        Dim objRequest As WebRequest = HttpWebRequest.Create(url)

        objRequest.Credentials = CredentialCache.DefaultCredentials

        Dim objResponse As WebResponse = objRequest.GetResponse()

        Using sr As New StreamReader(objResponse.GetResponseStream())

            result = sr.ReadToEnd()

            sr.Close()

        End Using

        Return result

    End Function

</script>

<html xmlns="http://www.w3.org/1999/xhtml" >

<head runat="server">

    <title>Untitled Page</title>

    <style type="text/css">

        .style1

        {

            width: 255px;

        }

        .style2

        {

            font-size: x-large;

        }

    </style>

    <style type="text/css">

        div,td,.n a,.n a:visited{color:#000}.ts td,.tc{padding:0}.ts,.tb{border-collapse:collapse}.ti,.bl{display:inline}.ti{display:inline-table}.f,.m{color:#666}.flc,a.fl{color:#77c}a,.w,.q:visited,.q:active,.q,.b a,.b a:visited,.mblink:visited{color:#00c}a:visited{color:#551a8b}a:active{color:red}.t{background:#d5ddf3;color:#000;padding:5px 1px 4px}.bb{border-bottom:1px solid #36c}.bt{border-top:1px solid #36c}.j{width:34em}.h{color:#36c}.i{color:#a90a08}.a{color:green}.z{display:none}div.n{margin-top:1ex}.n a,.n .i{font-size:10pt}.n .i,.b a{font-weight:bold}.b a{font-size:12pt}.std{font-size:82%}#np,#nn,.nr,#logo span,.ch{cursor:pointer;cursor:hand}.ta{padding:3px 3px 3px 5px}#tpa2,#tpa3{padding-top:9px}#gbar{float:left;height:22px;padding-left:2px}.gbh,.gb2 div{border-top:1px solid #c9d7f1;font-size:0;height:0}.gbh{position:absolute;top:24px;width:100%}.gb2 div{margin:5px}#gbi{background:#fff;border:1px solid;border-color:#c9d7f1 #36c #36c #a2bae7;font-size:13px;top:24px;z-index:1000}#guser{padding-bottom:7px !important}#gbar,#guser{font-size:13px;padding-top:1px !important}@media all{.gb1,.gb3{height:22px;margin-right:.73em;vertical-align:top}.gb2 a,.gb2 b{display:block;padding:.2em .5em}}#gbi,.gb2{display:none;position:absolute;width:8em}.gb2{z-index:1001}#gbar a{color:#00c}.gb2 a,.gb3 a{text-decoration:none}#gbar .gb2 a:hover{background:#36c;color:#fff;display:block}.sl,.r{display:inline;font-weight:normal;margin:0}.sl{font-size:84%}.r{font-size:100%}.e{margin:.75em 0}.m{font-size:84%}.sm{display:block;margin:0;margin-left:40px}.slk td{padding-top:5px;padding-left:40px;vertical-align:top;font-size:84%}.slk div{text-indent:-10px;padding-left:10px}.csb,.n div,#logo span{background:url(/images/nav_logo3.png) no-repeat;height:26px;overflow:hidden}.n .nr{background-position:-60px 0;width:16px}#np{width:44px}#nf{background-position:-26px 0;width:18px}#nc{background-position:-44px 0;width:16px}#nn{margin-right:34px;width:66px}#nl{width:46px}#nn,#nl{background-position:-76px 0}#logo{display:block;height:52px;margin:13px 0 7px;overflow:hidden;position:relative;width:150px}#logo span{background-position:0 -26px;height:100%;left:0;position:absolute;top:0;width:100%}.ss{background:url(/images/nav_logo3.png) no-repeat;background-position:0 -87px;display:block;left:0;overflow:hidden;position:absolute;top:0}.cps{overflow:hidden;height:18px;width:114px}.mbi{display:block;font-size:0;width:12px;height:12px;background-position:-114px -78px;margin-right:2px}.mblink{font-size:100%}body,td,div,.p,a{font-family:arial,sans-serif}.g{margin:1em 0}#sd{font-size:84%;font-weight:bold}#ap{font-size:64%}

</style>

</head>

<body>



    <script type="text/javascript" src="script/prototype.js"></script>





    <form id="form1" runat="server">

    <asp:ScriptManager ID="ScriptManager1" runat="server">

    </asp:ScriptManager>

        <asp:UpdatePanel ID="UpdatePanel1" runat="server">

        <ContentTemplate>

    <div>

        <table style="width: 100%;">

            <tr>

                <td class="style1">

                    Keywords</td>

                <td>

                    <asp:TextBox ID="txtKeywords" runat="server" Width="356px"></asp:TextBox>

                </td>

            </tr>

            <tr>

                <td class="style1">

                    Domain </td>   <td>

                    <asp:TextBox ID="txtDomain" runat="server" Width="358px"></asp:TextBox>

                </td>

            </tr>

            <tr>

                <td class="style1">

                    Results to search</td>

                <td>

                    <asp:DropDownList ID="cbxResults" runat="server">

                        <asp:ListItem>100</asp:ListItem>

                        <asp:ListItem>200</asp:ListItem>

                        <asp:ListItem>300</asp:ListItem>

                        <asp:ListItem>400</asp:ListItem>

                        <asp:ListItem>500</asp:ListItem>

                        <asp:ListItem>600</asp:ListItem>

                        <asp:ListItem>700</asp:ListItem>

                        <asp:ListItem>800</asp:ListItem>

                        <asp:ListItem>900</asp:ListItem>

                        <asp:ListItem>1000</asp:ListItem>

                    </asp:DropDownList>

                </td>

            </tr>

            <tr>

                <td class="style1">

                    <asp:Button ID="btnGo" runat="server" Text="Find page rank"  />

                </td>

                <td>

                    <asp:UpdateProgress

                        ID="UpdateProgress1" runat="server" AssociatedUpdatePanelID="UpdatePanel1" DynamicLayout=true>

                        <ProgressTemplate>

                        <div style="background-image: url('images/bg.jpg'); width: 179px;">

                        <img src="Images/loading.gif" /><span class="style2">Searching..</span>.

                        </div>

                            </ProgressTemplate>

                    </asp:UpdateProgress>

                </td>

            </tr>

        </table>

    </div>



    <div id=results>



    </div>



        <asp:Panel ID="panelResults" runat="server">

            <asp:Label ID="lblResult" runat="server" Text="Label" Visible="False"

                Font-Bold="True" Font-Size="X-Large"></asp:Label>

        </asp:Panel>

        </ContentTemplate>

    </asp:UpdatePanel>

    </form>

</body>

</html>

Have you ever noticed that when you click the Refresh button in Firefox, any form values you have filled in on the screen are retained after the refresh? Firefox is automatically altering these values at the end of the request, presumably as a convenience to you, the user. Hence the DOM and the page source are out of sync immediately after the refresh. There is nothing you can do to disable this behaviour within Firefox.

For the average web app, this is a desirable feature. But when combined with the ASP.Net viewstate model, it is anything but desirable, as ASP.Net uses a hidden form field to store serialized viewstate information.

The specific bug I have encountered is when using ASP.Net in conjunction with the built-in validation controls and AJAX. When using validation controls, if viewstate does not match that which last left the server, an error is produced :

The state information is invalid for this page and might be corrupted.

Because Firefox attempts to fill in the current form with current values after a refresh, a new viewstate, representing a fresh page, can be overwritten by an old viewstate (the result of 1 or many AJAX postbacks changing control states since the last "fresh" page load.)

The easiest solution I have found to this problem thus far is to change the form ID with each (non postback) page load. This will fool Firefox into thinking the form is different, and as such all values (including viewstate) will get their values from the incoming HTML rather than the browser's previous state.

Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load

    If Not Me.IsPostBack Then

        Me.form1.ID = "form" & Date.Now.Ticks().ToString()

    End If

End Sub

The only potential problem you could encounter with this solution is where the form ID has been hardwired in your application, either in client side script or server side code. This can be mitigated by always using myForm.clientID when referencing the form's ID. This is good practice in any case.

Hopefully this will help others having the same problems I was.

NOTE!!!
This solution will not work when using a Master Page. Use the solution suggested here instead (change form ID in pageLoaded AJAX event).

For those interested, the offending code is here in System.Web.UI.HtmlControls.HtmlForm.UniqueID: (from Reflector)

Public Overrides ReadOnly Property UniqueID As String
   Get
      If (Me.NamingContainer Is Me.Page) Then
         Return MyBase.UniqueID
      End If
      Return "aspnetForm"
   End Get
End Property

Lachlan's Blogzone

Thursday, 8 May 2008

Homebrew Google Position finder for Keywords / Domain

Tuesday, 29 April 2008

Firefox refresh viewstate updatepanel bug hell!!!