The sitemaps protocol is a great and universal way to tell the search engines about all you're web pages - If (like me) you develop alot of dynamic websites that are driven off large SQL databases, trying to make sure all you're pages get crawled/found is really important... This really easy sitemap is a must in you're web design toolkit.

In this tutorial I am going to grab all the products from a database and just loop through them (Using a SQLDataReader and then using the new XmlTextWriter), you can use the same logic for pretty much every website - I have used this script on about a dozen websites and just changd the SQL to the appropriate table names and field names etc..

Firstly you want to make sure you have a robots.txt file in the root of you're website, and add the following to it

User-agent: *
Sitemap: http://www.YOUR-DOMAIN-NAME.co.uk/sitemap.aspx

Obviously if you already have a robots.txt then just add the bolded line and change the domain name to you're website domain - This tutorial will create the sitemap.aspx file for you.

Now you have you're robots.txt file sorted, create a new page (Do not use a code behind) in Visual Web Developer or what ever code editor you use and name it sitemap.aspx (the name is not important to be honest).   Delete all the code in the file, and we'll start building out the page.

Because this is an XML sitemap, we want to send the XML directly to the browser - So even though the page is an .aspx extension, when its sent to the browser we will make sure it thinks its XML by adding certain bits of code in the page directive..  Copy and paste the below into you're newly made page

[CODE]

<%@ Page ContentType="text/xml" %>
<%@ Import Namespace="System.Xml" %>
<%@ Import Namespace="System.Text" %>
<%@ Import Namespace="System.Data" %>

<script runat="server">

 Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs)

End Sub 

</script> 

[/CODE] 

 This is simply adding a few things we are going to need and adding a Page_Load event to trigger our XML file...  Now we want to create a a SQL DataReader to grab our products from the DataBase - Add the below in the Page_Load sub.

[CODE]

        Dim MyDomain As String = "YOUR DOMAIN NAME"
        Dim connString As String = "YOUR CONNECTION STRING"
        Dim objConnection As New SqlConnection(connString)
        'Obviously add you're own SQL below
        Dim strSQL As String = "SELECT idProduct FROM Products"
        Dim objCommand As New SqlCommand(strSQL, objConnection)
        objConnection.Open()
        Dim objDataReader As SqlDataReader = objCommand.ExecuteReader(CommandBehavior.CloseConnection)

[/CODE] 

Ok we have our SQLDataReader setup and ready lets start the XMLTextWriter

[CODE]

        'Create a new XmlTextWriter instance
        Dim writer As New XmlTextWriter(Response.OutputStream, Encoding.UTF8)
        'start writing!
        writer.WriteStartDocument()
        writer.WriteStartElement("urlset", "http://www.sitemaps.org/schemas/sitemap/0.9")
       
           writer.WriteStartElement("url")
       
            writer.WriteElementString("loc", "MyDomain")
            writer.WriteElementString("priority", "0.5")
       
            'End URL
            writer.WriteEndElement()

[/CODE] 

Now we have started the XML - Lets loop through the SQLDataReader and add all our URLS - Again I am using some random SQL, you will need to change the product ID or whatever you're links are driven off to the appropriate field/text.  Once we have got all the links we are going to clear everything up and close everything.

[CODE]

         While objDataReader.Read()
        
            'Creating the <browserInfo> element
            writer.WriteStartElement("url")
        
            writer.WriteElementString("loc", MyDomain & "products.aspx?ProductID=" & objDataReader.Item("idProduct").ToString())
            writer.WriteElementString("priority", "0.5")
        
            'End URL
            writer.WriteEndElement()
        
        End While
        

        objDataReader.Close()
        objConnection.Close()
        objDataReader = Nothing
        objConnection = Nothing
        
        writer.WriteEndDocument()
        writer.Close()

[/CODE] 

Thats it!  Simple as that..  

Heres all the code in one go

if you open the page up in Firefox you will see all the XML formatted correctly

[CODE]

<%@ Page ContentType="text/xml" %>
<%@ Import Namespace="System.Xml" %>
<%@ Import Namespace="System.Text" %>
<%@ Import Namespace="System.Data" %>

<script runat="server">

 Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs)

         Dim MyDomain As String = "YOUR DOMAIN NAME"
        Dim connString As String = "YOUR CONNECTION STRING"
        Dim objConnection As New SqlConnection(connString)
        'Obviously add you're own SQL below
        Dim strSQL As String = "SELECT idProduct FROM Products"
        Dim objCommand As New SqlCommand(strSQL, objConnection)
        objConnection.Open()
        Dim objDataReader As SqlDataReader = objCommand.ExecuteReader(CommandBehavior.CloseConnection)

        'Create a new XmlTextWriter instance
        Dim writer As New XmlTextWriter(Response.OutputStream, Encoding.UTF8)
        'start writing!
        writer.WriteStartDocument()
        writer.WriteStartElement("urlset", "http://www.sitemaps.org/schemas/sitemap/0.9")
       
           writer.WriteStartElement("url")
       
            writer.WriteElementString("loc", "MyDomain")
            writer.WriteElementString("priority", "0.5")
       
            'End URL
            writer.WriteEndElement()

      While objDataReader.Read()
        
            'Creating the <browserInfo> element
            writer.WriteStartElement("url")
        
            writer.WriteElementString("loc", MyDomain & "products.aspx?ProductID=" & objDataReader.Item("idProduct").ToString())
            writer.WriteElementString("priority", "0.5")
        
            'End URL
            writer.WriteEndElement()
        
        End While
        

        objDataReader.Close()
        objConnection.Close()
        objDataReader = Nothing
        objConnection = Nothing
        
        writer.WriteEndDocument()
        writer.Close()

End Sub 

</script>

[/CODE] 

 There are a couple of other tags you can add to the sitemaps XML such as lastmod & changefreq - Have a look at the protocol page on the sitemaps website to read more on them and also about the priority tag and how you can also change the value of that.