How to Develop Google Chrome Extensions (draft)

2011-06-23T04:02:00.000-07:00

Create the extension

Create a folder somewhere on your computer to contain your extension's code.
Inside your extension's folder, create a text file called manifest.json, and put this in it:

{
"name": "MyExtension",
"version": "1.0",
"description": "This is a sample Google Chrome extension.",
"permissions": [ "http://www.ralin.net/" ]
}

Load the extension

Open the extensions management page by choosing Tools > Extensions menu.
Hit the Load unpacked extension button and a file dialog will appear.
Go to your extension folder and select it.

C# Programming: Web Scraping with Regular Expressions (draft)

2011-04-30T01:53:00.000-07:00

Overview

If you want to extract data from a public web site you have to scrape it.

Use Case 1: You need all products from a given online shop catalog to do price comparisons or re-sell them at your own shop.

Use Case 2: You need to know how your site is positioned in search engine result pages (SERP) for given keywords. Then you need to search for your keywords and scrape the SERPs to find the positioning of your web site.

There are many use cases where web scraping can be applied. Before scraping a web site check the terms and conditions of the web site as there could be legal issues related to scraping information from certain sites.

Crawling before Scraping

The scraping part is the easiest part of the exercise. The more complex thing is to get to the pages you want to scrape. And there is only one way to do it - like a spider you need to crawl your way to the desired page.

Use regular expressions

Here I show a simple class that receives the HTML string and then extracts all the links and their text into structs. It is fairly fast, but I offer some optimization tips further down. It would be better to use a class here and offer methods that act on its contents.

Use SingleLine mode

Many C# developers make the mistake of not specifying that the Regexes work on multiple lines, treating newlines as regular characters. MSDN states that SingleLine "Specifies single-line mode. Changes the meaning of the dot (.) so it matches every character (instead of every character except \n)."

Advanced C# Programming: Developing Internet Explorer Extensions

2011-04-30T01:42:00.000-07:00

Overview

A browser extension or plugin is a software program that lives inside the browser and add new features making the browser suitable for specific personal or business needs. The most common business needs that are easily accomplished with browser plugins are: modify default search, add side frames, inject new content into existing webpage, communicate with remote servers or web services, search highlight, show ads, etc. In this article I will explain how to inject new static and dynamic content into existing webpage.

Use Case

You want to develop an Internet Explorer extension which shows some fixed layer containing promotional offers and discounts for the goods you are selling.
To achieve that you need to develop a custom Internet Explorer plugin using Browser Helper Objects (BHO) that will track user browsing activities and display your discounts based on some URL or content filtering using regular expressions or XPath.

Step by Step Walk-Through

I will explain here a step by step guide on how to accomplish this task as it is better to learn how to do it instead of just take a ready solution which you don't know how it works and the moment you touch something to make it suit your needs it breaks down and you have no idea why. I will use .NET Framework and C#.

1. Open your MS Visual Studio 2010 :-)

So far so good.

2. Create a new project "Visual C#" -> "Windows" -> "Class Library"

A new project is opened for you containing a single class "Class1". Rename the class to "BHO". You can use F2 on the CS file in the solution explorer.
Now you have to define the IObjectWithSite interface which the BHO class will implement. The IObjectWithSite interface provides simple objects with a lightweight siting mechanism (lighter than IOleObject). Often, an object must communicate directly with a container site that is managing the object. Outside of IOleObject::SetClientSite, there is no generic means through which an object becomes aware of its site. The IObjectWithSite interface provides a siting mechanism. This interface should only be used when IOleObject is not already in use. By using IObjectWithSite, a container can pass the IUnknown pointer of its site to the object through SetSite. Callers can also get the latest site passed to SetSite by using GetSite.

3. Add references and imports to the following components

using System.Runtime.InteropServices; 
using SHDocVw;
using mshtml;

4. Define the IOleObjectWithSite interface just above your BHO class declaration

[
        ComVisible(true),
        InterfaceType(ComInterfaceType.InterfaceIsIUnknown),
        Guid("FC4801A3-2BA9-11CF-A229-00AA003D7352")
    ]
    public interface IObjectWithSite
    {
        [PreserveSig]
        int SetSite([MarshalAs(UnmanagedType.IUnknown)]object site);
        [PreserveSig]
        int GetSite(ref Guid guid, out IntPtr ppvSite);
    }

5. Now make the BHO class implement the IOleObjectWithSite interface like

[
        ComVisible(true),
        Guid("1056BA25-DA81-56E3-A671B-D38A9B1B2142"),
        ClassInterface(ClassInterfaceType.None)
    ]
    public class BHO : IObjectWithSite
    {
        private WebBrowser webBrowser;

        public int SetSite(object site)
        {
            if (site != null)
            {
                webBrowser = (WebBrowser)site;
                webBrowser.DocumentComplete += 
                    new DWebBrowserEvents2_DocumentCompleteEventHandler(
                    this.OnDocumentComplete);
            }
            else
            {
                webBrowser.DocumentComplete -= 
                    new DWebBrowserEvents2_DocumentCompleteEventHandler(
                    this.OnDocumentComplete);
                webBrowser = null;
            }

            return 0;

        }

        public int GetSite(ref Guid guid, out IntPtr ppvSite)
        {
            IntPtr punk = Marshal.GetIUnknownForObject(webBrowser);
            int hr = Marshal.QueryInterface(punk, ref guid, out ppvSite);
            Marshal.Release(punk);
            return hr;
        }

        public void OnDocumentComplete(object pDisp, ref object URL)
        {
            HTMLDocument document = (HTMLDocument)webBrowser.Document;

            
        }
    }

We have implemented the SetSite and GetSite methods and added an empty event handler for the document loaded event. Thus, we will receive an event every time the document is loaded into the browser.

6. Implement the OnDocumentComplete method to inject the Javascript code and the DIV element

public void OnDocumentComplete(object pDisp, ref object URL)
{
    HTMLDocument document = (HTMLDocument)webBrowser.Document;

    HTMLHeadElement head = (HTMLHeadElement) document.all.tags("head").item(null, 0);
    IHTMLScriptElement script = (IHTMLScriptElement)document.createElement("script");
    script.type = @"text/javascript";
    script.text = "function hidediv(){document.getElementById('myOwnUniqueId12345').style.visibility = 'hidden';}";
    head.appendChild((IHTMLDOMNode)scriptObject);

    string div = "<div id=\"myOwnUniqueId12345\" style=\"position:fixed;bottom:0px;right:0px;z-index:9999;width=300px;height=150px;\"><div style=\"position:relative;float:right;font-size:9px;\"><a href=\"javascript:hidediv();\">close</a></div>" My content goes here ...</div>";

    document.body.insertAdjacentHTML("afterBegin", div);
}

First we inject the Javascript that we will use to close the div popup. Then we inject the html for the div element in the body with style that it is displayed in the bottom right corner of the browser. And that it is. You may want to filter our URLs and content for which you display the div.

public void OnDocumentComplete(object pDisp, ref object URL)
{
    if (URL.ToString().Contains("www.google.com"))
    {
        // Show div in here ...
    }
}

7. Register your BHO to be loaded by the Internet Explorer

public const string BHO_REGISTRY_KEY_NAME = 
"Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Browser Helper Objects";

[ComRegisterFunction]
public static void RegisterBHO(Type type)
{
    RegistryKey registryKey = Registry.LocalMachine.CreateSubKey(BHO_REGISTRY_KEY_NAME, true);

    string guid = type.GUID.ToString("B");
    
    RegistryKey ourKey = registryKey.CreateSubKey(guid);

    ourKey.SetValue("NoExplorer", 1, RegistryValueKind.DWord);

    ourKey.Close();

    registryKey.Close();
}

[ComUnregisterFunction]
public static void UnregisterBHO(Type type)
{
    RegistryKey registryKey = Registry.LocalMachine.OpenSubKey(BHO_REGISTRY_KEY_NAME, true);

    string guid = type.GUID.ToString("B");

    registryKey.DeleteSubKey(guid, false);
}

The register method simply tells the IE which is the GUID of your extension so that it could be loaded. The "No Explorer" value simply says that we don't want to be loaded by the Windows Explorer.
Now all you have to do is create a simple installation project that will install your IE plugin or install it manually.

That is - quick and simple!

Ralin's Blog

How to Develop Google Chrome Extensions (draft)

Create the extension

Load the extension

C# Programming: Web Scraping with Regular Expressions (draft)

Overview

Crawling before Scraping

Use regular expressions

Use SingleLine mode

Advanced C# Programming: Developing Internet Explorer Extensions

Overview

Use Case

Step by Step Walk-Through

1. Open your MS Visual Studio 2010 :-)

2. Create a new project "Visual C#" -> "Windows" -> "Class Library"

3. Add references and imports to the following components

4. Define the IOleObjectWithSite interface just above your BHO class declaration

5. Now make the BHO class implement the IOleObjectWithSite interface like

6. Implement the OnDocumentComplete method to inject the Javascript code and the DIV element

7. Register your BHO to be loaded by the Internet Explorer