Extensible Storage Engine (ESE) : Storage option for large Sensor data

Currently I am evaluating the storage options to store large volume of data generated by sensors ( temperature,pressure,pH  … ) .  These sensor generate more than 1,000,000 data samples each day.  Goal is to select efficient storage option which is simple to deploy , easy to implement, and provides reliability against the application crashes.

Obvious choices include simple comma separated file ( CSV ) or embedded database solution. Each of the above mentioned options have its own pros/cons.

On searching  the internet for alternative solution found article on Extensible Storage Engine ( ESE ) which seems to be promising option . In this blog post I will discuss the same.

Extensible Storage Engine (ESE) is storage engine technology built into Windows since Windows 2000 . It is distributed has library (ESENT.DLL ) . It is used extensively inside many Microsoft  products  such as Active Directory, Microsoft Exchange  and Windows Desktop search. Outside Microsoft  it is  used in  RavenDB – a NoSQL document database built on top of “ESE”.

At high level “ESE” has following advantage over CSV and RDBMS

  • Zero-deployment  – Built into windows.
  • Extensively tested – Used inside Microsoft products and open source projects.
  • Easy to administer  –  Use file manipulation tools such as copy/move to backup the files.
  • Easy to use    – Managed .NET wrapper with LINQ support is available.
  • Others – Robust crash recovery, intelligent cashing, concurrent access to data and many more.

Even though ESE library is “C” API , there is managed .NET API available on codeplex website.

Codeplex project also provides an implementation of Persistent Dictionary . “PersistentDictionary” is a database-backed dictionary built on top of the ESENT database engine. It is a drop-in compatible replacement for the generic Dictionary<TKey,TValue>.

I did some  prototype implementation based on the sample code available at codeplex project  .This implementation  uses persistent dictionary to stores sensor data with key being the date time and value being the actual data.Initial performance results are good.

Here is the details about the sample application. This application generates 1,000,000 random sample data along with the timestamp. It uses date time has key and sensor data has value.

Make sure that you use the nuget to install managed ESENT .NET library.

Sample code

using Microsoft.Isam.Esent.Collections.Generic;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace SensorStorage
{
    ///
/// value type that stores sensor id
    /// and the actual sensor value.
    ///
    [Serializable]
    struct Sensor
    {
        public string Id;
        public float  Value;
    }

    class StoragePerfTest
    {
        public static int MAX_RECORDS = 1000000;
        public static int MAX_SENSORS = 16;
        // sensor id list
        public static List SensorIdList = new List
        {
            "46d77be8-1b6c-40e4-be90-657bba696db4",
            "84ccd877-e8be-4da8-81ae-1d7bda0ff2d7",
            "49be5ff1-bc44-4173-847f-33a9394d8787",
            "c41f8e8e-ca42-4248-8e2a-f047db76efcc",
            "b94bd698-9deb-4857-a685-f2b4920765ad",

            "3ff45aed-6c31-4079-8d83-6b0a6d2516ec",
            "d53d0cc8-debf-43a8-8ecf-37a9ff5a8afc",
            "2cb6ec16-da28-498f-baf3-9bcc661cd18e",
            "5b9843a4-db0d-4887-99f4-1b460eaa3c05",
            "7e3b9124-9816-4ecc-8d1c-95030bb93fc7",

            "01487c6a-cf7f-47ea-b870-58479ec5fd61",
            "ab271b46-b224-45e6-a8b7-1d04e20e0de2",
            "88c9d2e3-a411-446e-bf0a-d34e9278764d",

            "46a8c28f-07ab-4559-a9c7-5986d6e84baa",

            "df723ed8-70ee-4ca7-b8ae-df0902d481a3",
            "98e376bf-82e7-4bf3-96ab-f66cde5537bd"
        };

        ///
/// generate random value simulating the sensor data.
        ///
        ///
        static float NextFloat(Random random)
        {
            var buffer = new byte[4];
            random.NextBytes(buffer);
            return BitConverter.ToSingle(buffer, 0);
        }
        ///
/// gets next sensor id from the list simulating
        /// random sensor sending value to application
        ///
        ///
        ///
       static string GetNextSensorId(Random random)
        {
            int index = random.Next(0, MAX_SENSORS - 1);
            return StoragePerfTest.SensorIdList[index];
        }

        static void Main(string[] args)
        {

            if ( !PersistentDictionaryFile.Exists("SensorData"))
            {
                Console.WriteLine("No sensor data available");
                return;
            }

            if (args.Length == 0)
            {
                GenerateSensorData();

            }
            else if (args.Length == 2)
            {
                QueryRecords(DateTime.Parse(args[0]), DateTime.Parse(args[1]));
                return;
            }

            Console.WriteLine("Usage : StoragePerfTest.exe [<start_time> <end_time>] ");
            Console.WriteLine("To Generate sample data execute StoragePerfTest without any arguments . Example : StoragePerfTest.exe  ");
            Console.WriteLine("To query . Example : StoragePerfTest.exe \"7/27/2013 05:42:00 AM\" \"7/27/2013 05:44:00 AM\"");
            return;
        }
        ///
///  generate random records
        ///
        private static void GenerateSensorData()
        {
            Console.WriteLine("Deleting old sensor and generating new data");
            PersistentDictionaryFile.DeleteFiles("SensorData");
            Random random = new Random();
            Stopwatch watch = new Stopwatch();

            using (var dictionary = new PersistentDictionary<DateTime, Sensor>("SensorData"))
            {

                watch.Start();
                DateTime curr = DateTime.Now;

                for (int i = 0; i < MAX_RECORDS; i++)
                {
                    curr = curr.AddSeconds(1);
                    dictionary.Add(curr, new Sensor { Id = GetNextSensorId(random), Value = NextFloat(random) });

                }
                watch.Stop();
            }
            Console.WriteLine("Inserted {0} records in {1} miliseconds", MAX_RECORDS, watch.ElapsedMilliseconds);
        }

        ///
/// queries record
        ///
        ///
        ///
        private static void QueryRecords(DateTime st,DateTime et)
        {
            Stopwatch watch = new Stopwatch();
            using (var dictionary = new PersistentDictionary<DateTime, Sensor>("SensorData"))
            {
                watch.Start();
                var samples = from x in dictionary
                              where x.Key >= st && x.Key <= et
                              select x;
                int count = samples.Count();
                watch.Stop();
                Console.WriteLine("{0} Records Found for the query in {1} miliseconds", count, watch.ElapsedMilliseconds);
                foreach (var s in samples)
                {
                    Console.WriteLine("Key {0} = [{1},{2}]", s.Key, s.Value.Id, s.Value.Value);
                }
            }
        }

    }
}

Performance results

One my laptop ( 5400 rpm hd, dual core cpu with 4gb ) , I got following results.

  • Total records inserted : 1,000,000.
  • Database Size : 215MB
  • Add speed: ~18000 records/seconds into persistent dictionary.
  • Query speed : 120 records in 80 milliseconds to query the data in the middle of the sample data.

I will be doing further investigation to see the suitability of the library for the project .

Resource

  1. Codeplex project.
  2. MSDN Documentation.
  3. ESE Database Viewer.
  4. RavenDB developer blog on ESE

.

Advertisements

Simplify asynchronous programming with C# 5 async/await

One of key benefit of asynchronous programming is scalable & responsive applications. On server by eliminating threads blocked for network/disk I/O to complete, you can write scalable application using fewer server resources (threads). Similarly on the client side asynchronous programming helps in designing responsive UI without blocking the UI thread.

Traditionally designing asynchronous programming is hard that involves learning low level operating systems facilities such as POSIX select, epoll on Linux and IO completion ports on Windows. Even though newer platforms (Java and .NET) hide these complexities with nice object oriented API it is still hard to develop asynchronous applications as it involves dealing with callback and state machines.

.NET applications use APM or EAP to write asynchronous applications. Throughout .NET base class library, many classes support using the APM by implementing BeginXXX and EndXXX versions of functions. These methods allow applications to perform I/O operations asynchronously. For example, the FileStream class in System.IO namespace has a BeginXXX and EndReadXXX methods that reads data from a stream asynchronously.

Even with APM, asynchronous application development is still not as easy as synchronous applications because of callback and the state involved. C# 5 introduces new language features to simplify asynchronous programming. It introduces pattern where user can write asynchronous programs with sequential control flow by using async/await keywords. C#  implements this feature using Task Parallel library (TPL) and some compiler magic. If you interested to know more about how these feature implemented see the resource section.

Here is the simple TCP echo server that demonstrates C# async/await pattern .This program handle multiple client connections without creating threads explicitly. If you read the program, it looks like synchronous application with sequential control flow and easy to understand and maintain. Under the cover C# compiler is transforming code to use the asynchronous API with the help of Task parallel library. This transformation is similar to how C# implements yield keyword

 

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Runtime.CompilerServices;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace AsyncEchoServer
{
    public class AsyncEchoServer
    {
        private int _listeningPort;
        public AsyncEchoServer( int port)
        {
            _listeningPort = port;
        }
        ///
<summary>
        /// Start listening for connection
        /// </summary>
        public async void Start()
        {
            IPAddress ipAddre = IPAddress.Loopback;
            TcpListener listener = new TcpListener(ipAddre, _listeningPort);
            listener.Start();
            LogMessage("Server is running");
            LogMessage("Listening on port " + _listeningPort);

            while (true)
            {
                LogMessage("Waiting for connections...");
                try
                {
                    var tcpClient = await listener.AcceptTcpClientAsync();
                     HandleConnectionAsync(tcpClient);
                }
                catch (Exception exp)
                {
                    LogMessage(exp.ToString());
                }

            }

        }
        ///
<summary>
        /// Process Individual client
        /// </summary>
        ///
        ///
        private async void HandleConnectionAsync(TcpClient tcpClient)
        {
            string clientInfo = tcpClient.Client.RemoteEndPoint.ToString();
            LogMessage(string.Format("Got connection request from {0}", clientInfo));
            try
            {
                using (var networkStream = tcpClient.GetStream())
                using (var reader = new StreamReader(networkStream))
                using (var writer = new StreamWriter(networkStream))
                {
                    writer.AutoFlush = true;
                    while (true)
                    {
                        var dataFromServer = await reader.ReadLineAsync();
                        if (string.IsNullOrEmpty(dataFromServer))
                        {
                            break;
                        }
                        LogMessage(dataFromServer);
                        await writer.WriteLineAsync("FromServer-" + dataFromServer);
                    }
                }
            }
            catch (Exception exp)
            {
                LogMessage(exp.Message);
            }
            finally
            {
                LogMessage(string.Format("Closing the client connection - {0}",
                            clientInfo));
                tcpClient.Close();
            }

        }
        private void LogMessage(string message,
                                [CallerMemberName]string callername = "")
        {
            System.Console.WriteLine("[{0}] - Thread-{1}- {2}",
                    callername, Thread.CurrentThread.ManagedThreadId, message);
        }
    }
    class Program
    {
        static void Main(string[] args)
        {
            AsyncEchoServer async = new AsyncEchoServer(51510);
            async.Start();
            Console.ReadLine();
        }
    }
}

Echo Client Sample

class Program
    {
        static void Main(string[] args)
        {
            StartClient(Convert.ToInt32(args[0]));
            Console.ReadLine();
        }

        private static async void StartClient(int port)
        {
            TcpClient client = new TcpClient();
            await client.ConnectAsync(IPAddress.Loopback, port);
            LogMessage("Connected to Server");
            using (var networkStream = client.GetStream())
            using (var writer = new StreamWriter(networkStream))
            using (var reader = new StreamReader(networkStream))
            {
                writer.AutoFlush = true;
                for(int i = 0; i < 10;i++)
                {
                    await writer.WriteLineAsync(DateTime.Now.ToLongDateString());
                    var dataFromServer = await reader.ReadLineAsync();
                    if (!string.IsNullOrEmpty(dataFromServer))
                    {
                        LogMessage(dataFromServer);
                    }
                    
                }
            }
            if (client != null)
            {
                client.Close();
            }
           
        }
        private static void LogMessage(string message, 
                [CallerMemberName]string callername = "")
        {
            System.Console.WriteLine("{0} - Thread-{1}- {2}", 
                callername, Thread.CurrentThread.ManagedThreadId, message);
        }

    }

Resource

1. Visual Studio async home page.

2. DNT TV Screencast which explains async feature in .NET .

3. Jon Skeet multi part screencast on async/await. Part 1 , Part 2

4. Joseph Albahari talk at Tech·Ed Australia 2011

Compile and Run .NET program with just .NET framework on your system.

Compile and Run .NET program with just .NET framework on your system.

Did you know that every .NET framework installation also installs VB & C# compiler.

Using these compilers you can compile and run C# or VB.NET program on machine with just .NET framework installed. This can be useful on system where installing full visual studio is not an option.

Here is how to do it

If .NET framework installed on you system, C:\Windows\Microsoft.NET\Framework directory  should list all the versions. Here is what is on my machine.

image

Once you find required .NET version, directory content should list csc.exe and vbs.exe compiler executable. As shown below.

image

Now use C# or VB compiler to compile the program as follows. For example use .NET 2.0 compiler to generate .exe c:\Windows\Microsoft.NET\Framework\v2.0.50727\csc.exe Program.cs

image

Identifying process using TCP port in .NET

Currently I am working on a application  which uses third-party component for remote diagnostics.  This third party component internally uses 8081 tcp port for communication between the process and port was not configurable , which forced our application installer to make sure that no other process in the system is using port 8081 during install. We had to warn user during application install if any process is using required tcp port.

In .Net you can use System.Net.NetworkInformation.IPGlobalProperties to find out whether tcp port in use. But this API does not tell you which process using the port. We wanted to show user not only tcp port is in use , also provide information  about process , such as name , id and path .  We also had to make sure that it works from Windows XP to Windows 7 .

From MSDN documentation, I found that only to get the process id of the process using the tcp port is to use Win32 API . This means I had to use platform invoke(pinvoke) and carefully declare lot of Win32 structure in C#.

Other alternative is to use the netstat command , that provides the same details with appropriate command line switches. I used this technique various times ,when I was working on Linux systems. I decided to do the same here.

I wrote simple helper class which uses netstat and WMI to get the details about the process using the tcp port.
Here is how you use the sample

int port = 8081;
TcpHelperUtil tcpHelper = new TcpHelperUtil();
var details = tcpHelper.GetPortDetails(port);
if (details.Item1)
{
    Console.WriteLine("Port {0} in Use",port);
    Console.WriteLine(details.Item2.ToString());
}else
{
    Console.WriteLine("Port {0} is free ",port);
}

Here is the source code.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Management;

namespace ConsoleApp2
{

    internal class PortDetail
    {
        public int Port { get; set; }
        public int ProcessID { get; set; }
        public string ProcessName { get; set; }
        public string Path { get; set; }
        public override string ToString()
        {
            return string.Format(@" Process Name: {0} ,Process ID: {1} ,
                                    Port: {2} ,\nPath : {3}", ProcessName,
                                    ProcessID, Port, Path);
        }

    }
    /// <summary>
    /// Usage:
    /// int port = 8081
    /// TcpHelperUtil tcpHelper = new TcpHelperUtil();
    /// var details = tcpHelper.GetPortDetails(port);
    /// if (details.Item1)
    /// {
    ///     Console.WriteLine("Port {0} in Use",port);
    ///     Console.WriteLine(details.Item2.ToString());
    /// }else
    /// {
    ///     Console.WriteLine("Port {0} is free ",port);
    /// }
    ///
    /// </summary>
    class TcpHelperUtil
    {
        private const short MINIMUM_TOKEN_IN_A_LINE = 5;
        private const string COMMAND_EXE = "cmd";

        public TcpHelperUtil()
        {

        }

        public Tuple<bool, PortDetail> GetPortDetails(int port)
        {
            PortDetail PortDetail = new PortDetail();
            Tuple<bool, PortDetail> result = Tuple.Create(false, PortDetail);

            // execute netstat command for the given port
            string commandArgument = string.Format("/c netstat -an -o -p tcp|findstr \":{0}.*LISTENING\"", port);

            string commandOut = ExecuteCommandAndCaptureOutput(COMMAND_EXE, commandArgument);
            if (string.IsNullOrEmpty(commandOut))
            {
                // port is not in use
                return result;
            }

            var stringTokens = commandOut.Split(default(Char[]), StringSplitOptions.RemoveEmptyEntries);
            if (stringTokens.Length < MINIMUM_TOKEN_IN_A_LINE)
            {
                return result;
            }

            // split host:port
            var hostPortTokens = stringTokens[1].Split(new char[] { ':' });
            if (hostPortTokens.Length < 2)
            {
                return result;
            }

            int portFromHostPortToken = 0;
            if (!int.TryParse(hostPortTokens[1], out portFromHostPortToken))
            {
                return result;
            }
            if (portFromHostPortToken != port)
            {
                return result;
            }

            PortDetail.Port = port;
            PortDetail.ProcessID = int.Parse(stringTokens[4].Trim());
            Tuple<string, string> processNameAndPath = null;
            try
            {
                processNameAndPath = GetProcessNameAndCommandLineArgs(PortDetail.ProcessID);
                PortDetail.ProcessName = processNameAndPath.Item1;
                PortDetail.Path = processNameAndPath.Item2;
                result = Tuple.Create(true, PortDetail);
            }
            catch (Exception exp)
            {
                Console.WriteLine(exp.ToString());

            }

            return result;

        }
        /// <summary>
        /// Using WMI API to get process name and path instead of
        /// Process.GetProcessById, because if calling process ins
        /// 32 bit and given process id is 64 bit, caller will not be able to
        /// get the process name
        /// </summary>
        /// <param name="processID"></param>
        /// <returns></returns>
        private Tuple<string, string> GetProcessNameAndCommandLineArgs(int processID)
        {
            Tuple<string, string> result = Tuple.Create(string.Empty, string.Empty);
            string query = string.Format("Select Name,ExecutablePath from Win32_Process WHERE ProcessId='{0}'", processID);
            try
            {
                ObjectQuery wql = new ObjectQuery(query);
                ManagementObjectSearcher searcher = new ManagementObjectSearcher(wql);
                ManagementObjectCollection results = searcher.Get();

                // interested in first result.
                foreach (ManagementObject item in results)
                {
                    result = Tuple.Create<string, string>(Convert.ToString(item["Name"]),
                    Convert.ToString(item["ExecutablePath"]));
                    break;

                }
            }
            catch (Exception)
            {

                throw;
            }

            return result;

        }

        /// <summary>
        /// Execute the given command and captures the output
        /// </summary>
        /// <param name="commandName"></param>
        /// <param name="arguments"></param>
        /// <returns></returns>
        private string ExecuteCommandAndCaptureOutput(string commandName, string arguments)
        {
            string commandOut = string.Empty;
            Process process = new Process();
            process.StartInfo.FileName = commandName;
            process.StartInfo.Arguments = arguments;
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.CreateNoWindow = true;
            process.StartInfo.RedirectStandardOutput = true;
            process.StartInfo.RedirectStandardError = true;
            process.Start();

            commandOut = process.StandardOutput.ReadToEnd();
            string errors = process.StandardError.ReadToEnd();
            try
            {
                process.WaitForExit(TimeSpan.FromSeconds(2).Milliseconds);
            }
            catch (Exception exp)
            {

                Console.WriteLine(exp.ToString());
            }
            return commandOut;
        }
    }
}

Monitor process startup/shutdown using WMI & PowerShell

Recently I was working on a project, where I needed to monitor process for startup/shutdown events,  after searching internet  found that it can be easily done using WMI and PowerShell .

WMI (Windows Management Instrumentation) is a technology built in to Windows since Windows 2000, that provides standard interface to manage windows systems. For example it can used to find software products installed in single or multiple machines in network or to verify necessary OS service pack is installed on all machines in the network.

PowerShell provides easy way of accessing WMI functionality using simple scripting language.

Here is how you can monitor for process startup using PowerShell & WMI events

1. Launch PowerShell and enter following command. This command will register for process started event and prints statement in console whenever notepad.exe is executed.

Register-WMIEvent -query “SELECT * FROM Win32_ProcessStartTrace WHERE ProcessName=’notepad.exe'” -SourceIdentifier “testevent” -action { $e = $Event.SourceEventArgs.NewEvent

Write-Host $e.ProcessName,” started ” }

2. Now launch notepad and observer that console has message confirming the same.

3. Once you are done, unregister the event by entering following command

Unregister-Event testevent

4. You can also see list of event subscribers with “Get-EventSubscriber” command.

Some more examples of using WMI & PowerShell

1.Monitor process stop events

Register-WMIEvent -query “SELECT * FROM Win32_ProcessStopTrace WHERE ProcessName=’notepad.exe'” -SourceIdentifier “testevent” -action { $e = $Event.SourceEventArgs.NewEvent

Write-Host $e.ProcessName,” stopped ” }

2. Monitor Windows Service stop/start status. Replace “TargetInstance.Name” value with your service name

Register-WMIEvent -query “Select * From __InstanceOperationEvent within 1 Where TargetInstance ISA ‘Win32_Service’ and TargetInstance.Name=’Fax'” -sourceIdentifier “MyServMonitor” -action { Write-host “Service Name :”,$EventArgs.NewEvent.TargetInstance.Name ,” Service State :”, $EventArgs.NewEvent.TargetInstance.State }

image

You can also do the same event monitoring using WMI & C#


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Management;

namespace WmiEventTest
{
    class ServiceStatusMonitor
    {
        static void Main(string[] args)
        {

            if (args.Length < 1)
            {
                Console.WriteLine("Usage: ServiceStatusMonitor  ");
                Environment.Exit(0);
            }
            string WMI_EVENT_QUERY = @"SELECT * FROM __InstanceModificationEvent
                WITHIN 1 WHERE TargetInstance ISA 'Win32_Service'";

            string WMI_EVENT_QUERY_WITH_SERVICE_NAME = WMI_EVENT_QUERY
                    + " and TargetInstance.Name = '{0}'";
            WqlEventQuery serviceModificationEvent =
                    new WqlEventQuery(string.Format(WMI_EVENT_QUERY_WITH_SERVICE_NAME, args[0]));
            ManagementEventWatcher eventWatcher =
                    new ManagementEventWatcher(serviceModificationEvent);
            eventWatcher.EventArrived +=
                    new EventArrivedEventHandler(Watcher_EventArrived);
            Console.WriteLine("Waiting for service status change events ...");
            eventWatcher.Start();
            Console.ReadLine();
        }

        static void Watcher_EventArrived(object sender, EventArrivedEventArgs e)
        {
            string eventType = e.NewEvent.ClassPath.ClassName;

            switch (eventType)
            {
                case "__InstanceCreationEvent":

                    Console.BackgroundColor = ConsoleColor.Blue;
                    Console.WriteLine("'{0}' Service created ....",
                            Environment.GetCommandLineArgs()[1]);
                    Console.ResetColor();
                    break;
                case "__InstanceDeletionEvent":

                    Console.BackgroundColor = ConsoleColor.Green;
                    Console.WriteLine("'{0}' Service deleted ....",
                        Environment.GetCommandLineArgs()[1]);
                    Console.ResetColor();
                    break;

                case "__InstanceModificationEvent":

                    Console.BackgroundColor = ConsoleColor.Blue;
                    ManagementBaseObject obj = (ManagementBaseObject)e.NewEvent["TargetInstance"];
                    Console.WriteLine("'{0}' Service Modified ( {1} )",
                        Environment.GetCommandLineArgs()[1], obj["State"]);
                    Console.ResetColor();
                    break;
            }

        }
    }
}

Resources

1. Introduction to WMI

2. Receiving WMI Events

3. PowerShell for event monitoring.

Push notification from server to client using SignalR

There are number of scenarios where we need send push notification from server to client, client gets update from the server when ever server has new data.

Some of the scenario which benefit from push notification(real-time web) are

  • Monitoring Server application over the web(Diagnostic reports or Error reports )
  • User signup notification.
  • Stock tickers.
  • Chat application.
  • Monitoring of hardware instruments.

Now we know the possible usecases ,so how can we implement server to send arbitrary data to client.

Currently there are four established technique to do push notification on web

    • Polling : In case of polling Web browser make connection to server, server responds immediately with data if present else empty response. After predefined time(polling time) Web browser makes same request to server.This process repeats. Even though this solution works, it is not scalable solution. It can put heavy load on the server depending on the polling interval and number of clients . Polling from JavaScript shown below
	function getInfo() {

	$.ajax("url", function ( newInfo){
		if ( newInfo != null) {
			// do something with the data
		}
	});
	// poll again after 20 seconds
	setTimeout(getInfo,20000);
}
// start the polling loop
getInfo();
    • Long polling: It is similar to polling , where Web browser makes request to server , but server does not respond to client immediately , it will leave the request hanging open till it has some data to deliver. Eventually server responds and client uses the data and starts new request, whole process repeats. This technique is improvement over polling as it does creates unnecessary connections.  Long polling from JavaScript shown below
function getNewInfo(){
  $.ajax("url", function (newinfo) {
      // do something with the data
	  // start the new request
      getNewINfo();
  });
}
// start the polling loop
getNewInfo();
  • EventSource: It is similar to long polling client make request ,server leaves the request hanging open till it has some data to deliver, difference is after sending data back server does not close the connection. Server will keep the connection open .  This solution provide good scalability in handling requests , also it is implemented in natively by Web Browser it can perform better compared to previous techniques.But EventSource is  not iimplemented IE.  Event source from JavaScript shown below
var eventSrc = new EventSource("url");
	// register event handler for the message
    eventSrc.addEventListener( "message",function (evt) {
        //process the data
    });
  • WebSocket:Is the new approach to Web browser/server communication. It is bidirectional communication channel compared to unidirectional Event Source.  At high-level client sends HTTP Connection with “Upgrade: websockets” header;server leaves connection open , client and server negotiate which version of web socket protocol to use. Client and Server exchange messages once connection is upgraded to web socket. This technique is natively implemented by the Web Browser/Server . Refer to caniuse website to check the latest websocket implementation in Web Browser. WebSocket usage from JavaScript shown below
var socket = new WebSocket("url");
socket.onmessage = function (msg) {
	var newInfo = msg.data;
	// do something with the data
}
// client can also send request to server
socket.send(.... )

As we seen ,each technique has some pros and cons and not all browser have implemented all the features.
What we need is some abstraction over the different implementation  so that developer need not worry about various implementations.

SignalR to rescue.

SignalR is an abstraction on top of a persistent HTTP connection. It provides on consistent API over different implementations(long polling,websocket).

SignalR can use websocket if it is implemented by the Web browser and server and fallback to long polling if it is not implemented.
SignalR uses C# dynamic feature on the server side ( asp.net ) and Javascript dynamic features on the client side to create server and client proxy.

It is similar to Nodejs+socket.io implementation based on JavaScript.

CPU usage monitor sample

To illustrate the SignalR usage I wrote sample web application which allows to monitor % cpu usage on the server in real-time. Sample  application uses SignalR on server/client and Google web chart  for visualization.

Tools used: Free Microsoft Visual Web Developer 2010 Express, SignalR on Nuget and Google Chart Tools.

Here is the step by step instruction.

1. Create “ASP.NET Empty Web Application” in VS Web Developer express.

2. Use Nuget Package manager console in VS to install SignalR and associated dependency from Nuget.

3.Create class to hold the cpu usage information.

public class CpuUsageInfo
    {
        public double CPUUsage { get; set; }
        public String DateString { get; set; }
    }

4. Create new class on the server side which derives from SignalR PersistentConnection class.

 public class Perfmon : PersistentConnection
    {
        protected override System.Threading.Tasks.Task OnConnectedAsync(SignalR.Hosting.IRequest request, IEnumerable groups, string connectionId)
        {
            return base.OnConnectedAsync(request, groups, connectionId);
        }
    }

5.Update the Global.asax file to map ASP.NET route

protected void Application_Start(object sender, EventArgs e)
        {

            // add routing
            RouteTable.Routes.MapConnection("perfmon", "perfmon/{*operations}");

            Task.Factory.StartNew(() =>
                {
                    // get connection to global client connections
                    IConnectionManager connectonManager = AspNetHost.DependencyResolver.Resolve();
                    var connection = connectonManager.GetConnection();

                    var counter = new PerformanceCounter();
                    counter.CategoryName = "Processor";
                    counter.CounterName = "% Processor Time";
                    counter.InstanceName = "_Total";

                    while (true)
                    {
                        var item = new CpuUsageInfo()
                        {
                            DateString = DateTime.Now.ToString("hh:mm:ss"),
                            CPUUsage = Math.Round(counter.NextValue(), 2),
                        };

                        connection.Broadcast(item);

                        Thread.Sleep(500);
                    }
            });

        }

6. Inside Application_Start method add code to get send CPU usage events to connected clients using SignalR library.

7. Create new html file with that plots line chart using Google charting tools. In the JavaScript make connection to server using SignalR js library. As the data arrives from the server update the chart data collection and redraw the chart.

    var output;
    $(document).ready(function () {

        output = $("#output");

    });

    var data;
    var chart;
    var options;
    // Load the Visualization API and the piechart package.
    google.load('visualization', '1.0', { 'packages': ['corechart'] });
    // Set a callback to run when the Google Visualization API is loaded.
    google.setOnLoadCallback(drawChart);
    // Callback that creates and populates a data table,
    // instantiates the pie chart, passes in the data and
    // draws it.
    function drawChart() {

        // Create the data table.
        data = new google.visualization.DataTable();
        data.addColumn('string', 'Time');
        data.addColumn('number', 'CPU Usage in %');

        // Set chart options
        options = { 'title': 'CPU usage history', 'width': 600, 'height': 400 };

        // Instantiate and draw our chart, passing in some options.
        chart = new google.visualization.LineChart(document.getElementById('chart_div'));
        chart.draw(data, options);
    }

    connection = $.connection('/perfmon');

    connection.received(function (dataItem) {
        var newItem = dataItem;

        output.text("Received text from the server .: " + newItem.DateString + "," + newItem.CPUUsage);

        // update chart with data received from the server
        data.addRow([newItem.DateString, newItem.CPUUsage]);
        if (data.getNumberOfRows() > 100) {
            data.removeRow(0);
        }
        chart.draw(data, options);
    });

    connection.start();

Here is the demo output in Browser

Complete sample code
Resources
1. SignalR Channel9 Video
2. SignalR on GitHub
3. Socket.IO

C# 5 Callerinfo Attribute

As application developer we log information related to execution (information or error) to log files. This information is helpful for tracing, debugging, and creating diagnostic tools. If the log message has file name, line number and function name it is easy to narrow down the issue file and function.

In C/C++ developer can use  __FILE__ , __LINE__ and __FUNCTION__ macros to print file name, function name and line information along with the log message.

Following code c++ code illustrate the usage

// logger function
void LogMessage(std::string caller ,std::string LogMessage)
{
	std::cout<<__FILE__<<":"<<__LINE__<<" - "<<caller<<" "<<LogMessage<<std::endl;
}
// main function
int main(int argc, char* argv[])
{
	LogMessage (__FUNCTION__,"Starting Application");
	return 0;
}

If you are .Net developer  prior to .Net 4.0  you would have used System.Diagnostics.StackFrameclass to get this information .

public class Program
{
	public static void LogMessage(String message)
	{
		// get access to caller stackframe
		StackFrame frame = new System.Diagnostics.StackTrace(true).GetFrame(1);
		String file = frame.GetFileName();
		int line = frame.GetFileLineNumber();
		String member = frame.GetMethod().Name;
		
		var s = String.Format("{0}:{1} - {2}: {3}", file, line, member, message);
		Console.WriteLine(s);
	}
	static void Main(string[] args)
	{
		LogMessage("Starting the application ...");
	}
}

In C# 5 and VB.Net 11 there is easier way of getting this information using CallerInfo attribute.By using Caller Info attributes, you can obtain information about the caller to a method.

You can obtain file path of the source code, the line number in the source code, and the member name of the caller.

Here is the sample which uses C#5 feature ( available in .Net 4.5 )

public class Program
{
	public static void LogMessage(    
	string message,
	[CallerFilePath] string file = "",
	[CallerLineNumber] int line = 0,
	[CallerMemberName] string member = "")
	{
		var s = string.Format("{0}:{1} - {2}: {3}", file, line, member, message);
		Console.WriteLine(s);
	}
	static void Main(string[] args)
	{
		LogMessage("Starting the application ...");
	}
}

More information about Callinfo attribute can be found here
1. MSDN