Linux .bashrc equivalent on windows

In my current project I use Kubernetes & docker tools on Windows  10 . To interact with Kubernetes  and Docker environment typical we use kubectl & Docker command line tools , typing  these commands all day is not fun , so I was searching for solution to create some sort of shorter alias for these commands.

This is easy on Linux with ‘alias‘ command put it into  .bashrc , I was looking for similar solution on Windows . After googling around found that with .cmd file & some registry magic we can do the same.

Here are the steps .

        • Under HKCU\SOFTWARE\Microsoft\Command Processor added  value: AutoRun of type REG_EXPAND_SZ with value %USERPROFILE%\cmdrc.cmd. This is required to persist setting across reboots. reg
        • Created cmdrc.cmd in my userprofile directory.You can access the user profile with %USERPROFILE% environment variable.
        • Added following content in cmdrc.cmd file
          @echo off
          DOSKEY ls=dir
          DOSKEY cd=cd $1$Tdir
          DOSKEY clear=cls
          DOSKEY k8s=kubectl $*
          DOSKEY dk=docker $*


With this setting in place any new command instance will have shorter alias set in cmdrc.cmd

cmd

More informaton

Advertisements

Step by step guide install Windows 10 using boot to VHD feature

Windows 10 is coming this year, as technology enthusiast I always interested to tryout and explore features set and developer opportunities. In this blog post I will explain how to install and explore windows 10 safely on your computer without using virtual machine software or creating new partition in your hard drive.

Boot to VHD

Boot to VHD is feature in Windows 7 (higher) system , that allows to boot windows operating system from the VHD file ( virtual hard disk file ). Basically Windows 7 systems boot loader understands , how to boot from VHD file. I wrote blog post how to do it for Windows 7 (here). In this blog post I will show how to do it on Windows 8 .

First thing download windows 10 iso file from Microsoft site. You can download the iso file by be joining Microsoft Windows insider program.

Create VHD/VHDX file from the ISO.

  • Download the Convert-WindowsImage.ps1 PowerShell script from TechNet site that allows you to create VHD/VHDX file from ISO.Launch PowerShell command  prompt as administrator, execute command to enable execution of unsigned scripts ,

Set-ExecutionPolicy Bypass -Scope Process

  • Run downloaded script in interactive mode by executing .\Convert-WindowsImage.ps1 –ShowUI command . At the end of the execution it scripts creates vhdx file  in the working directory.

image

  • Mount the virtual hard disk image from working directory.

image

  • Once image is mounted  you should be able to see new drive in your explorer. In my case it is attached ‘G’ drive.

image

 

Update boot record.

  • Now run following command create new bootable entry. Replace G:\Windows with your location where vhdx image is mounted

bcdboot G:\windows

image

  • Verify that boot order is updated properly by running msconfig.exe. You should see new entry.

image

  • Restart computer , you should new boot option.

Uninstall Windows 10 (Removing boot entry).

If you want to remove boot entry you can follow the steps

  • Launch msconfig.exe from the admin command prompt  and select entry you want to delete and press delete, reboot.

image

 

 

Using Hazelcast ( In-Memory Data Grid )

Hazelcast is open source  in memory data grid that allows architects and developers easily design and develop faster, highly scalable and reliable applications.

Features

  • Distributed implementations of java.util.{Queue, Set, List, Map}.
  • Distributed Topic for publish/subscribe messaging.
  • Distributed listeners and events.
  • Dynamic clustering.
  • Dynamic fail-over.

Hazelcast is implemented in Java , it  uses Java NIO to implement non blocking I/O and it can dynamically discover and join the server cluster using IP Multicast .

Some use cases for  Hazelcast .

  • Designing highly scalable data store shared by multiple processes , multiple applications, or multiple servers.
  • Designing highly available distributed cache for your web application.
  • Designing distributed pub/sub.
  • Designing distributed queues.

Similar to Redis, Hazelcast can also be used to store session data in web application which can survive web server restarts and crashes. This setup also enables nice horizontal scalability of web application as shown below.

web-arch

As Event Router

One of the other uses case is to use it as distributed publish/subscribe server to  allows different application on the  same machine or across network to exchange messages in a loosely coupled fashion.

Following picture shows the architecture of such as system.  Clients ( publisher/subscribers ) connect to Event Router server to send/receive messages.

event-router

Server application source code

import com.hazelcast.core.*;
import com.hazelcast.config.Config;
import java.util.Map;
import java.util.Queue;
import java.io.*;
import java.util.*;
public class DemoRouter {	
    
    public static void main(String[] args) {
        
        Config cfg = new Config();		
        HazelcastInstance instance = Hazelcast.newHazelcastInstance(cfg);
        
        BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
        try {
            in.readLine();
        }catch(Exception exp) { }
        
        System.out.println("Exiting ..");		
    }
}

Subscriber client application source code

import com.hazelcast.core.*;
import com.hazelcast.client.config.*;
import com.hazelcast.client.*;
import java.util.Map;
import java.util.Queue;
import java.io.*;
import java.util.*;

public class SubscribeClient {

    public static void main(String[] args) {
        
        ClientConfig clientConfig = new ClientConfig();
        clientConfig.addAddress("127.0.0.1:5701");
        
        HazelcastInstance hz = HazelcastClient.newHazelcastClient(clientConfig);
        ITopic topic = hz.getTopic("default");
        topic.addMessageListener(new MessageListener<String>() {		
            public void onMessage(Message<String> message) {			
                String myEvent = message.getMessageObject();
                System.out.println("Message received = " + myEvent.toString());
            }
        });		
        BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
        try {
            in.readLine();
        }catch(Exception exp) { }
        System.out.println("Exiting ..");	
    }
}

Publisher  client application source code

import com.hazelcast.core.*;
import com.hazelcast.client.*;
import com.hazelcast.client.config.*;
import java.util.Map;
import java.util.Queue;
import java.io.*;
import java.util.*;

public class PublishClient {

    public static void main(String[] args) {
        ClientConfig clientConfig = new ClientConfig();
        clientConfig.addAddress("127.0.0.1:5701");
        
        HazelcastInstance hz = HazelcastClient.newHazelcastClient(clientConfig);
        ITopic topic = hz.getTopic("default");
        
        topic.publish("Hello from Client1");
        topic.publish("Hello from Client2");
        topic.publish("Hello from Client3");
        BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
        try {
            in.readLine();
        } catch (Exception exp) {
        }
        System.out.println("Exiting ..");
    }
}

Resource

1. Hazelcast documentation 

2. Using Hazelcast screencast

Extensible Storage Engine (ESE) : Storage option for large Sensor data

Currently I am evaluating the storage options to store large volume of data generated by sensors ( temperature,pressure,pH  … ) .  These sensor generate more than 1,000,000 data samples each day.  Goal is to select efficient storage option which is simple to deploy , easy to implement, and provides reliability against the application crashes.

Obvious choices include simple comma separated file ( CSV ) or embedded database solution. Each of the above mentioned options have its own pros/cons.

On searching  the internet for alternative solution found article on Extensible Storage Engine ( ESE ) which seems to be promising option . In this blog post I will discuss the same.

Extensible Storage Engine (ESE) is storage engine technology built into Windows since Windows 2000 . It is distributed has library (ESENT.DLL ) . It is used extensively inside many Microsoft  products  such as Active Directory, Microsoft Exchange  and Windows Desktop search. Outside Microsoft  it is  used in  RavenDB – a NoSQL document database built on top of “ESE”.

At high level “ESE” has following advantage over CSV and RDBMS

  • Zero-deployment  – Built into windows.
  • Extensively tested – Used inside Microsoft products and open source projects.
  • Easy to administer  –  Use file manipulation tools such as copy/move to backup the files.
  • Easy to use    – Managed .NET wrapper with LINQ support is available.
  • Others – Robust crash recovery, intelligent cashing, concurrent access to data and many more.

Even though ESE library is “C” API , there is managed .NET API available on codeplex website.

Codeplex project also provides an implementation of Persistent Dictionary . “PersistentDictionary” is a database-backed dictionary built on top of the ESENT database engine. It is a drop-in compatible replacement for the generic Dictionary<TKey,TValue>.

I did some  prototype implementation based on the sample code available at codeplex project  .This implementation  uses persistent dictionary to stores sensor data with key being the date time and value being the actual data.Initial performance results are good.

Here is the details about the sample application. This application generates 1,000,000 random sample data along with the timestamp. It uses date time has key and sensor data has value.

Make sure that you use the nuget to install managed ESENT .NET library.

Sample code

using Microsoft.Isam.Esent.Collections.Generic;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace SensorStorage
{
    ///
/// value type that stores sensor id
    /// and the actual sensor value.
    ///
    [Serializable]
    struct Sensor
    {
        public string Id;
        public float  Value;
    }

    class StoragePerfTest
    {
        public static int MAX_RECORDS = 1000000;
        public static int MAX_SENSORS = 16;
        // sensor id list
        public static List SensorIdList = new List
        {
            "46d77be8-1b6c-40e4-be90-657bba696db4",
            "84ccd877-e8be-4da8-81ae-1d7bda0ff2d7",
            "49be5ff1-bc44-4173-847f-33a9394d8787",
            "c41f8e8e-ca42-4248-8e2a-f047db76efcc",
            "b94bd698-9deb-4857-a685-f2b4920765ad",

            "3ff45aed-6c31-4079-8d83-6b0a6d2516ec",
            "d53d0cc8-debf-43a8-8ecf-37a9ff5a8afc",
            "2cb6ec16-da28-498f-baf3-9bcc661cd18e",
            "5b9843a4-db0d-4887-99f4-1b460eaa3c05",
            "7e3b9124-9816-4ecc-8d1c-95030bb93fc7",

            "01487c6a-cf7f-47ea-b870-58479ec5fd61",
            "ab271b46-b224-45e6-a8b7-1d04e20e0de2",
            "88c9d2e3-a411-446e-bf0a-d34e9278764d",

            "46a8c28f-07ab-4559-a9c7-5986d6e84baa",

            "df723ed8-70ee-4ca7-b8ae-df0902d481a3",
            "98e376bf-82e7-4bf3-96ab-f66cde5537bd"
        };

        ///
/// generate random value simulating the sensor data.
        ///
        ///
        static float NextFloat(Random random)
        {
            var buffer = new byte[4];
            random.NextBytes(buffer);
            return BitConverter.ToSingle(buffer, 0);
        }
        ///
/// gets next sensor id from the list simulating
        /// random sensor sending value to application
        ///
        ///
        ///
       static string GetNextSensorId(Random random)
        {
            int index = random.Next(0, MAX_SENSORS - 1);
            return StoragePerfTest.SensorIdList[index];
        }

        static void Main(string[] args)
        {

            if ( !PersistentDictionaryFile.Exists("SensorData"))
            {
                Console.WriteLine("No sensor data available");
                return;
            }

            if (args.Length == 0)
            {
                GenerateSensorData();

            }
            else if (args.Length == 2)
            {
                QueryRecords(DateTime.Parse(args[0]), DateTime.Parse(args[1]));
                return;
            }

            Console.WriteLine("Usage : StoragePerfTest.exe [<start_time> <end_time>] ");
            Console.WriteLine("To Generate sample data execute StoragePerfTest without any arguments . Example : StoragePerfTest.exe  ");
            Console.WriteLine("To query . Example : StoragePerfTest.exe \"7/27/2013 05:42:00 AM\" \"7/27/2013 05:44:00 AM\"");
            return;
        }
        ///
///  generate random records
        ///
        private static void GenerateSensorData()
        {
            Console.WriteLine("Deleting old sensor and generating new data");
            PersistentDictionaryFile.DeleteFiles("SensorData");
            Random random = new Random();
            Stopwatch watch = new Stopwatch();

            using (var dictionary = new PersistentDictionary<DateTime, Sensor>("SensorData"))
            {

                watch.Start();
                DateTime curr = DateTime.Now;

                for (int i = 0; i < MAX_RECORDS; i++)
                {
                    curr = curr.AddSeconds(1);
                    dictionary.Add(curr, new Sensor { Id = GetNextSensorId(random), Value = NextFloat(random) });

                }
                watch.Stop();
            }
            Console.WriteLine("Inserted {0} records in {1} miliseconds", MAX_RECORDS, watch.ElapsedMilliseconds);
        }

        ///
/// queries record
        ///
        ///
        ///
        private static void QueryRecords(DateTime st,DateTime et)
        {
            Stopwatch watch = new Stopwatch();
            using (var dictionary = new PersistentDictionary<DateTime, Sensor>("SensorData"))
            {
                watch.Start();
                var samples = from x in dictionary
                              where x.Key >= st && x.Key <= et
                              select x;
                int count = samples.Count();
                watch.Stop();
                Console.WriteLine("{0} Records Found for the query in {1} miliseconds", count, watch.ElapsedMilliseconds);
                foreach (var s in samples)
                {
                    Console.WriteLine("Key {0} = [{1},{2}]", s.Key, s.Value.Id, s.Value.Value);
                }
            }
        }

    }
}

Performance results

One my laptop ( 5400 rpm hd, dual core cpu with 4gb ) , I got following results.

  • Total records inserted : 1,000,000.
  • Database Size : 215MB
  • Add speed: ~18000 records/seconds into persistent dictionary.
  • Query speed : 120 records in 80 milliseconds to query the data in the middle of the sample data.

I will be doing further investigation to see the suitability of the library for the project .

Resource

  1. Codeplex project.
  2. MSDN Documentation.
  3. ESE Database Viewer.
  4. RavenDB developer blog on ESE

.

Install/Configure Jenkins continuous integration server on Linux.

In this blog post I will describe installing and configuring Jenkins Continuous integration server. This set-up was used for one of  my current project to build Linux Application software.

I used Ubuntu 12.04 virtual machine running inside VirtualBox for this tutorial.

Jenkins is an continuous integration server written Java . Jenkins monitor configuration management ( CVS,SVN,Perforce.. ) servers  for changes , such as source code check-ins. Once it detects changes it will update the local working directory with the source from CM and performs series of build steps defined by the user on the source code. These build steps could be simple as invoking the shell script or  build script such as make,ant .

Jenkins has many plug-ins available to extend its feature set and to integrate with other Software tools ( unit test, code coverage, code analysis ).

Jenkins is Java based software , hence it requires Java Runtime as prerequisite on a system.

Jenkins software is available as Debian package . To Install latest Jenkins software on Ubuntu  execute following steps from command line

image

Once the installation completes make sure that Jenkins server up and running by opening web browser and pointing to http://localhost:8080 .

If Jenkins running you should see following page in the web browser . This completes the installation.

image

Configuring Jenkins  with Perforce Configuration Management.

Jenkins natively support CVS and SVN CM. In my application I use Perforce for configuration management. Since Jenkins natively does not support Perforce CM I need to install Perforce plugin for Jenkins.   Installing plugin is easy, Select “Jenkins->Manage Jenkins” option from the menu , select available tab to list the plugin and select and install perforce plugin. Following picture shows the “Manage Jenkins” page.

Jenkins-inst-4-perforce

You can discover and install additional plug-in similar to perforce plugin from the page.

Configure source code build.

From the Jenkins main page select NewJob to create simple build step.

In the next page enter job name and select “Build free style software project” option . You can read about the various option by selecting help icon next to option.

Jenkins-inst-4-perforce-1

Next page allows user to enter Configuration management specific details, in case Perforce CM, it user name,password and server details as shown below. These options differ based on the selected Configuration management .

Jenkins-inst-4-perforce-2

Other configuration details such as source code depot ( Perforce specific ), workspace details , Poll interval ( how often Jenkins should poll CM for changes ), where to copy the build output   are entered on the same page . Once the Configuration Management specific details are entered next step is provide instruction Jenkins to build software.

For this demo I have selected option to “Execute Shell” option to enter build command. With this option you can enter any Linux shell command . I will Apache ant to build my source code . Jenkins will execute these command once it check-out source code from the CM . I have entered following command to illustrate the build setup

cd projecta-src/

echo “Build started…”

ant BuildRelease

echo “Build Ended”

Jenkins-inst-4-perforce-4

After saving this step Jenkins will start monitoring  Configuration Management for changes, if changes detected it will pull changes to local workspace and  execute the build scripts.

Resources:

1. Jenkins website.

2. Jenkins Plugin repository.

Embedded Linux application development

Currently I am working on project which is uses custom version of embedded Linux built using open embedded framework. This project also uses QT for all the UI development ,googletest for c++ unit test and boost c++ for all threading and asynchronous communication(boost.asio).

As part of development I have explored and collected web resources which helped me to explore ,understand and tryout the samples and tools. I thought  I would share the same in this blog post.

Boost C++ :

  1. The Boost C++ Libraries  : Excellent introduction Boost C++ libraries. It covers many boost libraries such smart pointers,function objects,boost.bind,boost asio,boost threads,boost file system and many more. It includes lot of examples on each topic.
  2. Guide to Boost asynchronous I/O :  Boost.asio is a library to develop responsive applications using  async I/O facilities built into moderns OS. It is built on top of the Operating System specific asynchronous IO facilities such as IO Completion ports on windows and  epoll and similar on UNIX based Operating systems. This tutorials covers  boost asio library in depth . It has numerous examples to illustrate the concepts used in library.
  3. Asio C++ Library : This tutorial is from the developer of boost.asio library. It covers tips and tricks and sample code to illustrate how to use boost.asio to execute arbitrary tasks using thread pool and asynchronous facilities.

QT :

  1. C++ QT training videos on YouTube :  Total 106 videos which cover QT basic to advanced concepts.
  2. QT Videos at QT project site : Collection screencast which covers wide range of QT topics from beginners to advanced presented by top QT training firms such as KDAB and ICSNetwork.

 

Embedded Linux

  1. Free Electrons :  One of best site for Linux and Embedded Linux. Lot of great free training materials.
  2. Embedded Linux conference video archive : Embedded Linux Conference is as vendor-neutral technical conference for companies and developers using Linux in embedded products.  This website has all session recordings of a conference from 2005 to till date .

googletest :

  1. Quick Intro to Google C++ Test framework : This article from IBM developer works technical library, gently introduces google test framework.
  2. Google C++ framework wiki :

C++ IDE

  1. C/C++ Remote development using Netbeans  : This is great article which explains how to setup Netbeans IDE for C/C++ development . It explains how develop/debug application on Windows targeting Linux Box. It uses gcc tools /gnu debugger  over SSH/Telnet.

Build Media Center PC using Raspberry Pi

Raspberry Pi is a credit card sized computer built using Broadcom SOC (system on chip) created by Raspberry Pi foundation. It costs $35 for model B board which has 700 MHz ARM CPU, 512 MB RAM, 2 USB2 ports, 1 HDMI port ,1 RCA video port,Ethernet and audio port. It has SD card for installing Linux distribution.

Here is screenshot of Raspberry Pi board

rasppi

There are number of Linux distribution that target Raspberry Pi , including distribution to run Media Center software.

In this post I will talked about how I built my media center pc using Raspberry Pi and OpenELEC ( open embedded Linux entertainment center ) Linux distribution.

OpenELEC is a small Linux distribution with complete XBMC media center software. Compared to Raspbmc which is another Linux based media center distribution , OpenELEC is very small around 80MB in size. Because OpenELEC distribution  packages only software necessary for media center it is small and fast compared to Raspbmc.

Here is the screenshot of my Raspberry Pi board with all necessary connection .(HDMI connection to monitor, micro usb power connection, Ethernet connection and USB keyboard and mouse).

2012-12-27 19.55.18

Creating bootable SD image

Before powering up the board , you need download OpenELEC distribution from the OpenELEC website. Make sure that you download image which as RPi-arm name in it,  example : OpenELEC-RPi.arm-devel-20121124031454-r12577.tar.bz2

To create bootable OpenELEC image on SD card you can Windows/Mac/Linux OS workstation with SD card reader. Here are the steps to create SD image on Ubuntu 12.10 .

1. Unzip and un-tar OpenELEC to some folder ,

2. Insert 2GB blank SD card to your machine.

3. Execute create_sdcard shell script from OpenELEC folder with device path of the SD card.

Example

linux-vm:~/OpenELEC-RPi.arm$ ./create_sdcard  /dev/sdb

4. Once step 3  completes without any errors, SD card will have bootable OpenELEC image.

5. Now insert SD card to Raspberry Pi board and power up.

6. You will see the boot screen with OpenELEC and Raspberry Pi logo.

2012-12-28 11.34.03

7. After while you see familiar XBMC media center interface.

2012-12-28 11.35.00

8. Now you navigate the menu using keyboard and add your data source where you stored your media collection.

OpenELEC supports UPnP, DLNA,NFS,CIFS,SMB,FTP,SFTP and others file sharing protocols. Using these you should be able to browse the media collection stored on your PC , MAC, Linux or NAS box.

You can also connect your USB2 hard drive to Raspberry Pi and browse media collection from OpenELEC.  OpenELEC supports hard drive formatted using NTFS ,FAT/FAT32 and other Linux file system formats.

Once you have contents you play any audio/video/photo formats.

OpenELEC also supports Apple AirPlay as target. You can enable this from settings menu. Once enabled , you can stream photo and video from you IOS device( iPhone ,iPad ,iPod touch and MAC computers )  to OpenELEC connected device ( TV or monitor )

In the next blog post I will explore installing Java 8 on Raspberry Pi and running Java FX application.

Resource

1. Raspberry Pi : Main Raspberry Pi website.

2. OpenELEC : Open Embedded  Linux media center website.

3. OpenELEC on Raspberry Pi :  Instructions to install OpenELEC on Raspberry Pi

4. XBMC : XBMC wiki page .

5. Raspbmc :  Raspberry Pi media center distribution based with XBMC software.

6. Run Java Application on Raspberry Pi. Oracle TechNet article for installing Java on Raspberry Pi

7. Raspberry Pi accessories :  Collection of Raspberry Pi compatible accessories.