Monday, January 6, 2014

Tx.Windows Parsing WC3 IIS Log files in LINQPad and PowerShell

I saw a tweet today from the @ReactiveX account about a new release called Tx (LINQ to Logs and Traces). It appears to have a lot of plumbing to use the Reactive Extensions (Rx) to parse through Event Logs and Event Tracing files. One of the slides on the Tx Codeplex site also mentioned parsing IIS W3C Logs, which is something I spend a fair amount of time doing.

Even better Tx has a custom LINQPad Driver, which is one of my favorite tools for playing around with LINQ and Rx. I downloaded the sample traces and was able to view low level event tracing details from the HTTP.sys driver used by IIS, but I didn’t immediately see how to parse the IIS log files using Tx. After downloading the source I found a W3CTest.cs file that showed how to parse log files using the Tx.Windows.W3CEnumerable.FromFile or FromFiles methods.

To use this in LINQPad you do NOT need the Tx Driver, but instead you use F4 to add a reference to the Tx.Windows Nuget package:

image

image

Once the reference is added to your LINQPad query, you may also want to add the following additional namespace imports:

image

This gives you access to the common LINQ functions used in the Tx samples. Now I can write a query in LINQPad that uses the W3CEnumerable to parse the log files. You can write complex LINQ queries or use the toDataGrid parameter of the Dump method to display the data in a grid format:

image

After I had it working in LINQPad, I next thought: “Why can’t I use this in Powershell?”. The Tx library targets .NET 4.0 and 4.5, so as long as you have PS 3.0 or later you can add a reference to the Tx.Windows.dll and then use the W3CEnumerable class to work with Log files.

#Load Tx.Windows.dll from current user’s LINQPad nuget local cache
[Reflection.Assembly]::LoadFile("C:\Users\$($env:username)\AppData\Local\LINQPad\NuGet\Tx.Windows\Tx.Windows.1.0.31229\lib\Net40\Tx.Windows.dll")

#Load log file and sent output to screen
$log = [Tx.Windows.W3CEnumerable]::FromFile("C:\inetpub\logs\LogFiles\W3SVC1\u_ex131230.log")
$log | select -Last 10 |  ft dateTime,cs_method,cs_uri_stem,cs_uri_query,cs_username,sc_status,time_taken

#Sent log to gridview
$log | out-gridview

image

The Gridview is very helpful since you can add filters to target specific entries similar to a LINQ query

image

All this in the first hour of playing with Tx! I am really excited to see if we can use Tx with IIS Failed Request Tracing or ASP.NET Tracing to track down issues in our .NET web apps.

Tuesday, February 12, 2013

Samsung Epic 4G CyanogenMod Notes and Links

Up until a few months ago I very much enjoyed having the Samsung Epic 4G as my personal phone. It is the best Android phone with full QWERTY keyboard that I have ever used, but the hardware is a bit dated and the WiMax speeds were not as fast as LTE, so I decided to upgrade to the Galaxy Note II on Verizon. Samsung stopped updating the software on the Epic 4G a while ago, but the CyanogenMod community has done a great job at supporting the phone and even has Android Jelly Bean 4.1 ROMs with full functionality. I still have friends with the Epic 4G so I thought I would post all the links I used for loading custom ROMs.
Most recent versions (As of January 2013)

Sprint has Steps to activate phone, which includes instructions for activating a used phone. These may not work in CM 10, but I usually put a temporary boot ROM on the SD card that will boot into the stock FC09 ROM
Temporary Stock Boot Instructions:
  1. Boot into CWM5 on your Epic 4G
  2. Install zip from sdcard > choose zip from sdcard > multiboot > FC09 > boot_FC09.zip
  3. WAIT.  It will take a while to boot.  You might see a "com.android.phone is not responding", do not worry, it is harmless.  Wait until Media Scanning is complete before doing anything.
  4. Follow instructions for activating phone or calibrating tilt sensor
You can also revert the Epic 4G to original locked boot loader and stock ROM if you have any issues with the custom ROMs or have to take it in for service/support.
Hope this helps any Epic 4G users out there. If Samsung ever decides to make another Galaxy S phone with a keyboard definitely will consider buying it, but for now I am enjoying the Galaxy Note II with flygrip.

Wednesday, May 25, 2011

Looking for a .NET programmer in Salt Lake City, Utah? You should hire me!



UPDATE 6/01/2011: I have accepted a great position at GE healthcare that interestingly has very little to do with .NET and everything to do with making better software.

UPDATE 5/15/2013: Although I am still working for GE (Now part of a joint venture with Microsoft) I am also actively looking for new challenges in Software Engineering or Developer Operations.

My 30 second over-hyped elevator pitch:
The most hirable man in the worldMost people don’t have the time to read through a full resume nowadays, so I’ll keep things short and to the point. You should hire me! Why? Because I am smart and I get things done. The last time I was in the job market I turned a 3 week software testing project into a 6 year pursuit helping to convert a small ISV into a cloud driven heavyweight. I don’t always code in .NET, but it is definitely my platform of choice. Life is too short to worry about pointers and memory leaks. If I wanted to do that I’d go back to working in assembly language on an 8086 like God intended. I can automate just about anything using a combination of BASH, PowerShell, AutoIT, VBA, WatiN, MSAA, a bunch of HWNDs, a packet sniffer, a few servos, and a roll of duct tape. I work well as part of the A-Team or sent out solo in the wild like Rambo. In short: I am the most hirable man in the world! :-P Keep coding my friends!

I live my life like an open book, indexed by Google. I have a thirst for knowledge and a passion for technology. I pay great attention to detail and strive for nothing less than perfection. A lot of my past projects are internal or backend systems, but below is a sample of what has kept me busy over the last decade or so.



Personal Coding projects:
PhraseMeme Scanner, WinFone Alpha (Windows Phone, 2010-2011)
Silverlight port of ZXing Barcode Scanning library (Silverlight, C#, 2010-2011)
Texas Holdem Monte Carlo Simulator (Silverlight, Windows Mobile, 2009)
Resolver one: Texas Holdem, Pivot Charts/Auto filter, Web page automation, WPF Charts, Mad Lib generator (IronPython spreadsheet, 2009)
GnuMap P2P Mapping project (Visual Basic, Web Spider, aiSee, 2002)




Samples of documents I have created:
High level overview graph (Visio 2007)
Detailed technical overview chart (Visio 2009)
In depth technical documentation (Word 2008)
Short section of web based training material (PowerPoint 2010)
Samples of feedback reports w/ synopsis (Acrobat 2006)
Memo to customers re: Software Upgrade (Word 2010)
Custom Reporting Tools using SOAP web service (C#, ClickOnce, Access, Excel 2009)
Raw Documentation Files (2006 – 2011, I created the majority of the documents on here)




Screencasts and videos of project I have created:
PhraseMeme scanner demo video (3 mins, Camtasia, PowerPoint 2010)
WinFone Alpha demo video (3 mins, Camtasia, PowerPoint 2010)
Texas Holdem Monte Carlo Simulator (5 mins, Camtasia, PowerPoint 2009)




Coding Competitions:
PhraseMeme Scanner received good rankings in Windows Phone 7 Competition sponsored by Red Gate (2010)
Texas Holdem app received honorable mention in 2009 INETA Code Challenge and 2009 Telerik Silverlight contest
Monte Carlo spreadsheet was the winner of $4,000 prize in April 2009 round of Resolver One Challenge

Monday, January 17, 2011

InvalidOperationException from IsolatedStorageSettings.Save on Windows Phone 7

I just had my first failed test from the marketplace certification for PhraseMeme Scanner v1.2. You can view the full error report here, but the important part is:

Comments: The application terminates unexpectedly when the user turns on/off one of the setting while trying to scroll to

another page.

Steps to reproduce:

1. Launch the application.

2. Scroll over to the setting menu.

3. At the same time, press and scroll on any of the settings.

4. Notice the application terminates unexpectedly when the user turns on/off one of the setting while trying to scroll to

another page.

The settings page uses ToggleSwitches from the Silverlight for Windows Phone Toolkit with a simple two-way binding to a values stored in an IsolatedStorageSettings class, so I was a bit confused where the error was coming from. Using Visual Studio I found that the IsolatedStorageSettings.Save method was throwing an InvalidOperationException whenever the user changed a setting and navigated way from the page at the same time.

The MSDN documentation doesn’t mention that the method can throw this error, but my guess is that either the value is in an indeterminate state or there is a conflict between the save method and the two-way binding. The error could have been there all along, but in this version I had moved all the IO calls to background threads to increase performance where as previously the exception was captured by a try/catch block on the calling method.

Luckily it was an easy fix: I added a try/catch statement to the save code block and inserted a delay to let whatever processing it was conflicting with finish before it tried to save the settings. Below is the new code that can be used for saving settings stored in an IsolatedStorageSettings class:

/// <summary>

/// Save settings to isolated storage. Will run using background thread to prevent blocking UI thread.

/// </summary>

public static void Save()

{

    ThreadPool.QueueUserWorkItem(func => {//Save items using background thread.

        try

        {

            //NOTE: may throw InvalidOperationException exception if user is in process of changing settings and navigating away from page

            System.Threading.Thread.Sleep(200);

            issSettings.Save();

        }

        catch (Exception ex)

        {

            Debug.WriteLine("Error saving settings: " + ex);

        }

    });

}

 

Hope that helps someone prevent a failed submission to the marketplace.

Monday, December 20, 2010

Enable Blogger Mobile Templates Support for WP7 and Opera Mobile

Blogger recently released a new feature to their Blogger in draft platform that allows the website to automatically detect mobile browsers and render the page using a small-screen friendly template. This fixes a HUGE issue I have been working on for the last few weeks ever since I released the PhraseMeme Scanner application, namely publishing content that will be consumed by both large and small screen devices. Previously I was planning on having two version of the website with one being optimized for mobile and one for desktop, but now I can store all the content on Blogger and use the mobile templates to render for small-screen devices.

The Show mobile template currently works with the default iPhone and Android browsers, but it does not work with the Opera Mobile, Pocket IE, or Windows Phone 7 Internet Explorer browsers. I assume there are other mobile browsers that also don’t work, but there is a simple way to add support for any browser by using JavaScript to redirect users.  Here is the code that I wrote to detect mobile browsers and redirect accordingly:

The templates work by appending a m=1 parameter to the blog URL or (m=0 to force full webpage on mobile devices), so all that the above code has to do is look for that parameter and add it if a mobile device is detected. You can add the above script to your blog using either the blog template or the HTML/Javascript Gadget. Adding it to the blog template will cause it to run before the page is fully loaded, but you should save your current template first to make sure you don’t mess anything up;

I added a feature request for Blogger to support more mobile browsers, but for now the above script should let you start using this amazing feature.

Wednesday, December 8, 2010

Windows Phone Barcode Library (Silverlight C# ZXing Port)

I have been working on a Barcode Scanning application for Windows Phone 7 for about 3 months now, but none of my efforts would have been possible without the prior work done by Google on the open source ZXing (“Zebra Crossing”) barcode scanning library. This library is released under the Apache License v2.0 and has spurred development of many different barcode scanners on many different platforms. The idea for PhraseMeme Scanner came when I saw there weren’t any good scanners available for Windows Phone 7 and then stumbled across the partial C# port of ZXing. Through a bit of testing I was able to modify the library to support Silverlight and Window Phone 7, and decided I would help push the library into a well maintained and supported state in the Silverlight world.

While the primary purpose for updating the library is to build the "Best barcode scanning app available" I also want to give back to the community and help others create great windows phone or Silverlight apps. ZXing has their own Google Code project, but that site is mainly devoted to Android or Java development, so I decided to store the code on Codeplex to get it some .Net love. This also means that I can use Mecurial instead of SVN/CVS, which so far has been a great experience and should be very helpful if anyone else decides to contribute patches or new features to the library.

So without further ado, please check out the Windows Phone 7 Silverlight ZXing Barcode Scanning Library based on the csharp ZXing port created by Suraj Supekar at revision 1202 in the SVN repository. Let me know what you think, and happy coding!

Update 3/11/2011: It looks like the project is getting a lot of interest from developers with over 6,000 page views and 750 total downloads in the last few months. This is probably due to the various tweets on twitter and a blog post from Rudi Grobler about reading barcodes on WP7 using the Silverlight ZXing port. That blog post was mentioned on This Week in Channel 9, which totally made my day even if it was only talked about for 40 seconds and never even mentioned my name. Still… pretty wicked cool

Smile with tongue out

Monday, October 4, 2010

VPS/VM vs Dedicated vs Cloud Servers: Hosting options and cost comparisons

I usually talk about Code on this blog, but eventually that code is going to have to run somewhere so I thought I would outline some of the different server options that are available. While there are free or low cost options for hosting simple or small websites, if you are using ASP.NET, PHP, or Java to create a dynamic website or if you are looking for a file or application server then you are going to have to pay for the resources that you use (bandwidth, hardware, software, etc). The price/performance ratio can vary greatly depending on what type of server you purchase. There are a lot of different configurations, but here are some of my recommendations.

First, what not to do:

While in certain cases it may be appropriate, my recommendation is to steer clear of two specific hosting options: Shared web hosting and Do-It-Yourself hosting. Shared hosting refers to the $5 a month plans offered by Godaddy, Yahoo, and thousands of “resellers” that use a control panel like Plesk or cPanel to divide up a single server into hundreds of small websites. These servers are often over-sold and under-powered, and you will only get FTP and control panel access, so you will have to rely on your hosting provider to make any advanced configuration changes or perform troubleshooting steps for you. If you cannot afford any of the other options then Shared Hosting might be a good way to start, but keep in mind that you get what you pay for and will  probably out-grow it very quickly.

As for Do-It-Yourself hosting, this is simply a case of economies of scale: You wouldn’t start a wheat farm just to make a sandwich, so why build and maintain your own servers? Unless you are a corporation with a full IT department, it is not in your best interest to setup and maintain your own web or application servers outside of your personal testing/development environment. Servers are more or less a commodity nowadays, so while you could setup a computer in a closet of your office and use it to host your company website, you then have to worry about what happens when the power goes out or when your Internet connection goes down. Setting up and maintaining a reliable network with redundant paths and uninterrupted power takes a considerable amount of time and money. I highly suggest that you outsource this effort to a 3rd party and focus your energy on improving your website or application instead. And in terms of running your own Email and Calendaring server, you really, REALLY should look at paying Google, Microsoft, or someone else to provide these services for you if at all possible.

Virtual Private Server / Virtual Machine:

It use to be that when looking for a hosting provider you were either stuck with shared hosting or a dedicated server. Now there are many other options, most of which are based on server virtualization. While it may sound similar to Shared Hosting, VPS or VM hosting options can provide guaranteed resources and an isolated execution environment in which you have full control of the server. The hosting provider will purchase high-end server hardware and then use a hypervisor to divide that server up into multiple virtual machines for different customers (usually 2-24 depending on the size of the VMs). Each VM gets a certain percentage of the CPU, Memory, and Disk Space and will function as its own server. From my experience VPS or VM hosting provides the best bang for your buck at around $20-$120 a month (see Choosing a Hosting Provider below).

Dedicated Servers:

Some applications require significant horsepower, in which case a dedicated server is usually a requirement. You can purchase standardized “cookie-cutter” servers from most hosting providers, or you can also get a custom quote if you need to have a specific type of RAID setup or maximize RAM size or CPU speeds. Until recently I have primarily worked with dedicated servers that usually run anywhere from $100 to $1000 a month, so they are only viable in business environments that can justify the cost of the server. You also may have the option to rent, lease or purchase the server, but if you are purchasing you must factor in the time and costs for replacing failed hardware components in the future. Also if you work with sensitive information such as credit cards or medical records then you may require a dedicated server to meet Payment Card Industry (PCI), Health Insurance Portability and Accountability Act (HIPAA) or various other federal regulations.

Cloud IaaS and PaaS Servers:

Cloud hosting is the new hot term, but a lot of the time it means the same thing as VPS or VM Hosting except you pay by the hour instead of by the month. There are typically 3 types of “Cloud Computing”: IaaS, PaaS, and SaaS. For the most part Infrastructure as a Service (IaaS) is exactly the same as VPS or VM hosting just with a fancy API and pay-by-the-hour terms. Software as a Service (SaaS) doesn’t really have servers in the sense that you usually are riding on top of another application (Ex: Google Apps or Salesforce). Platform as a Service (PaaS) still uses Virtual Machines, but the platform is designed to help you scale your application horizontally across multiple nodes. Amazon Web Services (AWS) has some PaaS products, but their EC2 product fits more into the IaaS category (you have to scale-out your code to work on multiple servers on your own). Windows Azure, Rackspace Cloud Sites and Google App Engine are examples of PaaS, but they each have different programing models so you have to build your application around their platform. PaaS is designed to scale to meet your traffic needs and use multiple server nodes to provide better reliability, but this also means you will play a higher cost if you are running a 24x7x365 application. For instance, running two small web roles in Windows Azure to create a high availability website would cost $0.12*2*24*30 = $172.80 a month.

Choosing a Hosting Provider:

So now that you have an idea of the different types of hosting, all you have to do is find a hosting provider that offers a solution that meets your needs. I have worked with ASP.NET websites of various sizes for the last 5-10 years and can highly recommend the following companies based on my experience with them:
  • Superb.net: Offers all types of hosting ranging from shared to dedicated to virtual machines and custom quotes. Their VPS product is very affordable starting at $35 per month for a Win 2003 server or $20 per month for a Linux server. They also have great rates on dedicated servers starting at around $70-$80 a month. I had a server with them for about 10 years and they have always had great support and great prices.
  • AccuWebHosting.com: Recently I moved my Windows server over to a VPS hosted at AccuWebHosting. They are not as large as Superb.net, but they offer some of the the best rates on VPS servers starting at $23 per month for Windows or $16.50 per month for Linux. While not as powerful as a dedicated server the VPS servers work great for light web or application servers.
  • Rackspace.com: Offers fully managed dedicated servers and colocation services with “Fanatical Support”. They are a premium provider and cost a bit more than their competitors (starting around $500-$600 a month for a mid-range server), but if your application is mission critical you can feel comfortable turning over the keys to them and they will make sure your servers remain online and operational. Also Rackspace is the only vendor I have ever been able to get to sign a HIPAA compliant Business Associate Agreement (BAA).
  • Rackspacecloud.com: Originally called Mosso it was rebranded in July 2009 and offers Cloud CDN, Website, and Server solutions. They have managed servers starting at less than $100 a month, so if you want “Fanatical Support” without the high costs this might be a good place to start.
  • Amazon EC2: EC2 is the most mature of the online cloud providers and offers the most pricing options including Spot and Reserved instances. If you envision scaling out to hundreds of servers it is well worth your time to check out EC2. Their Simple Storage Service (S3) is also one of the best storage providers as it offers two price points depending on the level of durability that you require.
  • Windows Azure: Designed for high reliability and easier development, Azure is worth looking into if you are starting a new Microsoft .NET project that will require multiple servers. They are the new kid on the block, having been officially released in February 2010, but their use of Roles instead of servers means an application can be easily scaled up or down as needed. Also SQL Azure is a great way to create a hosted MS SQL database, as you pay $10 per GB per month, which combined with $0.15/GB/Month for blob storage means you should be able to store all your data and images/files for relatively cheap and support a large number of simultaneous users. Check out an introduction to Windows Azure for more information and watch the Cloud Cover show on Channel 9 for great coverage.
I hope this helps as you look at designing or scaling out your application or website. If you have questions I highly suggest asking them on one of the StackExchange websites such as ServerFault.com or Webmasters.StackExchange.com. There are usually lots of great people there willing to help answer questions.

Blog.TheG2.Net - Your guide to life in the Internet age.