Latest Posts
Yes, Heroku made me sad today. Heroku!? What?! I can hear you saying it! =P Fortunately I was just coming down off of a fun trip after refreshing myself on the Codeacademy ruby course. Really nothing there if you already use ruby quite a bit, but I haven’t written any ruby in a couple years and it’s always nice to remember how pleasant that language is to work in. That ruby high is all that kept me plugging away at putting dex’s evil twin ruby brother ‘fwad’ out on Heroku. But first… how did Heroku make me sad? Well FIRST-first, a little backstory.
Lately I have been doing a lot of research into cloud providers and looking at various hosted solutions for all sorts of things. So fiddling with these two services over the last couple of days gave me some time to both play with ruby and play with Heroku and AppHarbor, two pretty popular application hosting/cloud services. I originally had the idea used for dex in ruby. I wrote a local client in ruby that did everything except stream the data in a response. I say “everything” but it should be in quotes the first time as well because I just wrote something simple that would download a file and write it to disk. On a whim, I also thought “hey, why not have it gzip the data as well?” So that’s what I wrote. In fact, here is the local client in all of it’s glory:
require "open-uri"
require "zlib"
url = 'http://somesite.with/an/exe/todownload/somefile.exe'
gzf = 'c:\temp\somefile.exe.gz'
open(url) { |rf| #open remote file
File.open(gzf, 'w+b') { |lf| #open local file/stream
Zlib::GzipWriter.open(lf) { |gz| #open gzipper on local file/stream
gz.write(rf.read) #write the compressed data
}
}
}
Not too glorious, eh? Well it was meant to be really really simple. I think I spent like 60 minutes getting ruby installed and writing this little dealio. It worked fine and thus began my campaign to somehow promote it, as a web app, to Heroku. I already had an account from when they were in beta, how hard could it be? Well, apparently for someone who works primarily on windows, without git, and without normal ruby DEV tools installed, it’s a bit of a pain. Now, I will say that it isn’t really Heroku’s fault that I was somewhat ignorant of what I wanted at the time. But in my defense the first thing I did was try and get a simple hello world site setup as described in their Getting Started with Ruby on Heroku tutorial here: https://devcenter.heroku.com/articles/ruby. Now, I’m not going to rehash my entire silly experience. But I will say that I tried to follow the directions and I still ended up having to piecemeal my solution together to get the initial Sinatra application to work. The main culprit in my whole story was this snip from that page:
Use a Procfile, a text file in the root directory of your application, to explicitly declare what command should be executed to start a web dyno. In this case, you simply need to execute the web.rb using Ruby.
Here’s a Procfile for the sample app we’ve been working on:
web: bundle exec ruby web.rb -p $PORT
If you’re instead deploying a straight Rack app, here’s a Procfile that can execute your config.ru:
web: bundle exec rackup config.ru -p $PORT
Somehow I missed the top one and put the second one in. So my app kept crashing and I had to become more familiar with debugging stupid self inflicted errors on Heroku than I wanted to. Anyway, once I figured all that out and all the other silly stuff, I ended up uploading a straight rack application. I found this link (http://rubylearning.com/blog/a-quick-introduction-to-rack/) to be very helpful. So this process took me on and off almost a day to figure out and get the code written/pushed/etc. I went with rack because by the time I was done with this I had already long since rewritten the app in C# (after downloading and installing the vs. web express as I didn’t have MS web DEV tools installed, only Win/Console App tools) and deployed it to AppHarbor. that includes setting up CodePlex, etc., etc. Since I hadn’t gotten my hello world app up, I hadn’t finished the ruby version, so I wrote the C# version first really. I say that only because I can’t say I ‘ported’ the ruby code. So back to why I went with rack, it was just because I could write all the code in one page and keep it really simple.
So what mad me sad is it was so much fun to write my little ruby script and then it was so much more of a pain to get it up to Heroku than I thought it would be. I realize that it was not Heroku’s fault really, but maybe they should have an even MORE handholding tutorial =P I felt like I had to click on 100 different links to go get a tutorial on this thing or that thing. really I would have been happy to just have them generate the app in their tutorial and then let me do a git from some prebuilt repo and I could edit it from there. I’m sure not everyone wants that, but it really would have helped me out today.
So here, finally, is FWAD( firewall avoidance downloader! =D )! I haven’t decided if I like the pronunciation ‘EF’-‘WAD’ or ‘fWAD’ better yet.
Anyway, the entire contents of the app is a Gemfile with:
source 'http://rubygems.org'
gem 'rack'
And a config.ru with:
require 'open-uri'
require 'zlib'
run lambda {|s|
headers = {}
response_body = ''
path = s['PATH_INFO']
path += '?' + s['QUERY_STRING'] if s['QUERY_STRING'] != ""
key = 'YaTl7a4akBMefeCZ'
op = path[1,16]
begin
if (op == key || op == key.reverse)
url = path[18,path.length-18] + '.exe'
response_body = open(url,'rb'){|i|i.read}
if (op == key)
ct = 'application/octet-stream'
cd = 'attachment; filename=renameme.txt'
else
ct = 'application/x-gzip'
cd = 'attachment; filename=file.gz'
bn = File.basename(url)
bn = bn.include?('?') ? 'renameme.txt' : bn
output = StringIO.new
gz = Zlib::GzipWriter.new(output)
gz.orig_name = bn
gz.write(response_body)
gz.close
response_body = output.string
end
headers["Content-Type"] = ct
headers["Content-Disposition"] = cd
else
response_body = 'no dude.... just... no.'
end
rescue
response_body = 'no dude.... just... no!'
end
headers['Content-Type'] ||= 'text/plain'
headers['Content-Length'] = response_body.length.to_s
[200, headers, [response_body]] #we lie and say we're always ok =D
}
Note: You do still need a procfile and a Gemfile.lock to push up to Heroku so it will work. It works similar to dex however, since I could intercept the call before it went anywhere, I went ahead and ghetto handled my own routes. I used the same key as a dex for uncompressed download to txt and I set the reverse of that key to be the route for compression. I also kinda went cheap on embedding the filename in the zip. If someone sends a filename that is in the querystring, I piecemeal that in earlier in the code, but File.basename in ruby will see a full filename as something like “filedownloader.php?myfile.exe”. So I just avoid that whole mess and if there is a ‘?’ (meaning a querystring) I just send gzipped renameme.txt.
Here’s a sample link to download LINQpad gzipped- http://fwad.herokuapp.com/ZCefeMBka4a7lTaY/http://www.linqpad.net/GetFile.aspx?LINQPad
Here’s a sample link to some weird googlecode uncompressed- http://fwad.herokuapp.com/YaTl7a4akBMefeCZ/https://lightpack.googlecode.com/files/PrismatikSetup_5.9.4
Here’s a link to the troublemaker exe that started all this - http://fwad.herokuapp.com/YaTl7a4akBMefeCZ/http://inedo.com/files/buildmaster/sql/3.5.8
Pretty simple, you just call the service with the appropriate route and some path to an exe without the exe on the end and it will send you a file.
Moral of the story: know what you are doing before you start doing it. don’t expect magic tutorials to save you. =)
A few days ago I was reading one of my usual news feeds The Daily WTF and saw Alex’s post (A (Long Overdue) BuildMaster Introduction) on their BuildMaster product. It looked pretty cool so I downloaded it myself and forwarded it to a colleague at another place of employment. Unfortunately for that individual their firewall policy prevents downloading .EXE files so they could not download any of the installs as they were all .EXE files.
So firstly, I would recommend that anyone that wants to have their cool/nifty product downloaded by someone in a large corp should at least *offer* the option of getting a compressed version that is not an MSI or an EXE file. I’ve seen these blocked at a number of institutions and while they are blocked for good reason, if you want to start a guerilla campaign with developers, it’s easier if they can get your stuff.
Secondly, to circumvent this PITA, I present ‘DEX’ which stands for ‘download EXE’. =) If you are one of the frustrated folks who can’t download EXE files, feel free to use this ‘new’ and ‘innovative’ service =P. You will probably want to let me know as I may change the key from time to time.
It’s very simple.
- Here is the URL format - http://royashbrook.apphb.com/dex.ashx?k=[key]&u=[urltoexe]
- It just processes the url string looking for the values of k,u, and one not shown here gzs.
- k must be YaTl7a4akBMefeCZ currently. It’s just a random key now and will change from time to time. I copied it from the text of a password generated here. This is where I remind you again to just drop me a line if you are using it.
- u should be the url to the exe file you want WITHOUT THE .EXE AT THE END.
- The service will then call ‘u’ with an added .EXE at the end and stream it down to you as renameme.txt.
- If you include gzs=1 it will gzip the stream and try and friendly name it for you although that doesn’t work all the time.
Here are some sample URLs:
Codeplex project is at http://dex.codeplex.com/.
so about… oh… 5 years ago, I posted this blog entry:
http://www.drowningintechnicaldebt.com/RoyAshbrook/archive/2007/03/09/passing-csv-to-sql-is-bad-mmkay.aspx
I recently needed to do something where I was passing in something like a csv to a sql proc and was reminded of it, so I looked it up. I noticed that I really didn’t have a good code sample on the old blog. I happened to notice a similar question on stackoverflow from 2010:
http://stackoverflow.com/questions/2312581/comma-delimited-sql-string-need-to-separated/15668344#15668344
So, I thought now would be a good time to post an updated code sample of this. Below is a method of simply passing a string to check (bad) and a way to pass that same value in xml (good). There are lots of ways to split the strings and do other things, but I like this way. Of course you can do this in LINQ now with a .contains pretty easily, but this is a good way to do it in pure sql.
--table with some test data
select * into #mytable from (
select 1 [id],'one' [value] union all
select 2,'two' union all
select 3,'three' union all
select 4,'four' union all
select 5,'five') st
--bad way passing raw csv string data in
declare @csv varchar(50)
set @csv = '1,3,5'
declare @sql nvarchar(2000)
set @sql = 'select * from #mytable where id in (' + @csv + ')'
exec sp_executesql @sql
--vs
--xml document with root, using attributes
declare @xml as xml
set @xml = '<root><v id="1" /><v id="3" /><v id="5" /></root>'
select
t.*
from
#mytable t join @xml.nodes('//v') x(n)
on n.value('@id','int') = t.id
--simple collection of xlm nodes
declare @xml2 as xml
set @xml2 = '<v>1</v><v>3</v><v>5</v>'
select
t.*
from
#mytable t join @xml2.nodes('/v') x(n)
on n.value('.','int') = t.id
drop table #mytable
I have been editing a *lot* of Visio diagrams recently. One of the issues I have with a lengthy session of Visio editing is the documents all get kind of wonky sometime because I have zoomed a ton or something. This isn’t a big deal but if I want to screen share and blow through the slides really quick to review it’s a pain. Here’s a macro to fit each visio window to page. I call it FWAP (Fit Window for All Pages) =).
Sub FWAP()
Dim OriginalPage As Visio.Page
Set OriginalPage = ActivePage
Dim PagObj As Visio.Page
For Each PagObj In ActiveDocument.Pages
ActiveWindow.Page = PagObj.Name
ActiveWindow.ViewFit = visFitPage
Next PagObj
ActiveWindow.Page = OriginalPage
End Sub
So far my only sadness is that Visio 2010 doesn’t appear to have an option to just add the macro to the quick bar. Ah well, hotkeys to the rescue I suppose.
A few months ago my kids and I started up another round of playing Minecraft. The only issue was my daughter had forgotten her password and she lost the password to the email she registered it on and she couldn’t remember her questions etc to recover the password.
So as I was looking for ways to recover her password I came across a way to recover it from the lastlogin file used by Minecraft if you check the ‘save password’ box, which she had done fortunately months previously when she last played. I used the Java code below and it worked like a champ. I can’t remember exactly what source I used for this and I modified it slightly, but if you search for “43287234L” you will probably be able to find a lot of the original links.
Hopefully it helps someone. Saved me a boatload of time ‘not playing’ while we worked to eventually recover all of the accounts and passwords.
import java.io.*;
import java.util.*;
import javax.crypto.*;
import javax.crypto.spec.*;
public class mcreco {
public static void main(String[] args) throws Exception {
String p = "Path To Your lastlogin file for minecraft";
String r = mcd(p);
System.out.println(r);
}
public static String mcd(String p) throws Exception {
PBEKeySpec pk = new PBEKeySpec("passwordfile".toCharArray());
String ci = "PBEWithMD5AndDES";
String output = null;
Random random = new Random(43287234L);
byte[] salt = new byte[8];
random.nextBytes(salt);
PBEParameterSpec pbeParamSpec = new PBEParameterSpec(salt, 5);
SecretKey pbeKey = SecretKeyFactory.getInstance(ci).generateSecret(pk);
Cipher cipher = Cipher.getInstance(ci);
cipher.init(2, pbeKey, pbeParamSpec);
DataInputStream dis = new DataInputStream(
new CipherInputStream(
new FileInputStream(new File(p)), cipher));
output = dis.readUTF() + " | " + dis.readUTF();
dis.close();
return output;
}
}
So we just moved and setup new internet. We are using Comcast for our internet here as we were at the last place. We got 30MB speed, but we were assured on the phone that they were upgrading the 30MB connections to 50MB on 4/1 for no charge to any of the customers. This was in response to our lamenting there being no package realistically priced between 30MB and 105MB.
It’s pretty fast so far. I took about 5 different samples from speedtest.net and they were all about the same. Here’s one of them:
I’ll have to check back in April to see if the sales guy was lying.
Note: I wanted to put a link to the current plan we were offered, but it doesn’t appear to be on Comcast’s site. Now *all* of the new plans are 20/50/105. Go figure. =P
So I updated SimpleSniffer with what I learned from all of my playing about lately. It seems quite fast to me. I have updated the latest release as ‘stable’ on CodePlex.
12 whole downloads so far, so maybe it helped someone else =P
Recently I have written a few blog entries about basic sniffer code. One of the most basic things that a sniffer has to do is gather packets and break out some data. My sniffers were all very simple, really only dealing with analyzing the packet headers themselves for IP and TCP headers. Within the headers there were a few values that I was concerned with that were multi-byte values. I have already written about this in my last blog post, but I thought I would post about something I noticed that really impacts the crazy problem of early optimization.
The three fields that I cared about that had multi-byte values were Size, Source Port, and Destination Port. For all of these values, given two bytes of data I found 2 ways to gather the data and present it. There are several conversions that must take place. The data is stored in a byte array, but it’s not in the correct order for most windows systems. The Endianness on windows systems (in general) is Little-Endian. But the order for network communication (in general) is Big-Endian. You can search on google for more info on Endianness and Endian related drama as I don’t really want to go into that here.
What I do want to go into is my false-positive optimization. I found that the following two methods can be used to get the data out of the Big-Endian value and convert it to a string:
((ushort)(buffer[0] << 8 | buffer[1])).ToString();
((ushort)IPAddress.NetworkToHostOrder(BitConverter.ToInt16(buffer, 0))).ToString();
Why does it need to be a string? Because ultimately I am presenting this data to the screen for consumption. The results are basically the same though even if you just get the raw data and don’t convert it to short or string.
In any case, basically the two methods are:
- shift the first bytes bits left by one byte and bitor it with the second byte to get a total
- use BitConverter.ToInt16 to get the int value from the 2 bytes, but it will be in incorrect byte order so you can fix it with the IPAddress.NetworkToHostOrder call.
There are most likely other permutations of this conversion, but these are kind of my ‘meta’ two scenarios to test and what is in the code above. I went with various different implementations of one or the other depending on my mood while I was fiddling with the sniffers, but once I was done with that, I wanted to pick one at least in my own mind as a ‘standard’ for me to use. So I decided to write up a quick test. The simplest way to test something seemed to be to just run it repeatedly and remove as many extra variables as I could. So put each method in it’s own action, wrap it with a stop watch, then run that action X times and record some total for which was fastest. In this case, I just ran them both in sequence and checked to see which ran fastest and created a ‘win’ counter for one of them to check against. I figured bitwise operations would be fastest, so I made that my first method, Method A. I then decided I wanted to run a large sample repeatedly against random data so I wrapped the loop call in another loop call that would produce a random value between 0 and 64k (the max value for 2 bytes) and then check it. Here is that code:
void Main()
{
//function to measure an actions execution time
Func<Action,int,TimeSpan> ma = (m,c) =>{
Stopwatch s = new Stopwatch();
s.Start();
for (int i = 0; i < c; i++)
m();
s.Stop();
return s.Elapsed;
};
Random rnd = new Random(); //randomizer for our random numbers
int aw = 0; //counter for how many times A Method was faster
int rnc = 10000; //random number counter - how many random numbers to try
int tc = 1000; //test counter - how many tests to run against each random number
byte[] buffer = null; //init buffer for our Actions to use
//the two Actions to test
Action a = () => { var v = ((ushort)(buffer[0] << 8 | buffer[1])).ToString();};
Action b = () => { var v = ((ushort)IPAddress.NetworkToHostOrder(BitConverter.ToInt16(buffer, 0))).ToString();};
//get this many random numbers
for (int r = 0; r < rnc; r++)
{
//populate the 'buffer' with some value up to 64k
buffer = BitConverter.GetBytes((UInt16)rnd.Next(0,65535));
//if the 'a' method is faster, increment it's win counter,
// test each random number 'tc' number of times
if(ma(a,tc) < ma(b,tc))
aw++;
}
//show the results
Console.WriteLine("Wins - A:{0} B:{1} - AWin% = {2}%", aw, rnc-aw, ((float)aw/rnc)*100);
}
This produced:
Wins - A:8834 B:1166 - AWin% = 88.34%
Wins - A:8968 B:1032 - AWin% = 89.68%
Wins - A:8834 B:1166 - AWin% = 88.34%
And lots of other numbers like that. Seems clear that Method A is the winner!!!
But wait! What happens if instead of running each random value for 1000 loops, we just run it once. I mean, that’s not *really* a true stress of each method, but that’s what we are going to be doing anyway (one packet, gets one cycle for it’s buffer). Hmm…. Let’s change ‘tc’ in the above code to 1… or maybe even 10…
int tc = 1;
Wins - A:2893 B:7107 - AWin% = 28.93%
Wins - A:3507 B:6493 - AWin% = 35.07%
Wins - A:3458 B:6542 - AWin% = 34.58%
int tc = 10;
Wins - A:3343 B:6657 - AWin% = 33.43%
Wins - A:3449 B:6551 - AWin% = 34.49%
Wins - A:3449 B:6551 - AWin% = 34.49%
Interesting. What gives? Well, I don’t know exactly. Here is the IL that is produced within LinqPad:
<>c__DisplayClass4.<Main>b__1:
IL_0000: nop
IL_0001: ldarg.0
IL_0002: ldfld UserQuery+<>c__DisplayClass4.buffer
IL_0007: ldc.i4.0
IL_0008: ldelem.u1
IL_0009: ldc.i4.8
IL_000A: shl
IL_000B: ldarg.0
IL_000C: ldfld UserQuery+<>c__DisplayClass4.buffer
IL_0011: ldc.i4.1
IL_0012: ldelem.u1
IL_0013: or
IL_0014: conv.u2
IL_0015: stloc.1 // CS$0$0000
IL_0016: ldloca.s 01 // CS$0$0000
IL_0018: call System.UInt16.ToString
IL_001D: stloc.0 // v
IL_001E: ret
<>c__DisplayClass4.<Main>b__2:
IL_0000: nop
IL_0001: ldarg.0
IL_0002: ldfld UserQuery+<>c__DisplayClass4.buffer
IL_0007: ldc.i4.0
IL_0008: call System.BitConverter.ToInt16
IL_000D: call System.Net.IPAddress.NetworkToHostOrder
IL_0012: conv.u2
IL_0013: stloc.1 // CS$0$0000
IL_0014: ldloca.s 01 // CS$0$0000
IL_0016: call System.UInt16.ToString
IL_001B: stloc.0 // v
IL_001C: ret
The calls out to the .NET libraries definitely make the code shorter. and we have to make two calls to the buffer for ours. I assume that the .NET team’s stuff is just faster than my humble shifting. I will also say that in my original code, I just converted the shift formula to a string without going to ushort first and that was slower every time. So it pays to do a little testing, but not too much. I could have just stuck with the BitConverter and the NetworkToHostOrder conversions. In many cases I didn’t even really register a tick difference unless I went with at least 10 cycles, that’s why I ever started at 1000 per number because it gave me a seemingly significant number to compare for the two operations. But even though the shifting is faster if it’s run a lot (and it’s much faster if you run tons and tons more cycles) it’s not relevant because you only will ever in practice (for my purposes anyway) run that cycle a single time.
So I guess as is normally the case, just use the libraries that MS gives you for tasks like this. If they gave you a method for it, it’s probably faster than just doing it yourself for normal use. =P
Some Links:
So I recently wrote a little sniffer in C# and decided I wanted to convert it to PowerShell. I ran into a little weirdness with the way that the bitwise operators worked in PowerShell. But before I talk about that let me describe the two ways to port your C# app to PowerShell.
- Just include the .NET code and call it
- Actually write it in PowerShell
Method 1
So #1 was very easy. Basically you just add the c# code as a type and then you can invoke it. Here’s some pseudo-code:
1: Add-Type = @'some code here with MyClass class and MyMethod method'
2: [MyClass]::MyMethod()
Below is a slightly modified version of my previous code. This runs as a synchronous
Add-Type @'
using System;
using System.Net;
using System.Net.Sockets;
public static class Sniffer{
public static string ToProtocolString(byte b){
switch (b){
case 1: return "ICMP";
case 6: return "TCP";
case 17: return "UDP";
default: return "#" + b.ToString();
}
}
public static void LogBuffer(byte[] buffer){
Console.WriteLine(
"{0} - {1} - {2}:{3} ===> {4}:{5} - {6} bytes"
, DateTime.Now.ToString("yyyy,MM,dd,HH,mm,ss,fff")
, ToProtocolString(buffer[9])
, buffer[12] + "." + buffer[13] + "." + buffer[14] + "." + buffer[15], buffer[20] << 8 | buffer[21]
, buffer[16] + "." + buffer[17] + "." + buffer[18] + "." + buffer[19], buffer[22] << 8 | buffer[23]
, buffer[2] << 8 | buffer[3]);
}
public static void Sniff(string ip)
{
Socket s = new Socket(AddressFamily.InterNetwork, SocketType.Raw, ProtocolType.Unspecified);
s.Bind(new IPEndPoint(IPAddress.Parse(ip), 0));
s.IOControl(IOControlCode.ReceiveAll, BitConverter.GetBytes(1), null);//promiscuous
byte[] buffer = new byte[s.ReceiveBufferSize];
while(true){
s.Receive(buffer);
LogBuffer(buffer);
}
}
}
'@
#call the sniffer
[Sniffer]::Sniff("127.0.0.1")
Method 2
So Method 2 is also ‘fairly’ straight forward. I ran into some issues with the bitwise operators that caused me enough pain to warrant mentioning it here. The basic problem was that .NET automatically would convert the values to int when it read them out of the byte and PowerShell did not. So all I had to do is cast the byte values to int prior to performing the bitwise operations and all was well. =) The only thing that was a bummer about this is I had less knowledge about bitwise operation than I would like and I couldn’t visualize the problem. so I had to write a third sniffer that simply output the different value portions of the bitwise operation to see which piece was breaking. In any case, here’s the PowerShell version:
#Resolves Protocol Numbers to Strings
function ToProtocolString($p){
switch ($p)
{ 1 {"ICMP"};6 {"TCP"};17 {"UDP"};default {"#$p"};}
}
#Lobs Buffer data we care about to the screen
function LogBuffer($b)
{
"{0} - {1} - {2}:{3} ===> {4}:{5} - {6} bytes" -F
([System.DateTime]::Now.ToString("yyyy,MM,dd,HH,mm,ss,fff")),
(ToProtocolString($b[9])),
([string]$b[12] + "." + [string]$b[13] + "." + [string]$b[14] + "." + [string]$b[15]),
([string] ([int]$b[20] -shl 8 -bor [int]$b[21])),
([string]$b[16] + "." + [string]$b[17] + "." + [string]$b[18] + "." + [string]$b[19]),
([string] ([int]$b[22] -shl 8 -bor [int]$b[23])),
([string] ([int]$b[2] -shl 8 -bor [int]$b[3]))
}
#Sniff traffic on an IP address on your machine
#create a socket, set it up, bind to it, switch to promiscious mode, loop, receive data, echo data
function Sniff($ipaddress)
{
$s = new-object system.net.sockets.socket(
[Net.Sockets.AddressFamily]::InterNetwork,
[Net.Sockets.SocketType]::Raw,
[Net.Sockets.ProtocolType]::Unspecified)
$s.bind((new-object system.net.ipendpoint(
[net.ipaddress]$ipaddress,0)))
$null = $s.iocontrol(
[Net.Sockets.IOControlCode]::ReceiveAll,
[BitConverter]::GetBytes(1),$null)
$buffer = new-object byte[] $s.ReceiveBufferSize
while($True){
$null = $s.Receive($buffer)
LogBuffer($buffer)
}
}
#call the sniffer
Sniff("127.0.0.1")
Conclusions, or something like them..
Both scripts ran for me with no noticeable differences. Piping the output to a file if they both were started at the same time yielded the same information. There was no appreciable difference visible in Task Manager when I was utilizing the two scripts simultaneously. Interestingly the PowerShell version appears to use more CPU although I couldn’t tell without querying the win32_process for the details. I guess PowerShell must be re-initializing some values or variables or something on each pass. Below is maybe 30 minutes worth of data. Every time I ran the packages simultaneously I got similar results.
PS C:\temp> gwmi win32_process | where {$_.CommandLine -like "*BasicSniffer*"} | select CommandLine, PeakPageFileUsage, ReadTransferCount, WriteTransferCount, KernelModeTime, UserModeTime | format-table -auto
CommandLine PeakPageFileUsage ReadTransferCount WriteTransferCount KernelModeTime UserModeTime
----------- ----------------- ----------------- ------------------ -------------- ------------
powershell .\BasicSniffer.ps1 68308 1361050 6205 2496016 28860185
powershell .\BasicSnifferDotNet.ps1 62796 1377036 8129 2652017 3588023
Unfortunately the inability to figure out the bitwise operator thing distracted me quite a bit. I will also point out that you could just use a call like ((ushort)IPAddress.NetworkToHostOrder(BitConverter.ToInt16(buffer, 20))).ToString() to get the values I am performing bitwise operations on. But I didn’t want to do it that way. Actually I didn’t really care to *not* do it that way but the fact that I couldn’t figure out how to directly move the code over using the same methodology as the C# code made me really want to figure it out. I hadn’t even intended to *really* blog anything but a translation of the code, but the PITA-factor related to not knowing I needed to cast that byte value to int caused me enough issues I wanted to document it.
For anyone who is not aware of what I am doing with the fun lines involving << and –shl and so on, I will summarize a bit.
- There are some values stored in the IP and TCP header (in this case, Size in IP header and Source/Destination Ports in the TCP header) that are multi-byte.
- Multi-byte values can go up to 64k , also known as 65535 in normal human terms (aaka, base 10).
- What this means is that we need to read the values out of two different byte fields. If we were reading the data bit by bit, we could simply add up the data where we need to, but we are getting the buffer back from Winsock as a byte array so I’m just dealing with it as a byte array.
- So the number we are looking to get to convert to an int is a 16 digit binary value, like 000001001000111, but what we will get are 2 byte values that look like this 0000010 01000111.
- In order to get that number, we have to do a bitwise shift and inclusive or
- First we pick the first ‘byte’ which is (in this example) 0000010. 0000010 is actually equal to 2 in decimal, but *REALLY* this number represents 000001000000000 (512) because it’s part of a multi byte value. we just can’t see the rest of the value because it’s in the next byte. This is ok though because we’ll kind of save this value and add the value to the next part by converting it to what it really is by using shift. Since we are shifting it over by one byte, we have to shift it 8 bits. The syntax in the code above shows examples of this. C#: b1 << 8 or PS: $b1 -shl 8 converts the 0000010 to 000001000000000. Basically we just tacked on 8 zeros. So now we have a part of a fixed number to work with, but we need the rest of the actual value.
- We now need to get the next portion of the value. In our example number here it is 01000111 (71). Really, we already have it. All you have to do is refer to the 2nd byte and you have it’s value as there is no transformation needed. So now we have it, what do we do with it?
- To combine these numbers we use bitwise inclusive or. C#: b1|b2 or PS: $b1 -bor $b2. Basically this is like ‘If either of the values when we check are 1’s, make the merged value a 1’. So we put 000001000000000 on top of 01000111 and move all the 1’s down. So you end up with 1001000111 which is 583.
- You can also just add the integer values of the shifted first byte shifted and the 2nd byte, but it’s easier just to Xor them together I think since you have to shift one anyway and are thus already doing bitwise operators.
- So the full syntax is something like this: C#: b1<<8|b2 PS: $b1 -shl 8 -bor $b2. But in the case of my code, I am not actually creating byte values, but I am getting them out of a byte array. because of this if you don’t cast the values to int values, the shifting doesn’t work right. so for PS you would do [int]$bArray[1] -shl 8 –bor [int]$bArray[1]
- Hopefully that helps someone who is/was on the cusp of understanding how some of the bitwise/binary stuff works. If you don’t know anything about binary or bitwise operators this may be a little much.
I wasn’t able to find a really simple way to do the eventing with PowerShell because the ‘onreceive’ method of the socket is something you have to call, not an event. I just went with this synchronous method that binds to a single local IP. I’m sure there is some way to do a console.read in PowerShell that will keep piping data out to the screen but I didn’t feel like messing with it too too much by the time I was done with the bitwise drama. Embedding the asynchronous C# code worked just fine so if I absolutely wanted to use the async method and I absolutely wanted to use it in PowerShell , I could just embed the C# code. Maybe one day I’ll revisit that whole issue. PowerShell is using .NET libraries to interface with Winsock anyway, so I have a hard time justifying spending too much time puzzling out how to do the harder stuff in PowerShell when I could just embed C# code just as easily. I would like to know exactly why there is so much more CPU time taken up by PowerShell, but that’ll have to wait for another day.
Sources
Some PowerShell Resources:
http://blogs.technet.com/b/heyscriptingguy/archive/2010/02/09/hey-scripting-guy-february-9-2010.aspx
http://blogs.technet.com/b/josebda/archive/2010/04/08/powershell-v2-get-date-vs-system-datetime-now-what-s-the-difference.aspx
http://learningpcs.blogspot.com/2012/06/powershell-v2-playing-with-bor-and-band.html
http://stackoverflow.com/questions/11329664/asynchronous-powershell-script
http://www.drdobbs.com/windows/building-gui-applications-in-powershell/240049898?pgno=2
http://blogs.technet.com/b/heyscriptingguy/archive/2011/06/16/use-asynchronous-event-handling-in-powershell.aspx
http://www.hofferle.com/embed-a-c-program-in-a-powershell-script/
http://stackoverflow.com/questions/5260125/whats-the-better-cleaner-way-to-ignore-output-in-powershell
Endianness:
http://en.wikipedia.org/wiki/Endianness
My Previous Post:
http://www.drowningintechnicaldebt.com/RoyAshbrook/archive/2013/02/13/simple-sniffer-in-c.aspx
In the past I blogged about a macro I use in outlook to automatically add a timedate stamp and my name to an outlook task to make updates easier to facilitate and read. I have updated that macro a little bit since that time and wanted to post the one I currently use. There are really only 2 major changes:
- Color Coding
- Insert at the beginning.
At my last job I used a sharepoint task list to manage most of my teams activities for quite some time. One of the guys on my team decided to add color coding to his macro and we all adopted it so we could see the different folks who edited a shared task at a glance. Typically the tasks weren’t passed around too much, but this was helpful for the collaborative tasks. I added this to my normal macro now and have also added a separate color for the Date and switch to the default for the delimiters. I also learned over time that adding a new line to the end wasn’t terribly helpful to me as I wanted to see the updates in reverse chronological order (latest updates first). This was particularly important once I began managing a team as I needed to be able to quickly see status on a task for someone on my team. So this macro goes to the beginning, then puts in a new line, then goes back to the beginning.
Here is an updated version of that macro:
1: ' need tools/references and add a reference to word
2: Sub InsertDateTime()
3:
4: ' strings to print
5: Dim t, n, v, delim As String
6: t = Format(Now(), "YYYY-MM-DD hh:mm:ss") & " EST"
7: n = "Roy Ashbrook"
8: delim = " | "
9:
10: Dim d As Word.Document
11: Dim s As Word.Selection
12: Set d = Application.ActiveInspector.WordEditor
13: Set s = d.Windows(1).Selection
14:
15: s.GoTo (0)
16: s.TypeText vbCrLf
17: s.GoTo (0)
18:
19: s.Font.ColorIndex = wdBlue
20: s.Font.Bold = True
21: s.TypeText t
22:
23: s.Font.ColorIndex = wdAuto
24: s.Font.Bold = False
25: s.TypeText delim
26:
27: s.Font.ColorIndex = wdRed
28: s.Font.Bold = True
29: s.TypeText n
30:
31: s.Font.ColorIndex = wdAuto
32: s.Font.Bold = False
33: s.TypeText delim
34:
35: Set s = Nothing
36: Set d = Nothing
37:
38: End Sub
Note that there are some ways to get the timezone, but they all added a *lot* of extra code so I just put it in myself. I generally just stick with whatever the default font is, but you can change that too if you wish. The code above essentially adds the line shown below to the top of the task. Sometimes I have had this hot-keyed, but generally I just create a button for the macro.
2013-02-27 23:45:50 EST | Roy Ashbrook |
Recently I had a need to audit network traffic on some servers. Believing we had appropriate monitoring in place I placed a request to the right folks to retrieve this information. During that process I decided to do something that’s been on my list of ‘programs to write’ for a while: write a sniffer.
My requirements:
- I only want Protocol Type, Source/Destination IPs and Ports (No Packet Data)
- Be promiscuous (meaning, get all of the data that we can see, even if it’s not for me)
- IPv4 only
You could argue that this is more of a packet header logger than sniffer, but the logic is there if you want to parse the rest of the packet. Using the references below you can find classes for the full IP and TCP headers as well as other info. Using that you could cache all of the packet data etc. I didn’t really want to build something to view cached data (like mjsniffer below), I just wanted the basics about the packets that are in flight. Anyway, the code is below. You can actually copy/paste this almost into LINQpad and run it there if you want. You have to convert the usings and a couple of other details but it works fine.
I have saved this project on codeplex with the project name SimpleSniffer, but included below is all of the code. This is “slightly” different than what’s on codeplex. The main difference is in this code I actually process the bytes as they come in and directly Console.Write the value from the buffer. In the code on codeplex I am shooting a subset of the resulting packet data (the last 15 bytes of the 24 I am grabbing) to a method and letting it write it. I thought this was a little bit smaller so I am leaving this version here.
1: using System;
2: using System.Linq;
3: using System.Net;
4: using System.Net.Sockets;
5:
6: namespace SimpleSniffer
7: {
8: static class Program
9: {
10: static void Main()
11: {
12:
13: // we are only listening to IPv4 interfaces
14: var IPv4Addresses = Dns.GetHostEntry(Dns.GetHostName())
15: .AddressList.Where(al => al.AddressFamily == AddressFamily.InterNetwork)
16: .AsEnumerable();
17:
18: // echo out a header line
19: Console.WriteLine("Protocol\tSourceIP:Port\t===>\tDestinationIP:Port");
20:
21: // start a sniffer for each interface
22: foreach (IPAddress ip in IPv4Addresses)
23: Sniff(ip);
24:
25: // wait until a key is pressed
26: Console.Read();
27:
28: }
29:
30: static void Sniff(IPAddress ip)
31: {
32:
33: // setup the socket to listen on, we are listening just to IPv4 IPAddresses
34: Socket sck = new Socket(AddressFamily.InterNetwork, SocketType.Raw, ProtocolType.IP);
35: sck.Bind(new IPEndPoint(ip, 0));
36: sck.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.HeaderIncluded, true);
37: sck.IOControl(IOControlCode.ReceiveAll, new byte[4] { 1, 0, 0, 0 }, null);
38:
39: //byte array to hold the packet data we want to examine.
40: // we are assuming default (20byte) IP header size + 4 bytes for TCP header to get ports
41: byte[] buffer = new byte[24];
42:
43: // Async methods for recieving and processing data
44: Action<IAsyncResult> OnReceive = null;
45: OnReceive = (ar) =>
46: {
47: Console.WriteLine( //echo the data. details at http://en.wikipedia.org/wiki/IPv4_packet#Packet_structure
48: "{0}\t{1}:{2}\t===>\t{3}:{4}"
49: , buffer.Skip(9).First().ToProtocolString()//todo: gotta be a cleaner way to do this one...
50: , new IPAddress(BitConverter.ToUInt32(buffer, 12)).ToString()
51: , ((ushort)IPAddress.NetworkToHostOrder(BitConverter.ToInt16(buffer, 20))).ToString()
52: , new IPAddress(BitConverter.ToUInt32(buffer, 16)).ToString()
53: , ((ushort)IPAddress.NetworkToHostOrder(BitConverter.ToInt16(buffer, 22))).ToString());
54: buffer = new byte[24]; //clean out our buffer
55: sck.BeginReceive(buffer, 0, 24, SocketFlags.None,
56: new AsyncCallback(OnReceive), null); //listen some more
57: };
58:
59: // begin listening to the socket
60: sck.BeginReceive(buffer, 0, buffer.Length, SocketFlags.None,
61: new AsyncCallback(OnReceive), null);
62: }
63:
64: // details at http://en.wikipedia.org/wiki/List_of_IP_protocol_numbers
65: public static string ToProtocolString(this byte b)
66: {
67: switch (b)
68: {
69: case 1: return "ICMP";
70: case 6: return "TCP";
71: case 17: return "UDP";
72: default: return "#" + b.ToString();
73: }
74: }
75:
76: }
77: }
I tried to figure out another way to do the Async stuff here but nothing seemed to be super elegant. I may come back to it. Although I really will probably end up tweaking this for something a little more in line with some other requirements I have developed as I was working on this. I also tried to find another way to do the stupid pull of that first Byte for the Protocol Version. I’m not sure why but I couldn’t find a way to just get ONE byte from the middle of the byte array without using the IEnumerable. I originally had this as a forward only read using a memorystream but it seemed easier to just put the BitConverter lines in for the others and pull it directly from the array. It was pretty difficult for me to personally come up with a good way to real world test this since it’s all different data on my home network. It works as it is so it’ll be fine for now. =)
Soooooooooooooooo (so so so so!) many references for this. I am going to try and just keep it narrowed to the ones I went back to repeatedly:
IPv4 Info: - http://en.wikipedia.org/wiki/IPv4_packet#Packet_structure
Byte Array Stuff:
http://stackoverflow.com/questions/409256/working-with-byte-arrays-in-c-sharp
http://stackoverflow.com/questions/123918/how-can-one-simplify-network-byte-order-conversion-from-a-binaryreader
http://stackoverflow.com/questions/589099/using-part-of-a-byte-array
Other C# Sniffers and Socket references:
http://www.c-sharpcorner.com/uploadfile/leonidmolochniy/simplesnifferincs11222005232804pm/simplesnifferincs.aspx
http://www.codeproject.com/Articles/17031/A-Network-Sniffer-in-C
http://innodesign.over-blog.com/article-netwrok-sniffer-54683945.html (mostly a copy of the mjsniffer)
http://stackoverflow.com/questions/2753743/socket-receiveall
http://www.winsocketdotnetworkprogramming.com/clientserversocketnetworkcommunication8b_1.html
Async/Parallel/Threading stuff:
http://stackoverflow.com/questions/3572554/c-sharp-parallel-vs-threaded-code-performance
MS References:
http://msdn.microsoft.com/en-us/library/ms700657(v=vs.85).aspx
http://msdn.microsoft.com/en-us/library/system.net.networkinformation.ipaddressinformation.aspx
http://msdn.microsoft.com/en-us/library/bb384066.aspx
Recently I have been using a software product from Sparx Systems called Enterprise Architect. I (and I believe most people) call it EA for short. You can read about it here.
I have enjoyed working with it for the most part. It seems to do a great job for what it is. I have the benefit of coming into an environment where we have a lot of documentation in EA already. Unfortunately none of it is really getting updated due to licensing but my team (and Enterprise Architecture group) is still utilizing it and updating it. The issue we have had is how to continue to work and update EA (at least for the existing solutions documented within it) and still provide that documentation to folks without EA. EA licenses are run in the hundreds.
My solution for this was to:
- Extract the Diagrams from EA (These house the majority of the information people want to see)
- Provide some mechanism to search the Diagrams
How I actually implemented this solution was:
- Extract the diagrams to PNG files
- Extract some searchable metadata to excel
- Import metadata document to a new SharePoint list
- Upload the PNG files to some location on a SharePoint list
- Add a calculated column that has an URL to the corresponding diagram
- Modify the list to show all items
- Implement JavaScript to search the SharePoint list page as I documented here.
During this process I wrote a little app and uploaded it to CodePlex. I called it EAPtoHTML as it takes an EAP file (this is an EA project file) and outputs a collection of PNG files and an index.html file with a bunch of keywords to search. I compressed a lot of words to make the keywords list shorter for a basic text search. So count, encounter, countered, and encountered would all be squished down to encountered. This isn’t ideal for everyone, but in the code you can simply remove the part of the code that ‘collapses’ the keywords and do something else with it if you want to actually put it into a real table with all of the words. You could also modify the extraction routine to get ‘more’ text. I just pulled the name and the notes from all of the objects in question as that seemed to have most of the keywords people would search for. You do lose unique combinations of words, but that trade-off seemed to work for my purposes/consumers. Ideally, if I get some time, I would like to go back and just hide the keywords column, keep the raw text, and allow searching. but this was pretty quick and dirty and worked fine.
Here is the project summary from CodePlex:
Project Description
EAPtoHTML serves a very simple purpose: Extract the diagrams from an Enterprise Architect Project file and save them as PNG files in a folder with an index.html that you can use to search for data.
Some Notes:
- There is currently no EXE download. You will have to download the solution and build it yourself to run it as it will require access to the EA Interop DLL which I am not providing as well as the LINQpad exe which I'm not providing.
- LINQpad is free and awesome. I am only using it for formatting the results. You could use anything you wanted I just didn't want to write anything fancy and I already use LINQpad all the time.
- EA is not free, but it's still pretty awesome for what it does. In my case (and I think many folks case) a group wants to adopt EA but then no one can see the data unless they have an EA license and since that costs money during a pilot it just puts a burden on the folks trying to adopt it as they have to continually extract data to show to people that do not have it.
- This project is really meant to show you how to do this. To use this extracted data, I personally have just extracted it to excel and then saved it to a SharePoint with a simple JavaScript search. But I wanted to put something up that would dump out some kind of results.
- When I have more time I'd like to tune the output, but I figured I would rather get it out there in case someone else just wanted to see how to get EA diagram data out.
- You can simply rename the .html index file to .xls and import into SharePoint if you want to expose it that way. I did this and then used a calculated column to build the URL to point to an uploaded copy of all of the PNG files also on SharePoint and put some JavaScript on the page to search within the list after modifying it to show all records. I had over 3000 diagrams and this worked well.
Hopefully this helps someone. =)
Sources
For some EA stuff:
C# Stuff:
Keywords:
- http://codereview.stackexchange.com/questions/11254/find-most-occurring-word-in-a-txt-file
- There were actually a *lot* of other articles I read regarding doing word search and matching. But the one above actually helped. I left this section here because deciding how to collapse words like count, counter, countered, encounter and encountered all into just ‘encountered’ for compressed searchability was a big pain. I’m not sure my way I is the ‘best’ but I liked it and it reduced the size of my data a ton and still made the items searchable pretty easily.
Outputting to LINQpad
For doing searching on SharePoint (which is where I ultimately put *my* data)
So recently I had this great idea to extract a bunch of documentation from a sort-of proprietary system, turn it into diagrams, and put it on SharePoint with some searchable data bout each diagram. So the solution looks like:
- Save all the Diagrams as files with some kind of unique name to some location.
- Extract a bunch of data bout the Diagrams (preferably the actual text in them) into excel with the same unique name as a key
- Upload the whole document into a new SharePoint list so it would be searchable and people that don’t have access to the source system could see the documentation.
Note: Make sure you add a column that points to the extracted diagrams.
Sounds pretty simple, but I ran into the little hitch that SharePoint apparently doesn’t seem to index list data by default. Bummer. So after looking for a way to search that data, the quickest way to get this to work (keeping in mind I don’t have control over the sharepoint server, just my site) was to utilize the Content Editor Web Part (CEWP) and JavaScript. This consisted of the following steps:
- Make sure your list shows *EVERYTHING*, this will only filter what is actually on the current page
- Add a Content Editor Web Part at the top of the list page
- Paste in the code down below
That’s it. The JavaScript listed will search whatever is on the page in that table and hide it if it doesn’t match the text within that row. The button text box will call the JavaScript when you hit enter after typing. Not perfect, but it works so far and everyone is happy with the results.
<script type="text/javascript">
function getElementsByCssClass(sTagName, sClassName) {
var results = new Array();
var allTagElements = document.getElementsByTagName(sTagName);
for (i = 0; i < allTagElements.length; i++) {
if(allTagElements[i].className == sClassName) results.push(allTagElements[i]);
}
return results;
}
function getTxt(obj) {
if(obj.innerText) return obj.innerText;
else return obj.textContent;
}
var lvRows = null;
function quickSearch(term) {
if(lvRows == null) {
var lvTable = getElementsByCssClass("table","ms-listviewtable")[0];
lvRows = lvTable.childNodes[0].childNodes;
}
for(i = 1; i < lvRows.length; i++) {
if(getTxt(lvRows[i]).toLowerCase().indexOf(term.toLowerCase()) > -1) {
lvRows[i].style.display = "table-row";
}
else lvRows[i].style.display = 'none';
}
}
Quick Search (type your query then hit 'Enter'): <input class="ms-long" id="inputboxID" type="text" onkeyup="if (event.keyCode == 13) quickSearch(this.value)"/>
Looked at a bunch of stuff, but these were the good sources:
So the other day I did an extract from another documentation system of a few thousand diagrams. I copied these all up to sharepoint but the library I was storing them on had versioning. So how to check them in?
Very simple! (assuming you have permissions, which I did)
- Click Site Actions > View Reports > Checked Out To Me
- Select the documents you want
- Actions > Check In
(edit: I had pictures here, but our image uploader is on the fritz, so now it’s just a list!)
Downloading videos from YouTube is something I think many people need to do from time to time. I always forget how to do it myself and have to re-look it up. So I’m self documenting this time.
1. Download and Install VLC Media Player (http://www.videolan.org/) Why VLC Media Player? Because it’s free, it’s a trusted source, and it works. Additionally, this article is about how to save videos from YouTube using VLC Media Player so if you are using another player, you are doing it wrong. =)
2. Open the video you want to save on YouTube, and copy the URL from the address bar.
3. Open VLC Media Player, click Media, click Open Network Stream, Paste in the URL, click Play.
4. Pause the video (not really needed, but I normally do), click Tools, click Media Information, right click on Location, click Select All, right click on Location again, click Copy.
5. Go back to your browser of choice (the one from step 2, probably), right click the address bar, paste in the location data.
6. The Final Step is very simple. Just save the file in the browser as an MP4 file. Depending on your browser and what plugins you have installed, the video will either start Playing in the browser, or it will ask you to save the file. For me if I paste the location data into IE, it asks to save immediately, but in Chrome it starts playing it. If it starts playing, you need to go to File and Save or Page and Save (depending on what browser you use) to initiate the saving. Otherwise just save it. Below is a screenshot of me saving a video from Chrome. Note that I renamed the video from videoplayback with no extension, which is what it starts as, to TestRayVideo.mp4. You can save it as any name you want as long as you END THE FILE NAME WITH .MP4!
That’s it, you should now be able to open the saved video or copy it or whatever you want to do with it.
I’ve found various sources for this method over the years, but my source this time was this video on YouTube. Thanks Innolea!
So I wrote a simple program I called NeverAway that will prevent your computer from showing ‘Away’ in Microsoft Office Communicator. ( Or any other program that tracks keyboard activity I would imagine. =P )
You can download the install, program, and the source code at http://neveraway.codeplex.com/ (click on downloads, then the recommended download that creatively is named ‘Extract to somewhere and install this’).
From my release notes:
Basic Usage Instructions:
Simple, just download, extract, install. It will start up automatically. Right click the little shield icon and click 'Never Away' and you won't show up as Away in MOC anymore.
This was mostly a result of my irritation with many corporate pc policies that would lock my screen after 15 minutes of inactivity even though I would have my MOC set to not show me as away unless I hadn’t done anything for an hour. I originally just used a LINQPad (http://www.linqpad.net/) program that made the appropriate call, but I decided I wanted to make an application so others could use it easily. Certainly not curing cancer, but it was something fun to do. I hadn’t written a traymenu application for probably 10 years so I thought this would be as good as any. =)
I know if someone would have shown me one when I was dealing with this irritation while on tons of conference calls, I would probably have paid 50 bucks for it on the spot. but here it is for free!
While I was working on this, I went to download stickies (http://www.zhornsoftware.co.uk/stickies/) and I decided to look through the other software on there. Lo and behold, there was caffeine (http://www.zhornsoftware.co.uk/caffeine/) which seems to do the same thing. I initially chose ‘mute’ (because everyone has a mute button) instead of F15 (as caffeine does), but then later picked F24 after testing some other items just because it seemed non-intrusive. I also added a config to where you could press a key more than once (because I initially pressed mute twice). Anyway, the point is caffeine from zhorn software is another option for this kind of thing. And Stickies is awesome. =P
Recently I was working to populate a Date Dimension table and I discovered that the script used was basing the first of the week on my local machine. This was fine except I wanted to line up my weeks against our financial system which hand the end of the week on Friday, not Saturday is it looks like on a normal calendar.
Basically making the effective change below:
To do this, I performed these steps:
- Open control panel
- Go to Clock/Language/Region
- Under Date and Time, click Set the date and time
- Click Change date and time button
- Click Change calendar settings link
- Change First day of the week to your desired day
- Click OK, OK, OK, OK to get out of the dialogue boxes
- You are done!
Images of the steps are below (excluding the first step of opening control panel):
Typically when asked to display a list data, lets say customers, we create a first name column, last name column and then iterate over the rows to produce a table. Something like this:
But recently I was asked to produce a pivot of this data, putting each customer in their own column instead of an individual row. After doing the obligatory internet searching I found no solution that I really liked. So I wrote my own…
Lets start by creating our customer object:
public class Customer
{
public int Id { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public int Age { get; set; }
}
Now that we got him, lets mock up some data in our controller:
public ActionResult Index()
{
var customers = new List<Customer>();
for (int i = 0; i < 100; i++)
{
customers.Add(new Customer()
{
Id = i,
FirstName = string.Format("First {0}", i),
LastName = string.Format("Last {0}", i),
Age = i
});
}
ViewBag.Customers = customers;
return View();
}
Finally, the meat, using a bit of reflection magic lets get a “list” of all the properties on our customer, iterate over each of them to generate our rows, then iterate over each customer and get the “value” for that row.
<div style="width: 600px; overflow: auto">
<table style="white-space: nowrap;">
<thead>
<tr>
<th></th>
@foreach (var customer in ViewBag.Customers)
{
<th style="">
Customer @customer.Id
</th>
}
</tr>
</thead>
<tbody>
@foreach (var prop in typeof(Customer).GetProperties())
{
<tr>
<td>
@prop.Name
</td>
@foreach (var customer in ViewBag.Customers)
{
<td>
@prop.GetValue(customer, null)
</td>
}
</tr>
}
</tbody>
</table>
</div>
And ta-da we have a table with dynamically generated columns based on the number of “customers” in our collection.
Ever had an excel file that looked like this?
and need an xml file that looks like this?
Converting takes 2 steps:
Step 1: Read the excel file into objects in memory
var products = new List<Product>();
using (var con = new OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0;Data Source=data.xlsx;Extended Properties=\"Excel 8.0;HDR=YES;\""))
using (var cmd = con.CreateCommand())
{
con.Open();
cmd.CommandText = "SELECT * FROM [Sheet1$]";
var dr = cmd.ExecuteReader();
while (dr.Read())
{
products.Add(new Product()
{
ProductGroup = dr[0].ToString(),
ProductSubGroup = dr[1].ToString(),
ProductNumber = dr[2].ToString()
});
}
}
Step 2: Use LINQ to group the objects into a hierarchy and convert them into xml
var xml = new XElement("ProductGroups",
products.GroupBy(x => x.ProductGroup).Select(x => new XElement("ProductGroup",
new XElement("name", x.Key),
new XElement("ProductSubGroups",
x.GroupBy(y => y.ProductSubGroup).Select(y => new XElement("ProductSubGroup",
new XElement("name", y.Key),
new XElement("Product", y.Select(z => new XElement("number", z.ProductNumber)))))))));
xml.Save("test.xml");
Orchard the open source CMS from Microsoft (http://www.orchardproject.net/) has a great tempting engine. The out of the box base template “The Theme Machine” is very powerful and flexible, however if you are as design challenged as I, creating something with it that looks decent is out of the question. To that end I use a very cool template builder, Artisteer (http://www.artisteer.com/) if that is not your speed there are about a million websites where you can download free/low cost templates, however most of them generate a standard HTML page with some javascript and css. I wanted to take a minute to document how I converted my HTML template into an Orchard theme.
(Disclaimer this worked for my site, your template might require some different html changes, especially when it comes time to edit the HTML and CSS)
Step 1: Get the html template. (Did I really need to tell you that?)
Step 2: The goal is to make a theme based on the base template “The Theme Machine”. You will need to follow the instructions on the Orchard site to “Generate a New Theme” http://www.orchardproject.net/docs/Writing-a-new-theme.ashx, however their instructions have you creating an empty theme. We need to create one based on “The Theme Machine”. To do that just alter the codegen step to use the following command:
>> codegen theme MyTheme /BasedOn:TheThemeMachine /CreateProject:true /IncludeInSolution:true
Step 3: Copy the CSS & related images into the “Styles” folder of your new theme
Step 4: Copy the javascript into the “Scripts” folder of your new theme
Step 5: Copy the template html file into the Views folder. I do this so that I have it for safe keeping and it stays with the project. Orchard will not be using it directly.
Step 6: Copy the “Layout.cshtml” file from the “Views” folder from the “TheThemeMachine” template into the “Views” folder of our new theme.
Step 7: This is the hard part, time to merge the HTML template that you found/created in Step 1 with the Layout.cshtml file you copied into your “Views” folder in Step 6.
There are 2 things that you will have to worry about here:
First you need to add references for the css and js files. My template requires 3 css files and 1 js file, so I needed to add the following at the top of the layout.cshtml file. I put mine just below the includes “Style.Include” statements that were already in the file.
My template looks like this:
Converted to the layout.cshtml file it looks like:
Now that we have the head tag filled with all its goodness, it is time to do the body part. Copy over the html from the template into your Layout.cshtml file. You want to copy the html just below the “@tag.StartElement” line of code. With the markup in place, start moving the placeholders from the Layout.cshtml file into the proper places in your new template.
Once you have moved the placeholders around, you will probably need to add a css file tweak everything just right.
Step 8: The menu. My template and my client required a drop down menu, however out of the box Orchard only supports a single level menu. Time to add a module, the “Advanced Menu” module from Piotr Szmyd to be specific. After installing this module you will now have the ability to create multi level menus. However it comes with its own look and feel. If this works for you great, if not, you will have to modify it. Since my template provides for a menu that I like I am going to use it. To do so I need to copy 2 more cshtml files into my "Views” folder. After adding the module you should see a folder for it listed in your “Modules” folder called “Szmyd.Orchard.Modules.Menu”. In there you will find a “Views” folder and in it you will see Menu.cshtml, and MenuItem.cshtml. Copy those two files into the "Views” folder of your theme. Now alter them to match your template, just like you did the layout.cshtml file.