Tag: coding

API: Top-Down? Bottom-Up? Somewhere in the Middle?

A Quick Brain-Dump on API Desgin

I’ll keep this one pretty brief as I haven’t totally nailed down my thoughts on this. I still thought it was worth a quick little post:

When you’re creating a brand new API to expose some functionality of a system, should you design it with a strong focus on how the internals work? Should you ignore how internals work and make it as easy to consume as possible? Or is there an obvious balance?

I find myself trying to answer this question without ever explicitly asking it. Any time I’m looking to extend or connect systems, this is likely to come up.

Most Recently…

Most recently I started trying to look at creating an API over AMQP to connect my game back-end to a Unity 3D front-end. I had been developing the back-end for a little while now, so I had a pretty good idea for how things needed to work from that perspective. The front-end? Not so much really. I knew some basic actions that I was considering, so I tried coding up an early API for them.

A lot of my focus was around how I was going to implement the code on the back-end to make this API work. It resulted in some of the API calls looking a little bit gross. But the idea was that if I settled on an API that would make the back-end easy to implement, I could get it up and running faster.

After a little while, I feel like my API isn’t getting any cleaner from a consumer perspective, and funny enough, it’s not actually as easy to implement as I was hoping on the back-end. Which had me reflect on a work example…

Once Upon a Time at Work…

Well, it was only a couple of months ago, really. I was working with a colleague on integrating a new system into an existing code base. We decided we wanted to approach the API problem from a consumer perspective. We said “let’s make this as easy to call and use as possible so that people ACTUALLY want to use it”.

We set out with this mission, and created a pretty simplistic API. The challenge? There was a lot of heavy lifting and a bit of voodoo going on behind the scenes. But you know what? We hid the magic in one spot of the code (instead of having ugly stuff scattered all over the code base) and it ended up being a very usable API.

So…

So does this consumer-first, top-down approach to API design always work? I’m not sure. Some similarities/differences in the scenarios:

  • In my current situation, I have a back-end implemented and very minimal code implemented on the caller side. In the work example, we had nothing implemented on the back-end, and a ton of code implemented where the caller side would be.
  • In both examples, at least one of the caller side or back-end side was reasonably well understood. For my game, the back-end was pretty well understood. For work, the caller side was pretty well understood and we had experience with what we’d call a “failed’ back-end implementation (that we were actually setting out to redesign).
  • The work example was a relatively small subset for an API, but the game example was about to be a very specific implementation that I’d need to adapt into a pattern for all messaging in my AMQP system

So there’s a few things to consider there. I think I’m at the point in my game where I’d like to revisit how I’m forming this API and try it from a client-first perspective. Now that I know some of the catches, maybe I’ll shed some new light!

How do you approach API design?


Should My Method Do This? Should My Class?

Whose Job Is It?

I wanted to share my experience that I had working on a recent project. If you’ve been programming for a while, you’ve definitely heard of the single responsibility principle. If you’re new to programming, maybe this is news. The principle states:

That every class should have responsibility over a single part of the functionality provided by the software, and that responsibility should be entirely encapsulated by the class

You could extend this concept to apply to not only classes, but methods as well. Should you have that one method that is entirely responsible for creating a database connection, connecting to a web service, downloading data, updating the database, uploading some data, and then doing some user interface rendering? What would you even call that?!

The idea is really this: break down your code into separate pieces of functionality.

Easier Said Than Done… Or Is It?

The idea seems easy, right? Then why is it that people keep writing code that doesn’t follow this guideline? I’m guessing it’s because even though it’s an easy rule, it’s even easier to just… code what works.

The recent experience I wanted to share was my work on a project that has a pretty short time frame to prove it was feasible. It was starting something from scratch, so I had all the flexibility in the world to design code however I wanted to. I really made an effort to keep asking myself this one question: Whose job is it?

Every time I asked that question and found that it was not my current method’s responsibility, I would ask “Is this class really responsible for that”? I’d either go make myself a new method in my class or I’d just go immediately make a new class with a single method on it. It seemed like a bit of extra overhead each time I had to do it, but was it worth it in the end?

Absolutely. After the project had proven itself and development continued on, I was easily able to refactor code (where necessary) and mock out functionality in my coded tests. Instead of trying to write test setup code that required a whack of classes I needed to initialize, I could mock out a couple of interfaces and test with ease. It was also really obvious which pieces were responsible for what functionality.

Final Thoughts

If you want to get better at following the single responsibility principle, I think it starts with one question: Whose job is it? Try it out!


Multiple C# Projects In Your Unity 3D Solution

Unity

Problem: Visual Studio and Unity Aren’t Playing Nice!

Disclaimer: I develop on Windows, so I have no idea if any of this even applies to other operating systems. I assume not. Sorry.

I just started poking around in Unity 4.6 and I’ve been having a blast. I’ve made it to the point where I want to actually start hammering out some code, but I came across a bit of a problem: I want to start leveraging other projects I’ve written in my Unity solution while I’m in Visual Studio, and things are blowing up. So, what gives?

Okay, so let me start by explaining why I want to do this. I understand that if I’m making a simple game, I should have no problem breaking out my unity scripts into sub folders and organizing them to be nice and pretty. The problem I’m encountering is that I have existing projects under source control and I don’t want to copy and paste all of the code as scripts into my Unity folder. I also want to be able to create re-usable code for my future games, so I’d like to start breaking things out into libraries as I see fit.

So, if you’ve been playing around in Unity for a bit, you might say “Oh, well you’re a dummy! Unity can totally leverage your C# DLLs once you drop them into your asset folder”! And you’d be 100% correct. But that’s not the workflow I want.

The underlying problem here is this: Unity will re-write your solution and project file when you flip between Unity and Visual Studio. But I’m sure they have it that way for a reason.

The Goal: Visual Studio and Unity Should… Play Nice!

My ideal state would be something like this:

  • Work in visual studio as much as I’d like to new projects to my solution, and reference them accordingly
  • Flip back and forth from Unity and Visual Studio without having to reset things to compile/run again
  • Build from visual studio and have things end up in the right spot… NOT copy DLLs
  • Not copy+paste my entire project(s) already under source control elsewhere

Is this something that can be achieved though? I was pretty determined that I should be able to do *something* to have this working. Could I get it perfect? I wasn’t sure… But I knew I could make it better.

The Solution: Give and Take with Unity

My *almost* perfect sution, which I’ll walk you through, is this: Leveraging Visual Studio tools for Unity, modify the Unity solution as you see fit and use directory junctions (symlinks) to the build output directories of other projects.

  1. Let’s get Visual Studio tools for Unity installed. Visit that link and download the version that you need for the version of Visual Studio that you use. After installing, I opened up my project within Unity and I had to import the Visual Studio Tools package.Import Package
    After selecting this menu item, I was presented with a dialog for picking the items to import. I left it as is.Import Package2After importing these items, I could see that Unity had successfully added these entries under my Assets folder. Okay, now we’re getting somewhere. Next up, I wanted to configure Unity to not modify my solution every time I go back and forth from Unity to Visual Studio. This is the part that kills whether or not I’ve added projects to my solution. For me, it’s critical to have code I’m working on immediately accessible so that I can jump back and forth between projects. Lucky for us, this part is pretty easy. Go to the menu to access your new Visual Studio Tools menu item:

    VS Tools Configure
    Selecting “Configuration” opens up a really simple dialog. Let’s make sure “Generate solution file” is unchecked! It’s that easy.

    VS Tools Configure2
    Once we have all of this setup, we should be able to go into Visual Studio and add other projects to our solution.

  2. The one thing that I *could not* get this solution to do is have Unity leave my main game project alone in Visual Studio. As a result, the rest of this walk through is allowing us to play by Unity’s rules. Unity is good at magically referencing all of the managed DLLs that you include within your assets folder. If you drop DLLs somewhere within “Assets” and switch to Visual Studio, Unity will likely have modified your main project to reference this DLL.My next step was creating a spot where I wanted to drop the build outputs of my extra projects I wanted to reference. In my Visual Studio solution, I have my original game project and some newly added projects I want to build from source. In Unity, I wanted these to end up in “Assets/Dependencies/bin”. No problem. Let’s make that folder structure (or your equivalent if you don’t like my naming):Bin Dependencies
    The next part is probably the “trickiest” part because it’s… well… unusual. You could technically stop here and manually copy DLLs back and forth, but I’m not about that life. I want things to happen automatically. For this, we’re going to use junction points. Browse to your newly created folder in an administrator command prompt. I say administrator because only certain users have permissions to create junction points. Your non-admin user might, but this is my “safe” way of instructing you. On the command prompt, we’re going to use “mklink” to create a junction. The command is “mlkink /D /J <NAME_OF_YOUR_PROJECT> <RELATIVE_PATH_TO_YOUR_PROJECT>”. For example, if you had a C# project you wanted to reference that was “MyCoolLibrary.csproj” and was located in the directory above your Unity project, you might use the command “mlkink /D /J MyCoolLibrary “……..MyCoolLibrarybindebug””. Note that I used two dots to go back up a directory several times (since we’re inside of AssetsDependenciesbin and want to get outside of our Unity project). you should get a success message when your junction is created.

    Repeat this step for as many extra projects as you want to include. You can always come back and add more projects this way too, or remove the junctions if you don’t want to include a project anymore.

    At this point, you’re technically done. If you build from Visual Studio, you should have your other projects’ DLLs end up in your Unity folder, and your main game project will be updated by Unity to reference these now!

  3. But… You’re not done if you use source control for your Unity project and have separate source control on your other projects. The scary thing here is that usually we don’t want our build outputs to be stored in source control… But if we do nothing else, your source control system will likely want to include the newly created “AssetsDependenciesbin” folder and any of the contents you’re building into there. I just modified my git ignore file (I’m sure there’s an equivalent for SVN or other source control) to exclude the contents of “AssetsDependenciesbin”.The reason I didn’t excluded dependencies all together is because I can add other folders and DLL references here that I don’t want to build (like… the normal way). This gives me the flexibility of building the projects I want to control and still be able to just reference other pre-built DLLs!

Summary

In three easy steps, you should be able to use Unity, Visual Studio, and multiple projects in one solution in a what-feels-like-normal way. Because there’s still some dynamic stuff going on with Unity updating your main project, you might find the odd time you need to build twice to fix up compilation problems. I’ve seen this happen maybe once or twice so far, but otherwise it feels like normal. It’s also  important to note that you can’t escape the Unity project updating… don’t add references to your main project manually. That’s what that “AssetsDependencies” folder is for that we made.

Here are a few shots of what my setup looks like (proof that it works):

Solution Explorer

Unity Dependencies

And of course… it’s not the perfect solution. There’s still these things:

  • Unity gets mad at you for using junctions within your project. It actually tells you not to do this because you can mess things up. It’s working awesome for me right now though… So I’m going to just ignore this warning.
  • Remember step 3 where we ignored the AssetsDependenciesbin location in git? This actually ignored your junction points you created too. As a result, anyone else who clones your code will need to create junctions too. I’m working solo, so I’m not too worried about this step… But it’s definitely something that should be fixed up (again, I’m sure it’s doable, but I’m in no rush).

Hope that helps you feel more at home in Unity and Visual Studio! It certainly made it nicer for me.

 


ProjectXyz: Enforcing Interfaces (Part 2)

Enforcing Interfaces

This is my second installment of the series related to my small side project that I started. I mentioned in the first post that one of the things I wanted to try out with this project is coding by interfaces. There’s an article over at CodeProject that I once read (I’m struggling to dig it up now, arrrrrghh) that really gave me a different perspective about using interfaces when I program. Ever since then I’ve been a changed man. Seriously.

The main message behind the article was along the lines of: Have your classes implement your interface, and to be certain nobody is going to come by and muck around with your class’s API, make sure they can’t knowingly make an instance of the class. One of the easiest ways to do this (and bear with me here, I’m not claiming this is right or wrong) is to have a hidden (private or protected) constructor on your class and static methods that let you create new instances of your class. However, the trick here is your static method will return the interface your class implements.

An example of this might look like the following:


public interface IMyInterface
{
    void Xyz();
}

public sealed class MyImplementation : IMyInterface
{
    // we hid the constructor from the outside!
    private MyImplementation()
    {
    }

    public static IMyInterface Create()
    {
        return new MyImplementation();
    }

    public void Xyz()
    {
        // do some awesome things here
    }
}

Interesting Benefits

I was pretty intrigued by this article on enforcing interfaces for a few reasons and if you can stick around long enough to read this whole post, I’ll hit the cons/considerations I’ve encountered from actually implementing things this way. These are obviously my opinion, and you can feel free to agree or disagree with me as much as you like.

  • (In theory) it keeps people from coming along and tacking on random methods to my classes. If I have an object hierarchy that I’m creating, having different child classes magically have random public APIs changing independently seems kind of scary. People have a harder time finding ways to abuse this because they aren’t concerned with the concrete implementation, just the interface.
  • Along with the first point, enforcing interfaces makes people think about what they’re doing when they need to change the public API. Now you need to go change the interface. Now you might be affecting X number of implementations. Are you sure?
  • Sets people up nicely to play with IoC and dependency injection. You’re already always working with interfaces because of this, now rolling out something like Moq or Autofac should be easier for you.
  • Methods can be leveraged to do parameter checks BEFORE entering a constructor. Creating IDisposable implementations can be fun when your constructor fails but and your disposable clean up code was expecting things to be initialized (not a terribly strong argument, but I’ve had cases where this makes life easier for me when working with streams).

Enforcing Interfaces in ProjectXyz

I’ve only implemented a small portion of the back-end of ProjectXyz (from what I imagine the scope to be) but it’s enough where I have a couple of layers, some different class/interface hierarchies that interact with each other, and some tests to exercise the API. The following should help explain the current major hierarchies a bit better:

  • Stats are simple structures representing an ID and a value
  • Enchantments are simple structures representing some information about modifying particular stats (slightly more complex than stats)
  • Items are more complex structures that can contain enchantments
  •  Actors are complex structures that:
    • Have collections of stats
    • Have collections of enchantments
    • Have collections of items

Okay, so that’s the high level. There’s obviously a bit more going on with the multi-layered architecture I’m trying out here too (since the hierarchies are repeated in a similar fashion in both layers). However, this is a small but reasonable amount of code to be trying this pattern on.

I have a good handful of classes and associated interfaces that back them. I’ve designed my classes to take in references to interfaces (which, are of course backed by my other classes) and my classes are largely decoupled from implementations of other classes.

Now that I’ve had some time to play with this pattern, what are my initial thoughts? Well, it’s not pure sunshine and rainbows (which I expected) but there are definitely some cool pros I hadn’t considered and definitely some negative side effects that I hadn’t considered either. Stay tuned

(The previous post in this series is here).


Refactoring For Interfaces: An Adventure From The Trenches

puzzle

Refactoring: Some Background

If you’re a seasoned programmer you know all about refactoring. If you’re relatively new to programming, you probably have heard of refactoring but don’t have that much experience actually doing it. After all, it’s easier to just rewrite things from scratch instead of trying to make a huge design change part way through, right? In any mature software project, it’s often the case where you’ll get to a point where your code base in its current state cannot properly sustain large changes going forward. It’s not really anyone’s fault–it’s totally natural. It’s impossible to plan absolutely everything that comes up, so it’s probable that at some point at least part of your software project will face refactoring.

In my real life example, I was tasked with refactoring a software project that has a single owner. I’m close with the owner and they’re a very technical person, but they’re also not a programmer. Because I’m not physically near the owner (and I have a full-time job, among other things I’m doing) it’s often difficult to debug any problems that come up. The owner can’t simply open an editor and get down to the code to fix things up.

So there was an obvious solution which I avoided in the first place… Unit tests. Duh. I need unit tests. So that’s an easy solution right? I’ll bust out my favourite testing framework (I’m a fan of xUnit) and start getting some solid code coverage. Well… in an ideal world, like every programming article is ever written, this would have been the case. But that wasn’t the case. My software project does a lot of direct HTTP/FTP requests and interacts with particular hardware on the machine. How awesome is writing a unit test that contacts an HTTP server? Not very awesome.

What was I to do? I need to be able to write unit tests so that I can validate my software before putting it in my customer’s hands, but I can’t test it with unit tests because I don’t have the hardware!

Refactoring for Interfaces

Okay, so the first step in my master plan is to refactor for interfaces. What do I mean by that? Well, I have a lot of code that will call out and make HTTP requests and it has a specific dependency on System.Net.WebRequest. The same thing holds true for my FTP requests I want to make. Because I have that dependency within my classes, it means I have no choice but to call out to the network and go do these things.

I could design it a different way though. What if I abstracted the web requests away so that I didn’t actually have to call that class directly? What if I could have a reference to some instance that met some API that would just do that stuff for me? I mean, my class knows all about it’s on particular job, but it truthfully doesn’t know the first thing about calling out to the Internet to go post some HTTP requests. This means if someone else is responsible for providing me with a mechanism for giving me the ability to post HTTP requests, this other entity could also fool me and not actually send out HTTP requests at all! That sounds like exactly what my test framework would want to do.

My first step was to look at the properties and methods I was using on the WebRequest class. What was shared between the HTTP and FTP requests that I was creating? The few things I had to consider were:

  • Some sort of Send() method to actually send the request
  • A URI to identify where the request is being sent
  • A timeout property

I then created an interface for a web request that had these properties/methods accessible and created some wrapper classes that implemented this interface but encapsulated the functionality of the underlying web requests. The next step was creating a class and interface for a “factory” that could create these requests. This is because my code that needs to make HTTP/FTP requests only knows it needs to make those requests–It doesn’t have any knowledge on how to actually create one.

With my interfaces for my requests and my factory that creates them, I was able to move onto the next step.

Create Mocks for the Interfaces

Now that I had my classes leveraging interfaces instead of concrete classes internally, I could mock the inner-workings of my classes. This would provide two major benefits:

  • I could create tests that wouldn’t have to actually go out to the Internet/network.
  • I could create instrumented mocks that would let me test whether certain web requests were being made.

I started off my writing up some unit tests. I tried to get as much code coverage as I could by doing simple tests (i.e. create an object, check default values, call a method and check a result, etc…). Once I had exhausted a lot of the simple stuff, I targeted the other areas that I wasn’t hitting. I mean, how was my coded test supposed to test my method that does an HTTP request and an FTP of a file under the hood? Mocks.

So this is where you probably draw the line between your integration tests or unit tests and get all pedantic about it. But I don’t care how you want to separate it: I need coded tests that cover a section of my class so that I can ensure it behaves as I expect. But if I’m mocking my dependencies, how do I know my class is actually doing what I expect?

Instrument your mocks! This was totally cool for me to play around with for the first time. I had to create dummy FTP/HTTP requests that met the interface my class under test expected. Pretty easy. But I could actually assert what requests my class under test was actually trying to send out! This meant that if my method was supposed to try and hit a certain URL, I could assert that easily by instrumenting my mocked instance to check just that. Was it supposed to FTP a certain set of bytes? No problem. Use my mocked instance to assert those bytes are actually the ones my class under test is trying to send.

Wrap Up

This was just a general post, and I didn’t put up any code to go along with it. Sorry. I really just wanted to cover my experience with refactoring, interfaces, mocking, and code coverage because it was a great learning for me.

To recap on what I said in this post:

  • Identify the parts you want to mock. These are the things your class or method probably isn’t responsible for creating directly. Going out to the network? Accessing the disk? Accessing the environment your test is running under? Creating complex concrete classes because they hook into some other system for you? Great candidates for this.
  • Create interfaces by looking at the API you’re accessing. You know what classes you want to mock, so look how you’re using the API. If you need to access a few properties and methods, then make that part of your interface. If you see commonality between a few similar things, you might be able to create a single interface to handle all of the scenarios.
  • Inject factories that can create instances for you. These factories know how to create the concrete classes that meet your interfaces. In a real situation, they can create the classes you expect. In a test environment, they can create your mocks.
  • Write coded tests with your mocks! The last part is the most fun. You can finally inject some mocked classes into your classes/methods under test and then instrument them to ensure your code under test is accessing them in the way you expect. Run some code coverage tools after to prove you’re doing a good job.

I hope my experiences down this path are able to help you out!


IronPython: A Quick WinForms Introduction

IronPython: A Quick WinForms Introduction

Background

A few months ago I wrote up an article on using PyTools, Visual Studio, and Python all together. I received some much appreciated positive feedback for it, but really for me it was about exploring. I had dabbled with Python a few years back and hadn’t really touched it much since. I spend the bulk of my programming time in Visual Studio, so it was a great opportunity to try and bridge that gap.

I had an individual contact me via the Dev Leader Facebook group that had come across my original article. However, he wanted a little bit more out of it. Since I had my initial exploring out of the way, I figured it was probably worth trying to come up with a semi-useful example. I could get two birds with one stone here–Help out at least one person, and get another blog post written up!

The request was really around taking the output from a Python script and being able to display it in a WinForm application. I took it one step further and created an application that either lets you choose a Python script from your file system or let you type in a basic script directly on the form. There isn’t any fancy editor tools on the form, but someone could easily take this application and extend it into a little Python editor if they wanted to.

Leveraging IronPython

In my original PyTools article, I mention how to get IronPython installed into your Visual Studio project. In Visual Studio 2012 (and likely a very similar approach for other versions of Visual Studio), the following steps should get you setup with IronPython in your project:

  • Open an existing project or start a new one.
  • Make sure your project is set to be at least .NET 4.0
    • Right click on the project within your solution explorer and select “Properties”
    • Switch to the “Application” tab.
    • Under “Target framework”, select  “.NET Framework 4.0”.
  • Right click on the project within your solution explorer and select “Manage NuGet Packages…”.
  • In the “Search Online” text field on the top right, search for “IronPython”.
  • Select “IronPython” from within the search results and press the “Install” button.
  • Follow the instructions, and you should be good to go!

Now that we have IronPython in a project, we’ll need to actually look at some code that gets us up and running with executing Python code from within C#. If you followed my original post, you’ll know that it’s pretty simple:


var py = Python.CreateEngine();
py.Execute("your python code here");

And there you have it. If it seems easy, that’s because it is. But what about the part about getting the output from Python? What if I wanted to print something to the console in Python and see what it spits out? After all, that’s the goal I was setting out to accomplish with this article. If you try the following code, you’ll notice you see a whole lot of nothing:


var py = Python.CreateEngine();
py.Execute("print('I wish I could see this in the console...')");

What gives? How are we supposed to see the output from IronPython? Well, it all has to do with setting the output Stream of the IronPython engine. It has a nice little method for letting you specify what stream to output to:


var py = Python.CreateEngine();
py.Runtime.IO.SetOutput(yourStreamInstanceHere);

In this example, I wanted to output the stream directly into my own TextBox. To accomplish this, I wrote up my own little stream wrapper that takes in a TextBox and appends the stream contents directly to the Text property of the TextBox. Here’s what my stream implementation looks like:


private class ScriptOutputStream : Stream
{
  #region Fields
  private readonly TextBox _control;
  #endregion

  #region Constructors
  public ScriptOutputStream(TextBox control)
  {
    _control = control;
  }
  #endregion

  #region Properties
  public override bool CanRead
  {
    get { return false; }
  }

  public override bool CanSeek
  {
    get { return false; }
  }

  public override bool CanWrite
  {
    get { return true; }
  }

  public override long Length
  {
    get { throw new NotImplementedException(); }
  }

  public override long Position
  {
    get { throw new NotImplementedException(); }
    set { throw new NotImplementedException(); }
  }
  #endregion

  #region Exposed Members
  public override void Flush()
  {
  }

  public override int Read(byte[] buffer, int offset, int count)
  {
    throw new NotImplementedException();
  }

  public override long Seek(long offset, SeekOrigin origin)
  {
    throw new NotImplementedException();
  }

  public override void SetLength(long value)
  {
    throw new NotImplementedException();
  }

  public override void Write(byte[] buffer, int offset, int count)
  {
    _control.Text += Encoding.GetEncoding(1252).GetString(buffer, offset, count);
  }
  #endregion
}

Now while this isn’t pretty, it serves one purpose: Use the stream API to allow binary data to be appended to a TextBox. The magic is happening inside of the Write() method where I take the binary data that IronPython will be providing to us, convert it to a string via code page 1252 encoding, and then append that directly to the control’s Text property. In order to use this, we just need to set it up on our IronPython engine:


var py = Python.CreateEngine();
py.Runtime.IO.SetOutput(new ScriptOutputStream(txtYourTextBoxInstance), Encoding.GetEncoding(1252));

Now, any time you output to the console in IronPython you’ll get your console output directly in your TextBox! The ScriptOutputStream implementation and calling SetOutput() are really the key points in getting output from IronPython.

The Application at a Glance

I wanted to take this example a little bit further than the initial request. I didn’t just want to show that I could take the IronPython output and put it in a form control, I wanted to demonstrate being able to pick the Python code to run too!

Firstly, you’re able to browse for Python scripts using the default radio button. Just type in the path to your script or use the browse button:

IronPython - Run script from file

Enter a path or browse for your script. Press “Run Script” to see the output of your script in the bottom TextBox.

Next, press “Run Script”, and you’re off! This simply uses a StreamReader to get the contents of the file and then once in the contents are stored in a string, they are passed into the IronPython engine’s Execute() method. As you might have guessed, my “helloworld.py” script just contains a single line that prints out “Hello, World!”. Nothing too fancy in there!

Let’s try running a script that we type into the input TextBox instead. There’s some basic error handling so if your script doesn’t execute, I’ll print out the exception and the stack trace to go along with it. In this case, I tried executing a Python script that was just “asd”. Clearly, this is invalid and shouldn’t run:

python_error_asd

Python interpreted the input we provided but, as expected, could not find a definition for “asd”.

That should be along the lines of what we expected–The script isn’t valid, and IronPython tells us why. What other errors can we see? Well, the IronPython engine will also let you know if you have bad syntax:

python_error_bad_syntax

Python interpreted the script, but found a syntax error in our silly input.

Finally, if we want to see some working Python we can do some console printing. Let’s try a little HelloWorld-esque script:

python_pass_hello_world

Python interpreted our simple Hello World script.

Summary

This sample was pretty short but that just demonstrates how easy it is! Passing in a script from C# into the IronPython is straight forward, but getting the output from IronPython is a bit trickier. If you’re not familiar with the different parts of the IronPython engine, it can be difficult to find the things you need to get this working. With a simple custom stream implementation we’re able to get the output from IronPython easily. All we had to do was create our own stream implementation and pass it into the SetOutput() method that’s available via the IronPython engine class. Now we can easily hook the output of our Python scripts!

As always, all of the source for you to try this out is available online:

Some next steps might include:

  • Creating your own Python IDE. Figure out some nice text-editing features and you can run Python scripts right from your application.
  • Creating a test script dashboard. Do you write test scripts for other applications in Python? Why not have a dashboard that can report on the results of these scripts?
  • Add in some game scripting! Sure, you could have done this with IronPython alone, but maybe now you can skip the WinForms part of this and just make your own stream wrapper for getting script output. Cook up some simple scripts in a scripting engine and voila! You can easily pass information into Python and get the results back out.

Let me know in the comments if you come up with some other cool ideas for how you can leverage this!


v6.2 of IEF from Magnet Forensics! – Weekly Article Dump

IEF v6.2 from Magnet Forensics

v6.2 Release: Mobile Forensics Upgrade

I like to be able to use these weekly article dumps for little summaries of what’s going on in my work life, and I think this is a perfect opportunity to acknowledge our latest product update at Magnet Forensics. We just pushed out v6.2 of Internet Evidence Finder and we’re incredibly proud of the work we’ve done. Like any release we have, we pour our hearts into making sure it’s a few big steps forward. We’ve done our best to listen to customers and work with them to address any bugs, but we’re always trying to push the boundaries in our features.

Some of the new offerings in v6.2 of Internet Evidence Finder include:

  • Dynamic App Finder: We now offer a solution for recovering mobile chat applications that we may not have otherwise supported. This is a great discovery tool and has proved to be very powerful even in our early tests. Read more about it here. v6.2’s secret weapon!
  • Chat Threading: Visualize chat threads within our software as they look in their native applications. If you’re looking at a Skype conversation between two or more people, it will show up just like it does from within Skype. A lot less jumping between records to piece together a conversation.
  • L2T CSV Support: L2T CSV files can now be loaded directly into our timeline viewer.
  • Case Merging: Combine multiple IEF cases together or pull in data from TLN/L2T CSV files.
  • More Artifacts: v6.2 is no different than previous releases when it comes to adding new artifacts!
    • AVI carving
    • Hushmail
    • TOR chat
    • Flash cookies
    • Offline gmail
    • Additional Chrome support
    • … and more.

If you’re a forensic investigator, v6.2 is going to be an awesome upgrade or addition to your suite of tools. If you’re not, then check out Magnet Forensics to see what we’re all about and so proud of what we do. Congrats to Magnet on an awesome release of v6.2!

Articles

  • In praise of micromanagement: I’m still very early on in my career, so it’s difficult for me to have an opinion on this article and back it up. It’s a bit controversial, so of course I want to take the other side and disagree with it.There’s that, and I have some discomfort when it comes to Apple so I like to turn off when I see articles on Apple or Steve Jobs. Regardless, I thought that there was an interesting perspective in this piece to share, and maybe even if I can’t see it right now, others would benefit from reading through it. Is there a place for micromanagement? Can it be done right? Are people like Steve Jobs just an exception to an otherwise good rule?
  • The Myth of the Rockstar Programmer: Scott Hanselman says that rockstar programmers don’t exist–rockstar teams do. I completely agree. When your company is so small that you essentially don’t have teams, this might not hold. Maybe you have three developers and each one is a rockstar in their own right. That’s probably a it different. More often than not, you’re not working with one or two people developing a product for a company. It’s not about having one rockstar with all the programming super-powers take charge on the team. It’s about creating a team where everyone has their own strengths and weaknesses and then organizing them to operate at full efficiency. Teams. Not individuals.
  • Strengthen and Sustain Culture with Storytelling: This is an article that I can really align myself with. Nancy Duarte writes about something that’s often lost when small startups are transitioning into small businesses. It’s entirely possible some companies don’t even make it out of the start up phase because this thing is already going south. Storytelling. It’s important to be able to share stories with people as you bring them on board to your company. They need to know where the company has been and how it’s gotten to where it is. New hires need to feel like their part of the family as they are brought on board, and without conveying your company’s mission and values properly you start to lose that alignment.
  • Ignoring Your Test Suite: Another programming article here, but this post by Jesse Taber has some deeper lessons to be learned, in my opinion. The article talks about something not all programmers do, but should: write code that tests their code. This lets developers catch problems early on (because catching a problem now might cost a bit of time, but catching the problem later could be devastating). Running code tests regularly is a process that allows you to ensure the foundation of your software product is structurally sound. But what happens when you have flaky tests? What happens when you introduce a new failure and don’t bother to fix it? After all, you have 3000 tests, and you know why test ABC is failing anyway. Don’t put processes in place just for the sake of having them. Everything you do should be done for a reason, because your business doesn’t have time for anything else. Don’t enable poor habits. If you’re noticing problems in your process, identify why they are happening and look to get them fixed. Maybe you need to adjust your process because it doesn’t fit anymore.
  • Cameron Sapp – Recognizing The New Guy: This one is from me. I wrote up a little recognition piece about a colleague and teammate, Cam Sapp. I want to be able to write more recognition posts, but I started with Cam. He’s been a great addition to our team both from a technical and work culture perspective. All of Magnet is glad to have him on board.
  • Don’t Work For Your Boss, Work For Your Company: I thought that Ilya Pozin had written something great when I cam across this article. Hierarchies in the workplace can often cause disconnect and disengage employees. So why do we have them? I’m not against hierarchies–I think they serve a purpose. However, I think necessary measures need to be put in place to ensure that hierarchies aren’t detracting from the company’s goals. In this article, Ilya says to not work for your boss. Your goals at work should not be to satisfy individuals or only do things for your boss so you can get your promotion. Align yourself to the company values and the mission of the company. You’ll remain engaged and happy to do the work you’re doing. In the end, if you’re not happy doing work that’s aligned with your company’s mission, vision, and values, you might be in the wrong place.
  • Creativity and the Role of the Leader: This article discusses where ideas come from and how leaders fit in to the grand scheme of things. The traditional mindset is that ideas come from the top and then are pushed down to employees to carry out the work necessary for bringing the idea to fruition. However, it’s increasingly more common where ideas are actually generated by employees, and it’s the responsibility of the leader for nurturing idea creation and ensuring that ideas that are aligned with the company’s mission can succeed.
  • Will Your Firm Endure?: In this article by Tim Williams, I took away two key points. In order for your business to be absolutely sure it can endure, everyone needs to be viewed as replaceable. I don’t mean in the sense where we can trade John for Joe and not care because we don’t value human qualities, I mean strictly from the skills and responsibility aspect. There shouldn’t be instanced in your business where if an individual were to disappear one day your company wouldn’t be able to carry on. The next is acknowledging strengths and weaknesses. When people have some obvious strengths, they have weak areas too. There’s nothing wrong with that. It’s normal. Make sure your teams are constructed of people with complementary skills.
  • Dynamic Programming with Python and C#: Another article from me, and another programming related post. This my follow up to a post about C# and Python integration that seems to have been received really well. It was a cool little experiment for me to take Python and C# and have them working together in my favourite IDE, but on top of that, I was actually able to learn a bit about C#’s “dynamic” keyword which was new for me. If you’re familiar with either of C# or Python I recommend checking it out. There’s some pretty cool stuff you can do, and I’ve only scratched the surface.
  • To Find Success, Forget Your PrioritiesClaire Diaz-Ortiz says that priorities are too general. We all have priorities, but how many of us are seeing ourselves achieve what we’d like? Claire suggests forgetting your priorities and breaking them down into goals you can achieve. By having conrete action plans, you can execute them properly.
  • Personality Tests: Modern-Day Phrenology: Ron Baker shares his perspective on why personality tests don’t have a place at work. He goes as far as calling them meaningless, but I believe his main argument is that simply siloing people into personality types is pointless. To that end, I agree. I thought this article had great timing because I’ve been discussing personality tests with our HR manager at work. I came across this article right before doing a personality test with her and we decided a few things. Firstly, if the results of the test don’t make sense, then don’t go any further with it. This means that either the test you’re using is flawed or perhaps you don’t understand the test. Regardless, how can you take action on something you don’t understand? We both agreed that simply identifying traits was useless on it’s own, so I think we agreed with Ron on this one, but we weren’t stopping there. The basic act of identifying personality traits had us sparking conversations about how our personalities were different and how acknowledging these differences could influence our interactions. Essentially, it was hard to just silo ourselves into a particular personality type without thinking about and acting on what we were observing. In the end, identifying personality types and sticking someone into some cookie-cutter process for it means nothing. The tests are all about ganining insight and understanding so that we can choose where to go from there.
  • How Open Should a Startup CEO Be with Staff?: Coming from a startup, this was another interesting article. Mark Suster writes a semi-controversial perspective about CEO transparency. The norm is that expecting CEO’s to share every bit of details with the employees achieved perfect transparency and makes everything better. Mark says this definitely isn’t the case and provides some excellent examples where total transparency came back to bite. It’s all about balance. Transparency is great,but total transparency is often too much for most employees to handle on a day-to-day basis.

Follow Dev Leader on social media outlets to get these updates through the week.

Nick Cosentino – LinkedIn
Nick Cosentino – Twitter
Dev Leader – Facebook
Dev Leader – Google+
Nick’s CodeProject Articles

You can also check out Dev Leader on FlipBoard.


Dynamic Programming with Python and C#

Dynamic Coding with C# and Python

Dynamic Code: Background

Previously, I was expressing how excited I was when I discovered Python, C#, and Visual Studio integration. I wanted to save a couple examples regarding dynamic code for a follow up article… and here it is! (And yes… there is code you can copy and paste or download).

What does it mean to be dynamic? As with most things, wikipedia provides a great start. Essentially, much of the work done for type checking and signatures is performed at runtime for a dynamic language. This could mean that you can write code that calls a non-existent method and you wont get any compilation errors. However, once execution hits that line of code, you might get an exception thrown. This Stack Overflow post’s top answer does a great job of explaining it as well, so I’d recommend checking that out if you need a bit more clarification. So we have statically bound and dynamic languages. Great stuff!

So does that mean Python is dynamic? What about C#?

Well Python is certainly dynamic. The code is interpreted and functions and types are verified at run time. You won’t know about type exceptions or missing method exceptions until you go to execute the code. For what it’s worth, this isn’t to be confused with a loosely typed language. Ol’ faithful Stack Overflow has another great answer about this. The type of the variable is determined at runtime, but the variable type doesn’t magically change. If you set a variable to be an integer, it will be an integer. If you set it immediately after to be a string, it will be a string. (Dynamic, but strongly typed!)

As for C#, in C# 4 the dynamic keyword was introduced. By using the dynamic keyword, you can essentially get similar behaviour to Python. If you declare a variable of type dynamic, it will take on the type of whatever you assign to it. If I assign a string value to my dynamic variable, it will be a string. I can’t perform operations like pre/post increment (++) on the variable when it’s been assigned a string value without getting an exception. If I assign an integer value immediately after having assigned a string value, my variable will take on the integer type and my numeric operators become available.

Where does this get us with C# and Python working together then?

Example 1: A Simple Class

After trying to get some functions to execute between C# and Python, I thought I needed to take it to the next level. I know I can declare classes in Python, but how does that look when I want to access it from C#? Am I limited to only calling functions from Python with no concept of classes?

The answer to the last question is no. Most definitely not. You can do some pretty awesome things with IronPython. In this example, I wanted to show how I can instantiate an instance of a class defined within a Python script from C#. This script doesn’t have to be created in code (you can use an external file), so if you need more clarification on this check out my last Python/C# posting, but I chose to do it this way to have all the code in one spot. I figured it might be easier to show for an example.

We’ll be defining a class in Python called “MyClass” (I know, I’m not very creative, am I?). It’s going to have a single method on it called “go” that will take one input parameter and print it to the console. It’s also going to return the input string so that we can consume it in C# and use it to validate that things are actually going as planned. Here’s the code:

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Scripting.Hosting;

using IronPython.Hosting;

namespace DynamicScript
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            Console.WriteLine("Enter the text you would like the script to print!");
            var input = Console.ReadLine();

            var script =
                "class MyClass:\r\n" +
                "    def __init__(self):\r\n" +
                "        pass\r\n" +
                "    def go(self, input):\r\n" +
                "        print('From dynamic python: ' + input)\r\n" +
                "        return input";

            try
            {
                var engine = Python.CreateEngine();
                var scope = engine.CreateScope();
                var ops = engine.Operations;

                engine.Execute(script, scope);
                var pythonType = scope.GetVariable("MyClass");
                dynamic instance = ops.CreateInstance(pythonType);
                var value = instance.go(input);

                if (!input.Equals(value))
                {
                    throw new InvalidOperationException("Odd... The return value wasn't the same as what we input!");
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Oops! There was an exception while running the script: " + ex.Message);
            }

            Console.WriteLine("Press enter to exit...");
            Console.ReadLine();
        }
    }
}

Not too bad, right? The first block of code just takes some user input. It’s what we’re going to have our Python script output to the console. The next chunk of code is our Python script declaration. As I said, this script can be loaded from an external file and doesn’t necessarily have to exist entirely within our C# code files.

Within our try block, we’re going to setup our Python engine and “execute” our script. From there, we can ask Python for the type definition of “MyClass” and then ask the engine to create a new instance of it. Here’s where the magic happens though! How can we declare our variable type in C# if Python actually has the variable declaration? Well, we don’t have to worry about it! If we make it the dynamic type, then our variable will take on whatever type is assigned to it. In this case, it will be of type “MyClass”.

Afterwards, I use the return value from calling “go” so that we can verify the variable we passed in is the same as what we got back out… and it definitely is! Our C# string was passed into a Python function on a custom Python class and spat back out to C# just as it went in. How cool is that?

Some food for thought:

  • What happens if we change the C# code to call “go1” instead of “go”? Do we expect it to work? If it’s not supposed to work, will it fail at compile time or runtime?
  • Notice how our Python method “go” doesn’t have any type parameters specified for the argument “input”? How and why does all of this work then?!

Example 2: Dynamically Adding Properties

I was pretty excited after getting the first example working. This meant I’d be able to create my own types in Python and then leverage them directly in C#. Pretty fancy stuff. I didn’t want to stop there though. The dynamic keyword is still new to me, and so is integrating Python and C#. What more could I do?

Well, I remembered something from my earlier Python days about dynamically modifying types at run-time. To give you an example, in C# if I declare a class with method X and property Y, instances of this class are always going to have method X and property Y. In Python, I have the ability to dynamically add a property to my class. This means that if I create a Python class that has method X but is missing property Y, at runtime I can go right ahead and add property Y. That’s some pretty powerful stuff right there. Now I don’t know of any situations off the top of my head where this would be really beneficial, but the fact that it’s doable had me really interested.

So if Python lets me modify methods and properties available to instances of my type at runtime, how does C# handle this? Does the dynamic keyword support this kind of stuff?

You bet. Here’s the code for my sample application:

using System;
using System.Collections.Generic;
using System.Text;

using Microsoft.CSharp.RuntimeBinder;

using IronPython.Hosting;

namespace DynamicClass
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            Console.WriteLine("Press enter to read the value of 'MyProperty' from a Python object before we actually add the dynamic property.");
            Console.ReadLine();

            // this script was taken from this blog post:
            // http://znasibov.info/blog/html/2010/03/10/python-classes-dynamic-properties.html
            var script =
                "class Properties(object):\r\n" +
                "    def add_property(self, name, value):\r\n" +
                "        # create local fget and fset functions\r\n" +
                "        fget = lambda self: self._get_property(name)\r\n" +
                "        fset = lambda self, value: self._set_property(name, value)\r\n" +
                "\r\n" +
                "        # add property to self\r\n" +
                "        setattr(self.__class__, name, property(fget, fset))\r\n" +
                "        # add corresponding local variable\r\n" +
                "        setattr(self, '_' + name, value)\r\n" +
                "\r\n" +
                "    def _set_property(self, name, value):\r\n" +
                "        setattr(self, '_' + name, value)\r\n" +
                "\r\n" +
                "    def _get_property(self, name):\r\n" +
                "        return getattr(self, '_' + name)\r\n";

            try
            {
                var engine = Python.CreateEngine();
                var scope = engine.CreateScope();
                var ops = engine.Operations;

                engine.Execute(script, scope);
                var pythonType = scope.GetVariable("Properties");
                dynamic instance = ops.CreateInstance(pythonType);

                try
                {
                    Console.WriteLine(instance.MyProperty);
                    throw new InvalidOperationException("This class doesn't have the property we want, so this should be impossible!");
                }
                catch (RuntimeBinderException)
                {
                    Console.WriteLine("We got the exception as expected!");
                }

                Console.WriteLine();
                Console.WriteLine("Press enter to add the property 'MyProperty' to our Python object and then try to read the value.");
                Console.ReadLine();

                instance.add_property("MyProperty", "Expected value of MyProperty!");
                Console.WriteLine(instance.MyProperty);
            }
            catch (Exception ex)
            {
                Console.WriteLine("Oops! There was an exception while running the script: " + ex.Message);
            }

            Console.WriteLine("Press enter to exit...");
            Console.ReadLine();
        }
    }
}

Let’s start by comparing this to the first example, because some parts of the code are similar. We start off my telling  the user what’s going to happen and wait for them to press enter. Nothing special here. Next, we declare our Python script (again, you can have this as an external file) which I pulled form this blog. It was one of the first hits when searching for dynamically adding properties to classes in Python, and despite having limited Python knowledge, it worked exactly as I had hoped. So thank you, Zaur Nasibov.

Inside our try block, we have the Python engine creation just like our first example. We execute our script right after too and create an instance of our type defined in Python. Again, this is all just like the first example so far. At this point, we have a reference in C# to a type declared in Python called “Properties”. I then try to print to the console the value stored inside my instances property called “MyProperty”. If you were paying attention to what’s written in the code, you’ll notice we don’t have a property called “MyProperty”! Doh! Obviously that’s going to throw an exception, so I show that in the code as well.

So where does that leave us then? Well, let’s add the property “MyProperty” ourselves! Once we add it, we should be able to ask our C# instance for the value of “MyProperty”. And… voila!

Some food for thought:

  • When we added our property in Python, we never specified a type. What would happen if we tried to increment “MyProperty” after we added it? What would happen if we tried to assign an integer value of 4 to “MyProperty”?
  • When might it be useful to have types in C# dynamically get new methods or properties?

Summary

With this post, we’re still just scratching the surface of what’s doable when integrating Python and C#. Historically, these languages have been viewed as very different where C# is statically bound and Python is a dynamic language. However, it’s pretty clear with a bit of IronPython magic that we can quite easily marry the two languages together. Using the “dynamic” keyword within C# really lets us get away with a lot!

Source code for these projects is available at the following locations:


Why Events? Flexibility.

Background

There are many different approaches to developing software, but in my opinion, the opposite ends of the spectrum end up being:

  • Knowing how the whole system looks, feels, and operates before coding a single line.
  • Having an idea of what the user wants and coding to make it happen.

Although I’m generalizing a lot here, it’s sort of like the battle between Waterfall and Agile. Okay, great. So what am I rambling on about here?

Well, in the first case, you know all the ins and outs of the system. You can structure your system so that almost no matter how complex it is, you can ensure that method A is always run immediately after method B which is etc… The design is completely controlled. You have a spec for how all the components work together. The problem? Well, most of this will be reserved for another write-up, but in our case there’s no flexibility. If that spec changes after your complicated system is already in place, how much code do you have to go change?

 

Enter: Events

Okay, so I hinted at why something like Waterfall might cause some headaches, but you don’t need to develop your software in an agile fashion to have flexibility. Events can help you with that!

If you’re not already familiar with what an event is, it’s simply a delegate that one would call when some particular criteria is met. In C#, which is what I’ll be assuming for this write-up, you add event handlers to an event so that when the event is invoked, the event handlers get called (and they get called in the order that they were “hooked” onto the event). Splendid.

So what’s great about events? How are they going to fix your brittle code? Well, if you approach software design in terms of components in a system and how they interact, it’s really beneficial to think about how certain things react to each other. If you’re planning your whole system out ahead of time, you could just always call method A, then method B, and then method C in order every time you want something in particular to happen. But is that flexible? What happens when method B now calls an additional method, let’s say method D, due to some architectural changes that had to be made. Well… That’s all fine and dandy unless you have scenarios where you don’t want to run D in your A,B,C method calls you have everywhere.

So, what if you had thought about it this way:

  • I want B to run once A is run.
  • I need an event for A
  • I need to call method B in my event handler for event A
  • I want C to be run once B is run.
  • I need an event for B
  • I need to call method C in my event handler for event B
And then someone comes along and throws method D into the mix. Well, you know that there is at least one particular case where you want D to be run after B is run… So make another event handler! Your listing now changes to:
  • I want B to run once A is run.
  • I need an event for A
  • I need to call method B in my event handler for event A
  • I want C to be run once B is run.
  • I need an event for B
  • I need to call method C in my event handler for event B
  • I want D to be run when condition X is met after B is run
  • I need an event for B (which I already have!)
  • I need to call method C in my event handler for event B
    • I can add a whole second event handler for event B (so I get C then D called)
    • OR I can modify the existing event handler that calls method C to conditionally call method D when necessary

And now how do you check the condition? Pass in some fancy event args with your state.

 

Summary

Events can help you keep your code flexible when your structure is changing because they provide hooks into the code when certain conditions are met. You don’t need to rely on a firmed-up design for how methods are required to be called in some particular order. If your design changes because you want something to execute under different conditions, just hook up to the event that meets those conditions instead and re-write minimal code.

I’m huge on using events in my architectures, so I’ll try to write up a few more posts on events and why I like them. The flexibility I just described is only one very small part of it, and it certainly requires that you shift your perspective on how components in a system interact.


  • Nick Cosentino

    Nick Cosentino

    I work as a team lead of software engineering at Magnet Forensics (http://www.magnetforensics.com). I'm into powerlifting, bodybuilding, and blogging about leadership/development topics over at http://www.devleader.ca.

    Verified Services

    View Full Profile →

  • Copyright © 1996-2010 Dev Leader. All rights reserved.
    Jarrah theme by Templates Next | Powered by WordPress