Sunday, July 10, 2011

Functional Object Manipulation

 

In the last couple of posts, we’ve looked at ways to copy scalar properties from one object to another, with the high performance of explicit assignments and the low maintenance of reflection.

The approach of creating a lambda to operate on the object instance as a whole shows a lot of promise. For example, we may want to compare two object instances property-by-property, and the code for that would look quite similar.

public static class Comparer
{
public static bool Compare<T>(this T source, T destination) { return CompareClosure<T>.Compare(source, destination); }

private static class CompareClosure<T>
{
private static Func<T, T, bool> BuildComparerLambda(Func<PropertyInfo, bool> propertyFilter = null)
{
propertyFilter = propertyFilter ?? (_ => true);

var sourceParameterExpression = Expression.Parameter(typeof(T), "_left");
var destinationParameterExpression = Expression.Parameter(typeof(T), "_right");

var properties = typeof(T).GetProperties().Where(propertyFilter);
var expressions = properties.Select(_pi => Expression.Equal(Expression.Property(destinationParameterExpression, _pi), Expression.Property(sourceParameterExpression, _pi)));
var conjoinedExpression = expressions.Aggregate<Expression, Expression>(Expression.Constant(true), Expression.AndAlso);

var lambdaExpression = Expression.Lambda<Func<T, T, bool>>(conjoinedExpression, sourceParameterExpression, destinationParameterExpression);
return lambdaExpression.Compile();
}

private static readonly Func<T, T, bool> _compare = BuildComparerLambda();
public static bool Compare(T source, T destination) { return _compare(source, destination); }
}
}

which gives us:


comparer


What are are doing in the code is mapping each property to a boolean value indicating whether the two instances have the same value for the property, and then folding that set of boolean values to a single boolean value by aggregating it with the && operator (and a seed value of ‘true’).


We can readily see a pattern emerging here:


There are two types of operations possible for a given arity. In the case of Compare and Copy where the operation arity is 2, we can create either an Action<T, T> or a Func<T, T, TResult>.


The Action<> case is a lambda that performs an operation on each property with reference to the given argument instances.


We can create a general operation builder by lifting a parameter which specifies the desired operation on a property, with reference to the lambda arguments.

public static Action<T, T> BuildOperationAction<T>(
Func<PropertyInfo, ParameterExpression, ParameterExpression, Expression> operationExpressionFactory)
{
var leftParameterExpression = Expression.Parameter(typeof (T), "_left");
var rightParameterExpression = Expression.Parameter(typeof (T), "_right");

var properties = typeof (T).GetProperties();
var expressions = properties.Select(_ => operationExpressionFactory(_, leftParameterExpression, rightParameterExpression));

var blockExpression = Expression.Block(expressions);
var lambdaExpression = Expression.Lambda<Action<T, T>>(blockExpression, leftParameterExpression, rightParameterExpression);

return lambdaExpression.Compile();
}

The Func<> case is a lambda that maps the set of properties to a set of values with reference to the argument instances, and then aggregates this set of values to a single value using an aggregator.


We can create a general operation builder by lifting:



  1. a parameter which specifies the desired operation on a property, with reference to the lambda arguments
  2. a parameter which specifies the aggregate operation to fold the set of computed values to a single value, and
  3. a parameter specifying the initial value for the aggregator.

Thusly:

public static Func<T, T, TResult> BuildOperationFunc<T, TResult>(
Func<PropertyInfo, ParameterExpression, ParameterExpression, Expression> operationExpressionFactory,
Func<Expression, Expression, Expression> conjunction, Expression seed)
{
var leftParameterExpression = Expression.Parameter(typeof (T), "_left");
var rightParameterExpression = Expression.Parameter(typeof (T), "_right");

var properties = typeof (T).GetProperties();
var expressions = properties.Select(_ => operationExpressionFactory(_, leftParameterExpression, rightParameterExpression));

var joinedExpression = expressions.Aggregate(seed, conjunction);
var lambdaExpression = Expression.Lambda<Func<T, T, TResult>>(joinedExpression, leftParameterExpression, rightParameterExpression);

return lambdaExpression.Compile();
}

This allows us to implement Copy() and Compare() as follows:

public static bool CompareWith<T>(this T source, T destination) { return CompareClosure<T>.Compare(source, destination); }
private static class CompareClosure<T>
{
private static readonly Func<T, T, bool> _compare =
BuildOperationFunc<T, bool>(
(_pi, _src, _dest) => Expression.Assign(Expression.Property(_dest, _pi), Expression.Property(_src, _pi)),
Expression.AndAlso,
Expression.Constant(true));

public static bool Compare(T source, T destination) { return _compare(source, destination); }
}

public static void CopyTo<T>(this T source, T destination) { CopyClosure<T>.Copy(source, destination); }
private static class CopyClosure<T>
{
private static readonly Action<T, T> _copy =
BuildOperationAction<T>(
(_pi, _src, _dest) => Expression.Assign(Expression.Property(_dest, _pi), Expression.Property(_src, _pi)));

public static void Copy(T source, T destination) { _copy(source, destination); }
}

Very neat.


We could develop more operations than Copy and Compare, building fast, automatically maintained mappers for data access (think ORMs) and UI field access, by simply specifying the equivalent expression. The development effort is simply in the development of the expression.


For example, here is a mapper which maps properties defined on type T, from a dynamic source object to backing fields on a target object of type T:

private static readonly Action<T, T> _copyDynamicToStaticBackingFields = 
BuildOperationAction<T>(
(_pi, _src, _dest) =>
{
var sourceBinder = RuntimeBinder.GetMember(CSharpBinderFlags.InvokeSpecialName, _pi.Name, typeof (T), new [ ] { ThisArgument });
var sourceGetExpression = Expression.Convert(Expression.Dynamic(sourceBinder, typeof (object), _src), _pi.PropertyType);

var fieldName = String.Format("_{0}{1}", _pi.Name.Substring(0, 1).ToLower(), _pi.Name.Substring(1));

var fieldInfo = typeof (T).GetField(fieldName, BindingFlags.Instance | BindingFlags.NonPublic);
if (fieldInfo == null) return Expression.Default(typeof (void));

var destinationAccessExpression = Expression.Field(_dest, fieldInfo);
return Expression.Assign(destinationAccessExpression, sourceGetExpression);
});

We could further develop this approach by developing the BuildOperationXXX functions for Action<>s and Func<>s of arity 1, which would allow us to quickly develop fast, low-maintenance object printers and serializers.


I believe that this set of generating functions is likely to be more generally useful, so I’ve created a CodePlex project for this. Please visit it and let me know how you used it!

The Object Copier, revisited.

 

We saw in the previous post how we could improve the maintainability of reflection-based code by building lambdas using reflection, and then gain the performance benefits by applying the lambdas at execution time.

Taking the idea a step further, we can easily see that what we are really doing is generating a function to do the copy operation on the entire object and applying the copy function.

(T _source, T _destination) =>
{
_destination.Id = _source.Id;
_destination.Name = _source.Name;
...
}

Based on the approach we took last time, this is easily achievable by wrapping the individual assignments into a block statement, wiring through the parameter expressions from the block into the individual lambdas, and then compiling out a full copy. Thusly:

public static class Copier
{
public static void CopyTo<T>(this T source, T destination) { CopyClosure<T>.Copy(source, destination); }

private static class CopyClosure<T>
{
private static Action<T, T> BuildCopierLambda()
{
var sourceParameterExpression = Expression.Parameter(typeof(T), "_source");
var destinationParameterExpression = Expression.Parameter(typeof(T), "_destination");

var properties = typeof(T).GetProperties();
var expressions = properties.Select(_pi => Expression.Assign(Expression.Property(destinationParameterExpression, _pi), Expression.Property(sourceParameterExpression, _pi)));

var blockExpression = Expression.Block(expressions);
var lambdaExpression = Expression.Lambda<Action<T, T>>(blockExpression, sourceParameterExpression, destinationParameterExpression);

return lambdaExpression.Compile();
}

private static readonly Action<T, T> _copy = BuildCopierLambda();
public static void Copy(T source, T destination) { _copy(source, destination); }
}
}


There are a few things to note about the code:


The outer, public, static  Copier class is non-generic to allow the CopyTo operation to be defined as an extension property on generic type T.


The compiler will create a special, unique CopyClosure class for each type T. This ensures that the _copy Action is unique for each type T. Also, the _copy Action is initialized in a thread-safe manner, because the compiler guarantees thread-safety on field initializers.


This approach actually improves performance, because there is no run-time iteration!


copy_block_expression


Stay tuned for more improvements…

Monday, June 27, 2011

A (Fast) Object Copier

 

A colleague of mine asked me today to help with the onerous task of “twisting-off” the data in an entity into a DTO. Something about serialization, he said…

He said that he didn’t mind if the copying was slow – just that he didn’t want to put in hundreds of lines of code each saying

destination.Id = source.Id;
destination.Name = source.Name;
destination.DateAndTimeOfBirth = source.DateAndTimeOfBirth;
destination.BirthWeightInKilograms = source.BirthWeightInKilograms;
destination.IsMultipleBirth = source.IsMultipleBirth;
destination.BirthOrder = source.BirthOrder;
...


So I sat down and knocked off the traditional reflection-based code:

internal static T Map<T>(T source, T destination) where T : class
{
foreach (var pi in typeof (T).GetProperties())
{
pi.SetValue(destination,
pi.GetValue(source, BindingFlags.GetProperty | BindingFlags.Instance, null, null, null),
BindingFlags.SetProperty | BindingFlags.Instance,
null,
null,
null);
}
return destination;
}


(I’ve removed the error checking just to keep things simple for this post)



Neat and all, and does the job.



But I couldn’t help wondering if I could do better – specially since I knew that the performance would potentially be too terrible to bear…



Express(ion) yourself!



It would be nice to use the tricks we learned with expressions to create, for a given type, a set of assignment expressions for each of the properties, and then simply apply all expressions one by one to the source and destination, and that way the cost of reflection is borne only once.



Fair enough. I got home, and after dinner, knocked off this piece of code:

public static class FastCopier
{
public static T Copy<T>(T source, T destination) where T : class
{
return FastCopierInternal<T>.Copy(source, destination);
}

private static class FastCopierInternal<T>
{
private static readonly IDictionary<PropertyInfo, Action<T, T>> _copyFunctions = typeof (T).GetProperties().ToDictionary(_pi => _pi, GetCopyFunc);

private static Action<T, T> GetCopyFunc(PropertyInfo propertyInfo)
{
Trace.WriteLine(String.Format("Creating mapping func for {0}.{1}", propertyInfo.DeclaringType.Name, propertyInfo.Name));
var sourceParameterExpression = Expression.Parameter(typeof (T), "_source");
var destinationParameterExpression = Expression.Parameter(typeof (T), "_destination");

var sourceAccessExpression = Expression.Property(sourceParameterExpression, propertyInfo);
var destinationAccessExpression = Expression.Property(destinationParameterExpression, propertyInfo);

var sourceAssignmentExpression = Expression.Assign(destinationAccessExpression, sourceAccessExpression);

var lambdaExpression = Expression.Lambda<Action<T, T>>(sourceAssignmentExpression,
sourceParameterExpression,
destinationParameterExpression);

return lambdaExpression.Compile();
}

public static T Copy(T source, T destination)
{
foreach (var func in _copyFunctions.Values)
{
func(source, destination);
}

return destination;
}
}
}


Here we have a static class with a single public method to do the Copy for a given type T.



Internally, we have a static generic copier class which only gets instantiated once for each type T, and which builds up an associative array of the properties of the type and the function required to copy each property from source to destination.



Here’s what the lambda expression looks like as it is built:



assignment_lambda



When the public Copy method on the outer class is called, it calls the Copy method on the (implicitly initalized) inner static class, which applies each of the copy functions in turn to copy the source to the destination one property at a time.



Pretty cool. So how much faster are we?

Explicit copy of object 1000000 times took 204 ms
Fast copy of object 1000000 times took 790 ms
Slow copy of object 1000000 times took 23801 ms


Running over a million objects, the fastest, as expected, is when we explicitly code the DTO initialization, and let the compiler optimize.



The lambda version is around 3.5x slower, but still pretty darn fast compared with the basic, slow, all-reflection version – which is two orders of magnitude slower!



Not a bad option to have, when the development and maintenance cost of creating the DTOs and keeping them up to date is factored in!

Sunday, June 5, 2011

Memoization Is Your Friend

 

I just came across some gems regarding memoization here, here, here and here. As a side note, I’d like to go on record and say that Wes Dyer’s derivation of fixed-point combinators is second to none! Even I could understand what he was saying! :) Another great discussion of fixed-point combinators is here. For more theoretical underpinnings, this paper discusses memoization of fixed-point functions.

We’re being plagued by performance issues in the project I’m currently working on, and it would be really nice to have a quick way to to automatically memoize some functions to gain the benefits of laziness and good book-keeping.

So, here is my implementation of a Memoize() function, which uses the fixed-point combinator concepts explained in Wes’ blog.

public static Func<TArgument, TResult> MemoizeFix<TArgument, TResult>(Func<Func<TArgument, TResult>, Func<TArgument, TResult>> func)
{
// ReSharper disable AccessToModifiedClosure
Func<TArgument, TResult> funcMemoized = null;

funcMemoized = func(_ => funcMemoized(_));
funcMemoized = funcMemoized.Memoize();

return funcMemoized;
// ReSharper restore AccessToModifiedClosure
}

private static Func<TArgument, TResult> Memoize<TArgument, TResult>(this Func<TArgument, TResult> func)
{
var _map = new ConcurrentDictionary<TArgument, TResult>();
return _ => _map.GetOrAdd(_, func);
}

In MemoizeFix(), we create the funcMemoized variable as null so that we get around the definite assignment problem. Because we defer the assignment to after the application of the memoization, we actually want to have access to the modified closure.

The other nice thing with .NET 4.0 is the ability to use the ConcurrentDictionary as a thread-safe result cache, with the added bonus of a very terse invocation to GetOrAdd().


We can use the memoization routines this way:

private static readonly Func<int, long> FastFibonacci = Functional.MemoizeFix<int, long>(
fib => n => (n <= 0 ? 0 : (n <= 2 ? 1 : fib(n - 1) + fib(n - 2)))
);

private static readonly Func<int, long> FastLucas = Functional.MemoizeFix<int, long>(
luc => (n => FastFibonacci(n) + FastFibonacci(n + 2))
);

private static readonly Func<int, double> FastFactorial = Functional.MemoizeFix<int, double>(
fact => n => n < 2 ? 1.0d : n*fact(n - 1)
);

For functions with more than one argument, I’d create a Tuple<> with all the arguments as the cache key as the simplest extension. Thusly:

private static Func<TArg1, TArg2, TResult> Memoize<TArg1, TArg2, TResult>(this Func<TArg1, TArg2, TResult> func)
{
var _map = new ConcurrentDictionary<Tuple<TArg1, TArg2>, TResult>();
return (_a1, _a2) => _map.GetOrAdd(new Tuple<TArg1, TArg2>(_a1, _a2), _ => func(_.Item1, _.Item2));
}

public static Func<TArg1, TArg2, TResult> MemoizeFix<TArg1, TArg2, TResult>(Func<Func<TArg1, TArg2, TResult>, Func<TArg1, TArg2, TResult>> func)
{
// ReSharper disable AccessToModifiedClosure
Func<TArg1, TArg2, TResult> funcMemoized = null;

funcMemoized = func((_a1, _a2) => funcMemoized(_a1, _a2));
funcMemoized = funcMemoized.Memoize();

return funcMemoized;
// ReSharper restore AccessToModifiedClosure
}


Pretty slick :)

Wednesday, April 20, 2011

C# Abuse

 

So I just watched this video presentation by Jon Skeet about some of the edges of our understanding of the C# language.

In this blog post, he threw open the challenge to define three static methods such that the following code compiles:

static void Main() 
{
Foo<int>();
Foo<string>();
Foo<int?>();
}

While watching the video and playing around on my own, I came up with:

class Program
{
static void Foo<T>() { Console.WriteLine(typeof(T).Name); }

class Inner : Program
{
struct R<T> where T : class { }

static void Foo<T>(params R<T>?[] _) where T : class { Console.WriteLine(typeof(T).Name); }
static void Foo<T>(params T?[] _) where T : struct { Console.WriteLine(typeof(T).Name); }

static void Main()
{
Foo<int>();
Foo<string>();
Foo<int?>();
}
}
}

which prints out the correct result. The analysis of how this works is given brilliantly over at Jon’s blog!


The key is the inner class, which forces the compiler to first check signature suitability for the two methods defined therein, before allowing the base class to catch-all with the most generalized signature.


Figure 1


The same caveat that Jon specified in his blog applies – such code should *never* be written!

Sunday, April 17, 2011

Lambda Wrangling For Fun : Part 4 – Sorting Multiple Columns

 

We can approach the problem of sorting a dataset by multiple properties using the same techniques as for the filter case.

A single-property sorting lambda looks like:

….OrderBy(_ => _.property) or ….OrderByDescending(_ => _.property)

If we wanted to sort by more than one property, we would do something like:

….OrderBy(_ => _.property1).OrderByDescending(_ => _.property2)

It’s important to notice two points:



  • The lambdas for each sort clause are self-contained – unlike the composite filter lambda where the same parameter is threaded through each predicate.
  • The direction of the sort is specified by the use of either the OrderBy or OrderByDescending extension method. This means that the composite sort criteria will be represented by a chain of OrderBy and/or OrderByDescending methods.

Bearing these points in mind, we can develop a general Sort action specification:

public interface ISort<TSource>
{
string PropertyName { get; }
SortDirection SortDirection { get; }

Expression<Func<TSource, object>> GetSelectorLambda();
}

The properties are as we expected, and we have encountered a variant GetSelectorLambda before.


Applying a composite sort on an enumerable can be achieved by:

public static IOrderedEnumerable<T> Sort<T>(this IEnumerable<T> _this, IEnumerable<ISort<T>> sortOperations)
{
var parameterExpression = Expression.Parameter(typeof (IEnumerable<T>), "_");

Expression items = parameterExpression;
foreach (var sortOperation in sortOperations)
{
// selector : __ => __.{sortOperation.PropertyName}
var selector = sortOperation.GetSelectorLambda();

// items.OrderBy<T, object>(__ => __.Property)
// is actually
// Enumerable.OrderBy<T, object>(items, __ => __.Property)
items = Expression.Call(typeof (Enumerable),
sortOperation.SortDirection == SortDirection.Ascending ? "OrderBy" : "OrderByDescending",
new [ ] { typeof (T), typeof (object) },
items,
selector);
}

// _ => _.OrderBy[Descending](__ => __.Property1)...OrderBy[Descending](__ => __.PropertyN)
var lambda = Expression.Lambda<Func<IEnumerable<T>, IOrderedEnumerable<T>>>(items, parameterExpression);

// compile the lambda...
var func = lambda.Compile();

// ...call the chain of extension methods on (_this)
return func(_this);
}

The function generates calls to either the ‘OrderyBy’ or the ‘OrderByDescending’ method on the Enumerable type, using the previous result as the first argument, thus generating the chain.


Figure1It finally compiles and executes the chain of methods on the actual enumerable instance passed in.


Here is a typical use:

var sorts = new List<ISort<IPerson>> {
new Sort<IPerson>("Timestamp", SortDirection.Descending),
new Sort<IPerson>("Name", SortDirection.Ascending),
};

var sortedPersons = persons.Sort(sorts);

Saturday, April 16, 2011

Lambda Wrangling For Fun : Part 3 – Filtering Multiple Columns

 

We now know how to generate a dynamic predicate to filter a collection of items by a single property value.

It is often useful to be able to apply multiple filters to a collection to support more sophisticated searching and filtering operations.

Let’s define a general-purpose filter interface to encapsulate the property, the target value and the comparison function.

public interface IFilter
{
string PropertyName { get; }
dynamic TargetValue { get; }
Expression<Func<dynamic, dynamic, bool>> Comparison { get; }
Expression<Func<TSource, bool>> GetPredicateLambda<TSource>();
Expression BuildComparisonExpression<TSource>(Expression parameterExpression = null);
}

The three properties are what we expect to see, and we’ve already seen a variant of the GetPredicateLambda function before.


Before we see what the other method is needed for, let’s analyse what it means to filter multiple columns first:


One Fish, Two Fish…


A filter with a single predicate comparing against a value looks thus:

….Where(_ => predicate0(_.property0, value0))

and if we had to filter by more than one predicate would look like:

….Where(_ => predicate0(_.property0, value0) 
&& predicate1(_.property1, value1))

Notice that what we need to do is to create just the comparison part part of the lambda for each filter, with the same parameter injected into each predicate call.


This is exactly what the BuildComparisonExpression function does. Here is the implementation:

public virtual Expression BuildComparisonExpression<TSource>(Expression parameterExpression = null)
{
// This gives us * _ *
parameterExpression = parameterExpression ?? Expression.Parameter(typeof (TSource), "_");

var propertyInfo = typeof (TSource).GetProperty(PropertyName, true);
if (propertyInfo == null)
{
throw new MemberAccessException(String.Format("No such property exists: {0}", PropertyName));
}

// This gives us * _.property *
var accessorExpression = Expression.Property(parameterExpression, propertyInfo);

// This gives us * value *
var valueExpression = Expression.Constant(TargetValue);

// This gives us * comparison(_property, value) *
return Expression.Invoke(Comparison, accessorExpression, valueExpression);
}

The key to this method is the last line. Expression.Invoke generates an expression which invokes the Comparison expression with the accessorExpression and valueExpression as its arguments.


Also, we do not generate the Lambda with this expression, as we may have more than one Comparison to chain together.


We do that as follows:

public static Expression<Func<TSource, bool>> GetFilterExpression<TSource>(this IEnumerable<IFilter> filterCollection)
{
if (filterCollection == null) return _ => true;

var parameterExpression = Expression.Parameter(typeof (TSource), "_");

var aggregateFilterExpression = filterCollection.Aggregate(
(Expression) Expression.Constant(true),
(current, filter) => Expression.AndAlso(current,
filter.BuildComparisonExpression<TSource>(parameterExpression)));

return Expression.Lambda<Func<TSource, bool>>(aggregateFilterExpression, parameterExpression);
}

This function aggregates all the filter expressions, threading through the same parameterExpression, and builds the lambda function to call.


This is what a complex filter expression looks like:


Figure 1


Clean and straightforward lambdas.


We can now extend this approach to use conjunctions other than AndAlso, and perhaps special-case some comparison operations.