Decide between In-Process or Inter-Process Communication at Deploy Time - Part 3

This is the last post from the series on how we could implement a design that allows to decide only at deploy time how two services communicate: in process if they are deployed on the same server or inter-process if they are on different machines. The first post shows where such a design is useful and what are the benefits it brings. The second one, takes an example from the financial systems world, and outlines the key design ideas on how to implement it. In this post we'll only focus on code. We'll take the example presented in the previous post and code it from scratch in C#.

Dependency Injection is the base technique to implement all the key design ideas detailed in the previous post. In this demo, we'll use the iQuarc.AppBoot, which brings on top of a classic DI Container, the support for implementing simple conventions for the type discovery and the proxies. (More on iQuarc AppBoot Library here)

The entire source code that we are going to walk through is available on Github, here, as part of my Code Design training. Each implementation step is marked with a tag in the git repo. We have ipc-step0 for the starting point and so on, until ipc-step10b. You can see how the implementation evolves step by step.

Running the Demo

If we checkout the final step (ipc-step10b), we have what we wanted in the previous post: the ability to change the communication between the OrdersService and QuotationService to be in-process without changing code or recompile.

To get that running, after you build the TradingApp.sln you can find into the bin\.Deploy folder different deployments configurations (there are some post build events that copy the binaries in these folders). If you run the startAll.bat you will start three console processes, one for each of our three services from the example. They are all hosted by themselves, and each of them listens for REST calls on a different port (I've mentioned previously that we're implementing the example using REST services). Now, we can open Postman (or other similar tool) and fire some calls to them:

http://localhost:9002/api/Quotation/GetByExchange?exchange=NYSE&instrument&from=2017-01-01&to=2017-01-01

this gets from the QuotationService the quotations for the NYSE in a time window. You can notice that each console writes the calls it received and the remote that made it, so we can see what happens in our demo app.

If we make a POST to the OrdersService, to place an order:

http://localhost:9003/api/Orders/PlaceSellLimitOrder?securityCode=AAPL.S.NASDAQ&sellingPrice=11.45&validUntil=2017-01-01

we see the call logged into the console of the process that hosts the OrdersService (the remote is a Postman-Token outlined with yellow below, as for the previous call)

and further more, we can see another REST call made by the OrdersService to the QuotationService. We see it logged in the console of the process that loaded the Quotation Module (the one at the bottom below). We see that the remote is a Service-Proxy with the GUID of the proxies loaded by the process that hosts the Sales Module, as outlined with red below. This means that the one that made the call to the Quotation Module host was the Sales Module host.

So we had an inter-process communication, the SalesService making a REST call to the QuotationService, when someone else made a call to it.

Now, lets change the communication. We first close the process with the Sales Module. Then we copy in its output folder (bin\Sales\) the QuotationServices.dll. This is the binary that has the implementation of the QuotationService. Mainly we've changed the deployment configuration, by having now in the same output folder the binaries with implementations of the SalesService and of the QuotationService. Now, we just restart the host by executing the bin\Sales\ConsoleHost.exe. We see that it loaded both Quotation Module and Sales Module.

The other two console hosts remain running.

Now, if we repeat the same POST from Postman to the SalesService

http://localhost:9003/api/Orders/PlaceSellLimitOrder?securityCode=AAPL.S.NASDAQ&sellingPrice=11.45&validUntil=2017-01-01

we see that no other inter-process call is made. Noting gets logged in the other consoles. The SalesService did call the QuotationService since it depends on it, but the call was just a function call in the process that hosts them both.

So, only by the fact that we've changed the deployment by copying the QuotationsService.dll into the same \bin folder with the SalesService.dll and then restart the host process to load them both, we have switched from a inter-process communication to an in process communication.

We started with a deployment like below,

where each service was hosted in its own console host and an external call to the OrdersService triggered an inter-process communication with the QuotationServce, and we ended with a deployment like below

where OrdersService and QuotationService are hosted by the same process and an external call to the OrdersService can by satisfied with an in process communication with the QuotationService.

Writing the Code

If we checkout the ipc-step0 we have the solution structure to start with. At this step all the projects are empty.

We have a console project (ConsoleHost), which will be our generic host. Then we have class library projects where we'll implement the services, and another one for the contracts.

The solution structure is a key element of our design and I have detailed in the previous post here, the basic rules we want to enforce with it.

Basic Code for Contracts and Services

Next, if we checkout ipc-step1 we have the contracts of the QuotationService, an interface for the service contract and a DTO for the data contract:

public interface IQuotationService
{
    Quotation[] GetQuotations(string exchange, string instrument, DateTime from, DateTime to);
    Quotation[] GetQuotations(string securityCode, DateTime from, DateTime to);
}

public class Quotation
{
    public DateTime Timestamp { get; set; }
    public decimal BidPrice { get; set; }
    public decimal AskPrice { get; set; }
    public string SecurityCode { get; set; }
}

Next, we write a simple implementation for it, as a class in the Quotation.Services project (checkout ipc-step2):

class QuotationService : IQuotationService
{
    private readonly Quotation[] array = 
        {
            new Quotation {AskPrice = 10.50m, BidPrice = 10.55m, SecurityCode = "ING.S.NYSE"},
            new Quotation {AskPrice = 12.50m, BidPrice = 12.55m, SecurityCode = "ING.B.NYSE"},
            ...
        };
    
    public Quotation[] GetQuotations(string exchange, string instrument, DateTime @from, DateTime to)
    {
        var result = array.Where(q => q.SecurityCode.Contains(exchange));

        if (!string.IsNullOrWhiteSpace(instrument))
            result = result.Where(q => q.SecurityCode.Contains(instrument));

        return result.ToArray();
    }

    public Quotation[] GetQuotations(string securityCode, DateTime @from, DateTime to)
    {
        return array.Where(q => q.SecurityCode == securityCode).ToArray();
    }
}

Similar, we add the contract and a simple implementation for the OrderingService in the Sales.Services project (checkout ipc-step3). Notice here the dependency to the IQuotationService:

public class OrdersService : IOrdersService
{
    private readonly IQuotationService quotationService;
    public OrdersService(IQuotationService quotationService)
    {
        this.quotationService = quotationService;
    }

    public void PlaceSellLimitOrder(string securityCode, decimal sellingPrice, DateTime validUntil)
    {
        var todayQuotations = quotationService.GetQuotations(securityCode, DateTime.Today.AddDays(-1), DateTime.Today);
        foreach (var quotation in todayQuotations)
        {
            if (quotation.AskPrice >= sellingPrice)
                limitOrders.Add(new LimitOrder
                {
                    SecurityCode = securityCode,
                    PlacedAt = DateTime.UtcNow,
                    Type = OrderType.Sell,
                    Price = sellingPrice,
                    ValidUntil = validUntil
                });
        }
    }
...
}

Similar at ipc-step4 we have the PortfolioService implementation in the Portfolio.Services project. It also has a dependency to the IQuotationService

As pointed in the previous post here it is a critical requirement for this design that the dependencies among the services are towards interfaces. So, the OrdersService class depends on the IQuotationService interface and not on the QuotationService class. The same is for the PortfolioService.

Add iQuarc.AppBoot

At this point we have service contracts and implementations for them but they are not linked in any way. If we look at the Project Dependencies Diagram we see that we have no references nor strong dependencies among them.

In the previous post here, we've detailed why this is important.

The iQuarc.AppBoot will bring things together. We'll use it for type discovery to find the service implementations deployed, and then to satisfy the dependencies by configuring the underlying DI Container.

So, the next step is to install and configure it. We can install it as a NuGet Package to all the projects under the \Modules folder.

PM> Install-Package iQuarc.AppBoot -Version 2.0.3-disposables002

The iQuarc.AppBoot package doesn't have dependencies to any DI Container, and it is not a container by itself. It uses a container by an abstraction. It is good to also keep our modules (the place where we implement the business logic of our system) independent of a specific DI framework.

However, the host project needs to use a specific container, so for it only, we will install the iQuarc.AppBoot.Unity, which adapts the Unity Dependency Injection Container to AppBoot.

PM> Install-Package iQuarc.AppBoot.Unity -Version 2.0.3

The ConsoleHost is our generic server process. It should find all the services deployed, then publish them to be called from other processes and configure the dependencies. In a real system this would be an web app in IIS or a Cloud Service, or a Windows Service etc. For this demo it is a simple console app.

To use AppBoot to find the deployed services, first we need to decorate all the service implementations with the ServiceAttribute.

[Service(typeof(IQuotationService))]
class QuotationService : IQuotationService
{
...
}

With this we declare that the QuotationService class is the implementation of the IQuotationService interface. The same goes for the other service we have implemented in the \Modules folder.

[Service(typeof(IPortfolioService))]
public class PortfolioService : PortfolioService
{
...
}

[Service(typeof(IOrdersService))]
public class OrderingService : IOrdersService
{
...
}

Now, as a first step, when the ConsoleHost starts, in its main() it will call AppBoot.Bootstrapper.Run() which will scan through reflection all the binaries in the output folder and look after the types decorated with the ServiceAttribute. The ones that are found are registered in the DI Container as pairs of interface - implementation. This configuration is in the helper class AppBootBootstrapper:

public static class AppBootBootstrapper
{
    public static Bootstrapper Run()
    {
        var assemblies = GetApplicationAssemblies().ToArray();  //returns all the assemblies that should be scanned (filters out Microsoft.*, System.* and others)
        Bootstrapper bootstrapper = new Bootstrapper(assemblies);
        bootstrapper.ConfigureWithUnity(); // says to AppBoot to use Unity Container

        // configures the registration convention, by saying to use the ServiceAttribute convention
        bootstrapper.AddRegistrationBehavior(new ServiceRegistrationBehavior()); 

        bootstrapper.Run();
        return bootstrapper;
    }
...
}

To get this running we need to make sure that at build all the binaries are put in the same folder with the ConsoleHost. This helps in debug, and is the place where we can come later to define different deployment configurations. The easiest is to set the build output path for all the projects. Here is how it should be for a *.Services project:

Output path =  ..\..\..\bin\Debug\

So at the root level, we'll have a \bin folder where all the projects get build.

Self-Host ASP.NET Web API

For our demo purpose this is not that important. We just need to make the services available for REST calls, so a simple self host in the console app of the Web Api will do.

static void Main(string[] args)
{
    string baseAddress = "http://localhost:9000/";
    using (WebApp.Start<Startup>(url: baseAddress))
    {
        Console.WriteLine($"Server runs at: {baseAddress}");
        Console.WriteLine("Press ESC to stop the server\n");

        ConsoleKeyInfo keyInfo;
        do
        {
            keyInfo = Console.ReadKey();
        }
        while (keyInfo.Key != ConsoleKey.Escape);
    }
}

public class Startup
{
    // This code configures Web API. The Startup class is specified as a type
    // parameter in the WebApp.Start method.
    public void Configuration(IAppBuilder appBuilder)
    {
        // Configure Web API for self-host. 
        HttpConfiguration config = new HttpConfiguration();
        config.Routes.MapHttpRoute(name: "DefaultApi", routeTemplate: "api/{controller}/{action}");

        // kicks the AppBoot Bootstrapper
        AppBootBootstrapper.Run().ConfigureWebApi(config);

        appBuilder.UseWebApi(config);
    }
}

We see that at startup we kick the AppBootBootstrapper.Run() and then we call the ConfigureWebApi() helper. This configures the WebApi to use the AppBoot DI Container to inject the dependencies into the controller. Our controllers are trivial:

public class OrdersController : ApiController
{
    private readonly IOrdersService ordersService;
    public OrdersController(IOrdersService ordersService)
    {
        this.ordersService = ordersService;
    }

    public IHttpActionResult PlaceSellLimitOrder(string securityCode, decimal sellingPrice, DateTime validUntil)
    {
        ordersService.PlaceSellLimitOrder(securityCode, sellingPrice, validUntil);
        return Ok();
    }
...

They get through constructor DI the service they should make available through REST and they just forward the calls to it.

To make our ConsoleHost truly generic we should make a generic ApiController which gets the service through DI and forwards the call to it, and then we should customize the WebApi to use through some conventions our generic controller. Definitely it could be done, but it doesn't worth for our demo, so we just create three dummy controllers for each of our services.

A Fat Server

Now if we checkout ipc-step7b we have all of the above implemented. If we execute the ConsoleHost.exe we see that it loaded all the three modules, because they were all copied in this \bin folder. If we make the same POST call from Postman to the OrdersService it works and it will do an in process communication with the QuotationService.

Everything is hosted in one process, like a fat server, which does not scale.

If we delete the Quotations.Services.dll and run again the ConsoleHost.exe we see that it only loaded the two remaining modules: Portfolio and Sales. Now if we make the same POST call from Postman it returns a 500 Internal Server Error because the OrdersService did not find an implementation for its dependency to the IQuotationService.

http://localhost:9003/api/Orders/PlaceSellLimitOrder?securityCode=AAPL.S.NASDAQ&sellingPrice=11.45&validUntil=2017-01-01

Implement the Proxies

What we wanted above, is that when a dependency is not found in the same process to make a REST call to another process where that dependency is hosted. To make this happen, we need to create other implementations to the interfaces describing our contracts, which know how to forward the call over REST. These are the Proxies.

The proxies are part of the infrastructure. They are just plumbing code, no business logic, and we want them deployed on all our ConsoleHost instances to be the fail over when a real service implementation was not deployed. Therefore, we put them in the \Infrastructure folder.

Their code is not that important for our demo

class QuotationServiceProxy : IQuotationService
{
    public Quotation[] GetQuotations(string exchange, string instrument, DateTime @from, DateTime to)
    {
        using (HttpClient client = HttpHelpers.CreateNewClient<IQuotationService>())
        {
            string path = HttpHelpers.GetServicePath<IQuotationService>("GetByExchange");
            string uri = $"{path}?exchange={exchange}&instrument={instrument}&from={from}&to={to}";
            HttpResponseMessage response = client.GetAsync(uri).Result;
            if (response.IsSuccessStatusCode)
            {
                Quotation[] value = response.Content.ReadAsAsync<Quotation[]>().Result;
                return value;
            }

            throw new HttpException((int)response.StatusCode, response.Content.ReadAsStringAsync().Result);
        }
    }
...
}

They use a HttpClient, they build the URL to the service they want to call, based on some conventions, and they make the REST call. The response is then returned to the caller. We can checkout icp-step8b to have the proxies for all our services.

Implement the ServiceProxyAttribute

Now, we have two implementations for each of our contracts: the proxy and the real service implementation. The proxies are not yet used by the AppBoot. If we were to also decorate them with the ServiceAttribute as we did for the service implementations, the AppBoot will register both implementations for one interface and the last one found will overwrite the first. This is not deterministic, so it is not good. When a service is not deployed, as we've seen above, sometimes things will work, sometimes not, depending on which type was found last by the AppBoot while it scanned the deployed assemblies.

To fix this, we need to extend the conventions AppBoot uses. The ServiceAttribute we've used so far, was just a convention that says to register to DI interface - implementation pairs decorated with this attribute. We can create a new attribute say ServiceProxyAttribute and register a new convention to AppBoot.

[AttributeUsage(AttributeTargets.Class)]
public sealed class ServiceProxyAttribute : Attribute
{
    public ServiceProxyAttribute(Type exportType)
    {
        ExportType = exportType;
    }
    public Type ExportType { get; private set; }
}

public sealed class ServiceProxyRegistrationBehavior : IRegistrationBehavior
{
    public IEnumerable<ServiceInfo> GetServicesFrom(Type type)
    {
        IEnumerable<ServiceProxyAttribute> attributes = type.GetAttributes<ServiceProxyAttribute>(false);
        return attributes.Select(a => new ServiceInfo(a.ExportType, type, string.Empty, Lifetime.AlwaysNew));
    }
}

The convention is implemented as a IRegistrationBehavior, where we just return what should be registered to DI from a type that was found during the assemblies scanning. Checkout ipc-step9a to get these implemented and have all the proxies decorated with the ServiceProxyAttribute.

To make AppBoot use the new convention we go into the AppBootBootstrapper class and in the Run() function we add to the bootstrapper an instance of the ServiceProxyRegistrationBehavior

public static Bootstrapper Run()
{
    var assemblies = GetApplicationAssemblies().ToArray();
    Bootstrapper bootstrapper = new Bootstrapper(assemblies);
    bootstrapper.ConfigureWithUnity();
    bootstrapper.AddRegistrationBehavior(new ServiceProxyRegistrationBehavior());
    bootstrapper.AddRegistrationBehavior(new ServiceRegistrationBehavior());

    bootstrapper.Run();
    return bootstrapper;
}

The order we add the registration behaviors to the bootstrapper matters. The latter overwrites the registrations returned by the previous. So we add first the ServiceProxyRegistrationBehavior which puts into the DI Container proxies as implementations for all the contracts. If we'd leave it like this, all the service calls will be REST calls to other processes through the proxies. The ServiceRegistrationBehavior is the second, so it overwrites the proxies registrations with the real implementation for the services that were deployed. This means, that if the real implementation of a service was deployed, thus found at the assemblies scan, it will overwrite the proxy implementation, therefore the call to that service will be direct to real implementation in the current process. If it is not found, the proxy registration is not overwritten and the call will be done in another process.

With these done (checkout ipc-step10), we have the design completely implemented. Depending on what *.Services.dll are copied at deploy time into the \bin of a ConsoleHost.exe we have in processes communication or inter-process communication. Checkout ipc-step10a to get all the configurations tweaked for an easy run, and then you can re-run the demo as we did in the beginning of the post. You can play with different deployment configurations by copying the *.Services.dll in different ConsoleHost.exe instances. Here's another deployment configuration

This design is part of my Code Design Training where you will learn how to implement it in your context and to maximize the benefits for your requirements

Featured image credit: DENYS Rudyi via 123RF Stock Photo