Software projects rarely collapse because of lack of talent. They collapse because complexity grows faster than it is controlled.

When technical failure happens, the root cause is often unmanaged complexity.

You have seen it:
- Small changes become expensive.
- Adding developers does not increase speed.
- Deadlines move again and again.
- The system becomes difficult to reason about.

In some organizations, four engineers deliver in two years what thirty could not deliver in six. The difference is not effort. It is controlled structure.

Complexity does not disappear on its own. It must be designed under control.

Structure Creates Predictability

The primary mechanism for controlling complexity is structure.

A system that follows consistent patterns becomes understandable. When code design enforces structure, change becomes localized. Impact becomes predictable.

Well-defined structure limits the surface area of change. It encapsulates complexity instead of letting it spread across the system.

When constraints are built into the codebase, patterns emerge naturally. Those patterns reinforce maintainability, reuse, and long-term stability.

Predictability is not an accident. It is the outcome of disciplined Code Design.

In my work, I rely on three techniques to enforce structure and predictability:

• Hide the Frameworks
• Depend on Contracts, not Implementations
• Apply Consistent Design Patterns across Services

This article focuses on the first: hiding the frameworks.

Hide the Frameworks

Hiding frameworks behind application-specific abstractions is one of the most powerful Code Design techniques for long-term scalability.

It strengthens structure.
It improves predictability.
It protects your Infrastructure from uncontrolled drift.

There is an inherent conflict in software reuse.

Framework providers design generic APIs that fit as many use cases as possible. Broader adoption means greater success.

Application teams, on the other hand, need APIs tailored to their specific domain and delivery model. They care about their product, not universal flexibility.

By placing your own abstractions between the application code and external libraries, you resolve this tension.

Your application no longer depends directly on ASP.NET, Entity Framework, RabbitMQ, or a DI container. Instead, it depends on interfaces designed specifically for your domain.

This gives you control over:

• The allowed patterns
• The constraints
• The conventions
• The architectural rules

Instead of allowing each developer to use a framework differently, you enforce consistent usage patterns across the entire system.

Framework changes remain isolated. Volatility does not propagate through your codebase.

You move from framework-driven development to structure-driven Code Design.

Why This Matters Even More with Agentic AI

This approach becomes even more important when adopting Agentic AI in your development workflow.

AI agents generate code based on patterns and instructions. If your codebase exposes raw framework APIs everywhere, agents will replicate inconsistency.

When frameworks are hidden behind structured Infrastructure and clear contracts, you give agents:

• A constrained surface area
• Clear rules to follow
• Stable abstractions
• Explicit architectural boundaries

These become enforceable instructions inside your agent definitions.

Instead of generating ad-hoc framework calls, agents generate code aligned with your architectural patterns.

Structure enables feedback.
Infrastructure enables guardrails.
Predictability becomes systematic.

Without structure, Agentic AI amplifies chaos.
With structure, it amplifies disciplined Code Design.

Abstract the Data Access

Consider data access with Entity Framework.

EF supports multiple usage patterns: stateless vs stateful contexts, read-only optimizations, change tracking, concurrency strategies, and more.

Different scenarios require different patterns.

If developers use EF directly inside use cases, inconsistency emerges quickly. Query logic spreads. Data access rules diverge. Cross-cutting concerns multiply.

Instead, define your own repository abstractions. Implement them using EF internally.

Now:

• All LINQ queries flow through a controlled Infrastructure layer
• Read-only and read-write patterns are clearly separated
• Concurrency strategies are standardized
• Data access patterns are consistent across services

This centralization enables extensibility without rewriting business logic.

Need multi-tenancy with a TenantID discriminator?
Extend the Infrastructure layer to append tenant filters automatically.

Need auditing, authorization, or logging?
Implement them once inside the data access Infrastructure.

The use cases remain untouched.

This is scalable Code Design.
This is structured Infrastructure.

Abstract the Messaging

Messaging systems such as RabbitMQ or Kafka introduce their own complexity.

Concepts like exchanges, bindings, queues, routing keys, acknowledgments, and configuration strategies are powerful but generic.

They are designed for a wide range of use cases:

• Pub-sub
• Fire-and-forget commands
• Request-reply
• High throughput
• High reliability
• Distributed scale

But your system likely uses only a subset of these patterns.

Instead of exposing raw messaging APIs to application code, define your own abstractions:

• PublishEvent
• SendCommand
• SubscribeToEvent
• HandleCommand

Behind those abstractions, configure the message bus consistently.

Now:

• All services follow the same messaging patterns
• Reliability and throughput strategies are standardized
• Infrastructure configuration is centralized
• Cross-team predictability increases

Messaging becomes part of your Infrastructure, not a distributed design decision.

Patterns become explicit.
Structure becomes enforced.

This is critical when scaling delivery teams.

Application Infrastructure

All these abstractions form what I call Application Infrastructure.

Application Infrastructure is a structured layer that sits between:

• Application use cases
• External frameworks and libraries

It is feature-agnostic.
It does not implement business logic.
It defines how business logic interacts with technical concerns.

This Infrastructure layer establishes:

• Conventions
• Constraints
• Patterns
• Architectural guardrails

It shapes how Code Design is executed across the system.

When done correctly, Infrastructure becomes the foundation of predictability.

Developers follow established patterns instead of inventing new ones.
Architectural decisions are embedded in code structure.
AI agents operate within clearly defined boundaries.

This is how structure scales.

Key Takeaways

Hiding frameworks is not about rejecting tools. It is about reclaiming control over them.

By introducing structured Infrastructure between your application and external libraries, you:

• Reduce uncontrolled complexity
• Enforce consistent patterns
• Improve long-term maintainability
• Increase delivery predictability

In an Agentic AI era, structured Code Design becomes even more critical. Clear abstractions and consistent patterns give agents the boundaries they need to generate aligned, reliable output.

Complex systems require structure.
Structure enables predictability.
Predictability enables scalable delivery.

That is the real value of hiding the frameworks.

Energy systems must operate continuously. Orders must always be processed. Data loss is unacceptable. Business processes must remain visible and auditable at all times.

Most of these systems operate in heavily regulated environments and integrate directly with critical infrastructure. As a result, they are often deployed in private clouds or on-premises data centers, where reliability, security, and compliance take priority over raw scalability.

Architectural Trade-offs in Energy Systems

Designing resilient systems is largely about making the right trade-offs.

In the energy domain, Reliability and Consistency are usually prioritized over performance metrics such as latency or throughput. This often means:

• Accepting higher latency to support retries, timeouts, and fault tolerance
• Investing in redundant infrastructure, geographic replication, and failover mechanisms
• Adding architectural complexity through circuit breakers, health checks, and recovery logic
• Choosing strong consistency models, even if they introduce temporary unavailability

Applying these constraints uniformly across all services would lead to systems that are slow, rigid, and expensive to evolve. That approach does not scale well over the Long-Term.

A more effective strategy is to clearly separate responsibilities. Some services are designed as the system’s source of truth, optimized for reliability and consistency. Others prioritize responsiveness and usability, accepting eventual consistency to improve user experience and integration performance.

This balance is essential for Predictability, both technically and financially.

Structuring Systems Through Service Categories

A key principle in Code Design is reducing complexity through clear structure. Categorizing services helps enforce architectural boundaries, encourages reuse, and simplifies decision-making.

Common service categories include:

Core Services
These services focus on Reliability and Consistency. They typically rely on:

• Relational databases
• Reliable, durable messaging
• Workflow engines for resilient execution of business processes

They represent the authoritative state of the system and are critical for regulatory and operational correctness.

Support Services
These services are optimized for availability and responsiveness. They often use:

• NoSQL databases
• Event-driven communication or lightweight messaging
• Data models optimized for fast reads or high-volume ingestion, even if data is not always fully up to date

Their role is to shield users and external systems from the complexity and latency of core services.

Integration Gateways
Integration services handle communication with external parties such as Power Exchanges, TSOs, or asset control systems.
Their challenges include:

• Rate limiting and security
• Retries, circuit breakers, and fault isolation
• Protocol mismatches, data model differences, and data quality issues
• Monitoring and observability

Explicitly isolating these concerns prevents external complexity from leaking into the core system.

Redundancy as the Foundation of Resilience

Resilience is achieved through redundancy at both the compute and storage levels.

With redundant compute, each service runs multiple instances simultaneously. If one instance fails, others continue processing requests or handling workloads. Load balancers and controllers that automatically restart failed instances are essential for maintaining availability.

With redundant storage, data is replicated across disks, nodes, or locations. This protects against hardware failures and ensures that data remains accessible even when components fail. The same guarantees must apply not only to databases, but also to queues and long-lived caches.

Redundancy is a direct investment in Predictability and delivery On Time and On Budget.

Kubernetes as a Platform for Resilient Systems

Container orchestration is the standard approach for managing resilient compute and data access services.

Kubernetes has become a strong choice due to its mature ecosystem, broad adoption, and vendor neutrality. It allows teams to avoid lock-in while deploying consistently across public clouds, private clouds, or on-premises environments.

Kubernetes also supports geo-distributed deployments. Running clusters across multiple data centers enables traffic routing to healthy locations in case of regional failures, supporting high availability and disaster recovery requirements.

Reliable Messaging in Distributed Architectures

Asynchronous communication is fundamental for building scalable and resilient distributed systems.

When resilience is the goal, reliable messaging ensures that communication between services remains durable and correct, even in the presence of failures. This typically involves:

• Persistent queues
• Explicit acknowledgments
• Retry and delay mechanisms
• Guarantees around message delivery

RabbitMQ is one common solution, particularly in environments where cloud-managed messaging services are not an option. Running messaging infrastructure inside Kubernetes can offer operational consistency across deployment models.

However, message brokers introduce significant complexity. They come with their own concepts, failure modes, and tuning requirements. Encapsulating this complexity behind a dedicated messaging component is critical to keep the overall system maintainable over the Long-Term.

Workflow Engines for Long-Running Business Processes

Energy systems are driven by time-bound and stateful business processes. Examples include:

• Submitting bids by fixed deadlines
• Sending production or consumption plans
• Executing activation orders
• Validating and settling imbalance costs

These processes often span hours or days, involve external systems or user interaction, and must complete reliably despite failures.

Implementing such workflows directly in application code is risky. Process state would be lost on crashes and could not resume on another node. A workflow engine is required to persist state, handle retries, and provide visibility into execution progress.

There are off-the-shelf solutions such as Temporal or Azure Durable Functions. In some cases, a custom-built workflow engine is a better fit, as it can focus on the exact needs of the domain while reducing operational overhead.

Final Thoughts

Resilient software in the energy sector is not the result of isolated technical choices. It is the outcome of deliberate Code Design, clear architectural boundaries, and a constant focus on Predictability.

By structuring systems around service categories, embracing redundancy, and investing in reliable messaging and workflow execution, organizations can build platforms that evolve safely over the Long-Term and consistently deliver On Time and On Budget.

This is what enables energy systems to remain trustworthy, adaptable, and ready for the future.

This article outlines the principles and techniques I apply when building predictability into complex software projects. These are part of a broader Delivery Architecture mindset, where Code Design and System Design align with project execution to reduce uncertainty and increase control.

Why Predictable Delivery Is Hard

Software systems are inherently complex. Requirements shift, understanding deepens as work progresses, and visibility into progress can be limited. Many teams plan based on gut feeling or past experience, which breaks down quickly in larger or evolving projects.

Predictable outcomes don’t come from hope—they come from systems thinking and structured design. Just as you wouldn’t construct a building without blueprints, you shouldn’t start delivery without an architecture that supports it.

Key Practices for Predictable Software Delivery

Below are the core practices that I use to bring predictability into the delivery process, grounded in both Code Design and Project Design principles.

1. Map Work as a Dependency Network

Project activities should never be treated as a flat backlog. Each task interacts with others—some are prerequisites, others rely on shared resources or outputs. The first step in a reliable Delivery Architecture is to model your project as a graph of dependent activities.

By using an Activities Network (or Dependency Graph), you gain visibility into how tasks relate. This enables float analysis and critical path estimation—tools that are essential for accurate forecasting and risk management.

Dependencies may be:

• Technical: A feature needs a shared service or API.
• Resource-based: A specialist is needed in multiple places.
• Output-based: One component generates input for another.

This model becomes the foundation of your project planning and informs how teams are aligned and work is scheduled.

2. Stable Architecture Enables Predictable Change

A resilient System Design should not shift every time new features are introduced. Changing the architecture midway through a project is expensive and disruptive. A good Delivery Architecture assumes the system will evolve functionally without needing to be structurally redesigned.

To achieve this, avoid feature-based decomposition. Instead, design around volatility. Encapsulate areas that are expected to change frequently, so that those changes don’t ripple through the entire system.

This kind of separation makes change measurable and impact easier to assess—one of the cornerstones of predictable delivery.

3. Build in Stable Increments

Software delivery should progress incrementally. Once a module, service, or layer is complete, it should be locked unless there's a clear justification to revisit it. Otherwise, the project gets trapped in a loop of rework, and forward momentum stalls.

In Code Design, this means being intentional about which parts of the system are stable foundations, and which are extensions or configurable layers.

The analogy with construction applies: you pour the foundation, then build the structure floor by floor. You don’t pour the foundation, finish the kitchen, tear it down to redo the plumbing, and then go back to the basement.

You can still stagger work for speed, but components must be layered properly. For example, in real-world construction, while upper levels are being poured, the lower ones are already being finished.

Design, however, is cheap to iterate—before coding begins. That’s why System Design must be done interactively and iteratively, allowing multiple passes to validate assumptions before implementation begins.

4. Don’t Accumulate Bugs

Defects introduce chaos into schedules. A single unresolved bug can have cascading effects on other teams, timelines, and budgets.

The best way to control this uncertainty is to adopt a zero-tolerance policy for defects. Every bug is addressed immediately—not necessarily fixed right away, but fully understood and scoped. The unknown is the real problem, not the defect itself.

In a well-structured Delivery Architecture, bugs are treated as blockers to progress and eliminated as soon as they are found.

5. Use Estimate Ranges, Not Wishful Thinking

Estimates are necessary, but they should serve the plan—not the ego. The goal is to produce accurate, not overly precise, estimates.

Focus on activity duration buckets (5, 10, 15, 20 days). If something takes more than that, it should be broken down. This allows for averaging effects, where some overruns are offset by faster-than-expected completions.

The structure of your activity network will ultimately shape your delivery timeline more than individual estimates. That’s why detailed network modeling is a critical part of Project Design.

6. Track with Engineering Discipline

Planning is only half the equation. Without tracking, predictability fades fast.

Use your activities network as a live reference. Update it based on progress. Recalculate float and critical paths weekly. Redefine “done” as “reviewed and usable by the next activity.”

This tracking method enables timely corrective actions and gives real-time insight into how far off-plan you are and why.

A Delivery Architecture approach treats the project like a system—observe, measure, and adjust continuously.

Conclusion: Design + Execution = Predictability

Predictable delivery doesn’t come from luck or rigid processes. It comes from smart System Design, structured Code Design, and thoughtful Project Design—all aligned under a cohesive Delivery Architecture.

It also depends on tight collaboration between the architect and the project manager. The architect ensures the system is designed for change without disruption. The PM ensures the plan adapts to reality without losing control.

When design and execution stay in sync, delivery becomes something you can count on—not just hope for.

A structured architecture removes ambiguity. It gives both humans and AI a clear model of what good code looks like. In my approach to Code Design, the Application Infrastructure defines strict boundaries and provides the building blocks that guide the implementation. These constraints make it easy to follow the intended Software Design and harder to introduce patterns that would break Clean Architecture principles.

Originally, this setup was meant to support developers. The goal was simple: let them focus on implementing features in the problem domain and avoid accidental complexity. But the same idea now applies to Copilot and Agentic AI as well. A strong architectural foundation doesn’t just help teams—it also helps AI generate better, more consistent code.

Application Infrastructure

A good Application Infrastructure isn’t about business features. It’s about the technical substrate that keeps the system healthy through clear abstractions, predictable behavior, and enforced boundaries.

By hiding external frameworks behind project-specific interfaces, the infrastructure shapes how feature code is written. This creates a high level of Structure and Consistency, which directly supports maintainability. It also establishes conventions that show “how things are done here.”

These conventions translate well into AI instructions. Copilot can follow rules, recognize patterns, and replicate examples with surprising accuracy—especially when the architectural boundaries are explicit.

Two main goals drive the design of the infrastructure:

• Establish a consistent structure that guides how features are implemented
• Hide complexity behind stable abstractions

Both goals help Copilot produce cleaner and more predictable code. A well-defined API gives it one correct way of completing a task. And a coding environment with fewer choices tends to reduce mistakes.

One rule remains critical: AI should not modify the infrastructure itself.
That foundation defines the architecture. It sets the boundaries, maintains Clean Architecture principles, and keeps the system stable. Copilot should operate on top of it, not change it.

Copilot for Feature Implementation

Feature code sits above the Application Infrastructure and depends on it. This is where Copilot shines. Most systems rely on a limited number of core use cases, and many features are just variations of those patterns. With clear constraints in place, an AI Agent can generate feature logic effectively and safely.

Why offload feature code to Copilot?

• Feature code changes more often than the infrastructure. Using AI here reduces the cost of change.
• Rich context improves accuracy. Copilot can use user stories, acceptance criteria, and examples to guide generation.
• The layering enforces separation. Even if Copilot writes imperfect code, other modules remain unaffected.
• Feature code is less critical. Reliability and cross-cutting concerns are already handled elsewhere.
• Each feature has a bounded context. Rules, abstractions, and local conventions prevent architectural drift.

This combination—strong boundaries and flexible feature logic—creates a productive environment where AI can accelerate development without compromising design quality.

Experiment on Workshop Labs

In my Application Infrastructure for Clean Architecture workshop, participants work through eight labs designed to show how the structure comes together. The same environment is a great testbed for AI.

The workshop repo includes foundational components such as AppBoot and DataAccess. They are production-ready building blocks that anyone can adopt, adapt, and test in their own projects.

For this experiment, I asked Copilot to complete the labs—or heavily assist in them.
The setup:

• Visual Studio 2026
• .NET 10
• Agent Mode enabled
• Models used: GPT-5 mini and Claude Sonet 4.5
• One shared copilot-instructions.md file

Experiment Results

What Copilot handled well

• Understood and respected the dependency rules defined in the infrastructure
• Followed the structure and boundaries described in the instruction file
• Performed well on repetitive or verbose tasks, such as user interaction or logging
• Created new modules correctly when examples were available
• Applied Dependency Inversion and layering principles correctly

Where Copilot struggled

• Data access code required corrections—extra round trips or misuse of abstractions
• DTO generation occasionally hallucinated
• Did not improve or extend the infrastructure (as expected)
• Sometimes produced overly verbose code, easy to clean up but not ideal

Overall, Copilot performed best when the task lived inside clear boundaries. Predictability came from the architecture—not the model.

Details

Instruction File

The experiment started by generating a copilot-instructions.md file.
There are more advanced ways to approach this—such as model-specific variants with an index—but I used a simpler method to move fast.

I generated a first draft with ChatGPT-5.1, providing it with my training materials and pointing it to the workshop repo. It produced a solid outline that captured the core ideas of the infrastructure. I refined it further to match the Labs repo and the demo Application Infrastructure.

This file became the backbone of the experiment, guiding both models through the architectural rules and conventions.

Some sections to highlight:

1) Folder & Layering Rules (must follow)

repo-root/
├─ Infra/                         # Application Infrastruture (Application Framework, DataAccess, Logging, Messaging etc) 
│  ├─ AppBoot/                    # dependency injection, modules composition, app startup, plugins dynamic load
│  ├─ AppBoot.UnitTests/          # Unit tests for AppBoot
│  ├─ DataAccess/                 # Hides EF Core, IRepository and IUnitOfWork implementations
├─ Modules/                       # Functionalities grouped by domain (Sales, Notifications, Export etc).
│  ├─ Contracts/                  # Contracts shared between modules (e.g., Events, Messages, DTOs). No logic here!
│  ├─ Sales/                      # Sales module (example)
│  │  ├─ Sales.Services/          # Use-cases implementations, domain services.
│  │  ├─ Sales.DataModel/         # [Example] Entities, DTOs mapped to DB tables. No logic here! (no if, while, logical expressions etc.). NO reference to EF Core!
│  │  ├─ Sales.DbContext/         # [Example] EF DbContext for Sales module
│  │  └─ Sales.Console/           # [Optional] Console UI commands specific to sales module.
│  └─ Notifications/              # Notifications module (example)
│     └─ Notifications.Services/  # Use-cases implementations, domain services.
└─ UI/                            # User Interface layer / Clients   
   └─ ConsoleUi/                  # Console application (CLI)

I think this helped a lot, because I didn’t get any misplaced files.

2) Dependency boundaries

- `Infra/*` → implements ports for DB, messaging, HTTP, files, dymamic load of plugins; registers via DI; no domain logic.
- `Modules/Contracts` → **no** references to anything. Only pure DTOs and interfaces. No logic
- `Modules/*` → **no** references to other modules. Only references to **Contracts** and **Infra**.
- `Modules/*/*.DataModel` → **no** logic; only entities/DTOs; no references to EF Core or other frameworks.
- `Modules/*/*.Services` → references **Contracts** and **DataModel**; NO references ot EF Core or other frameworks. Contains domain logic and use-cases.
- `UI/*` → references **Modules/Contracts** and **Infra**; NO references to **Modules/*/Services**. No domain logic.


> **Copilot:** If a change violates these rules, raise an error instead of making the change.

 The cheaper models like “GPT-5 mini” did not raise any errors, but they still followed the rules.

The more advanced models like “Claude Sonet 4.5” refused to make changes that would violate them.

3) Registering in DI

- Use `ServiceAttribute` from `Infra/AppBoot` to register services in DI.
- The `ServiceAttribute` decorates the implementation class, specifying the service lifetime and the interface to register.
- Register only interfaces, not concrete classes.
- Example of the `PriceCalculator` class registered as the implementation of the `IPriceCalculator` interface:

[Service(typeof(IPriceCalculator), ServiceLifetime.Transient)]
class PriceCalculator : IPriceCalculator
{
    public decimal CalculateTaxes(OrderRequest o, Customer c)
    {
    }
}

This made a big difference. The Copilot, even with the cheaper models, consistently registered services correctly.

5) AppBoot Plugin Model

- AppBoot supports dynamic loading of modules as plugins at runtime.
- In `Program.cs` where AppBoot is configured, use `.AddPlugin()` to specify the modules that should be loaded as plugins.
- Load as plugins all assemblies that are not referenced by any other assembly at compile time (i.e., have no incoming `ProjectReference` in the solution).
- Each call to `.AddPlugin()` creates a LoadContext isolated for that module assembly; dependent assemblies passed in the dependency array are loaded into the same LoadContext.
- `AddPlugin()` accepts modules names, which are built by convvention as `{ModuleName}.{AssemblySuffix}`. 
  - The `ModuleName` correspond to the folder name under `Modules/` (e.g., `Sales`, `Notifications`).
  - The `AssemblySuffix` is the assembly name without the module name prefix (e.g., `Services`, `DbContext`).
  - Assembly are named by convention as `{ModuleName}.{AssemblySuffix}` (e.g., `Sales.Services`, `Notifications.Services`, `Sales.DbContext`).
- When a module has dependent assemblies that are not referenced by the assembly that gives the plugin name, specify their names in the `.AddPlugin()` dependency parameter.
    - Example: `.AddPlugin("Sales.Services", new[] { "Sales.DbContext" })` — each string is a simple module name (not a file path or DLL filename).

This was handled well by the advanced models, but not always by the cheaper ones. Still, this is part of setting up new projects, which wasn’t the focus of my experiment. And it’s easy enough to do manually.

6) Build for Dev/Debug

- Some plugin assemblies are not referenced by the host or by other projects. These assemblies are loaded dynamically at runtime and must be built for Dev/Debug.
- Ensure those assemblies are included in the Visual Studio build by adding them as build dependencies of the host or plugin root project using the __Project Build Dependencies__ feature in the solution.
    - Steps: right‑click the solution → choose __Project Build Dependencies__ → select the dependent projects (for example, add plugin projects as dependencies of `UI/ConsoleUi` or the plugin root).
    - The selection is saved in the `.sln` file and is not part of individual project files.
- If a plugin has additional assemblies that are not directly referenced, add those dependent projects as build dependencies of the plugin root project as well.
    - Example: `Sales.DbContext` is a dependency of the `Sales.Services` plugin; add `Sales.DbContext` as a build dependency of the `Sales.Services` project so both are built in Debug.

This was not done by Copilot. For some reason it could not edit the Solution File (sln) to setup Build Project Dependencies, nor to add new projects.

Again, not a problem as this is more into setting up the structure, which was not the target of Copilot in my experiment anyhow.

Copilot did not handle this. It couldn’t edit the solution file to set up project build dependencies, nor could it add new projects.

Again, not a real issue, since this is about setting up the structure, which wasn’t the focus of the experiment anyway.

7) Data & Persistence

- `Infra/DataAcces` abstractions only, like `IRepository` or `IUnitOfWork`. Do not use directly EF Core. Do not take hard dependencies to EF Core. 
- Use `IRepository` for read only cases; Get the `IRepository` via DI.
- Use `IUnitOfWork` for transactional operations; Get the `IUnitOfWork` via a factory function (`IRepository.CreateUnitOfWork`).

In general the Copilot didn’t do the best work in using the DataAccess. Simple, but impactful fixes were needed.
However, these instruction helped the more advanced models to get things right more often.
Maybe, I would refine this section more, and add some code examples, things will be better. Those samples are part of the course, so adding them here might help.

In general, Copilot didn’t do a great job with the DataAccess layer. It needed simple but important fixes.

The instructions did help the more advanced models get it right more often.

I might refine this section further and include a few code examples. Those samples are already part of the course, so adding them here would be easy and helpful.

8) Console UI

- Host project is `UI/ConsoleUi/`.
- Each module has its own subfolder under `Modules/` for console commands (e.g., `Modules/Sales/Console/`).
- The modules do not directly depend `UI/ConsoleUi/`; instead, commands implement interfaces defined in `Modules/Contracts/Console/`.

This helped a lot. All features accessed through the console (we’re building a CLI) were placed correctly in the corresponding Console project, which only depends on the Contracts.

The structure was followed consistently.

9) Files Copilot Must Not Modify

- Any file under `Infra/**`
- Any file under `*/DbContext`
- Any `*.csproj` file

> **Copilot:** If a change is requested in these paths, reply with an alternative that keeps generated/third-party code intact (e.g., partial class, extension method, adapter).

The advanced models took this very seriously. Claude refused to touch these files and explicitly pointed back to rule nine as the reason. It generated the code in the chat window instead and told me to apply it manually.

I appreciated that behavior.

The Labs

You can find the lab instructions in the repo at this path: .Practice/readme.md

The instructions are structured and clear. They’re written for developers attending the workshop, and they work well for an AI agent too.

My approach was simple. I prompted Copilot with the description of each lab, one at a time. At the second prompt, I added more details and clarified what needed to be done. Then I asked for a detailed plan. Only after that did I ask it to implement the code.

Afterwards, I reviewed the generated code and applied fixes where they were quick. In most cases, the fixes were straightforward.

Lab 1 - Notify IsAlive for the Sales Module

The first lab is straightforward. It asks you to replicate an existing mechanism in a new module.

Besides the written instructions, I also provided Copilot with the existing implementation to give it more context. I treated this as a small upfront investment to help it understand the pattern.

I used the cheap model for this one, and it handled it well.

Lab 2 - Refactor the Console Application

This lab required some restructuring. The goal was to use the infrastructure mechanisms to achieve low coupling between the UI and each feature implementation.

The cheap model struggled here. It didn’t respect the dependency rules and introduced incorrect references.

I had to refine the prompt several times and revert some of its changes. It didn’t feel like an efficient workflow.

When I switched to the advanced model, things improved a lot.

Copilot was very useful for building a clean CLI. This part involved some verbosity and user interaction code, and the model handled it well.

Lab 3 - Create a Composite Console Application

This lab focused on using the Composite pattern support built into the infrastructure.

The goal was to discover all implementations of the IConsoleCommand interface across all modules and build a CLI from them.

Copilot made this lab very efficient.

I only needed to break the lab into smaller steps, validate each step, and guide it in the right direction.

The only downside was the speed. It often took between thirty and seventy seconds to complete a task, which broke my workflow and focus.

One thing I started to appreciate about Agent Mode is that it builds the solution after each step. If the build fails, it tries to fix the issue on its own. In most cases, it solved it after one or two attempts. A solid plus for autonomy.

Still, the result was worth it. It generated a nicer CLI than the one I would have had the patience to write by hand.

Lab 4 - DataAccess in Sales Module

This lab focused on using the DataAccess component from the infrastructure.

Copilot handled the read-only part well, but it made several mistakes when adding or changing data. The code worked, but it was far from optimal. There were unnecessary round trips to the database and incorrect use of the IRepository and IUnitOfWork abstractions.

The fixes were easy, but they required a careful review.

Labs 5 and 6 - DataAccess Interceptors

These labs focused on using the interceptors provided by the DataAccess component.

Copilot had a good grasp of the concept and the structure. It created the right files, placed them in the correct locations, and set up the dependencies properly.

It was also very helpful with the repetitive work of making all the DTOs implement the IAuditable interface.

Overall, a great help.

Labs 7 and 8 - Adding new Modules

These last two labs are meant to test and recap the understanding of the structure created by the App Infra.

It asks to add new modules, which do DataAccess and have UI. Plus, it makes you use services from one module into another to prove the loose coupling between modules.

Again the Copilot was of a great help. I was just using the advance model and by now I also got a good sense on how to build the prompts more efficient. 

I got into a workflow that I was reviewing files while others were generated. This helped not only with the speed, but aslo with not getting out of the flow.

It proved that it had a good understanding of the structure and the low coupling principles and of the dependency rules.

I appreciated summaries like:

Verify Solution Structure

After adding, your solution structure should look like:

AppInfraDemo
├── UI
│   └── ConsoleUi
├── Modules
│   ├── Contracts
│   ├── Sales
│   │   ├── Sales.DataModel
│   │   ├── Sales.DbContext
│   │   ├── Sales.Services
│   │   └── Sales.ConsoleCommands
│   ├── Notifications
│   │   └── Notifications.Services
│   ├── Export
│   │   ├── Export.DataModel
│   │   └── Export.Services
│   └── ProductsManagement          ← NEW
│       ├── Products.DataModel      ← NEW
│       ├── Products.DbContext      ← NEW
│       ├── ProductsManagement.Services          ← NEW
│       └── ProductsManagement.ConsoleCommands   ← NEW
└── Infra
    ├── AppBoot
    ├── AppBoot.UnitTests
    └── DataAccess

Or

Architecture Highlights:
Cross-Module Communication:

```
Sales.Services
  ↓ (depends on interface)
IPersonService (in Contracts)
  ↑ (implemented by)
PersonsManagement.Services
```

These show that it has a good understanding of the structure and gives confidence that the changes will be right.

Conclusion

This experiment reinforced a key principle in Code Design: an AI Agent becomes effective when it operates inside a clear structure. When the architecture has well-defined boundaries, stable abstractions, and predictable patterns, Copilot can generate code that aligns with the design instead of fighting it. The Application Infrastructure provides the guardrails, and the AI simply follows them.

Strong conventions reduce ambiguity. Clean Architecture reduces decision noise. Together they create an environment where AI coding becomes reliable rather than accidental. The agent does not need to “understand” the whole system — it only needs to work within the rules. And when those rules are sharp and consistent, both the developer and the AI move faster with far fewer mistakes.

In short, structure amplifies the strengths of AI. A disciplined foundation makes Copilot more accurate, more predictable, and more useful. And that combination points toward a future where Agentic AI and good Software Design reinforce each other, rather than compete.

Code Design Example from a Real ETRM Platform

When building large-scale distributed systems, integration with third-party platforms is often one of the most sensitive and critical aspects. These external dependencies can easily become points of failure or sources of complexity if not handled properly. In this article, we explore a Code Design approach for implementing an Integration Gateway that helps isolate and standardize external communication.

The Role of the Integration Gateway

The Integration Gateway is a dedicated layer of services responsible for managing all interactions with systems outside our control. These may include TSOs (Transmission System Operators), Power Exchanges, asset control platforms, or data providers.

These integrations often share common technical requirements—security, monitoring, reliability, fault tolerance, diagnostics, and performance. At the same time, each external API has its own peculiarities. The Integration Gateway is designed to bridge that gap: it speaks the language of each external system while translating everything to a common contract understood by our core platform.

This approach protects the internal system from changes, errors, and unexpected behaviors coming from external parties.

Real-World Context: Technical Design for an ETRM System

As part of a greenfield development project in the energy sector, I led the software design and implementation of a platform that enables grid balancing entities to:

• Trade ancillary services with TSOs
• Buy and sell energy on Power Exchanges
• Monitor and control production or consumption assets

The system had to be multi-tenant, support multiple countries and energy markets, and run either as an on-premise solution or in the cloud using a SaaS model.

The Integration Gateway became a key part of this software design, as we needed to connect to many different external platforms while enabling customers to license only the specific integrations they required.

Technical Design Approach

To ensure maintainability and predictability, we applied a mix of Code Design patterns and supporting infrastructure:

• Contract-First Approach
• Pluggable Application Structure
• Runtime Type Discovery
• Clean Architecture Principles

These decisions shaped both the detailed design and the structure of our codebase, offering strong separation of concerns and promoting reusability.

Contract-First Design

We defined interfaces at two different abstraction levels:

1. Internal Contracts – describing what our system expects from the external integration (e.g., data or behavior needed from a TSO)
2. External Contracts – defining what each adapter needs to fulfill when connecting to a specific external API

All contracts were isolated in "Contract Assemblies"—assemblies that include only interfaces, DTOs, and exceptions, without any logic.

This separation ensured loose coupling and clarity at the design level.

Pluggable Applications

Each external system (e.g., each TSO or Power Exchange) had a dedicated adapter packaged as a separate assembly. These acted as plugins.

The adapters translated between the contracts we defined and the actual APIs provided by the external parties.

At runtime, different versions of the same integration service (e.g., for TSOs) were deployed, each with its own plugin based on customer-specific configuration.

This allowed us to maintain a single codebase and dev team per integration type, while deploying multiple services for scalability, fault isolation, and operational independence.

Runtime Type Discovery

We used a type discovery mechanism to detect and register all plugin implementations at application startup.

When a service instance launched, it scanned all loaded assemblies and registered relevant implementations into the Dependency Injection container based on naming conventions.

This enabled us to customize deployments and behaviors without needing to recompile the application.

A lightweight implementation of this pattern can be found in our GitHub account part of the AppInfra-Training repository.

Clean Architecture

A core idea behind Clean Architecture is that business logic should be independent of frameworks or external SDKs.

In this Integration Gateway, the core logic did not reference any third-party libraries. Only the plugin assemblies were allowed to depend on SDKs provided by TSOs or Power Exchanges.

This separation made the codebase easier to test, maintain, and evolve.

Summary

In distributed platforms that rely on third-party services, the Integration Gateway acts as a protective boundary. It translates diverse external protocols into a common language, making the system resilient and maintainable.

By applying Contract-First principles, pluggable structures, runtime discovery, and Clean Architecture, you can build integration layers that scale, adapt, and hold up over time. This kind of technical design is essential when building modern platforms that must remain robust in the face of constant external change.

This example showcases how disciplined software design and clear separation of concerns can support both flexibility and control in real-world systems.

A Structured View for Project Planning and Replanning

When the project plan is designed using such a network, and combined with realistic constraints, it forms a foundation that stakeholders can trust. This approach replaces intuition with a strategy grounded in Code Design principles—building confidence, improving predictability, and making it easier to manage complexity.

Even as execution progresses and changes occur, the network perspective enables proactive replanning, better alignment across teams, and clearer communication of impact on scope, budget, and delivery.

Simply put: analyzing the Activities Network is a practical tool for planning and managing complex software systems.

Why Use an Activities Network Diagram?

• Clarifies Dependencies: Shows the real order in which activities must be done.
• Improves Project Management: Plans are based on calculation, not guesswork.
• Supports Change Control: Makes it easier to understand how new requirements or risks affect progress.
• Improves Communication: Helps explain structure, scheduling, cost, and trade-offs to both technical and non-technical stakeholders.

Building the Activities Network

The starting point is the System Design, which outlines all the building blocks of the solution and the dependencies between them, based on core use-case analysis. While valuable, this design view has some limits:

• It is structural, not execution-oriented.
• It is incomplete for planning.
• It can be too detailed to use directly for scheduling.

To transform it into an Activities Network Diagram:

1. List Activities for each building block.
2. Add Non-Development Work such as requirements, documentation, testing, training, or integration.
3. Define Dependencies between tasks (what must be finished before something else can start).
4. Draw the Diagram as a directional graph.

When represented as an Arrow Diagram—nodes as events and arrows as activities—the network becomes easier to follow and scalable to projects with 100+ activities.

Integration Points

In this network, integration events are key. Integrating too many streams at once creates risk. Two activities are enough for an integration; more than three often leads to instability.

Whenever integration looks complex, it’s worth revisiting the Project Design and refining how the system supports core use-cases.

Critical Path Analysis

The Critical Path represents the longest sequence of dependent activities, and it defines the absolute shortest time the project can be completed.

• The critical path duration is the minimum project timeline possible.
• No project can be finished earlier than its critical path allows.

To calculate it, activities are estimated in multiples of 5 days (5, 10, 15, 20, 25, 30). Anything longer should be broken down. This keeps estimation consistent and allows the law of large numbers to balance over- and under-estimates.

Critical Path Analysis is the only reliable way to answer: How long will it take to deliver the system?

Since any delay on the critical path delays the whole project, it must be monitored and recalculated regularly as execution evolves.

Staffing the Project

The Activities Network also helps answer a critical Project Management question:

What is the minimum staffing level that ensures the critical path moves forward without interruption?

This makes it possible to optimize cost, manage budget, and control risk while still ensuring quality and predictable delivery.

• Always assign the strongest developers to the critical path.
• Assign the next strongest to near-critical paths.
• Use the network to see the maximum parallelism achievable.

The analysis also helps with staffing distribution:

• Not every resource is needed at the same time.
• Hiring and offboarding monthly is inefficient and costly.

Instead, staffing should follow four phases: initial ramp-up, enabling activities, peak staffing, and ramp-down. Activities can be shifted within available float to keep this distribution balanced, without delaying the critical path.

Change Management and Replanning

As the project evolves, changes are inevitable. An Activities Network provides a structured way to measure their impact:

• Dependencies may shift.
• New activities may appear.
• The critical path may change.
• Staffing or scheduling may need to be adjusted.

This makes it possible to clearly quantify the effect of changes on budget, cost, quality, and delivery—and communicate those impacts with confidence.

Conclusion

Viewing projects through an Activities Network transforms Project Management from uncertain estimation into a predictable, Code Design-driven process. It improves project design, clarifies dependencies, and helps balance budget, cost, quality, and delivery—making it an essential tool for planning and executing complex software systems.

You can’t jump directly from architecture—which provides the overall System Design—into coding. Not if you want reasonable development costs, long-term maintainability, and predictable delivery.

System Design outlines the system’s services and building blocks, as well as how they interact. Detailed Technical Design, on the other hand, specifies how each of those elements should actually be built.

Why Detailed Technical Design Matters

Like any design discipline, its main role is to reduce costs by finding the right solution early. It is always cheaper to test a design (sometimes through a Proof of Concept or custom testing tools) than to fully implement a component only to later realize it doesn’t fit.

A common misconception is that design “emerges” from code. In reality, engineering follows the opposite direction: first design, then build. Skipping this step usually undermines predictability.

Establishing Technical Design as a Process

Technical Design should exist as a clear activity in the development process.

On the projects I lead, I always emphasize this stage. Usually an architect or senior developer owns the task, since it requires strong analytical skills and deep system knowledge.

Scrum often misses this step. I’ve seen estimates made on half-understood solutions, riddled with unknowns, or based on assumptions the team didn’t share. Grooming, backlog refinement, or planning meetings rarely leave enough space for the analytical thinking needed here.

When estimates fail, defects pile up, or rework becomes unavoidable, developers often get blamed. But the real cause is skipping a proper Technical Design phase. Running Scrum ceremonies on top of well-defined Technical Designs leads to better quality, higher efficiency, and improved predictability.

What a Detailed Technical Design Should Cover

A solid Detailed Technical Design usually answers three key questions:

1. What are the requirements of the service/component?
2. What contracts define how others interact with it?
3. How will those contracts be implemented? (including internal code design, structure, and technologies).

1. Service Requirements

System Design defines services and their interactions to support requirements like scalability, reliability, or security. Each service then has its own responsibilities to meet.

These need to be written down and validated against the system’s use cases.

Requirements are rarely tied directly to a feature.
Example: Just as a jet’s fuel pump has nothing to do with serving meals, an IdentityProviderService has nothing to do with placing an order. Its requirement is clear: provide a security token from username and password.

2. Service Contracts

Strong contracts are the foundation of modular systems. They should be:

• Cohesive
• Logically consistent
• Independent
• Reusable

The level of detail depends on the project and the team’s experience—from draft interfaces with placeholder functions to fully defined APIs with detailed parameters.

3. Implementation

The Technical Design should define:

• How contracts are implemented
• Which frameworks and technologies will be used
• How those choices meet the service requirements

Even for something like a web client—where frameworks set many conventions—the design should specify best practices, exceptions, and customizations.

In larger systems with many services, keeping a consistent structure is crucial. Components should look and feel like part of the same system, which is only possible through consistent design practices.

The design should also capture:

• Key technical decisions
• Alternatives considered
• The reasoning behind choices

An Iterative Activity

Design is not linear. Each step gets revisited, refined, and validated against core use cases. Mistakes are expected—they’re much cheaper to fix in design than in code.

Peer review sessions are often invaluable, as a fresh set of eyes helps challenge assumptions and validate ideas.

Deliverables of Technical Design

The output of a Technical Design can include:

• A design document
• Diagrams and interface definitions (sometimes written in code)
• Proofs of Concept to validate design choices
• Small demos (vertical slices) showing how the design applies in practice

Case Study: Messaging Component in an Energy Project

In one energy system I worked on, the architecture defined a Messaging Component that had to support both Pub/Sub messaging and Fire-and-Forget commands.

Requirements:

• Message durability, reliability, and deduplication
• Ownership and security
• High performance under load
• Publishing transactions
• Monitoring and diagnostics

Contracts:

We defined the main concepts first (Message, Event, Task, Endpoint, Handler, MessageOwner, etc.). After several iterations, these were refined into clear interfaces in C#.

Implementation:

We compared Kafka and RabbitMQ. Through PoCs, RabbitMQ proved to be the better fit.

The design also established the code structure:

• Assemblies with clear roles and dependencies
• Rules for referencing across services
• How and where messages owned by other services would be written

Finally, we created demos to show how events and tasks would be published and handled.

Consistency Across Services

The same project had over a dozen services. Two examples—DataManagement and LongRunningFlows—were very different in purpose, yet shared the same coding structure and dependency principles. This consistency was achieved through detailed code design and technical guidelines.

The Value of Detailed Technical Design

The effort required for Detailed Technical Design is modest compared to implementation.

Designing a service or component usually takes two to five days. Sometimes, more time is needed for PoCs and technology evaluations, which should be handled separately.

By formalizing the design before writing code, you save time, reduce risks, and build systems that are easier to maintain.

Over time, I’ve refined a set of working practices that help ensure stable execution, technical clarity, and predictability in delivery. Especially in recent years, my approach has evolved further, shaped by both advanced training and the lessons learned from real project work.

Here’s what I rely on:

1. Avoid Splitting by Functionality

One of the most impactful changes I made was to stop decomposing systems by business functions. Instead, I identify areas that are likely to evolve independently—grouping logic around those axes of change. This method leads to a code design that localizes change and protects the overall structure from ripple effects, supporting long-term maintainability.

2. Maintain Separation of Concerns in Code

Even with a solid architectural model, it’s easy for the actual implementation to drift. I enforce separation of concerns by putting clear boundaries in the code design, especially when it comes to dependencies between modules. This keeps complexity in check and ensures that architectural intent is preserved over time.

3. Stay Out of Feature Development

It may sound counterintuitive, but I intentionally avoid building features. Developers should own the domain. As an architect, I stay focused on structure and delivery. Getting involved in individual features creates distractions and pulls attention toward short-term urgency at the cost of long-term system health.

4. Build a Custom Application Infrastructure

Creating infrastructure components tailored to the project’s needs is how I bridge design and implementation. These are abstracted building blocks that simplify the work of feature development teams, reduce duplication, and standardize behavior. This kind of foundational setup makes writing unit tests and adding new features much easier.

5. Shape the Development Process

Good software design alone isn’t enough. The process matters too. I actively work with teams to define how design, development, and validation happen. It’s important to create space for technical design to happen independently, and to structure work in a way that aligns with the system’s complexity and business priorities.

6. Use an Activities Network, Not a Task List

Instead of linear backlogs, I plan around a network of activities and dependencies. This approach improves visibility for all roles—developers, managers, testers—and helps us manage change effectively. It’s a practical tool that supports predictable delivery in dynamic environments.

7. Actively Support Project Staffing

Staffing decisions directly affect technical outcomes. I collaborate with PMs to plan who joins the team, at what point, and with what level of experience. Matching skills to the structure of the system—rather than leaving it to chance—can significantly improve delivery quality and speed.

8. Communicate Progress Early and Often

Early in the project, there's little visible functionality. Still, I make sure to show progress—milestones in setting up tools, making architectural decisions, or validating approaches. Keeping stakeholders informed helps build trust and reduces unnecessary pressure on the team.

9. Delay Features Until the Foundation Is Ready

Most of the initial work in a serious project is foundational: CI/CD setup, infrastructure, and system design. That’s deliberate. Once enough groundwork is in place, I look for opportunities to start building features in parallel. Done right, this staged delivery helps manage risk without delaying visible outcomes.

10. Estimate Time and Cost with Structure

Reliable estimates require more than guesswork. I correlate the activities network with staffing plans to produce grounded forecasts. This helps manage expectations and leaves room to adapt. Predictability comes from this discipline—not just optimism.

Final Thoughts

These practices help me keep a clear focus on design quality, technical clarity, and the ability to deliver predictable results. Whether it’s through clean code design, structured estimation, or modular infrastructure that supports unit testing, these principles give me the tools to build systems that scale and last.

Even after applying it on real projects, I still find it hard to explain clearly—and even harder to convince others of its value.

At its heart, Project Design is an engineering approach to planning that aims to deliver on time and within budget. For teams unfamiliar with a truly engineering-led method of planning and running software projects, it often sounds too good to be true.

Many dismiss it outright, calling it “waterfall” without understanding what it actually is.

What Does Project Design Mean?

Just as you Design a software system, you also have to Design the project itself.

Project Design includes:

• Accurate Estimation of Duration and Cost
  Done like an engineer would: no guesswork or wishful thinking, but clear calculations based on realistic assumptions.
• Building Multiple Executable Options
  Each option considers staffing levels, team skills, milestones, and business goals. These options will differ in time, cost, and risk—giving leadership the ability to choose deliberately.
• Validating the Plan
  Can this team really deliver this plan? Would I be able to do it with these people? The plan must match the team’s real capabilities.

Who Is Responsible for Project Design?

This is the architect’s job—not the project manager’s.

The architect works closely with the PM, but takes the lead in making the plan. It's an engineering process with trade-offs, calculations, creative solutions, and constraints. Fundamentally, it’s about making informed decisions.

Project Design vs. Project Management

Project Design is to project management what System Design is to coding: it’s the blueprint.

Architecture is the design of the system; coding implements it. Similarly, Project Design is the plan for how the project will be executed, and project management is the actual execution of that plan.

How Does Project Design Work in Practice?

Project Design builds on System Design. The system architecture describes what needs to be built and the technical dependencies. From there:

• The architect and PM identify all activities and their dependencies.
• This is turned into an Activities Network Diagram—a visual representation of the plan.
• Analysis and calculations are done on this diagram to support better decisions.
• Staffing requirements are planned in detail.
• The result is a clear, actionable set of “assembly instructions” for the project.

The Lego Analogy

Think about a Lego set. The picture on the box shows the finished model. The bags of bricks give you the parts. But neither is enough.

What you really need is the assembly guide—the instructions telling you the sequence to put pieces together, what can be done in parallel, and how components fit.

Project Design delivers these assembly instructions for software projects: defining the order of work, identifying parallel tasks, integration points, and dependencies. Architecture alone isn’t enough.

Tools and Monitoring

A well-done Project Design equips project management with:

• A plan they can actually track.
• Tools to detect when execution is off-course.
• The ability, together with the architect, to take early corrective action and avoid big delays.

The Value of Project Design

Project Design takes practice and learning to master. But even partial use of this approach can greatly improve predictability in delivery.

Bringing Code Design discipline to project planning makes it far more likely to deliver successfully, on time, and on budget.

For more than 15 years, I’ve relied on this approach while leading the design and implementation of complex systems. It’s helped me build software that remains maintainable and predictable — even as teams evolve, systems scale, and requirements shift.

What Is Application Infrastructure?

Application Infrastructure refers to a set of technical building blocks that are specific to your application. These components define and enforce the structure of your code, making it easier to follow architectural guidelines and harder to go against them.

You might wonder: isn’t that what frameworks or foundations are for? They’re close, but not quite. Application Infrastructure is more tailored. It includes structure and guidance like a framework but also provides tooling — a toolkit — that helps reduce complexity and enables developers to move faster while doing the right thing.

Why It Matters

Even the best project design can fall apart in implementation. I’ve seen good architectures fail because the code wasn’t structured to support them. Whether due to time pressure, misunderstandings, or lack of training, the result is the same: a system that doesn’t meet its goals.

In one of my earlier projects, we followed a 4-tier architecture. The structure was sound, but developers started bypassing the service layer and accessing the database directly from the UI. It worked, feature-wise, but broke the core non-functional requirements. When we introduced Application Infrastructure, the rules became enforced through the structure itself — and code that violated them simply couldn’t be written anymore.

This isn’t limited to traditional architectures. The same principles apply to microservices, cloud-native apps, and modern web or desktop clients. The goal is the same: predictability in how code is written, maintained, and evolved.

How It Works

The heart of Application Infrastructure lies in enforcing rules for how code is organized and interacts. For example, in a .NET environment, I usually:

• Separate technical components into an Infrastructure layer, which contains no business logic
• Create Contract assemblies for interfaces and DTOs only — no logic here either
• Place use-case implementations under a Modules folder, organized by functional boundaries

Clear reference rules govern which parts can depend on others, and these rules are verified through static analysis and CI pipelines.

To keep Dependency Injection clean and automated, we use a Type Discovery mechanism — a lightweight library that handles registration while respecting the defined structure.

Real Benefits in Real Projects

Application Infrastructure delivers concrete benefits:

• Code consistency: Patterns emerge naturally, and similar problems are solved in similar ways.
• Team Volatility: New team members learn faster because the structure is self-explanatory.
• Modular ownership: Even in a monolith, teams can take ownership of individual modules, because there are no hard dependencies at build among them
• Reduced framework coupling: Wrapping external libraries behind internal APIs aligns with Clean Architecture.
• Lower complexity: Infrastructure handles the plumbing — messaging, security, communication — and exposes clean APIs tailored to your app.

Trade-offs and Considerations

This approach isn’t without effort. You need to build a good portion of the infrastructure early. With experience, though, you learn how to deliver just enough to support early features and evolve it along the way.

From a team perspective, you can either rotate developers into infrastructure work or dedicate a small team — especially when senior devs are limited. Either way, the return on investment is high.

Why It Matters for Learning and Training

If you’re doing software training, mentoring, or investing in team learning, Application Infrastructure becomes even more valuable. It reinforces the lessons you teach about design patterns, separation of concerns, and maintainable architecture — directly in the day-to-day coding process.

It’s one of the most effective ways I’ve found to bridge the gap between theory and practice in software development.

If you're looking for more predictability in your software delivery, or if you're teaching teams how to build systems that last, start with your Application Infrastructure. It’s one of the best tools we have for making design real — and keeping it that way.

Until recently, I served as the Solution Architect for a distributed system in the energy sector—a system critical to grid balancing and enabling energy trading. It was a greenfield project that we built from the ground up.

Developing and evolving such a complex system over several years, while keeping it in production, required us to uphold a high level of technical quality—especially in the face of team changes. This project allowed me to put into practice the experience I've accumulated for designing distributed systems with long-term development in mind.

In this session, I'll share the story of this project, highlighting design strategies that supported long-term development. Drawing on over 15 years of experience in high-pressure projects that demanded availability, reliability, and precision, I’ll provide practical insights into creating systems that will stand the test of time.

Resources

• Slides: Slide Share
• Recording: Codecamp 2024
• Referenced code snippets: Code Design Training on GitHub
• iQuarc.AppBoot: on GitHub
• iQuarc.DataAccess: on GitHub

Has implementing Clean Architecture become more of an ideal than a reality in your projects? Despite its clear rules and intended separations, the complexity of growing codebases and the crunch of time often render these principles invisible in practice. When projects technically fail, the reason is often uncontrolled complexity – where Clean Architecture remains a concept on paper, not in code.

In this session, I will show how to get predictability by implementing Clean Architecture through structure, rather than relying on discipline and code reviews only. I’ll show how to create a structure that makes it easy to write the code that follows your architecture and at the same time it makes it difficult to write the code that doesn’t.

You will walk away with a recipe and building blocks for creating a foundation in code that sustains Clean Architecture and the level of quality in your code that can control the complexity of the project.

This structure will enforce separation of concerns and how dependencies are created. It will deliver predictability by creating a Code Design that is maintainable, extensible and reusable.

Resources

• Slides: Slide Share
• Recording: Codecamp 2024
• Referenced code snippets: Code Design Training on GitHub
• iQuarc.AppBoot: on GitHub
• iQuarc.DataAccess: on GitHub

When projects fail for reasons that are primary technical, the reason is often uncontrolled complexity. The complexity goes out of hand when the code lacks structure, when it lacks Clean Architecture. In large software projects where many developers work on the same code base one of the biggest challenges is to get consistency in code, to create development patterns for common problems, so you can control the complexity and size of the system.

In this session I will show how we can achieve consistency through structure, rather than relying on discipline only. We will look at some basic building blocks of an application infrastructure which will enforce the way dependencies are created, how dependency injection is used or how separation of the data access concerns is enforced.

Resources

• Slides: on SlideShare
• Code snippets on the slides: Code Design Training on GitHub
• Code Design Training: description
• Implementing Clean Architecture Training: description
• iQuarc.AppBoot: on GitHub
• iQuarc.DataAccess: on GitHub

How it usually starts it's like this: the product owner comes and says that they need a project to do a task, be it simple or complex. The architect and the team think about the design, think about how data should be stored, processed and displayed. The project starts, latest and best tools and frameworks are used. This would be a great project. All the team ever hoped for was a new, greenfield project. Everyone is motivated. This time it would be different.

The first version is great. The UI looks very posh, it's latest fashion, the database has the perfect balance between the 3rd normal form and the performance requirements. You even asked the stakeholders about what are the nonfunctional requirements and did some automated tests around that. You may have some small performance problems, in some strange cases, but after you'll see how the application is used, you will find the best solution for that in v2.

People are seeing the application and they are charmed and want more out of it. This is where things get tricky. You built a product for one purpose. And having that and all the best practices in mind you made the design. Now, as the things evolve, the purpose changes. It happens almost every time, and numerous books where written in order to helping us, developers, protect our project from this. Robert C. Martin is just the most famous one to having built a writer and speaker carrier on this topic. The Open/Closed Principle is the best example of a good code design practice for minimizing the risk of breaking the project in v2+. But nobody tells you, the developer, how to approach the new requirements, how to try to understand the core need and model it in order to come with a solution that is the best compromise between the business and the technical sides.

Let's say for example you need to display all the orders created in a store. First, you'll have a table with columns for the number of the order, the date, the state, the value, and the possibility to see the details. This is great, clear and straight forward. Then the users realize that they not only want to filter the table, but want to see in separate tabs new, processing and closed orders. Because it would be easier to just have a tab right there, click on it. Of course, the product owner still wants to see everything aggregated, someone must have gotten used to that. After seeing them separately new functional ideas come to mind. The processing orders should have a new column with a new status, like "ordered from supplier/in transit from supplier/in stock/sent for delivery". These do not make sense for new or closed orders. So the "All" tab will be inconsistent in some way. Either you have the new column populated just for processing orders, or you take this column out, or you will populate it with closed or new for these two statuses. Either way, your model will not be 100% compatible with your view. And this is where a well written code should either signal a problem in the specs, and you, as a professional can either challenge as much as you can the requirements, or take it as it is, listen to the product owner, admit that it's just one new column, shouldn't be that much work, and end up, without realizing in a legacy project.

There is a trap in trying to align the magnitude of the business requirements with the magnitude of the changes needed in code. It's "just" a new column, it's "just" a new view like the old one with minor changes. What we as developers should do is detect those moments when the functionality evolves even a little bit away from the old one. In that moment we should either make a point that the purpose of the application is no longer the same, or that we need to make structural changes, probably even in the requirements.

We lack the concept of "functional refactoring". Wouldn't it be nice from time to time to analyse the capabilities of our application and reorganize or delete the legacy ones? An application that lived more than a few years and had grown during that time it's usually a monster, functionally speaking: users can do stuff nobody remembers it's possible, old functionalities are still there even if nobody uses them, users sometimes discover flows no one thought about and are using them day by day. It looks a little bit like Istanbul: it used to be magnificent, it's still magnificent here and there, but it's growth is out of control and it looks like an unpredictable amazing monster these days.

So that's the problem. What's the solution?

1. Listen to your code. If you think you must add a forced conditional, whether it's an if, of some kind of polymorphism, or dynamically composed screens, no matter how cool it is from the technical point of view, take a closer look. Is it a natural evolution of the code and functional behavior? If not you should challenge it.

2. Know your business domain. You should know what every stakeholder is gaining from the product. You should know how the users are behaving inside the app, how the roadmap for the product development is aligned to the current requirements, why and when are they needed. Try to find out all these things. Log flows in the app to see what the users are doing, ask around the organization if people are familiar with the app, ask the decision makers what they think about it, what it's missing, what do they think are it's strengths. Listen carefully to the boring organizational status meetings if products are presented. Look to your competitors. These all are sources of information that we developers are constantly ignoring because we think it would to be too much to fit in our heads. But by knowing these things, and paying attention to these details we can add real value because this is where you learn how the product should evolve. Everyone should be able to write code these days; internet is helping, IDEs are helping, there are a lot of good books and talks about good programming practices. What a great developer should be able to do is identify the real needs from his project and make them happen.

3. Always ask the stakeholders what is the basic need. When you are required to make some modifications, ask why if it's not obvious. Maybe the need can be satisfied better. Maybe, going back to our example, a new screen would be better than adding a new column. The user might need afterwards to be able to also update the processing orders in ways that would not apply to new and closed orders.

4. Challenge requirements. The idea of adding a new column on the tab with all orders is a bad one. Not uncommon, just bad. The users will have mixed information, will need to do classifications in their own mind in order to understand the presented information. You should ask the product owner why that tab is needed. If it's because aggregating all the information is found in every app, that is a bad answer. If it's because the user is accustomed to seeing this as a first screen, it's also bad. If it's because the aggregated data provides important insights, like how many orders are received in a day, then you have a better answer. But then the display of all information available for all order types it's useless. Users will, most probably, change tabs for the detailed and filtered information. In my experience, inconsistencies in the requirements, which of course are reflected in the code, are the most fragile and volatile parts of an application. If you sense that your design is unnaturally modified in order to implement a shaky solution, raise the problem. Try to find a better solution for the basic need.

5. Do not be afraid to provide out of the box solutions. Your most important advantage is that you see things from a different perspective. The PO can be stuck on one solution and the user can be stuck on one face of the problem. You, with the knowledge of the code, and without the pressure of being expected to come up with the perfect solution are in the best position to be able to provide a new perspective. A new way to approach the problem, a new way to solve it.

6. Do not over reuse UI code. I am not providing technical solutions here, in this article. On the contrary, I'm saying that for once we should trust our code when it screams that the functionality is bad, and help it stay clean by changing the requirements. That being said... The UI will change often, it's the most volatile part of the project. And I'm not talking about CSS or overall style, which of course should be uniform and shared. I'm taking about the two screens that seem almost identical in v1, and became totally different in v2. It's usually stated like this: "We have these two screens, which should look the same, but the second one will have some small additions, like being able to edit, delete, listen for updates, nothing much". Never, and I mean it, never reuse the UI code between these two because it will come and haunt you, and this is what you will be talking about in the project's lessons learnt meeting.

7. Do not forget that "Thinking is always a best practice". All our processes, all the measurements, all the paradigms, all the patterns, are made to made our work easier, i.e. to help us increase productivity without increasing brain effort. It's easy then to follow the good or bad practices blindly, because everybody does it. I dare you to stop for a moment and think if what you do every day makes sense to you, your project, your organization. If not, challenge it.

In my experience the worst code decisions come from the worst business decisions. In those moments when it just didn't feel right what was done in the code, a wrong feature was implemented. Almost always in these cases, after releasing, there was another stakeholder that was under the impression that a feature should look different, or behave different. And it is normal. For a good user experience a product should be very clear in its intentions. Tweaking functionalities in order to add abilities in unorthodox places destroys this clarity, and the first place where you can see that happening is in the code. This is why it is the responsibility of every developer to challenge their tasks when those don't seem right. Remember that at the end of the day, your work's purpose is to have an excellent product, and not to be able to say: I was able to do what I was told to.

Featured image from Let Go and LIVE

It turned out that upgrading to newer versions of Ghost wasn't as easy as I thought. I've installed Ghost in an Azure App Service from this Github repository by Felix Rieseberg, and I was relying on his Ghost-Updater-Azure tool for the updates. The update tool worked only once for me. That took me to Ghost v0.78. For the next versions it didn't work anymore. It was just doing nothing without any error message. Each time I tried to manually upgrade to a newer version it took more than a couple of hours, so I've quit each time.

Last year, Ghost announced a major release, version 1.0, and I was still on v0.78, so I had to migrate. I started with three migration goals in mind:

1. easy to do
2. have an easy path for future upgrades
3. host on Azure using PaaS

I started from the official Migrating to 1.0.0 guide, which made it clear that I will need a new fresh installation to which I should copy and import my theme and the content.

Ghost CLI

Given my previous experiences with upgrading Ghost, I was happy to see that they now provide this Ghost-CLI tool that is supposed to install, configure and upgrade Ghost through simple command line commands. Unfortunately, it doesn't work well on an Azure Web App based on Windows, so I didn't succeed in using it.

Ghost v1.x on Azure App Service Linux

Trying to get to a more reliable way of upgrading in the future, I insisted to use the Ghost-CLI and I tried to install it on an Azure App Service running Linux. I have found this tutorial which should help me do it: https://ourwayoflyf.com/ghost-v1-0-on-app-service-linux/.

This may be a good path for someone who is familiar with NodeJS, MySQL, Linux stack. That's not me :( I got stuck at the Knex Migration error that is also mentioned in the above blog.

After the deployment which I did with the provided template, I got Ghost v1.22, so a good opportunity to use the ghost update command (part of the Ghost-CLI tool) to try an update. It gave the error:

root@a8fbdc1b2630:/var/lib/ghost# ghost update
Checking for latest Ghost version
Downloading and updating Ghost to v1.22.0 > Installing dependencies > info
...
Running database migrations
An error occurred.
Message: 'Command failed: knex-migrator-migrate --init --mgpath /var/lib/ghost/current
[2018-04-02 07:36:00] ERROR

NAME: MigrationScript
MESSAGE: task.execute is not a function

level:normal

Error occurred while executing the following migration: 1-add-webhooks-table.js
MigrationScript: task.execute is not a function
    at MigrationScript.KnexMigrateError (/usr/local/lib/node_modules/ghost-cli/node_modules/knex-migrator/lib/errors.js:7:26)
    at new MigrationScript (/usr/local/lib/node_modules/ghost-cli/node_modules/knex-migrator/lib/errors.js:26:26)
    at /usr/local/lib/node_modules/ghost-cli/node_modules/knex-migrator/lib/index.js:353:19
...

TypeError: task.execute is not a function
    at /usr/local/lib/node_modules/ghost-cli/node_modules/knex-migrator/lib/index.js:308:25
    at tryCatcher (/usr/local/lib/node_modules/ghost-cli/node_modules/bluebird/js/release/util.js:16:23)
...

A database migration error. Might be a bug in the Ghost-CLI or in this Ghost version or simply something wrong with the Docker image I was using. Anyhow, it didn't seem like a good or a simple path for me to continue on, so I've dropped it.

Deploy from GitHub to Azure App Service

Not having success with the Ghost-CLI, I've decided to make a deployment from GitHub to Azure App Service, using the Ghost-Azure repository made by Radoslav Gatev, which configures Ghost for a deployment in an Azure App Service. This worked, and the entire migration steps that I'll detail next are part of this approach.

First, I have made a fork, so I can do the customizations needed for my blog.

The Upgrading Plan

For upgrading to next version of Ghost, my plan is to keep this fork up to date and Sync it with the Azure App.

This means to get the latest version of Ghost from the upstream Ghost-Azure repo. (Here is how to sync a fork). In the case that this repo is not updated I could get the latest Ghost version from its original repo. Then, I could make a pull request back to Ghost-Azure repo.

Once the latest Ghost version is merged in my fork, the next step would be to test that it works for my blog. Having the repo cloned locally, I can easily run the blog on my machine and do the checks. If I want I could also copy the latest content (DB and images) from the server, through FTP, to see that the DB gets migrated to latest version and that everything works as it should.

Then, if all good, I should push the changes to the onCode branch, which is the branch connected with my Azure Web App. The only thing left would be to go in the Azure portal and click Sync in the Deployment Options

This is the plan :) I'll let you know how it works when I'll do the first upgrade.

The Deployment

I did the initial deployment using the Azure deployment template from the repository, following Radoslav Gatev blog post.

It did create all the resources, but it didn't set the application settings, which I had to configure afterwards in the portal.

This wasn't a problem except for the WEBISTE_NODE_DEFAULT_VERSION. After the deployment, it was the only setting, and was 6.9.1. This setting says which node version the web app should use.

When I went into the App Service Editor, (which is very cool by the way) and executed in the console node db.js everything went well and it created the database. However, when I executed the npm rebuild it ended with some errors:

D:\home\site\wwwroot\node_modules\dtrace-provider>if not defined npm_config_node_gyp (node "D:\Program Files (x86)\npm\3.10.8\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "" rebuild ) 
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
D:\home\site\wwwroot\node_modules\dtrace-provider\build\DTraceProviderStub.vcxproj(20,3): error MSB4019: The imported project "D:\Microsoft.Cpp.Default.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
gyp ERR! build error 
gyp ERR! stack Error: `D:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (D:\Program Files (x86)\npm\3.10.8\node_modules\npm\node_modules\node-gyp\lib\build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
gyp ERR! System Windows_NT 10.0.14393
gyp ERR! command "D:\\Program Files (x86)\\nodejs\\6.9.1\\node.exe" "D:\\Program Files (x86)\\npm\\3.10.8\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd D:\home\site\wwwroot\node_modules\dtrace-provider
gyp ERR! node -v v6.9.1
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok 

Also, the web site wasn't starting... Ghost console output was Error: Cannot find module at npm install sqlite3

After some google-ing, I've realized that the problem was the node version, so I've set the WEBISTE_NODE_DEFAULT_VERSION to 8.9.0, reran the npm rebuild, and then everything was working. I had a fresh installation of Ghost in Azure App Service.

Migrate the Content from Ghost v0.78

Following the migration guide this should have been an easy two steps thing:

1. Export the content from the old site
2. Import the content in the new one

My problem was that the export from the old site didn't work. It didn't give any error in the Settings page, but when I ran it locally with verbose logging, I've seen the error in the console:

ERROR: Cannot read property 'client' of undefined 

Somewhere was a bug in Ghost. Either when it migrated the DB from a previous version to 0.78, either in the export functionality.

The trick I did to get my content was the following:

• install a new Ghost site with version 0.11, locally
• copy the /content/data/ghost.db from the old site to this v0.11 installation
• run the v0.11 site
  • when it started it migrated the database to its version
• export the content from the Settings\Labs screen from the v0.11 site

Add the Theme as a Git Submodule

The next step was to add the onCodeDesign theme to my new Ghost 1.19 installation. There are usually two ways to do this:

• upload the theme using the Settings\Design screen as shown in the migration guide
• copy the theme in git, in the content\themes folder and redeploy from GitHub

I didn't choose any of these. The reason is that I already keep my theme in a git repository on visualstudio.com in VSTS, and I don't want to loose all the history on how it was developed and modified. None of the above were a good option to have an easy deployment of future changes on the theme, and also to keep the history in git.

My solution was to add the theme as a git submodule to the GitHub repository of my blog. The submodule went into the content\themes\oncode folder, and it pointed to the same remote in VSTS. This, not only that preserved the history, but it also allowed me to keep the same repo for the theme which saved me some work.

The difficulty here was to setup the authentication to the theme repository on VSTS, which was not a public repo. When, Azure fetches the changes from the GitHub repo, it also needs to update the submodule, so it needs to have access to it. One solution would have been to use a SSH git URL to the submodule to the VSTS. This wasn't an option for my case because I use git-lfs (the theme repo has many images) and VSTS does not support git-lfs over SSH. Therefore, I had to configure a Personal Access Token for a new user in VSTS, and keep a HTTPS git URL.

The biggest advantage of this setup is that now, in my local clone of the repo I can do and test any changes to the theme. When I'm done, I just push the changes to the onCode branch, I go in the Azure portal and make a Sync and the changes are up.

Upgrading the Theme

The next step was to upgrade the theme to work with Ghost 1.x. This was a straight forward process. When I've activated the theme, it gave me some errors that I had to fix.

I've just followed the guide here and having the environment locally everything went fast and smooth.

Copy the Images

The next step was to copy the images from the old blog to the new one. This again, went smoothly. I've downloaded the images through FTP from the old site, and them I've uploaded them through FTP on the new one. The folders structure does not differ from v0.x to v1.x, so no issues here.

Also, most of my images are on Cloudinary, so I didn't have too many to copy from one place to the other.

Final Settings

The last steps were to review the settings on the new installation. Things like title & description, publication logo, publication icon (much nicer than on v0.x), publication cover etc.

To make the email settings you could put them in the config.production.json as shown here in the docs. However I prefer to use the App Service - Application settings, so I don't put my Sendgrid credentials on GitHub in a public repo.

Redirects

I do my redirect in the web.config as I've detailed in my post about migrating from Wordpress here.

In the previous installation I didn't had a nice way to deal with these. I set them up using the App Service Editor and that was it. Luckily they didn't need changing. Now, with my new setup of deploying from GitHub, I've just copy them in the new web.config which is part of my repo and just redeploy.

Ghost v1.x has support for uploading the redirects in a json file (Labs screen), but I just prefer the above to have them in the git history.

Switch the DNS and Setup the SSL

I run my site through Cloudflare for all the benefits in security (SSL, DDoS, etc) and performance.

This first thing was to go in Cloudflare and change the DNS setting to point to the new Azure App Service. Then, I've removed the domains from the old App Service and added them to the new one.

The oncodedesign.com got validated and was responding very fast. Almost instantaneously. I've added the SSL Binding using the Cloudflare certificate I already had setup in Azure and it was all good.

The www.oncodedesign.com didn't get validated even if it was changed in the DNS. I waited a day, thinking that I should be patient with the DNS propagation, but it still didn't work. After some reading I've switched off the HTTP proxy leaving DNS only for the www record

This did the trick. The domain was validated by Azure, and I could setup the SSL Bindind with the same certificate. Afterwards, I've switched it back to DNS and HTTP proxy and everything went well. Probably something was cached, and the new DNS record didn't push through.

Summary

To summaries:

• I've migrated from Ghost v0.78 to v1.19 wanting to keep hosting in Azure
• I wanted to get a easy way to future updates, so I looked into the Ghost CLI, but it doesn't play nice on Azure
• I ended up using an automated deployment from GitHub on Azure App Service
• I keep my theme development in its own git repo on VSTS, which is a git submodule in the main one
• I ended up with a nice environment that allows me to dev & test in my local clone and then easily push to Azure
• From now on the upgrades to the new versions should be easier given that I can keep my repo in sync with the one of Ghost or Ghost-Azure.

If you find yourself doing such a migration and you get stuck don't hesitate to ask me, I might have gone through the same.

This is the last post from the series on how we could implement a design that allows to decide only at deploy time how two services communicate: in process if they are deployed on the same server or inter-process if they are on different machines. The first post shows where such a design is useful and what are the benefits it brings. The second one, takes an example from the financial systems world, and outlines the key design ideas on how to implement it. In this post we'll only focus on code. We'll take the example presented in the previous post and code it from scratch in C#.

Dependency Injection is the base technique to implement all the key design ideas detailed in the previous post. In this demo, we'll use the iQuarc.AppBoot, which brings on top of a classic DI Container, the support for implementing simple conventions for the type discovery and the proxies. (More on iQuarc AppBoot Library here)

The entire source code that we are going to walk through is available on Github, here, as part of my Code Design training. Each implementation step is marked with a tag in the git repo. We have ipc-step0 for the starting point and so on, until ipc-step10b. You can see how the implementation evolves step by step.

Running the Demo

If we checkout the final step (ipc-step10b), we have what we wanted in the previous post: the ability to change the communication between the OrdersService and QuotationService to be in-process without changing code or recompile.

To get that running, after you build the TradingApp.sln you can find into the bin\.Deploy folder different deployments configurations (there are some post build events that copy the binaries in these folders). If you run the startAll.bat you will start three console processes, one for each of our three services from the example. They are all hosted by themselves, and each of them listens for REST calls on a different port (I've mentioned previously that we're implementing the example using REST services). Now, we can open Postman (or other similar tool) and fire some calls to them:

http://localhost:9002/api/Quotation/GetByExchange?exchange=NYSE&instrument&from=2017-01-01&to=2017-01-01

this gets from the QuotationService the quotations for the NYSE in a time window. You can notice that each console writes the calls it received and the remote that made it, so we can see what happens in our demo app.

If we make a POST to the OrdersService, to place an order:

http://localhost:9003/api/Orders/PlaceSellLimitOrder?securityCode=AAPL.S.NASDAQ&sellingPrice=11.45&validUntil=2017-01-01

we see the call logged into the console of the process that hosts the OrdersService (the remote is a Postman-Token outlined with yellow below, as for the previous call)

and further more, we can see another REST call made by the OrdersService to the QuotationService. We see it logged in the console of the process that loaded the Quotation Module (the one at the bottom below). We see that the remote is a Service-Proxy with the GUID of the proxies loaded by the process that hosts the Sales Module, as outlined with red below. This means that the one that made the call to the Quotation Module host was the Sales Module host.

So we had an inter-process communication, the SalesService making a REST call to the QuotationService, when someone else made a call to it.

Now, lets change the communication. We first close the process with the Sales Module. Then we copy in its output folder (bin\Sales\) the QuotationServices.dll. This is the binary that has the implementation of the QuotationService. Mainly we've changed the deployment configuration, by having now in the same output folder the binaries with implementations of the SalesService and of the QuotationService. Now, we just restart the host by executing the bin\Sales\ConsoleHost.exe. We see that it loaded both Quotation Module and Sales Module.

The other two console hosts remain running.

Now, if we repeat the same POST from Postman to the SalesService

http://localhost:9003/api/Orders/PlaceSellLimitOrder?securityCode=AAPL.S.NASDAQ&sellingPrice=11.45&validUntil=2017-01-01

we see that no other inter-process call is made. Noting gets logged in the other consoles. The SalesService did call the QuotationService since it depends on it, but the call was just a function call in the process that hosts them both.

So, only by the fact that we've changed the deployment by copying the QuotationsService.dll into the same \bin folder with the SalesService.dll and then restart the host process to load them both, we have switched from a inter-process communication to an in process communication.

We started with a deployment like below,

where each service was hosted in its own console host and an external call to the OrdersService triggered an inter-process communication with the QuotationServce, and we ended with a deployment like below

where OrdersService and QuotationService are hosted by the same process and an external call to the OrdersService can by satisfied with an in process communication with the QuotationService.

Writing the Code

If we checkout the ipc-step0 we have the solution structure to start with. At this step all the projects are empty.

We have a console project (ConsoleHost), which will be our generic host. Then we have class library projects where we'll implement the services, and another one for the contracts.

The solution structure is a key element of our design and I have detailed in the previous post here, the basic rules we want to enforce with it.

Basic Code for Contracts and Services

Next, if we checkout ipc-step1 we have the contracts of the QuotationService, an interface for the service contract and a DTO for the data contract:

public interface IQuotationService
{
    Quotation[] GetQuotations(string exchange, string instrument, DateTime from, DateTime to);
    Quotation[] GetQuotations(string securityCode, DateTime from, DateTime to);
}

public class Quotation
{
    public DateTime Timestamp { get; set; }
    public decimal BidPrice { get; set; }
    public decimal AskPrice { get; set; }
    public string SecurityCode { get; set; }
}

Next, we write a simple implementation for it, as a class in the Quotation.Services project (checkout ipc-step2):

class QuotationService : IQuotationService
{
    private readonly Quotation[] array = 
        {
            new Quotation {AskPrice = 10.50m, BidPrice = 10.55m, SecurityCode = "ING.S.NYSE"},
            new Quotation {AskPrice = 12.50m, BidPrice = 12.55m, SecurityCode = "ING.B.NYSE"},
            ...
        };
    
    public Quotation[] GetQuotations(string exchange, string instrument, DateTime @from, DateTime to)
    {
        var result = array.Where(q => q.SecurityCode.Contains(exchange));

        if (!string.IsNullOrWhiteSpace(instrument))
            result = result.Where(q => q.SecurityCode.Contains(instrument));

        return result.ToArray();
    }

    public Quotation[] GetQuotations(string securityCode, DateTime @from, DateTime to)
    {
        return array.Where(q => q.SecurityCode == securityCode).ToArray();
    }
}

Similar, we add the contract and a simple implementation for the OrderingService in the Sales.Services project (checkout ipc-step3). Notice here the dependency to the IQuotationService:

public class OrdersService : IOrdersService
{
    private readonly IQuotationService quotationService;
    public OrdersService(IQuotationService quotationService)
    {
        this.quotationService = quotationService;
    }

    public void PlaceSellLimitOrder(string securityCode, decimal sellingPrice, DateTime validUntil)
    {
        var todayQuotations = quotationService.GetQuotations(securityCode, DateTime.Today.AddDays(-1), DateTime.Today);
        foreach (var quotation in todayQuotations)
        {
            if (quotation.AskPrice >= sellingPrice)
                limitOrders.Add(new LimitOrder
                {
                    SecurityCode = securityCode,
                    PlacedAt = DateTime.UtcNow,
                    Type = OrderType.Sell,
                    Price = sellingPrice,
                    ValidUntil = validUntil
                });
        }
    }
...
}

Similar at ipc-step4 we have the PortfolioService implementation in the Portfolio.Services project. It also has a dependency to the IQuotationService

As pointed in the previous post here it is a critical requirement for this design that the dependencies among the services are towards interfaces. So, the OrdersService class depends on the IQuotationService interface and not on the QuotationService class. The same is for the PortfolioService.

Add iQuarc.AppBoot

At this point we have service contracts and implementations for them but they are not linked in any way. If we look at the Project Dependencies Diagram we see that we have no references nor strong dependencies among them.

In the previous post here, we've detailed why this is important.

The iQuarc.AppBoot will bring things together. We'll use it for type discovery to find the service implementations deployed, and then to satisfy the dependencies by configuring the underlying DI Container.

So, the next step is to install and configure it. We can install it as a NuGet Package to all the projects under the \Modules folder.

PM> Install-Package iQuarc.AppBoot -Version 2.0.3-disposables002

The iQuarc.AppBoot package doesn't have dependencies to any DI Container, and it is not a container by itself. It uses a container by an abstraction. It is good to also keep our modules (the place where we implement the business logic of our system) independent of a specific DI framework.

However, the host project needs to use a specific container, so for it only, we will install the iQuarc.AppBoot.Unity, which adapts the Unity Dependency Injection Container to AppBoot.

PM> Install-Package iQuarc.AppBoot.Unity -Version 2.0.3

The ConsoleHost is our generic server process. It should find all the services deployed, then publish them to be called from other processes and configure the dependencies. In a real system this would be an web app in IIS or a Cloud Service, or a Windows Service etc. For this demo it is a simple console app.

To use AppBoot to find the deployed services, first we need to decorate all the service implementations with the ServiceAttribute.

[Service(typeof(IQuotationService))]
class QuotationService : IQuotationService
{
...
}

With this we declare that the QuotationService class is the implementation of the IQuotationService interface. The same goes for the other service we have implemented in the \Modules folder.

[Service(typeof(IPortfolioService))]
public class PortfolioService : PortfolioService
{
...
}

[Service(typeof(IOrdersService))]
public class OrderingService : IOrdersService
{
...
}

Now, as a first step, when the ConsoleHost starts, in its main() it will call AppBoot.Bootstrapper.Run() which will scan through reflection all the binaries in the output folder and look after the types decorated with the ServiceAttribute. The ones that are found are registered in the DI Container as pairs of interface - implementation. This configuration is in the helper class AppBootBootstrapper:

public static class AppBootBootstrapper
{
    public static Bootstrapper Run()
    {
        var assemblies = GetApplicationAssemblies().ToArray();  //returns all the assemblies that should be scanned (filters out Microsoft.*, System.* and others)
        Bootstrapper bootstrapper = new Bootstrapper(assemblies);
        bootstrapper.ConfigureWithUnity(); // says to AppBoot to use Unity Container

        // configures the registration convention, by saying to use the ServiceAttribute convention
        bootstrapper.AddRegistrationBehavior(new ServiceRegistrationBehavior()); 

        bootstrapper.Run();
        return bootstrapper;
    }
...
}

To get this running we need to make sure that at build all the binaries are put in the same folder with the ConsoleHost. This helps in debug, and is the place where we can come later to define different deployment configurations. The easiest is to set the build output path for all the projects. Here is how it should be for a *.Services project:

Output path =  ..\..\..\bin\Debug\

So at the root level, we'll have a \bin folder where all the projects get build.

Self-Host ASP.NET Web API

For our demo purpose this is not that important. We just need to make the services available for REST calls, so a simple self host in the console app of the Web Api will do.

static void Main(string[] args)
{
    string baseAddress = "http://localhost:9000/";
    using (WebApp.Start<Startup>(url: baseAddress))
    {
        Console.WriteLine($"Server runs at: {baseAddress}");
        Console.WriteLine("Press ESC to stop the server\n");

        ConsoleKeyInfo keyInfo;
        do
        {
            keyInfo = Console.ReadKey();
        }
        while (keyInfo.Key != ConsoleKey.Escape);
    }
}

public class Startup
{
    // This code configures Web API. The Startup class is specified as a type
    // parameter in the WebApp.Start method.
    public void Configuration(IAppBuilder appBuilder)
    {
        // Configure Web API for self-host. 
        HttpConfiguration config = new HttpConfiguration();
        config.Routes.MapHttpRoute(name: "DefaultApi", routeTemplate: "api/{controller}/{action}");

        // kicks the AppBoot Bootstrapper
        AppBootBootstrapper.Run().ConfigureWebApi(config);

        appBuilder.UseWebApi(config);
    }
}

We see that at startup we kick the AppBootBootstrapper.Run() and then we call the ConfigureWebApi() helper. This configures the WebApi to use the AppBoot DI Container to inject the dependencies into the controller. Our controllers are trivial:

public class OrdersController : ApiController
{
    private readonly IOrdersService ordersService;
    public OrdersController(IOrdersService ordersService)
    {
        this.ordersService = ordersService;
    }

    public IHttpActionResult PlaceSellLimitOrder(string securityCode, decimal sellingPrice, DateTime validUntil)
    {
        ordersService.PlaceSellLimitOrder(securityCode, sellingPrice, validUntil);
        return Ok();
    }
...

They get through constructor DI the service they should make available through REST and they just forward the calls to it.

To make our ConsoleHost truly generic we should make a generic ApiController which gets the service through DI and forwards the call to it, and then we should customize the WebApi to use through some conventions our generic controller. Definitely it could be done, but it doesn't worth for our demo, so we just create three dummy controllers for each of our services.

A Fat Server

Now if we checkout ipc-step7b we have all of the above implemented. If we execute the ConsoleHost.exe we see that it loaded all the three modules, because they were all copied in this \bin folder. If we make the same POST call from Postman to the OrdersService it works and it will do an in process communication with the QuotationService.

Everything is hosted in one process, like a fat server, which does not scale.

If we delete the Quotations.Services.dll and run again the ConsoleHost.exe we see that it only loaded the two remaining modules: Portfolio and Sales. Now if we make the same POST call from Postman it returns a 500 Internal Server Error because the OrdersService did not find an implementation for its dependency to the IQuotationService.

http://localhost:9003/api/Orders/PlaceSellLimitOrder?securityCode=AAPL.S.NASDAQ&sellingPrice=11.45&validUntil=2017-01-01

Implement the Proxies

What we wanted above, is that when a dependency is not found in the same process to make a REST call to another process where that dependency is hosted. To make this happen, we need to create other implementations to the interfaces describing our contracts, which know how to forward the call over REST. These are the Proxies.

The proxies are part of the infrastructure. They are just plumbing code, no business logic, and we want them deployed on all our ConsoleHost instances to be the fail over when a real service implementation was not deployed. Therefore, we put them in the \Infrastructure folder.

Their code is not that important for our demo

class QuotationServiceProxy : IQuotationService
{
    public Quotation[] GetQuotations(string exchange, string instrument, DateTime @from, DateTime to)
    {
        using (HttpClient client = HttpHelpers.CreateNewClient<IQuotationService>())
        {
            string path = HttpHelpers.GetServicePath<IQuotationService>("GetByExchange");
            string uri = $"{path}?exchange={exchange}&instrument={instrument}&from={from}&to={to}";
            HttpResponseMessage response = client.GetAsync(uri).Result;
            if (response.IsSuccessStatusCode)
            {
                Quotation[] value = response.Content.ReadAsAsync<Quotation[]>().Result;
                return value;
            }

            throw new HttpException((int)response.StatusCode, response.Content.ReadAsStringAsync().Result);
        }
    }
...
}

They use a HttpClient, they build the URL to the service they want to call, based on some conventions, and they make the REST call. The response is then returned to the caller. We can checkout icp-step8b to have the proxies for all our services.

Implement the ServiceProxyAttribute

Now, we have two implementations for each of our contracts: the proxy and the real service implementation. The proxies are not yet used by the AppBoot. If we were to also decorate them with the ServiceAttribute as we did for the service implementations, the AppBoot will register both implementations for one interface and the last one found will overwrite the first. This is not deterministic, so it is not good. When a service is not deployed, as we've seen above, sometimes things will work, sometimes not, depending on which type was found last by the AppBoot while it scanned the deployed assemblies.

To fix this, we need to extend the conventions AppBoot uses. The ServiceAttribute we've used so far, was just a convention that says to register to DI interface - implementation pairs decorated with this attribute. We can create a new attribute say ServiceProxyAttribute and register a new convention to AppBoot.

[AttributeUsage(AttributeTargets.Class)]
public sealed class ServiceProxyAttribute : Attribute
{
    public ServiceProxyAttribute(Type exportType)
    {
        ExportType = exportType;
    }
    public Type ExportType { get; private set; }
}

public sealed class ServiceProxyRegistrationBehavior : IRegistrationBehavior
{
    public IEnumerable<ServiceInfo> GetServicesFrom(Type type)
    {
        IEnumerable<ServiceProxyAttribute> attributes = type.GetAttributes<ServiceProxyAttribute>(false);
        return attributes.Select(a => new ServiceInfo(a.ExportType, type, string.Empty, Lifetime.AlwaysNew));
    }
}

The convention is implemented as a IRegistrationBehavior, where we just return what should be registered to DI from a type that was found during the assemblies scanning. Checkout ipc-step9a to get these implemented and have all the proxies decorated with the ServiceProxyAttribute.

To make AppBoot use the new convention we go into the AppBootBootstrapper class and in the Run() function we add to the bootstrapper an instance of the ServiceProxyRegistrationBehavior

public static Bootstrapper Run()
{
    var assemblies = GetApplicationAssemblies().ToArray();
    Bootstrapper bootstrapper = new Bootstrapper(assemblies);
    bootstrapper.ConfigureWithUnity();
    bootstrapper.AddRegistrationBehavior(new ServiceProxyRegistrationBehavior());
    bootstrapper.AddRegistrationBehavior(new ServiceRegistrationBehavior());

    bootstrapper.Run();
    return bootstrapper;
}

The order we add the registration behaviors to the bootstrapper matters. The latter overwrites the registrations returned by the previous. So we add first the ServiceProxyRegistrationBehavior which puts into the DI Container proxies as implementations for all the contracts. If we'd leave it like this, all the service calls will be REST calls to other processes through the proxies. The ServiceRegistrationBehavior is the second, so it overwrites the proxies registrations with the real implementation for the services that were deployed. This means, that if the real implementation of a service was deployed, thus found at the assemblies scan, it will overwrite the proxy implementation, therefore the call to that service will be direct to real implementation in the current process. If it is not found, the proxy registration is not overwritten and the call will be done in another process.

With these done (checkout ipc-step10), we have the design completely implemented. Depending on what *.Services.dll are copied at deploy time into the \bin of a ConsoleHost.exe we have in processes communication or inter-process communication. Checkout ipc-step10a to get all the configurations tweaked for an easy run, and then you can re-run the demo as we did in the beginning of the post. You can play with different deployment configurations by copying the *.Services.dll in different ConsoleHost.exe instances. Here's another deployment configuration

This design is part of my Code Design Training where you will learn how to implement it in your context and to maximize the benefits for your requirements

Featured image credit: DENYS Rudyi via 123RF Stock Photo

This post continues the previous by giving an example on how we could implement a design that allows to decide only at deploy time how services communicate: in process if they are deployed on the same server or inter-process if they are on different machines. If you haven't read the previous post, you should go through it to get the context and where such a design is useful. Now, we are going to implement it for a simple example.

As stated previously there are three key design ideas we're going to put into practice:

1. Depend only on Contracts, which are expressed by abstract types (interfaces)
2. Use Proxies to forward the call to a contract to the actual implementation
3. Use Type Discovery to determine what implementations were deployed on each process

Let's pick an example inspired from a financial system. We need some services that depend one on the other, so we have some communication to play with. Say we have a PortfolioService which can get the current value of a portfolio, then we have an OrdersService to place buy or sell orders and we also have a QuotationService for getting quotations. The PortfolioService and the OrdersService call (depend on) the QuotationService, like in below diagram

To keep things simple, we say that each of these services is implemented by one class. Notice, that each of the classes in the diagram implement a correspondent interface, which represents the contract of the service.

In one deployment we would want that OrdersService directly calls (in the same process as a function call) the QuotationService (the blue arrow) and the PortfolioService makes a inter-process communication to call the QuotationService (green arrow). Then, without changing any code we'd want to change this and have all to use inter-process communication or all in-process communication.

Depend on Contracts

Another important thing to notice is that the PortfolioService and the OrdersService depend on the interface IQuotationService and not on the implementation class. This is an important constraint that our design employs.

To enforce this constraint we will place all the public contracts in a separate assembly (Contracts) which will only have contracts and which will be available on all the servers where we'll deploy. This Contracts assembly may be referenced by any service implementation from any module. This allows any service to consume any service, but it will depend on its contract and not on its implementation. In fact it will not know if it will talk to the actual implementation or to a proxy which forwards its call to another process on a different server. The IQuotationService contract will be like:

public interface IQuotationService
{
    Quotation[] GetQuotations(string exchange, string instrument, DateTime from, DateTime to);
    Quotation[] GetQuotations(string securityCode, DateTime from, DateTime to);
}

public class Quotation
{
    public DateTime Timestamp { get; set; }
    public decimal BidPrice { get; set; }
    public decimal AskPrice { get; set; }
    public string SecurityCode { get; set; }
}

An important building block of this design is to have a process which is able to host any of the services we may have. It should be able to:

• host one or more services
• publish these services to be consumed from another processes on another server
• discover at startup which services were deployed, host and publish them

In our simple example we will build it as a console app and we'll publish the services as REST endpoints. In a real distributed application on Azure this may be a Cloud Service or it could leverage the benefits of the Azure Service Fabric. On premises it may be a Windows Service.

Now in a deployment like this

where we have each service hosted alone, in its own process, they will all do inter-process communication. In a real system, such a deployment where everything is self hosted may be the first one we try out. It maximizes scalability. Then based on collected metrics we may group things to reduce the communication overhead.

For our example, let's assume that we observe that the OrdersService and the QuotationService have a very intense communication and that the communication overhead is significant. We can deploy a copy of the QuotationService in the same place with the OrdersService and load them in the same host like this

Now these two will communicate in the same process, only through function calls, and the PortfolioService will continue to use inter-process calls to a QuotationService instance that remained hosted individually.

Even more, for testing purposes maybe, we could deploy all the services in the same place and load them all in the same process like below.

We can make all these new deployments, without making any code change.

Solution Structure

The way we organize the code in a Visual Studio solution (or into folders) and how we allow references to be created is a critical aspect of this design. We want to gain a huge flexibility at deployment, so we need loose, well managed and controlled dependencies. The folders structure should lay this out. Also we should have clear rules on which assembly can be referenced by whom.

Here is a view of the structure for our example.

The first level of separation is outlined by the root folders: Infrastructure and Modules.

The Infrastructure will contain code that has nothing to do with the functional use-cases and the business logic of the application. The code here is to implement the non-functional requirements and to support the implementation of the business logic. Here we have the Proxies, the utilities for the host process and some extensions for the AppBoot.

The Modules folder, on the other hand, contains the implementation of the functional use cases. Here is where the business logic is. Within it, we have a second level of separation: the functional modules, each represented by its own folder. We also have here the Contracts assembly which has the functional contracts among the modules, again separated in its own folders and namespaces.

If we look at the references or dependencies in below diagram,

we see that we don't have nor allow references between the modules. This is important if we want to be able to deploy them in separate processes. They all depend on the Contracts, and they may also depend on the Infrastucture. The Contracts and the Infrastructure binaries will be deployed and available on all the servers where we deploy.

Proxies

The proxies are another key element of this design. Each interface from the Contacts assemblies will have at least two types of implementations: the real implementation and the proxies implementations. The proxies will just forward the call to the real implementation.

In our example we use REST for inter-process communication, so the proxy will create a HttpClient, will call the REST endpoint and will return the result. Here's an example for the QuotationService REST Proxy:

class QuotationServiceProxy : IQuotationService
{
    public Quotation[] GetQuotations(string exchange, string instrument, DateTime @from, DateTime to)
    {
        using (HttpClient client = HttpHelpers.CreateNewClient<IQuotationService>())
        {
            string path = HttpHelpers.GetServicePath<IQuotationService>("GetByExchange");
            string uri = $"{path}?exchange={exchange}&instrument={instrument}&from={from}&to={to}";
            HttpResponseMessage response = client.GetAsync(uri).Result;
            if (response.IsSuccessStatusCode)
            {
                Quotation[] value = response.Content.ReadAsAsync<Quotation[]>().Result;
                return value;
            }

            throw new HttpException((int)response.StatusCode, response.Content.ReadAsStringAsync().Result);
        }
    }
...

We should have at least two types of proxies:

• Inter-Process Proxies, which forward the call to another process using an inter-process communication protocol (like the one in the above example), and
• In-Process Proxies, which forward the call to the real implementation in the same process (basically this is just a wrapper over the real implementation)

In our implementation example, we will skip the In-Process Proxies and use directly the real implementation. However, in a real application it is needed because we need to have a consistent contract implementation from the caller perspective, no matter if it calls in process or inter-process.

For example, let's assume that the real implementation of the IQuotationService.GetQuotations() throws under certain conditions (a bug maybe) IndexOutOfBoundsException. If it is called in the same process directly, the caller will get this exception. At the same time, if it is called through the Inter-Process Proxy the caller will get an HttpException. This is not good. The caller calls an interface and we want it not to care if it calls in process or inter-process. In this example the fault contract is not consistent.

So the proxies should wrap the implementation and make sure that the public contracts in the Contracts assembly are consistently implemented for both in-process and inter-process calls.

Type Discovery

Having only dependencies against interfaces, Dependency Injection is a handy technique to get this duality on using an In-Process Proxy or an Inter-Process Proxy to get to the real implementation of a service we depend on.

At startup, before it is ready to receive requests, the host process (the ConsoleHost in our example) will scan all the deployed binaries in its output folder to determine which services were deployed and which of them should be published to be called from other processes. Using conventions, it will do the proper configuration of the Dependency Injection Container.

For example if for a contract it has the real implementation deployed it will register into the container a In-Process Proxy that will call it. If it doesn't have the real implementation it will register the Inter-Process Proxy, which will know haw to forward the call through HTTP, in our case to the service implementation from another process.

Also discovering which services are deployed, the host process may use some other conventions to know which to publish get calls from other processes. The easiest convention is to publish all deployed services.

Summary

As a continuation of the previous post, we have defined an example to show how to implement this design. We have detailed the key building blocks on it: Depend on Contracts, Generic Process Host, Solution Structure, Proxies and Type Discovery. Now we should have a good idea on how to implement this.

In the next post, we will continue with the implementation, focusing on the code of the ConsoleHost and the Proxies to get to an example that we can run and with which we can have different deployments for these three services without changing their code.

By attending my Code Design Training you will learn how to implement this design in your context, to maximize the benefits for your requirements

Featured image credit: bluebay via 123RF Stock Photo

After a long vacation in October last year, followed by some intensive work at MIRA and InfiniSwiss, now I can make some time to share some more design ideas that I have implemented along the years.

I thought to resume blogging with showing how we could design for something which sounds quite amazing:

Without changing any code nor any configuration file, without recompiling, just by copying binaries on the same or on different servers, we can change how same two classes communicate: in the same process or inter-process communication.

In other words, at Deploy Time we can decide if the same two classes communicate through simple function calls (in-process communication) or through an inter-process communication protocol (like HTTP), without recompiling nor changing anything else.

I presented this last year at some conferences and I got very good feedback, so I think it worth a few posts.

A good example where this design brings an important advantage is a financial system. Here performance is critical and at the same time it deals with large loads (in data and transactions), so it needs to scale well. With this technique we can decompose the system in more micro-services. This assures a good scalability, each of the micro-service may be deployed more times on more servers. Then, after we get some metrics from running it in production, if we see that two services have a very intensive communication and the inter-process communication affects the performance, we may redeploy those two services on the same server, load them in the same process and have them communicate directly without any overhead. And, we can do all these without changing any code, without recompile.

To explain it, we can simplify everything to the communication between a Client and a Service. If they are hosted in the same process as shown below they communicate through function calls as we'd expect

Then, if we deploy the same two classes on different servers (different processes), without changes or recompilation they will do an inter-process communication.

This may be useful in any Enterprise Architecture, because we could scale out, or on the contrary we could bring more components on the same box, without changing code. If we don't change code we have no risks of breaking things, so we don't need extensive regression testing. Even more, we don't need to bother the development team for this deployment optimization. This flexibility that we gain at Deployment Time, may be very cost effective.

The first time I've implemented this design was with a client that came after a migration from a Monolith Architecture like below (one big Windows Service for the backend on top of a database and one fat client),

to a Distributed Architecture, which had more sub-systems that composed the backend, more databases, a Service Bus for orchestrating the communication and different clients offering the user interface. A modern architecture.

With this architecture they got many benefits in implementing scalability, availability, reliability and even better security. More important for them were the benefits in the maintainability and testability. They moved from one big thing to more and smaller things. Each of the resulted sub-system was now developed, changed and tested on its own, in isolation. They were able to reach one release per month from one release per year which they had with the monolith.

However, all these came with a major drawback: performance problems due to communication overhead. They had cases in which to satisfy a business flow, say Place Order, they had like a dozen service call hops in the backed to complete it and give the response to the user. The overhead with serialize the request, place it on the wire to send it to the service provider, which has to deserialize it, and then serialize the response and place it on the wire back to the client, was significant. And this had to be done for each call, within that business request. They split the monolith, but they were having performance problems due to the communication between the resulted pieces.

One could argue that the decomposition wasn't right. And probably that was the case, but in a large enterprise system it is very hard to get it right from the start. Every refactoring, migration and new feature has to be added gradually so you can continuously perform and deliver business value. So they couldn't redo the composition all at once. Another idea was to merge back some sub-systems, but they would loose the benefits they've gained in maintainability, testability and frequent releases and they were risking to go back to a monolith.

So in this context a design that takes the best from both cases was the solution.

We wanted to:

• continue to develop, test and maintain each sub-subsystem in isolation, as if it was hosted individually in its own process, and
• be able to load more sub-subsystem in the same process and have them communicate through simple function calls, to
• be able to think about the decomposition of the system regardless of the deployment and communication concerns (primary focus on volatility and sources of change, rather than communication when making the decomposition)
• decide only at Deploy Time (configuration only), which sub-subsystem are loaded in the same process to communicate directly and which are loaded on different servers to scale and have inter-process communication between them

There are three key design ideas to achieve all these:

1. Depend only on Contracts, which are expressed by abstract types (interfaces). This means that:

• the contracts between the sub-systems are written as interfaces, DTOs and Exceptions only
• they do not contain logic
• they are the only types that have business knowledge and are shared (referenced by) all the sub-systems
• there are no references (hard dependencies) among the sub-systems implementations nor binaries

2. Use Proxies to forward the call to a contract to the actual implementation. The communication between a client and a service will be materialized through proxies. By convention:

• if the implementation is available in the same process, a proxy that forwards the call in the same process will be used
• if the implementation is not available in the same process, a proxy that can forward the call through a inter-process communication protocol (HTTP) will be used

3. Use Type Discovery to determine what implementations were deployed on each process

• at startup each process will discover (with reflection or other similar means) the implementations deployed and their dependencies
• based on conventions it will configure the Dependency Injection container on what proxies to use as implementation for the contracts that are implemented by other sub-systems

In the next posts we'll take a simple example with some classes that depend one on the other and we'll demo how this design can be implemented to achieve all the benefits outlined here. The demo will start from a high level overview and will go deep into code until we'll get to a runnable solution, in which just by copying binaries from one output folder to another we'll change the way communication happens.

To implement this I use the iQuarc.AppBoot. It offers all the features needed to implement such a design in C#/.NET. I wrote about it in my last post. The demo will be in C#/.NET, but the same ideas could be applied in other technologies. You just need some form of Dependency Injection and Reflection.

By attending my Code Design Training you will learn how to implement this design idea in your context, to maximize the benefits for your requirements

Featured image credit: bluebay via 123RF Stock Photo

AppBoot is a generic .NET application bootstrapper, we at iQuarc, have put on GitHub a while ago.

It started few years back when we were about to begin developing a large enterprise application. Back then, we wanted to use the Unity Container for dependency injection, but we liked the way MEF was declaring, with attributes, the interfaces and their implementation. The next day my colleague Cristian put in some code which did exactly that: bring the MEF like attributes for configuring dependency injection to Unity. Since then, it got refined and we've started to see many other benefits from using the AppBoot.

Now, most of the projects we begin start with the AppBoot as the first piece of infrastructure.

What Is iQuarc.AppBoot

A lightweight library that handles the startup of any .NET application. It also abstracts the composite application as concept.

AppBoot is a reusable implementation of the Separate Configuration and Construction from Use principle.

The two most important steps performed at application startup are:

• configure the Dependency Injection Container, and
• initialize the Composite Application

AppBoot is not a dependency injection container by itself. It uses one. It abstracts and hides the underlying container.

One of the most common questions I get when introducing it is:

Why use the AppBoot and not the Dependency Injection framework directly?

My answer is that by hiding the dependency injection framework from the rest of your code you gain consistency. You have consistency on how DI is done (only through constructor or only against interfaces, etc.) and you know that DI is used on all of your code. It is the same technique that I describe in my Enforce Consistency with Assembly References post.

There are also other benefits that arise from using the AppBoot. I will detail the most relevant in the rest of the post. Some are like adding the missing features to the DI framework and having a central place where such features can be added, others are in the support it brings on applying and following design patterns and programming principles.

Where Is AppBoot Available

You can find all the source code on GitHub here. If you go through the read.me file you will get the basics on how to use it. You'll see examples for the Annotation Based Configuration, for the Convention Based Configuration for the Instances Lifetime and for the Composite Application support. I won't repeat them here.

The AppBoot design is extensible. You can bring in the Dependency Injection framework that you like, by implementing the IDependencyContainer interface with an adapter for it. You can also implement new conventions on how to configure the dependency injection by adding more implementation of IRegistrationBehavior interface.

It is also available as NuGet Packages:

• iQuarc.AppBoot - the core library, which does not depend on any type of .NET application, nor on a specific dependency injection framework
• iQuarc.AppBoot.Unity - the adapters to use it with Unity Container
• iQuarc.AppBoot.WebApi - the helpers to configure it for an ASP.NET Web API project

What AppBoot Provides

Even if the AppBoot is a lightweight library, and it has a very simple code, when combined with some principles it may be very powerful in some scenarios. It supports the implementation of some good design principles like Separation of Concerns, Modularity and Loose Coupled Implementations. Its most valuable benefits detailed below, are very much connected and derive one from the other.

Dependencies Discovery

Instead of explicit registration of the pairs between Service Contract (interface) and Service Implementation (class), AppBoot promotes a way to discover them at application startup.

These pairs can be specified in a declarative manner, with attributes like:

[Service(typeof (IPriceCalculator), Lifetime.AlwaysNew)]
public class PriceCalculator : IPriceCalculator
{
	...
}

The ServiceAttribute says that the class it decorates, implements the interface it has in the constructor. Above, PriceCalculator class implements the IPriceCalculator interface.

It also specifies that the lifetime is AlwaysNew, meaning that a new instance is created each time when it is injected as a dependency to another implementation. At the other end is the Lifetime.Application which is equivalent to Singleton.

By making this kind of declaration on the implementation class, not only that we get rid of the long registration config files, but the lifetime specification is very close to the implementation. This may help in avoiding implementation mistakes. For example, if we say that this implementation is Lifetime.Application (Singleton) and it is statefull we'd better synchronize it in a multi-thread environment.

We could also specify such a pair through conventions, like:

    conventions.ForTypesDerivedFrom<Repository>()
			.ExportInterfaces(i => i.Name.EndsWith("Repository"))
			.WithLifetime(Lifetime.AlwaysNew);

This says that all the classes that inherit the base Repository class should be registered as implementations of the specific interface they implement. Like: for PersonsRepository : Repository, IPersonsRepository the (IPersonsRepository, PersonsRepository) pair should be registered.

At application startup, on Bootstrapper.Run(), AppBoot will scan with reflection all the types from all the assemblies that build our application, will look after the types that have the ServiceAttribute or match a convention and will make the registrations into the DI Container.

Modular Application

AppBoot defines an application as being composed by a set of modules. The modules are defined by a simple interface:

public interface IModule
{
	void Initialize();
}

We can create any number of modules as implementations of the IModule interface and declare them as such. For example:

[Service(nameof(MyModule), typeof(IModule))]
public class MyModule : IModule
{
	public void Initialize()
	{
		// some configuration code that runs at app startup
	}
}

At application startup, on Bootstrapper.Run(), AppBoot will find all the modules and for each of them will execute their Initialize() method. Therefore, if we have some (configuration) code that needs to run on the application startup, the AppBoot defines where that code should be and it takes care of running it.

Separation between Configuration and Construction from Use

AppBoot delegates everything related to constructing object instances to the underneath Dependency Injection framework. By hiding it from the rest of the code it assures a separation between its configuration and the rest of the code.

For example, the application code cannot call functions of the Unity Container to do additional registrations mixed with business logic, because it doesn't reference it. It can only make declarations of interface - implementation pairs, in a declarative manner through the AppBoot.

Moreover, with the modules support the AppBoot brings, we can isolate our application code from the .NET host process. This means that the application could be hosted by a ConsoleApp or by a WindowsService, or by some other kind of .NET process because it does not depend on it.

The dependency has been inverted by the IModule abstraction, and all the configuration code that should execute on the main() function of the host process will be executed as part of Bootstrapper.Run() -> IModule.Initialize().

With these two pieces: Dependencies Discovery and Modular Application support the AppBoot completes the Separation between Configuration and Construction from Use principle implementation. After the Bootstrapper.Run() the application is ready to use: the DI can construct instances and the configuration has been made.

Separates the Contracts from Implementations

The Dependencies Discovery the AppBoot does, opens many design possibilities and is a powerful technique in some scenarios. For example we could separate the contracts from their implementations into different assemblies. At deploy time, we could decide which implementations we want on each deployment, and then the AppBoot will do the registrations depending on what types it finds with reflection.

Even more we could have more functional modules isolated in different assemblies. We can make these functional modules assemblies to have no references among them (as the above diagram shows). Even if they depend one on the other functionally, they don't need to have direct dependencies one on the other, and this could be enforced by not allowing references among them. So we'll not have direct dependencies between the implementations, they will only depend through abstractions which are the contracts. (The above diagram shows that the two Application Module assembly reference the Contracts assembly and the AppBoot assembly, but they don't reference each other).

This leads to a loosely coupled system and opens the possibility to change the behavior by having different deployment configurations.

This approach puts a lot of power and responsibility at deploy time. We could make some decisions that will influence the system behavior only at deploy time, without changing any code or any configurations. I will show in some future posts how we can do this, to change if some services communicate in the same process through simple function calls or inter processes through a REST API, just by doing a different deployment, without changing any code.

Facilitates Parallel Development with Multiple Teams

With these separations we could also benefit in organizing parallel development with different teams for very large projects.

If we have several modules that are developed by team A and other modules developed by team B, we could structure them in different assemblies with no references between each team's assemblies. They could only depend through the Contracts assembly, as the above diagram shows.

Once the contracts have been decided and written into the Contracts, each team can work on their implementation without the need to even load in Visual Studio the projects belonging to the other team. They just need in the \bin\Debug\ the compiled assemblies made by the other team, or they could use some fake implementations.

This may bring important flexibility in planning and monitoring the work on very large projects.

Promotes Constructor Dependency Injection

By hiding the dependency injection framework, the AppBoot controls how DI is done. In the GitHub implementation, DI is limited to constructor. Beside having consistency, this makes that the dependencies of a class are visible into the constructor. This means that we can easily:

• see classes with more than 5±2 dependencies and be critical about them
• easily stub or mock external dependencies in tests
• prevent circular dependencies (when all the classes use constructor dependency injection, most likely an error will occur if circular dependencies exist. Depending on the DI framework this may be further enforced with extensions to AppBoot)

Moreover, we could add additional constraints in the AppBoot to allow only dependencies to interfaces, promoting with this programming against interfaces practice.

Creates Patterns in the Code

AppBoot also encourages patterns in the way code is written in a project.

For example it inherits from the Unity Container the notion of having a default implementation for an interface and also more named implementations for the same interface.

This means that for the interface:

interface IApprovalService
{
    bool Approve(ApproveRequest approveRequest);
}

we could have more implementations, which we declare by specifying a name with the ServiceAttribute, like:

[Service(nameof(BannedCustomer), typeof(IApprovalService))]
class BannedCustomer : IApprovalService
{
    public bool Approve(ApproveRequest approveRequest)
    {
        if (IsBanned(approveRequest.Customer))
            return false;
        return true;
    }

    private bool IsBanned(Customer customer)
    {
        // maybe check this in the DB or in a cache
    }
}

[Service(nameof(PriceForCustomer), typeof(IApprovalService))]
class PriceForCustomer : IApprovalService
{
    public bool Approve(ApproveRequest approveRequest)
    {
        // check if the order price is to high for the trust we have in this customer
        return true;
    }
}

At the same time we could also have the default implementation of the same interface:

[Service(typeof(IApprovalService))]
class ApprovalService : IApprovalService
{
        ...
}

When some other class asks in the constructor (depends on) an array of IApprovalService[] it will get all the named implementations except the default implementation. At the same time, when another class asks (depends on) the IApprovalService it will receive only the default implementation. The name used to declare named implementations is not used by the AppBoot. It just needs to be unique. No one could ask for a specific named implementation, it is not meant for that.

With this in mind, we could make the default implementation of the IApprovalService as a composite of the named implementations, like:

[Service(typeof(IApprovalService))]
class CompositeApprovalService : IApprovalService
{
    private readonly IApprovalService[] approvals;

    public CompositeApprovalService(IApprovalService[] approvals)
    {
        this.approvals = approvals;
    }

    public bool Approve(ApproveRequest approveRequest)
    {
        foreach (var  approval in approvals)
        {
            bool isOk = approval.Approve(approveRequest);
            if (!isOk)
                return false;
        }
        return true;
    }
}

Now, the client code which only needs an IApprovalService doesn't need to know that the implementation it gets (the default implementation) is the CompositeApprovalService which depends in its turn on the named implementations of the same interface. Moreover, it doesn't need to know that named implementations of the IApprovalService even exist.

The result resembles with the Chain of Responsibility design pattern. Separating the IApprovalService implementation into more classes each of them dealing with another concern is a far better design than having them all in a class like:

[Service(typeof(IApprovalService))]
class ApprovalService : IApprovalService
{
    public bool Approve(ApproveRequest approveRequest)
    {
        if (IsBanned(approveRequest.Customer))
        {
            // deal with banned customer
        }
        else if (IsOverPrice(approveRequest))
        {
            // deal over priced requests
        }
        else // add other cases 
    }

    private bool IsBanned(Customer approveRequestCustomer)
    {
        // maybe check this in the DB or in a cache
    }

    private bool IsOverPrice(ApproveRequest approveRequest)
    {
        // check if the price for this order is over the limit for the customer
    }
}

IDisposable Support

When using Dependency Injection (Inversion of Control at creating object instances), there is the question on who is going to call Dispose() on the IDisposable objects that the DI framework has created, and when. The answer is even more tricky when the interface that the client code depends on (like IApprovalService from above) is not inheriting the IDisposable, but only one of its implementations is IDisposable. So the client code doesn't even know that it got injected an IDisposable instance.

One valid expectation is that the DI framework should dispose them, when the operation (a web request handling for example) within which they were created, ends.

Not many DI frameworks offer a good support for this. AppBoot adds this support. It offers a way to define an operation scope, it creates a child DI container for that scope, and it disposes all the IDisposable instances which were created within that scope, when the operation ends.

In an older post Disposing Instances when Using Inversion of Control I detail the problem, the possible solutions and how this is implemented in AppBoot. The post is part of a series on dealing with disposable instances.

What Is Next?

The AppBoot is a mature library. We're using it for a while now, in production, in various projects. However, there are still a few things that could be added.

One is to have a version for .NET Core which also works with the new DI from ASP.NET Core.

Another thing would be to add support for Type Containers. At startup, AppBoot scans anyhow all the types from all the assemblies that form the application. Therefore, we could define custom attributes to decorate classes with them, and have the AppBoot initialize containers with types based on the metadata from these attributes. This kind of feature would encourage declarative programming, which would improve Separation of Concerns and reduce the complexity.

Other features would be to add support for more design patterns like Factory, Decorator, etc. These could also lead towards a better design of the client app.

My Code Design Training goes in depth on how to use the AppBoot and how to build the application infrastructure

Featured image credit: jura13 via 123RF Stock Photo

A few posts back I wrote about the benefits of having a well encapsulated data access implementation. One of the benefits outlined there was the advantage it may bring when we need to implement row-level authorization at the data access level. In this post, I will detail this implementation on top of the iQuarc Data Access library.

By row-level authorization, we mean that we want to restrict the access to the rows of one or more entities based on the rights or the role of the current user. For example we want to ensure that users can only access those data rows that belong to their department, or we want to restrict a manager to see only the data rows related to the projects she manages.

To implement this at the data access level, it means that we should build a filter based on the current user roles (or claims) and apply it to each query that is issued on the entities for which rows we should restrict the access. If we have a well encapsulated data access it means that we can use it as the central place through which all the queries go and we could extend it to intercept and append the filter to the queries. This assures us a consistent implementation for row-level authorization. Otherwise, we would need to go through all the controllers or services that send queries and append the filter "by hand", which is error prone.

Lets take a more specific example. Say we have the following entities:

, and we want to restrict the access to the Order rows such that the current user, if she is an account manager, to see only the orders of the customers that are in her area. For this case, we need to append the order.Customer.AreaID == currentUserAreaID filter to all the queries that go on the Order entity.

To get this done in a consistent manner and for all the entities where we want to have row-level authorization, we need to have a generic approach. Here, a well encapsulated data access implementation, which is Linq based (like iQuarc.DataAccess is) plays an important role. We can build a Lambda Expression for the authorization filter and add a .Where() with it on the IQueryable<T> which we'll pass to the caller. Having it implemented on top of Entity Framework (EF), we can add the .Where() on to the DbContext.DbSet<T> property, like this:

public class Repository : IRepository
{
     public IQueryable<T> GetEntities<T>() where T : class
     {
         int currentUserValue = GetFilterValueFromCurrentUser();
         Expression<Func<T, bool>> authFilter = BuildWhereExpression<T>(currentUserValue);

         return Context.Set<T>()
                  .Where(authFilter)
                  .AsNoTracking();
     }
...
}

This is very much similar with what we did in the previous post, where we built a filter for filtering tenant specific data based on the .TenanatID property. The difference is that here, we cannot make all the entities for which we need row-level authorization to implement an interface like ITenantEntity was, and to have a TenantID property to use in the filter. This is a more generic case than the multitenacy example from the previous post. Here for the Order we need order.Customer.AreadID, but for the Customer we need customer.AreadID, and maybe for other entities we want to filter based on something else like OrganizationID or CountryCode.

Therefore, the BuildWhereExpression<T>() function is more complicated here. For each entity it would need to know the two operands from the filter expression:

For each entity we can specify these as row-level authorization policies and register them in a container that will be used from the repository. Such a policy should have a Lambda Expression which can select, starting from the current entity type, the property to be used in the authorization filter, and a function that can get the filer value from the current user. In a simplified form, the class that represent such a policy may look like this:

class RowAuthPolicy<TEntity, TProperty>
{
    public RowAuthPolicy(Expression<Func<TEntity, TProperty>> selector, IRowAuthPoliciesContainer parent)
    {
        Selector = selector;
        FilterValueGetter = () => default(TProperty);
        EntityType = typeof(TEntity);
    }

    public Expression<Func<TEntity, TProperty>> Selector { get; private set; }
    public Func<TProperty> FilterValueGetter { get; private set; }
    public Type EntityType { get; private set; }
}

The class is generic by the type of the entity to which it applies and by the type of the property that will be used in the filter. The property type and the filter value type must match.

Now, we can create a RowAuthPoliciesContainer with a fluent API that allows a nice and easy way to specify these policies for the entities we want row-level authorization. We'd like something like this:

public static IRowAuthPoliciesContainer ConfigureRowAuthPolicies()
{
    return new RowAuthPoliciesContainer()
        .Register<Order, int>(o => o.Customer.AreadID).Match(CurrentUserAreaId)
        .Register<Customer, int>(c => c.AreadID).Match(CurrentUserAreaId)
        .Register<SalesArea, string>(sa => sa.CountryCode).Match(CurrentUserCountryCode);
}

The Register() function receives the Lambda Expression that will be used to select the property to filter by, and then by calling the Match() function we can pass the function which will get the filter value from the current user. The CurrentUserAreaId() is just a static helper function. It is quite simple, something like this:

private static int CurrentUserAreaId()
{
    const string areaKeyClaim = "area_key";
    Claim areaClaim = ClaimsPrincipal.Current.FindFirst(areaKeyClaim);
    int areaId = ClaimsValuesCache.GetArea(areaClaim.Value);
    return areaId;
}

we may have more such helper functions, which we may use in more policies.

The RowAuthPoliciesContainer class will create the policy classes when Register() is called and will store them in a dictionary:

class RowAuthPoliciesContainer : IRowAuthPoliciesContainer

   readonly Dictionary<Type, object> policies = new Dictionary<Type, object>();

   public RowAuthPolicy<TEntity, TProperty> Register<TEntity, TProperty>(Expression<Func<TEntity, TProperty>> selector)
   {
       var policy = new RowAuthPolicy<TEntity, TProperty>(selector, this);
       policies.Add(policy.EntityType, policy);
       return policy;
   }

   public IRowAuthPolicy<TEntity> GetPolicy<TEntity>()
   {
       return (IRowAuthPolicy<TEntity>) policies[typeof(TEntity)];
   }

   public bool HasPolicy<TEntity>()
   {
       return policies.ContainsKey(typeof(TEntity));
   }
...
}

Now, lets go back to the Repository and its BuildWhereExpression<T>() function. One first thing to notice is that the filter value is now part of the row-level authentication policy, so it shouldn't be passed through the currentUserValue parameter to the function as we've written the code in the beginning. It may be of different types, not only int, but also string or others, like we have in the CountryCode policy in the above example. Moreover, the Repository.BuildWhereExpression<T>() can only be generic by the type of the entity, which means that it does not know the type of the property, and therefore it wouldn't be able to build the Lambda Expression. To fix all these we can delegate the building of the Lambda Expression to the policy class. So by refactoring it, we'll have:

public class Repository : IRepository
{
    private IRowAuthPoliciesContainer container;

    public IQueryable<T> GetEntities<T>() where T : class
    {
         Expression<Func<T, bool>> authFilter = BuildWhereExpression<T>(currentUserValue);
         return Context.Set<T>()
                  .Where(authFilter)
                  .AsNoTracking();
    }

    private Expression<Func<T, bool>> BuildWhereExpression<T>()
    {
        if (container.HasPolicy<T>())
        {
            IRowAuthPolicy<T> policy = container.GetPolicy<T>();
            return policy.BuildAuthFilterExpression();
        }
        else
        {
            Expression<Func<T, bool>> trueExpression = entity => true;
            return trueExpression;
        }
    }
...
}

The RowAuthPolicy<TEntity, TProperty> class also gets refactored to hide the Selector and the FilterValueGetter and to use them internally to build the Lambda Expression for the authentication filter:

class RowAuthPolicy<TEntity, TProperty> : IRowAuthPolicy<TEntity>
{
    private readonly Expression<Func<TEntity, TProperty>> selector;
    private Func<TProperty> filterValueGetter;
    private readonly IRowAuthPoliciesContainer parent;
    
    public RowAuthPolicy(Expression<Func<TEntity, TProperty>> selector, IRowAuthPoliciesContainer parent)
    {
        this.selector = selector;
        this.parent = parent;
        this.filterValueGetter = () => default(TProperty);
        EntityType = typeof(TEntity);
    }

    public Type EntityType { get; private set; }

    public Expression<Func<TEntity, bool>> BuildAuthFilterExpression()
    {
        TProperty value = filterValueGetter.Invoke();
        Expression<Func<TProperty>> filterValueParam = () => value;

        var filterExpression = Expression.Lambda<Func<TEntity, bool>>(
            Expression.MakeBinary(ExpressionType.Equal,
                Expression.Convert(selector.Body, typeof(TProperty)),
                filterValueParam.Body),
            selector.Parameters);

        return filterExpression;
    }

    public IRowAuthPoliciesContainer Match(Func<TProperty> filterValueGetFunc)
    {
        this.filterValueGetter = filterValueGetFunc;
        return parent;
    }
...
}

And with this we have completed the implementation. We now have row-level authentication consistently implemented for any entity we need. We added it only by extending the data access, so we could add it with a minimum effort even at a later stage of the project if we have a well encapsulated data access implementation, which assures that all the queries and commands go in a consistent manner through it. Now, with the policies we have defined, all the screens that show Customers, Orders or SalesAreas will automatically be filtered based on the access rights of the current user. If later we want that all the screens that show Products to also be filtered we just add a new registration in the policy container:

    .Register<Product, int>(p => p.Producer.AreaID).Match(CurrentUserAreaId)

You can find the entire source code of this sample in my Code Design Training GitHub repository here.

This can be further extended:

• You may have more policies for one entity and use them based on some context or combine them in different ways. You can also add a .When() function to the RowAuthPolicy class to specify that the rule should only apply on a condition. For example the below registration says that the rule should only apply if the user is a sales person:

  .Register<SalesArea, string>(sa => sa.CountryCode).When(CurrentUserIsSales).Match(CurrentUserCountryCode);

• Another thing to consider is if you have use cases when the related entities collections should also get filtered by this mechanism. This sample implementation only takes care of filtering the root queries. If you eager load the related entities some more work is needed. For example if you have the scenario with OrderLines which have Products that belong to an area which makes them not accessible by the current user, a rule like:

   .Register<OrderLine, int>(ol => ol.Product.Producer.AreaID).Match(CurrentUserAreaId)

will get applied when the OrderLines are loaded like:

var q = repository.GetEntities<OrderLine>()
     .Where(ol => ol.OrderID == orderId);
 return q.ToArray();

However, it will not be applied when they are loaded through the related entity collection like:

var c = repository.GetEntities<Order>()
        .Include(o => o.OrderLines)
    .Where(o => o.ID == orderId);

or like:

var q = repository.GetEntities<Order>()
    .Select(o => new
    {
        o.ID,
        o.OrderLines,
        o.OrderDate
    })
    .Where(o => o.ID == orderId);

For both these examples you need to parse the Lambda Expression and to rewrite it to append the authorization filter.

The .Include() is specific to EF, so you will need to make an EF specific parser. EF does not provide a way to filter the rows loaded with Include(), so you could use the EntityFramework-Plus 3rd party project that provides an .IncludeFilter() operator.

For the second example, you would need to rewrite the query into something like:

var q = repository.GetEntities<Order>()
    .Select(o => new
    {
        o.ID,
        OrderLines = o.OrderLines.Where(ol=>ol.Product.Producer.AreaID == areaId),
        o.OrderDate
    })
    .Where(o => o.ID == orderId);

This will filter the related entity collection when it is loaded and projected into the anonymous type.

More discussions on data access security are part of my Code Design Training

Featured image credit: maxkabakov via 123RF Stock Photo

This is the last part of the "Data Isolation and Sharing in a Multitenant System" article, which is also a continuation of my post that outlines the benefits of a well encapsulated data access.

In this post, we'll look in detail at the implementation of the Shared Database strategy, explained in the first part. We'll see how to refactor the database schema for multitenancy and then how to build a Lambda Expression to filter the tenant specific data.

With this strategy we'll have one database for all the tenants, which holds both tenant shared data and tenant specific data:

This means that all the tables that hold tenant specific data will have the TenantID as the discriminant column. Here is a small view on the database diagram for our example, where with blue are the tenant specific tables.

If we're starting the project with multitenacy in mind, we'll add the TenantID from the start, so we can skip the next section and go directly at the data access. Otherwise, if we are in the case where we add multitenancy at a later stage of the project, we need to refactor the database to add the TenantID column to the tenant specific tables.

Refactor the Database Schema

Doing this is not as easy as it sounds, especially if we have data which we want to preserve.

Basically, we need to create the Tenants table and to insert in it, a row for the first tenant. (We assume that all existent data belongs to this First Tenant).

GO
BEGIN TRAN

CREATE TABLE [dbo].[Tenants] (
    [ID]   INT           IDENTITY (1, 1) NOT NULL,
    [Name] NVARCHAR (50) NOT NULL,
    [Key]  NVARCHAR (50) NOT NULL,
    CONSTRAINT [PK_Tenants] PRIMARY KEY CLUSTERED ([ID] ASC)
);

CREATE UNIQUE NONCLUSTERED INDEX [UK_Tenants_Key]
    ON [dbo].[Tenants]([Key] ASC);

SET IDENTITY_INSERT [dbo].[Tenants] ON
INSERT INTO [dbo].[Tenants] ([ID], [Name], [Key]) 
	VALUES (1, 'First Tenant', 'FirstTenant')
SET IDENTITY_INSERT [dbo].[Tenants] OFF

COMMIT TRAN

Then, we should alter each table with tenant specific data to add the TenantID, plus a FK to the Tenants table. Here, existent data complicates things. We should first add the TenantID as nullable, then update all the rows with SET TenantID = 1 and then alter again the table to make the the TenantID not nullable. Here is a simplified version of the script for the Patients table:

GO
BEGIN TRAN

ALTER TABLE [dbo].[Patients] 
	ADD [TenantID] INT NULL

ALTER TABLE [dbo].[Patients] WITH CHECK
    ADD CONSTRAINT [FK_Patients_Tenants] FOREIGN KEY ([TenantID]) REFERENCES [dbo].[Tenants] ([ID])

GO
UPDATE Patients SET [TenantID] = 1 WHERE [TenantID] IS NULL

ALTER TABLE [dbo].[Patients] 
	ALTER COLUMN [TenantID] INT NOT NULL

COMMIT TRAN

Based on this, we can create a small tool (it may be just another T-SQL script) which generates such scripts for all the tables that keep tenant specific data.

Another option to do the database schema refactor, is to use an existent schema compare tool. Such a tool may be the SQL Server Data Tools (aka Visual Studio Database Project).

Here are the steps we should do with it (you can follow the code changes for all these steps on my Code-Design-Training GitHub repository. For each step I've made a tag, so you can easily follow the progress):

1. Create a database project with the current schema of the database (the one without multitenancy things) [Tag on GitHub: mt-shared-step1]
2. Edit the schema in the tool, to add the Tenants table, the TenantID column and the foreign keys [Tag on GitHub: mt-shared-step2]

• we can do this in Visual Studio, or we could publish the schema in a temporary database (which has no data), do the changes there (with SQL Management Studio) and then update back the schema from the Database project

3. Generate the Publish Script from the Database project to our original database [Tag on GitHub: mt-shared-step3]

• this should be a script that prevents data loss and assures a safe way to refactor the database

4. Edit the generated Publish Script to insert the row for the First Tenant, to set the FKs to it, and to make the FKs not nullable [Tag on GitHub: mt-shared-step4]

• for each table, the generated script creates temporary table with the new schema (which contains the TenantID column) and copies the existent data to it. Then it drops the original table and renames the temporary one as the original. So, we do the following:
  • search for all CREATE TABLE statements and make the TenantID NOT NULL
  • add in all INSERT INTO ... SELECT (..) statements the TenantID and its PK value 1
  • add the INSERT INTO [dbo].[Tenants] ([ID], [Name], [Key]) VALUES (1, 'First Tenant', 'FirstTenant') after the CREATE TABLE [dbo].[Tenants] (...

5. Run the Publish Script against the database to execute the refactor [Tag on GitHub: mt-shared-step5]

• after this we could update the database project with the changes we did to the database with this refactor

Isolate Tenant Data through Data Access

The next step is to make sure that tenant data is isolated. This means that each query or command we send to the database must have a WHERE TenantID = ... clause appended. Here, having a well encapsulated data access implementation makes a huge difference, because it assures us that all the queries and commands go through it, so we can intercept them to append the WHERE.

If we are using a Linq based data access, we can build a Lambda Expression for the TenantID = currentTenantId filter and add a .Where() with it, to IQueryable<T> we're going to pass to the caller. If the data access is built on top of Entity Framework (EF) this simplifies to adding the .Where() to the DbContext.DbSet<T> property.

Lets take the iQuarc.DataAccess as an example for the data access implementation (for more on this, you can go to my older post "Separating Data Access Concern"). We should modify the Repository.GetEntities<T>() and the UnitOfWork.GetEntities<T>() functions, from:

public class Repository : IRepository, IDisposable
{
  public IQueryable<T> GetEntities<T>() where T : class
  {
    return Context.Set<T>().AsNoTracking();
  }
...
}

into

public class Repository : IRepository
{
     public IQueryable<T> GetEntities<T>() where T : class
     {
         int tenantId = GetCurrentUserTenantId();
         Expression<Func<T, bool>> condition = BuildWhereExpression<T>(tenantId);

         return Context.Set<T>()
                  .Where(condition)
                  .AsNoTracking();
     }
...
}

Actually, we don't have to modify the existent Repository or UnitOfWork classes. We could apply the Decorator pattern and write a MultitenancyRepository which wraps the existent implementation and does the filtering based on the current tenant.

Lets look at the two helper functions we've introduced.

The GetCurrentUserTenantId() is similar with the one from the previous post, where we've looked at separate databases strategy implementation.

private int GetCurrentUserTenantId()
{
    const string tenantKeyClaim = "tenant_key";
    Claim tenantClaim = ClaimsPrincipal.Current.FindFirst(tenantKeyClaim);
    int ternanId = tenantsCache[tenantClaim.Value];
    return ternanId;
}

It relays on the existent tenant_key claim of the current user, which should be set by the authentication mechanism. Then, it uses a cache build from the Tenants table to return the tenantId which corresponds to the key. Nothing fancy.

The BuildWhereExpression<T>()is a bit more complex. It needs to build a Binary Expression with the equals operator. The left operand should be the TenantID property of the current entity and the right operand the tenantId which is passed as a parameter. For the Patient entity this whould be: patient.TenantID == tenantId.

This means that all the tenant specific entities should have the TenantID property mapped to the TenantID column. So we should change the EF code generator to make these entities implement the ITenantEntity interface

interface ITenantEntity
{
    int TenantID { get; set; }
}

Having this, we can easily build the left operand and also we can know if the current entity is tenant specific or not. The operands are:

• left operand: tenantIdSelector = entity => entity.TenantID;
• right operand: tenantIdParam = () => tenantId;

so the entire function code goes like this:

private Expression<Func<T, bool>> BuildWhereExpression<T>(int tenantId)
{
    if (IsTenantEntity<T>())
    {
        Expression<Func<ITenantEntity, int>> tenantIdSelector = entity => entity.TenantID;
        Expression<Func<int>> tenantIdParam = () => tenantId;

        var filterExpression= Expression.Lambda<Func<T, bool>>(
            Expression.MakeBinary(ExpressionType.Equal,
                Expression.Convert(tenantIdSelector.Body, typeof(int)),
                tenantIdParam.Body),
            tenantIdSelector.Parameters);

        return filterExpression;
    }
    else
    {
        Expression<Func<T, bool>> trueExpression = entity => true;
        return trueExpression;
    }
}

If the current entity type is not implementing ITenantEntity interface, then on the else branch we will just build and return an expression which is always true so the .Where() we append has no effect for the tenant shared tables.

Now, if we do the same for the UnitOfWork class we have consistently achieved data isolation for each tenant data at the data access level.

You can see a full running demo of this implementation if you go to my Code Design Training GitHub repository and open the SharedDbConsoleDemo sample.

More discussions multitenancy design are part of my Code Design Training

Featured image credit: alinoubigh via 123RF Stock Photo

This post is the second part of the article that shows how a well encapsulated data access implementation can play a key role in implementing a multitenant application. It is also a continuation of my post that outlines the additional benefits such a data access implementation may bring, multitenancy being one of them.

The previous post (the first part of the article) focuses on presenting the main two strategies for addressing Data Isolation and Sharing in a multitenant application:

• Separate Databases one for each tenant, or
• a Shared Database used by all tenants

It also shows how we can make an informed decision on which of these strategies fit best a certain context.

Now, we go into the implementation details for these strategies. Both are in C# and rely on a well encapsulated data access implementation, for example one like the iQuarc Data Access, which I've presented a while ago in this post. This post covers the first strategy and the next one the second.

We'll use the same example as in the first part: the multitenant application used in clinics for physiotherapy, where the clinics are the tenants, the patient related data represents tenant specific data and the diagnostics or common exercises related data is the tenant shared data.

Separate Databases Implementation

With this strategy we'll have one database for each tenant. If we'll have n tenants we'll have n databases. When a new tenant comes, we'll have to create a new database for it.

Being in a multitenant architecture, we have the same instance of the application serving all the users from all the tenants. Based on the tenant to which the current user belongs to, the application should connect to the database that corresponds for that tenant. To maximize the cost savings that the multitenancy may bring we'll have the same database schema (same version) for all the databases from all the tenants.

At the database schema level, we do not need to do any changes from a design of a non-multitenant application. Each database will have the tables both for the tenant specific data and for the tenant shared data. The tenant shared data will be duplicated in all the databases. Also the DataModel assembly, which contains the DTOs that are mapped by EF to the tables of the database will suffer no changes. Moreover, having the same database schema for all tenants, means that we can have the same DataModel for all. (for more details on why the EF DTOs would be generated in a separate assembly DataModel from the data access implementation DataAccess you can refer to the Separating Data Access Concern post)

The only place where we need to do intervene to implement multitenancy with this strategy is in the DataAccess. More precisely where we create a connection to the database. This is the only place, because we are relying on a well encapsulated data access implementation, which assures us that only here a connection to the database is made. Otherwise, we would have to go through all the services or controllers that might create an entity framework DbContext (or a connection to the database with other means) and deal with those places as well.

First we need a configuration of the connection strings which allows us to identify them for each tenant. We'll keep them in the config file.

One option is to have more connection strings and use a convention for the name attribute based on the tenant_key.Something like:

<connectionStrings>
  <add name="Tenant1_PhysioDb" connectionString="..." />
  <add name="Tenant2_PhysioDb" connectionString="..." />
  ...
</connectionStrings>

Another option is to have a template connection string and to have a convention for the database name based on the tenant_key. The application will have to replace the <tenant_key> with the key of the current tenant, before the connection string will be used. This would look like:

<connectionStrings>
  <add name="PhysioDb" 
     connectionString="...;initial catalog=<tenant_key>_PhysioDb;..." />
</connectionStrings>

This option works fine if we deploy all the databases on the same server and we'll use the same credentials for the application to connect to any of the databases. Otherwise, the template gets too complicated.

Next we need to make the application to use the connection string based on the tenant of the current user. The Entity Framework (EF) DbContext receives into the constructor the name of the connection string to use. We need to change the code generator, to make this constructor public for the class it generates for our specific data model (which inherits the DbContext). It should look like:

public class PhysioEntities : DbContext
{
    public PhysioEntities(string nameOrConnectionString)
        : base(nameOrConnectionString)
    {
    }

    public virtual DbSet<PatientFile> PatientFiles { get; set; }
    public virtual DbSet<RehabSession> RehabSessions { get; set; }
    ...
}

Having a well encapsulated data access, it means that the PhysioEntities class is hidden from the rest of the code. It is instantiated inside the data access and used by a Repository or an Unit of Work implementation. If we look into the iQuarc Data Access library code, we see that this is already well separated. The IDbContextFactory abstraction takes the responsibility of constructing the context, and the Repository or UnitOfWork clases only use it as a DbContext instance. (An application of the Separate construction and configuration from use principle.) Here is a simplified snippet of this code.

public interface IDbContextFactory
{
    DbContext CreateContext();
}

public class Repository : IRepository, IDisposable
{
    private IDbContextFactory contextFactory;    
    public Repository(IDbContextFactory contextFactory, IInterceptorsResolver interceptorsResolver, IDbContextUtilities contextUtilities)
    {
        this.contextFactory = contextFactory;
        ...
    }

    private DbContext context
    protected DbContext Context
    {
        get 
        { 
            if (context == null)
                context = contextFactory.CreateContext()    
            return contextBuilder.Context; 
        }
    }

    public IQueryable<T> GetEntities<T>() where T : class
    {
        return Context.Set<T>().AsNoTracking();
    }
    ...
}

If we have this abstraction already done, it means that we don't even need to change the core code of the data access implementation to add the multitenanacy. We only need to come with another implementation for the IDbContextFactory. If we don't, we just need to create the abstraction as above. It's simple, just delegate the new PhysioEntities() code to the IDbContextFactory.CreateContext().

The multitenancy IDbContextFactory implementation, will have to take the tenant_key of the current user, to determine the connection string to use and to instantiate the PhysioEntities with it. Here is a snippet of the code:

public class MultitenancyDbContextFactory : IDbContextFactory
{
    public DbContext CreateContext()
    {
        string tenantKey = GetTenantKeyFromCurrentUser();
        string connectionName = $"{tenantKey}_PhysioDb";
        return new PhysioEntities(connectionName);
    }

    private string GetTenantKeyFromCurrentUser()
    {
        const string tenantKeyClaim = "tenant_key";
        Claim tenantClaim = ClaimsPrincipal.Current.FindFirst(tenantKeyClaim);
        return tenantClaim.Value;
    }
}

Next we only need to register in the DI Container the MultitenancyDbContextFactory class as the default implementation of the IDbContextFactory, so the Repository and UnitOfWork use it, and we're done.

One thing to notice is that we take the tenant_key from the claims of the current user. We expect that an authenticated user will have this claim, with the correspondent value for the tenant she belongs to. This should be assured by the authentication part, which is another important aspect in a multitenant application. We should have a separate Identity Provider Service which handles the identity management and user authentication. For each authenticated user it should add this tenant_key claim, so our application can use it.

The Identity Management and Authentication in a multitenant application is not in the scope of these posts, but here is a good guidance on how to do this using OpenID Connect, and Azure AD as the Identity Provider Service. Identity Server is an alternative to Azure AD, which you can host yourself.

Another important aspect for this Data Isolation and Sharing strategy is to maintain the database schemas in sync for all the databases of all the tenants. The best way to do this is to rely on a tool which can generate SQL scripts that create the schema of the database. These should include the tables, view, stored procedure, system data and everything else. The SQL Server Data Tools (also known as Visual Studio Database Project) is a good tool for this. Once we generate the database schema, we can add it in git and use it as the single source of truth for the state of the database. With this, when we have a new tenant is easy to create its database. Even more, such a tool can also generate a diff T-SQL script which results from comparing the one in git with a deployed database. This diff T-SQL could update the schema of the deployed database. If we integrate this into the tools that do the deployments in all our environments (testing, acceptance, production) the overhead of having more databases and keeping their schema in sync can be very much reduced.

Closing

This covers the implementation of the Separate Databases strategy for data isolation and sharing in a multitenant application. In the next post, which is the third part of this article, I will detail the implementation of the other strategy we've discussed in the first part, the Shared Database strategy.

More discussions on implementations for multitenancy are part of my Code Design Training

Featured image credit: alinoubigh via 123RF Stock Photo

When I've migrated my blog to Ghost (My Wordpress to Ghost Journey) I've left a few things not done. Quite a few actually. Integrating a comments service was one of them.

Last year, I didn't have much time for the blog, and the little time I had went into writing posts, so the TODOs list remained quite the same. Now, its time take them one by one, so expect some more improvements soon.

Today I've integrated Disqus, and I invite you to comment on my posts!

Below are a few technical details on how this went in my case.

Integrating Disqus into a Ghost blog

As expected it was very simple to add Disqus. I've just followed the steps from this tutorial, which even if it is a bit out of date, it tells very well the steps that are needed.

After I've created the account and accepted the policy, at the Select Platform step, I had Ghost as an option,

so I didn't go for the Universal Code, like the tutorial says, even if I think it would have been pretty much the same.

Next, at the Install Instructions Disqus tells the exact installation steps:

Howerver, the step 2, is not very accurate. Actually, it should say to insert the code anywhere between the {{#post}} and its ending tag {{/post}}.

Depending on your theme, you might have a footer section, like I have, and you'd probably want the Disqus comments to go in there and not at the end of the article. In my case, I wanted it inside the footer, so it gets its styling, but after the author and the share this post sections, so the code went inside <footer> which is inside {{#post}}.

and the result is

For the Configuration Variables from the discus_config function, I've just followed the instructions from the tutorial, so I had:

var disqus_config = function () {
    this.page.url = '{{url absolute="true"}}';  
    this.page.identifier = 'gh-{{id}}';
    };

The page.identifier might require some thinking. If we look into Disqus documentation it says that this variable is used to uniquely identify the comments thread that should be shown in the post. So, indeed the {{post.id}} seems a good idea. However, this id is nothing more than an integer that is incremented by ghost with each page or post we add. What if I will migrate from Ghost to another platform? The alternative would be not to use it and to rely only on the post.url, which change is supported by Disqus Migration Tools. However, on the other hand, I might need to change the URL before a migration for other reasons, like a title rename, or a typo. In the end, I've concluded that I'll use it with the gh- prefix, because:

• it is more probable to change the post URL due to a typo than to do a migration ;)
• even if I migrate, I could probably keep this ID on the new platform. If not, it will just not be set. If it is not set, Disqus will fall back and use the URL. So, after an eventual migration, I should not change the URLs of the migrated posts.
• the gh- prefix makes it more explicit saying that this is an ID generated by Ghost. Plus, I can use another prefix like dev-gh- when I test the comments from my test env, using the same Disqus account.

The last thing was to do a quick test and look at the page source to see if these variables are set as expected:

And, with this you can comment on my blog, including on the previous posts. If you have questions, comments or suggestions on this, you can, now, directly ask them below.

Featured image credit: rawpixel via 123RF Stock Photo

In the previous post I've shown few of the additional benefits a well encapsulated data access implementation brings. Implementing multitenancy is one of them. This article continues the previous, picks up the multitenancy context from there, and shows how we can address one of the most important design challenges of multitenant applications, which is Data Isolation and Sharing.

The article turned out quite long, so I have structured it in three parts. This first part focuses on describing the two strategies for Data Isolation and Sharing and more important on how to choose between them. The second and the third parts will show the implementation details on for each strategy.

Multitenancy

Before we go into details, lets review a bit what multitenancy is. The wikipedia definitions says:

The term software multitenancy refers to a software architecture in which a single instance of software runs on a server and serves multiple tenants.

A tenant is a group of users who share a common access with specific privileges to the software instance.

With a multitenant architecture, a software application is designed to provide every tenant a dedicated share of the instance.

We can think of a tenant as an organization which is a customer of our application. The users that belong to that organization is the group of users that form that tenant.

We do multitenant systems because they allow for cost savings. If we have one instance of the application for all our customers we may save money on hardware, software licenses and operational costs. Almost any application has a certain overhead on the resources it needs. If we spread it over more customers of the same app instance we can reduce it. There is memory and processing which is usually not used or used only at peak time; the effort to monitor and operate one instance of the app does not grow with its size as it grows with monitoring more instances of the same app; similar, the number of licences (servers and tools) is more dependet on the number of instances rather than on the size of one instance. All these make a good case for multitenancy architectures over multi-instance architectures. A perfect case for SaaS.

The cost savings can be eclipsed by two main things:

• the difficulty of scaling, and
• the difficulty of satisfying each tenant specific needs.

The one application instance should be able to scale when demand grows, when we add more tenants. There are many aspects to consider when designing for scalability, but I won't detail them in this post. For the second, we should aim for having the same functionality for all the tenants. If one client asks for a new feature and we implement it, then all clients will get it. Maybe some will not use it, or we configure the system, so that they don't have access to it, but it will be there. This also points another important aspect in a multitenant design, which is configurability.

Data Isolation and Sharing

At this point, we should have a good understanding of what multitenancy is and when it makes sense to implement a multitenant architecture. Now, lets focus on the Data Isolation and Sharing aspect, and go back to the scenario from the previous post.

To make it simpler, lets take an example. Lets think of an application which is used in clinics for physiotherapy. The application provides functionalities to assist the doctor, the physiotherapist and the patient during the recovering from surgery or injury. The doctor may give prescriptions, and the physiotherapist may compose a set of exercises that the patient should do, and also to monitor the patient progress. Such an application may be used by large hospitals and at the same time by very small physiotherapy clinics. They would all need a similar functionality, so it makes sense to make it a multitenant application, where each clinic or hospital is a tenant. (This is a simplified example of what we do at MIRA)

Here, we'll have data which belongs to each tenant. For example all the patient related data, which includes: the patient profile, the patient file, the progress the patient had during any rehabilitation program at that specific clinic, etc. These are tables like Patient, PatientFile, RehabProgram, PatientHistory etc. This data needs to be isolated for each tenant. Each clinic has to feel like that it is the only one on the platform, so a doctor at a clinic cannot see the patients of another clinic and so on. At the same time there is also data which should be shared by all tenants. Here we may have diagnostics, a common set of exercises, a common set of tools, etc. These are tables like: Diagnostics, Exercises, RehabTools etc. This is typically system data, which we want to be used by all tenants and we want that any changes of it to be available to all tenants. Usually it is maintained by system administrators, but there are cases when we want that users that belong to any tenant to be able to edit it. For example we might want that any doctor to be able to enrich the diagnostics that the application knows.

We'll find the same in any multitenant application. We're always have:

• tenant specific data that needs to isolated for each tenant, and
• tenant shared data which needs to be shared between tenants

Another good case for multitenancy are applications that are used in the insurance field. By hosting on the same application instance, large insurance companies with the very small ones we could reduce the operational costs by sharing the resources. Tenant specific data would be the: insurance products, the policies which belong to each insurance company (tenant) etc. and tenant shared data would be the regulations data or the actuarial tables (which are used by all insurance companies to calculate the premiums), etc.

Separate Databases or Shared Database

Once we've identified the tenant specific data and the tenant shared data, the next step would be to decide between the two main strategies for storing data:

• Separate Databases one for each tenant, or
• a Shared Database used by all tenants

Both of the above strategies have pluses and minuses, and there are many tradeoffs to consider when making this choice.

Having a Separate Database for each tenant is easier to implement, especially in the case when we add multitenancy at a later stage of the project (after part of the functionality was already implemented as if this would not be a multitenant app). In this case each tenant will have access only to its own database and the data schema should not be changed from the non-mutitenant version.

This is also simple for isolating the tenants data. Based on the current user, we know the tenant it belongs to and we'll connect to that tenant database. So there no risk that we'll mix multiple tenants data. (Of course if we have caching at higher levels those should also be tenant aware, but that's another topic.)

To maximize the cost benefits of the multitenancy app, we should keep identical schemas for each database of each tenant. This means that when we make a schema change (because we add a column for a new feature) this change needs to be applied to all the databases. Therefore, we'll have the same database version (in fact the same app version) for all the tenants. This means that our automated deployment tool will need to upgrade the schemas of all the databases when we deploy a new version in testing, acceptance or production environments.

For the tenant shared data one approach is to duplicate it in the databases of each tenant. This means that we need to synchronize its changes on all the databases. If it is only system data, then this may be done in the same process with updating the databases schemas from version to version. Otherwise, if users can change it (operational data), then we need to put a sync mechanism in place. In most of the cases it does not need to be a real-time sync, because changes done by one tenant, on the shared data, do not need to be instantaneusly available to the others.

Another approach is to have a separate database for the tenant shared data only. So for a tenant we will connect to its own database for its specific data and to the tenant common database for the shared data. We keep the simplicity on isolating the tenant data and we don't need to do any data sync. However, we would need to take out the tenant shared data tables from the tenant specific database and use some GUID as IDs to link to it. This approach may make even more sense if we have a large set of tenant shared data, which frequently changes and it which is not very interconnected with the tenant data. With this approach we could even consider a non-relational database for the tenant common database.

The other strategy, a Shared Database used by all tenants, means that we keep data from all the tenants in the same database. To isolate tenant specific data, we will have to add a discriminator column like TenantID to every table which is tenant specific, and to make sure that all the queries and commands will filter the data based on it.

With this strategy dealing with tenant shared data is simple, we just don't filter it. Isolating data is what we need to deal with. For this we need to make sure that ALL the queries and the commands that deal with tenant specific data get filtered by the TenantID. Here, having a well encapsulated data access through which all the queries and command go can play a key role in assuring this. The data access would be the place where we intercept each query and command, we determine if it deals with tenant specific tables, and if yes we append a WHERE clause that filters it based on the current tenant. With Linq we would like to add something like:

    int currentTenantId = GetCurrentUserTenanatId();
    IQueryable<PatientFile> originalQuery
           .Where(p => p.TenanatID == currentTenantId);

In the second part of this article (next post), I will detail this code and show how we can make it generic as part of the data access implementation.

This strategy is easier to implement if we know from the beginning that we should build a multitenant system, because we will consider it while designing the database and it will lead to a better design. Also it will be easier to set and maintain the conventions based on which we can implement the generic code that appends the tenant filter. This doesn't mean that we cannot add it at a later stage of the project, IF we can rely on a well encapsulated data access and if we have a good database design in place. After we decide which is tenant specific data and which is tenant shared data, we can build some SQL scripts that add the TenantID column. This database refactoring may be more difficult if we're already in production, because we might also need to migrate production data to the new schema of the database.

Both these strategies have pluses and minuses and the choice which one is better depends on many aspects including the application field and business, the moment when multitenancy should be added and also on the advantages we want to achieve from the multitenancy model. Below tables illustrate a summary of the benefits and the liabilities of these two main strategies for isolating and sharing data in a multitenant system:

Separate Databases

Shared Database

The tenant shared data may play an important role on deciding between these two strategies. For example, an application where the tenant shared data is operational data (users from all tenants may modify it), and it represents a significant part from the entire application data model, and it is highly connected (related) to the tenant specific data, it makes a good case for the Shared Database strategy. In the same example, where the tenant shared data is operational data, is big, but it is not very related to the tenant specific data we could go for the Separate Databases strategy, even more if we're adding multitenancy later or if we have very few tenants with large data sets or if we cannot assure data isolation through other means.

There are many other metrics we should measure or estimate before making a decision on one of these strategies. Here are a few:

• the number of tenants
• the number of users per tenant
• amount of data or data transactions per tenant
• frequency of adding tenants or removing (disabling) tenants
• the ratio between small and big tenants
• the frequency of database schema changes

For example a scenario with many small tenants where we want to have the flexibility to add or disable tenants often, would make a good case for the Shared Database strategy. On the other hand the scenario with a fairly fixed number of tenants, which are big (in data amount and data transactions) and similar in size makes a better case for the Separate Databases strategy.

The development model may also play a role in making this decision. For example if we want to go very fast into production, with minimum business functionality and then increase the functionality by doing Continuous Delivery the Shared Database strategy may be a better choice because we'll need to change the database schema quite often. On the other hand if we go into production after most of the functionality is done, and we don't foresee many changes to the database schema Separate Databases strategy may be a good choice because the liability of high costs on updating more databases schemas won't be paid that often.

So, if we put some numbers on these metrics and we balance the benefits and the liabilities of these two strategies we could make a good decision in our context. Even more, we could end up with a strategy which is a hybrid between these two if we have external constraints and we try to take benefits from both or to minimize the liabilities of one of them. For example, in a scenario with one big tenant and many other small tenants we might have one database for the big tenant and one shared for the small ones. With this separation we may target benefits in planning the activities for the backup and restore, or the schema changes for the big fish from the small ones. Another scenario which may lead to such an hybrid is when we go with the Shared Database and at a certain point we need to scale at the database level. We can create a new database and distribute the tenants among them.

Closing

At this point we should have a good image on the strategies for the Data Isolation and Sharing in a multitenancy application and how to choose the one that suits best a given context. In the second part of the article (the next post) we will go in details on how to implement each of them, by looking at some code snippets that give an implementation direction based on a well encapsulated data access.

More discussions on designing for multitenancy, high scalability and SaaS scenarios are part of my Code Design Training

Featured image credit: alinoubigh via 123RF Stock Photo

We all know that achieving a good Separation of Concerns is good, because it increases the maintainability and it reduces the cost of changing our code. This pays off big time in long running projects, with large code bases where changing code is a big part of the coding activity and where the predictability of the effects of our changes is very important.

A while ago I have written a post that shows how to achieve a good separation of the Data Access Concern, even when using an ORM like Entity Framework. I've outlined there that just using the EF for the data access is not enough because it does not give consistency. I have also presented the iQuarc Data Access library (available on GitHub and on NuGet.org), which implements the Repository and Unit of Work patterns making a good encapsulation of the data access concerns and creating consistent development patterns on how the data access will be used from the above layers in the entire code base. We've used this library in many projects we've started since then, and we've seen many benefits on the cost of adding technical features later, because of the well encapsulated data access. In this post I'm going to give some examples of them.

The context of these examples is the one of large projects where after a few months of developing business functionalities (lets say data entry screens) we need to add some technical functionalities (let's say auditing each query execution). Without a central place through which all queries' execution goes, we would need to go back thorough all of the Controllers and the Services and log the query execution. So the effort to consistently add such a technical feature depends on the size of the project (code) at the moment we add it. The more Controllers and Services we have, the more places we need to modify and test, so the more effort is needed. Even more, the effort/cost is not linear to size, because complexity is not linear to size and the complexity grows because we don't have a consistent way to doing data access. Below graphic shows this:

The red line represents the case where we need to go screen by screen and add the technical feature and the blue line represents the case where we have a well encapsulated data access implementation that allows such extensions. We see that in the second case the cost hardly grows by size, after we have such a data access implementation in place.

Now lets see the examples one by one. I'll list them here for an easier navigation. For some, I might write separate posts, so this one doesn't get way too long :)

• Write in an audit log when an entity was read, modified or deleted
• Data Audit
• Data Localization
• Multitenancy
• Authorization on data records

Write in an audit log when an entity was read, modified or deleted

Context: We need to write in an audit log the name of the user that has read, modified or deleted a Patient entity related data and the date and time when the action happened.

Without a centralized place through which all our queries and commands go, we would need to go through all our code where the Patient entity is added, modified or deleted, and make a call to the IAuditLog in a consistent way.

If we use the iQuarc Data Access library, we can create an entity interceptor for this by implementing the IEntityInterceptor<T> interface, and that would be all:

[Service(nameof(PatientAuditLogInterceptor), typeof(IEntityInterceptor<Patient>)]
class PatientAuditLogInterceptor : EntityInterceptor<Patient>
{
    private readonly IAuditLog auditLog;

    public PatientAuditLogInterceptor(IAuditLog auditLog)
    {
        this.auditLog = auditLog;
    }

    public void OnLoad(IEntityEntry<Patient> entry, IRepository repository)
    {
        User user = GetCurrentUser();
        Patient patient = entry.Entity;
        auditLog.Write(AuditType.Read, $"Patient data was read. Patient Name: {patient.Name}", user);
    }

    public void OnSave(IEntityEntry<Patient> entry, IUnitOfWork unitOfWork)
    {
        User user = GetCurrentUser();
        Patient patient = entry.Entity;

        if (entry.State == EntityEntryState.Added)
            auditLog.Write(AuditType.Added, $"Patient was added. Patient Name: {patient.Name}", user);
        else
            auditLog.Write(AuditType.Added, $"Patient was modified. Patient Name: {patient.Name}", user);
    }

    public void OnDelete(IEntityEntry<Patient> entry, IUnitOfWork unitOfWork)
    {
        User user = GetCurrentUser();
        Patient patient = entry.Entity;
        auditLog.Write(AuditType.Deleted, $"Patient was deleted. Patient Name: {patient.Name}", user);
    }
...
}

The entity interceptors are nothing more than extension points that the iQuarc Data Access library provide (a simple OCP application). They are somehow similar with database triggers in the sense that we can write code that executes when an entity is saved, loaded or deleted, but it is at a higher level, not in the database. Even more, the code we write does not depend on the EF and it can sit in the Business Logic Layer (DIP application). You can find more details on how they work in my older post or if you just look in the code on GitHub.

There are many other technical features we can easily add in the same way as we did in this example. Here we have implemented a specific interceptor, which executes only for entities of type Patient, but we can also have global interceptors that execute for entities of any type, like in the following example.

Data Audit

Context: We need to set the CreatedBy, LastModifiedBy, CreatedDate and LastModifiedDate columns in most of the tables from the database. We add this feature at a later stage of the project.

To implement this in a generic way for all the entities we need to define an interface. Something like:

public interface IAuditable
{
    DateTime? LastModifiedDate { get; set; }
    DateTime CreatedDate { get; set; }
    string LastModifiedBy { get; set; }
    string CreatedBy { get; set; }
}

After we create the correspondent columns in the tables of the database (we can make a T-SQL script that adds them for all tables), the next step is to make all the DTOs, which the ORM maps to the tables, to implement this interface. If we use EF, we can easily modify the code generator to generate the entity classes to implement the interface and the audit columns to be mapped to its properties.

The above needs to be done regardless of the fact that we add this feature at the beging of the project or later. It is almost the same effort / cost.

However, to make sure that we implement this feature in a in a constant manner and these properties are correctly set in all of the Controllers and Services where we modify data, we can again benefit from having an encapsulated data access implementation, which gives us a central place to set this data. Again, by having this, we will not need to go through all the code we've already written, if we are at a later stage of the project.

This time we will make a global interceptor, so it gets executed for all entities types.

[Service("AuditableInterceptor", typeof(IEntityInterceptor))]
public sealed class AuditableInterceptor : GlobalEntityInterceptor<IAuditable>
{
    public override void OnSave(IEntityEntryFacade<IAuditable> entity, IRepository repository)
    {
        var systemDate = DateTime.Now;
        var userName = GetUserName();

        if (entity.State == EntityEntryStates.Added)
        {
            entity.Entity.CreationDate = systemDate;
            entity.Entity.CreatedBy = userName;
        }

        entity.Entity.LastEditDate = systemDate;
        entity.Entity.LastEditBy = userName;
    }
...
}

The data access implementation, finds in the Dependency Injection Container all the implementations of IEntityInterceptor, including the above, and for each entity that was modified, deleted, added or loaded it calls the correspondent functions of each interceptor found. The GlobalEntityInterceptor<T> is just an implementation helper (Template Method design pattern applied), which casts each modified entity to IAuditable and if that succeeds it forwards the call to the specific implementation.

public abstract class GlobalEntityInterceptor<T> : IEntityInterceptor
    where T : class
{
    public abstract void OnLoad(IEntityEntryFacade<T> entry, IRepository repository);
    public abstract void OnSave(IEntityEntryFacade<T> entry, IRepository repository);
    public abstract void OnEntityRemoved(IEntityEntryFacade<T> entry, IRepository repository);

    void IEntityInterceptor.OnLoad(IEntityEntryFacade entry, IRepository repository)
    {
        if (entry.Entity is T)
            this.OnLoad(entry.Convert<T>(), repository);
    }

    void IEntityInterceptor.OnSave(IEntityEntryFacade entry, IRepository repository)
    {
        if (entry.Entity is T)
            this.OnSave(entry.Convert<T>(), repository);
    }
...
}

Data Localization

Context: Our application is an e-commerce site which is in production. Now, we need to enter the French market, where most of the products we sell have different names in French than in English. Therefore, we need to add data localization.

I have written before about data localization, explaining what it is and how it can be implemented. In this older post I give the starting point of implementing it as part of an encapsulated data access implementation. It fits well the above context. It parses each Linq and then recreates the lambda expression with a join to the translation table. Doing this we will not need to go through all the existent code where Products are read to modify it. Instead we intercept the existent queries and we rewrite them.

public class EfRepository : IRepository
{
...
  public IQueryable<T> GetEntities<T>(bool localized = true) where T : class
  {
     DbSet<T> dbSet = GetContext().Set<T>();
     return localized ? new LocalizedQueryable<T>(dbSet, this .cultureProvider) : dbSet;
  }
...
}

For more details on how to implement the LocalizedQueryable<T> to rewrite the Linq, you can look into the code of the iQuarc Data Localization library that I presented in the previous post.

Multitenancy

Context: Our application is in a late stage of development, or even deployed in production, and we decide that the same instance of the application should be used by more clients. Therefore, we need to add support for the multitenancy scenario.

There are more strategies to implement a multitenant application. The most common are to have separate databases, one for each client (tenant), or to have one database shared by all clients and use a discriminant column like TenanatID to separate each tenant data. Both these strategies could be implemented later in a project, and having a well encapsulated data access where you can intervene to consistently implement it, makes a huge difference. The rest of the code will need minimum changes, given that you will want the same functionality to all your tenants and that your application was built with scalability in mind.

If you go with multiple databases strategy (one for each tenant), the data access implementation will be the place to decide to which database you will connect to execute the current query or command, based on the tenant of the current user. If you go with one database for all the tenants, the data access will be the place where you can intercept each Linq and rewrite it to append a WHERE condition that filters the data by the TenantID of the tenant to which the current user belongs to.

I will detail in a future post more about multitenancy, what should we consider when we choose a strategy and also give some code snippets on how to implement it taking the benefits of an encapsulated data access implementation.

Authorization on data records

Context: The authorization rules say that users from certain roles can read, modify or delete only certain records of some entities. This means that data in the list screens, for example, needs to be filtered based on the access rights of the current user. We need to add this, at a later stage of the project.

In most of the applications when we implement the authorization (what the current user is allowed to do) we restrict some functionalities for some users. For example a user from the Guest role can only see the list of products. She does not have access to Buy functionality. Only users from the Customer role can buy products. Even more, the Edit Product functionality is only available to users from Sales Manager role.

Authorization on data records goes a step further. It says that a Sales Manager user has access to Edit Product functionality, but she can edit ONLY the products that are at sale in the area managed by that user.

To implement this it means that we need to go in all the screens that show products for edit and filter them by some data from the current user. Going back through all these screens and mix this authorization logic with existent queries might be costly and may result in a hard to maintain code.

However, if we have a well encapsulated data access through which all our queries go, it means that we could intercept them and based on some conventions rewrite the Linq to append a WHERE condition that filters the result by some data from the current user. Again, we can rely on the data access to add this functionality later.

I will detail this more and I'll also give some code snippets on how it could be done, in a future post.

To summarize, we have seen some examples of some extra benefits an encapsulated data access brings. In all theses cases we could add technical functionalities at a later stage of the project with costs that do not depend on how much business functionality was already implemented.

All of the implementations rely on the fact that all the queries and the commands go through a central place: the data access implementation. This means that we can either use some extensibility points like the entity interceptors to execute custom code when data is loaded, modified or deleted; or we could rewrite the Linq to enrich it with the functionality we want. If our data access is implemented with a Linq based framework like EF, the resulted code, even if it may be complex, is testable and maintainable. Even more, this complexity remains separated from the business functionalities, somewhere in the infrastructure of our project, rather than being spread all over the code base.

More about how to implement an encapsulated data access and how to benefit from it is discussed in my Code Design Training

Featured image credit: alinoubigh via 123RF Stock Photo

A while ago I have written about the Localization Concern, and one of the topics that I have detailed there was Data Localization.

By Data Localization we mean the cases when we need to store some text properties of some business entities in different languages. For example in an e-commerce app you may need to store the name and the description in more languages for the same product. The name of the mouse product will be Mouse in English, Souris d’ordinateur in French and maybe Computermaus in German.

In that post, I have emphasized why it is wise to separate this translation concern from the rest of the query logic and to build a generic mechanism that based on some conventions does the translation for all the queries. I've also given some code snippets that give the direction on how this can be implemented as part of a generic Repository or an UnitOfWork implementation.

Recently my colleague Catalin Pop has published on our iQuarc Github account a library that does just that. To quote him:

The library provides a set of helper methods for querying localized data split in multiple tables. The library works by rewriting Linq that perform projections on the main table to retrieve data from localization tables when available.

The library is very easy to use and it can be plugged in any Linq based data access framework. You install it via NuGet and you just add the .Localize() extension method after the .Select(), and boom: your data is translated.

var products = dbContext.Products
                 .Select(p => new ProductData
                     {
                         Name = p.Name,
                         Description = p.Description,
                         Quantity = p.Quantity,
                         ...
                      })
                  .Localize()
                  .ToList();

Of course, you need to have the translations tables in the database. Under the hood, the query is rewriten to join with them. You can do this call in your repository implementation as I was suggesting in my older blog post, or you can just call .Localize() from your controllers or services depending on your context.

To make it more usable, the library provides the TranslationForAttribute which can be added on the classes that are mapped through the ORM to the translation tables. In my example this would go on the ProductTrans class that is mapped to Products_Trans table. This means that you don't have to use name based conventions on your tables, because you can use this attribute to tell the library which is the table it needs to join with.

To see more usage examples and how to get started with this you can go to the read.me file on the Github repository, where Catalin has written more details. Also, if you have feedback or questions you can use the Issues on Github and I'm sure Catalin is going to answer or to help you to get stared.

More about implementing localization is addresses in my Code Design Training

Featured image credit: eyematrix via 123RF Stock Photo

Update, 19.03.2018: Daniel Kvist, has made a detailed step by step tutorial on how to setup the Data Localization Library in an .NET Core MVC app, in a 4 part blog series:

• Simple data localisation with .NET core – Part 1 – Set up basic .NET core MVC project
• Simple data localisation with .NET core – Part 2 – Add models, database and migrations with code first
• Simple data localisation with .NET core – Part 3 – Add API Controller to view model data
• Simple data localisation with .NET core – Part 4 – Add data localisation for models using iQuarc.DataLocalization

Even if it has been a while since my last post (I've been very busy lately with giving my training), one piece of feedback that I got back then, stuck into my head:

You can make a better implementation of it. I remember somewhere a better implementation of the Visitor Pattern in C#.

So, last week, having some dead time while flying, I've pulled out my laptop and tried to improve the same sample code I did in the previous post.

In that post we have taken as example a CommandsMananger which holds a structure of commands, and it has to do different operations on these commands, like PrettyPrint() and Approve().

The implementation we have ended up with has a ReportVisitor which builds a report while each command is visited, and at the end it prints the report.

class ReportVisitor : IVisitor
{
	private StringBuilder report = new StringBuilder();

	public void VisitCustomerCommand(CustomerCommand customerCommand)
	{
		report.AppendLine($"VisitCustomerCommand customer command: {customerCommand.Name} in business: {customerCommand.BusinessDomain}");
	}

	public void VisitSalesOrderCommand(SalesOrderCommand salesOrderCommand)
	{
		report.AppendLine("Sales order command: ");
		foreach (var line in salesOrderCommand.OrderLines)
		{
			report.AppendLine($"\t Product={line.Product} Quantity={line.Quantity}");
		}
	}

	public void VisitPurchaseOrderCommand(PurchaseOrderCommand purchaseOrder)
	{
		report.AppendLine($"Purchase order command: Product={purchaseOrder.Product} Quatity={purchaseOrder.Quantity}");
	}

	public void Print()
	{
		Console.WriteLine(report);
	}
}

This is a visitor which holds state (the report), and when each command is visited the state accumulates.

For the Approve() operation we ended up with more visitors. One for each type o the commands. For example the visitor for the PurchaseOrderCommand looked like this:

class PurchaseOrderCommandApprover : IVisitor
{
	public void VisitPurchaseOrderCommand(PurchaseOrderCommand purchaseOrder)
	{
		// code that approves the command of creating a new purchase order.
		// this code may use external classes or services to the approval
	}
	public void VisitCustomerCommand(CustomerCommand customerCommand)
	{	// we do nothing here because we only deal with new purchase orders approval
	}

	public void VisitSalesOrderCommand(SalesOrderCommand salesOrderCommand)
	{	// we do nothing here because we only deal with new purchase orders approval
	}
}

These are visitors without state. When a command is visited it gets approved or not.

The thing which is not nice about this implementation is that we have to have functions for all the command types even if we don't want to do anything with some commands when they are visited. Above, the PurchaseOrderCommandApprover class is only interested on the PurchaseOrderCommand and it will have no code on the functions VisitCustomerCommand() or VisitCustomerCommand().

The most common implementation that I've seen to get rid of the unwanted functions is to have an abstract base class (like in the Template Method pattern) for the visitor and then we just override the methods that we want. In our example it would be like this (here in v4 is the full code):

abstract class Visitor : IVisitor
{
    public virtual void VisitCustomerCommand(NewCustomerCommand customerCommand)
    {
    }

    public virtual void VisitSalesOrderCommand(NewSalesOrderCommand salesOrderCommand)
    {
    }

    public virtual void VisitPurchaseOrderCommand(NewPurchaseOrderCommand purchaseOrder)
    {
    }
}

and for a concrete visitor we would have:

class PurchaseOrderCommandApprover : Visitor
{
	public override void VisitPurchaseOrderCommand(NewPurchaseOrderCommand purchaseOrder)
	{
		// code that approves the command of creating a new purchase order.
		// this code may use external classes or services to the approval
	}
}

Nice! We have small classes for the specific visitors.

This is a simple and a good solution for most of the cases. Especially for those where the types of the items visited do not change. This is the one chosen for the Lambda Expression Tree Visitor in .NET. There, you will inherit from the ExpressionVisitor class and override the function that interests you.

Even if this is simple, it doesn't mean it is the nicest solution. I don't like it the most for two reasons:

1. it uses inheritance, and I have a heavy bias against inheritance
2. in the case that a new type of command needs to be added we will have to add new functions to the IVisitor interface and to the Visitor class.

So for the sake of exercise, lets go forward and see if we can make it better.

There are two directions from where we can get to a better implementation. Both have to do with generics (here in v5 is the full code for this version).

One direction is to start from the visitors implementations. Another way to have only the functions that we need in a visitor class is to make it implement a generic interface:

public interface IVisitor<TElement>
{
    void Visit(TElement element);
}

With this, the visitors get only the functions they want. Even more the resulted objects will have only those functions. The PurchaseOrderCommandApprover is as simple as above:

class PurchaseOrderCommandApprover : IVisitor<NewPurchaseOrderCommand>
{
	public void VisitPurchaseOrderCommand(NewPurchaseOrderCommand purchaseOrder)
	{
		// code that approves the command of creating a new purchase order.
		// this code may use external classes or services to the approval
	}
}

and the Report visitor which is interested in all the commands will implement the generic interfaces for each command type:

class Report : IVisitor<NewCustomerCommand>,
               IVisitor<NewSalesOrderCommand>,
               IVisitor<NewPurchaseOrderCommand>
{
    readonly StringBuilder report = new StringBuilder();

    public void Print()
    {
        Console.WriteLine(report);
    }

    public void Visit(NewCustomerCommand customerCommand)
    {
        report.AppendLine($"New customer request: {customerCommand.Name} in business: {customerCommand.BusinessDomain}");
    }

    public void Visit(NewSalesOrderCommand salesOrderCommand)
    {
        report.AppendLine("Sales order request: ");
        foreach (var line in salesOrderCommand.OrderLines)
        {
            report.AppendLine($"\t - Product={line.Product} Quantity={line.Quantity}");
        }
    }

    public void Visit(NewPurchaseOrderCommand purchaseOrder)
    {
        report.AppendLine($"Purchase order request: Product={purchaseOrder.Product} Quantity={purchaseOrder.Quantity}");
    }
}

The other direction is to start from the IVisitable interface, which on the Accept(IVisitor visitor) function needs a parameter of a non-generic interface. So, the non-generic interface IVisitor is still needed, but it doesn't mean it needs to have one function for each command type. Instead it may have a generic function, like:

public interface IVisitor
{
    void Visit<TElement>(TElement element);
}

Now, if we look again at the interfaces we've obtain we have the non-generic IVisitor above, the generic IVisitor<TElement> below and the IVisitable below:

public interface IVisitor<TElement>
{
    void Visit(TElement element);
}

public interface IVisitable
{
     void Accept(IVisitor visitor);
}

Nice! None of the interfaces have any knowledge of the types of the commands (or more abstract said: the items) which are visited. When a new type of command is needed none of the interfaces, nor its implementation must be changed!

To make all this work one last piece of the puzzle remains: the link between the two IVisitor and IVisitor<TElement> interfaces. This is realized by a general implementation of the IVisitor, which is very simple. It wraps the specific visitor and delegates the visit to it:

sealed class Visitor : IVisitor
{
    private readonly object specificVisitor;

    public Visitor(object specificVisitor)
    {
        this.specificVisitor = specificVisitor;
    }

    public void Visit<TElement>(TElement element)
    {
        IVisitor<TElement> v = visitor as IVisitor<TElement>;
        v?.Visit(element);
    }
}

So, when a IVisitable command is visited, on the Accept() function the above Visitor.Visit<TElement>(TElement element) is called. It casts the specificVisitor to the generic IVisitor<TElement> and if that succeeds (the specific visitor is interested in this element type), it forwards the visit to it.

The client code wires up the visitors, by wrapping the specific visitor into the general one:

public class CommandsManager
{
    private readonly List<IVisitable> items = new List<IVisitable>();

    public void PrettyPrint()
    {
        Report report = new Report(); // the specific visitor
        IVisitor reportVisitor = new Visitor(report); // constructs the general visitor as a wrapper over the specific one.

        foreach (var item in items)
        {
            item.Accept(reportVisitor);
        }

        report.Print();
    }
....
}

And voilà! Everything works nicely! (the full code for this implementation is here in v5, where you can also run the demo and see how everything works together).

I find this implementation to be nicer than the previous ones. It is very flexible, we can easily adapt it to changes and if we'd use a smart Dependency Injection Container the wire-up of the general visitor and the specific visitor could be more elegant.

A small variation of it is to make the link between the two IVisitor interfaces visible in the interfaces themselves. We can do it by adding the AsVisitor() function like this:

public interface IVisitor<TElement>
{
     IVisitor AsVisitor();

     void Visit(TElement element);
}

Another advantage of this variation is that it makes the client code simpler, because it does not need to know nor to wire the two IVisitor interfaces:

public class CommandsManager
{
    private readonly List<IVisitable> items = new List<IVisitable>();

    public CommandsManager()
    {
        this.items.AddRange(DemoData.GetItems());
    }

    public void PrettyPrint()
    {
        ReportVisitor reportVisitor = new ReportVisitor(); //we just construct the visitor

        foreach (var item in items)
        {
            item.Accept(reportVisitor.AsVisitor()); // we pass the general visitor
        }

        reportVisitor.Print();
    }
...
}

On the other hand it makes the visitor implementations a bit more complicated, because now the wiring happens here:

class ReportVisitor :   IVisitor<NewCustomerCommand>,
                        IVisitor<NewSalesOrderCommand>,
                        IVisitor<NewPurchaseOrderCommand>
{
    readonly StringBuilder report = new StringBuilder();
    private readonly IVisitor visitor;

    public ReportVisitor()
    {
        visitor = new Visitor(this);
    }

    public IVisitor AsVisitor()
    {
        return visitor;
    }
...
}

To sum it up, we have started from the "by the book" implementation of the Visitor Pattern that we've done in the previous post, and we've tried to improve it gradually. We have reached to a flexible implementation that uses generics and which does not hard code the types of the items being visited. I think this is one of the greatest differences. This opens the possibility to embed a generic Visitor Pattern implementation somewhere in the infrastructure of a project, but... more about this in a future post.

The entire source code with all the versions is available on Github as part of my Design Patterns Explained course, structured in a folder for each version:

• v3 - is the state where we left off in the previous post. A "by the book" implementation
• v4 - uses a base class to let specific visitors override only the methods they are interested in
• v5 - is a better implementation based on generics
• v6 - is a small variation of v5, by adding an explicit link between the IVisitor interfaces

Many more patterns are explained with examples in my Design Patterns Explained course

Featured image source: IMDb - Vikings

The Gang of Four book says:

Represents an operation to be performed on the elements of an object structure.

Visitor lets you define a new operation without changing the classes of the elements on which it operates.

Go back and read the last sentence once more. Quite an ambitions desire to add operations without modifying the classes, isn't it?

Lets see what this actually means. I like to explain this pattern using the following example:

Say we have some classes that represent commands:

class PurchaseOrderCommand
{
	public Product Product { get; set;}
	public int Quantity { get; set;}
}

class SalesOrderCommand
{
	public IEnumerable<OrderLine> OrderLines { get; set;}
	public string CustomerCode { get; set; }
	public DateTime Date { get; set; }
}

class CustomerCommand
{
	public string Name { get; set; }
	public string BusinessDomain { get; set; }
}

and some client code which keeps these in a data structure, a List lets say:

public class CommandsManager // refered as the Client code
{
	List commands = new List();

	// The client class has a structure (a list in this case) of the items (commands).
	// The client knows how to iterate through the structure
	// The client would need to do different operations on the items from the structure when iterating it
}

(List is one of the simplest data structure we use. Even if the Visitor Pattern addresses the cases when the data structure is complex and difficult to iterate, for the simplicity of this example, I use only a list.)

Now, lets add some operations with these items. Say we want to add:

• PrettyPrint() - a function that will produce a nice report with all the commands which are pending
• Approve() - a function that will approve each command for execution
• Save() - a function that will persist all the commands, so we don't loose them when the application is restated

These operations belong to different areas of concern. The PrettyPrint() would have presentation concerns, the Approve() would have business logic concerns and the Save() would have data access concerns.

One way to do these operations on all the items we have, is to add these functions to the client code, the CommandsManager class in our example. The result would be:

and the code would be:

public class CommandsManager
{
	readonly List<object> items = new List<object>();

	public void PrettyPrint()
	{
		foreach (var item in items)
		{
			if (item is PurchaseOrderCommand)
			   Print((PurchaseOrderCommand)item);
			else if (item is SalesOrderCommand)
				Print((SalesOrderCommand)item);
			else if (item is CustomerCommand)
				Print((CustomerCommand)item);
		}
	}

	private void Print(PurchaseOrderCommand item)
	{
		Console.WriteLine($"Purchase order command: Product={item.Product} Quatity={item.Quantity}");
    }

	private void Print(SalesOrderCommand item)
	{
		Console.WriteLine("Sales order command: ");
		foreach (var line in item.OrderLines)
		{
			Console.WriteLine($"\t Product={line.Product} Quantity={line.Quantity}");
		}
	}

	private void Print(CustomerCommand item)
	{
		Console.WriteLine($"New customer command: {item.Name} in business: {item.BusinessDomain}");
	}

	public void ApproveAll()
	{
		foreach (var item in items)
		{
			if (item is PurchaseOrderCommand)
				Approve((PurchaseOrderCommand)item);
			else if (item is SalesOrderCommand)
				Approve((SalesOrderCommand)item);
			else if (item is CustomerCommand)
				Approve((CustomerCommand)item);
		}
	}

	private void Approve(CustomerCommand item)
	{
		// Interact w/ the databse and use external services to process a new purchase order command
	}

	private void Approve(SalesOrderCommand item)
	{
		// Interact w/ the databse and use external services to process a new purchase order command
	}

	private void Approve(PurchaseOrderCommand item)
	{
		// Interact w/ the databse and use external services to process a new purchase order command
	}

	public void Save()
	{
		// This might mix DA concerns w/ UI concerns
	}
}

This approach is not a good solution in most of the contexts. The separation of concerns is poor and the costs of change will be high. Each time a new type of command will appear we need to change this client class. Even more, changes in the presentation logic may affect the Approve() or the Save() code, because all these are implemented in same class for all our commands.

Another way is to add these operations to each of the command classes. This would look like:

and the code like:

public class CommandsManager
{
	readonly List<ICommand> commands = new List<ICommand>();

	public void ApproveAll()
	{
		foreach (var item in commands)
		{
			item.Approve();
		}
	}

	public void PrettyPrint()
	{
		foreach (var item in commands)
		{
			item.PrettyPrint();
		}
	}
}

class PurchaseOrderCommand : ICommand
{
	public void Approve()
	{
		// Interact w/ the databse and use external services to process a new purchase order command
	}

	public void PrettyPrint()
	{
		Console.WriteLine($"Purchase order command: Product={Product} Quatity={Quantity}");
	}

	public Product Product { get; }
	public int Quantity { get;  }
}
...

Here the code that iterates through the data structure (the list of commands) is isolated from the code that implements the commands. This is an advantage from the previous approach. However, now each time we need to add a new operation we will need to change all the existent command classes to add that new operation. This may induce a high cost of change for two reasons: first, we still have a poor separation of concerns (one command class has presentation code, mixed with business logic code and mixed with data access code); second, changing the interface of these classes will trigger changes in all the other classes that use them.

So, can we come with a better design than these two approaches? Yes, if we apply the Visitor Pattern.

We apply it by evolving our previous designs. We define two interfaces: IVisitable and IVisitor. The class diagram is now like this:

Lets go through the code as and see how they interact.

The IVisitable interface is implemented by all the items in the data structure (by all the command classes in our example). The operations we wanted are not functions of this interface, because the operations may vary. This interface has only one function that will accept a visitor.

public interface IVisitable
{
	void Accept(IVisitor visitor);
}

By implementing this, each item from the data structure, will pass itself to the visitor and let the visitor to implement whatever operation it wants on it.

The implementation is trivial. It just calls the appropriate VisitXXX() method of the visitor.

public class PurchaseOrderCommand : IVisitable
{
	public void Accept(IVisitor visitor)
	{
		visitor.VisitPurchaseOrderCommand(this);
	}
	...
}

public class SalesOrderCommand : IVisitable
{
	public void Accept(IVisitor visitor)
	{
		visitor.VisitSalesOrderCommand(this);
	}
	...
}

public class CustomerCommand : IVisitable
{
	public void Accept(IVisitor visitor)
	{
		visitor.VisitCustomerCommand(this);
	}
	...
}

The client code (CommandsManager) iterates through the data structure, and for each item it calls the Accept(IVisitor) method, passing the visitor that visits that item.

public class CommandsManager
{
	private readonly List<IVisitable> items = new List<IVisitable>();

	public void PrettyPrint()
	{
		ReportVisitor report = new ReportVisitor();
		foreach (var item in items)
		{
			item.Accept(report);
		}

		report.Print();
	}
        ...
}

Now, the client code does one thing only: it iterates through the data structure. The implementation of any operation on the data structure or on its elements is forwarded to the visitor classes.

The visitors are classes that focus on the behavior only and on one operation only. They implement the operations we want. When we need a new operation we will create a new visitor and so on. For example the ReportVisitor creates a printable report with all the commands. It is used to implement the PrettyPrint() operation.

class ReportVisitor : IVisitor
{
	public void VisitCustomerCommand(CustomerCommand customerCommand)
	{
		report.AppendLine($"VisitCustomerCommand customer command: {customerCommand.Name} in business: {customerCommand.BusinessDomain}");
	}

	public void VisitSalesOrderCommand(SalesOrderCommand salesOrderCommand)
	{
		report.AppendLine("Sales order command: ");
		foreach (var line in salesOrderCommand.OrderLines)
		{
			report.AppendLine($"\t Product={line.Product} Quantity={line.Quantity}");
		}
	}

	public void VisitPurchaseOrderCommand(PurchaseOrderCommand purchaseOrder)
	{
		report.AppendLine($"Purchase order command: Product={purchaseOrder.Product} Quatity={purchaseOrder.Quantity}");
	}

	public void Print()
	{
		Console.WriteLine(report);
	}
}

We can design the visitors as we want. We may have one visitor that does one operation (the printing) for all the items (commands) as the above, or we can have one visitor for one operation for one type of item, as in the following example.

class PurchaseOrderCommandApprover : IVisitor
{
	public void VisitPurchaseOrderCommand(PurchaseOrderCommand purchaseOrder)
	{
		// code that approves the command of creating a new purchase order.
		// this code may use external classes or services to the approval
	}
	public void VisitCustomerCommand(CustomerCommand customerCommand)
	{	// we do nothing here because we only deal with new purchase orders approval
	}

	public void VisitSalesOrderCommand(SalesOrderCommand salesOrderCommand)
	{	// we do nothing here because we only deal with new purchase orders approval
	}
}

class CustomerCommandApprover : IVisitor
{
	private ICrmService crmService;
	public CustomerCommandApprover(ICrmService crmService)
	{
		this.crmService = crmService;
	}

	public void VisitCustomerCommand(CustomerCommand customerCommand)
	{
		// code that approves the command of creating a new customer.
		// uses the ICrmService to do the approval
	}

	public void VisitSalesOrderCommand(SalesOrderCommand salesOrderCommand)
	{	// we do nothing here because we only deal with new purchase orders approval
	}

	public void VisitPurchaseOrderCommand(PurchaseOrderCommand purchaseOrder)
	{	// we do nothing here because we only deal with new purchase orders approval
	}
}

The way we design the visitors depends on the operation we implement. For the ReportVisitor we wanted a report with all the types of the commands, and it made sense to have code when any element from the structure was visited. For the Approve() operation is the other way around. Here, because we have a very different logic for approving a purchase order command from approving a new customer command, we need a better separation. We created one visitor class for the approval of each element type. It only has code on the VisitXXX() method that corresponds to the element type it is interested in. These classes may have different dependencies the CustomerCommandApprover needs a the ICrmService and the others will not. Also, they will change and evolve separately, so it makes sense to make this kind of separation.

By applying the Visitor Pattern we have achieved a design with a good separation of concerns. Now the client code that iterates the data structure (CommandManager), is separated from the implementation of the operations. The items (the command classes) only hold their specific data and do not have behavior. The operations are implemented by the visitors and we can easily define and add new of them.

The Visitor Pattern brings most value when applied in the contexts where the data structure is complex (a tree or a graph), the types of items it holds (the nodes) are quite fixed and the operations we want on those items vary a lot. Good examples of these are the syntax trees, which are source code representation in a tree data structure. Here the programming language grammar is fixed, which means that the nodes in the tree are fixed. After the syntax tree is created by parsing the code, then we can define different visitors which will visit each node and we can add any operation on the code. This is useful for doing static code analyses, generating code, printing code etc.

A good example of Visitor implementation in .NET Framework is the ExpressionVisitor class. We can inherit from it to implement different operations on a lambda expression. I have used it many times to parse LINQ queries to alter them before they are sent further.

If we go back to our initial example we can make another important observation: the Visitor Pattern helps us to follow the principle that says "Separate Data from Behavior". A principle that I've first read in the Clean Code book. We have started with our command classes which were only data. Data that represents a command. First we wanted to keep them clean of the behavior and we added the operations in the client code, resulting a switch on the object type. Not an OOP design. Then we moved the behavior into the command classes. This was not separating it from data, and we've seen the drawbacks. In the end by applying the Visitor Pattern we have managed to get to a good separation and to get a reduced cost of change as pursued by the Separate Data from Behavior principle. I won't go in more details about this principle here, even if it would turn into an interesting discussion. I'll do it in a future post.

I believe that the Visitor Pattern is not too complex and if used in the correct context it can lead to a better design, a design that embraces change. I hope that the example that I've described here gives an useful addition to all the other good writings on this pattern. The entire source code, including a running demo is available on Github here as part of my Design Patterns Explained course.

Many more patterns are explained with examples in my Design Patterns Explained course

Featured image credit: FWSAM via 123RF Stock Photo

Migrating my blog from Wordpress.com to a self hosted blog using Ghost, was a lot of work. More then I have anticipated. If I look at my Trello board where I keep track of it, I see that I've started to work on this on January the 20th and I am not yet done. It is up, it looks good, but there are still some items on my TODO list. It takes me that long for three main reasons:

• it is a lot of work
• I work on it only on weekends or at night (I'm going through a very busy period with the real work), and
• I'm a very pretentious with the result (I can spend hours on a small detail until it looks / works as I want).

But it's worthy! I'm very happy with the result. I like how it looks and how it works and I've achieved everything that I wanted when I've started this. However, there are a few things that I miss from Wordpress (I'll get to them below).

In all of this work I was guided by the experience of others who went through a similar process. I've learned by reading many blogs, like Troy's Hunt, who not only that contributed in making my choice of going with Ghost, but he was also migrating his own blog at the same time with me :)

@florincoros nice one!

— Troy Hunt (@troyhunt) April 19, 2016

So, I've decided to take some time and detail how I did it, maybe it'll be useful to other who consider such a migration. Troy did the same here, and even if I've learned a lot from him and there are similarities, we had different contexts so there are also differences.

It is going to be a long post, but I have made below a list with all the sections, so you can easily jump up and down to what might be useful for you.

Sections

• Choosing the Platform
• Migrate the Content
• The Theme
• Hosting
• Code Syntax Highlighting
• Redirect from Wordpress
• HTTPS
• What I Miss from Wordpress
• TODOs

Choosing the Platform

The goal wasn't Ghost. As I said in the post that announced the new look I wanted to get away from a free Wordpress.com site, with a free theme and to give the blog a more professional look. The options I had in mind were:

• build or buy a theme for Wordpress and migrate to a professional Wordpress host provider
• Ghost self host or Ghost Pro
• Github Pages

I was biased against Wordpress, because I had a lot of pain with it. It always felt like it is too complex and rigid for what I wanted. I was still considering it thinking that in a payed service it should be better. My biggest pain with it was with writing the posts. Some of my posts have a lot of code, and pasting the code from Visual Studio and then formatting it too look good in the post was a cumbersome process. I was waisting a lot of time with that.

Ghost was attractive because of the Markdown. I can paste code from Visual Studio in the editor and it keeps the formatting. No more inserting spaces and break lines manually. Plus I can easily mark a code structure using the code inline construct like this.

I've started by reading how other technical bloggers feel about it. I've read the following posts (at least):

• Michele Bustamante: i. love. ghost.
• Troy Hunt: Creating a blog for your non-techie significant other; the path to Ghost
• Scott Hanselman: How to install the nodejs Ghost blogging software on Azure Websites
• Ryan Hayes: Ghost VS WordPress (and Why I Migrated Back to WordPress)
  

After reading these I was pretty much convinced. I know I should have given a better chance to Wordpress or Github Pages, make a more informed decision, but... this felt right. Wordpress failed me once and Github Pages feels more focused on building sites for projects, rather then blogs.

Next to make a final call on this I needed to get some info about:

• what it means to migrate the content from Wordpress to Ghost
• how hard is to build / buy / make a theme for Ghost

Migrate the Content

go up

The posts content is the most valuable thing you want to migrate. This is in fact the point of the migration, otherwise we would talk about a new blog. For me it was also the most time consuming thing.

Before making the final decision of going with Ghost, I've started to search and read about the migration to it. Among others, I've read these:

• Ghost for Beginners: Migrating from WordPress to Ghost
• All About Ghost: Moving your Blog from WordPress To Ghost

Seams not only doable, but quite easy. Lets see how it went.

I've had 31 posts and a few pages to migrate. Not too many.

I could not just move over the HTML of the old posts in Ghost markdown, because it was an ugly HTML. It had inline CSS, because in Wordpress I was using a free theme that was not offering much, so I was colouring some text headers in the editor and that resulted in HTML mixed with CSS, like this:

<span style="color:#50b4c8;">Part 1: reviewing few concepts</span>

This colour wouldn't look right with the new blog design.

Another blocker was the code snippets. The code would have been unreadable elsewhere than in that particular Wordpress theme.

So I've used the Wordpress Ghost Plugin to migrate the content as explained in above articles. But it wasn't straight forward. First you cannot install plugins on a free Wordpress.com site. Secondly, the images are not migrated by the plugin.

I've followed the instructions from this post by Hugh Rundle, and I have used a self-hosted Wordpress install as an intermediate step, as follows:

1. I've installed a fresh Wordpress on Azure. I found in Azure Marketplace a Wordpress + Mysql machine, published by Docker, which I deployed easily as a standalone VM (any other Wordpress installation will do).
2. I've exported the content from Wordpress.com using this guide and then I've imported it on the installation from Azure.

At this point I had my content on the self-hosted Wordpress on Azure. On this one I could install plugins.

The images are still on the Wordpress.com site, so when I open a post from the self-hosted site, the images are downloaded from florincoros.wordpress.com. If now I'd migrate the posts to Ghost the images remain on florincoros.wordpress.com. I didn't like this. I can't download them from Wordpress.com to upload them on Ghost, because Wordpress does not offer such an export and even if it would, all the URLs would need to be changed and I'd be limited to a self-hosted Ghost.

What I did was to use Cloudinay. Cloudinary is an image management solution in the cloud. You can upload your images there and then you can easily use them anywhere with some neat image manipulation features based on the URL format. They also have an useful Wordpress plugin.

Having the blog copied on the self-hosted Wordpress on Azure, I did the following steps to migrate to Ghost:

1. Install the Cloudinary plugin for Wordpress and upload all my images on Cloudinary. The plugin also modified all the URLs from my posts, and now all posts take the the images from Cloudinary and not from florincoros.wordpress.com.

• Install the Wordpress Ghost Plugin and use it to export the content in Ghost format
• Import the content on my new Ghost installation

After this I've had the bulk of my content available on the new blog. There were still a few things to adjust:

• The code snippets
• Wordpress uses [code lang=”csharp”] ... [/code] to mark a code block. This is what I got in the post text :( . It doesn't look like code.
• some snippets had poor formatting due to the spaces I've manually added with Wordpress editor
• Feature images
• Because I had a poor free theme on Wordpress, for each post I have inserted its image as content in the post and also selected the same image as the featured image of the post. Now in Ghost, each post gets the image twice :(
• Images URL
• Cloudinary plugin did a good job for most of the cases, but there are few images who point to an IP address. I guess somewhere along the import/export plugins there was a bug that caused this (that IP address is the one that Azure has given to my Wordpress VM).
• all the URLs were rewritten with HTTP not with HTTPS. I am not sure why. Maybe I've missed a setting in plugin, maybe because they may have been HTTP on the original Wordpress too.
• Words in italics
• in the Wordpress editor it often happened that when I wanted one word from a phrase to be in italics to also make the following space char in italics. When this is converted to markdown we get a space between the word and the *. The result looks like this: *word in italics **next word, instead of word in italics next word.

All of these I had to fix manually by editing each post. Here is the time consuming part. I get a lot of speed by using an smart editor like Visual Studio Code, where I can find & replace all [code lang=”csharp”] with ````language-csharp` or I can use the multi-cursor or other neat features for fast editing to format the code. However, its still a lot of manual work. Maybe if I had hundreds of posts I would have built a tool that parses the json the Wordpress Ghost Plugin generates, and fix all of the above there before import it in Ghost.

The Theme

go up

I knew from the beginning that I can't build or customise a theme. I don't have web design skills and I'm not good with HTML or CSS. So, even before deciding to go with Ghost, I've asked my good friends at Dali Media if they have a preference of building a my blog theme for Wordpress or Ghost. Even if they have a lot of experience with building Wordpress sites, I got the feeling that it wasn't all that pleasant. They preferred Ghost event though they didn't know it.

For me the theme was a payed service. I've sit down with them, explained what I'd want and in a few days, after a few iterations, I've got it. I'm very happy with the result, it looks good on the phone, on the tablet or on the desktop, and everything I wanted was doable quite fast. I don't know all the details, but I can say that if you are good with HTML, CSS and JavaScript you can figure out quite fast how Ghost works and how to build or customise a theme.

I know that they had to do some custom work to have the dropdown in the menu I wanted for the Training entry, because Ghost does not support this yet. The drawback for me is that I cannot maintain it from the /ghost/settings/navigation/ and I'll need to change the theme files when I want something changed in there. Another thing hardcoded in the theme is the bar I have on the right. This also needs to be maintained from the theme's code and not from the editor. The Recent posts box uses the API Ghost offers, so that is not hardcoded, but if I'd want to take out this box, I'd need to change the template for the right bar. I don't mind these. I will change them rarely and the code is clean. I've put it in git, so I have history and from there I can easily deploy updates to these parts.

The entire work for the theme was only in its folder, so with what I've got from the guys at Dali Media I can go either to a self-host, either to Ghost Pro.

Having the content migration figured out, and a theme for Ghost built the next thing was to decide on hosting.

Hosting

go up

From the very beginning, I wanted to go with a SaaS offer. After I've decided to go with Ghost, I was convinced to go with Ghost Pro offer. So the entire work on the migration and on the theme went on this assumption. I was using an installation on Azure, but only for development and testing purposes.

When I've reached the Migrate to GhostPro item in my TODO list, this changed. The price was 29$ / month. I was remembering it to be a lot cheaper when I've first read about it on Troy's blog. I've searched a bit and indeed the price grew from 5$ / month to 8$ / month, then to 10$ / month and now to 29$ / month for the smallest plan. Troy makes in his post a very compelling argument in GhostPro favour, but 228$/year seems a bit too much for hosting to me. Adding to this that their price grew more then tree times in about a year... I decided to wait a bit before I go on Ghost Pro. So, I've turned my Azure installation in the production one, at least for now. The first time I'll feel that the extra work that I need to do to manage the self-hosted installation on Azure is significant, I'll move to GhostPro.

On Azure, I have easily installed it as an App Service on the free plan, using this Github repository from Felix Rieseberg. I have also made an upgrade of Ghost to see how complicated that is, before making the final call with the self-hosted installation. I have used this upgrade tool from the same Felix Rieseberg. Again it worked with a simple button hit. So, it seems that self host is not going to be that difficult. I get the PaaS from the Azure, so I don't need to think about the OS, web server etc. and I just need to upgrade Ghost.

For developing and testing the free plan was more then enough. When I've decided to use it for production I've upgraded to the basic plan, because I wanted a better service. I've used features like AlwaysOn, Custom Domain, etc. This makes sense, money wise, only because I have enough credits in my Azure subscription. A B2 costs 55.21EUR a month, which is a lot more than GhostPro. Maybe the shared plan (D1), which is 8.16EUR a month would also be enough.

To setup my domain I have followed this guide. After everything was done, I also had to setup the domain name in the Azure portal in the Application settings, so Ghost knows it.

With this done, I had my site migrated and working at http://oncodedesign.com. At this point it wasn't HTTPS, yet. I was using https://oncodedesign.azurewebsites.net/ghost to login and access the admin page, but the rest was HTTP. I wanted to get to a version that I could show to the world, before fixing this.

Code Syntax Highlighting

go up

For code syntax highlighting I have used PrismJS. I appreciate the most two key aspects of it:

• very easy to install - I've just followed the steps here and I was done in a few minutes
• highly extensible - The CSS is very clean. Even if I'm not good at CSS I could easily change the colours of some C# tokens that did not fit well in my theme design

Having the code syntax highlighting done, the theme installed and configured, next I have spent some time to adjust the menu, the content of the pages it links and to fix the content of most of the posts after the migration. With this done, the first version of my blog on Ghost at oncodedesign.com was ready to be shown to the world. To make this happen, the next step was to redirect the requests from florincoros.wordpress.com to the new address.

Redirect from Wordpress

go up

The hole idea here is that when someone uses a link to a page from my old blog, it works. She should be redirected to that page on the new blog. This is useful for all the cases when links are saved in bookmarks, or referenced in other sites or on twitter, facebook, etc., and most important for SEO. I want all the Google / Bing ranking that my posts and pages made on the old blog to be used by the new one as well. So all the redirects need to be 301.

The first step here is to make the florincoros.wordpress.com/<some_url> to redirect to oncodedesing.com/<some_url>. This is easy to achieve with some money :) Wordpress.com offers this as a service for 13$/year. To configure it I just had to follow the instructions here.

The next step was to rewrite the Wordpress URL into Ghost URL. By default an Wordpress URL is like: florincoros.wordpress.com/2015/05/06/the-post-title. Now this gets redirected to oncodedesign.com/2015/05/06/the-post-title and the request fails because in Ghost this post is at oncodedesign.com/the-post-title. So, to fix it I have followed the instructions from this blog post, only that I've used the IIS URL rewrite module and specified the rewrite rules in the web.config of the Azure Web App.

Another thing that I did, here was to redirect the www. subdomain to naked domain, meaning that a www.oncodedesign.com is redirected to oncodedesing.com. Here Ryan Hayes gives a good explanation on why this is important and how to do it.

In the end my redirect rules look like this, in the web.config:

<system.webServer>
    ...
    <rewrite>
      <rules>
               
        <rule name="Redirect to non-www" stopProcessing="true">
            <match url="(.*)"></match>
            <conditions>
                <add input="{HTTP_HOST}" pattern="^oncodedesign\.com$" negate="true"></add>
            </conditions>
            <action type="Redirect" url="https://oncodedesign.com/{R:1}"></action>            
        </rule>
                             
        <rule name="Redirect Wordpress posts" stopProcessing="true">
            <match url="\d{4}\/\d{2}\/\d{2}\/(.*)$"></match>
            <action type="Redirect" url="{R:1}"></action>            
        </rule>    
        
        <rule name="Redirect Wordpress training pages" stopProcessing="true">
            <match url="training\/(.*)-training"></match>
            <action type="Redirect" url="training-{R:1}"></action>            
        </rule>                                                
                             
        <rule name="StaticContent">
          <action type="Rewrite" url="public{REQUEST_URI}"/>
        </rule>
        <rule name="DynamicContent">
          <conditions>
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="True"/>
          </conditions>
          <action type="Rewrite" url="index.js"/>
        </rule>
      </rules>
    </rewrite>
  </system.webServer>

The rules that I've added are: Redirect to non-www, Redirect Wordpress posts and Redirect Wordpress training pages. It is needed that these rules are before the existent ones (StaticContent and DynamicContent), which are needed by Node / Ghost. It gave me some headaches to figure this out...

Visual Studio Online "Monaco" was of great help to edit the web.config and test the redirects directly on Azure.

If I were on Ghost Pro, I couldn't have written the redirects myself. They also don't yet have an admin console where you could configure them. However, from what I've read, if you email the support with the redirects you want they configure them for you.

With this done, my blog was migrated. From this point on I could publish new posts and add new pages. It was the moment I have announced the new look.

HTTPS

go up

The last thing I did was to set everything to go through HTTPS. For this I rely entirely on CloudFare. Troy Hunt describes it very nicely here and I have just followed his steps. Indeed it only took a few minutes to setup and leverage all the benefits. Besides the HTTPS, and the potential performance gain, I like the most the email obfuscation feature.

What I Miss from Wordpress

go up

Even if I am very happy with Ghost and I think it was a good call to leave Wordpress. However, there are a few features that I am missing:

1. review functionality

I value a lot the reviews I get for my writings, any of my writings: code, articles, documents, e-mails etc. All the posts or pages that I published on this blog are reviewed at least by one person. I am in luck that my girlfriend is also a developer, she reviews most of my posts.

Wordpress has a review functionality which I used for this. It is not much, but you could ask for a review by e-mail, which was convenient. Now, I either make an account on my blog for all people that I want to ask for a review, or... send the article by email.

2. social networks integration

I want to tweet and post on Facebook, LinkedIn, Google+ and the others when I publish a new post.

Wordpress has many plugins for social networks, which give this out-of-the-box. Now, I am posting by hand, and I am planning to setup some IFFFT recipes to automate it.

I also like to have my tweets on the blog. I had this easily setup on Wordpress. I don't have yet any idea on how to put this on the new blog.

3. comments

I love to get feedback or questions on my posts. Ghost does not have this out of the box. I plan to integrate Disqus for this. Until then please use my contact info when you want to send me feedback, questions, ideas or thoughts.

TODOs

go up

At this point my new blog is ready. I can easily publish new posts and everything should work well. However, there are still a few items on my Checklist in my Migrate the blog at oncodedesign.com Trello card:

None of these are critical. I think most are nice to have. I will try to do them when a find a few hours here and there. However, I would like to spend the little time I have for my blog to write new technical articles, so it may take a while until I will move this card to the Done column.

This was pretty much the work that I did to redo my blog. I hope you like the result. I like it for sure, and excepting the manual fix of the content, I also enjoyed working on it and learning new things on the way.

If you find yourself doing such a migration and you get stuck don't hesitate to ask me, I might have gone through the same.

It is already more than one year and a half since I started to give my Code Design training. I have developed this training as I say here, out of the desire to teach others the way I link the learnings from the best practices books to the code I write and how this helps to write a better code in general, a code that is inexpensive to change.

Along these months I've had the opportunity to give the training to various audiences in different companies. There were cases when the audience was with only really experienced programmers, cases with mixed levels of experience and also with only juniors. Mainly they were .NET or Java developers, but there were also C++, ObjectiveC, JavaScript or Android devs among them. The training, even if it was the same material, had a different dynamic depending on the audience and I enjoyed this a lot. My greatest satisfaction is when I see that "Aha" moment. It may be like "Aha, now I finally get how to apply that into my project", or "Aha, I see now when to apply this technique or to stress on this matter", or "Ah... an interesting solution, I'll try it out" or "Aha, you've just confirmed that I'm not (the only one) crazy". To my satisfaction, after each instance of the training I have got very good and positive feedback. This is also reflected by the testimonials I collect on the training page.

Now, is the time to act on the feedback I've collected and to try to improve it even more. I have spent some time in the last few weeks to reflect and to work on this. I wanted to target the most common improvement suggestions that I've got. Thinks like: "More code would have been nice", "More exercises might have helped", or "More time on the Cross-Cutting concerns rather then on Design Patters".

I think that most of these came either because there were cases when we tried to cover too much material in the allocated time or because in some cases it wasn't very clear what to expect from a lesson or the other. When there is too much material planned, it always feels like there wasn't enough time for discussing over code or exercising. Also, from a lesson titled "Design Patterns" it is quite clear what to expect, which may not be the same for "Separation of Concerns" or for "From Principles and Patterns to Practices". Sometimes it is a pity to allocate a full day to something classic like Design Patterns and maybe not enough to the others, when the others may bring more value since are practices that I've applied and collected from the projects I did along the years.

Many of the above can be addressed through a deeper discussion when planning the training for a specific audience, but I think that restructuring the course helps even more, so here is what I did.

I have taken out from the main course a few lessons and spawned new courses for them. This gives more time to the main course and also makes the courses description more focused, more precise, so it is easier to explain what to expect. The main course remains the "Code Design Practices" of four days, but now there is much more time for coding and exercising. Now, we could start implementing a demo application from day one, and exercise on it each day as we advance through the material. If we can plan the training over four weeks, one day each week, than we can have a very nice learning experience by giving people time to reflect from one course session to the other.

From the lessons that I've taken out I have now the "Design Patterns Explained", the "SOLID Principles Insights" and the "Managing Dependencies with Dependency Injection" courses. They are all for one day, which gives enough time for discussions, debates and exercises. The Dependency Injection, also remains as a lesson in the main course, but because there are teams who maybe can spent only one day and would benefit a lot from understanding and using Dependency Injection and Service Locator I also present it as a stand alone training.

I have updated the outline for all the courses, so if you go here you can see more exactly what changes.

In the end I'd like to thank to all who attended my courses and helped me to improve them through the discussions we had, through the suggestions they gave me and through the feedback and testimonials they have submitted. Thank you!

Featured image credit: olegdudko via 123RF Stock Photo

I've been quiet for a while. I didn't publish a new post in a few weeks. It wasn't because I got lazy or because I ran out of ideas or experiences worth sharing. I was really busy.

As you probably know I've quit my job last year and since then I'm working as an independent. It couldn't have been better so far! I've got involved interesting projects, I'm working with smart people and I enjoy it a lot. But... as an independent the workload may vary a LOT, especially at the beginning. So, I've been through some very busy weeks.

The Blog

One of the things I've been busy with, was this: to give my blog a new look. I like writing here my ideas and my experiences. I also find it imperative in our job to always be up to date with the technology and part of this is done through reading blogs. So, I want to continue to give something back by writing about my work and by sharing my ideas here. Beside this, I enjoy it a lot when I hear that some of my posts are mentioned and debated in various places. All these motivate me not only to continue writing, but also to invest in making the overall blog better.

The first thing I wanted to do was to give it a proper name and to move it from a free Wordpress account. I've settled at On Code Design (oncodedesign.com was also available :) ). I wanted a name that sets the theme of the blog. I'll continue to focus on practical ideas rooted in real live projects, about how to structure the code to maximize the chances of success. So I think that the new name fits this theme well. Until now, I've had a lot of influence from building software in the large organizations, and now since I'm very much anchored in the start-ups challenges this may be completed with another perspective. In the future, I'd also like to have guest posts on code design from colleagues or friends whom share same values with me on software quality and good practices, and who can bring different views in this space.

The next thing, after getting rid of the ads that came with the free account, was to give it a more modern look. Because web design isn't my thing I've asked the help of professionals. Together with my very good friends at Dali Media we've came up with this theme, and as it happens when designers get involved, with a logo :).

I've also decided to migrate from Wordpress to Ghost. I didn't do a research or comparison in depth, but after reading about some other tech bloggers experiences, Ghost seems a good alternative for getting away from Wordpress issues and complexity. At first I wanted to go with Ghost Pro and take the SaaS advantages, but after I've seen that their price grew from 8$ / month (when I've first read about it on Troy's Hunt blog) to 26$ / month in just a few months, I've decided to host it myself on an Azure website.

...And, after some weeks of hard work on it, this is the end result. I hope you'll like it! (and that Wordpress redirects from my old blog work well :) ).

MIRA

Refreshing my blog was't the only thing that kept me busy from writing posts in the last weeks. Another big thing that happened was our (iQuarc) partnership with MIRA. We've worked with the MIRA co-founders for a few months last year, and we've easily concluded that a partnership between MIRA and iQuarc would be highly beneficial for all parties, so it happened. Therefore, in the past weeks we all did a hard work to re-architect and implement their core product preparing it for the cloud and for being sold at a large scale. I think more writings inspired from this product will come in the future, because it has some nice technical challenges and I enjoy a lot working with the guys at MIRA.

Training

Another thing that kept me busy was restructuring my Code Design training. I wanted to put in all the good feedback that I've collected along the years. More about this in a new post. (I promise not long from now :) )

My talk at Microsoft Summit created a nice discussion with some of the participants about writing isolated unit tests when using the Service Locator.

It started from the part where I was showing how the AppBoot helps in dependencies management by enforcing consistency on how the Dependency Injection is done. With the AppBoot we make sure that Dependency Injection is used and that it is done only through the constructor. The question that started the discussion was:

Does this mean that I will have constructor parameters for everything? Including utility services like ILog? If so, it means that I will pollute the constructors with these details and…. things may get over complicated

My answer was that for logging or other similar utilities we could make static helpers that make them easier to be called. Such a helper would wrap a ServiceLocator, so we do not make a strong dependency on the logging implementation or library. Something like this:

public static class Logger  
{ 
	public static void Error(string headline, string message)  
	{  
	    ILogSrv log = ServiceLocator.Current.GetInstance<ILogSrv>();  
	    log.WriteTrace(new Trace(headline, message, Severity.Error));  
	}
	public static void Warning(string headline, string message)  
	{  
	    ILogSrv log = ServiceLocator.Current.GetInstance<ILogSrv>();  
	    …  
	}
	public static void Trace(string functionName, string message)  
	{  
	    ILogSrv log = ServiceLocator.Current.GetInstance<ILogSrv>();  
	    …  
	}
	public static void Debug(string message, object[] variables)  
	{   
	    ILogSrv log = ServiceLocator.Current.GetInstance<ILogSrv>();  
	    …  
	}  
}

This makes my class code to depend on some static functions (Logger.Error()), but that seems a good compromise as long as the underneath things remain as simple as in the above snippet.

Now, if we are to write some unit tests in isolation, we would like to use a stub for the ILogSrv interface, and we can do that by making a setup like this:

 [TestClass]  
 public class UnitTests  
 {  
   private Mock<IServiceLocator> slStub;

   [TestInitialize]  
   public void TestInitialize()  
   {  
     slStub = new Mock<IServiceLocator>();  
     ServiceLocator.SetLocatorProvider(() => slStub.Object);  
   }

   [TestMethod]  
   public void PlaceNewOrder_FromPriorityCustomer_AddedOnTopOfTheQueue()  
   {  
      Mock<ILogSrv> dummyLog = new Mock<ILogSrv>();
      slStub.Setup(l => l.GetInstance<ILogSrv>()).Returns(dummyLog.Object); 
      …  
   }  
    …  
 }  

This code configures the ServiceLocator.Current to return an instance which gives the dummy ILogSrv when needed. Therefore, the production code will use a dummy ILogSrv, which probably does nothing on WriteTrace().

For logging this may be just fine. It is unlikely that we would need different stub configurations for ILogSrv in different tests. However, things may not be as easy as this, for other services that are taken through the ServiceLocator.Current. We might want different stubs for different test scenarios. Something like this:

// — production code —
public class UnderTest  
{  
  public bool IsOdd()  
  {  
     IService service = ServiceLocator.Current.GetInstance<IService>();  
     int number = service.Foo();  
     return number%2 == 1;  
  }  
}

// — test code —
private Mock<IServiceLocator> slStub = new Mock<IServiceLocator>();  
ServiceLocator.SetLocatorProvider(() => slStub.Object);

[TestMethod]  
public void IsOdd_ServiceReturns5_True()  
{  
  Mock<IService> stub = new Mock<IService>();  
  stub.Setup(m => m.Foo()).Returns(5);

  slStub.Setup(sl => sl.GetInstance<IService>()).Returns(stub);  
  …  
}

[TestMethod]  
public void IsOdd_ServiceReturns4_False()  
{  
  Mock<IService> stub = new Mock<IService>();  
  stub.Setup(m => m.Foo()).Returns(4);

  slStub.Setup(sl => sl.GetInstance<IService>()).Returns(stub);  
 …  
}

Because our production code depends on statics (uses ServiceLocator.Current to take its instance), when these tests are ran in parallel, we will run into troubles. Think of the following scenario: Test1 sets up the slStub to return its setup for the IService stub. Then, on a different thread Test2 overwrites this setup and runs. After that, when the code exercised by Test1 gets the IService instance through the static ServiceLocator.Current it will receive the Test2 setup, therefore the surprising failure.

By default MS Test or VS Test will run tests from different test classes in parallel, so if we have more test classes which do different setups using the ServiceLocator.SetLocatorProvider(), we will run into the nasty situation that sometimes our tests fail on the CI server or on our machine.

So, what should we do?

One option is to avoid the dependencies to the statics and to get the service locator through Dependency Injection through constructor. This would make the above example like below:

 // — production code —

public class UnderTest  
{  
    private IServiceLocator sl;  
    public UnderTest()  
    {  
        sl = ServiceLocator.Current;  
    }

    public UnderTest(IServiceLocator serviceLocator)  
    {  
        this.sl = serviceLocator;  
    }

    public bool IsOdd()  
    {  
        IService service = sl.GetInstance<IService>();  
        int number = service.Foo();  
        return number%2 == 1;  
    }  
 }

// — test code —

[TestMethod]  
 public void IsOdd_ServiceReturns5_True()  
 {  
    Mock<IService> stub = new Mock<IService>();  
    stub.Setup(m => m.Foo()).Returns(5);

    Mock<IServiceLocator> slStub = new Mock<IServiceLocator>();  
    slStub.Setup(sl => sl.GetInstance<IService>()).Returns(stub);

    var target = new UnderTest(slStub.Object);  
    …  
 }

[TestMethod]  
 public void IsOdd_ServiceReturns4_False()  
 {  
    Mock<IService> stub = new Mock<IService>();  
    stub.Setup(m => m.Foo()).Returns(4);
 
    Mock<IServiceLocator> slStub = new Mock<IServiceLocator>();  
    slStub.Setup(sl => sl.GetInstance<IService>()).Returns(stub);
 
    var target = new UnderTest(slStub.Object);  
    …  
 }  

This would be a good solution and I favour it in most of the cases. Sometimes I add, as in the above snippet, a constructor without parameters that is used in the production code and one which receives the ServiceLocator as a parameter for my unit tests code.

The other option, which is the answer to the question at the start of the post, looks a bit more magical :). It fits the cases when we need and want to keep the simplicity the static caller brings. Here, we keep the production code as is and we make the unit tests to safely run in parallel. We can do this by creating one stub of the IServiceLocator for each thread and record it on a thread static field. We can do it with a ServiceLocatorDoubleStorage class that wraps the thread static field and gives the tests a clean way to setup and access it.

 public static class ServiceLocatorDoubleStorage  
 {  
  [ThreadStatic]  
  private static IServiceLocator current;

   public static IServiceLocator Current  
   {  
      get { return current; }  
   }
  
   public static void SetInstance(IServiceLocator sl)  
   {  
      current = sl;  
   }
  
   public static void Cleanup()  
   {  
      SetInstance(null);  
   }  
}  

Now, the unit tests will use the ServiceLocatorDoubleStorage.SetInstance() instead of the ServiceLocator.SetLocatorProvider(). So the test code from the above sample transforms into:

 [TestClass]  
 public class UnitTest  
 {  
   [AssemblyInitialize]  
   public static void AssemblyInit(TestContext context)  
   {    
        // the production code will get it through  
        // ServiceLocator.Current, so this is needed  
        ServiceLocator.SetLocatorProvider(  
            () => ServiceLocatorDoubleStorage.Current);  
   }
  
   private Mock<IServiceLocator> slStub;
  
   [TestInitialize]  
   public void TestInitialize()  
   {  
        slStub = new Mock<IServiceLocator>();  
        ServiceLocatorDoubleStorage.SetInstance(slStub.Object);  
   }
  
   [TestMethod]  
   public void IsOdd_ServiceReturns5_True()  
   {  
        Mock<IService> stub = new Mock<IService>();  
        stub.Setup(m => m.Foo()).Returns(5);
        
        slStub.Setup(sl => sl.GetInstance<IService>()).Returns(stub);  
        …  
   }  
   …  
 }  

With this, each time a new thread is used by the testing framework, the test on it will first set its own stub of the ServiceLocator and then it will run. This makes that the ServiceLocator stubs even if they are static resources, not to be shared among different tests on different threads. On the code samples from my Code Design Training, on github here, you can find a fully functional example that shows how this can be used and how it runs in parallel.

To conclude, I would say that Dependency Injection and Service Locator should be used together. I strongly push towards using Dependency Injection in most of the cases because it makes the dependencies clearer and easier to manage, but definitely there are cases where Service Locator is needed or makes more sense. In both cases writing isolated unit tests should be easy and may be a good check of our design and dependencies.

This is discussed and explained in detail in my Code Design Training

Featured image credit: noche via 123RF Stock Photo

Last week I had the opportunity to speak for the first time at the Microsoft Summit.

It was a nice and pleasant experience. I have talked about how we could achieve a high quality code design by enforcing consistency with the support of an Application Infrastructure in a large and complex enterprise system.

I think I did one of my best presentations. I placed my story into the context of re-architecting and re-implementing a legacy system using modern techniques and technologies. I have focused my talk on the challenge of building, testing and deploying for On-Premises, but at the same time being able to easily migrate to Azure and leverage the advantages of PaaS. The focus went on implementing a loosely coupled design, so we can replace certain components when migrating to the cloud.

At the end I had some very interesting questions, which turned into good suggestions for me too. One was about Dependency Injection vs Service Locator and how would I unit test, when Service Locator is chosen. I will detail this in my next technical blog post. Its a good idea for a technical topic, so thanks!

Another question was if I am thinking to make the design practices that I have presented available to a broader audience through other means then my Code Design training, like writing on MSDN for example. Definitely I will put some time and thought into this. Maybe besides articles in MSDN, making my course available on an online platform like Pluralsight would also be an idea worth investing in. Thanks for this suggestion as well!

The overall conference was a successful event in my opinion. I’ve liked the mixture of business and technology. I’ve also liked the area with the partners’ stands where you could see demos on how technology can optimise businesses. I had a pleasant surprise to see that our friends from ConSix startup, had a stand presenting their product.

All in all it was a nice week. Thank you Microsoft for inviting me! I’m looking forward to the next editions.

I have uploaded my slides over here if you’d like to take a look before the recordings are made available.

The Crosscutting Concerns are the areas in which high-impact mistakes are most often made when designing an application. There are common causes that lead to this and there are common practices that can help to avoid such mistakes. In this post I will talk about the way I usually address most of the crosscutting concerns at the start of a project, when usually there are many other more urgent things to think of.

The crosscutting concerns are the features of the design that may apply across all layers. This functionality typically supports operations that affect the entire application. If we are thinking of a layered architecture, usually the components from more layers (Presentation Layer, Business Layer or Data Access Layers) use components that address in a common way a concern, or which assures that a critical non-functional requirement is going to be met.

The crosscutting concerns are very much related to the quality requirements (also named non-functional requirements or quality attributes). In fact the crosscutting concerns should derive from these requirements. They are usually implemented in a separate area of the code and specifically address these requirements. For example:

• Logging concern derives from Diagnostics, Monitoring or Reliability requirements
• Authentication and Authorization concern derives from Security requirements
• Caching concern derives from Performance requirements
• Error Handling concern derives from Diagnostics, Availability and Robustness requirements
• Localization concern derives from Multi-language and Globalization requirements
• Communication concern derives from Scalability, Availability or Integration requirements

One of the most common mistake with the crosscutting concerns is that we tend to neglect their importance at the start of the project, and we poorly consider them or we don’t do it at all. Then, when we already have a ton of code written, somewhere in the second half of the project, from various reasons these quality requirements become actual again. But now… because of all the code we have, the cost of addressing them in a consistent and robust manner becomes very high. To add consistent and useful logging for example, now, would mean to go back through all the code and all the tested functionality and change it to call a logging function. This may be costly. The same goes for authorization, localization and many others.

The challenge comes from the fact that the crosscutting components are like support code for the components that implement the functional requirements. They should be done before we start developing functionality. But this is often not a good idea. At the start of the project it is good to start implementing the functionalities, so we can get feedback and show progress, not to spend too much time on things that may be postponed.

The key is to address them at the very beginning, but not to implement them. We should just identify them and design only the most important aspects. We should postpone the making of any time consuming decision. I’m not saying make uninformed decisions because of lack of time and change them later, I am saying to design in a way that allows postponing these decisions.

Lets take Logging for example. We can easily define the logging interface, looking at the monitoring and diagnostics requirements and considering the complexity of the application. It will be something like this:

public static class Log  
{  
 	public static void Error(string healine, string message)  
 	{ }
 
 	public static void Warning(string healine, string message)  
 	{ }
 
 	public static void Trace(string functionName, string message)  
 	{ }
 
 	public static void Debug(string message, object[] variables)  
 	{ }  
}

For the start the functions can do nothing. They can be left unimplemented. Later, we can come back to this and invest time to decide which logging framework should we use. We will think about the configurability of the logging, whether we should log in the database or not, weather we should send some logs by email, whether we should integrate with a monitoring tool and which one, at a later time. Until then, all the code that implements the functional use cases, has a support for logging and if we make a clear and simple logging strategy then we will have meaningful calls to these functions.

Maybe in a few sprints, we’ll come with a simple implementation of this, which will write logs in a text file, to help us in debugging. Then later, we could integrate the framework we know best to write the logs in csv file so we can easily filter and search. By the time we get near the deployment in different test environments, we will know more about the logging the system needs and it will be easier to make a good decisions on how to address this concern’s implementation better. However, in all this time the code we are writing calls the logging interface, so we don’t need to go back and search for meaningful places to insert these calls.

The same practice can be applied for all of the crosscutting concerns. I also show it in a previous post Localization Concern which covers the localization aspects in detail.

So, the idea is that at the start of the project we should abstract each component that addresses a crosscutting concern. We can do this by thinking on how the other layers will interact with it, and define the abstraction from their perspective. The other layers should depend on this abstraction only. We can assure this by hiding (or encapsulating) the implementation of the crosscutting concern. By encapsulating the implementation we have the flexibility to change it or  to replace the frameworks we use without a significant impact on the other code. In most of the cases the dependencies will be like in this diagram, where ApplicationComponents do not know about the Log4Net library, which is just an implementation detail.

As a conclusion, following this approach, we can have the interface (abstractions) of the crosscutting concerns components very early in the project and have it called from all the use cases. With this we postpone the real implementation of these concerns and we limit the cost of adding it later. The time to define these abstraction is usually short. It goes from a few hours to a few days depending on the size and complexity of the application, but it always pays off rather than having this tackled for the first time in the second half of the project.

This topic is discussed in more detail in my Code Design Training

Featured image credit: noche via 123RF Stock Photo

While reviewing my previous post another great discussion, which may arise from paying attention to your references, came to my mind: Don’t Repeat Yourself (DRY) vs Coupling. Each time you add a new reference it means that you want to call the code from the other assembly, therefore you are coupling the two assemblies.

We have always been told that we should not tolerate duplication. We should always DRY our code. We have even become obsessed with it and each time we see few lines of code, which resemble we immediately extract them into a routine and have it called from the two places. What we often don’t realize is that we’ve just coupled those two places. DRY comes at a cost, and that cost is coupling.

Loosely coupled designs is another desiderate that we have. Lack of coupling means that the elements of our code are better isolated from each other and from change. In general the looser the coupling is the better the design is, because it is less resistant to change. And, change is certain!

On the other hand, of course that duplication is bad. It is one of the primary enemies of a well-designed system. It usually adds additional work, additional risk and additional unnecessary complexity. It is even more problematic when the code is copied & pasted and then small changes are done on it. The code is not identical, but it has teeny-tiny, hard to observe variations. Like an equals (=) operator changed with a differs (!=) or a greater or equals (>=) changed with a strict greater (<). The code is not identical, and the commonality is not abstracted. In such a context abstracting the commonality in an interface and making all the callers to depend on the abstraction, but not on the implementation, which should be well encapsulated / hidden, is the key to a good design. Here we couple them together, we pay the coupling cost, but with clear benefits.

Don’t tolerate coupling without benefits! Don’t pay a cost if it doesn’t bring you something back. If by DRYing stuff out you don’t make things simpler but rather more complex, then something may be wrong in your design. In this example if we have created a correct abstraction, which truly represents the commonality then changes of its implementation should not trigger changes into its callers, and changes in the callers will not trigger changes into the implementation.

Coming back to my previous post where I’ve talked about the benefits that come from monitoring our references, which is just another way of managing the dependencies in our code, we can have this discussion on the example there, too.

The application there is well isolated from the frameworks it uses. It has more types of UI. One was in WPF as a desktop app, and one was just a console app. It may have had a web UI as well. Another thing that I’ve emphasised there was that we might have some rules that say which references are allowed and which aren’t. Here is the diagram of the references:

When we are working on the WpfApplication we might find ourselves rewriting some of the code that we’ve already had written in the ConsoleApplication. The first thing that comes to mind would be to reference the assembly and reuse the code. But we can’t. Its against the rules, because we want that the different UIs to be independent. Making references among them would mean that the WPF needs the console UI, or even more strange the web UI would need the desktop UI. So, we are left with two options:

1. duplicate the code in both assemblies
2. create a new assembly (CommonUI) and put the code there

Option 2. reduces duplication, but it still creates coupling. Now all the UI assemblies will reference this common thing, and if it is not well abstracted then we may have indirect dependencies among the UIs. A change in WPF may trigger a change in the common assembly, which will trigger a change in the console or in the web UI. Tricky! We should see if it pays off. If it is just about some helpers it might be better to tolerate the duplication and don’t increase the coupling. On the other hand if it is something that has to be commonly presented in all the UIs of our application, then abstracting and encapsulating it in a common assembly might make our design better.

This is also a good example that we should try to DRY as much as possible in a bounded context, but we should not DRY among different context, because we will couple them together with low benefits.

In the end it is again about making the correct tradeoffs and realising that each time we make a decision we are also paying a cost. Dan North puts this very nicely in a talk that I like very much called Decisions, Decisions

This is discussed in more detail in my Code Design Training, when talking about programming principles.

Featured image credit: bond138 via 123RF Stock Photo

A few posts back I talked about how we can use the assembly references to enforce consistency and separation of concerns (here and here are the old posts). I argue there that if we derive from the architecture the assemblies of our application, the kind of code (responsibility) each one holds and how they can reference each other, then monitoring this can become a useful tool to assure that the critical architectural aspects are followed by the implementation.

In this post I will show how am I verifying the assembly references and the tool that I use. It is usually the first thing I do when I start a review on a new project. I find it an efficient way to spot code design smells, bad questionable design decisions or implementation “shortcuts”, which may hurt really badly the project on the long run.

Let’s dive into details. My favorite tool for this is ReSharper. More exactly the Project Dependency Diagram. It can be generated from the ReSharper | Architecure menu. What I like the most about it is that in ReSharper 9, it is augmented with the Type Dependency Diagram, and now you can easily drill down in any reference to see the dependencies among the classes, and even further to the lines of code that make that reference needed. Now, when you see a reference that shouldn’t be there you can easily find the code that created it and reason about it. (In ReSharper 8, I was using the Optimize Refernces view to drill down to the lines of code that made a reference needed.)

I’m not going to explain in detail how the tool works, you can read about on the ReSharper blog. Let’s look at an example instead.

Here I have generated the reference diagram for a demo project from my Code Design Training (code available on GitHub here). A few things to notice as architectural principles to follow for this project:

• The application has more UI clients (a WPF and Console for now). They cannot have dependencies (references) one on the other, because we want them independent.
• The application consists of more modules. Each module has its own assemblies and they cannot have dependencies (references) among each other. The modules interact only through the Contracts assembly which has only pure interfaces (service contracts) and DTOs (data contracts). - This is applying the DIP (see here) and makes that the implementation of the modules is encapsulated and abstracted through the types in the Contracts assembly.
• The UI gest the functionality implemented by the modules only through the contracts. The UI cannot have direct references to the implementation of the modules, neither to the Data Access. This enforces that the application logic does not depend on the UI, but the other way around (again applying DIP).

Any new reference, which does not obey the above architectural principles will easily be found when we re-generate the diagram from code.

If we go in more details we see other development patterns.

Each module has a .Services assembly which implements or consumes Contracts. The module assemblies may reference and use the DataAccess or the Common assembly from the Infrastructure. These are not necessarily rules as strict as the above, but more like conventions which create a consistency on how a module is structured. The reference diagram can help a lot to see how these evolve.

Look again to the diagrams. Which reference do you think is strange and might be wrong? The Sales module references the DataAccess. This is fine. It needs to use the IRepository / IUnitOfWork to access data. But, one of the Sales module assemblies is referenced back by the DataAccess. This is wrong. We would want that when the implementations of any of the modules changes, the Infrastructure assemblies not to be affected, because if they are then their change may trigger changes into the other modules as well. So, we’ll have a wave of changes which starts from one module and propagates to other modules. If you look to the first diagram, this reference looks like it creates a circular dependency which may be a better smell that something is wrong. If we right-click the reference we can open the Type Dependency Diagram.

Here we see that the SalesEntities, from the DataAccess, is the class that created this reference. If I hold the cursor on the dependency arrow I get all the classes it depends on. This class is the Entity Framework DbContext for the Sales module. It should not be here, but the DataAccess needed it to new it up. (in fact this is a TODO that I have postponed for a while in my demo project). To fix this ‘wrong’ reference we have to invert the dependency. So we should create an abstraction. We can make a IDbContextFactory interface into the DataAccess, move the SalesEntities into one of the Sales module assemblies and implement there the factory interface to new it up.

This is a good example of how this tool can help us to find code at wrong level of abstraction, by spotting wrong dependencies. SalesEntities is a high level concept class. It describes the domain of the Sales module. It was wrongly placed into a lower level mechanism implementation, that implements data access.

If you can spread this code review practice among your team more benefits will follow. Each time a new reference or a new assembly appears the team will challenge if it fits into the rules and into the reasoning behind those rules. It will get you towards contextual consistency: this is how we build and structure things in this project. It may not apply to other contexts, but it makes sense in ours and we know why. Consistency is the primary mechanism for simplicity at scale. Reviewing dependencies easily and fast, shared idioms and guiding principles help create and sustain consistency. Once you have consistency, difference is data! There has to be a good reason why things that break consistency in a particular spot are tolerated.

To conclude, the tool that we use to generate the references diagram is not the most important thing here. I like ReSharper, but you can get almost the same with the architecture tools from Visual Studio Enterprise / Ultimate. What is important is to use a tool that can generate useful dependencies diagrams from the code and to constantly monitor them. The entire team should be doing this. By reviewing these dependencies regularly you make sure that the critical architectural principles and requirements are still met.

Featured image credit: vska via 123RF Stock Photo

Last Friday marked an important milestone for me – it was my last day working for ISDC. After 10 years with ISDC I have decided to put an end to my job there. I think it is a good moment in my life and in my career to try something else, to try to work as an independent programmer / software architect. For those that are interested, I’d like to tell you what I’ve been doing in recent years and give a hint of what will come next.

ISDC

I’ve started in the .NET Department as a junior developer and I’ve exited as a software architect. A long way. I had the opportunity to work with most of the Microsoft technologies, to switch projects and contexts quite often. This helped me a lot in learning and expanding my experience. I have also found in ISDC great people to work with and to learn from. I have found the right models and good mentors. I am grateful for all that. I have also given a lot back. I have pushed for doing things the right way, close to the highest level of the industry standards. I pushed for quality and I had an important contribution in increasing the technical quality delivered by the .NET teams. I remember being characterised as the quality guy that not only talks the talk, but actually walks the walk. I was the one that introduced Unit Testing in .NET teams and worked hard to make it part of the development process and a common practice in all of the teams. I also was one of the key members in some of the most difficult and important projects we had.

In the last years I have focused on starting projects. It starts with envisioning the technical solution that fits the requirements, and continues with working close with the project management to build the right team and to define and implement the strategy that may lead to reaching project goals in budget and time. In the beginning I was also leading the development of the application infrastructure that sets the project on the right path from the technical perspective. I think this defines quite well my software architect role for a project in ISDC. It is not the same as the architect in a product company or as the Enterprise Architect. It focuses on one project. It involves making difficult decisions, making tradeoffs and explaining to all the stakeholders the consequences of their choices. It’s a worth having experience.

The Future

So where to next? I like this graphic from The Future of Work by Jacob Morgan. It shows nicely how I’ll try to work next. I find myself standing right where the half-grey half-green guy is (though ISDC did some of the things better than this illustrates).

As an independent I would like to use my experience to help various teams or companies to start projects on the right track or to get out from a nasty situation related to software development. I think that I can bring a lot of value by coming in, work with the team to set things on a right path, make sure that the team can take it over and be independent of me and then gradually step out. I don’t see myself staying for a very long time in a project. I may stay close, help whenever something new comes up, but if I do my work well the team should do fine without me after a while. Sometimes my involvement may be only to give training or coaching on certain topics or technologies.

Beside my previous experience there are two more things that make me believe I can do this. First is that I’m not entirely new to it. Since 2013 I only work with a part time job. This year I had a three days/week job, so I had two days to work as independent consultant. I’ve already helped some teams in different companies with training, coaching or to start new projects.

The other thing is that I’m not alone. I am part of iQuarc. This gives me a great deal of assurance. I know I will always find the kind of support, advice or help that I need from my colleagues. At core, we all have the same level of expertise and at the same time we have complementary skills. We share same values, principles and passion. Together we can make a great team and we can successfully respond to many kinds of requests.

So, what will change? Now, I have all the time for this. I’m all in. I’m available for hire. Here’s a brief summary of what I can do for you:

• Custom software development. I love writing code, so I’d also like writing code for you. I can take small or big tasks. It does’t really matter. I can work in a team or by myself, remote or on-site, as necessary. My experience is with C# and related technologies / frameworks.
• Software architecture. I have experience in designing small and large applications. I can design the entire project, not only the technical part. This may include a complete strategy from requirements to deployment with needed staff for each phase of the project.
• Reviews. If you need an external party to review your code or your design I’ll be happy to do so. I can do code reviews at different levels, from looking at the big picture, at the way the code is structured and the way dependencies go, towards lower details on how classes and functions are written or tested. When reviewing code or design I can focus on specific quality requirements like security, performance, maintainability, scalability or other that may be of interest for you or I could do a more general check.
• Training and coaching. In the past years I have developed and given two standard trainings: Code Design and Unit Testing, in which I address a wide range of subjects about coding. I can visit your company to deliver lectures and workshops. Beside these topics I could easily spin out workshops on others, like estimations or time management, depending on your needs. I am also a strong believer on learning on the job, so I could join your team only to coach you on a specific technique or a specific issue, working on your project. We can also pair while doing so.
• Development process. Along the years I have experienced different ways of organising software development teams. If you need help with this I can do so. We can see together whether Scrum fits your context or not and how to tweak it. I can also help with Continuous Integration, Continuous Delivery, TFS, Git etc.
• Round table. Sometimes people simply want to have a meeting with someone to validate certain topics or ideas. I’m happy to visit you for a meeting with you and your team where we can discuss your questions, sketch together on a whiteboard, look at code, etc. in an ad hoc fashion.

This list isn’t exhaustive, so if you have other ideas for how you think I may be able to help you, please contact me.

Localization (also known as internationalization) is one of the concerns that is most of the times overlooked when we design an application. We almost never find it through the requirements, and if we do or if we ask about it, we usually postpone thinking about it and we underestimate the effort of adding it later. In this post I will summaries few key aspects which take little time to take into consideration in our design, and they can save a lot of effort on the long term.

Localization Service

One of the first things that I do is to define, what I call the Localization Service. It is nothing more than a simple interface, with one or two simple methods:

 public interface ILocalizationService  
 {  
 	string GetText(string localizationKey);  
 	string Format<T>(T value);  
 }  

Notice that the functions do not take as input parameters the language or culture code. The implementation will take care of taking them from the current user. The interface stays simple.

At the beginning of the project I don’t do more than this. I just put in a trivial implementation that doesn’t do much and I postpone the rest of the decisions. Now when screens are built we can just call this interface, and later we’ll make a real implementation of it. We already have a big gain: when we’ll build the localization we don’t need to go through all the screens and modify them to translate the texts. The localization service is called from the beginning.

To make it a simple call, we can have a static wrapper:

 public static class Localizer  
 {  
 	public static string GetText(string localizationKey)  
 	{  
 		var s = ServiceLocator.Current.GetInstance<ILocalizationService>();  
 		return s.GetText(key);  
 	}

 	// same for Format<T>(..)  
 }  

We can decide later if the translations are stored in resource files or in the database or somewhere else. For now we can just pick the quickest implementation (hardcoded in a dictionary maybe) and move on. We can change it later without modifying the existent screens.

Localization Key

The next thing to consider are some conventions for building the localization keys. We are going to have many texts and it will make a big difference to have some consistent and meaningful keys rather than randomly written strings.

To do this I usually try to define some categories for the translated strings. Then for each category we can define conventions of patterns on how we will create the keys. In most of the applications we’ll have something similar with the following:

• Labels on UI elements - these are specific texts that appear on different screens. Things like buttons, menus, options, labels, etc
  • Pattern:  <EntityName>.<ControllType>.<LabelKey>
  • Example: Person.Button.ChangeAddress
• Specific messages or text - These are text that are specific to a functionality or a screen
  • Pattern: <MessageType>.<Functionality>.<MessageKey>
  • Example: Message.ManagePersons.ConfirmEditAddress
• Standard (or generic) labels or messages - these are text that appear o different screens of the applications
  • Pattern: <MessageType>.<MessageKey>
  • Example: ErrorMessage.UnknownError
• Metadata - These are names of business entities or their properties that need to be displayed. Usually these are column names in list screns or labels in edit screens
  • Pattern: <EntityType>.<Property>
  • Example: Person.Name

With such categories and conventions in place, we get many benefits in debugging, managing translations and even in translating texts.

If the application screens are built by templates (like all of the list or edit screens are similar and are around one business entity), later, we could go even further and write generic code which builds the localization key based on the type of the screen and the type of the entity, and it could automatically call the localization service. For example in a razor view, we could write a html helper like:

// usage  
 @Html.LabelForEx(m => m.Subject);

// implementation  
 public static MvcHtmlString LabelForEx<TModel, TValue>(this HtmlHelper<TModel> html, Expression<Func<TModel, TValue>> expression)  
 {  
 	string entityName = ModelMetadata.GetEntityName(expression, html.ViewData);  
 	string propName = ModelMetadata.GetPropName(expression, html.ViewData);
 
 	string localizationKey = $"{entityName}.{propName}";  
 	string text = Localizer.GetText(localizationKey);
 
 	return html.LabelFor(expression, text);  
 }  

I think that these two: the Localization Service Interface and the Conventions for the Localization Keys are the aspects that should be addressed by the design at the beginning of the project. Next I will go through other two important aspects of localization: Managing Translations and Data Localization.

Managing Translations

One of the aspects that is usually ignored when designing for localization is the process of entering and managing the translated texts in different languages: translating the application.

This process can be difficult if the one that does the translation does not have the context of the text she is translating. Usually a word by word translation in a table is not working well. It can be even more difficult if she does not get fast feedback of the changes into the application screens. Emailing the translations to the developers and waiting for a new release can be very annoying. This difficult process can be even more costly if it was postponed until the last moment and it happens a few weeks before the release into production, when usually there are many other urgent things.

The conventions for the localization keys can play an important role in this. They could give some context, and if the one that translates the application can upload the translations into the app and get fast feedback is usually good enough. This means that we need to design and implement some functionality to upload and show the translated text, to avoid a painful process.

Another approach that works well is to implement into the application functionality to translate it. For a web app, the one that does the translation will access the application in “translate mode” and when she hovers the mouse on a text a floating div with an input is shown where she can input the translated text. The text is saved into the database and the page reloaded with the translation in it.

Even if this sounds difficult to implement, it is not and for an application that has a large variety in the texts it displays and needs to be translated in many languages, it worths the effort and makes the translation changes easy.

Data Localzation

Data Localization is about keeping some of the properties of the business entities in more languages. Imagine that your e-commerce app gets used in France and it would be better to have a translation for the name and the description of your products. For instance for the name of the mouse product, you will need to store its name in french: souris d’ordinateur.

One of the solution to implement this is to create a translation table for each table that has columns, which should be kept in more languages. This allows us to add more languages over time.

The columns of the Products will keep the data in the default language (or language agnostic) and the Products_Trans table will keep the translations in a specific language. Here we’ll have only the columns that need translations: Name and Description.

If we are to add this functionality later into our project, we need to go back in all the existent screens and change them not to read data from one table (Products), but also to join it with the translations table (Products_Trans).  This may be very costly, because changing tens of screens built months ago may put our project in jeopardy.

The alternative, is to build some generic mechanism that automatically does the join under the scenes, based on some conventions and metadata. If we use Entity Framework, LINQ and we have the data access made though a single central point as I’ve described in Separating Data Access Concern post, then this can be achieved.

We need to rely on some conventions:

• the translation tables and EF mapped entities have the same name with _Trans suffix
• the translated columns have same name with the ones in the main table
• some catalog that gives the entity names (tables), for which there is a translation table

With this, and by knowing that the all the LINQ queries go through our one Repository and UnitOfWork implementation as described in the above post, we intercept the lambda expression of each query, parse it, and recreate it with the join for the translation table.

To implement this we make that all the IQueryable our Repository returns to be a wrapper over the one returned by the EF.

 private IQueryable<T> GetEntitiesInternal<T>(bool localized) where T : class  
 {  
 	DbSet<T> dbSet = context.Set<T>();
 
 	return localized ? new DataLocalizationQueryable<T>(dbSet, this .cultureProvider) : dbSet;  
 }  

The DataLocalizationQeryable wrapper uses a visitor to go through the lambda expression and for each member assignment  node, from the Select statement, which needs to be translated, gets the value from the related property of the translation entity. Here is a code snippet that gives an idea of how the wrapper is implemented:

 public class DataLocalizationQueryable<T> : IOrderedQueryable<T>  
 {  
 	private IQueryable<T> query;  
 	private ICultureProvider cultureProvider;  
 	private ExpressionVisitor transVisitor;
 
	public DataLocalizationQueryable(IQueryable<T> query, ICultureProvider cultureProvider)  
	 {  
	 	…  
	 	transVisitor = new DataLocalizationVisitor(this.cultureProvider.GetCurrentUICulture());  
	 	this.Provider = new DataLocalizationQueryProvider(query.Provider, this.translationVisitor, 		cultureProvider);  
	 }
 
	public IEnumerator<T> GetEnumerator()  
	 {  
	 	return query.Provider.CreateQuery<T>(  
	 	this.transVisitor.Visit(query.Expression)).GetEnumerator();  
	 }
 
	class DataLocalizationQueryProvider : IQueryProvider  
	 {  
	 	private IQueryProvider efProvider;  
	 	private ExpressionVisitor visitor;  
	 	private readonly ICultureProvider cultureProvider;
	 
	 public IQueryable<TElement> CreateQuery<TElement>(Expression expression)  
	 {  
	 	return new DataLocalizationQueryable<TElement>(  
	 	efProvider.CreateQuery<TElement>(expression), cultureProvider);  
	 }  
 }

class DataLocalizationExpressionVisitor : ExpressionVisitor  
 {  
 	private const string suffix = "_Trans";  
 	private const string langCodePropName = "LanguageCode";  
 	private readonly CultureInfo currentCulture;
 
 	public DataLocalizationExpressionVisitor(CultureInfo currentCulture)  
 	{ … }
 
 	protected override MemberAssignment VisitMemberAssignment(MemberAssignment node)  
 	{ … }  
 …  
 }  

Even if modifying lambda expressions at runtime isn’t a trivial task, we do it only once as an extension to the data access and we avoid going back and modifying tens of screens.

With this, we have covered the most common aspects of localization and we’ve seen that if we pay some thought to it when we design our application we can easily avoid high costs or painful processes on the long run.

This topic is discussed in more detail in my Code Design Training

Featured image credit: noche via 123RF Stock Photo

In my last posts I have talked about using assembly references to preserve critical design aspects. In Enforce Consistency with Assemblies References I talk about how we can use references to outline the allowed dependencies in code and how to use a references diagram to discover code at wrong levels of abstractions. In Separating Data Access Concern I show how we can enforce separation of concerns by using references and I detail this with the data access example. In this post I will talk about the relation between assembly references, in the context of above posts, and the Dependency Inversion Principle (DIP).

When we reference another assembly we take a dependency on it. If assembly A references assembly B it means that A depends on B. Taking this to the data access example, it means that a business logic assembly depends on the data access assembly.

This seems to be in contradiction with DIP which says that

High level modules should not depend on low level modules

The business logic is the high level module, and the data access is just details on how we get and store data. The contradiction may be even more clear if we refer to the The Clean Architecture of Uncle Bob, where he points that the application should not depend on frameworks.

Let’s look more closely into DIP, and let’s focus on the INVERSION word. DIP doesn’t only say that we invert the dependency, but more importantly we do it by inverting the ownership of the contract (the interface).

After DIP is applied as above diagram shows, the contract is owned by the high level layer, and no longer by the low layer. The essence in DIP is that the changes on the contract should be driven by the high level modules, not by the low level ones. When the contract ownership is inverted, the dependency is also inverted, because now the low level module depends on the high level one by complying with its contract.

In our example, the business logic assemblies depend on the DataAccess assembly because the IRepository and IUnitOfWork interfaces are placed into the DataAccess. If we would move them into the business logic assemblies, then we would invert the reference. Even more, now we could have more DataAccess assemblies which have different implementations, one  with Entity Framework one with NHibernate and at the application startup we could choose which one to use for that specific deployment or configuration.

However, this is not practical. We may have more business logic assemblies that need to access data, so which one should contain these references? We could make an assembly only with the data access interfaces, to solve it. With this, we would also keep the possibility to have more data access implementations. But do we really need more data access implementations? In most cases we don’t. So, it doesn’t worth to separate them.

Now, coming back to the initial question: if we keep the data access interfaces into the DataAccess assembly and the rest of the assemblies reference it, are we following DIP?

YES, as long as these interfaces change ONLY based on the needs of the business logic modules and not because of implementation details, we follow DIP. From a logical separation point of view they are owned by the business logic layer, and the data access implementation depends on them. For practical reasons they are placed in the same assembly with the implementation, because it doesn’t worth creating one only with the interfaces for this case.

As long as the implementation details and specifics do not leak into these interfaces, they represent correct abstractions and the implementation remains well encapsulated.

Following the same reasoning, I some times create a Contracts assembly, which contains the underlying abstractions of the applications. These abstract concepts that are specific to the application and not to one module. They are the truths that do not vary when the details are changed. I may have more functional modules, which have assemblies that implement or consume these contracts.

This figure shows this, by outlining that the functional modules do not reference one on each other but they all reference the Contracts assembly. If you go deep into DIP description in Uncle’s Bob paper here, you will find this approach very similar with the Button-Lamp example from Finding the Underlying Abstraction section.

This topic is discussed in more detail in my Code Design Training, when talking about SOLID Principles.

Featured image credit: jacephoto via 123RF Stock Photo

In this post I’ll describe some key aspects that I consider when designing the assemblies that build a system.

When we structure our code into assemblies (generally named binaries, libraries or packages in other platforms than .NET) we are reasoning about three main things:

• Deployment: different assemblies are deployed on different containers. Some assemblies end up on the UI client, some on the application server and some on third party servers we may use;
• Separate Concerns: we put code that addresses similar concerns in one assembly and we separate it from the code that addresses different concerns. This may translate into encapsulate the implementation of one functional area into a module and offer it through an abstract interface to others. It may also translate into separate the data access concern from business logic;
• Assure Consistency: we want that certain things to always be done in the same way through the entire application, so we define an assembly that will be reused by other assemblies around our application

Another important aspect of referencing assemblies (or linking binaries) is that we cannot have circular references. This may play an important role in managing dependencies. It may help us to avoid circular dependencies among modules or components if we carefully map them to assemblies.

Each time I design the main structure of a new application I do it with all the above in mind. These, used well can bring huge advantages in managing the growth of the application. Even if it is about a rich desktop UI client, which has in same process the business logic and a local database, so the entire application deploys in one container, I will still have more assemblies because I want to use the other advantages. I want to use assembly references to enforce separation of concerns and to enforce consistency. These are the most important tools to manage the complexity of the application, which is critical in large applications developed by more people or even more teams.

When the initial assembly structure is defined we have: the assemblies (or assembly types) and the rules of how they reference each other. This should satisfy the deployment requirements and it should reflect the concerns that must be separated and the things that must be done consistently. I usually put it in a simple diagram with boxes for assemblies and arrows for allowed references. Where there are no arrows, there cannot be references. This diagram not only that helps to explain and verify the design, but it can also be used when reviewing the implementation. If in code we see references that are not in the diagram it may be a fault in the implementation (encapsulation or abstraction leaking, code at wrong level of abstraction, etc.) or it may be a case which was not handled by the design, so the diagram needs to be adjusted.

Let’s dive into details, by looking at some examples.

Logging, is a cross-cutting concern, which in most of the cases we implement by using a third party library. We look for a library (Log4Net for example) which has a high degree of configurability and it can log in files, in databases or send the traces to a web service. In all the cases where we write a log trace, we want to specify in the same way the type, the criticality, the priority and the message. We want to use Log4Net in the same way everywhere in our app. Consistency is important. When something needs to be changed, we want to be able to do the change following the same recipe in all the places where we log.

We can easily enforce this by wrapping the external library in one of our assemblies. Our assembly defines the Log interface which we’ll use in the application. This interface shapes the logging library to our application specific needs. All the configuration and tweaking is done now in one single place: our Logging assembly which implements the Log interface. It is the only one that may reference Log4Net. The rest of the code of the application doesn’t even know that Log4Net is used.

In general any external library gives a very generic API and it is very extendable to many kind of applications. The most applications the library fits, the most successful the library is. When we plug such library in our application we need to tweak it to our specifics and we need to use it in the same way in all the cases.

Even if wrapping it is a very simple solution, it is very powerful. It isolates the change. If something needs to be changed in how the external library is configured, now we don’t need to go through the entire application where it was used. It is directly used in only one place: our wrapper assembly. Even more when we need to replace the external library or to upgrade it to a new version the changes are again isolated in our wrapper. We can isolate in our wrapper all the concerns of communicating with the external library which may include concerns about communication with external systems, security concerns, error handling and so on.

An example for using assembly references to enforce separation of concerns is to separate data access implementation.

In this example the only assembly that can make a connection to the database is the DataAccess assembly. It implements all the data access concerns and offers an abstract interface to above layers. Even more, it does not contain the data model classes, so the business logic (validations or business flows) are kept outside. For more details on how this could be implemented you can refer to my previous post: Separating Data Access Concern.

In the end, here is a simplified references diagram.

Here we can see that we do not have references between the assemblies that implement the business logic, the Functional Modules. They communicate only through abstract interfaces placed into the Contracts assembly. The Contracts assembly contains only interfaces and DTOs. No logic.  With this we make sure that we will not create dependencies between implementation details of the functional modules. The functional modules can access data through the DataAccess assembly, but they cannot go directly to the database. They don’t have any UI logic since they do not have references to UI frameworks assemblies (like System.Web or System.Windows). The UI assembly gets the functionality and data only through the abstract interfaces from Contracts assembly. They can’t do data access otherwise. All are linked through Dependency Injection, which is abstracted by the AppBoot assembly.

To conclude, even if you didn’t start with all these in mind when you’ve created the assemblies of your app, I think it worth the effort to draw such a diagram at any moment, because it will show opportunities to bring more order, clarity and better ways to manage the size of your project.

This design approach is discussed in detail in my Code Design Training

Featured image credit: aspect3d via 123RF Stock Photo

At this time, last week, I was getting ready to get on the stage at the fifth edition of IT Camp. I was starting to feel butterflies in my stomach. Even if it was the third time in a row that I was speaking here I was getting nervous. I remember a friend telling me, before my talk, that this is a good sign. That it means that I care, even if I am comfortable with the talk and the topic. I guess it is true. I do care a lot about these opportunities, I always prepare them carefully and I do my best to say something meaningful to the audience, to have something that sticks in their mind, to make a small impact.

This year I talked about refactoring. I’ve presented a pattern of refactoring that showed how we can get to lower coupling and to a better separation of concerns by trying to increase the cohesion of our classes. I hope that I have mange to transmit that refactoring is part of software development and we cannot code without refactoring. I would be glad to see now everyone from the audience with their hand up when asked if they refactor, knowing that refactoring is not a way they compensate for their mistakes, is not something that developers like and managers hate, but is part of what we do. It is normal for every good developer. At the end I was asked the question I always get when I talk about good practices: “How do I convince my manager that this is right? that this is something that is needed?” This time I answered that we should try to translate it in no technical terms by using metaphors and I pointed a short video, that I value a lot, where Ward Cunningham presents the Debt Metaphor. An answer maybe inspired by Dan North whom, I’ve recently met at Craft Conference.

What I love the most about IT Camp is that it manages to create this great learning and experience sharing atmosphere. Even if it is at its fifth edition, the enthusiasm is everywhere. It’s like the holiday we were waiting for. People are eager to comment the sessions, to share the good and the bad things from their work. I always get back to work with higher energy and with revitalized believe that small things matter and that we all can make a difference even when we feel too far from decision making. IT Camp has proven that if you stick to strong principles, and if you can learn from the past editions, you can constantly improve and you can keep high standards once you get there.

If I were to pick only two things that characterized this edition from content perspective I would say security and a track appealing to managers.

Security topics were very present. There were many experts on security among the speakers and it was the subject of many discussions during the breaks or at the beers at day ending. I think that creating awareness on security is much needed for IT industry of Cluj. We need this. There are too many cases where under the delivery pressure we stop thinking about security after we are done with the login screen and we take huge risks for us and for our customers without even knowing. Putting it on the top of the agenda at one of the most relevant conferences in our area helps.

From the business track, which targeted CxOs and managers in general, I hope to get more managers attending developers’ conferences and meetups. I believe that in general, we need to make a better team with management. We need to understand each other and to really work together to make a common strategy that leads us to achieve common goals. We need to close the gap between these worlds. I think that if managers come to developers events and if we get closer to the business, this gap may shrink. And I believe that community events can play an important role in this.

Mihai, Tudy and all organizing team, THANK YOU! for doing it again and I’m looking forward to the next edition.

UPDATE: You can see my slides on slideshare and the code demo on github.

When writing unit tests one of the challenges is to isolate your tests from everything. To isolate them from the code that is not in their target and also from the other tests. As Roy Osherove puts it in his book “The Art of Unit Testing

[...] a unit test should always run in its little world, isolated from even the knowledge that other tests there may do similar or different things

Test isolation makes a key difference between tests suites which are maintainable and bring high value and the ones which are a burden and bring little or no value.

In data intensive applications one of the most common difficulties when writing unit tests is to isolate them from the database. We would like that the tests which verify the business logic, not to hit the database, so we can easily configure test data for different test cases and we can easily assert the result.

One of the best approaches to achieve this, is to implement the Repository Pattern with a good abstraction that gives all the necessary functions to do data access. In my previous posts I present some examples. Having this, in our tests we can use stubs or mocks as implementations of the IRepository interface, and we can test the caller code in isolation. The code snippet below shows such a test which verifies that the where clause filters out data. Similarly, tests which verify that the result is ordered by a certain criteria, or that some business calculations are done correctly when the data is read or saved can easily be done.

 [Test]  
 public void GetOrdersForShipment_AlsoPendingOrders_PendingOrdersFilteredOut()  
 {  
 	Order orderInPending = new Order {Status = OrderStatus.Pending};  
 	Order orderToShip = new Order {Status = OrderStatus.Processed};  
 	IRepository repStub = GetRepStubWith(orderInPending, orderToShip);
 
 	var target = new OrderingService(repStub);
 
 	IQueryable<Order> orders = target.GetOrdersForShipment();
 
 	var expected = new[] {orderInPending};  
 	AssertEx.AreEquivalent(orders, expected);  
}
 
private IRepository GetRepStubWith(params Order[] orders)  
{  
 	Mock<IRepository> repStub = new Mock<IRepository>();  
 	repStub.Setup(r => r.GetEntities<Order>())  
 			.Returns(orders.AsQueryable());
 
 	return repStub.Object;  
}

Entity Framework supports testable code designs and the DbContext in itself is a repository implementation. So what would it mean to stub or mock the DbContext directly and write isolated tests in a similar way as we did in the example above? We might need to do this when we don’t wrap the DbContext into another repository implementation or we want to test the code that does the wrapping (as we did in the DataAccess implementation here).

To get this going the first thing we need, is to make sure that the code we want to test does not use directly our specific context class, but its base class which is DbContext. A factory which will be used by the target code instead of newing up a MyDatabaseContext instance, gets this done. In the test code we will have the factory return a stub or a mock for the context.

Let’s start with a simple test: verify that filtering data at read works. So this simple test would look like this:

 // target (production) code  
 class UsersService  
 {  
 	private DbContext dbContext;
 
 	public User GetUserById(int id)  
 	{  
 		return dbContext.Set<User>().FirstOrDefault(x => x.Id == id);  
 	}  
 	…  
 }

// unit test code, test method  
 	… 	 
	var contextFactory = GetFactoryWithTestData();  
 	var target = new UsersService(contextFactory);

 	User actual = target.GetUserById(2)

 	User expected = new User {Id = 2};  
 	Assert.AreEqual(expected, actual);

The test data is not visible here. This is because setting it up requires quite some code, and I’ve put it into the GetFactoryWithTestData() function. First, this function needs to build a stub for DbSet<User>, which contains a few user DTOs among which one has the Id == 2. Second, it has to build and configure a DbContext stub which returns the DbSet stub. In a simplified version the code looks like below:

 …  
 private static DbContext CreateContextWithTestData()  
 {  
 	List<User> users = // newup a list with some users DTOs  
 	  DbSet<User> userSet = GetDbSetStub(users);
 
 	Mock<DbContext> contextStub = new Mock<DbContext>();  
 	contextStub.Setup(x => x.Set<User>())  
 	.Returns(() => userSet);
 
 	return contextStub.Object;  
 }

…  
 private static DbSet<T> GetDbSetStub<T>(List<T> values) where T : class  
 {  
 	return new FakeSet<T>(values);  
 }  
 …  
 class FakeSet<T> : DbSet<T>, IQueryable where T : class  
 {  
 	List<T> values;  
 	public FakeSet(IEnumerable<T> values)  
 	{  
 		this.values = values;  
	}
	
 	IQueryProvider IQueryable.Provider  
 	{  
 		get { return values.AsQueryable().Provider; }  
 	}
 
 	Expression IQueryable.Expression  
 	{  
 		get { return values.AsQueryable().Expression; }  
 	}
 
 	Type IQueryable.ElementType  
 	{  
 		get { return values.AsQueryable().ElementType; }  
 	}
 
 	public IList<T> Values  
 	{  
 		get { return values; }  
 	}
 
 	public override T Add(T entity)  
 	{  
 		values.Add(entity);  
 		return entity;  
 	}
 
 	public override T Remove(T entity)  
 	{  
 		values.Remove(entity);  
 		return entity;  
 	}  
 }  

This works well for testing simple queries. For more complex scenarios, setting up data for one-to-many or many-to-many relations gets quite complex. You could set it once with a model of Users and Roles and use it for more tests, but it is hard to do the same for testing other areas of the application and all the business logic.

Another thing to notice in the above snippet is that we have written the FakeStub class instead of using Moq. This is because we want to keep some state on it (the values) and use it in test cases that involve adding or removing entities from the context.

Until at this point, we were able to stub or mock the DbContext and *DbSet  *because all the methods our code used, were overridable. This allowed our us or Moq to replace their behavior in the tests. However, not all public members (or their dependencies) of DbContext are like this, therefore it gets more difficult isolate the tests for some scenarios.

For example, if we would like to test the code that executes when an entity is read from the database, we would need to be able to raise the ObjectMaterialized event on the stubbed context. This event is not on the DbContext, but on the ObjectContext. The ObjectCotext property is nor public nor overridable, which makes it almost impossible to replace it with a stubbed ObjectContext on which we could trigger the event. To overcome this we can create a DbContext wrapper that just pushes up this event. Like this:

 public sealed class DbContextWrapper : IDbContextWrapper  
 {  
 	private readonly ObjectContext objectContext;
 
 	public DbContextWrapper(DbContext context)  
 	{  
 		Context = context;  
 		objectContext = ((IObjectContextAdapter) context).ObjectContext;  
 		objectContext.ObjectMaterialized += ObjectMaterializedHandler;  
 	}
 
 	private void ObjectMaterializedHandler(object sender, ObjectMaterializedEventArgs e)  
 	{  
 		EntityLoadedEventHandler handler = EntityLoaded;  
 		if (handler != null)  
 			handler(this, new EntityLoadedEventHandlerArgs(e.Entity));  
 	}
 
 	public DbContext Context { get; private set; }
 
 	public event EntityLoadedEventHandler EntityLoaded;
 
 	public void Dispose()  
 	{  
 		objectContext.ObjectMaterialized -= ObjectMaterializedHandler;  
 		Context.Dispose();  
 	}  
 }  

Now we need to modify all our test code and production code to use the IDbContextWrapper instead of the DbContext. The factory will return a stub for it and the stub can be configured to raise the event.

This is quite inconvenient. Our tests have too much knowledge of implementation details of the production code. Even more, when trying to tests more code that accesses data, things will get more complex and this wrapper will grow creating a hard to manage mess. It is also an example of how the tests may damage the production code design. Maybe by more refactoring this wrapper will lead to the IRepository interface as the abstraction of the repository pattern which hides EF from the production code, but… it seems unlikely and a very long an painful route.

All these point to the conclusion that for testing in isolation the business logic code, abstracting the data access in with some clear interfaces and testing from there up, is a better approach. Implementing the repository pattern on top of EF, not only gives better separation of concerns and may enforce consistency, but is also helps in testing.

However, testing on the DbContext directly may be useful when it is wanted to write some tests on the repository implementation that wraps the EF, and we want these tests to be isolated from the database and from the caller code. Such implementation is available on iQuarc github account, in the DataAccess repository.

More discussions about writing good unit tets are part of my Unit Testing Training

Featured image credit: geargodz via 123RF Stock Photo

In my previous post I have presented a way to separate your data access from the business logic, when a relational database is used. I have shown another implementation of the well-known Repository pattern. Since Martin Fowler described it in his book Patterns of Enterprise Application Architecture it became one of the most implemented pattern in enterprise applications. As it happens with all the design patterns, design principles or any other widely known best practice, we find them in many different implementations, mixed in different ways, but they all have identical things, because they all implement the same solution to a common problem. They all have classes and interfaces named the same. Things like: Repository, UnitOfWork, IEntity, BaseEntity, ConcurencyException and so on.

I like the quote of Christopher Alexander in GoF (originally in his book A Pattern Language):

Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice

In most of the projects I’ve been involved in, I’ve seen the Repository pattern implemented differently. Tweaked to the context of that project, to the specifics of that team, according with their own style. All different implementations of the same patterns and practices.

Out of the curiosity of how many other blog posts are out there, which show similar implementation with the one I’ve presented, I have googled: “generic repository implementation”. As expected there are quite a few, and as expected they are very similar with mine and with each other. I have randomly picked a few (the first 5 returned by my google search at the time of writing this article) that I will comment in this post.

Blog Post: Implement Step-by-Step Generic Repository Pattern in C#

I like the idea of defining an IEntity interface which should be implemented by all the entities that your repository can persist. This opens the possibility of writing a more generic code into a default implementation of IRepository<T>, (a Repository<T> class), where you could push all the common code like a generic implementation of .FindById(int Id). This will only work if you can enforce the convention that all the tables in your database have a surrogate primary key of type integer.

I appreciate a lot the discussions in the comments of the post. I am closer to the point of view of ardalis regarding the use of such abstraction on top of EF. However, I don’t see IQueriable<T> as a leaky abstraction. In my view it abstracts a query, which may be executed on different query providers, EF being one of them. It also helps a lot when Unit Testing, because I can stub the IRepository to return array.AsQueriable().

Blog Post: Generic Repository and Unit of Work Pattern, Entity Framework, Unit Testing, Autofac IoC Container and ASP.NET MVC [Part 1]

The implementation in this blog post also has the IEntity, but with a slight variation. It is generic by the type of the primary key. With this, we can have different types for the primary key, but it increases the complexity of writing generic code in a common repository implementation.

I like the IAuditableEntity, which should be implemented by all entities on which we want to store information like CreatedBy, CreatedDate etc. With this, in a generic implementation of the repository before we save an entity we can check if it implements IAuditableEntity and if yes, then we can set the correct values in CreatedBy, CreatedData properties and then continue with the save.

The implementation of the UnitOfWork looks a bit strange. It has a Commit() function, which should be called when the client wants to persist a set of changes. What I don’t like is that the client service will need to use one instance of a Repository, another instance of an UnitOfWork and these instances should be built in such way that they wrap the same DbContext instance. The way objects are created (Dependency Injection, Service Locator or custom factories), needs to take care of this, and all the client services need to be aware of it and use it consistently. Things may get too complex when a client service will need to use repositories of two different entities. It will have to deal with four dependencies only for this.

This implementation goes even further and defines a generic service for maintaining business entities. It is called EntityService. A suggestive name I’ve also used in different projects, for a similar generic service with the same scope. These set of services should use the repository (data access), encapsulate the logic around maintaining a business entity (business logic) and give to the controller (UI) functions for the basic CRUD operations.

Blog Post: Understanding Repository and Unit of Work Pattern and Implementing Generic Repository in ASP.NET MVC using Entity Framework

This blog post addresses the UnitOfWork implementation in a better way than the previous two. Here, the UnitOfWork has set of Repository instances that it creates and makes them available through get properties. This solves in a better way the thing that I didn’t like in the previous post. The DbContext is created and owned by the UnitOfWork, and because it is the one that creates the repositories it may pass it to the Repository class.

The drawback of this is that for a client class is quite difficult to know which are the repositories that are available on a specific UnitOfWork instance. It does not know what get properties to expect from an UnitOfWork. This may lead to inconsistency. As a client I might end up reaching the entities I want through strange paths starting from different repositories. It may also lead developers to add repository properties to UnitOfWork classes as they need them in different context, making the UnitOfWork classes to get fat and difficult to maintain.

The generic UnitOfWork class presented in the article helps, but even that may become quite complex when there are many specific repository implementations that need to be created and cached in the dictionary. Mainly the UnitOfWork turns into a Service Locator implementation for repositories of different entity types.

Blog Post: Generically Implementing the Unit of Work & Repository Pattern with Entity Framework in MVC & Simplifying Entity Graphs

This implementation has the same way of implementing a UnitOfWork as the previous. The UnitOfWork has a generic repository through a get property.

The article shows the idea of abstracting the DbContext under an IDbContext interface. With this, the UnitOfWork and Repository no longer depend on a specific context, but on this abstraction. This can help in separating the data access implementation in another assembly than the one in which we have generated the code for a specific context. This interface may be useful when you want that Dependency Injection or Service Locator to get directly involved on creating context instances and you need an interface for configuring the container. Otherwise, the same separation may be achieved with an abstract factory.

Another interesting thing is the RepositoryQuery<T>. The Repository.Get() does not return an IEnumerable nor an IQueryable. It returns this RepositoryQuery<T> which is somehow in the middle. It limits the queries which may be defined, by giving Include, Filter or OrderBy functions and when you say .Get() it composes the expressions and executes the query on the underlying repository.

It is hard to see the cases where the complexity added by this extra indirection pays off. It brings this common place where we can make sure that the queries written by upper layers may be adapted to work the same on any query provider, but unless we know that we need to easily replace EF with something else… might not worth doing it. Another advantage, I can think of, is that it enforces consistency in the client queries, but this needs to be balanced with the limitations it brings.

Blog Post: Implementing the Repository and Unit of Work Patterns in an ASP.NET MVC Application

This article is part of a tutorial on www.asp.net about getting started with EF and MVC. It presents the step by step the process of evolving from a simple repository, which does nothing more than wrapping a DbContext, towards a generic repository and then towards a unit of work that has more repository instances. It follows the same idea. An UnitOfWork class which provides a generic repository through a get property. The UnitOfWork takes care of creating and caching multiple repositories instances, one for each entity type. This may turn it into a complex factory.

Here, the Get() function of the repository returns an IEnumerable. It receives through input parameters different parts of a query. There are parameters to give a filter expression, an order by expression or a string to specify the related entities. In my view this is an unfortunate choice. It does not give to the client a fluent API to specify the query. The client code has to tweak the parameters in different ways which increases the complexity on its side. There will also be cases when new Get() overloads are needed to write in them the LINQ queries directly against the DbContext. This may lead to inconsistency. I would rather return IQueriable, or if not then have a Get() without parameters which returns IEnumeralble and explicitly ask for specific repository implementations which define other Get() overloads for the specific queries.

Conclusions

I think the biggest take away from these blog posts is that there are many ways we can implement the repository and unit of work patterns. All implementations will achieve the same goal which is separating data access concern from the other concerns. The implementations differ because they are used in different context. All have pluses and minuses. What may work in one project, may not work in other, because the tradeoffs that are made differ from project to project. What is a critical plus for some teams in some contexts, may not be useful for others. What is not a big minus in some cases may prove to ruin projects in other contexts.

The more implementations of a pattern you see, the more design ideas you will get when you need to implement it for your current case. The more projects you’ve done where such patterns were used, the easier and faster it will be to implement it again for a new context or to evaluate possible implementations.

Many implementations of data access are discussed in detail in my Code Design Training

Featured image credit: 1tjf via 123RF Stock Photo

In our days most of the applications that have a relational database as storage, use an ORM to access the data. The ORM (Entity Framework, Hibernate, etc.) does most of the data access implementation. Many of them have a modern API for querying data and for creating sessions of editing and saving changes. They also provide mechanisms to hook on events of data being loaded or before data is saved.

However, using the ORM API directly into the classes which implement the business logic (and business flows) or into the Controllers (or ViewModels) which have the UI logic is not a good idea in most of the cases. It would lead to low maintainability and high costs of change. The main reasons for this are poor consistency on how the data access is done throughout the entire app and mixture of business logic with data access concerns.

Think that we are going to use EF in the entire application to access the database. In a large code base there will be places where the EF context is created in the constructor of the class that uses it, and there will be places where the context will be created in the method that needs to get or alter some data. In some cases the context will be disposed by the class that created it and in others it will not be disposed at all. There will be cases where it is passed in as a parameter by the caller code because of the entities which are bound to it. There will also be cases where under certain circumstances entities are going to be attached to newly created contexts. Does this sound familiar?

All these signal poor consistency which leads to increased complexity. When we add error handling, logging or multiple contexts for multiple databases the complexity increases exponentially and becomes uncontrollable. Adding features like: data auditing, data localization, instrumentation (for performance measurements) or enhancing the data access capabilities in any other ways, becomes very costly. It implies going through the entire code base where EF context was used and do these changes. When business logic is not well separated from data consistency validations, enhancing data access capabilities will most likely affect the use case functionalities. We’ll introduce bugs. Our code will smell as rigid and fragile.

To avoid all the above, we can abstract the data access and encapsulate its implementation by hiding the EF from the caller code. The caller code will not be able to do data access in any other ways, but the ones defined by our abstraction. The abstraction has to be good enough not to limit the capabilities of the underlying ORM, and to allow the implementation to hide it, without leaking any dependencies to above layers.

In the rest of the post I will detail such an implementation. Its code source is available on iQuarc github account, in the DataAccess repository. It is designed as a reusable component, which can be installed as a NuGet package.

The main goals of this library are:

• to support a consistent development model for reading or modifying data in the entire application
• to enforce separation of concerns by separating data access from business logic

One of the main ideas is that the assemblies that need to do data access (the ones that implement the business logic) are not allowed to have any references to the Entity Framework assemblies. They can only depend and use the public interface exposed by the DataAccess assembly. They will reference things like System.Data or System.Core and will take full advantage of LINQ or IQueryable, but they do not know that EF is behind. As far as they are concerned any ORM that gives a compatible IQueryProvider implementation may be used as implementation of the abstract DataAccess interfaces they use. The figure below illustrates this:

This enforces consistency on how data access is done in the entire application. Any class that needs to access data has to use the DataAccess interfaces, because they can no longer create or get an EF context. Now, each time we need to change or enhance the data access implementation, there is one central place to do this.

Another important aspect revealed by above diagram is that the database model classes are separated into their own assembly (DataModel). Since with the code first support, we can generated simple POCOs, which have no dependencies to the EF, as classes mapped to the database tables. These POCOs remain as simple as they are, and they change only when the database model changes. They will not have logic.

Having this plus the constraint that the DataAccess assembly cannot reference the DataModel assembly we assure that the business logic does not get mixed with the data access concerns. Inside the DataAccess we cannot write any business logic (not even validations), because it does not know about domain entities and in the other assemblies we cannot have data access concerns because they are well encapsulated into the DataAccess assembly.

Now, let’s explore in more detail the code.

IRepository and IUnitOfWork are the main interfaces of the public API the DataAccess library offers. Besides them, there are few more types, but these give the development patterns of doing data access.

IRepository is meant to read data. It is a generic interface that supports queries starting from any entity.

/// <summary>  
/// Generic repository contract for database read operations.  
/// </summary>  
public interface IRepository  
{  
	/// <summary>  
	/// Gets the entities from the database.  
	/// </summary>  
	/// <typeparam name="T">The type of the entities to be retrieved from the database.</typeparam>  
	/// <returns>A <see cref="IQueryable" /> for the entities from the database.</returns>  
	IQueryable<T> GetEntities<T>() where T : class;
	
	/// <summary>  
	/// Creates a new unit of work.  
	/// </summary>  
	/// <returns></returns>  
	IUnitOfWork CreateUnitOfWork();  
}

In most of the cases it can be received through dependency injection into the constructor of a class that needs to deal with data. Because its implementation is light (entities are for read only), its scope may be larger and its implementation can be disposed when the operation ends (see my Disposable Instances Series blog posts on how to handle the IDisposable with Dependency Injection).

The examples below, show the code patterns for reading data:

private readonly IRepository rep; // injected w/ Dependency Injection  
public IEnumerable<Order> GetAllLargeOrders(int amount)  
{  
	var orders = rep.GetEntities<Order>()  
					.Where(o => o.OrderLines.Any(ol => ol.Ammount > amount)  
	return orders.ToList();  
}

Queries may be reused and returned to be enhanced or composed by the caller code, in the same operation scope:

private readonly IRepository rep; // injected w/ Dependency Injection  
private IQueriable<Order> GetAllLargeOrders(int amount)  
{  
	var orders = rep.GetEntities<Order>()  
		.Where(o => o.OrderLines.Any(ol => ol.Ammount > amount)  
	return orders;  
}

public IEnumerable<OrderSummary> GetRecentLargeOrders(int amount)  
{  
	int thisYear = DateTime.UtcNow.Year;  
	var orders = GetAllLargeOrders(amount)  
		.Where(o.Year == thisYear)  
		.Select(o => new OrderSummary
	
	return orders;  
}     

The IUnitOfWork interface is used to modify data.

/// <summary>  
/// A unit of work that allows to modify and save entities in the database  
/// </summary>  
public interface IUnitOfWork : IRepository, IDisposable  
{  
	/// <summary>  
	/// Saves the changes that were done on the entities on the current unit of work  
	/// </summary>  
	void SaveChanges();
	
	/// <summary>  
	/// Saves the changes that were done on the entities on the current unit of work  
	/// </summary>  
	Task SaveChangesAsync();
	
	/// <summary>  
	/// Adds to the current unit of work a new entity of type T  
	/// </summary>  
	/// <typeparam name="T">Entity type</typeparam>  
	/// <param name="entity">The entity to be added</param>  
	void Add<T>(T entity) where T : class;
	
	/// <summary>  
	/// Deletes from the current unit of work an entity of type T  
	/// </summary>  
	/// <typeparam name="T">Entity type</typeparam>  
	/// <param name="entity">The entity to be deleted</param>  
	void Delete<T>(T entity) where T : class;
	
	/// <summary>  
	/// Begins a TransactionScope with specified isolation level  
	/// </summary>  
	void BeginTransactionScope(SimplifiedIsolationLevel isolationLevel);  
}  

The pattern here is: read data, modify it and save the changes. These operations should be close one to the other, in a short and well defined scope, like below:

public void ReviewLargeAmountOrders(int amount, ReviewData data)  
{  
	using (IUnitOfWork uof = rep.CreateUnitOfWork())  
	{  
		IQueryable<Order> orders = uof.GetEntities<Order>()  
			.Where(o => o.OrderLines.Any(ol => ol.Ammount > amount);  
	
		foreach(var order in orders)  
		{  
			order.Status = Status.Reviewed;  
			order.Cutomer.Name = data.CustomerNameUpdate;  
			…  
		}
		
		ReviewEvent re = new ReviewEvent {…}  
		uof.Add(re)
		uof.SaveCanges();  
	}  
}  

We want the IUnitOfWork to always be used inside a using statement. To enforce encourage this, we do not register its implementation into the Dependency Injection Container, but we make a factory. The factory may be the IRepository itself. We prefer this, rather than forcing a class, that needs data, to have an extra dependency to an IUnitOfWorkFactory.

Another thing to notice here is that the IUnitOfWork inherits from IRepository. This gives two advantages:

• queries defined for the repository may be reused on an IUnitOfWork instance to get data for changing it
• the IUnitOfWork instance may be passed as an IRepository param to code that should only read data in the same editing context

The classes that implement these two interfaces are not part of the public API of the DataAccess. The code from above layers should not use them. To enforce this they may be made internal, which I do when the DataAccess is a project in my VS solution. On the other hand, if it is a reusable library this may be too restrictive and may not work with some Dependency Injection frameworks. Leaving them public requires some development discipline to make sure that they are not newed up in the client code.

Another important part of the public API are the IEntityInterceptorinterfaces. They give the extensibility to run custom logic at specific moments when entities are being loaded, saved or deleted.

/// <summary>  
/// Defines a global entity interceptor.  
/// Any implementation registered into the Service Locator container with this interface as contract ill be applied to  
/// all entities of any type  
/// </summary>  
public interface IEntityInterceptor  
{  
	/// <summary>  
	/// Logic to execute after the entity was read from the database  
	/// </summary>  
	/// <param name="entry">The entry that was read</param>  
	/// <param name="repository">A reference to the repository that read this entry. It may be used to ead 	additional data.</param>  
	void OnLoad(IEntityEntry entry, IRepository repository);
	
	/// <summary>  
	/// Logic to execute before the entity is written into the database. This runs in the same transaction 	ith the Save  
	/// operation.  
	/// This applies to Add, Update or Insert operation  
	/// </summary>  
	/// <param name="entry">The entity being saved</param>  
	/// <param name="repository">A reference to the repository that read this entry. It may be used to ead 	additional data.</param>  
	void OnSave(IEntityEntry entry, IRepository repository);
	
	/// <summary>  
	/// Logic to execute before the entity is deleted the database. This runs in the same transaction with 	he Save  
	/// operation.  
	/// </summary>  
	/// <param name="entry">The entity being deleted</param>  
	/// <param name="repository">A reference to the repository that read this entry. It may be used to ead 	additional data.</param>  
	void OnDelete(IEntityEntry entry, IRepository repository);  
}  

There are global interceptors, which are triggered for all the entities of any type and there are specific entity interceptors, which are triggered for the all entities of one specific type. Interceptors are a good candidate to implement data consistency validations or data auditing. Their implementations belong to above layers and are a key element for keeping business logic outside of the data access assembly.

Another element to mention is the custom exception classes. To prevent breaking encapsulation at error handing, the DataAccess implementation wraps all the EF exceptions, for the errors that need to be passed to the client code, into custom exceptions that are defined. It abstracts the errors and lets the client code to do the error handing without depending on EF specifics.

You can explore further the DataAcess code on its GitHub repository. To see more examples on how it is used, you can look over the sample from my Code Design training, which are also available as a GitHub repository.

This implementation is discussed in detail in my Code Design Training

Featured image credit: kirillm via 123RF Stock Photo

In the past few weeks I have published a set of four posts that deal with disposable instances. These posts describe in detail a working implementation that automatically disposes all the instances that are no longer needed, in a deterministic way. This solution works when we use Dependency Injection or Service Locator to create instances and we want to prevent, by design, leaks of expensive resources.

The posts started from a discussion on how to deal with a repository implementation, which is disposable by nature, and it evolved into a broader one, because the same challenges and approaches apply to any disposable type.

Even if I didn’t intend them from the beginning to be a series, these posts continue each other and describe the design of a component that may be part of the Application Software Infrastructure of any complex app. They include the problem description, the challenges, the different approaches which may be considered and implementation examples.

The posts are:

• Who Disposes Your Repository – where I describe the context and the challenges of dealing with a disposable Repository and Dependency Injection. In this post I also compare different alternatives of handling the dispose in a repository implementation
• Extending Unity Container for IDisposable Instances (1st approach) – here I detail the challenges of achieving automatic dispose with Unity Container and I describe one implementation approach with its pros and cons
• Extending Unity Container for IDisposable Instances (2nd approach) – here I continue the post above with another solution for extending Unity with this behavior. This implementation overcomes the shortcomings of the previous one, but raises some design concerns
• Disposing Instances when Using Inversion of Control – here I complete the solution by addressing the question of when the disposables are  going to be disposed. I give a solution for defining a scope for any operation in any kind of app in a similar way as we have the Request in an web app

many solutions and discussions like the above are included in my Code Design Training

Image by: eteimaging / 123RF Stock Photo

In the last few posts I have written about how to deal with IDisposable instances when using Dependency Injection. In the Who Disposes Your Repository I talk about the possibilities and challenges of disposing a repository which is injected. Then in the Extending Unity Container for IDisposable Instances (part 1 and part 2) I show how automatic dispose of all IDisposable instances can be achieved with Unity Container. This post completes the solution by detailing when the Container Hierarchies (aka Scoped Containers) are built and how they work with a Service Locator.

What we want to achieve is that all the IDisposable instances are disposed when no longer needed. Making this more specific, we want that all the IDisposable instances created within one defined scope to be disposed when that scope ends. In C# we have language support for this. We can use the using statement like this:

using (IDisposable o = new DisposableClass())  
{ // scope begins

	// do stuff with o within this scope

} // scope ends  

Inside the brackets of this using statement we can use the IDisposable instance as we want and we are assured that it will get disposed when the scope ends.

We can use the using statement only if:

1. the begin and the end of the scope is in the control of our code (we write the code that defines the scope)
2. the creation of the IDisposable instance is in the control of our code (we write the code that calls its constructor)

By doing Inversion of Control we give the control of the above to the frameworks we are using.

When we are using Dependency Injection, we no longer call the constructors. A framework (Unity Container maybe) will call it for us. However, we can still dispose instances, if we put the container itself in the using statement:

using (var scopeContainer = mainContainer.CreateChild())  
{ // scope begins

	// do stuff with o within this scope  
	// all the instance created within this scope are created by the scopeContainer
	
} // scope ends  

Mainly we create one container for each scope and we dispose it when the scope ends. When the scoped container gets disposed, it will dispose all the IDisposable instances that it created (previous posts show how this can be done with Unity Container). This is how the idea of using Container Hierarchies for disposing instances came into place. Therefore, if we leave to the framework the control of building instances, we also expect the framework to dispose the instances it created. We still need to make sure that when the scope begins a new container is associated with it, and within that scope all the instances are built with the associated container.

When we are in a web application, we are not in the control of defining the scope neither (Inversion of Control again). We might want to consider this scope to be the same with handling a web request or with a web session. For example, we would want to nest all the code that handles a request within a using statement as we did above. Something quite similar would be: when the request begins create the scoped container, then keep it on the request so all the code running on that request could use it to get new instances, and when the request ends dispose it. Again, if the framework is in control of defining the scope we are interesting in, the framework should give us some hooks which we can use to run some code when the scope begins or ends and to give us some object that represents a context for this scope, so we can keep a reference to the scoped container on it.

Most of the web frameworks give such hooks. We are signaled when a request or session begins or ends. There is also an object which is easy to access, which represents the current request or session and on which we can store context information. If we are using ASP.NET MVC, which is designed with Dependency Injection in mind, we can get this done quite easy. Below is a code snippet from DevTrends github repository, which contains small projects that integrate Unity Container with ASP.NET MVC and ASP.NET Web API.

public class UnityDependencyResolver : IDependencyResolver  
{  
	private readonly IUnityContainer _container;
	public UnityDependencyResolver(IUnityContainer container)  
	{  
		_container = container;  
	}
	
	public object GetService(Type serviceType)  
	{  
		if (typeof(IController).IsAssignableFrom(serviceType))  
		{  
			return ChildContainer.Resolve(serviceType);  
		}  
		
		return IsRegistered(serviceType) ? ChildContainer.Resolve(serviceType) : null;  
	}
	
	protected IUnityContainer ChildContainer  
	{  
		get  
		{  
			var scopeContainer = HttpContext.Current.Items[HttpContextKey] as IUnityContainer;
			if (scopeContainer == null)  
				HttpContext.Current.Items[HttpContextKey] = scopeContainer = _container.CreateChildContainer();
				
			return scopeContainer;  
		}  
	}  
	...  
}    

As you can see here the child container is created first time the framework needs it and then it is recorded on the request object.

When the Service Locator is used it is important that the scoped container gets called whenever a new instance is needed. If the container is used directly, it means that all the code that needs to request a new instance has to be able to go to the current request object, obtain the reference to the scoped container and ask the instance. If we are in an ASP.NET app this is easier because we can use the DependencyResolver.Current, which implements the Service Locator pattern and which with the above integration code will go to the Unity Container recorded on the current request. If we are using another implementation which wraps a dependency container like Common Service Locator does, you will need to set it up in a way that it uses the current container. The snippet below shows an example for the Common Service Locator.

...  
private void ConfigureDependencyContainer()  
{  
 	Microsoft.Practices.ServiceLocation.ServiceLocator.SetLocatorProvider(() =>  
	{  
 		var scopeContainer = HttpContext.Current.Items[HttpContextKey] as IUnityContainer;

 		if (scopeContainer == null)  
 			HttpContext.Current.Items[HttpContextKey] = scopeContainer = _container.CreateChildContainer();
 	
 		return scopeContainer;  
 	});  
 }  
 ...  

All the above work well when we are in a web application, but how can we do the same in a context where we do not have a request object (given by a framework) on which we can keep the scoped container? What if we want to scope our context to a custom operation? How can we make sure that whenever ServiceLocator.Current is called from any class or any function in any thread, it will wrap over the current scoped container if the calling code is within an operation or it will go to the main container if it is outside of any operation? Such application examples may be:

• a windows service which listens on a TCP-IP socket and handles concurrently all the commands that come on the socket. The custom operation would be all the code that handles such a command.
• a console application which executes commands received through inline parameters. Here each command implementation would be part of the custom operation to which we would like to scope a child container.
• a desktop application, where we would like that all the code behind a screen to be part of an operation to dispose all the instances used while that screen was opened or used.

In all these cases we can create the scoped container and put it in a using statement that nests all the code within that operation. The difficulty comes in associating the scoped container with the operation. Where to keep it so all the code that runs within that operation uses it to build new instances. We need an object (like the request in the web app) which can keep this reference. That object should be available (through a static call) to any class or function on any thread which is within the operation. In short ServiceLocator.Current needs to wrap the scoped container of the current operation.

We can implement this by creating an OperationContext class, which represents the context information of a custom operation. When it is built, it creates and stores a scoped container and it makes it available through a static getter. Here is a code snippet of this class

public class OperationContext : IDisposable  
{  
 private readonly IDependencyContainer container;

 private OperationContext()  
 {  
 	container = ContextManager.GlobalContainer.CreateChildContainer();  
 }

 public IServiceLocator ServiceLocator  
 {  
 	get { return container.AsServiceLocator; }  
 }

 public IDictionary Items  
 {  
 	get  
 	{  
 		if (items == null)  
 			items = new Hashtable();  
 		return items;  
 	}  
 }

 public void Dispose()  
 {  
 	DisposeItems();

 	IDisposable c = container as IDisposable;  
 	if (c != null)  
 		c.Dispose();  
 }

 public static OperationContext Current  
 {  
 	get { return ContextManager.Current; }  
 }

 public static OperationContext CreateNew()  
 {  
 	OperationContext operationContext = new OperationContext();  
 	ContextManager.SwitchContext(operationContext);  
 	return operationContext;  
 }  
 ...  
 }  

Our code which defines the scope will create a new OperationContext when the operation starts and it will dispose it when the scope ends. We can do this with an using statement. The OperationContext.Current gives access to it. It can be called from any class, function on any thread and it gives the current operation. ServiceLocator.Current can wrap OperationContext.Current.ServiceLocator and all existent code which we are nesting within this using statement shouldn’t be modified. This class makes sure that the current operation context information is thread static, but is it passed to new threads which are created within current operation. It also assures that when the operation ends all the disposables it holds (including the dependency injection container) are disposed.

The OperationContext class implementation is inspired from the HttpContextclass. It uses a ContextManager static class to manage the access to the storage of the context. The context store is abstracted by IContextStore interface. Its implementation has to provide thread static storage for more operations that may exist simultaneous. When we are in a console application or in a windows service, its implementation is based on the CallContext class. This assures that the context is passed along on any function call, and also to new threads which may be created from current one.

Having this, we can now define custom operations in any application in a easy way:

 using(OperationContext.CreateNew())  
 { //scope begins

 	// code that implements the operation  
 	// ServiceLocator.Current wraps the scoped container created for this operation.

} // scope ends. OperationContext and all its content are disposed  

The OperationContext is an abstract solution for giving context information to any operation regardless of the type of application. When used in a web application the IContextStore may be implemented over the HttpContext.Current and the ASP.NET remains in control of managing the context of our operation (web request).

// IContextStore implementation for an ASP.NET app  
public class HttpRequestContextStore : IContextStore  
{  
	public object GetContext(string key)  
	{  
		return HttpContext.Current.Items[key];  
	}
	
	public void SetContext(object context, string key)  
	{  
		HttpContext.Current.Items[key] = context;  
	}  
	
	// setup at app startup (Global.asax.cs)  
	protected void Application_Start()  
	{  
		...  
		ContextManager.SetContextStore(new HttpRequestContextStore());  
	}
	
	// bind OperationContext with a web request  
	class RequestLifetimeHttpModule : IHttpModule  
	{  
		public void Init(HttpApplication context)  
		{  
			context.BeginRequest += (sender, args) => OnBeginRequest();  
			context.EndRequest += (sender, e) => OnEndRequest();  
		}
		private void OnBeginRequest()  
		{  
			OperationContext.CreateNew();  
		}
		private void OnEndRequest()  
		{  
			OperationContext.Current.Dispose();  
		}
	}  
	...  
}  

An integrated implementation of the OperationContext, class can be found on iQuarc github repository for the AppBoot library. The OperationContext there can be used in the same way in any .NET app. An isolated sample of the OperationContext implementation code can be downloaded here:

To summarise, OperationContext class gives us the means to easily achieve what we wanted: all the IDisposable instances created within one defined scope to be disposed when the operation ends. It does this by using scoped dependency containers which are bound to the defined scope and are disposed when the scope ends. It also gives an abstract way to easily define such a scope when our code is in control or to bind it to one created and managed by a framework.

many examples and discussions like the above are included in my Code Design Training

I have enrolled in John’s Sonmez free e-mail course titled “How to Create a Blog That Boosts Your Career”.

I’ve joined this because I wanted to focus more on improving my blog, and because I care about John’s advice on how to do this. I was already acting on some of the advice he gave when talking about how developers can promote their selves or how they can improve their careers. I’ve picked these from his blog (simpleprogrammer.com), when he was on .NET Rocks! or in other of his materials that I get by following him on twitter.

I was also curious on how an e-mail course would go. Each lesson is in an e-mail that we (the trainees) receive twice a week. One on Monday, and one on Thursday. The lessons are quite short, few minutes to read them. Each speak about a key aspect to care and think about when building a blog. I see it like developing a set of habits that can lead you towards a fulfilling blog. Each lesson also has a short homework, which in most of the cases we should e-mail back. The nice reward I’ve gotten by doing my homework was the feedback that I’ve received from John. So far, he replied with some extra advice specific to my context to each of my homework e-mails. It makes me feel that this is not just a set of articles sent by e-mail, but it is an actual course with interaction with the trainer over the e-mail. I think it is one of the key factors of success in course ran by e-mail.

One of the things that it made me think hard about, was how to focus my posts on a topic. Which is the theme of by blog? Conclusion: Quality Code Design. The key reason I’ve started a blog in the first place was to share from my experiences. I was lucky enough to be part of many software development projects, in different roles. I’ve been dealing with difficult contexts both from technical and corporate politics perspective. There are many learnings I’ve taken along the way. One of the believes I’ve formed is that the way we organize our code, the way we design the structures in which we place the code, the way we do code design matters a lot in achieving the needed flexibility to accommodate changes in the success factors of a project (requirements, time and budget). I consider a code design to be of good quality if it stands the test of time. If it can be changed at low costs. Therefore, the conclusion I’ve reached is that I will try to focus my writings in this area. Not as stories from the projects I did in the past, but rather experiences or advices that I pick from my current work. Learnings I find worth sharing from the training and coaching I do in different companies, from the consulting and the development I do as part of iQuarc and from the work I do as a Software Architect at ISDC. And all, focused on achieving a quality code design.

At this point the course reached its second half. I can’t say I have found many new things, so far. However, it outlined some important aspects and made me think harder of them. The blog theme is one example. Another great benefit for me is that I get a strong confirmation on the practices I was already following. For me this is a good encouragement that I am going on a good path, especially when it comes from someone who is successful in this.

In the lessons to come I hope to get some guidance on aspects like:

• How to put links from a post. Should all the words that may point to more details, have a link?
• How to find and chose good images for the posts. Should all the posts be accompanied by a featured image?
• How long an article should be? When is it better to finish a subject in one post and when to split it over more posts?
• When is it better to put source code into the post and when is it better to give it as a downloadable zip.
• Which are the pros and cons of hosting your blog under a company domain (blog.iquarc.com/florin) or come up with a name for the blog and host it on its own domain (blogname.com)

For me it was a good choice to enroll. It cost me just some time and I’ve got good advice and strong confirmation in return. A great deal. If you’d want to do the course I think you can still enroll (for a future session maybe) or you could probably get the lessons and go through them at your own pace.

In my previous blog post I detailed an approach of making the Unity Dependency Injection Container ț̦o automatically call Dispose() on all the IDisposable instances it builds and injects. The implementation described there, makes use of custom lifetime managers and it works fine for most of the cases except for the PerResolveLifetimeManager. Unity was extended with the PerResolve functionality in a tricky way, which makes my previous IDisposable implementation not to work for it. Therefore, I’ve came up with another solution which works in all cases. I will go into details about it here. Both these blog posts are a continuation of the Who Disposes Your Repository post, I’ve written a while ago, where I describe a context where such an automatically disposing mechanism is desired.

For both implementations we consider that a child container is created and associated with each new scoped operation. A scoped operation may be a request or a session in a web application, or it may be a window or a view in a desktop one. During that operation the child container will be used to inject the dependencies. This is also known as using Container Hierarchies or Scoped Containers. When the operation ends, it will dispose the child container, and what we want is that this also calls Dispose() on all the IDisposable instances that were created within that operation.

Core Solution

In the previous implementation, we were recording all the IDisposable instances that are created on the lifetime manager instance associated with a type registration. Now, because this does not work for the PerResolve case (where new PerResolveLifetimeManager instances are constructed during build), we need to come with another place where to record these instances. This new place also needs to be signaled when the owning container gets disposed, so we can dispose them.

The next thing to look at is to use a custom Unity builder strategy. It is one of the most powerful mechanisms that can be used to extend the build mechanism in Unity. To create one, we need to inherit from the BuilderStrategy base class and to override one of its methods to add the code we want to be executed when any new instance gets built or torn down. Here is the code of the base class:

// Represents a strategy in the chain of responsibility. 
// Strategies are required to support both BuildUp and TearDown. 
public abstract class BuilderStrategy : IBuilderStrategy 
{ 
	// Called during the chain of responsibility for a build operation. The 
	// PreBuildUp method is called when the chain is being executed in the 
	// forward direction. 
	public virtual void PreBuildUp(IBuilderContext context) 
	{ 
	}
	
	/// Called during the chain of responsibility for a build operation. The 
	/// PostBuildUp method is called when the chain has finished the PreBuildUp 
	/// phase and executes in reverse order from the PreBuildUp calls. 
	public virtual void PostBuildUp(IBuilderContext context) 
	{ 
	}
	
	/// Called during the chain of responsibility for a teardown operation. The 
	/// PreTearDown method is called when the chain is being executed in the 
	/// forward direction.
	public virtual void PreTearDown(IBuilderContext context) 
	{ 
	}
	
	/// Called during the chain of responsibility for a teardown operation. The 
	/// PostTearDown method is called when the chain has finished the PreTearDown 
	/// phase and executes in reverse order from the PreTearDown calls.  
	public virtual void PostTearDown(IBuilderContext context) 
	{ 
	} 
} 

For our case, we can override the PostBuildUp() and record all the IDisposable instances that get constructed. In comparison with the previous implementation, here we would keep on the strategy object all the IDisposable instances of different types (all types that implement IDisposable). There, on the lifetime manager we were keeping references only to the instances of the type registered with that lifetime manager object.

The next step is to trigger the dispose, down from the container to the recorded IDisposable instances. In its Dispose() the container disposes two things: the lifetime managers, and the extensions. See below:

protected virtual void Dispose(bool disposing) 
{ 
	if (disposing) 
	{ 
		if (lifetimeContainer != null) 
		{ 
			lifetimeContainer.Dispose(); 
			lifetimeContainer = null;
	
			if (parent != null && parent.lifetimeContainer != null) 
			{ 
				parent.lifetimeContainer.Remove(this); 
			} 
		}

		// this will trigger the Dispose() into our strategy object 
		extensions.OfType().ForEach(ex => ex.Dispose()); 
		extensions.Clear(); 
	} 
} 

The extensions dispose is our trigger point. We hold the IDisposable instances on the builder strategy object, not on the extension. To have them disposed, we can make the strategy object to implement IDisposable. There we can dispose all the instances. Then, going upwards, we can also make the custom extension that adds the strategy to the build strategies chain, to be IDisposable. It may keep a reference to the strategy and when it gets disposed by the container, it can dispose the strategy, which in its turn will dispose all the IDisposable instances that it records on the PostBuildUp(). So, when we put all the things together the code is like below

public class DisposeExtension : UnityContainerExtension, IDisposable 
{ 
	private DisposeStrategy strategy = new DisposeStrategy();

	protected override void Initialize() 
	{ 
		Context.Strategies.Add(strategy, UnityBuildStage.TypeMapping); 
	}
	
	public void Dispose() 
	{ 
		strategy.Dispose(); 
		strategy = null; 
	}
	
	class DisposeStrategy : BuilderStrategy, IDisposable 
	{ 
		private DisposablesObjectList disposables = new DisposablesObjectList();
	
		public override void PostBuildUp(IBuilderContext context) 
		{ 
			if (context != null) 
			{ 
				IDisposable instance = context.Existing as IDisposable; 
				if (instance != null) 
				{ 
					disposables.Add(instance); 
				} 
			}
		
			base.PostBuildUp(context); 
		} 
		… 
		public void Dispose() 
		{ 
			disposables.Dispose(); 
			disposables = null; 
		} 
	} 
} 

Having this done, if we add the extension to the child container after it is created, things work. All the IDisposable instances get disposed when the child container is disposed. The code snippet below shows the usage case:

// code in the application start-up area 
UnityContainer container = new UnityContainer(); 
DisposeExtension disposeExtension = new DisposeExtension(); 
container.AddExtension(disposeExtension);

container.RegisterType(new PerResolveLifetimeManager()); 
container.RegisterType(new PerResolveLifetimeManager());

// some instance created out of the scoped operation 
outerScopeSrv = container.Resolve();

// code in the area that marks and new operation. It creates a child container 
using (IUnityContainer childContainer = container.CreateChildContainer()) 
{ 
	var childDisposeExt = new DisposeExtension(); 
	childContainer.AddExtension(childDisposeExt);
	
	// instances built within the operation 
	scopedSrv1 = childContainer.Resolve(); 
	scopedSrv2 = childContainer.Resolve(); 
	…
} // end of the operation -> childContainer.Dispose()

AssertIsDisposed(scopedSrv1); // -> Pass 
AssertIsDisposed(scopedSrv2); // -> Pass 
AssertNotDisposed(outerScopeSrv); // -> Pass

// at application end 
container.Dispose();

AssertIsDisposed(outerScopeSrv); // -> Pass

However, there are two more aspects that need to be taken care of.

Issues

The first is about singletons. What happens when a singleton which is IDisposable gets injected through the child container? Now, it will be disposed by the first child container that used it when it gets disposed. From now on, all the other parts of the application will use the already disposed singleton instance. This is certainly a behavior we wouldn’t want. If a singleton is IDisposable (it’s another discussion why would we do that in the first place) it should not be disposed in the first operation that uses it. The code snippet below shows this case.

UnityContainer container = NewContainer();

// register as singleton 
container.RegisterType(new ContainerControlledLifetimeManager());

IService1 singletonSrv = container.Resolve(); 
IService1 scopedSrv;

using (IUnityContainer childContainer = CreateChildContainer(container)) 
{ 
	scopedSrv = childContainer.Resolve(); 
	Assert.AreSame(s, scopedService); 
}

AssertNotDisposed(singletonSrv); // -> Fail

To fix this, inside the strategy, we need to verify if the instance that is being built is singleton. If yes, it should not be recorded for dispose, in the strategy that belongs to the child container. To do this check, we look if the lifetime manager is ContainerControlledLifetimeManager and if it comes from the parent container. Below is the code that does it

… 
private bool IsControlledByParrent(IBuilderContext context) 
{ 
	IPolicyList lifetimePolicySource; 
	ILifetimePolicy activeLifetime = context.PersistentPolicies 
										.Get(context.BuildKey, out lifetimePolicySource);
	
	return activeLifetime is ContainerControlledLifetimeManager 
						&& !ReferenceEquals(lifetimePolicySource, context.PersistentPolicies); 
} 

This code already starts to make our extension a bit ugly. It hardcodes that the singletons are being registered with this particular lifetime manager type. If one will create a new custom lifetime manager to implement singletons, but with some additional behavior our extension may not work. Another thing that is a bit ugly, is that we need to search each time an instance is built, for the lifetime manager recursively upwards. However, I don’t see a better way to make this work for this case.

The second thing that needs to be fixed is more difficult to observe. The issue appears when we add this extension both to the parent and to the child container. This may happen either because we want the automatic dispose for the parent container too, or when we have deeper container hierarchies, meaning that we create a child container from another child container because of nested operations (probably not the case in a web app, but it could be the case in a desktop app).

By design, when a child container is created it inherits (by referencing) all the build strategies of the parent container. In general this is a good idea, because the strategies the parent container was configured with are also used by its children, and this brings consistency. However, for our case this is problematic. When the child container builds an IDisposable instance, that instance will be recorded twice: once in the child container DisposeStrategy object, and one more time in the parent DisposeStrategy object (we need to have different DisposeStrategy objects in the parent and in the child, because on these objects we record the IDisposable instances). Recording the instances built by the child container into the parent container strategy object may lead to memory leaks. The child containers are disposed once their operation ends, but the parent container will live typically as long as the application does. Its strategy object will also live with it, and it will grow with WeakRefernce objects for each instance built by all the child containers in all operations. This is bad!

It is also a sign that this approach to implement the automatic dispose, by keeping the reference on a builder strategy may not be in line with the design of Unity. The strategies were not meant to keep state from build to build. They were rather meant to extend the behavior of building instances, by adding logic that operates on the data from the IBuilderContext. However, because PerResolveLifetimeManager is implemented in a questionable way, now we are working around it with another questionable extension :(.

The fix for this case is not nice. In the PostBuildUp() method we need to determine if this strategy object belongs to the container that initiated the current build (the child) or it is an inherited strategy. If doesn’t belong to current container, we should not record currently built instance. The IBuilderContext which we get into the PostBuildUp(), does not contain any information on the container that builds this instance. So we cannot use a container match to distinguish if currently built instance should be registered in current builder strategy object or not. The only way we could distinguish between these cases is to relay on the order in which the strategies are put in the strategy chain which is constructed before a build starts. Always the strategy chain is created by placing the inherited strategies first, and at the end the specific strategies of the current container. Therefore, if the current strategy object is not the last in the strategy chain, it means that it is inherited and we should not record the current instance. We are relying on the ordering done in the StagedStrategyChain class which implements the IStagedStrategyChain interface. It is not very clear if the ordering we are relying on is by design, meaning that it should be preserved in future versions or it is just an implementation detail. Therefore, this should be an attention point when new versions of Unity will be released. The code for this fix is shown in the snippet below:

private bool IsInheritedStrategy(IBuilderContext context) 
{ 
	// unity container puts the parent container strategies before child strategies when it builds the chain 
	IBuilderStrategy lastStrategy = context.Strategies 
									.LastOrDefault(s => s is DisposeStrategy);

	return !ReferenceEquals(this, lastStrategy); 
}

In the end, if we put everything in together, we get the desired dispose behavior both for PerResolveLifetimeManager and for TrainsientLifetimeManager with this approach. The entire source code that implements this approach can be downloaded from here.

Conclusion

Both of the two approaches that I’ve presented in the previous blog post and in this one, have pluses and minuses. There is a trade-off to be made when picking one over the other. The first one, which uses the lifetime managers to record the IDisposable instances has a cleaner design. It is in-line with the way Unity was meant to be extended for lifetime and dispose management. Its disadvantage is that it does not work with the PerResolveLifetimeManager.

The second approach, which uses a builder strategy to record the IDisposable instances works with all the lifetime managers, including the PerResolveLifetimeManager. Its disadvantage is that it is an ugly extension. It holds state on a builder strategy and it may rely on some of the current implementation details. These makes it vulnerable to future versions of Unity or when combining it with other extensions.

If I were to choose, I would use the first one if I do not need the PerResolveLifetimeManager. If I need it, then I would fall back to the second one and carefully test if it works with the version of Unity that I am using and with the other extensions that I need. The good part is that switching from one implementation to the other is done by changing the way the container is configured. Usually this code is separated from the rest of the application, so by doing this switch there should be little if any change in the code that implements the use cases. Therefore, a switch with small costs.

Many examples like the above are included in my Code Design training

A few weeks ago, in my blog post ‘Who Disposes Your Repository’ I wrote about the challenges of implementing an IDisposable repository, which takes full advantage of the deferred execution of the IQueryable and, which is being injected through Dependency Injection (DI) into the classes that need to read or store data. I have focused the discussion on the case when the repository is abstracted by an interface, and its implementation is injected through DI. I have detailed more alternatives on how it could be disposed and the challenges that arise from the stateful nature of a repository, the deferred execution of the IQueryable and the DI.

In the end I have argued that especially for large applications, I would prefer that the Dependency Injection Container (DIC) disposes all the IDisposableinstances it created and it injected, including the repository. In short, the code that creates an IDisposable instance should also be responsible to call Dispose() on it.

In this post I will discuss some of the challenges of achieving this with Unity Container. By working to extend Unity with this functionality I have ended up implementing two approaches. In this post I will detail the first implementation I did, and in a next post I’ll explain the other.

There are many other writings on this. For example I have found useful the posts of Rory Primrose and of Marcel Veldhuizen who describe different approaches for disposing instances created by Unity. They both describe very well how Unity works and how it can be extended for this purpose. However, I was looking for an approach in which Unity remains unknown to the client code. I didn’t want the client code to call Teardown() or other Unity specific methods.

I wanted that when an operation ends all the IDisposable instances that were created and injected by Unity within that operation to be automatically disposed. Such an operation may be a request or a session in a web application, or may be a window or a view in a desktop one. In general any well-defined scope in any application.

My approach is based on using the Container Hierarchies (also known as Scoped Containers). Unity, as many other dependency containers, supports this. When you call CreateChildContainer() method, it will create a container that inherits all the configurations form the current container. The idea is that when an operation begins, a child container will be created and associated with it. During that operation all the dependency injection will be done using the child container. This means that all the new instances injected during the operation, will be created by the child container. When the operation ends, the child container will be disposed. The Dispose() of the child container should trigger the Dispose() for all IDisposable instances that were created.

Here is a good example on how to associate a child container with a web request on an ASP.NET MVC app or here for a WebApi app. I will come back in a future post on this, to show how this association can be done in a more abstract way which can also work when we are not in a web application.  Now, I will dive into how to extend to trigger the Dispose() from the child container to all the IDisposable instances that it created.

The problem to solve boils down to these:

• Keep weak references to all the IDisposable instances that were created by the container (the child container)
• Make sure that the Dispose() function of Unity will also call Dispose() on the instances we are referencing

The first thing I tried, was to make use of the lifetime managers. Unity defines this as an extension point to allow external code to control how references to object instances are stored and how the container disposes these instances. For example they are used to implement Singleton-like instances. When the container is configured, using RegisterType() method, an instance of a LifetimeManager has to be also given. The container will keep as part of its configuration, all the instances of the lifetime managers associated to the configured types, and it will call them when it injects instances of those types. A lifetime manager class is in general simple. It inherits the LifetimeManager base class and by overriding GetValue() and SetValue() methods, it can control the lifetime of the instances of the type it was configured with.

For example if you want that a certain type to be Singleton like, you would use the ContainerControlledLifetimeManager like this:

 container.RegisterType<IMySingletonService, MySingletonService>(
 									new ContainerControlledLifetimeManager());

If we look into the codeof the ContainerControlledLifetimeManager we see that the lifetime manager keeps a reference to the instance and it will return it each time GetValue() is called (for simplicity I modified a bit the code of this class):

public class ContainerControlledLifetimeManager : LifetimeManager, IDisposable  
{  
	private object value;
	
	/// <summary>  
	/// Retrieve a value from the backing store associated with this Lifetime policy.  
	/// </summary>  
	/// <returns>the object desired, or null if no such object is currently stored.</returns>  
	public override object GetValue()  
	{  
		return this.value;  
	}
	
	/// <summary>  
	/// Stores the given value into backing store for retrieval later.  
	/// </summary>  
	/// <param name="newValue">The object being stored.</param>  
	public override void SetValue(object newValue)  
	{  
		this.value = newValue;  
	}
	
	/// <summary>  
	/// Remove the given object from backing store.  
	/// </summary>  
	public override void RemoveValue()  
	{  
		this.Dispose();  
	}
	
	public void Dispose()  
	{  
		this.Dispose(true);  
		GC.SuppressFinalize(this); // shut FxCop up  
	}
	
	protected virtual void Dispose(bool disposing)  
	{  
		if (this.value != null)  
		{  
			if (this.value is IDisposable)  
			{  
				((IDisposable)this.value).Dispose();  
			}  
			this.value = null;  
		}  
	}  
}

The first time an instance of MySingletonClass needs to be injected, the Unity container will first call the GetValue() of the lifetime manager. If that returns null, the container will build a new instance, will call the SetValue() of the lifetime manager to pass it back, and then it will inject it further. Next time an instance of same time is needed, the GetValue() will return the previously created one, so the container will not build a new one. Therefore, the Singleton-like behavior.

The TransientLifetimeManger is at the other end. It is used when you want new instances to be created each time they need to be injected. It is also the default. Its code is even simpler because GetValue() always returns null, which makes the container to build new instances each time.

 /// <summary>  
 /// An <see cref="LifetimeManager"/> implementation that does nothing,  
 /// thus ensuring that instances are created new every time.  
 /// </summary>  
 public class TransientLifetimeManager : LifetimeManager  
 {  
 	/// <summary>  
 	/// Retrieve a value from the backing store associated with this Lifetime policy.  
 	/// </summary>  
 	/// <returns>the object desired, or null if no such object is currently stored.</returns>  
 	public override object GetValue()  
 	{  
 		return null;  
 	}
 
 	/// <summary>  
 	/// Stores the given value into backing store for retrieval later.  
 	/// </summary>  
 	/// <param name="newValue">The object being stored.</param>  
 	public override void SetValue(object newValue)  
 	{  
 	}
 
 	/// <summary>  
 	/// Remove the given object from backing store.  
 	/// </summary>  
 	public override void RemoveValue()  
 	{  
 	}  
 }  

Unity design and documentation encourages using the lifetime managers to control the disposing of created instances. On its Dispose() the container will call the Dispose() on all the lifetime manager instances that implement IDisposable. However, the TransientLifetimeManger is not IDisposable. This is somehow normal because it does not keep reference to anything, so nothing to Dispose(). To achieve our goal, I have created a DisposableTransientLifetimeManger like this:

 public class DisposableTransientLifetimeManager : TransientLifetimeManager, IDisposable  
 {  
 	private DisposableObjectList list = new DisposableObjectList();

 	public override void SetValue(object newValue)  
 	{  
 		base.SetValue(newValue);
 	
 		IDisposable disposable = newValue as IDisposable;  
 		if (disposable != null)  
 		list.Add(disposable);  
 	}
 
 	public void Dispose()  
 	{  
 		list.Dispose(); // this will call Dispose() on all the objects from the list  
 	}  
 }  

The SetValue() will populate a list which keeps weak references to all instances which are disposable. On Dispose() it will just dispose all the elements in the list. Simple. The ContainerControlledLifetimeManager is already IDisposable and it does dispose the instance it references. So now, if we use the disposable lifetime managers when we configure the container, all the instances which are IDisposable be disposed when the container gets disposed.

This works fine when we dispose the container that is directly configured, with the disposable lifetime managers. However, what we wanted was to configure once the main container, and then to use child containers for each operation (request) and to dispose the child containers only. The child containers, would use the configuration of the parent. The code snippet below shows this usage case:

 // configuring the main container  
 UnityContainer container = new UnityContainer();  
 container.RegisterType<IService1, Service1>(new DisposableTransientLifetimeManager());  
 container.RegisterType<IService2, Service2>(new DisposableTransientLifetimeManager());

 using (var childContainer = container.CreateChildContainer()) // child container should be associated with an operation (request)  
 {  
 	// some instances created within the operation (request)  
 	s11 = childContainer.Resolve<IService1>();  
 	s12 = childContainer.Resolve<IService1>();
 
 	s21 = childContainer.Resolve<IService2>();  
 	s22 = childContainer.Resolve<IService2>();  
 } //childContainer.Dispose()

 AssertIsDisposed(() => s11.Disposed); //–> fail  
 AssertIsDisposed(() => s12.Disposed); //–> fail

 AssertIsDisposed(() => s21.Disposed); //–> fail  
 AssertIsDisposed(() => s22.Disposed); //–> fail  

This does not work as we expected, because when the child container is disposed it does not have any lifetime managers instances in it. This is because it uses the configuration from the parent, including the instances of the lifetime mangers given there. This is by design in Unity. If the lifetime mangers instances from the parent would not be used, we would get into short lived singletons issues. Think that you configure a certain type to be Singleton. If you get objects of it through main container you get instance1, but if you get it through a child container you would get instance2. To prevent this, the same instance of lifetime manager (the one given at RegisterType() is used by child containers too.

For our case, what we would like, is that for our disposable lifetime managers, the child container to create new instances of them and to use those to manage the lifetime of the objects it builds and injects. We can achieve this in by creating a custom Unity extension. Extensions are a more powerful way to extend the behavior of Unity and are often used in combination with the lifetime managers. As any powerful extension mechanism you can tweak the default behavior a lot, but when not used carefully you can work against the original design of the framework and create complexity. In our case we want to achieve the exact same thing as the HierarchicalLifetimeStrategy does for the HierarchicalLifetimeManager. So, I pretty much copied its code, into a new generic extension for any hierarchical lifetime manager, like this:

 public class HierarchicalLifetimeExtension<T> : UnityContainerExtension where T : LifetimeManager, new()  
 {  
 	protected override void Initialize()  
 	{  
 		Context.Strategies.AddNew<HierarchicalLifetimeStrategy<T>>(UnityBuildStage.Lifetime);  
 	}

 	/// <summary>  
 	/// A strategy that handles hierarchical lifetimes across a set of parent/child  
 	/// containers.  
 	/// </summary>  
 	private class HierarchicalLifetimeStrategy<T> : BuilderStrategy  where T : LifetimeManager, new()  
 	{  
 		/// <summary>  
 		/// Called during the chain of responsibility for a build operation. The  
 		/// PreBuildUp method is called when the chain is being executed in the  
 		/// forward direction.  
 		/// </summary>  
 		public override void PreBuildUp(IBuilderContext context)  
 		{  
 			IPolicyList lifetimePolicySource;
 	
 			var activeLifetime = context.PersistentPolicies.Get<ILifetimePolicy>(context.BuildKey, out lifetimePolicySource);  
 			if (activeLifetime is T && !object.ReferenceEquals(lifetimePolicySource, context.PersistentPolicies)	)  
 			{  
 				// came from parent, add a new lifetime manager locally  
 				var newLifetime = new T();
 		
 				context.PersistentPolicies.Set<ILifetimePolicy>(newLifetime, context.BuildKey);  
 				context.Lifetime.Add(newLifetime);  
 			}  
 		}  
 	}  
 }  

Now, if we put all the pieces together, we get the behavior we wanted. The snippet below shows the result.

 // configuring the main container  
 UnityContainer container = new UnityContainer();  
 container.RegisterType<IService1, Service1>(new DisposableTransientLifetimeManager());  
 container.RegisterType<IService2, Service2>(new DisposableTransientLifetimeManager());

 var outterScopeSrv = container.Resolve<IService1>();

 using (var childContainer = container.CreateChildContainer()) // child container should be associated with an operation (request)  
 {  
 	// adding this extension to the child, makes the difference from previous code snippet  
 	childContainer.AddExtension(  
 	new HierarchicalLifetimeExtension<DisposableTransientLifetimeManager>());
 
 	// some instances created within the operation (request)  
 	s11 = childContainer.Resolve<IService1>();  
 	s12 = childContainer.Resolve<IService1>();
 
 	s21 = childContainer.Resolve<IService2>();  
 	s22 = childContainer.Resolve<IService2>();  
 } //childContainer.Dispose()

 AssertIsDisposed(() => s11.Disposed); //–> success  
 AssertIsDisposed(() => s12.Disposed); //–> success

 AssertIsDisposed(() => s21.Disposed); //–> success  
 AssertIsDisposed(() => s22.Disposed); //–> success

 AssertIsNotDisposed(() => outerScopeSrv.Disposed); //–> success

So, when the child container is disposed it will call the Dispose() of all the IDisposable lifetime mangers within the child container. The DisposableTransientLifetimeManager we have created and use is IDisposable and on its Dispose() it will call the Dispose() of all the IDisposable instances that it references. The HierarchicalLifetimeExtension that we have created and added to the child container, makes sure that when an instance is to be build, for a type that was configured in the parent container, a new instance of the same lifetime manager is created and added into the child container to be used from now on when building objects of that particular type.

This approach works well for transient lifetime manager and for most of the other lifetime managers, if are extended with the IDisposable implementation in the same way. It is inline with the Unity design and not difficult to understand and use.

You can download all the source code in a zip file here.

However, this approach does not work for the PerResolveLifetimeManager. Yes, it was a bad surprise for me too :(. PerResolveLifetimeManager is not IDisposable, but that’s not the issue. We could make a DisposablePerResolveLifetimeManager as we did for the transient, and collect in it weak references to all IDisposable instances. However, the PerResolveLifetimeManager behavior is implemented with the DynamicMethodConstructorStrategy, and this is problematic. Under certain conditions this strategy creates new instances of the PerResolveLifetimeManager. This seems to work against the original design of Unity, because by creating more new instances of a lifetime manager within same container instance, the lifetime manager is striped by its main purpose, which is to control the lifetime of the instances of a particular type. Here, the used instance of the lifetime manager is no longer given through RegisterType() method, but it is being created during the object build. So, our current approach does not work for this lifetime manager for two main reasons: not all of the new instances of the PerResolveLifetimeManager get stored into the container to be disposed, and the DynamicMethodConstructorStrategy will build new instances of PerResolveLifetimeManager not of DisposablePerResolveLifetimeManager as we’d want.

For the cases when we would like to use both the PerResolveLifetimeManager and the TransientLifetimeManger I ended up making another implementation to extend Unity with same behavior of automatically disposing all the IDisposableinstances, when the child container gets disposed. I will detail it in a future post.

many examples like the above are included in my Code Design Training

Low coupling and high cohesion go hand in hand. In a low coupled design, the modules, classes and functions have a high cohesion. The other way around is also true: making high cohesive modules, classes or functions leads to a loosely coupled design.

Why is this useful or interesting? The thing is that in designing software we never get directly to a good design. We can’t make a good design from the start. We improve the design of our code by gradually refactoring it. We put into the code the knowledge we learn by experimenting with it. This is code refactoring. Therefore, I think it is important to identify and to know patterns of refactoring, in order to increase our efficiency in getting to a good design. Refactoring towards high cohesive classes and functions is a pattern of refactoring our code to result a lower coupled design. I will present it bellow by walking through an example.

This article turned out to be quite long. For easier reading I have splat it in three. The first part reviews few concepts and the second part dives into the code, walks us through the small refactoring steps and explains the reasoning behind. At the end it summaries the refactoring pattern steps and a few smells to pay attention to.

Part 1: reviewing few concepts

Coupling refers to how tight the connection is between two elements of our system. Lower the coupling is the better isolated the elements of our system are. This means that changes in one element would not trigger changes in the others. Generally, the looser the coupling is the better the design is because it is less resistant to changes. However, some coupling is needed. The system elements do need to interact, and in a loosely coupled design this interaction is well defined through abstractions. The inner details of an element (module or class) are well encapsulated (hidden) and are not relevant when thinking on how to interact with others. When details change, they do not trigger waves of changes in all the system making it resistant to change. So coupling is tolerated where it is needed, meaning when it brings benefits and it should be done through good abstractions. In other words: Don’t tolerate coupling without benefits!

Identifying the good abstractions and implementing them through well encapsulated elements is not trivial, because we have a limited view on the system in the beginning. We couple things by nature. We put things together when we do not see the whole picture. Then when it is revealed, we need to stop, we need to refactor to break them up. Then we can identify the good abstractions which lead towards a better design.

A class has high cohesion when most of (all off) its fields are used by most of (all of) its methods. The more fields a method manipulates the more cohesive that method is to the class. A class in which each field is used by each method is maximally cohesive. When cohesion is high, the methods and fields of the class are co-dependent and hang together as a logical whole. Generally, it is neither advisable nor possible to create such maximally cohesive classes, but on the other hand we would like the cohesion to be high. For a function, cohesion refers to how closely the operations in a routine are related. The function code should consistently use all the variables it declare and all the input parameters to compute the result.

When we observe a class with poor cohesion, for example a class which has two parts, one made of three public methods which operate on two fields and another one made of two public methods which operate on other three fields, the general pattern to increase its cohesion is to split it in two classes and to make one to use the other. This way we can identify good abstractions and more code reuse opportunities, which will improve our design.

The example we will dive into shows a pattern to refactor to lower coupling, by improving the cohesion. The first step is to improve the public interface of a class by reducing the number of parameters of its functions. This reveals better the low cohesion of the class and then in successive refactoring steps we increase it, by splitting the class into more.

The pattern to reduce the number of parameters from public methods is to transform them into class fields, which are set through the constructor. This makes the class interface to be more cohesive and it reduces the complexity of the caller code. As consequence, it may also reveal the poor cohesion of the class, by now having groups of fields used by groups of methods. The refactor goes in small steps improving the design a little by little in controllable small chunks.

Part 2: refactoring example

The class that we are looking at is a class that is used to export XML files that represent pages of data about customers and their orders. The class gives functions that can export XML pages with detailed customer information or only with orders. The core functionality this class implements stays in the logic to fill in a XML with data (to simplify the example I have taken out this code)

To follow easier this refactoring example, you can download the source code file from each refactoring step from github here. We start with 00_PageXmlExport.cs, which is the initial state of the code and then when we advance you can look at 01_PageXmlExport.cs after first refactor step, then 02_PageXmlExport.cs for the second refactor step and so on until the fifth refactor step.

 // file: http://tinyurl.com/00-PageXmlExport-cs  
public class PageXmlExport  
{  
  private const string exportFolder = "c:temp";
 
  public bool ExportCustomerPage(
			string fileNameFormat,  
			bool overwrite,  
			string customerName,  
			int maxSalesOrders,  
			bool addCustomerDetails)  
  {  
  	string fileName = string.Format(fileNameFormat, "CustomerPage", customerName, DateTime.Now);  
  	string filePath = Path.Combine(exportFolder, fileName);
 
  	if (!overwrite && File.Exists(filePath))  
           return false;
  
  	PageXml content = new PageXml {Customer = new CustomerXml {Name = customerName}};
 
  	using (var repository = new EfRepository())  
  	{  
  	  if (maxSalesOrders > 0)  
  	  {  
  		var orders = repository.GetEntities<Order>()  
  				.Where(o => o.Customer.CompanyName == customerName)  
  				.OrderBy(o => o.OrderDate)  
  				.Take(maxSalesOrders);
 
  		//enrich content with orders  
  		// …  
  	  }
 
  	  if (addCustomerDetails)  
  	  {  
  	      var customer = repository.GetEntities<Customer>()  
  		       .Where(c => c.CompanyName == customerName);
		  
  		  // enrich content with customer data  
  		  // …  
	     }  
  	}
	 
  	XmlSerializer serializer = new XmlSerializer(typeof (PageXml));  
  	using (StreamWriter sw = File.CreateText(filePath))  
  	{  
  		serializer.Serialize(sw, content);  
  	}

    return true;  
  }  
  …  
}  

ExportCustomerPage(…) function will write on the disk an XML page containing customer details and customer orders, which are read from the database using a Repository implementation. It receives as input parameters a fileNameFormat with the pattern to generate the name of the file and a flag that says whether it should overwrite the file in case it already exits. It also receives the name of the customer to export the data of, and the maximum orders that should be exported. It gets an extra setting weather it should include or not the customer details.

This function evolved over time in this shape. The desire to reuse some of the code that implements the logic of enriching the XML with orders into the one that also enriches the XML with customer data, made the developers to add a the addCustomerDetails flag and correspondent code in here. Also the logic of composing the file name and/or overwriting it was added later, in the easiest way to implement a new request.

Looking further, the same desire to reuse the code that enriches the XML, made it that later a new function which includes external data into the exported XML was added to the same class. This is how ExportCustomerPageWithExternalData(…) got written. It does the same as the above, but if external data is present, then it is included in the XML.

 // file: http://tinyurl.com/00-PageXmlExport-cs  
 public class PageXmlExport  
 {  
  …  
  public bool ExportCustomerPageWithExternalData(  
			string fileNameFormat,  
			bool overwrite,  
			string customerName,  
			int maxSalesOrders,  
			bool addCustomerDetails,  
			PageData externalData,  
			ICrmService crmService,  
			ILocationService locationService)  
  {  
    string fileName = string.Format(fileNameFormat, "CustomerPage", customerName, DateTime.Now);  
    string filePath = Path.Combine(exportFolder, fileName);
   
    if (!overwrite && File.Exists(filePath))  
      return false;
 
    PageXml content = new PageXml {Customer = new CustomerXml {Name = customerName}};

    if (externalData.CustomerData != null)  
    {  
  		// enrich content with externalData.CustomerData  
  		// …  
    }  
    else  
    {  
       CustomerInfo customerData = crmService.GetCustomerInfo(  
       content.Customer.Name);
      
       // enrich content with customer data  
       // …  
    }
 
    using (EfRepository repository = new EfRepository())  
    {  
      if (maxSalesOrders > 0)  
      {  
        var orders = repository.GetEntities<Order>()  
             .Where(o => o.Customer.CompanyName == content.Customer.Name)  
             .OrderBy(o => o.OrderDate)  
             .Take(maxSalesOrders);
 
        //enrich content with orders  
      }
 
      if (addCustomerDetails)  
      {  
         var customer = repository.GetEntities<Customer>()  
         .Where(c => c.CompanyName == customerName);
        
         // enrich content by merging the external customer data with what read from DB  
         // …  
       }  
    }
   
    if (locationService != null)  
    {  
      foreach (var address in content.Customer.Addresses)  
      {  
        Coordinates coordinates = locationService.GetCoordinates(address.City, address.Street, address.Number);  
        if (coordinates != null)  
          address.Coordinates = string.Format("{0},{1}", coordinates.Latitude, coordinates.Longitude);  
      }  
    }
   
    XmlSerializer serializer = new XmlSerializer(typeof (PageXml));  
    using (StreamWriter sw = File.CreateText(filePath))  
    {  
       serializer.Serialize(sw, content);  
    }
  
    return true;  
 }  
  …  
}  

Following the same approach to reuse the core code that enriches the XML, other functions we added in time. ExportOrders(..) exports a XML with the same schema, but with all the orders the customer has and without additional customer data.

// file: http://tinyurl.com/00-PageXmlExport-cs  
public class PageXmlExport  
{  
  …  
  public bool ExportOrders(string fileNameFormat, bool overwrite, string customerName)  
  {  
    string fileName = string.Format(fileNameFormat, "CustomerOrdersPage", customerName, DateTime.Now);  
    string filePath = Path.Combine(exportFolder, fileName);
 
    if (!overwrite && File.Exists(filePath))  
      return false;
   
    PageXml content = new PageXml {Customer = new CustomerXml {Name = customerName}};
   
    using (EfRepository repository = new EfRepository())  
    {  
      var orders = repository.GetEntities<Order>()  
      		.Where(o => o.Customer.CompanyName == content.Customer.Name)  
      		.OrderBy(o => o.ApprovedAmmount)  
      		.ThenBy(o => o.OrderDate);
 
      //enrich content with orders  
    }
 
    XmlSerializer serializer = new XmlSerializer(typeof (PageXml));  
    using (StreamWriter sw = File.CreateText(filePath))  
    {  
        serializer.Serialize(sw, content);  
    }
 
    return true;  
  }  
  …  
}  

Later on, because this was the class that knew how to produce XMLs with customer orders, it got two new methods: GetPagesFromOrders(…) and ExportPagesFromOrders(…), which were useful when the data should not be taken from the database, but it is given as input parameter.

 // file: http://tinyurl.com/00-PageXmlExport-cs  
public class PageXmlExport  
{  
  …  
  public IEnumerable<PageXml> GetPagesFromOrders(  
		IEnumerable<Order> orders,  
		int maxSalesOrders,  
		ICrmService crmService,  
		ILocationService locationService)  
  {  
    Dictionary<string, IEnumerable<Order>> customerOrders = GroupOrdersByCustomer(orders);  
    foreach (var customerName in customerOrders.Keys) 
    {  
       PageXml content = new PageXml {Customer = new CustomerXml {Name = customerName}};
 
       if (crmService != null)
       {  
           CustomerInfo customerData = crmService.GetCustomerInfo(content.Customer.Name);  
           //enrich with data from crm  
       }
 
       var recentOrders = customerOrders[customerName]  
                    .OrderBy(o => o.OrderDate)  
                    .Take(maxSalesOrders);  
       foreach (var order in recentOrders)  
       {  
           // enrich content with orders  
           // …  
       }
 
       if (locationService != null)  
       {  
           foreach (var address in content.Customer.Addresses)  
           {  
               Coordinates coordinates = locationService.GetCoordinates(address.City, address.Street, address.Number);   
               
			   if (coordinates != null)  
               		address.Coordinates = string.Format("{0},{1}", coordinates.Latitude, coordinates.Longitude);  
           }  
       }
 
       yield return content;  
    }  
  }
 
  public bool ExportPagesFromOrders(  
			string fileNameFormat,  
			bool overwrite,  
			IEnumerable<Order> orders,  
			int maxSalesOrders,  
			ICrmService crmService,  
			ILocationService locationService)  
  {  
     IEnumerable<PageXml> pages = GetPagesFromOrders(orders, maxSalesOrders, crmService, locationService);  
     foreach (var pageXml in pages)  
     {  
        string customerName = pageXml.Customer.Name;  
        string fileName = string.Format(fileNameFormat, "CustomerOrdersPage", customerName, DateTime.Now);  
        string filePath = Path.Combine(exportFolder, fileName);
 
        if (!overwrite && File.Exists(filePath))  
            return false;
 
        XmlSerializer serializer = new XmlSerializer(typeof (PageXml));  
        using (StreamWriter sw = File.CreateText(filePath))  
        {  
           serializer.Serialize(sw, pageXml);  
        }  
     }
 
     return true;  
  }  
  …  
}  

Now, look at the public methods signature only. They have many and redundant parameters. This makes them hard to be used by the caller code.

 // file: http://tinyurl.com/00-PageXmlExport-cs  
public class PageXmlExport  
{  
  …  
  public bool ExportCustomerPage(  
			string fileNameFormat,  
			bool overwrite,  
			string customerName,  
			int maxSalesOrders,  
			bool addCustomerDetails)  
  {…}
 
  public bool ExportCustomerPageWithExternalData(  
			string fileNameFormat,  
			bool overwrite,  
			string customerName,  
			int maxSalesOrders,  
			bool addCustomerDetails,  
			PageData externalData,  
			ICrmService crmService,  
			ILocationService locationService)  
  {…}
 
  public bool ExportOrders(  
			string fileNameFormat,  
			bool overwrite,  
			string customerName)  
  {…}
 
  public IEnumerable<PageXml> GetPagesFromOrders(  
			IEnumerable<Order> orders,  
			int maxSalesOrders,  
			ICrmService crmService,  
			ILocationService locationService)  
  {…}
 
  public bool ExportPagesFromOrders(  
			string fileNameFormat,  
			bool overwrite,  
			IEnumerable<Order> orders,  
			int maxSalesOrders,  
			ICrmService crmService,  
			ILocationService locationService)  
  {…}  
  …  
}  

The first step of refactor is to reduce the number of parameters of the first two methods. We move some common parameters into the constructor. It results the following code (file 01_PageXmlExport.cs).

 // file: http://tinyurl.com/01-PageXmlExport-cs  
public class PageXmlExport  
{  
  private const string exportFolder = "c:temp";
 
  private readonly string fileNameFormat; //used in 3/5 methods  
  private readonly bool overwrite; //used in 3/5 methods
 
  private readonly int maxSalesOrders; // used in 3/5 methods  
  private readonly bool addCustomerDetails; // used in 2/5 methods
 
  public PageXmlExport(string fileNameFormat,  
			bool overwrite,  
			int maxSalesOrders,  
			bool addCustomerDetails)  
  {  
  	this.fileNameFormat = fileNameFormat;  
  	this.overwrite = overwrite;  
  	this.maxSalesOrders = maxSalesOrders;  
  	this.addCustomerDetails = addCustomerDetails;  
  }
 
  public bool ExportCustomerPage(string customerName)  
  {…}
 
  public bool ExportCustomerPageWithExternalData(  
		string customerName,  
		PageData externalData,  
		ICrmService crmService,  
		ILocationService locationService)  
  {…}  
  …  
}  

This already makes the functions to look better. They are easier to be called. The caller can now instantiate this class with the settings it needs, and then call ExportCustomerPage(…) for all the customer names it wants. Previously, it had to repeat most of the parameters for each customer call. If we look at the resulted fields we can see that most of them are used in 3 of the 5 functions. The rest of the code remains pretty much the same.

Continuing on this path, the next small refactor step is to reduce the number of parameters for the next functions. The resulted code is the following (file 02_PageXmlExport.cs).

 // file: http://tinyurl.com/02-PageXmlExport-cs  
public class PageXmlExport  
{  
  private const string exportFolder = "c:temp";  
  private readonly string fileNameFormat; //used in 3/5 methods  
  private readonly bool overwrite; //used in 3/5 methods
 
  private readonly int maxSalesOrders; // used in 3/5 methods  
  private readonly bool addCustomerDetails; // used in 2/5 methods
 
  private readonly ICrmService crmService; // used in 3/5 methods  
  private readonly ILocationService locationService; // used in 3/5 methods
 
  public PageXmlExport( string fileNameFormat,  
			bool overwrite,  
			int maxSalesOrders,  
			bool addCustomerDetails,  
			ICrmService crmService,  
			ILocationService locationService)  
  { … }
 
  public bool ExportCustomerPageWithExternalData(  
			string customerName,  
			PageData externalData)  
  { … }
 
  public bool ExportOrders(string customerName)  
  { … }
 
  public IEnumerable<PageXml> GetPagesFromOrders(IEnumerable<Order> orders)  
  { … }
 
  public bool ExportPagesFromOrders(IEnumerable<Order> orders)  
  { … }  
  …  
}  

Now looking again only at the signature of the public functions of this class. They look much better. The public interface is more consistent and this simplifies the caller code.

Analyzing further the refactor result, we see that the class has in the constructor 6 dependencies. This is quite a lot. Looking at the fields, it has 7 fields, from which 2 are not complex (ICrmService and ILocationService). Having so many fields and dependencies is a bad smell. On a closer look we may observe that some fields are used only by some functions and not all functions use all fields. This is a clear symptom of a class with poor cohesion.

To refactor towards a better cohesive class, we should identify groups of fields, used by same methods. For example, the exportFolder, fileNameFormat and overwrite are used by all functions that write to disk. Even more, they are only used in the logic that relates to writing the file. When we identify such a group of dependencies it is an opportunity to reduce the dependencies of the class by moving them into a new class, which will be used by the current one. While doing this, is good to also think about a good abstraction for the class we are going to create, an abstraction that express well the functionality the new class is going to provide. For our example we will have it as an interface which has a function that writes a PageXml. If it is hard to come up with a good abstraction, we can postpone this and do it in two steps: first split the classes and then find a good abstraction. Doing this refactor step it results the following code (file 03_PageXmlExport.cs).

 // file: http://tinyurl.com/03-PageXmlExport-cs  
public interface IPageFileWriter  
{  
   bool WriteFile(PageXml page, string filePrefix);  
}

public class PageXmlExport  
{  
  private readonly IPageFileWriter fileWriter;
  
  private readonly int maxSalesOrders; // used in 3/5 methods  
  private readonly bool addCustomerDetails; // used in 2/5 methods
  
  private readonly ICrmService crmService; // used in 3/5 methods  
  private readonly ILocationService locationService; // used in 3/5 methods
  
  public PageXmlExport( IPageFileWriter fileWriter,  
  			int maxSalesOrders,  
  			bool addCustomerDetails,  
  			ICrmService crmService,  
  			ILocationService locationService)  
  { … }
  
  public bool ExportCustomerPage(string customerName)  
  {  
     PageXml content = new PageXml {Customer = new CustomerXml {Name = customerName}};
     
     using (EfRepository repository = new EfRepository())  
     {  
        if (maxSalesOrders > 0)  
        {  
           var orders = repository.GetEntities<Order>()  
                .Where(o => o.Customer.CompanyName == content.Customer.Name)  
                .OrderBy(o => o.OrderDate)  
                .Take(maxSalesOrders);
                
           //enrich content with orders  
           // …  
        }
     
        if (addCustomerDetails)  
        {  
           var customer = repository.GetEntities<Customer>()  
           .Where(c => c.CompanyName == customerName);
           
           // enrich content with customer data  
           // …  
        }  
     }
   
     return fileWriter.WriteFile(content, "CustomerPage");  
  }  
  …  
}  

This refactor step has reduced the number of fields by groping three of them into one. As a consequence it also reduced the dependencies of the class and it made all its methods better by taking out the code that deals with the details of writing a file.

Now, is the moment to think about a better abstraction for the interface we have created: the IPageFileWriter. We could make it more abstract if we rename it and take out or at least rename the filePrefix parameter of its method. The interface in its form limits its implementation to writing to files. A better one would be:

public interface IPageWriter  
{  
    bool Write(PageXml page);  
}  

This interface may have implementations that could write the XML anywhere. By increasing this abstraction we are increasing the reusability of the PageXmlExport class, by being able to reuse the same code and logic to export in multiple mediums. We are not going to pursue this refactoring path, but we are going to continue with increasing the cohesion of the PageXmlExport.

Lets see other groups of fields that could be extracted. Looking at maxSalesOrders or addCustomerDetails it is hard to see with which to group them. On the other hand we observe that crmService and locationService are used by the functions that enrich the XML with external data. Therefore one idea is to group these two together in a new depenedency. Because it gives data for the export class we name it as IExportDataProvider interface. It results the following code (file 04_PageXmlExport.cs)

 // file: http://tinyurl.com/04-PageXmlExport-cs  
public interface IExportDataProvider  
{  
   CustomerInfo GetCustomerInfo(string name);  
   Coordinates GetCoordinates(string city, string street, string number);  
}

public class PageXmlExport  
{  
   private readonly IPageFileWriter fileWriter;  
   private readonly IExportDataProvider dataProvider;
  
   private readonly int maxSalesOrders; // used in 4/5 methods  
   private readonly bool addCustomerDetails; // used in 2/5 methods
  
   public PageXmlExport( IPageFileWriter fileWriter,  
			int maxSalesOrders,  
			bool addCustomerDetails,  
			IExportDataProvider dataProvider)  
   { … }  
   …  
}  

The number of fields and dependences got smaller after this step. However, the code of the class did not improve much. It is the same, but instead calling functions from ICrmService respectively ILocationService, now all the calls go to the same IExportDataProvider. IExportDataProvider implementations will wrap together the external services for the export class, but something doesn’t smell good about it. The first bad signal comes from the name. It is named data provider, but it also gets geo location coordinates. Looking  at it from another perspective, we can also see that the interface mixes different levels of abstraction. The CustomerInfo is a high level concept in comparison with Coordinates.

Looking again at the resulted code in PageXmlExport class, we see that it now has this IExportDataProvider, but it also uses a repository to get data from the database. Maybe a better idea would have been to wrap also the repository in the IExportDataProvider. The repository is another dependency our class has, but it is not that visible. The refactoring that we have done made us think to the dependencies more, and now with the IExportDataProvider as a visible dependency in the constructor and the repository being used in most of the methods, it is much more obvious that they should be grouped together as one dependency.

Lets rollback the last refactor step that we did, and wrap the ICrmService together with the repository in a real IExportDataProvider. Thinking on doing this we also observe that the other dependencies our class has (maxSalesOrders and addCustomerDetails settings) are only used together with the hidden dependency towards the repository. They all could be wrapped into the IExportDataProvider. The IExportDataProvider implementations should depend on these settings and them should instruct the repository accordingly. In fact, by doing this we are taking out from the PageXmlExport class the responsibility of being setup about how to read data and we push that lower to IExportDataProvider implementations. The resulted code after this refactor step is the following (file 05_PageXmlExport.cs).

 // file http://tinyurl.com/05-PageXmlExport-cs  
public class PageXmlExport  
{  
  private readonly IPageFileWriter fileWriter;  
  private readonly IExportDataProvider_5 dataProvider;  
  private readonly ILocationService locationService;
  
  public PageXmlExport( IPageFileWriter fileWriter,  
			IExportDataProvider dataProvider,  
			ILocationService locationService)  
  {  
     this.fileWriter = fileWriter;  
     this.dataProvider = dataProvider;  
     this.locationService = locationService;  
  }
  
  public bool ExportCustomerPage(string customerName)  
  {  
     PageXml content = new PageXml {Customer = new CustomerXml {Name = customerName}};
  
     IEnumerable<CustomerData> orders = dataProvider.GetCustomerOrders(customerName);
  
     // enrich content with orders  
     // ..
     
     // enrich content with customer data  
     // ..
  
    return fileWriter.WriteFile(content, "CustomerPage");  
  }
  
  public bool ExportCustomerPageWithExternalData(  
			string customerName,  
			PageData externalData)  
  {  
     PageXml content = new PageXml {Customer = new CustomerXml {Name = customerName}};
  
     if (externalData.CustomerData != null)  
     {  
         // enrich with externalData.CustomerData  
         // …  
     }  
     else  
     {  
         CustomerInfo customerData =  
         dataProvider.GetCustomerInfo(content.Customer.Name);  
     }
  
     IEnumerable<CustomerData> orders = dataProvider.GetCustomerOrders(customerName);
     
     // enrich content with orders  
     // …
     
     // enrich content by merging the external customer data with what read from DB  
     // …
  
     foreach (var address in content.Customer.Addresses)  
     {  
        Coordinates coordinates = locationService.GetCoordinates(address.City, address.Street, address.Number)   ;     
        if (coordinates != null)  
           address.Coordinates = string.Format("{0},{1}", coordinates.Latitude, coordinates.Longitude); 
     }
     
     return fileWriter.WriteFile(content);  
  }
  
  public bool ExportOrders(string customerName)  
  {  
     PageXml content = new PageXml {Customer = new CustomerXml {Name = customerName}};
     
     IEnumerable<CustomerData> orders =  dataProvider.GetCustomerOrders(customerName);
  
     //enrich content with orders
  
     return fileWriter.WriteFile(content);  
  }  
  …  
}  

Looking now at the result, we are definitely in a much better place than when we were when we started. Maybe our code design is not perfect nor the best, but it is definitely better. Now our PageXmlExport class has three dependencies, which are used by most of its functions. Most of the code in the class got smaller and simpler. The methods follow a simple pattern: they get data using the IExportDataProvider, they enrich the content XML with it (which is the core functionality and knowledge of our class), and they output the result. The details of reading data and output the result are not important. Our class can focus on XML enrichment only.

Maybe we do not yet have the best abstractions in place, but now after we have separated better the concerns it is a good moment to think more about abstraction and encapsulation. The code that gets data from the database or other sources is now decoupled from the core code that enriches the XML. We have achieved the same decoupling for the code that writes the XML on the disk. The details of how these are done are no longer mixed with XML enrichment logic and by abstracting them we can now reuse the XML enrichment code to read data from any medium and output the results to any medium. In conclusion we have achieved a better Separation of Concerns, and a more loosely coupled design, by following a simple refactoring pattern. We did this almost in a mechanical way looking to reduce the number of parameters or the number of dependencies.

Part 3: summarizing the refactoring pattern

If we recap the refactoring pattern we have followed to achieve loose coupling by improving coherence we have the following steps or recommendations:

1. Be critical with functions that have more than about 4 parameters. It may be that they are part of poor cohesive classes or in they have a poor cohesion themselves.
   • reduce the number of parameters of functions by transforming the common (redundant) ones into class fields received through the constructor.
   • do this in small refactoring steps
   • this may reveal a class with poor cohesion
2. Be critical with classes that contain more than about 7±2 fields (more toward the high end of 7±2 if the fields are primitives and more toward the lower end of 7±2 if the fields are pointers to complex objects or services)
   • the number 7±2 has been found to be a number of discrete items a person can remember while performing other tasks
   • reduce the number of fields by grouping the ones that are used by same functions into one new field. This leads to taking out code, and it increases the coherence of the original class
3. Be critical with classes that have more than about 4±1 dependencies - most probably they have poor cohesion
   • most probably the dependencies may be reduced by grouping and extracting them in new classes. This increases the cohesion and leads to a loosely coupled code
   • think about putting in place good abstractions for new extracted classes. This will increase the reusability and extensibility of the code.
   • think about different levels of abstractions and different levels reasons of change when you try to improve the abstractions that define the interaction between your classes

By following these with the above example we have refactored from a class that mixes code with details of creating and writing into files, with code that implements high level business logic (XML enrichment) and with code that deals with data access concerns, into a more decoupled design where all these concerns are separated and may be abstracted better.

many examples like the above are included in my Code Design Training

Recently, I’ve went again through the discussion of how the Repository Pattern works with Dependency Injection (DI) in one of the projects I’m involved in. Even if these patterns are around for a while and there are many examples on how to use them together, discussing some particular implementation aspects is still interesting. So, I think it might help to go once again through it, maybe from a different angle.

The case I want to focus on, is when the repository implementation is abstracted through a generic interface, and its implementation is injected through DI to the classes that need to read or store data.

Using DI in an application it is often a good idea. It is an easy way to follow the Separate Construction and Configuration from Use principle (Uncle Bob explains it in his Clean Code book and also Martin Fowler does it here). Abstracting the repository through an interface is also good. It doesn’t only benefit you in writing isolated Unit Tests, but it brings a good separation of the data access concerns from the business logic. It prevents the data access specifics to leak into the upper layers, if you encapsulate them well into the implementation of the interface. Using them together is also good, but you need to consider the stateful nature of a repository and the deferred execution of an IQueryable.

The question I want to address is who disposes my repository implementation and when. The repository will use some connection to the storage, which should be released when is no longer needed. And that should be in a deterministic way, not when the garbage collector kicks in. If the implementation of my repository is through the Entity Framework (EF), it means that I should dispose the DbContext when I no longer need it. For the EF case this may not be too problematic since it does a clever connection management. Here is a response of the EF team on this. However, it may not be the same for other ORMs. Even more I think that IDisposable pattern should be correctly implemented and followed in all cases. I read IDisposable as a clear statement that says that the instances of its implementation must be cleaned in a deterministic way.  I don’t think it’s a good idea to just leave instances undisposed, because I may risk nasty leaks.

I prefer to have an open interface for my repository by taking the benefits the IQueryable brings. This means that the user of my repository can write or alter the query by itself, and I will not need to add a new methods to my repository each time a new use case needs a different kind of filtering or a different data projection. This is easily achieved with an interface like this:

interface IRepository  
{  
	Queryable<T> GetEntities<T>();    
	…  
}  

The clients of my repository may add additional filtering or data selection before the query is actually executed on the database server. Combining it with DI we may have class services like this using it:

class SalesOrdersService : ISalesOrdersService  
{  
 // The concern of calling repository.Dispose()  
 // is not in this service  
 private readonly IRepository repository;  
 …

public SalesOrdersService(IRepository repository, …)  
{  
   this.repository = repository;  
   …  
}

public decimal GetHighRiskOrdersAmount(int year)  
{  
    IQueryable<Order> orders = GetHighValueOrders()  
    		                    .Where(o => o.Year == year);
   
    decimal amount = 0;  
    foreach (var order in orders)  
    {  
         if (IsHighRisk(order))  
         amount += order.OrderLines.Sum(ol => ol.Amount);  
         // only read DB the order lines of high risk orders.  
         // important if expect that only 10% of orders are high risk  
    }

    return amount;  
 }

public int CountHighRiskOrders(int startWithYear, int endWithYear)  
{  
    IQueryable<Order> orders = GetHighValueOrders()  
 								.Where(o =>  o.Year >= startWithYear &&  
 											 o.Year <= endWithYear);

    // lets say I’m keen on performance and I only want to iterate  
    //    once through the resultset.  
    // If I would use the return order.ToArray().Count(IsHighRisk),  
    //    instead of the foreach, there is one iteration for ToArray 
	//    and one for the Count()  
    int count = 0;  
    foreach (var o in orders)  
    {  
        if (IsHighRisk(o))  
        count++;  
    }
   
    return count;  
 }

 public IEnumerable<Order> GetOrders(int startingWith, int endingWith)  
 {  
    return GetHighValueOrders()  
              .Where(order => order.Year >= startingWith &&  
              order.Year <= endingWith);

  // I have the flexibility to:  
  // -return IEnumerable, so the query executes when iterated first  
  // -return IQueryable, so the query can be enriched by the client before execution  
  // -return IEnumerable, but with an List underneath so the query executes here (call .ToList())  
 }

 // This functions makes reuseable the is HighValue evaluation.  
 // The evaluation of the condition can be translated through  
 // LINQ to SQL into a WHERE filtering which runs in the database  
 private IQueryable<Order> GetHighValueOrders()  
 {
       var orders = from o in repository.GetEntities<Order>()  
                          join ol in repository.GetEntities<OrderLine>() on o.Id equals ol.OrderId  
                    where ol.Amount > 100  
                    select o;

       return orders;  
  }  
 …  
}  

With this approach, I also get the benefit that for a new use case, I could maybe only wire-up in a different way some of my existent services, reusing them, and all will get through DI the same instance of the repository. Therefore, the underlying implementation has the opportunity to send all the queries from one session / request on the same storage connection. Even more, if my services return the IQueryable as IEnumerable, the query will be executed only when the client needs it.

Coming back to disposing the underling DbContext, one option is to make the repository implementation to also implement IDisposable, and to call DbContext.Dispose() when it is disposed:

class Repository : IRepository, IDisposable  
{  
   private MyDbContext context = new MyDbContext();

   public IQueryable<T> GetEntities<T>()  
   {  
      return context.Set<T>().AsNoTracking();  
   }

   public void Dispose()  
   {  
      context.Dispose();  
   }  
 …  
 }  

Now, I have all the benefits I described above, but who is going to call the Dispose() of my repository implementation. The services should not do it. They do not know that the implementation they got injected is an IDisposable and they shouldn’t know it. Making them also to implement IDisposable, and to check on Dispose() if some of their members are IDisposable, is not a solution either. My rule of thumb is that the code that creates an IDisposable instance, should also call dispose on it. It is also according with the Code Analyses Rules. In our case this is the DI Container. Our code does not create, nor explicitly asks for the instance, it is just injected into it. By using DI I am inverting the control of creating instances from my own code to the framework. The framework should also clean them up when no longer needed.

For this, we need to make sure that two things happen:

1. the DIC will call Dispose() on all the disposable instances it created, and
2. it will do it in a deterministic way (when no longer needed, meaning when the request  / session ends)

If you’re using Unity Container you have to take care of both. When an instance of the UnityContainer is disposed, it will call Dispose() on all the lifetime managers which are IDisposable. However, the built-in lifetime managers Unity provides for short lived instances, are not disposable, so you need to write your own. Here are some examples on how to do it. Other DIC, like MEF, have this built in.  The other thing to care of is: when the Dispose() call chain will be kicked off. For this you need to use Scoped Containers (aka Container Hierarchies). In short, this means that you will need to associate a request / session with a child container instance, and dispose that child container when the request ends. The dispose of the child container will trigger the dispose of all IDisposable instances it created. Here is a simple example on how to do this for an ASP.NET MVC application, where a child container is associated with each HTTP Request.

Even if this approach gives a lot of flexibility and advantages, is not easy to setup. It requires some non-trivial code to be written. In some applications the added complexity may not be justified. Let’s explore other ideas, too.

We could make the IRepository interface to be IDisposable and not use DI to get the repository implementation, but use Service Locator instead. The main difference from the above is that we are no longer inverting the control. Our code is now in charge of explicitly ask for a repository, so it should also take care of cleaning it up when it is no longer needed. Now, we don’t need to go through all the trouble of making the DI to call Dispose(), because our code will do it. The services now can use a using statement for each repository instance, like:

class SalesOrdersService : ISalesOrdersService  
{  
  private readonly IServiceLocator sl;

  public SalesOrdersService(IServiceLocator serviceLocator)  
  {  
      this.sl = serviceLocator;  
  }

  public decimal GetHighRiskOrdersAmount(int year)  
  {  
     using (IRepository repository = sl.GetInstance<IRepository>())  
     {  
        IQueryable<Order> orders = GetHighValueOrders(repository)  
                 .Where(o => o.Year == year);

        decimal amount = 0;  
        foreach (var order in orders)  
 	    {  
 			if (IsHighRisk(order))  
 				ammount += order.OrderLines.Sum(ol => ol.Ammount);  
 	    }

 	    return ammount;  
 	 }  
  }

  public IEnumerable<Order> GetOrders(int startingWith, int endingWith)  
  {  
      using (IRepository rep = sl.GetInstance<IRepository>())  
      {  
      		return GetHighValueOrders(rep)  
      					.Where( order => order.Year >= startingWith &  
      							order.Year <= endingWith);  
      }  
  // Is the above code correct? Have another look
 
  // the returned IEnumerable will be enumerated by the caller  
  // code after the rep.Dispose() –> error.
 
  // To avoid this more (uncontrollable) approaches may be taken  
  // –> error prone –> low maintainability
 
  // -receive the repository instance created by the caller code.  
  // The caller code creates it, so it needs to dispose it, not me
 
  // -call .ToList() on this, but still return it as IEnumerable  
  // What happens, when the caller iterates the  
  // order.OrderLines? –> error
 
  // -this function is refactored to only send a query,  
  // which should be wrapped and executed by the caller  
}  

We are losing some of the flexibility and advantages of the above. The main one is that now an IQueryable issued by the repository is scoped to the using statement, not to the request / session. If it is passed as return value to another class, and it gets executed after the using statement ends, an error will occur because the underlying connection is closed. Therefore, I need to be careful that the return values of the services that use a repository should be executed. This means that my clients cannot add to the query additional filters, nor they can decide when / if to execute it. This reduces the flexibility and reusability of my code. This may not be that bad in some applications. One of the biggest risks I see with this is that by trying to workaround these limitations, there may be code that passes around an undisposed repository to different classes or functions (like GetHighValuesOrders() function has). Without discipline, this may get out of hand.

Another different approach is not to make the repository implementation disposable, and to dispose the underlying connection (DbContext), immediately after it is no longer needed. This implies that the IQueryable (better said LINQ to SQL), will not leave the repository scope. This makes the repositories to be use case specific, with a close interface.

class OrdersRepository : IOrdersRepository  
{  
 	public IEnumerable<Order> GetOrders(int startingWith,  
 	int endingWith,  
 	decimal highValueAmount)  
 	{  
 		using (var db = new MyDbContext())  
 		{  
 			var orders = from o in db.Orders  
 				join ol in db.GetEntities<OrderLine>() on o.Id equals ol.OrderId  
 				where 	o.Year >= startingWith && 
				 		o.Year <= endingWith &&	
						ol.Amount > highValueAmount  
 				select o;
		 
 			return orders.ToList();  
 		}  
 	}  
 	…  
}  

This approach is the most rigid and it makes me add new functions and new repositories each time a functionality is added or changed. I usually do not use it, unless I am dealing with a storage which doesn’t support LINQ.

I prefer to implement the first approach described above, even more in large applications, with many use cases. It is a better choice when you are dealing with more types of implementations which use expensive external resources, which need to be released as soon as possible in deterministic way. In other words, I am addressing with this all the IDisposable implementations (not only the repository), which may be injected through DI.

The first approach and the second one are not exclusive. I may get through DI the repository for some cases and use the service locator where I explicitly want to be more closed to my clients and I would need to dispose the repository even sooner than the request / session would end. I very often use them together: the repository through DI for read only cases, where I would want to more flexibility (more reads on same repository instance), and the repository through Service Locator for read write cases, where I want a smaller scope for my unit of work.

many design discussions like the above are included in my Code Design Training

Unit Testing has been one of the dearest technical subjects for me in the past years. A great influencer was a TDD Workshop lead by J. B. Rainsberger to which I was lucky enough to attend, somewhere in 2010. It was a very good fit with the moment of my career. I was starting to realize that we are quite far in the way we are developing software from the principles and disciplines advocated by different industry leaders. I was looking for concrete methods to improve the way me and my team colleagues w software.

JB not only that has taught us what Unit Testing and TDD are, but he also demonstrated a powerful technique: getting to a modular design driven by unit tests and by some simple rules. He showed how to achieve a design that is inexpensive to change, by minimizing the accidental complexity. This has made me to see in unit testing one of the most efficient tools for assuring a sustainable quality level of the code in the projects I was involved in. It motivated me to teach others to write Unit Tests that benefit them in the quality level of their production code.

I wanted that the next projects I will start to work on to begin with a training on unit testing. I wanted that the entire team writes and cares about unit testing in a consistent manner. This is how my Unit Testing training was born. I have initially developed it for a newly created team that was starting a challenging project on a tight schedule. I knew from the start that it will not be possible for the very few seniors to review enough and do enough pair-programming with everyone, so we can have a sustainable quality level in our code. We needed a flexible design which could accommodate the volatile requirements we were supposed to deal with in a predictable way and at reasonable costs. I thought that if everyone would cover with good written unit tests their code, than we would end up with a design that is good enough. Maybe not the best one, but good enough. And, it worked! Even if we were on a tight schedule, the initial investment in learning and writing unit testing, paid off in the end.

Since then I have redone the training many times for different teams in ISDC and for some of their clients. We have a seen many benefits not only at the project startup, but also for ongoing projects with existent teams. It is always about reading and changing code, rather than writing code. Unit testing, if done the right way, can significantly increase the efficiency and predictability of the changes we do in code.

Another motivation to build this training was to share with others my experiences of learning unit testing. For many teams it is very important to start on the right track with unit testing. When unit testing is done incorrectly it can cause schedules to slip, waste of time, and it can low motivation, and code quality. It is a double-edge sword, which many teams learn to master the hard way. One of the lessons I’ve learned the hard way was: “Never treat your unit tests as second class citizens”. It was in one of the first projects I was learning unit testing on. We’ve ended up after a few weeks with a tests suite, which was breaking at any changes we were doing to the production code. We couldn’t refactor or redesign the production code, because the tests would break. It wasn’t because of poor design of the production code side, it was because of the poor quality of our unit testing code. We have created a too strong coupling in test code, which turned out hurting in the production code as well. I and my team colleague ended up in drawing sticks on who is going to spend the weekend to fix the tests. I wouldn’t want anyone to repeat this painful experience.

In the past few weeks I have taken the time to review, refresh, enrich and restructure the entire training material of my course. I have added the topics and examples for which, until now, I never had the time to detail them as I wanted.

At start, the content was largely built based on Roy Osherove’s The Art of Unit Testing book. Now I have enriched it with new elements from the books and articles of Kent Beck or Martin Fowler and others. I have also taken to opportunity to reshape the training with my experiences from last few years and to add the new things I have learned from pairing with different people at code-retreats or on other occasions.

The training focuses on two main aspects: maintainability of the tests and the benefits on the quality of the production code. The tests code may be a significant amount from all the lines of code forming a project. However, even if it is big in size it is important that it does not increase the complexity, so it has low maintainability costs. The greatest benefit I see from doing Good Unit Tests is in the design of the production code. By learning a few aspects about how to write your tests you can exercise a positive pressure on your production code leading it towards a better design.

UPDATE: I have published a presentation page with the full format of the entire course here.

Most of the times when I talk about unit testing or I ask encourage my colleagues to write unit tests, I emphasis on writing GOOD Unit Tests which are easy to write, they test only one thing and they run in isolation. I always make a clear distinction between these very granular and highly isolated unit tests, which I call GOOD Unit Tests and the Integration Tests. I roughly name Integration Tests any automated test that has more than one point of failure or it exercises more external objects (other classes objects, services, databases, etc.). These tests verify that all those pieces work well together. What I want to focus on instead, are the very granular unit tests, which verify the basic correctness of each unit in isolation and they assert only one thing. When such test fails I know exactly where the problem is, from the test name only.

The greatest benefit we get from writing GOOD Unit Tests, which are easy to write, they test only one thing and they run in isolation, is a better code design that results into the production code. These unit tests put positive pressure on the production code making it better. I am not chasing a bug free code, nor a full coverage in regression testing with these unit tests, but a higher quality code design when I am working with teams that are less experienced in doing a good design.

Yes, there are drawbacks as well. We might get into the other extreme and have the production code design suffer from this granular unit testing, resulting into too small classes, with code smells like feature envy or unnecessary complexity. I totally see the point of DHH during the ‘Is TDD Dead?’ debate, that this might lead to creating interfaces or abstract classes only for the sake of mocking. Indeed, it may get to making interfaces which do not create abstractions and which break encapsulation. Yes, granular unit testing will not help in preventing these, on the contrary. When our code design falls into being too granular, practices like eliminate duplication and reduce the dependencies of a class while refactoring, plus a consistent code structure, are helpful to keep things in balance. In the end it all depends on your context: what problem you want to solve, which are your constraints, how experienced is your team and many other aspects.

We couple things by nature. It is more common to put everything in one class or one function rather than thinking about separating the concerns by factors of change. In most of the cases when I am looking over a poorly designed code I am seeing too much coupling (or coupling without benefits) and poor cohesion. Everything is done in one or two big classes. In all most of the cases it is clear that if someone would have tried to write GOOD Unit Tests on that code, it would have been forced to refactor it towards a better design. It wouldn’t be possible to write small unit tests, otherwise. This would have turned the code into a better one. Maybe not the best design, but certainly better. It would break the one thing into more, and the breaking would be driven by test cases, which leads to a good separation of concerns. I often see that the GOOD Unit Tests make the design to move from one like this

towards one like this

which is clearly better than the first one, maybe even better than this

, and clearly better than this

, which is at the other extreme than the first one. We want to balance things and to get somewhere in the middle. (These design examples are copied from Juval Lowy presentation on ‘Modular Application Design’, which I value a lot.)

Another important advantage of the Good Unit Tests is that they are easy to write and to maintain, therefore they have low costs. For example if we consider the following setup:

where, function f() of class A, calls g() of class B, which in its turn calls h() of class C. After the return of g(), f() may make a call to h’() depending of the result of g(). A quite common setup in an OO program. Now, if we want to write a test to verify the correctness of f() and only f(), it will not be easy. When we write the arrange part of the test, we need to take into account the preconditions and post conditions of all the functions, and find the test data that will make the code flow to be the one we want in our test scenario. This will make our arrange code hard to write. When we assert, things it will also be difficult. The expected result of f() may need to take into account calculations made by the other functions as well. Even if we manage to write the test, it will be hard to maintain. Also, there are high changes for this test to break when refactoring or other changes occur to classes B or C. Another, disadvantage of such integration test is that if it fails we cannot clearly say where the bug is. It may be in classes B or C even though those are not the classes under our test. As a conclusion, this kind of test has high costs (difficult to write and maintain) and low benefits (I don’t know where the bug is, when it fails).

The alternative, is to refactor my code and change B and C with interfaces which abstract them. This will allow me to replace them with fakes (mocks or stubs) in my tests, which are totally under the control of my test code. Now I can easily write many GOOD Unit Test for class A, in isolation to verify its basic correctness. I can write one test to verify ONE thing. Looking more closely to the production code, we’ll see design patterns and SOLID principles that may emerge from this refactoring. For example, we’ll tend to program against interfaces, we’ll depend on abstractions and not on implementation details (partially what DIP says), we’ll have our class dependencies visible, we’ll move towards a more extensible and reusable code (partially what OCP says).

It is good to focus to cover with unit test all the logic and calculations. This means the code that has ifs, whiles, fors, etc. and mathematical or other data structure operations. Covering plumbing code, data conversion or basic variable assignment brings little value. When we strive to achieve a good coverage for the logic, because it is easier to fully cover it with GOOD Unit Test rather than Integration Tests, another positive effect happens: the logic gets pushed away from the external frameworks and libraries. The external libraries tend to get abstracted, and our core logic separated from the plumbing code that integrates them. This leads to a better separation of concerns. Our business logic code tends to be pushed away from the presentation and from the data access towards the middle. Wasn’t this something we’ve always wanted?

Another aspect of GOOD Unit Tests is that we can fully and easily test all the scenarios that the code needs to handle in a very detailed way. I usually think of the car running example. If the car does not move, it is clearly broken. However, we don’t know what part is not working, so we need to verify each part separately to find and fix the problem. Moreover, even if the car moves, we can’t say by this integration test only, if all the parts function at correct parameters. It may be that only after driving more than 1000 km without stopping the engine will get overheated, because of a malfunction in the cooling parts. Coming back to code, I rely on Good Unit Tests to verify the basic correctness in detail, with high coverage. Like in the car example, there may be scenarios that I cannot test otherwise or I would just ignore because it is too hard to be done with integration test. For instance how would you test that your code behaves correctly when the hard-disk is full? Would you fill it to run a tests? Can you do that on the CI server?

Therefore, we target a high coverage with unit tests, and we verify that each of the services work well in isolation.  The unit tests can go in detail at lower costs. Then on top we may have few Integration Tests, which do not go in many details because they are costly. They just check that all the services and components can work together and that all the configurations are consistent. The integration tests usually follow some happy flow scenarios and focus that all the components they touch are running. They do not target correctness as the unit tests do. Below figure, shows three classes fully tested in isolation by unit tests (blue arrows) and two integration tests (purple arrows) on top, which touch all the working components, but fewer code paths.

In theory both with isolated Unit Tests and Integration Tests we could reach a full coverage of the entire code we write. With unit tests we need to write many pairs of isolated Collaboration Tests and Contract Tests to reach a full coverage. With Integration Tests we need to be able to specify and simulate all the test scenarios. In general, the costs of writing and maintaining such integration tests which cover all the details, grows exponentially and they are very fragile to change. Any change in the system is very likely to break them, which makes them not feasible. That is why we need to test on intermediate results. We break the end-to-end tests in smaller tests, until we get to the Good Unit Tests, which are at the lowest level checking each detail. By doing this, not only that we get to full a high coverage at lower costs, but we also get better production code in the process. Integration Tests rarely bring any benefits to the design of the production code. However, they contribute to the test suite covering the component integration and the configurations consistency.

To conclude, I always treat my unit test code as first class citizen. It is separated from production or other test code. I like my unit tests to be small, easy to write, to test only one thing and to run in isolation. The highest benefit I want from them is to put pressure on my production code to make it of better quality. I write them in very short cycles, with the production code, even if I do code first, because I want to refactor in small steps. I rely on them for basic correctness and I target a high coverage of the code that does logic and calculations. Separately, I write different levels of Integration Tests. They check that more services or components (which I already know are functioning well in isolation) can work together. They may also check interactions with external services, frameworks or data sources. Sometimes, I also have integration tests for regression testing, performance or load testing.

More discussions about writing good unit tets are part of my Unit Testing Training

It was few months ago more than a year ago (22^nd of February 2013 in my Trello card, Ideas column), when I came up with the thought of developing a training about how to write good code. At that time I was doing a lot of thinking about how to best use my one extra free day, as I started to work part time. The idea to use part of this time to develop trainings came rather naturally as I was already teaching and coaching colleagues about good code and also holding trainings about good unit tests in ISDC and for some of their clients. I also realized that holding trainings was a really enjoyable experience for me, so the fit was quite perfect.

What I intended with this kind of course is to express and explain, how all the best practices from the object oriented design world are structured in my mind. It is about how my links and connections among these various design principles and patterns results in the code I write. During my previous coaching sessions, I realized that explaining my vision helped more than just presenting the theoretical concepts. It helped both to get to a better understanding of their usefulness and in writing better code. The Code Design training was born from this desire: put in a clear map all the patterns, principles and practices, and present them using code examples from real projects to any team from any company.

The entire training is structured around of the idea of designing code that embraces change. Code that is easy to change. Code that is inexpensive to change. Code with predictable effects of change. Changeable code. Change is the leitmotif of my training. What I want to teach is how to drive all the design decisions to minimize the effect of change. That is why this is not an Object Oriented Design training. It is not only about that. It is a Code Design training that focuses on minimizing the cost of change. Because, at the end of the day, the most important benefit quality code is the low cost of change.

I have started to actually work on developing this training somewhere in October 2013 and I worked regularly in my free days left from the regular part time job. I have taken an interactive approach. First I have put on paper the main ideas and topics I wanted to teach. Then I have defined the learning objectives and I have also summarized what would one learn from this course. Next was to put on paper a draft script for the entire course. After that I have  enriched this draft in an iterative way, by identifying materials that would lead to reaching the learning objectives. I have gathered concepts, examples, explanations and ideas from my favorite books, like: Clean Code: A Handbook of Agile Software Craftsmanship or Code Complete: A practical handbook of software construction, and others that I have read along the years. It ended up to be a selection of concepts and examples from these books structured, interpreted and explained in my own way shaped by my own experiences of writing code and explaining it to others. At some point the training lessons got the shape I was looking for. Each lesson was targeting roughly one or more of the objectives. In the end the training lessons are:

• Part 1: Good Code Design and Design Patterns
  • Lesson 1: Why Code Quality and Code Design
  • Lesson 2: Clean Code
  • Lesson 3: Separation of Concerns
  • Lesson 4: Design Patterns
• Part 2: Object Oriented Design Principles and Practices
  • Lesson 5: Object Oriented Design Principles
  • Lesson 6: Dependency Injection
  • Lesson 7: From Principles and Patterns to Practices
  • Lesson 8: Application Infrastructure

Somewhere along the way I have started to think more seriously about another important stakeholder in this: the companies that would ask me to hold this training for their employees. I have made a commercial offer for the training, which details the outcomes, the benefits and the costs of the training. I have also considered more carefully how I would customize the training for different needs. The training lessons are built in such way that they can be taught individually and still bring value, or only a part of them can be selected into a shorter version of the course for an audience that doesn’t need to go through all the subjects.  Having the offer done, I have sent it through my contacts, to some companies in Cluj. I have got good feedback, which encouraged me even more to dedicate more time and energy to developing the training.

Once I had the course script complete enough, I have started to build the training materials: power point slides, exercises, code samples, and to select videos which can contribute to the learning experience.

Tomorrow is the first time I will hold it as an internal company training, for Wirtek. A company here in Cluj, which shows high desire to increase the quality of their services by investing in people. In a way I am glad it happened that the first run of the training is not in the company I regularly work for, because it is in a new context, which makes it a better test for me and for the training itself. I am quite confident that it will go well, and that it will help them, because I have put a lot of effort, energy and thought in it. I’m looking forward to a great day tomorrow!

UPDATE: I have published a presentation page with the full format of the entire course here.

Tomorrow the 4^th edition of IT Camp, starts in Cluj-Napoca. It’s a good time to stop from day to day work and look towards the community to share and to learn from the other fellow developer’s experiences.

This year I am going to add to the story of how to reach a good Quality Code Design, which can sustain your project over the time. Last year I have talked about why quality is important in software and how it can be achieved by benefiting from the positive pressure the GOOD Unit Tests put on our design. This year I am talking about the other part: the Application Infrastructure. They both go hand in hand, by completing each other, in leading your code towards a good quality that can pass the test of time. Usually the Application Infrastructure comes first, but I am saying the story in reverse order.

So, if you’ve seen my last year talk come Friday morning in Verdi room to hear the first part of the story. If you didn’t see it yet, you can see the recording here, after Friday, so you can get the story in its natural order.

See you at IT Camp! Have a great conference!

PS:
I hope the title: “Quality Code through Application Software Infrastructure” doesn’t mislead you in thinking that it is going to be an IT Pro talk about servers and operating systems. Noting like that. It’s going to be about Code Design.

IT Camp happened at the end of this week in Cluj. I am proud that the town I am living in hosts the biggest premium conference on Microsoft technologies in Romania. It gathered almost 400 participants, from which a significant part were from other cities. My impression was that most of them were experienced software professionals. I hope to see in next editions people coming also from neighbor countries. I think it worth the trip. All the sessions are in English, there are a lot of international speakers and there is a great opportunity to meet experienced professionals from Microsoft technologies world.

The first day was opened by two inspirational keynotes given by Richard Campbell and Tim Huckaby. I was impressed about how friendly, open and close to the audience Richard is. I think anyone could approach him during the breaks and lunch for any questions or discussions. Same goes for Tim. It was a rare opportunity to meet and speak to such great guys. Thanks for being here!

I was honored to be one of the speakers. I am glad that I have had the opportunity to speak in front of a large audience and to share my experience. I was a bit nervous to have my session in the 2^nd day, on the ‘Architecture and Best Practices’ track, after notorious speakers like: Martin Kulov, Peter Leeson, Richard Campbell and Sergiu Damian. However, my story on how you can get to high code quality by doing good unit tests, fitted well after the talks of Peter, Richard and Sergiu. Peter and Richard touched other faces of quality in software. My good friend and RABS colleague, Sergiu walked us through architect’s challenges. I think, I added to both subjects by talking about code quality and the challenge to get there with the whole team. I had a good time speaking, I liked it and I’ll do it again. I hope I’ve inspired my audience in a way or another, and even if maybe some were not convinced about the technique I presented, I hope that they went away with a new perspective, maybe a new author to read or at least with a good story. I was very happy to hear someone that one of his takeaways from the conference was one of the key points of my talk. It made me to feel good J

My slides are published here. I did not have the time to include a code demo in this talk, but I will do a full demonstration on how good unit tests put a positive pressure on the production code design, next week in Bucharest at I TAKE Unconference as an Open Session.

One of the sessions I liked a lot was the one given by Peter Leeson. It was perfect! I hope to see him at next editions or other similar events in Romania. I hope that some companies active in Cluj or Romania were inspired to pay more attention to improving the quality of their products or services, because in the end this will be the only thing that would differentiate us from cheaper services in eastern countries or Asia.

All the sessions from IT Camp were recorded and will be available soon on the conference web site.

IT Camp was for me a great experience. I’ve met great people and I made some new friends and contacts. Big thanks to Mihai and Tudy for organizing this, thank you for giving me the opportunity to speak and thanks you all for attending. See you next year to an even better edition!

I will speak this year to ITCamp conference on 23^rd of May.

When I have received the invitation (thanks Mihai and Tudy for inviting me) my first thought was whether I should talk about Unit Testing or something else. Then I started to wander why do I like to talk that much about it, so much that I am seen as the unit tests guy by my colleagues, all the projects I’m involved in have a high accent on unit tests, and I even receive birthday cards with: “no unit tests? You’re doing it wrong!”. There are definitely other interesting subjects to talk about. Then I realized it. Unit testing was the only technique with which I managed to persuade one of my teams to write good quality code, which allowed us to easily change it. All the talks about Design Patterns, programming principles didn’t do it. Unit testing did it.

Therefore, this year I will talk at ITCamp about how to drive your team towards quality code, by writing good unit tests. Today I have started to put on paper some ideas on how to structure and build my talk. I have few more free Fridays to refine it and make it a good talk in which to show my view on how to achieve good code and why would you want that. I hope to see you there!

On December the 8^th, we had the Global Day of Code Retreat in Cluj-Napoca. We have joined the other 150 cities all over the world in a great global event.

My first contact with a code retreat was almost a year ago when some friends from iQuest invited me to join them for a code retreat organized internally. I loved it. I liked the idea, the format, and the most the fact that I had the opportunity to pair with developers I didn’t know and to experience different programming languages. Since then I had the idea to organize with RABS a large code retreat with a lots of developers. When somewhere in October, I heard that there is going to be a global day of code retreat, I said that this is the best opportunity to organize it, so I’ve registered Cluj, among the other locations in the world. I wanted diversity: many programming languages and many developers with different experiences and views. We’ve succeeded: we had 70 developers who programmed in 8 languages: C#, Java, C, C++, javascript, Ruby, Python and F#.

I needed help. I’ve asked Andrei Olar, who facilitated the code retreat at iQuest to take over the facilitator role. He gladly accepted. Knowing we are going to be more than 50, we have also asked Costin Morariu who facilitated more code retreats, to be another facilitator. Even if he lives in Sibiu, he said yes. I was the third facilitator and Sergiu took over the host responsibilities. We decided at RABS that we are going to organize it, but we wanted to have an event for all developers, regardless the technology or experience, so we asked the major programmers communities in Cluj: Code Camp, JUG, Cluj.rb and RONUA to join us as co-organizers. They all accepted our invitation, so we’ve published the event and started registrations. We’ve chosen Cluj Cowork to be our host. It was perfect for such event (lot of space and more rooms). After a few days we had over 50 people registered. When we’ve reached 80, the maximum capacity, we’ve opened a waiting list. In the end we had 135 registrations. Part of them announced us that they cannot make it and leave the seat to someone else. From 85 confirmed registrations, 70 showed up. Given the large number of people we splat from the beginning in three groups: red team, blue team and green team. Each team had its own facilitator, its own rooms, and basically we had three code retreats which ran in parallel. The facilitators consulted regularly to do the same exercises in each session so we allow people to switch teams after lunch.

The first session was to allow everyone to get familiar with the problem. We also asked them to write tests. Given the fact that most of the participants were not familiar with the format we have chosen Conway Game of Life. The second session asked for TDD, but we did not make anyone to change the pair if he didn’t want to. The third session was also about TDD and pair programming, but now anyone had to pair with someone else. At lunch we were already counting around 7 programming languages. After lunch we started with a ping pong TDD session, meaning that one starts writing the failing test and his/her pair makes it pass and writes the next failing test and then they switch again. The last session was a mute session, no talking with the pair. This forced everyone to make their code very expressive. We had some pairs who also tried the ‘no-ifs’ exercise for the last session.

I have facilitated for the blue team. It was the first time for me as a facilitator and a great experience. The advices from Andrei and Costin helped a lot. Also of great help was the facilitator training registered by Jim Hurne, which I watched a day before on the plane and in the airport. Thanks guys! A large part of the participants didn’t try TDD before, so during the second and third session I insisted that the three steps of TDD are done in very short cycles. I remember that at the beginning of the third session when everyone had to switch the pair, there was a lot of loudly arguing in each pair. Everyone knew a different way to solve it. They all did it two times before. I went down stairs for a few minutes and when I’ve returned it was quite again. They’ve reached the common ground. Pair programming worked. In the last session I had a pair of three who were intrigued by the no-ifs exercise and wanted to try it instead of the mute session. They almost did the implementation in 45 minutes. At the closing circle everyone shared what they’ve learned. If I had asked the same questions at the beginning of the day, I am sure I would have got very different answers then the ones at the closing circle. This proves that it was a good experience for everyone. It was a nice reward, to see developers happy because they have done TDD and pair programming and that they have learned something. Some did this for the first time.

Most of the participants asked to have more code retreats. There were many suggestions to have one each month or one every two months. Given the interest I’ve seen in Cluj for this I think we are going to organize it again and again, but next time I will be also coding. Andrei and Costin said the same, they also want to code in the next code retreat. We all three were a bit jealous on the ones who were having fun coding and experimenting new approaches. So, who wants to facilitate the next code retreat in Cluj?

In the end, I want to thank to everyone who contributed to this great day starting with each participant and ending with Corey who started it. I had a great experience. Thanks!

You can see some more pictures on our facebook page.

P.S. Special thanks to my girlfriend Georgiana and my brother Mihai for helping me with the organising work.

P.P.S. A big thanks to our local sponsors: ZenCash.com, ISDC, accesa, TSE Development Romania, Endava, Kno and ULLINK.

I’m not sure if the term of Mirrored Unit Tests actually exists, but I’ll explain in this post what I mean by it.

You get this situation when you have a class under test, which has symmetric public methods, which change the internal state of the class and have logic that depends on the fact that one method was called before or not. To test this for one method, you will have to have tests for scenarios in which the other method was called or not. Then you will have to rewrite the same tests for the other method, in a very similar way, like in a mirror. From where the term: Mirrored Unit Tests. No matter how much you will try to remove duplicate code by extracting it into helper methods, your test code will get messy.

The solution, is an abstraction (of course J ). You make an abstract test class, in which you implement the tests for one method. Then you implement the abstraction for each method you wanted to test and you override the protected members that abstract the method under test. With this test design you get rid of the duplication and it works nicely with most of the unit testing frameworks.

Below I will detail this more, by giving an example.

I have a class which is used in an implementation which builds a lambda expression, from some other representation of code (less important for this example). The class looks like this:

[sourcecode language=”css”]
class BinaryExpressionBinder
{
private Expression left;
private Expression right;

public Expression ResultedExpression { get; private set; }

public void NotifyForRightNode(Expression rightExpr)
{
bool constantExpr = TrySetAsConstantExpression(rightExpr, LeftWasSet);

if (!constantExpr)
{
right = rightExpr;
SetFieldExpression();
}
}

public void NotifyForLeftNode(Expression leftExpr)
{
bool constantExpr = TrySetAsConstantExpression(leftExpr, RightWasSet);

if (!constantExpr)
{
left = leftExpr;
SetFieldExpression();
}
}

private bool LeftWasSet
{
get { return left != null; }
}

private bool RightWasSet
{
get { return right != null; }
}

private void SetFieldExpression()
{
if (LeftWasSet && RightWasSet)
{
BinaryLogicalExpression be = new BinaryLogicalExpression
{
Left = left,
BooleanOperator = op,
Right = right
};
ResultedExpression = be;
}
}

private bool TrySetConstantExpression(Expression memberExpr, bool otherMemberWasSet)
{
// some logic which is not relevant for this sample.
// This logic may set the result with a constant expression
}
}
[/sourcecode]

The important thing to notice is the two public methods: NotifyForRightNode() and
NotifyForLeftNode(). They are symmetric: both in signature and also in implementation. They try to set the result as a constant expression. If that fails it remembers the notification argument and if the notification for the other node was called, then a binary expression is set as result. It is important to remember that the binary expression can be built only when both members are known.

Now let’s see how to test this. I will need to write unit tests against the public interface and not to relay at all on the implementation details. In this case I would need to write tests for NotifyForRightNode() with at least the following scenarios: left node was not set yet, and a constant expression can or (other scenario) cannot be built; left node was already set and a constant expression can or (other scenario) cannot be built. So at least four scenarios for NotifyForRightNode(). Now the same tests have to be for the other method
NotifyForLeftNode(), but in mirror (configuring that right node was set or not). As I said above, I will write the tests only for one method, in an abstract test class and then I will override just to specify the method under test.

[sourcecode language=”css”]
public abstract class BinaryExpressionBinderTests
{
protected abstract void NotifyForNodeUnderTest(Expression expr);
protected abstract void NotifyForOtherNode(Expression expr);

protected BinaryExpressionBinder Binder;

[TestMethod]
public void NotifyForNode_OtherNodeWasNotifiedAndConstantExpressionCannotBeBuild_ResultIsBinaryExpression()
{
//arange
Binder = ConfigureBuilder();
Expression dummyExpr = GetDummyExpression();
NotifyForOtherNode(dummyExpr);

//act
NotifyForNodeUnderTest(dummyExpr);

//assert
Expression actual = Binder.ResultedExpression;
Assert.IsInstanceOfType(actual, typeof(BinaryExpression));
}

// other tests …
}

[TestClass]
public class BinaryExpressionBinderTestsForLeftNode : BinaryExpressionBinderTests
{
protected override void NotifyForNodeUnderTest(Expression expr)
{
Binder.NotifyForLeftNode(expr);
}

protected override void NotifyForOtherNode(Expression expr)
{
Binder.NotifyForRightNode(expr);
}
}

[TestClass]
public class BinaryExpressionBinderTestsForRightNode : BinaryExpressionBinderTests
{
protected override void NotifyForNodeUnderTest(Expression expr)
{
Binder.NotifyForRightNode(expr);
}

protected override void NotifyForOtherNode(Expression expr)
{
Binder.NotifyForLeftNode(expr);
}
}
[/sourcecode]

So, less code, lower maintenance costs. It wouldn’t be right to test only one method, even though the VS (or other measuring tool) would say that I have a good coverage. The good coverage comes only because my implementation avoids duplicates and calls helper methods. What I want is to have a good coverage and this not to depend on the implementation. When I will refactor the production code, my tests have to stay green and valid without the need of being adjusted. Because that’s how good tests are!

I’ve talked this weekend at Code Camp Cluj about good unit tests and quality code.

Here are my slides and  here is the code I’ve demoed.

I am sharing here my resharper templates that I use. You should be able to easily import them.

The most useful one is the TestMethod template:

[TestMethod]  
public void MethodUnderTest_Scenario_ExpectedBehaviour()  
{
     Assert.Fail("Not yet implemented");  
}

It follows the unit test naming convention proposed by Roy Osherove in his book “The Art Of Unit Testing”. Naming your unit tests like this it helps you in writing good unit tests, because it foces you to keep your tests simple and test only one thing. If you cannot name the test method like this either you’re testing more than one thing, either your test is too complex, so you cannot name the test scenario. If you’re in the last case consider to refactor your production code.

Another great advantage I’ve found by following Roy’s convention is that I can easily know, just by looking at the test run report, where the bug is when a test fails. This way I reduce my debug time to almost zero for the code under unit tests and this gives me a higher efficiency.

I’m starting a worldpress blog. I’m not sure how much inspiration will I have to post here articles. We’ll see…

I am starting this, thinking to have a place where I can easily share experiences, thoughts or any kind of other stuff….