Sunday, November 10, 2013

Gaining Domain Knowledge of ISO-8583 Messages

Let's talk about how we improved code readability and gained domain knowledge in creating ISO 8583 messages in jPOS.

At first, we were writing code like this. And the team started asking about field 2. The code didn't help in providing much domain knowledge about financial transaction card originated messages.

import org.jpos.iso.*;

ISOMsg m = new ISOMsg();
m.setMTI("0100");
m.set(2, "...");
m.set(3, "000000"); // purchase; no account type specified
m.set(4, "000000001500"); // in acquirer's currency (e.g. USD 15.00)

So, we wanted to improve things and here's what we had in mind.

import static org.junit.Assert.*;
 
import java.util.*;
import org.junit.*;
 
import org.jpos.iso.*;

public class AuthorizationRequestBuilderTest {
  @Test
  public void test() throws Exception {
    ISOMsg msg = new AuthorizationRequestBuilder()
            .withPrimaryAccountNumber(...)
            .withProcessingCode("000000")
            .withTransactionAmount(...)
            .build();
    assertEquals("0100", msg.getMTI());
    assertEquals("...", msg.getString(2));
    assertEquals("000000", msg.getString(3));
    assertEquals("...", msg.getString(4));
  }
}

We wanted to use the names of the fields, instead of referring to them as field numbers. This helps improve readability and adds to the team's domain knowledge, as they now know that field number 2 is the primary account number (or PAN for short). We also applied the builder pattern and used a fluent interface.

public class AuthorizationRequestBuilder {
  …

  public AuthorizationRequestBuilder() {
  }

  public ISOMsg build() {
    ISOMsg msg = new ISOMsg();
    msg.setMTI("0100");
    msg.set(2, this.pan);
    msg.set(3, this.processingCode);
    msg.set(4, this.transactionAmount);
    return msg;
  }

  public AuthorizationRequestBuilder withPrimaryAccountNumber(String pan) {
    if (!pan.matches("[0-9]{12,19}")) {
      throw new IllegalArgumentException("PAN must be a minimum of 12 digits");
    }
    this.pan = pan;
    return this;
  }

  public AuthorizationRequestBuilder withProcessingCode(String processingCode) {
    this.processingCode = processingCode;
    return this;
  }

  public AuthorizationRequestBuilder withTransactionAmount(String transactionAmount) {
    this.transactionAmount = transactionAmount;
    return this;
  }

  …
}

Primary Account Number

As it turns out, PANs are not just a minimum of 12 (and maximum of 19) digits. It consists of three primary components:

To illustrate, say we have the following PAN, 55417710000xxxx3.

  • 554177 is the IIN
  • 10000xxxx is the individual account identification number
  • 3 is the PAN check digit

With this added knowledge, we can enhance the builder to validate the PAN.

public class AuthorizationRequestBuilder {

  …

  public AuthorizationRequestBuilder withPrimaryAccountNumber(String pan) {
    if (!pan.matches("[0-9]{12,19}")) {
      throw new IllegalArgumentException("PAN must be a minimum of 12 digits and a maximum of 19 digits");
    }
    if (!CheckDigit.isValid(pan)) {
      throw new IllegalArgumentException("PAN contains invalid check digit");
    }
    this.pan = pan;
    return this;
  }

  …
}

ISO specification 7812 and 7813 details the specific requirements for PAN composition. All PANs used in ISO 8583–1987 messages must conform to the ISO PAN encoding requirements.

Processing Code

The processing code (DE 3) contains even more knowledge to be gained. At first, we thought they were just digits. Later, we found out (thanks to the domain experts and supporting documents) that it was a series of six (6) digits used to describe the effect of a transaction on the customer account and the type of accounts affected.

These six (6) digits are composed of three (3) subfields:

  1. Cardholder Transaction Type Code
  2. Cardholder Account Type (From)
  3. Cardholder Account Type (To)

To get a better sense of what transaction types can be used in an authorization request, here are some transaction types (NOTE: Your payment network may differ. Please refer to its documents/manuals):

ValuesDescription
00Purchase
01Withdrawal
28Payment
30Balance Inquiry
40Account Transfer

Cardholder account types can have the following values:

ValuesDescription
00No account specified (NAS)/Default Account
01Savings Account
02Checking Account
03Credit Card Account

So, when a business user says, "balance inquiry on savings account", s/he means processing code 300100.

Did you get a light-bulb moment like I did when I first found out? Smile! If so, hit the comments.

Given the above domain knowledge, we initially set out to create a builder for the processing code to do something like this.


    ISOMsg msg = new AuthorizationRequestBuilder()
            .withPrimaryAccountNumber(...)
            .withProcessingCode(new ProcessingCode.Builder()
                .purchase()
                .from(AccountType.NOT_SPECIFIED)
                .to(AccountType.NOT_SPECIFIED)
                .build())
            .withTransactionAmount(...)
            .build();

But then, we later found out that the payment network only supports specific processing code combinations. Here are some (NOTE: Table below does not provide a complete list of valid processing codes):

ValuesDescription
000000Purchase; no account specified
001000Purchase from savings account
002000Purchase from checking account
280000Payment; No account specified
280010Payment to savings account
280020Payment to checking account
280030Payment to credit card account
300000Balance inquiry; no account specified.
When no account is specified on a balance inquiry transaction, the issuer may return both checking and savings account balances if applicable.
301000Balance inquiry on savings account
302000Balance inquiry on checking
303000Balance inquiry on credit card (credit line)

Since not all combinations (between transaction type and to-/from- account types) are valid, we thought it would be best to create a builder that helps with the creation of valid processing codes (and not just a simple string of six digits). Here's our rough idea.

public enum AccountType {
  NOT_SPECIFIED, SAVINGS, CHECKING, CREDIT_CARD
}

. . .

public class PurchaseProcessingCodeBuilder {
  public PurchaseProcessingCodeBuilder from(AccountType type) {. . .}
  // does not support a To- account
  public String build() {. . .}
}

. . .

public class PaymentProcessingCodeBuilder {
  // does not support a From- account
  public PaymentProcessingCodeBuilder to(AccountType type) {. . .}
  public String build() {. . .}
}

. . .

public class BalanceInquiryProcessingCodeBuilder {
  public BalanceInquiryProcessingCodeBuilder from(AccountType type) {. . .}
  // does not support a To- account
  public String build() {. . .}
}

. . .

    ISOMsg msg = new AuthorizationRequestBuilder()
            .withPrimaryAccountNumber(...)
            .withProcessingCode(new PurchaseProcessingCodeBuilder()
                // From- account type is NOT_SPECIFIED
                .build())
            .withTransactionAmount(...)
            .build();

    ISOMsg msg2 = new AuthorizationRequestBuilder()
            .withPrimaryAccountNumber(...)
            .withProcessingCode(new PurchaseProcessingCodeBuilder()
                // From- account type is NOT_SPECIFIED
                .to(...) // <-- results into a compiler error!
                .build())
            .withTransactionAmount(...)
            .build();

Notice that purchase transactions only support a "from" account type, but no "to" account type. Payment transactions support a "to" account type, but no "from" account type. And, balance inquiry only supports a "from" account type.

The astute reader would probably notice that in the given sample transaction types, only one account type is used (either "from" or "to"), but not both. So, you might ask, "Is there a transaction type that needs both 'from' and 'to' account type values?" Yes, there is — transfers.

Another possible idea is to create separate builders for the transaction types. Something like this,


    ISOMsg msg = new AuthorizationRequestBuilder()
            .withPrimaryAccountNumber(...)
            .balanceInquiry()
                // no account type is specified
                // .withTransactionAmount(...) <-- no transaction amount is needed
            .build();

    . . . = new AuthorizationRequestBuilder()
            .withPrimaryAccountNumber(...)
            .balanceInquiry()
                .onSavingsAccount() // or .onCheckingAccount() or .onCreditCardAccount()
            .build();

    . . . = new AuthorizationRequestBuilder()
            .withPrimaryAccountNumber(...)
            .accountTransfer()
                .fromSavingsAccount()
                .toCheckingAccount()
            .withTransactionAmount(...)
            .build();

Transaction Amount

At first, we simply thought that the transaction amount was a left-zero-padded string with two decimal places, but without the separator (i.e. decimal point). Again, after learning much more from the domain, the amount was actually based on the acquirer's currency. The sample from the document helps explain this.

DE 4 (Amount, Transaction)DE 49 (Currency Code)Currency ExponentCurrency NameActual Monetary Value of DE 4
0000000015009490New Turkish Lira1500 Lira
0000000015001242Canadian Dollar15.00 Dollars
0000000015007883Tunisian Dinar1.500 Dinars

Notice that while the transaction amount (DE 4) value is the same, it means differently based on the value of the currency (DE 49). We've used java.util.Currency#getDefaultFractionDigits() for this.

Message- vs. Domain- Centric

I consider the above ideas to be rather message-centric. After gaining more domain knowledge, I believe a domain-centric design would be of greater help. This domain-centric design would revolve around issuers, acquirers, card holders, merchants, and more. I hope to write more about this when I get some free time in the near future.

Acknowledgements

There is just so much more to learn about ISO-8583 and payment networks. One blog post is definitely not enough. Hopefully, I was able to share some of the things I've learned. Thanks to my team mates, Edge, Claire, and JC, for encouraging me to write this. I've learned so much while working with you guys.

More power to the team, and have fun learning more about the domain.

Wednesday, October 23, 2013

Long Loading Requests in App Engine for Java

In this post, I revisit the loading requests in Google App Engine for Java. The test results are shown below. The following modes are:

Skeleton
Plain-vanilla skeleton webapp with SDK (1.8.2) dependencies.
With Spring Framework Web
Skeleton webapp plus Spring Framework Web (3.2.4.RELEASE) dependencies. This adds almost 3 MB of JARs.
spring-aop-3.2.4.RELEASE.jar335455
spring-beans-3.2.4.RELEASE.jar607755
spring-context-3.2.4.RELEASE.jar863688
spring-core-3.2.4.RELEASE.jar869674
spring-expression-3.2.4.RELEASE.jar196807
spring-web-3.2.4.RELEASE.jar625875
With Spring Web MVC
Webapp with Spring Framework Web MVC. This add one additonal JAR.
spring-webmvc-3.2.4.RELEASE.jar636993
With Spring Web MVC and applicationContext
Webapp with Spring Framework Web MVC and running an empty (zero beans) application context on startup via ContextLoaderListener.
With Google Guice
Skeleton webapp with Google Guice (3.0) dependencies.
With Google Guice (one servlet module)
Webapp with Google Guice servlet context listener, filter, loading a servlet module.
With Google Guice (one servlet module)
Same as before, but built without AOP. The following dependency declaration was used:
        <dependency>
            <groupId>com.google.inject</groupId>
            <artifactId>guice</artifactId>
            <version>${guice.version}</version>
            <classifier>no_aop</classifier>
            <exclusions>
             <exclusion>
              <groupId>aopalliance</groupId>
              <artifactId>aopalliance</artifactId>
             </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.google.inject.extensions</groupId>
            <artifactId>guice-servlet</artifactId>
            <version>${guice.version}</version>
            <exclusions>
             <exclusion>
              <groupId>com.google.inject</groupId>
              <artifactId>guice</artifactId>
             </exclusion>
            </exclusions>
        </dependency>  
  

Test Results (Summarized)

Times are measured by shutting down instance, hitting a URL, looking at request time in logs. All using a frontend instance class of F1 (600MHz, 128MB).

ModeAverage Time (ms)
Skeleton4322.6
With Spring Web4836.0
With Spring Web MVC4819.0
With Spring Web MVC and applicationContext8845.6
With Google Guice (no modules)4437.6
With Google Guice (one servlet module)8260.8
With Google Guice (one servlet module, no AOP)5332.2

Observations

  • Adding some JARs (dependencies) to the web application (even without using any of its classes) increases the time it takes to complete a loading request.
  • Adding an empty (zero beans) application context to the web application increases the time by about 4 seconds.
  • Using Google Guice to load servlets and provide dependency injection can add about 4 seconds (similar to using Spring Framework).
  • Removing AOP from the use of Google Guice can lower the additional loading request time.

Test Results (Raw)

ModeTryTime (ms)
Skeleton14050
Skeleton24384
Skeleton34305
Skeleton44609
Skeleton54265
SkeletonAverage4322.6
With Spring Web15001
With Spring Web25115
With Spring Web34783
With Spring Web44724
With Spring Web54557
With Spring WebAverage4836.0
With Spring Web MVC15205
With Spring Web MVC24900
With Spring Web MVC34486
With Spring Web MVC45164
With Spring Web MVC54340
With Spring Web MVCAverage4819.0
With Spring Web MVC and applicationContext19187
With Spring Web MVC and applicationContext28946
With Spring Web MVC and applicationContext38417
With Spring Web MVC and applicationContext48777
With Spring Web MVC and applicationContext58901
With Spring Web MVC and applicationContextAverage8845.6
With Google Guice (no modules)14578
With Google Guice (no modules)24473
With Google Guice (no modules)34480
With Google Guice (no modules)44223
With Google Guice (no modules)54434
With Google Guice (no modules)Average4437.6
With Google Guice (one servlet module)17845
With Google Guice (one servlet module)28426
With Google Guice (one servlet module)38476
With Google Guice (one servlet module)48394
With Google Guice (one servlet module)58163
With Google Guice (one servlet module)Average8260.8
With Google Guice (one servlet module, no AOP)15702
With Google Guice (one servlet module, no AOP)25020
With Google Guice (one servlet module, no AOP)36036
With Google Guice (one servlet module, no AOP)45223
With Google Guice (one servlet module, no AOP)54680
With Google Guice (one servlet module, no AOP)Average5332.2

Related Links

Sunday, September 29, 2013

Early UI, Early Feedback, Better Results

A few months back, I was reviewing some internal apps we had in our organization, and this thing we humbly call the "library app" got my attention. We have a lot of books (physical copies) lying around, and we encourage everyone in the organization to read and apply/share what they've read. So far, it seems to be working.

A group of young developers was learning Grails, and they built this "library app". I wanted to learn too, so I pushed myself to do something that ran on Google's App Engine, and I ended up learning more.

UI and User Experience

I started looking at the current user interface of the one built using Grails. Here's what I saw — a login page.

After logging in, I get the following.

Thanks to PJ for the screenshots.

I asked myself, "What would a typical user do if s/he was greeted with this landing page?" My first response was, I wouldn't even think of clicking any of the two icons. And here's what I saw after clicking on the "Book Listing" icon/link.

There was nothing there that "entices" me to explore and look at the available titles. So, I set out to create a landing page that would make we want to look at what's available.

Early Feedback

Here's what I came up with (largely inspired by iBooks and other similar apps).

libraryapp.onb-demo.appspot.com

I didn't build the code behind it (at least, not yet). I just started with some static HTML, deployed it to the cloud, and asked some colleagues to try it out. I was pleasantly surprised with their reaction. They asked if the app was a modified version of the one built using Grails. They also asked if I got help from our creatives team, and if it was written using Grails. Anyway, the early UI allowed me to get early feedback. And letting the landing page be seen without the need to login also helped (I guess everyone is too lazy busy to login).

So, I proceeded to turn the static HTML into something more dynamic. I started populating the database with book titles. Then, it occurred to me that I was spending so much time entering the titles, authors, publishers, etc. Even though we have about 50+ titles, it was taking too much time. So, I asked around how the current version was being maintained. I was surprised that they did it pretty much the same way I did (adding titles one by one, and typing everything in while looking at the physical copy). I started thinking of improving this. (Seems like an unarticulated need here)

I saw myself Googling the book's title every time I needed to enter its details. Light bulb moment! Why don't I enhance the entry form and allow the user to Google it up, and populate the entry form?

After some rough JavaScript, here's what I ended up with.

I added a "Search" button that would do the search (using Google Books API). The results are shown, and a button is added to use the book information to populate the form.

After I added this, I was able to add titles in no time! This would not have been possible for a user to articulate or express as a requirement. How would they know that APIs exist to help with finding book information? It's critical to observe and experience what the user goes through when using your app to achieve their goals and/or perform their tasks.

I'm no design guru. This simple enhancement is far from a breakthrough. I hardly observed our office admin team use the existing library app. But I'm glad I had this experience, and I hope sharing it here would help others. I indeed learned more than just building apps!

Future Enhancements

As you can see from the screenshot, I have not implemented the borrowing part of the app (yet). But I'm thinking of ways to add a "comments" section to allow users to post their comments about the book. I'm hoping that these comments shall further "entice" others to grab a copy, read, apply, and share.

Tuesday, September 3, 2013

Model-View-Controller (MVC) Done Right?

I was pleasantly reminded of how I, as a budding junior software engineer (a couple of years back), misunderstood MVC and application architecture, and how others may misunderstand this too. So, please allow me to explain my journey in understanding MVC better.

MVC and Application Architecture

Martin Fowler provides a good explanation of MVC in his Patterns of Enterprise Application Architecture.

When developing applications, one application architecture I'm usually exposed to is the layered architecture (also known as multi-layered, or multi-tiered). Each layer would usually be dependent on layers underneath it, but not above it. These layers provided abstraction (via interfaces), which is useful in tackling complex systems and allowing concurrent development.

A multilayered software architecture is a software architecture that uses many layers for allocating the different responsibilities of a software product.

Presentation Layer
Business Layer
Data Access (or Infrastructure) Layer

Missing Link Between MVC and Layered Architecture

So, I started thinking that MVC was mapped to layers like this:

LayerMVC Component
PresentationView
Business (or Domain)Controller
Data AccessModel

But that's the wrong way! A crucial point that I missed is that MVC is a presentation layer pattern. It is really intended for the user interface. So, what did I end up doing?

Because of this poor understanding of MVC, I ended up exposing domain entities as models to my views. This resulted in entities that exposed setters and getters, which made it difficult to implement invariants and DDD. This also resulted in business rules that were mixed with UI-related implementations. So, what is a better understanding of MVC?

LayerContent
PresentationMVC
Business (or Domain)Services / Domain Objects
Data AccessDAO / Repository Implementations / Gateways

As it was originally intended, MVC is found inside the presentation layer. It's important to note that the Model in MVC is not your Service Layer or the Domain Objects. It's the user interface's state (or View state).

If your application is simple enough, you could choose to have domain objects (usually entities) as your view state for simplicity. But please don't confuse MVC's Model with the Domain Objects.

If you find yourself having difficulty dealing with entities (or domain objects) in the presentation layer, perhaps you can consider converting domain objects to more portable DTOs that can be tailored to specific use cases (or specific UIs) and add an application service (typically using a facade pattern) that would handle the DTO-serializing responsibilities before exposing them to the presentation layer.

Friday, July 26, 2013

Google Maps and Google Earth Synchronized

Inspired by Google Map Maker's visualization of updates, I created this mashup of Google Earth and Google Maps. Here, if the user pans the map (2D), the earth (3D) follows, and vice-versa.

NOTE: You'll need Google Earth Plug-in installed for this to work.

Below is a demo that is initially showing Manila, Philippines. Go ahead and pan either view and see if it works.

Loading...
Loading...

Synchronizing Earth and Maps

Using Google Earth and Google Maps is straight-forward using their JavaScript APIs. The tricky part is synchronizing them. When a user pans the map (2D), the earth (3D) should follow. Likewise, when the user pans the earth (3D), the map (2D) should follow.

My first attempt was to listen to the earth's viewchangeend event, and the map's center_changed event.

function syncEarthWithMap() {
  …
}

function syncMapWithEarth() {
  …
}
…
google.maps.event.addListener(map, 'center_changed', syncEarthWithMap);
…
google.earth.addEventListener(ge.getView(), 'viewchangeend', syncMapWithEarth);
…

Unfortunately, this resulted in an infinite loop, since the event handlers would update the view, which will trigger more events. To fix this, we'll need to remove the event handlers and restore them at a later time. The idea looks something like this:

function syncEarthWithMap() {
  …
  if (earthEventHandler.isEnabled()) {
    earthEventHandler.disableAndEnableLater();
  }
  …
  // Now that the earth event handler is disabled,
  // we can safely change it without triggering more calls.
  ge.getView().setAbstractView(lookAt);
}

function syncMapWithEarth() {
  …
  if (mapEventHandler.isEnabled()) {
    mapEventHandler.disableAndEnableLater();
  }
  …
  // Now that the map event handler is disabled,
  // we can safely change it without triggering more calls.
  map.setZoom(zoom);
  map.panTo(center);
}

Here's how the event handler would look like:

function createEventHandler(enableHandler, disableHandler, enableLater) {
  var enabled = false;
  return {
    enable: function() {
      enableHandler();
      enabled = true;
    },
    isEnabled: function() {
      return enabled;
    },
    disableAndEnableLater: function() {
      disableHandler();
      enabled = false;
      enableLater(enable);
    }
  };
}

Creating the map event handler looks something like this.

var mapEventHandler;

mapEventHandler = createEventHandler(function() {
  /* enable event handler */
  … = google.maps.event.addListener(map, 'center_changed', syncEarthWithMap);
  … = google.maps.event.addListener(map, 'bounds_changed', syncEarthWithMap);
}, function() {
  /* disable event handler */
  google.maps.event.removeListener(…);
  google.maps.event.removeListener(…);
}, function(callback) {
  /* later, enable event handler */
  google.earth.addEventListenerOnce(map, 'idle', function() {
    callback();
  });
});
mapEventHandler.enable();

The earth event handler is similar.

var earthEventHandler;

earthEventHandler = createEventHandler(function() {
  /* enable event handler */
  google.earth.addEventListener(
            ge.getView(), 'viewchangeend', syncMapWithEarth);
}, function() {
  /* disable event handler */
  google.earth.removeEventListener(
            ge.getView(), 'viewchangeend', syncMapWithEarth);
}, function() {
  /* later, enable event handler */
  …
});
earthEventHandler.enable();

Synchronizing both views can be a bit tricky. But thanks to good documentation from Google, synchronizing them can be quite fun!

While developing this, I was looking for interesting places in the Philippines that would look good on Google Earth (largely due to the presence of 3D structures). If you know of some, please let me know (include latitude and longitude if possible) via comments, and I'd add them as links that will reposition the views to those locations. Cheers!

Monday, July 22, 2013

Delegation versus Abdication

I've had several instances where I was being asked about management work at an IT software/services company. They were interesting conversations, and I thought I'd write about it here. Some of the senior developers and project managers I've worked with indicated they wanted to either stay away from management (and remain with coding), or get out of management (and get back to coding).

These discussions usually boiled down to the difference between management by delegation and management by abdication. I've read about this before in E-Myth Revisited and here too.

The dictionary defines delegating as "entrusting (a task or responsibility) to another person, typically one who is less senior than oneself". And defines abdicating as "to renounce or relinquish a throne, right, power, claim, responsibility, or the like, especially in a formal manner".

I'd usually get a chuckle and a reaction that indicates that they're not abdicating, since they are obviously still occupying the position of power. But I believe they are missing the point.

Delegation requires a manager to understand what needs to be done, and then assigns someone to perform or be responsible for that task. A delegating manager designs what they are managing, and then delegates the responsibilities of that management to the people underneath him.

Abdication, on the other hand, involves assigning tasks to others, while not having any understanding of what they are doing.

Are you delegating? Or are you abdicating?

Real-world Example: IT Manager

Abdicator

Hires a team of software developers, analysts, and testers, for a project. Never gives them clear roles and responsibilities, but instead just says "do your job". While the team is busy with the project, the abdicator doesn't monitor and check if the project is progressing the way he wants it to progress (as s/he does not understand how it needs to be done).

When a project team member would ask the manager for suggestions, the abdicating manager would simply say something like "I surround myself with people smarter than me. That's why I hired you. So, what do you think would be best? ... Then go with it." (Sound familiar?)

Then, when something goes wrong with the project (e.g. missed deadlines, etc.), the abdicator gets mad and fires the team because they weren't doing their job.

And worse, this repeats again with another project team, since the abdicator still hasn't learned his/her lesson. S/he still has not designed a way (strategies/tactics/processes) for the team to manage the project, and failed to ensure that the team is following it.

Delegator

Hires a team of software developers, analysts, and testers, for a project. Understands that the following processes need to happen for a successful project: contract, schedule, backlog, sprints, iterations, version control, coding, testing, deployment, etc. Delegates the: backlog to the product owner, user stories to an analyst, design and coding to the programmers, testing to the testers, etc. Understands that each of these tasks happens continuously and then tracks the progress of these. S/he knows before something goes wrong in the project, and understands how to remedy them.

S/he continues to refine the design with structure and forethought. This frees the manager from the crushing load of day-to-day issues, and allows him/her to expand and scale out.

Micro-manage?

Remember our dictionary definition of "delegate" is to entrust someone else a task? Regardless of who has the responsibility for a task or function, the delegating manager is ultimately accountable for the outcome.

If it's such a big risk to delegate (and ensure things are done correctly), then should we micro-manage? Please don't!

Delegate, Do Not Abdicate

There are ways to help ensure that delegated tasks are done correctly (without micro-managing). That's why most IT managers define processes and checklists to help ensure that things are done well (e.g. require a signed contract before starting a project, ensure that work effort is charged to the correct project account, how are margins affected when team members are promoted, consider effects of long-term engagements over short-term, etc), and if things go wrong, managers can readily see it and take action.

In the E-Myth Blog post, it provides some guidelines for effective delegation. These include:

  • Document the correct way to perform this task, step by step, including the quality control standards for each step.
  • Clearly specify the expected results of the delegated task. Give information on what, why, when, who, where and how.
  • When the new system document is ready, provide it to the employee responsible for that task, train them on how to successfully run this "system" and insert a copy of this process document into your company's Operations Manual.

In an organization that provides IT software/development services, these step by step guides to performing (delegated) tasks may mean: using TDD or ATDD, setting up a version-controlled repository for team members, using user stories and story points, having a daily 15-minute stand-up, proper on/off-boarding for new team members, etc.

Note that the processes need to be continuously refined. Or, if the process was defined by someone who does not have a good understanding of the business of software development, then perhaps the process itself is flawed.

If you're joining a company to become part of management (or whatever role), this is effectively being delegated the task of managing (or whatever the role entails). It would be good to determine if the company (and its leaders or managers) is good with delegation, and check if they have processes to bring you up to speed with how to do things correctly. Ask how you'll be evaluated to see if you're doing a good job or not. Otherwise, it might be a sign that the company doesn't really know what it's doing (and is just abdicating to hired people and telling them to "do your job").

Tuesday, July 9, 2013

Monopoly Deal Cards with CSS3

I've been learning some CSS techniques. And I thought it would be good to continue using CSS on something I can show to others. So, here's a rough attempt to render Monopoly Deal cards with CSS3.

Thanks to the creative team of Edwin and Renee for teaching me and answering some of my questions. Also thanks to Sofia for letting me borrow some Monopoly Deal cards. And the Hong Kong edition looks great too.

What's next? Can we add some JavaScript and make an online game with this?

WARNING: Only tested on Firefox 20+.

My apologies for not explaining the CSS behind this. Perhaps, if I get enough time (and requests), I'll write about it. Besides, I'm sure I'm not the first to attempt this. For now, use your favorite HTML/Browser tool to checkout the CSS behind it.

4
4
4
Boardwalk
4
(No. of properties owned in set) RENT
1
3
1
2
8
FULL SET
New York
Avenue
2
(No. of properties owned in set) RENT
1
1
1
2
3
1
2
3
5
FULL SET
North
Carolina Avenue
4
(No. of properties owned in set) RENT
1
2
1
2
4
1
2
3
7
FULL SET
太平山
Victoria Peak
4
(No. of properties owned in set) 租金 RENT
1
3
1
2
8
全套
FULL SET

DISCLAIMER: I still don't know how to play the game.

Update (Dec 2013): Thanks to Jason for asking. Here's an action card.

3
ACTION CARD
SLY DEAL
Steal a property from
the player of your choice.
(Cannot be part of a full set.)
Play into center to use.
3

Thursday, June 20, 2013

Domain-Driven Design: Accounting Domain Model

In my previous post, I talked about two things that helped me improve my implementation of domain-driven design. In this post, I'll try to use the ubiquitous language and develop an accounting model.

Disclaimer: Note that I'm no accountant by any means.

This is based on "Accounting for Non-accountants" by Daniel R. Cruz (as listed here) — a text book from De la Salle University Press that I was fortunate enough to read. Thanks to Garen and her MBA studies.

There are eight steps in the accounting cycle. For purposes of this post, I'll limit only to journalizing and posting.

Journalizing
Record transactions in journal.
Posting
Post transactions to ledger accounts.

An accounting plug-in/sub-module would enable users to enter transactions and generate journal entries. These entries can be exported (posted) to the real or main accounting system which would tally the account balances of the enterprise using transactions from different systems/sources.

So, what's a journal entry? Here's what Wikipedia gives us:

A journal entry, in accounting, is a logging of transactions into accounting journal items. The journal entry can consist of several items, each of which is either a debit or a credit. The total of the debits must equal the total of the credits or the journal entry is said to be "unbalanced".

Journal Entries

Let's start modeling this. The JournalEntry is an entity. An aggregate can be formed around a journal entry and its items. The JournalEntry is the aggregate root. Note how I'm trying to use the ubiquitous language in naming our entities.

package accounting.domain.model;

@Entity
public class JournalEntry {
  private Date date;
  private String description;
  private List<JournalEntryItem> items = new ArrayList<JournalEntryItem>();

  public JournalEntry() {
  }

  public List<JournalEntryItem> getItems() {
    return this.items;
  }

  public void setItems(List<JournalEntryItem> items) {
    // TO DO: Ensure items are "balanced"
    this.items = items;
  }
}
package accounting.domain.model;

@Entity
public class JournalEntryItem {
  public JournalEntryItem(…) {…}
}

At this point, we can see that keeping the items "balanced" has not yet been implemented. We can add a validate method to the JournalEntry entity, and have it called before it gets persisted. But what about keeping the entity valid all the time by doing something like this?

@Entity
public class JournalEntry {
  private Date date;
  private String description;
  private List<JournalEntryItem> items;

  public JournalEntry(…, JournalEntryItem items[]) {
    if (!JournalEntry.isBalanced(items)) {
      throw new IllegalArgumentException(
          "The total of debits must equal the total of credits");
    }
    this.items = Arrays.asList(items);
    …
  }

  public static boolean isBalanced(JournalEntryItem items[]) {
    … total = ….ZERO;
    for (JournalEntryItem item : items) {
      if (item.type() == Type.DEBIT) {
        total = total.subtract(item.amount());
      } else { // item.type() == Type.CREDIT
        total = total.add(item.amount());
      }
    }
    return total.equals(….ZERO);
  }

  // getters and setters removed

  public List<JournalEntryItem> items() {
    return Collections.unmodifiableList(this.items);
  }

  public String description() { return this.description; }
  public Date date() { return this.date; }
}

Here's how the expanded journal entry item can be implemented.

@Entity
public class JournalEntryItem {
  public static enum Type {
    DEBIT, CREDIT
  }

  private String accountId;
  private Type type;
  private … amount;

  public JournalEntryItem(String accountId, Type type, … amount) {
    // TO DO: Throw IllegalArgumentException if any argument is null
  }

  public String accountId() { return this.accountId; }
  public Type type() { return this.type; }
  public boolean isCredit() { return this.type == Type.CREDIT; }
  public boolean isDebit() { return this.type == Type.DEBIT; }
}

The amount in the entry item was simplified for brevity. In a real implementation, it would be a value object representing money (an amount of a certain currency).

Based on the domain language, journal entries are created via business transactions, like buying merchandise inventory using cash, business owner depositing money to corporate bank account as capital, create invoice, receiving payment for invoice, paying bills, etc.

Now, let's give it a test run. Let's say the business purchased merchandise inventory worth $4,000, paying $3,000 in cash, and the balance on credit. This shall be represented as:

new JournalEntry(…, new JournalEntryItem[] {
    new JournalEntryItem("Inventory", ….DEBIT, 4000),
    new JournalEntryItem(    "Cash",     ….CREDIT, 3000),
    new JournalEntryItem(    "A/P",      ….CREDIT, 1000) // accounts payable
});

The Accounting Equation

To understand why certain business transactions result into journal entries with debit/credit against certain accounts, we need to understand the accounting equation.

Assets = Equity

Assets can be in any form of property owned or controlled by the business, whather tangible or intangible. Equity refers to the owners or claimants to the assets or properties of the business. Therefore, if the business has $100,000 worth of assets, there must necessarily be $100,000 worth of claimants.

- "Accounting for Non-accountants" by Daniel R. Cruz

The basic accounting equation is:

Assets = Liabilities + Capital

In actual businesses, there are normally two types of claimants to the assets of the business. The claim(s) of the owner(s) who invested in the business is called owners' equity or capital. The claim(s) of the creditor(s) who extended financial assistance to the business, whether in cash or in kind, is called creditors' equity or liability(ies). Total owners' equity (capital) plus total creditors' equity (liabilities) equals total equity.

If the business has $100,000 worth of assets and total capital investment of the owner is $60,000, then total liabilities (or claimants) must necessarily be $40,000. The left and right sides of the equation must always balance for every business transaction as well as for the summary of all transactions.

- "Accounting for Non-accountants" by Daniel R. Cruz

There are five accounting elements. Accounts are classified under one of the five elements. Depending on the account's classification, an increase or decrease may mean a debit or credit.

IncreaseDecrease
AssetDebitCredit
LiabilityCreditDebit
Income/RevenueCreditDebit
ExpenseDebitCredit
Equity/CapitalCreditDebit

Let's go back to the previous example, where the business buys merchandise inventory worth $4,000, paying $3,000 in cash, and the rest on credit.

  • The "inventory" account is an asset account. Buying merchandise inventory increases it. Thus, there's a debit entry item against it.
  • The "cash" account is an asset account. Paying in cash decreases it. Thus, there's a credit entry item against it.
  • The "A/P" (accounts payable) account is a liability account. Paying on credit increases the accounts payable. Thus, there's a credit entry item against it.

Let's have another business transaction example. Let's say the business sold merchandise inventory (Cost of Goods Sold or CGS) worth $500 for $900, receiving $600 in cash, and the balance on credit (Accounts Receivable or A/R). How would this look using our domain model?

// for the sale of goods transaction
new JournalEntry(…, new JournalEntryItem[] {
    new JournalEntryItem("CGS",  ….DEBIT, 500), // cost of goods sold
    new JournalEntryItem(  "Inventory", ….CREDIT, 500)
});
// for the sale on account and cash sales transaction
new JournalEntry(…, new JournalEntryItem[] {
    new JournalEntryItem("Cash", ….DEBIT, 600),
    new JournalEntryItem("A/R",  ….DEBIT, 300), // accounts receivable
    new JournalEntryItem(  "Sales",     ….CREDIT, 900)
});
  • When we move an item from inventory to sell as a product, we move the product from an asset (inventory) to an expense (cost of goods sold). Thus, we increase from "cost of goods sold", and decrease "inventory".
  • The "cash" and "accounts receivable" accounts are asset accounts. Receiving cash and credit increases it. Thus, there are debit entry items against it.
  • The "sales" account is an income/revenue account. Making a sale increases it. Thus, there is a credit entry item against it.

More examples can be found here (at Wikipedia).

Special Journals

Special Journals are designed to facilitate the process of journalizing and posting transactions. They are used for the most frequent transactions in a business. For example, in merchandising businesses, companies acquire merchandise from vendors, and then in turn sell the merchandise to individuals or other businesses. Sales and purchases are the most common transactions for the merchandising businesses. A business such as a retail store will record the following transactions many times a day for sales on account and cash sales.

Automated systems would usually create interfaces that have a pre-determined set of journal entry items for the most frequent transactions.

Let's take the sales journal for example. An entry is made when creating invoices (or to record transactions that involve sales on credit). The journal entry items are as follows:

  • Debit on "A/R" account (increasing an asset-type account)
  • Credit on "Sales" account (increasing an income-type account)

The interface for the sales journal would look more like creating an invoice, rather than creating journal entry items. The description of the journal entry shall contain a reference to the number (or unique ID) of the created invoice.

Ledger Accounts

The astute reader would have seen that each entry item references an (ledger) account. From my previous post on DDD, we're referencing other aggregates by ID (and not by type).

The list of ledger accounts is called the "chart of accounts". Typically, an organization (or enterprise, or business) would setup a chart of accounts, and each account is classified under a certain type of account. The type shall determine how increases and decreases affect the balance of the account. The account is uniquely identified using an alpha-numeric string that follows some kind of convention.

So, how can we model the account?

@Entity
public class LedgerAccount {
  private String id;
  private String name;
  private String description;
  private String type;
  private List<LegderAccount> childAccounts;
  // private … balance; // can be calculated
  …
}

It's a pretty straight-forward entity. Generally, accounts are arranged in a hierarchy.

In automated systems, the journal entry items already reference the account. So, to get the balance of an account, it's just a matter of doing a simple query that joins the ledger account and its related journal entry items. The posting step is now optional.

Closing

This post has gotten quite lengthy. And we've only touched the journal entries. Hopefully, it's enough to get some accounting domain model started. If you've got questions/suggestions, please hit the comments.

Thursday, May 30, 2013

Domain-Driven Design: Referencing Aggregates and Using DTOs

After seeing and reviewing some existing systems, I've been thinking about how to improve the way I practice domain-driven design. In this entry, let me talk about two things that I think helps when implementing DDD:

  • Reference other aggregates by ID.
  • Use data transfer objects.

Reference Other Aggregates by ID

Let aggregates reference other aggregates by ID (identity), not the aggregate itself.

Vaughn Vernon has written about this.

It's quite often that I'd model aggregates and reference other entities/aggregates. But this makes the system more difficult to maintain, as an aggregate would need to know every other entity/aggregate. Take the order and product domain model as an example.

We have an order entity that has one-or-more order items. The order entity forms the aggregate root. Each order item would refer to a product, a quantity, and a total price.

Just to help paint a better picture, we would also have a repository for order (OrderRepository) and a repository for product (ProductRepository).

Usually, we would have the order item reference a product by type. Something like:

class Order {
  private Long id;
  private List<OrderItem> items;
  . . .
}

class OrderItem {
  private Product product;
  private int quantity;
  private Money unitPrice;
}

class Product {
  private Long id;
}

Instead of referencing product by type, we can reference it by ID, like this:

class Order {
  private Long id;
  private List<OrderItem> items;
  public Long id() { return this.id; }
  public List<OrderItem> items() { return Collections.unmodifiableList(this.items); }
  . . .
}

class OrderItem {
  private Long productId;
  private int quantity;
  private Money unitPrice;
  . . .
}

class Product {
  private Long id;
  . . .
}

NOTE: I'm using the Long type for simplicity here. Otherwise, a domain-related value object like ProductId could be used.

NOTE: Notice how getters are not used to expose read-only properties in the Order entity. This seems to be a pattern to help developers get off the JavaBean/getter-setter mindset, and stay with a domain object/entity mindset.

This allows a cleaner implementation, since the order aggregate would not be referencing the product. This also makes the ProductRepository more useful. Otherwise, products would have been retrieved using orders.

If you happen to be using JPA for persistence, you can use the @ManyToOne annotation and specify the targetEntity element:

@Entity
class Order {
  @Id
  private Long id;
  @OneToMany
  private List<OrderItem> items;
  . . .
}

@Entity
class OrderItem {
  @ManyToOne(targetEntity = Product.class)
  private Long productId;
  private int quantity;
  private Money unitPrice;
  . . .
}

@Entity
class Product {
  @Id
  private Long id;
  . . .
}

NOTE: Notice that a uni-directional relation between order and order items is used (not a bi-directional relation). This helps make the implementation simpler, as the order item does not reference its parent order.

The above results to the following DDL.

create table "order" (. . .);

create table order_item (
  . . .
  product_id bigint,
  . . .
);

create table product (
  id bigint generated by default as identity (start with 1),
  . . .
);

. . .
alter table order_item
    add constraint FKxxxxxxxxxxx2
    foreign key (order_id)
    references order;
. . .
alter table order_item
    add constraint FKxxxxxxxxxxx7
    foreign key (product_id)
    references product;
. . .

Given the above simple adjustment, we now have an order aggregate that does not directly reference the product aggregate. This creates a cleaner separation and a more modular domain model. It might even be possible to have an order that can reference a different type of product (as long as the ID type is compatible). This might allow a more re-usable order domain model.

Use Data Transfer Objects

Use simple structures (DTOs) to transfer data from the domain model to an interface and vice-versa.

I often see the domain model entities with all of its fields being modifiable via getters and setters. Also, they have zero-arguments constructors. This makes it difficult to apply invariants or business rules (as the entity can be in an invalid state). And the usual excuse I get for this is because they need to create a UI that allows the fields/properties to be modifiable.

For me, I'd rather have an entity that is always valid.

To preserve the business rules or invariants of the domain model, we cannot afford to have all of the entity fields being modifiable. This is where DTOs are needed.

package order.domain.model;

@Entity
class Order {
  . . .
  public int addItem(Long productId, int quantity, Money unitPrice) {
    . . .
  }
}

@Entity
class OrderItem {
}

. . .

package order.interfaces.dto;

class OrderDTO {
  private List<OrderItemDTO> items;
  public List<OrderItemDTO> getItems() {...}
  public void setItems(List<OrderItemDTO> ...) {...}
}

class OrderItemDTO {
  private Long productId;
  private int quantity;
  // getters and setters
}

The UI shall use the DTOs when displaying and receiving inputs. Once the data is captured in DTOs, assemblers are used to apply the changes to domain objects. This pattern can be seen in the dddsample project.

package order.interfaces.assembler;

class OrderAssembler {
  . . .
  public OrderDTO toDTO(Order order) {
    OrderDTO dto = new OrderDTO();
    . . .
    dto.setOrderDate(order.orderDate());
    dto.setEntryDate(order.entryDate());
    dto.setCustomerId(order.customerId());
    . . .
    return dto;
  }
  . . .
  public Order fromDTO(OrderDTO dto) {
    Order order;
    if (dto.getId() != null) {
      order = orderRepository.find(dto.getId());
    } else {
      order = new Order(..., dto.getCustomerId());
    }
    . . .
    Long productIds[] = dto.getProductIds();
    // Can use IN operator to get products with given IDs
    Map<Long, Product> products = productRepository.find(productIds);
    for (OrderItemDTO itemDTO : dto.getItems()) {
      Product product = products.get(itemDTO.getProductId());
      order.addItem(
          item.getProductId(),
          item.getQuantity(),
          product.getPrice());
    }
    return order;
  }
  . . .
}

This has helped me understand DDD implementations better. I hope it helps others too.

Monday, May 6, 2013

JSTL <c:url> and jsessionid Parameter

Here's a brief entry about the jsessionid parameter that appears whenever you use <c:url/> tags in your JSPs. I've seen several questions on how to remove it, and how to disable <c:url/> tags from generating it. After some research, and some quiet time to think about it, the solution is quite simple, and has something to do with JSPs having session defaulting to true.

The solution is to make sessions optional for pages that don't require them. Simply add a JSP directive <% @page session="false" %>.

JSPs, By Default, Create Sessions

Yes, JSPs create sessions, unless you specify otherwise. By default, JSP files get a HttpSession using request.getSession(true) method. So, by default, JSP files create a new session if none existed.

But what does this have to do with <c:url/> generating/adding a jsessionid parameter to the resulting URL?

The <c:url/> tag checks if the request supports cookies. In cases where the request is the first request (with no cookies sent, at least, none yet), the server could not be sure if the request supports cookies. In these cases, the <c:url/> tag implementation adds the jsessionid parameter when the request does not support cookies. This is called URL session tracking.

When will cookies not be supported?

There are a couple of possibilities. One would be when the browser has it disabled. The other would be when a search bot is crawling your web site.

So, it's not the tag's fault. It's doing what it is supposed to do. It also escapes characters to form a valid URL.

Others might think that simply not using the <c:url/> tag would solve the problem. But I think that creates more problems, as your URLs may not have been escaped properly.

Still others have conjured up filters that would remove the session ID from the URL. Please don't. Making sessions optional may be a better option.

JSPs and Template Frameworks like SiteMesh and Tiles

If you're using template frameworks like SiteMesh and Tiles, you're likely to be using JSPs as decorators or templates (for headers, navigation bars, footers, and others). Note that these JSPs may also create a session, unless you explicitly specify it using the page directive. So, be careful. It might be a good practice to keep these template pages with session creation set to false.

URLs and Crawlers

The problem of inadvertently creating session is more apparent when developing web sites that are crawled by bots (like Googlebot). If you happen to see URLs crawled by the search bot having jsessionid parameters, then you should be looking at the pages that should not have created sessions.

Not only are these session parameters ugly, they simply ruin your web site's search engine results page. Who would want to visit your site with a pre-existing jsessionid?

Conclusion

A good practice would be to keep your landing page (usually an index.jsp, or whatever your welcome page is in your web.xml) with a page directive to turn-off session creation. Another page that should not create sessions would be your login page. The session is only required after the user has logged in (not before).

If a user visits your page with an existing session, it will still work. Remember, we're just making session creation optional. We're not making it invalid.

I hope this brief entry can help several Java developers and save some of our precious time.