Monday, June 7, 2021

Changing The Way We Use JPA

I've been updating some training materials recently, and thinking about better ways of teaching and talking about JPA. One of the things I've been thinking about is how we have typically used JPA, and how that should change given the pains I've experienced (and observed).

JPA is often seen as a set of annotations (or XML files) that provide O/R (object-relational) mapping information. And most developers think that the more mapping annotations they know and use, the more benefits they get. But the past few years of wrestling with small to medium monoliths/systems (with about 200 tables/entities) have taught me something else.

TL;DR

  1. Reference entities by ID (only map entity relationships within an aggregate)
  2. Don't let JPA steal your Identity (avoid @GeneratedValue when you can)
  3. Use ad-hoc joins to join unrelated entities

Reference Entities by Identifier

Only map entity relationships within an aggregate.

Tutorials (and training) would typically go about teaching and covering all possible relationship mappings. After basic mappings, many would start from simple uni-directional @ManyToOne mapping. Then proceed to bi-directional @OneToMany and @ManyToOne. Unfortunately, most often than not, they fail to explicitly point out that it is perfectly fine to not map the relationship. So, beginners would often complete the training thinking that it would be a mistake to not map a related entity. They mistakenly think that a foreign key field must be mapped as a related entity.

In fact, it is not an error, if you change the @ManyToOne mapping below…

@Entity
public class SomeEntity {
    // ...
    @ManyToOne private Country country;
    // ...
}

@Entity
public class Country {
    @Id private String id; // e.g. US, JP, CN, CA, GB, PH
    // ...
}

…into a basic field that contains the primary key value of the related entity.

@Entity
public class SomeEntity {
    // ...
    @Column private String countryId;
    // ...
}

@Entity
public class Country {
    @Id private String id; // e.g. US, JP, CN, CA, GB, PH
    // ...
}

Why is this a problem?

Mapping all entity relationships increases the chances of unwanted traversals that usually lead to unnecessary memory consumption. This also leads to an unwanted cascade of EntityManager operations.

This may not be much if you're dealing with just a handful of entities/tables. But it becomes a maintenance nightmare when working with dozens (if not hundreds) of entities.

When do you map a related entity?

Map related entities only when they are within an Aggregate (in DDD).

Aggregate is a pattern in Domain-Driven Design. A DDD aggregate is a cluster of domain objects that can be treated as a single unit. An example may be an order and its line-items, these will be separate objects, but it's useful to treat the order (together with its line items) as a single aggregate.

@Entity
public class Order {
    // ...
    @OneToMany(mappedBy = "order", ...) private List<OrderItem> items;
    // ...
}

@Entity
public class OrderItem {
    // ...
    @ManyToOne(optional = false) private Order order;
    // ...
}

More modern approaches to aggregate design (see Vaughn Vernon's Implementing Domain-Driven Design) advocate a cleaner separation between aggregates. It is a good practice to refer to an aggregate root by storing its ID (unique identifier), not a full reference.

If we expand the simple order example above, the line-item (OrderItem class) should not have a @ManyToOne mapping to the product (since it is another aggregate in this example). Instead, it should just have the ID of the product.

@Entity
public class Order {
    // ...
    @OneToMany(mappedBy = "order", ...) private List<OrderItem> items;
    // ...
}

@Entity
public class OrderItem {
    // ...
    @ManyToOne(optional = false) private Order order;
    // @ManyToOne private Product product; // <-- Avoid this!
    @Column private ... productId;
    // ...
}

But… what if the Product (aggregate root entity) has its @Id field mapped as @GeneratedValue? Are we forced to persist/flush first and then use the generated ID value?

And, what about joins? Can we still join those entities in JPA?

Don't Let JPA Steal Your Identity

Using @GeneratedValue may initially make the mapping simple and easy to use. But when you start referencing other entities by ID (and not by mapping a relationship), it becomes a challenge.

If the Product (aggregate root entity) has its @Id field mapped as @GeneratedValue, then calling getId() may return null. When it returns null, the line-item (OrderItem class) will not be able to reference it!

In an environment where all entities always have a non-null Id field, referencing any entity by ID becomes easier. Furthermore, having non-null Id fields all the time, makes equals(Object) and hashCode() easier to implement.

And because all Id fields become explicitly initialized, all (aggregate root) entities have a public constructor that accepts the Id field value. And, as I've posted long ago, a protected no-args constructor can be added to keep JPA happy.

@Entity
public class Order {
    @Id private Long id;
    // ...
    public Order(Long id) {
        // ...
        this.id = id;
    }
    public Long getId() { return id; }
    // ...
    protected Order() { /* as required by ORM/JPA */ }
}

But beware! When using Spring Data JPA to save() an entity that does not use @GeneratedValue on its @Id field, an unnecessary SQL SELECT is issued before the expected INSERT. This is due to SimpleJpaRepository's save() method (shown below). It relies on the presence of the @Id field (non-null value) to determine whether to call persist(Object) or merge(Object).

public class SimpleJpaRepository // ...
	@Override
	public <S extends T> save(S entity) {
		// ...
		if (entityInformation.isNew(entity)) {
			em.persist(entity);
			return entity;
		} else {
			return em.merge(entity);
		}
	}
}

The astute reader will notice that, if the @Id field is never null, the save() method will always call merge(). This causes the unnecessary SQL SELECT (before the expected INSERT).

Fortunately, the work-around is simple — implement Persistable<ID>.

@MappedSuperclass
public abstract class BaseEntity<ID> implements Persistable<ID> {
	@Transient
	private boolean persisted = false;
	@Override
	public boolean isNew() {
		return !persisted;
	}
	@PostPersist
	@PostLoad
	protected void setPersisted() {
		this.persisted = true;
	}
}

The above also implies that all updates to entities must be done by loading the existing entity into the persistence context first, and applying changes to the managed entity.

Use Ad-hoc Joins to Join Unrelated Entities

And, what about joins? Now that we reference other entities by ID, how can we join unrelated entities in JPA?

In JPA version 2.2, unrelated entities cannot be joined. However, I cannot confirm if this has become a standard in version 3.0, where all javax.persistence references were renamed to jakarta.persistence.

Given the OrderItem entity, the absence of the @ManyToOne mapping causes it to fail to be joined with the Product entity.

@Entity
public class Order {
    // ...
}

@Entity
public class OrderItem {
    // ...
    @ManyToOne(optional = false) private Order order;
    @Column private ... productId;
    // ...
}

Thankfully 😊, Hibernate 5.1.0+ (released back in 2016) and EclipseLink 2.4.0+ (released back in 2012) have been supporting joins of unrelated entities. These joins are also referred to as ad-hoc joins.

SELECT o
  FROM Order o
  JOIN o.items oi
  JOIN Product p ON (p.id = oi.productId) -- supported in Hibernate and EclipseLink

Also, this has been raised as an API issue (Support JOIN/ON for two root entities). I really hope that it will become a standard soon.

In Closing

What do you think about the above changes? Are you already using similar approaches? Do you use native SQL to explicitly retrieve a generated value (e.g. sequence object) to create an entity with a non-null Id field? Do you use entity-specific ID types to differentiate ID values? Let me know in the comments below.

Thursday, September 17, 2020

Reduce Repetitive Code in Spring MVC Controllers

After spending some time doing sustained engineering (a.k.a. maintaining legacy code), I ventured to reduce repetitive code in Spring MVC @Controllers. I started with an abstract base controller class. But I soon found out that it was a dead-end because @RequestMapping is not inherited from (or combined with) parent classes and/or interfaces (see Spring MVC @RequestMapping Inheritance).

With some free time to further think about this, I took different approach.

@Controller
public class RepositoryCrudController {

	private final Repositories repositories;
	private final RepositoryInvokerFactory repositoryInvokerFactory;

	// ...

	@RequestMapping("/{repository}")
	String index(@PathVariable("repository") String repositoryKey,
			Pageable pageable, Model model) {
		// ... (only if repository has findAll(Pageable) method)
		return repositoryKey + "/index";
	}

	@GetMapping("/{repository}/{id}")
	String show(@PathVariable("repository") String repositoryKey,
			@PathVariable("id") Object id, Model model) {
		// ... (only if repository has findById method)
		return repositoryKey + "/show";
	}

	@GetMapping(path = "/{repository}", param = "create")
	String create(...) {
		// ... (only if repository has save method)
		return repositoryKey + "/create";
	}

	@PostMapping("/{repository}")
	String save(...) {
		// ... (only if repository has save method)
		return "redirect:/" + repositoryKey + "/{id}";
	}

	@GetMapping(path = "/{repository}/{id}", param = "edit")
	// ... edit (only if repository has findById and save methods)
	@PutMapping("/{repository}/{id}")
	// ... update (only if repository has save method)
	// @DeleteMapping("/{repository}/{id}")
	// ... delete (only if repository has deleteById method)
}

This approach is largely inspired by RepositoryEntityController of Spring Data REST.

Instead of defining an abstract base controller class, I created a concrete controller class with the default (or repetitive) behavior. The default behavior relies on invoking methods on Spring Data repositories.

For custom controllers, instead of creating subclasses (of the abstract base class), the controller can simply define handler methods that behave differently. For example:

@Entity
public class Article {...}

// Spring Data JPA
public interface ArticleRepository extends CrudRepository<Article, ...> {...}

@Controller
@RequestMapping("/articles")
public class ArticlesController {

	// no need for index() handler method
	// just set-up "articles/index" view

	// no need for show() handler method
	// just set-up "articles/show" view

	@GetMapping(param = "create")
	String create(Model model) {
		// Do something that is _not_ *default* behavior
		// e.g. provide options for dropdowns, or use form-backing object/JavaBean
		// ...
		return "articles/create";
	}

	// no need for save() handler method
	// just set-up "articles/show" view

	@GetMapping(path = "/{id}", param = "edit")
	String edit(@PathVariable("id") ... id, Model model) {
		// Do something that is _not_ *default* behavior
		// e.g. provide options for dropdowns, or use form-backing object/JavaBean
		// ...
		return "articles/edit";
	}

	// no need for update() handler method
	// just set-up "articles/show" view

}

The above will work because Spring MVC chooses the more specific mapping first. When a GET /articles?create request is received, the ArticlesController will be chosen to handle it (and not RepositoryCrudController). But if ArticlesController handler methods were not defined, the GET /articles?create request would have been handled by the RepositoryCrudController.

With this simple fall-back controller that has the default behavior, developers (team mates) can then focus on the domain, creating views, or creating controllers with custom behavior (e.g. Ajax, Server-generated JavaScript Responses).

That's all for now.

Tuesday, June 25, 2019

Spring with Rails' jQuery UJS

I’ve always wanted to try to see if I could use Rails’ jQuery UJS in a Spring Boot project. The UJS in jquery-ujs stands for unobtrusive JavaScript. I really like how UJS wires event handlers to eligible DOM elements marked with HTML5 data-* attributes. I find myself wanting to see more of this approach being used in Spring Boot web apps. I wonder why there’s very little mentioned on the web about this. Or, may be I’ve been looking at the wrong places.

Anyway, here are some things jQuery UJS can do, and the related source code is on GitHub (albeit using a different example).

Non-GET Links (e.g. DELETE)

When I need to render a link that deletes an item, I would use a <button> wrapped in a <form> with a hidden _method field with a value of delete. The <form> is not visible to the user. But the visible button is used to submit the <form>. Some CSS is used to make the button look like a link.

<form action="/articles/42" method="post">
  <input type="hidden" name="_method" value="delete" />
  <button class="btn btn-link">Delete</button>
</form>

Thanks to Spring’s HiddenHttpMethodFilter (also automatically configured in Spring Boot), when this <form> is submitted, it will be received as a request with a method of DELETE. It maps to @DeleteMapping(path="/articles/{id}") in the related @Controller.

While the above works, there is an easier way with jQuery UJS. All that is needed to render a link to delete an item is this:

<a href="/articles/42" data-method="delete">Delete</a>

jQuery UJS will enhance links that have data-method attribute. When the above example link is clicked, the JavaScript will create a <form>. The action attribute of this <form> is set to the value of link’s href. The method is set to post. A hidden _method field is added to the <form> and set to the value of the link’s data-method. Finally, the <form> is submitted (and the link is not followed).

Confirmation Dialogs

Most often than not, when it comes to deleting anything, it would be good to confirm with the user. This could be implemented as a simple dialog via window.confirm(). If we build from the previous example, the <form> would look like this:

<form action="/articles/42" method="post"
    onsubmit="return confirm('Are you sure?')">
  <input type="hidden" name="_method" value="delete" />
  <button>Delete</button>
</form>

While the above works, jQuery UJS shows us a better way. All that is needed to confirm before deleting is this:

<a href="/articles/42?delete" data-method="delete"
    data-confirm="Are you sure?">Delete</a>

jQuery UJS will enhance links (and <form>s too) that have data-confirm attribute. When the above example link is clicked, the JavaScript will call confirm() to show a dialog containing the text that is the value of the attribute. If the user chooses to cancel, the creation/submission of the <form> (due to data-method) does not take place.

Ajax Links

After deleting an item, the page usually reloads to show that the deleted element has indeed been removed.

Let's say the items are displayed in a table. Each row has a unique id.

<table>
  <tr id="article:18">
    <!-- data cells for /articles/18 -->
    <td><a href="/articles/18?delete"
        data-method="delete" data-confirm="Are you sure?">Delete</a></td>
  </tr>
  <tr id="article:42">
    <!-- data cells for /articles/42 -->
    <td><a href="/articles/42?delete"
        data-method="delete" data-confirm="Are you sure?">Delete</a></td>
  </tr>
  <!-- other rows -->
</table>

Assuming that we can simply remove the HTML element that represented the deleted item, then we can probably send the DELETE request asynchronously and get a response that would remove the related HTML element. jQuery UJS makes this as easy as adding data-remote="true" and some minor changes to the server-side controller.

For the HTML, all we need is data-remote="true".

<a href="/articles/42?delete" data-remote="true"
    data-method="delete"
    data-confirm="Are you sure?">Delete</a>

When the link is clicked, jQuery UJS will again send the DELETE request. But this time, it will be sent asynchronously using Ajax. Doing so enables the browser not to reload the entire page. And depending on the server’s response, can update just a portion of the page. Thus, providing a slightly better user experience.

For the server-side controller, we need to send a different response when the request is expecting text/javascript. We add a handler method that will respond with text/javascript by using the produces element of @RequestMapping. The response will remove the related HTML element(s).

// inside a @Controller
@DeleteMapping(path="/articles/{id}")
String delete(... id) {
    // ... delete article with given identifier
    return "redirect:/articles";
}

@DeleteMapping(path="/articles/{id}",
    produces="text/javascript")
String delete(... id) {
    // ... delete article with given identifier
    return "articles/delete";
}

The view is a JSP that contains text/javascript. This will be executed by jQuery UJS.

<%-- articles/delete.js.jsp --%>
<%@ page contentType="text/javascript" %>
$('#article:${id}').remove();

Partials and Server-generated JavaScript Responses

Now what happens if we wanted to have an edit link to get some HTML content and show it up in a modal (without a page refresh)?

Here's what we can do. We send a GET request asynchronously. The response is expected to contain JavaScript that would append the HTML in targeted places in the document, and then trigger the modal to appear.

  <a href="/articles/42?edit" data-remote="true">Edit</a>

When the response is expected to be text/javascript, articles/edit.js.jsp is rendered. Otherwise, the usual articles/edit.jsp is rendered.

// inside a @Controller
@GetMapping(path="/articles/{id}", params={"edit"})
String edit(... id, ...) {
    // ...
    return "articles/edit";
}

@GetMapping(path="/articles/{id}", params={"edit"},
    produces="text/javascript")
String editViaAjax(... id, ...) {
    // ...
    return "articles/edit";
}

The edit.jsp renders the <form> (a partial, not a complete HTML document) which has been refactored to its own JSP to avoid repetition.

<%-- articles/edit.jsp --%>
<!-- -->
  <jsp:include page="_form.jsp" />
<!-- -->

The edit.js.jsp renders the same <form> (a partial, not a complete HTML document) as a string in JS. Then includes it in the modal. It was tricky to render _form.jsp as a string. I had to use <c:import>.

<%-- articles/edit.js.jsp --%>
<%@ page contentType="text/javascript" %>
<c:import var="html" url="…_form.jsp" />
<!-- escape double quotes and remove new lines -->
(function() {
  const $modal = $('#...'); // ID of modal element
  $modal.find('.modal-body').html('${html}');
  if (!$modal.is(':visible')) {
    $modal.modal('show');
  }
})()

For this to work, another InternalResourceViewResolver (IRVR) bean with text/javascript as the contentType is configured. This bean uses the same prefix and a slightly different suffix: .js.jsp. That way, when the request is expecting text/javascript, the CNVR will favor using the IRVR bean with text/javascript and it ends up rendering articles/edit.js.jsp.

Ajax Forms

The data-remote="true" attribute can also be applied to <form>s. With it, jQuery UJS will handle the form submission as an Ajax request. And when the form is being submitted, the buttons can be disabled by adding data-disabled-with. For example,

<form ...>
  <!-- ... -->
  <button data-disable-with="Saving...">Save</button>
</form ...>

When the above form is submitted, jQuery UJS will disable the button and change its content to "Saving...".

Closing Thoughts

I’ve barely touched the surface of Rails’ jQuery UJS. There is so much more that it can offer. I look forward to using it (and similar techniques) in my web apps.

Monday, November 12, 2018

Revisions and Immutability

Here's a brief post. I'm not sure how to start it. It's one of those "why didn't I think of that" moments while reviewing some existing code. Due to NDAs, I cannot share the actual code. It has something to do with handling revisions. The closest thing I can relate to is how WordPress (WP) handles blog posts and revisions.

In WP, the wp_insert_post function inserts or updates a post. It checks the ID field to determine if it will carry out an INSERT or an UPDATE. If the post is being updated, it checks if changes were made. If so, a revision is saved. A limit for the number of revisions to keep can be set. If so, the oldest ones are deleted.

This sounds like something that can be modeled as a rich domain entity. Here's a first try.

@Entity
... class Post {
    @Id @GeneratedValue ... id;
    ... name;
    ... title;
    ... content;
    ... excerpt;
    ... status; // e.g. 'draft', 'publish', 'inherit'
    ... type; // e.g. 'post', 'revision'
    @OneToMany @JoinColumn(name="parent_post_id") ... List<Post> revisions;
    ...
    // setters and getters
}
Post post = new Post();
post.setTitle("Lorem Ipsum");
post.setContent("...");
// save post
...
post = // retrieve existing post for updates
post.setContent("..."); // how can we ensure that revision is created?

In the first try, the setter methods pose a challenge to ensuring that a revision is created when the post is updated. Let's give it another try. Here's our second try.

// Immutable class
@Embeddable
... class PostData {
    ... title;
    ... content;
    ... excerpt;
    // getters only
    ... getTitle() { return title; }
    ... getContent() { return content; }
    ... getExcerpt() { return excerpt; }
    // equals() method to compare with another post data
    // to see if there are changes
}

@Entity
... class Post {
    @Id @GeneratedValue ... id;
    ... name; // for a revision, will contain parent ID and revision #
    @Embedded ... PostData postData; // read-only
    ... status; // e.g. 'draft', 'published', 'inherit'
    ... type; // e.g. 'post', 'revision'
    @OneToMany @JoinColumn(name="parent_post_id") ... List<Post> revisions;
    ...
    ... getTitle() { return this.postData.getTitle(); }
    ... getContent() { return this.postData.getContent(); }
    ... getExcerpt() { return this.postData.getExcerpt(); }
    ... getName() { return name; }
}

This is when I got my "why didn't I think of that" moment!

Note how we encapsulated the post data into its own type — PostData. It is immutable. This makes it possible to ensure that a revision is created when the post is updated.

PostData postData = new PostData("Lorem Ipsum", "...", "...");
Post post = new Post(postData);
// save post
...
post = // retrieve existing post for updates
// post.setContent("..."); // not possible
post.updateData(new PostData("...", "...", "...")); // ensure that revision is created

And here's how we create revisions.

@Entity
... class Post {
    ...
    @Embedded ... PostData postData; // read-only
    ...
    @OneToMany @JoinColumn(name="parent_post_id") ... List<Post> revisions;
    ...
    public Post(PostData postData) {
        this(postData, null);
    }
    /* package private */ Post(PostData postData, Post parent) {
        if (postData == null) {
            throw new IllegalArgumentException(...);
        }
        this.postData = postData;
        if (parent == null) {
            this.type = "post";
            this.status = "draft";
            this.name = null;
            this.revisions = new ArrayList<>();
        } else {
            this.type = "revision";
            this.status = "inherit";
            this.name = "" + parent.getId() + "-revision" + (parent.getRevisionsCount() + 1);
            this.revisions = null;
        }
        ...
    }
    ...
    ... void updateData(PostData newPostData) {
        if (this.postData.equals(newPostData)) {
            // no changes, no revisions added
            return;
        }
        ...
        // creates a revision
        PostData beforePostData = this.postData;
        this.revisions.add(0, new Post(beforePostData, this));
        // store latest changes
        this.postData = newPostData;
        // limit to number of revisions to keep
        if (this.revisions.size() > ...) {
            // delete the excess ones
            for (...) {
                this.revisions.remove(this.revisions.size() - 1);
            }
        }
        ...
    }
    ...
}

Like I said, this one is a brief post. Let me know in the comments below if it's something you've seen before, or, just like me, it gave you a "why didn't I think of that" moment.

Wednesday, July 18, 2018

Caching in Spring Boot with Spring Security

In this post, I’d like to share a lesson learned by one of the teams at O&B. They were using Spring Boot with Spring Security.

By default, anything that is protected by Spring Security is sent to the browser with the following HTTP header:

Cache-Control: no-cache, no-store, max-age=0, must-revalidate

Essentially, the response will never be cached by the browser. While this may seem inefficient, there is actually a good reason for this default behavior. When one user logs out, we don’t want the next logged in user to be able to see the previous user’s resources (and this is possible if they’re cached).

It makes sense to not cache anything by default, and leave caching to be explicitly enabled. But it’s not good if nothing is cached, as it will lead to high bandwidth usage and slow page loads.

Good thing it is very easy to enable caching of static content in Spring Boot (even with Spring Security). Simply configure a cache period. And that’s it!

# Boot 2.x
spring.resources.cache.cachecontrol.max-age=14400

# Boot 1.x
spring.resources.cache-period=14400

But there are some gotchas! With some versions, it ain’t that simple! Let me explain further.

There are several ways content can be returned:

  1. Static content through Spring Boot auto-configured static resource request handler
  2. Controller method returning view name (e.g. resolves to a JSP)
  3. Controller method returning HttpEntity (or ResponseEntity)

Enable Caching of Static Content

The first (serving static content) is handled by configuring the said property (usually in application.properties as shown above).

Set via HttpServletResponse

In the second case, the controller handler method may choose to set "Cache-Control" headers through a HttpServletResponse method parameter.

@Controller
... class ... {
    @RequestMapping(...)
    public String ...(..., HttpServletResponse response) {
        response.setHeader("Cache-Control", "max-age=14400");
        return ...; // view name
    }
}

This works, as long as Spring Security does not overwrite it.

Set via HttpEntity/ResponseEntity

In the third case, the controller handler method may choose to set "Cache-Control" headers of the returned HTTP entity.

@Controller
... class ... {
    @RequestMapping(...)
    public ResponseEntity<...> ...(...) {
        return ResponseEntity.ok().cacheControl(...).body(...);
    }
}

This works, as long as Spring Security has not written its own "Cache-Control" headers yet.

Under the Hood

To understand when and why it works, here are the relevant sequences.

With Spring Security Web 4.0.x, 4.2.0 up to 4.2.4 and above, the following sequence occurs:

  1. The HeaderWriterFilter delegates to CacheControlHeadersWriter to write the "Cache-Control" headers (including "Pragma" and "Expires"), if no cache headers exist.
  2. Controller handler method (if matched) is invoked. The method can:
    • Explicitly set a header in HttpServletResponse.
    • Or, set a header in the returned HttpEntity or ResponseEntity (refer to the handleReturnValue() method of HttpEntityMethodProcessor).
      • Note that HttpEntityMethodProcessor only writes the headers (from HttpEntity) to the actual response if they do not exist yet. This becomes a problem, since back in #1, the headers have already been set.
  3. If no controller handles the request, then the Spring Boot auto-configured static resource request handler gets its chance. It tries to serve static content, and if configured to cache, it overwrites the "Cache-Control" headers (and clears the values of "Pragma" and "Expires" headers, if any). The static resource handler is a ResourceHttpRequestHandler object (refer to the applyCacheControl() method in its WebContentGenerator base class).
    • However, in Spring Web MVC 4.2.5, the WebContentGenerator only writes the "Cache-Control" headers only if it does not exist!. This becomes a problem, since back in #1, the headers have already been set.
    • In Spring Web MVC 4.2.6 and above, it adds the "Cache-Control" headers even if it already exists. So, no problem even if the headers have been set in #1.

With Spring Security Web 4.1.x, 4.2.5, and above (version 4.2.5 is used in Spring Boot 1.5.11), the sequence has changed. It goes something like this:

  1. Controller handler method (if matched) is invoked. The method can:
    • Explicitly set a header in HttpServletResponse.
    • Or, set a header in the returned HttpEntity or ResponseEntity (refer to the handleReturnValue() method of HttpEntityMethodProcessor).
      • Note that HttpEntityMethodProcessor only writes the headers (from HttpEntity) to the actual response if they do not exist yet. No problem, since no headers have been set yet.
  2. If no controller handles the request, then the Spring Boot auto-configured static resource request handler gets its chance. It tries to serve static content, and if configured to cache, it overwrites the "Cache-Control" headers (and clears the values of "Pragma" and "Expires" headers, if any).
  3. The HeaderWriterFilter delegates to CacheControlHeadersWriter to write the "Cache-Control" headers (including "Pragma" and "Expires"), if no cache headers exist.
    • No problem, since it will not overwrite if cache headers have already been set.

Working Versions

The above three cases of controlling caching all work in Spring Boot 1.5.11 and Spring Boot 2.x. But in case upgrading to those versions is not possible, please see the following classes and check if it has your desired behavior (using the above sequences):

  • HeaderWriterFilter (see doFilterInternal method)
  • CacheControlHeadersWriter (see writeHeaders() method)
  • WebContentGenerator (see applyCacheControl() method)
  • HttpEntityMethodProcessor (see handleReturnValue() method)

Also, be aware that Spring Security Web 4.2.5 and above will write the following HTTP headers (overwrite it, even if they are already set, like in a controller for example):

  • X-Content-Type-Options via XContentTypeOptionsHeaderWriter
  • Strict-Transport-Security via HstsHeaderWriter
  • X-Frame-Options via XFrameOptionsHeaderWriter
  • X-XSS-Protection via XXssProtectionHeaderWriter

This is because, unlike CacheControlHeadersWriter, the header writers for the above do not check if the headers already exist. They simply set their respective HTTP headers. Please refer to their respective header writer classes and issue #5193.

Another option is to have Spring Security ignore static resource requests. That way, the configured cache period will not be overwritten.

@Configuration
@EnableWebSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    public void configure(WebSecurity web) throws Exception {
        web.ignoring().antMatchers("/css/**", "/js/**");
        // If the above paths are served by the
        // Spring Boot auto-configured
        // static resource request handler,
        // and a cache period is specified,
        // then it will have a "Cache-Control"
        // HTTP header in its response.
        // And it would NOT get overwritten by Spring Security.
    }
}

That's all for now. Hope this clears things up.

Wednesday, June 20, 2018

Dealing with Domain Objects in Spring MVC

I was recently surprised by how one code base had public default constructors (i.e. zero-arguments constructors) in all their domain entities, and had getters and setters for all the fields. As I dug deeper, I found out that the reason why the domain entites are the way they are is largely because the team thinks it was required by the web/MVC framework. And I thought it would be a good opportunity to clear up some misconceptions.

Specifically, we'll look at the following cases:

  1. No setter for generated ID field (i.e. the generated ID field has a getter but no setter)
  2. No default constructor (e.g. no public zero-arguments constructor)
  3. Domain entity with child entities (e.g. child entities are not exposed as a modifiable list)

Binding Web Request Parameters

First, some specifics and some background. Let's base this on a specific web/MVC framework - Spring MVC. When using Spring MVC, its data binding binds request parameters by name. Let's use an example.

@Controller
@RequestMapping("/accounts")
... class ... {
    ...
    @PostMapping
    public ... save(@ModelAttribute Account account, ...) {...}
    ...
}

Given the above controller mapped to "/accounts", where can an Account instance come from?

Based on documentation, Spring MVC will get an instance using the following options:

  • From the model if already added via Model (like via @ModelAttribute method in the same controller).
  • From the HTTP session via @SessionAttributes.
  • From a URI path variable passed through a Converter.
  • From the invocation of a default constructor.
  • (For Kotlin only) From the invocation of a "primary constructor" with arguments matching to Servlet request parameters; argument names are determined via JavaBeans @ConstructorProperties or via runtime-retained parameter names in the bytecode.

Assuming an Account object is not added in the session, and that there is no @ModelAttribute method, Spring MVC will end up instantiating one using its default constructor, and binding web request parameters by name. For example, the request contains "id" and "name" parameters. Spring MVC will try to bind them to "id" and "name" bean properties by invoking "setId" and "setName" methods, respectively. This follows JavaBean conventions.

No Setter Method for Generated ID Field

Let's start with something simple. Let's say that we have an Account domain entity. It has an ID field that is generated by the persistent store, and only provides a getter method (but no setter method).

@Entity
... class Account {
    @Id @GeneratedValue(...) private Long id;
    ...
    public Account() { ... }
    public Long getId() { return id; }
    // but no setId() method
}

So, how can we have Spring MVC bind request parameters to an Account domain entity? Are we forced to have a public setter method for a field that is generated and read-only?

In our HTML form, we will not place the "id" as a request parameter. We will place it as a path variable instead.

We use a @ModelAttribute method. It is called prior to the request handling method. And it supports pretty much the same parameters as a regular request handling method. In our case, we use it to retrieve an Account domain entity with the given unique identifier, and use it for further binding. Our controller would look something like this.

@Controller
@RequestMapping("/accounts")
... class ... {
    ...
    @ModelAttribute
    public Account populateModel(
            HttpMethod httpMethod,
            @PathVariable(required=false) Long id) {
        if (id != null) {
            return accountRepository.findById(id).orElseThrow(...);
        }
        if (httpMethod == HttpMethod.POST) {
            return new Account();
        }
        return null;
    }

    @PutMapping("/{id}")
    public ... update(...,
            @ModelAttribute @Valid Account account, ...) {
        ...
        accountRepository.save(account);
        return ...;
    }

    @PostMapping
    public ... save(@ModelAttribute @Valid Account account, ...) {
        ...
        accountRepository.save(account);
        return ...;
    }
    ...
}

When updating an existing account, the request would be a PUT to "/accounts/{id}" URI. In this case, our controller needs to retrieve the domain entity with the given unique identifier, and provide the same domain object to Spring MVC for further binding, if any. The "id" field will not need a setter method.

When adding or saving a new account, the request would be a POST to "/accounts". In this case, our controller needs to create a new domain entity with some request parameters, and provide the same domain object to Spring MVC for further binding, if any. For new domain entities, the "id" field is left null. The underlying persistence infrastructure will generate a value upon storing. Still, the "id" field will not need a setter method.

In both cases, the @ModelAttribute method populateModel is called prior to the mapped request handling method. Because of this, we needed to use parameters in populateModel to determine which case it is being used in.

No Default Constructor in Domain Object

Let's say that our Account domain entity does not provide a default constructor (i.e. no zero-arguments constructor).

... class Account {
    public Account(String name) {...}
    ...
    // no public default constructor
    // (i.e. no public zero-arguments constructor)
}

So, how can we have Spring MVC bind request parameters to an Account domain entity? It does not provide a default constructor.

We can use a @ModelAttribute method. In this case, we want to create an Account domain entity with request parameters, and use it for further binding. Our controller would look something like this.

@Controller
@RequestMapping("/accounts")
... class ... {
    ...
    @ModelAttribute
    public Account populateModel(
            HttpMethod httpMethod,
            @PathVariable(required=false) Long id,
            @RequestParam(required=false) String name) {
        if (id != null) {
            return accountRepository.findById(id).orElseThrow(...);
        }
        if (httpMethod == HttpMethod.POST) {
            return new Account(name);
        }
        return null;
    }

    @PutMapping("/{id}")
    public ... update(...,
            @ModelAttribute @Valid Account account, ...) {
        ...
        accountRepository.save(account);
        return ...;
    }

    @PostMapping
    public ... save(@ModelAttribute @Valid Account account, ...) {
        ...
        accountRepository.save(account);
        return ...;
    }
    ...
}

Domain Entity with Child Entities

Now, let's look at a domain entity that has child entities. Something like this.

... class Order {
    private Map<..., OrderItem> items;
    public Order() {...}
    public void addItem(int quantity, ...) {...}
    ...
    public Collection<CartItem> getItems() {
        return Collections.unmodifiableCollection(items.values());
    }
}

... class OrderItem {
    private int quantity;
    // no public default constructor
    ...
}

Note that the items in an order are not exposed as a modifiable list. Spring MVC supports indexed properties and binds them to an array, list, or other naturally ordered collection. But, in this case, the getItems method returns an unmodifiable collection. This means that an exception would be thrown when an object attempts to add/remove items to/from it. So, how can we have Spring MVC bind request parameters to an Order domain entity? Are we forced to expose the order items as a mutable list?

Not really. We must refrain from diluting the domain model with presentation-layer concerns (like Spring MVC). Instead, we make the presentation-layer a client of the domain model. To handle this case, we create another type that complies with Spring MVC, and keep our domain entities agnostic of the presentation layer.

... class OrderForm {
    public static OrderForm fromDomainEntity(Order order) {...}
    ...
    // public default constructor
    // (i.e. public zero-arguments constructor)
    private List<OrderFormItem> items;
    public List<OrderFormItem> getItems() { return items; }
    public void setItems(List<OrderFormItem> items) { this.items = items; }
    public Order toDomainEntity() {...}
}

... class OrderFormItem {
    ...
    private int quantity;
    // public default constructor
    // (i.e. public zero-arguments constructor)
    // public getters and setters
}

Note that it is perfectly all right to create a presentation-layer type that knows about the domain entity. But it is not all right to make the domain entity aware of presentation-layer objects. More specifically, presentation-layer OrderForm knows about the Order domain entity. But Order does not know about presentation-layer OrderForm.

Here's how our controller will look like.

@Controller
@RequestMapping("/orders")
... class ... {
    ...
    @ModelAttribute
    public OrderForm populateModel(
            HttpMethod httpMethod,
            @PathVariable(required=false) Long id,
            @RequestParam(required=false) String name) {
        if (id != null) {
            return OrderForm.fromDomainEntity(
                orderRepository.findById(id).orElseThrow(...));
        }
        if (httpMethod == HttpMethod.POST) {
            return new OrderForm(); // new Order()
        }
        return null;
    }

    @PutMapping("/{id}")
    public ... update(...,
            @ModelAttribute @Valid OrderForm orderForm, ...) {
        ...
        orderRepository.save(orderForm.toDomainEntity());
        return ...;
    }

    @PostMapping
    public ... save(@ModelAttribute @Valid OrderForm orderForm, ...) {
        ...
        orderRepository.save(orderForm.toDomainEntity());
        return ...;
    }
    ...
}

Closing Thoughts

As I've mentioned in previous posts, it is all right to have your domain objects look like a JavaBean with public default zero-arguments constructors, getters, and setters. But if the domain logic starts to get complicated, and requires that some domain objects lose its JavaBean-ness (e.g. no more public zero-arguments constructor, no more setters), do not worry. Define new JavaBean types to satisfy presentation-related concerns. Do not dilute the domain logic.

That's all for now. I hope this helps.

Thanks again to Juno for helping me out with the samples. The relevant pieces of code can be found on GitHub.

Wednesday, January 24, 2018

JasperReports: The Tricky Parts

If you have been programming in Java long enough, chances are you needed to generate reports for business users. In my case, I've seen several projects use JasperReports® Library to generate reports in PDF and other file formats. Recently, I've had the privilege of observing Mike and his team use the said reporting library and the challenges they faced.

JasperReports in a Nutshell

In a nutshell, generating reports using JasperReports (JR) involves three steps:

  1. Load compiled report (i.e. load a JasperReport object)
  2. Run report by filling it with data (results to a JasperPrint object)
  3. Export filled report to a file (e.g. use JRPdfExporter to export to PDF)

In Java code, it looks something like this.

JasperReport compiledReport = JasperCompileManager.compileReport(
        "sample.jrxml");
Map<String, Object> parameters = ...;
java.sql.Connection connection = dataSource.getConnection();
try {
    JasperPrint filledReport = JasperFillManager.fillReport(
            compiledReport, parameters, connection);
    JasperExportManager.exportReportToPdf(
            filledReport, "report.pdf");
} finally {
    connection.close();
}

Thanks to the facade classes, this looks simple enough. But looks can be deceiving!

Given the above code snippet (and the outlined three steps), which parts do you think takes the most amount of time and memory? (Sounds like an interview question).

If you answered (#2) filling with data, you're correct! If you answered #3, you're also correct, since #3 is proportional to #2.

IMHO, most online tutorials only show the easy parts. In the case of JR, there seems to be a lack of discussion on the more difficult and tricky parts. Here, with Mike's team, we encountered two difficulties: out of memory errors, and long running reports. What made these difficulties particularly memorable was that they only showed up during production (not during development). I hope that by sharing them, they can be avoided in the future.

Out of Memory Errors

The first challenge was reports running out of memory. During development, the test data we use to run the report would be too small when compared to real operating data. So, design for that.

In our case, all reports were run with a JRVirtualizer. This way, it will flush to disk/file when the maximum number of pages/objects in memory has been reached.

During the process, we also learned that the virtualizer needs to be cleaned-up. Otherwise, there will be several temporary files lying around. And we can only clean-up these temporary files after the report has been exported to file.

Map<String, Object> parameters = ...;
JRVirtualizer virtualizer = new JRFileVirtualizer(100);
try {
    parameters.put(JRParameter.REPORT_VIRTUALIZER, virtualizer);
    ...
    ... filledReport = JasperFillManager.fillReport(
            compiledReport, parameters, ...);
    // cannot cleanup virtualizer at this point
    JasperExportManager.exportReportToPdf(filledReport, ...);
} finally {
    virtualizer.cleanup();
}

For more information, please see Virtualizer Sample - JasperReports.

Note that JR is not always the culprit when we encountered out-of-memory errors when running reports. Sometimes, we would encounter an out-of-memory error even before JR was used. We saw how JPA can be misused to load the entire dataset for the report (Query.getResultList() and TypedQuery.getResultList()). Again, the error does not show up during development since the dataset is still small. But when the dataset is too large to fit in memory, we get the out-of-memory errors. We opted to avoid using JPA for generating reports. I guess we'll just have to wait until JPA 2.2's Query.getResultStream() becomes available. I wish JPA's Query.getResultList() returned Iterable instead. That way, it is possible to have one entity is mapped at a time, and not the entire result set.

For now, avoid loading the entire dataset. Load one record at a time. In the process, we went back to good ol' JDBC. Good thing JR uses ResultSets well.

Long Running Reports

The second challenge was long running reports. Again, this probably doesn't happen during development. At best, a report that runs for about 10 seconds is considered long. But with real operating data, it can run for about 5-10 minutes. This is especially painful when the report is being generated upon an HTTP request. If the report can start to write to the response output stream within the timeout period (usually 60 seconds or up to 5 minutes), then it has a good chance of being received by the requesting user (usually via browser). But if it takes more than 5 minutes to fill the report and another 8 minutes to export to file, then the user will just see a timed-out HTTP request, and log it as a bug. Sound familiar?

Keep in mind that reports can run for a few minutes. So, design for that.

In our case, we launch reports on a separate thread. For reports that are triggered with an HTTP request, we respond with a page that contains a link to the generated report. This avoids the time-out problem. When the user clicks on this link and the report is not yet complete, s/he will see that the report is still being generated. But when the report is completed, s/he will be able to see the generated report file.

ExecutorService executorService = ...;
... = executorService.submit(() -> {
    Map<String, Object> parameters = ...;
    try {
        ...
        ... filledReport = JasperFillManager.fillReport(
                compiledReport, parameters, ...);
        JasperExportManager.exportReportToPdf(filledReport, ...);
    } finally {
        ...
    }
});

We also had to add the ability to stop/cancel a running report. Good thing JR has code that checks for Thread.interrupted(). So, simply interrupting the thread will make it stop. Of course, you'll need to write some tests to verify (expect JRFillInterruptedException and ExportInterruptedException).

And while we were at it, we rediscovered ways to add "listeners" to the report generation (e.g. FillListener and JRExportProgressMonitor) and provide the user some progress information.

We also created utility test classes to generate large amounts of data by repeating a given piece of data over and over. This is useful to help the rest of the team develop JR applications that are designed for handling long runs and out-of-memory errors.

Further Design Considerations

Another thing to consider is the opening and closing of the resource needed when filling the report. This could be a JDBC connection, a Hibernate session, a JPA EntityManager, or a file input stream (e.g. CSV, XML). Illustrated below is a rough sketch of my design considerations.

1. Compiling
         - - - - - - - - - - - - - -\
         - - - -\                    \
2. Filling       > open-close         \
         - - - -/   resource           > swap to file
                                      /
3. Exporting                         /
         - - - - - - - - - - - - - -/

We want to isolate #2 and define decorators that would open the resource, fill the report, and close the opened resource in a finally block. The resource that is opened may depend on the <queryString> element (if present) inside the report. In some cases, where there is no <queryString> element, there is probably no need to open a resource.

<queryString language="hql">
    <![CDATA[ ... ]]>
</queryString>
...
<queryString language="csv">
    <![CDATA[ ... ]]>
</queryString>

Furthermore, we also want to combine #2 and #3 as one abstraction. This single abstraction makes it easier to decorate with enhancements, like flushing the created page objects to files, and load them back during exporting. As mentioned, this is what the JRVirtualizer does. But we'd like a design where this is transparent to the object(s) using the combined-#2-and-#3 abstraction.

Acknowledgements

That's all for now. Again, thanks to Mike and his team for sharing their experiences. Yup, he's the same guy who donates his app's earnings to charity. Also, thanks to Claire for the ideas on testing by repeating a given data again and again. The relevant pieces of code can be found on GitHub.

Tuesday, January 2, 2018

DataSource Routing with Spring @Transactional

I was inspired by Carl Papa's use of aspects with the Spring Framework to determine the DataSource to use (either read-write or read-only). So, I'm writing this post.

I must admit that I have long been familiar with Spring's AbstractRoutingDataSource. But I did not have a good idea where it can be used. Thanks to Carl and team, and one of their projects. Now, I know a good use case.

@Transactional

With Spring, read-only transactions are typically marked with annotations.

public class ... {
    @Transactional(readOnly=true)
    public void ...() {...}

    @Transactional // read-write
    public void ...() {...}
}

To take advantage of this, we use Spring's TransactionSynchronizationManager to determine if the current transaction is read-only or not.

AbstractRoutingDataSource

Here, we use Spring's AbstractRoutingDataSource to route to the read-only replica if the current transaction is read-only. Otherwise, it routes to the default which is the master.

public class ... extends AbstractRoutingDataSource {
    @Override
    protected Object determineCurrentLookupKey() {
        if (TransactionSynchronizationManager
                .isCurrentTransactionReadOnly() ...) {
            // return key to a replica
        }
        return null; // use default
    }
    ...
}

Upon using the above approach, we found out that the TransactionSynchronizationManager is one step behind because Spring will have already called DataSource.getConnection() before a synchronization is established. Thus, a LazyConnectionDataSourceProxy needs to be configured as well.

As we were discussing this, we figured if there was another way to determine if the current transaction is read-only or not (without resorting to LazyConnectionDataSourceProxy). So, we came up with an experimental approach where an aspect captures the TransactionDefinition (from the @Transactional annotation, if any) as a thread-local variable, and an AbstractRoutingDataSource that routes based on the captured information.

The relevant source code can be found on GitHub. Thanks again, Carl! BTW, Carl is also an award-winning movie director. Wow, talent definitely knows no boundaries.

Monday, April 24, 2017

Apache Spark RDD and Java Streams

A few months ago, I was fortunate enough to participate in a few PoCs (proof-of-concepts) that used Apache Spark. There, I got the chance to use resilient distributed datasets (RDDs for short), transformations, and actions.

After a few days, I realized that while Apache Spark and the JDK are very different platforms, there are similarities between RDD transformations and actions, and stream intermediate and terminal operations. I think these similarities can help beginners (like me *grin*) get started with Apache Spark.

Java StreamApache Spark RDD
Intermediate operationTransformation
Terminal operationAction

Java Streams

Let's start with streams. Java 8 was released sometime in 2014. Arguably, the most significant feature it brought is the Streams API (or simply streams).

Once a Stream is created, it provides many operations that can be grouped in two categories:

  • intermediate,
  • and terminal.

Intermediate operations return a stream from the previous one. These intermediate operations can be connected together to form a pipeline. Terminal operations, on the other hand, closes the stream pipeline, and returns a result.

Here's an example.

Stream.of(1, 2, 3)
        .peek(n -> System.out.println("Peeked at: " + n))
        .map(n -> n*n)
        .forEach(System.out::println);

When the above example is run, it generates the following output:

Peeked at: 1
1
Peeked at: 2
4
Peeked at: 3
9

Intermediate operations are lazy. The actual execution does not start until the terminal operation is encountered. The terminal operation in this case is forEach(). That's why, we do not see the following.

Peeked at: 1
Peeked at: 2
Peeked at: 3
1
4
9

Instead, what we see is that the operations: peek(), map(), and forEach(), have been joined to form a pipeline. In each pass, the static of() operation returns one element from the specified values. Then the pipeline is invoked: peek() that prints the string "Peeked at: 1", followed by map(), and terminated by forEach() that prints the number "1". Then with another pass starting with of() that returns the next element from the specified values, followed by peek(), and map(), and so on.

Executing an intermediate operation such as peek() does not actually perform any peeking, but instead creates a new stream that, when traversed, contains the same elements of the initial stream, but additionally performing the provided action.

Apache Spark RDD

Now, let's turn to Spark's RDD (resilient distributed dataset). Spark's core abstraction for working with data is the resilient distributed dataset (RDD).

An RDD is simply a distributed collection of elements. In Spark all work is expressed as either creating new RDDs, or calling operations on RDDs to compute a result. Under the hood, Spark automatically distributes the data contained in RDDs across your cluster and parallelizes the operations you perform on them.

Once created, RDDs offer two types of operations:

  • transformations,
  • and actions.

Transformations construct a new RDD from a previous one. Actions, on the other hand, compute a result based on an RDD, and either return it to the driver program or save it to an external storage system (e.g., HDFS).

Here's an example with a rough equivalent using Java Streams.

SparkConf conf = new SparkConf().setAppName(...);
JavaSparkContext sc = new JavaSparkContext(conf);

List<Integer> squares = sc.parallelize(Arrays.asList(1, 2, 3))
        .map(n -> n*n)
        .collect();

System.out.println(squares.toString());

// Rough equivalent using Java Streams
List<Integer> squares2 = Stream.of(1, 2, 3)
        .map(n -> n*n)
        .collect(Collectors.toList());

System.out.println(squares2.toString());

After setting up the Spark context, we call parallelize() which creates an RDD from the given list of elements. map() is a transformation, and collect() is an action. Transformations, like intermediate stream operations in Java, are lazily evaluated. In this example, Spark will not begin to execute the function provided in a call to map() until it sees an action. This approach might seem unusual at first, but it makes a lot of sense when dealing with huge amounts of data (big data, in other words). It allows Spark to split up the work and do them in parallel.

Word Count Example

Let's use word count as an example. Here, we have two implementations: one uses Apache Spark, and the other uses Java Streams.

Here's the Java Stream version.

public class WordCountJava {

 private static final String REGEX = "\\s+";
 
 public Map<String, Long> count(URI uri) throws IOException {
  return Files.lines(Paths.get(uri))
   .map(line -> line.split(REGEX))
   .flatMap(Arrays::stream)
   .map(word -> word.toLowerCase())
   .collect(groupingBy(
    identity(), TreeMap::new, counting()));
 }

}

Here, we read the source file line by line and transforming each line in a sequence of words (via the map() intermediate operation). Since we have a sequence of words for each line and we have many lines, we convert them to a single sequence of words using flatMap(). In the end, we group them by their identity() (i.e. the identity of a string is the string itself) and we count them.

When tested against a text file that contains the two lines:

The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog

It outputs the following map:

{brown=2, dog=2, fox=2, jumps=2, lazy=2, over=2, quick=2, the=4}

And now, here's the Spark version.

public class WordCountSpark {

 private static final String REGEX = "\\s+";
 
 public List<Tuple2<String, Long>> count(URI uri, JavaSparkContext sc) throws IOException {
  JavaRDD<String> input = sc.textFile(Paths.get(uri).toString());
  return input.flatMap(
     line -> Arrays.asList(line.split(REGEX)).iterator())
    .map(word -> word.toLowerCase())
    .mapToPair(word -> new Tuple2<String, Long>(word, 1L))
    .reduceByKey((x, y) -> (Long) x + (Long) y)
    .sortByKey()
    .collect();
 }

}

When run against the same two-line text file, it outputs the following:

[(brown,2), (dog,2), (fox,2), (jumps,2), (lazy,2), (over,2), (quick,2), (the,4)]

The initial configuration of a JavaSparkContext has been excluded for brevity. We create a JavaRDD from a text file. It's worth mentioning that this initial RDD will operate line-by-line from the text file. That's why we split each line into sequence of words and flatMap() them. Then we transform a word into a key-value tuple with a count of one (1) for incremental counting. Once we have done that, we group by words (reduceByKey()) our key-value tuples from the previous RDD and in the end we sort them in natural order.

In Closing

As shown, both implementations are similar. The Spark implementation requires more setup and configuration, and is more powerful. Learning about intermediate and terminal stream operations can help get a Java developer started with understanding Apache Spark.

Thanks to Krischelle, RB, and Juno, for letting me participate in the PoCs that used Apache Spark.

Friday, December 30, 2016

Isolating the Domain Logic

In one design patterns class, I had an interesting discussion about modelling domain logic. Specifically, it was about isolating the domain logic. An application would typically be divided into three parts:

  1. Presentation (e.g. desktop GUI, browser, web service)
  2. Domain logic
  3. Infrastructure (e.g. persistence storage, e-mail)

The class found it interesting that the dependency arrows were pointing towards the domain logic part. They asked, “Is the diagram intentionally made wrong? Shouldn’t the domain logic part be dependent on the persistence storage?” It was a great question. And I wanted to share and post the discussion and explanation here.

Often Misunderstood

Most developers would usually have this misunderstanding in mind.

Misunderstood
vs.
Proper

And this misunderstanding is largely due to the sequence of operations. It usually starts with a trigger (e.g. a user clicking a button or a link) in the presentation layer, which then calls something within the domain logic layer, which then calls something within the infrastructure layer (e.g. update a database table record).

While this is the correct sequence of operations, there’s something subtle in the way in which the domain logic layer can be implemented. This has something to do with dependency inversion.

Dependency Inversion Principle

The domain logic layer may need something from the infrastructure layer, like some form of access to retrieve from persistence storage. The usual patterns for this are: DAO and repository. I won’t explain these two patterns here. Instead, I would point out that the interface definitions are placed within the domain logic layer, and their implementations are placed in another separate layer.

Placing the (DAO and repository) interface definitions inside the domain logic layer means that it is the domain logic layer that defines it. It is the one that dictates which methods are needed, and what return types are expected. This also marks the boundaries of the domain logic.

This separation between interface and implementation may be subtle, but key. Placing just the interface definitions allows the domain logic part to be free from infrastructure details, and allows it to be unit-tested without actual implementations. The interfaces can have mock implementations during unit testing. This subtle difference makes a big difference in rapid verification of (the development team’s understanding of) business rules.

This separation is the classic dependency inversion principle in action. Domain logic (higher-level modules) should not depend on DAO and repository implementations (low-level modules). Both should depend on abstractions. The domain logic defines the abstractions, and infrastructure implementations depend on these abstractions.

Most novice teams I’ve seen, place the DAO and repository interfaces together with their infrastructure-specific implementations. For example, say we have an StudentRepository and its JPA-specific implementation StudentJpaRepository. I would usually find novice teams placing them in the same package. While this is fine, since the application will still compile successfully. But the separation is gone, and domain logic is no longer isolated.

Now that I’ve explained why and how the domain logic part does not depend on the infrastructure part, I’d like to touch on how the presentation part is accidentally entangled with the domain logic.

Separated Presentation

Another thing I often see with novice teams is how they end up entangling their domain logic with their presentation. And this results into this nasty cyclic dependency. This cyclic dependency is more logical than physical. Which makes it all the more difficult to detect and prevent.

I won’t use a rich GUI presentation example here, since Martin Fowler has already written a great piece on it. Instead, I’ll use a web-browser-based presentation as an example.

Most web-based systems would use a web framework for its presentation. These frameworks usually implement some form of MVC (model-view-controller). The model used is usually the model straight from the domain logic part. Unfortunately, most MVC frameworks require something about the model. In the Java world, most MVC frameworks require that the model follow JavaBean conventions. Specifically, it requires the model to have a public zero-arguments constructor, and getters and setters. The zero-arguments constructor and setters are used to automatically bind parameters (from HTTP POST) to the model. The getters are used in rendering the model in a view.

Because of this implied requirement by MVC frameworks used in the presentation, developers would add a public zero-arguments constructor, getter and setters, to all their domain entities. And they would justify this as being required. Unfortunately, this gets the in the way of implementing domain logic. It gets entangled with the presentation. And worse, I’ve seen domain entities being polluted with code that emits HTML-encoded strings (e.g. HTML code with less-than and greater-than signs encoded) and XML, just because of presentation.

If it is all right to have your domain entity implemented as a JavaBean, then it would be fine to have it used directly in your presentation. But if the domain logic gets a bit more complicated, and requires the domain entity to lose its JavaBean-ness (e.g. no more public zero-arguments constructor, no more setters), then it would be advisable for the domain logic part to implement domain logic, and have the presentation part adapt by creating another JavaBean object to satisfy its MVC needs.

An example I use often is a UserAccount that is used to authenticate a user. In most cases, when a user wishes to change the password, the old password is also needed. This helps prevent unauthorized changing of the password. This is clearly shown in the code below.

public class UserAccount {
  ...
  public void changePassword(
      String oldPassword, String newPassword) {…}
}

But this does not follow JavaBean conventions. And if the MVC presentation framework would not work well with the changePassword method, a naive approach would be to remove the erring method and add a setPassword method (shown below). This weakens the isolation of the domain logic, and causes the rest of the team to implement it all over the place.

public class UserAccount {
  ...
  public void setPassword(String password) {…}
}

It’s important for developers to understand that the presentation depends on the domain logic. And not the other way around. If the presentation has needs (e.g. JavaBean convention), then it should not have the domain logic comply with that. Instead, the presentation should create additional classes (e.g. JavaBeans) that have knowledge of the corresponding domain entities. But unfortunately, I still see a lot of teams forcing their domain entities to look like JavaBeans just because of presentation, or worse, having domain entities create JavaBeans (e.g. DTOs) for presentation purposes.

Arrangement Tips

Here’s a tip in arranging your application. Keep your domain entities and repositories in one package. Keep your repository and other infrastructure implementations in a separate package. Keep your presentation-related classes in its own package. Be mindful of which package depends on which package. The package that contains the domain logic is preferrably at the center of it all. Everything else depends on it.

When using Java, the packages would look something like this:

  • com.acme.myapp.context1.domain.model
    • Keep your domain entities, value objects, and repositories (interface definitions only) here
  • com.acme.myapp.context1.infrastructure.persistence.jpa
    • Place your JPA-based repository and other JPA persistence-related implementations here
  • com.acme.myapp.context1.infrastructure.persistence.jdbc
    • Place your JDBC-based repository and other JDBC persistence-related implementations here
  • com.acme.myapp.context1.presentation.web
    • Place your web/MVC presentation components here. If the domain entities needed for presentation do not comply with MVC framework requirements, create additional classes here. These additional classes will adapt the domain entities for presentation-purposes, and still keep the domain entities separated from presentation.

Note that I’ve used context1, since there could be several contexts (or sub-systems) in a given application (or system). I’ll discuss about having multiple contexts and having multiple models in a future post.

That’s all for now. I hope this short explanation can shed some light to those who wonder why their code is arranged and split in a certain way.

Thanks to Juno Aliento for helping me with the class during this interesting discussion.

Happy holidays!

Thursday, October 27, 2016

Architectural Layers and Modeling Domain Logic

As I was discussing the PoEAA patterns used to model domain logic (i.e. transaction script, table module, domain model), I noticed that people get the impression (albeit wrong impression) that the domain model pattern is best. So, they set out to apply it on everything.

Not Worthy of Domain Model Pattern

Let's get real. The majority of sub-systems are CRUD-based. Only a certain portion of the system requires the domain model implementation pattern. Or, put it in another way, there are parts of the application that just needs forms over data, and some validation logic (e.g. required/mandatory fields, min/max values on numbers, min/max length on text). For these, the domain model is not worth the effort.

For these, perhaps an anemic domain model would fit nicely.

Anemic Domain Model Isn't As Bad As It Sounds

The anemic domain model isn't as bad as it sounds. There, I said it (at least here in my blog post).

But how does it look like?

package com.acme.bc.domain.model;
...
@Entity
class Person {
 @Id ... private Long id;
 private String firstName;
 private String lastName;
 // ...
 // getters and setters
}
...
interface PersonRepository /* extends CrudRepository<Person, Long> */ {
 // CRUD methods (e.g. find, find/pagination, update, delete)
}
package com.acme.bc.infrastructure.persistence;
...
class PersonRepositoryJpa implements PersonRepository {
 ...
}

In the presentation layer, the controllers can have access to the repository. The repository does its job of abstracting persistence details.

package com.acme.bc.interfaces.web;

@Controller
class PersonsController {
 private PersonRepository personRepository;
 public PersonsController(PersonRepository personRepository) {...}
 // ...
}

In this case, having the Person class exposed to the presentation layer is perfectly all right. The presentation layer can use it directly, since it has a public zero-arguments constructor, getters and setters, which are most likely needed by the view.

And there you have it. A simple CRUD-based application.

Do you still need a service layer? No. Do you still need DTO (data transfer objects)? No. In this simple case of CRUD, you don't need additional services or DTOs.

Yes, the Person looks like a domain entity. But it does not contain logic, and is simply used to transfer data. So, it's really just a DTO. But this is all right since it does the job of holding the data stored-to and retrieved-from persistence.

Now, if the business logic starts to get more complicated, some entities in the initially anemic domain model can become richer with behavior. And if so, those entities can merit a domain model pattern.

Alternative to Anemic Domain Model

As an alternative to the anemic domain model (discussed above), the classes can be moved out of the domain logic layer and in to the presentation layer. Instead of naming it PersonRepository, it is now named PersonDao.

package com.acme.bc.interfaces.web;

@Entity
class Person {...}

@Controller
class PersonsController {
 private PersonDao personDao;
 public PersonsController(PersonDao personDao) {...}
 // ...
}

interface PersonDao /* extends CrudRepository<Person, Long> */ {
 // CRUD methods (e.g. find, find/pagination, update, delete)
}
package com.acme.bc.infrastructure.persistence;

class PersonDaoJpa implements PersonDao {
 ...
}

Too Much Layering

I think that it would be an overkill if you have to go through a mandatory application service that does not add value.

package com.acme.bc.interfaces.web;
...
@Controller
class PersonsController {
 private PersonService personService;
 public PersonsController(PersonService personService) {...}
 // ...
}
package com.acme.bc.application;
...
@Service
class PersonService {
 private PersonRepository personRepository;
 public PersonService(PersonRepository personRepository) {...}
 // expose repository CRUD methods and pass to repository
 // no value add
}

Application Services for Transactions

So, when would application services be appropriate? The application services are responsible for driving workflow and coordinating transaction management (e.g. by use of the declarative transaction management support in Spring).

If you find the simple CRUD application needing to start transactions in the presentation-layer controller, then it might be a good sign to move them into an application service. This usually happens when the controller needs to update more than one entity that does not have a single root. The usual example here is transferring amounts between bank accounts. A transaction is needed to ensure that debit and credit both succeed, or both fail.

package sample.domain.model;
...
@Entity
class Account {...}
...
interface AccountRepository {...}
package sample.interfaces.web;
...
@Controller
class AccountsController {
 private AccountRepository accountRepository;
 ...
 @Transactional
 public ... transfer(...) {...}
}

If you see this, then it might be a good idea to move this (from the presentation layer) to an application-layer service.

package sample.interfaces.web;
...
@Controller
class AccountsController {
 private AccountRepository accountRepository;
 private TransferService transferService;
 ...
 public ... transfer(...) {...}
}
package sample.application;
...
@Service
@Transactional
class TransferService {
 private AccountRepository accountRepository;
 ...
 public ... transfer(...) {...}
}
package sample.domain.model;
...
@Entity
class Account {...}
...
interface AccountRepository {...}

Domain Model Pattern (only) for Complex Logic

I'll use the double-entry accounting as an example. But I'm sure there are more complex logic that's better suited.

Let's say we model journal entries and accounts as domain entities. The account contains a balance (a monetary amount). But this amount is not something that one would simply set. A journal entry needs to be created. When the journal entry is posted, it will affect the specified accounts. The account will then update its balance.

package ….accounting.domain.model;
...
/** Immutable */
@Entity
class JournalEntry {
 // zero-sum items
 @ElementCollection
 private Collection<JournalEntryItem> items;
 ...
}
...
/** A value object */
@Embeddable
class JournalEntryItem {...}
...
interface JournalEntryRepository {...}
...
@Entity
class Account {...}
...
interface AccountRepository {...}
...
@Entity
class AccountTransaction {...}
...
interface AccountTransactionRepository {...}

Now, in this case, a naive implementation would have a presentation-layer controller create a journal entry object, and use a repository to save it. And at some point in time (or if auto-posting is used), the corresponding account transactions are created, with account balances updated. All this needs to be rolled into a transaction (i.e. all-or-nothing).

Again, this transaction is ideally moved to an application service.

package ….accounting.application;

@Service
@Transactional
class PostingService {...}

If there's a need to allow the user to browse through journal entries and account transactions, the presentation-layer controller can directly use the corresponding repositories. If the domain entities are not suitable for the view technology (e.g. it doesn't follow JavaBean naming conventions), then the presentation-layer can define DTOs that are suitable for the view. Be careful! Don't change the domain entity just to suit the needs of the presentation-layer.

package ….interfaces.web;

@Controller
class AccountsController {
 private AccountRepository accountRepository;
 private AccountTransactionRepository accountTransactionRepository;
 private PostingService postingService;
  ...
}

In Closing...

So, there you have it. Hopefully, this post can shed some light on when (and when not) to use domain model pattern.