Not Quite Callbacks

I’ve been working on application architectures based on Uncle Bob’s Ruby Midwest talk, following the hexagonal architectural pattern. I posted an article a couple months ago showing a way that works fairly well in Rails, and some accompanying Rails example code. But there was one thing I wasn’t quite happy with.

The problem is that we used callbacks (actually, a publish/subscribe mechanism) in a situation where they don’t seem to quite fit:

  def show
    interactor.on(:display) { |order| render order }
    interactor.on(:not_found) { |order_id| render status: 404 }
    interactor.get(params[:id])
  end

What we really want is to respond in different ways, depending on the result of the call to interactor.get(). There’s no good reason to define the responses before the call. It makes a lot more sense to define the responses after the call, because they’ll happen after the call. I’d much prefer that the code be written in the order that it will be run.

I discussed this problem with my friend and colleague, Amos King. We came up with a better solution, which puts things back in the right order:

  def show
    interactor.get(params[:id]) do |on|
      on.display { |order| render order }
      on.not_found { |order_id| render status: 404 }
    end
  end

He even wrote a small library to do this, which he called Riposte. I’m not sure what to call this pattern, but it seems to work pretty well in this situation. I suppose that they’re still technically callbacks, because they’re passed in in the block that’s passed in to the call to interactor.get(). But due to the magic of Ruby blocks, we get to put them in the order they should be.

Riposte also gives you the option of using the response object directly, instead of passing a block:

  def show
    on = interactor.get(params[:id])
    on.display { |order| render order }
    on.not_found { |order_id| render status: 404 }
  end

This shows that it’s just returning an object, with the twist that the response object has methods that take blocks. The nested blocks variant is really the same thing, except that it’s yielding to the response object instead of returning it.

I’ve decide that is the pattern I’d like to use for interactions and their callers within Ruby hexagonal architecture.

Architectural Thoughts

I’ve started working on my own framework in Ruby in the past couple days. It’s built upon my recent work at understanding Uncle Bob’s Ruby Midwest 2011 talk, and his article on Clean Architecture, and the resulting hexagonal architecture (AKA ports and adapters).

Somehow my research in that vein led me to Gary Bernhardt’s Boundaries talk. I’ve read a lot about the talk, and knew about the idea of “functional core / imperative shell”. And I’ve worked with a lot of similar ideas lately. But I believe this is the first time that I actually watched the whole video.

Even after having read a lot about similar ideas, it was pretty mind-expanding. Gary’s really good at presenting these kinds of ideas in a simple way.

OOP as usually taught includes encapsulation of data together with behavior, with mutable objects. Functional programming separates data and behavior, with mostly immutable data. From experience, encapsulating data and behavior together seems helpful. But experience also shows that immutability is useful. So it would be good to have both of those together. This is something I’ve been thinking for a few years — how best do we get both?

Gary calls the combination “FauxO”. Logic and data are still combined, but there’s no mutation. Anywhere OOP would normally have mutation would just generate a new object. There’s no language restriction involved in enforcing immutability — just discipline.

But without mutability, it’s hard to do IO and maintain state. So Gary’s solution is to encapsulate as much as possible into an immutable (functional or FauxO) core, and around that, use an imperative (traditional OOP) shell. The functional core contains the bulk of the logic, and the imperative shell is a glue layer that handles the real world, including disk, network, and other I/O.

The result of this is that the shell has fewer paths, but more dependencies. The core contains no dependencies, but encapsulates the different logic paths. So we’re encapsulating dependencies on one side, and business logic on the other side. Or put another way, the way to figure out the separation is by doing as much as you can without mutation, and then encapsulating the mutation separately.

I love how this naturally breaks things up, so that the core is all testable with unit tests, and the imperative shell is tested with integration tests. And since the shell has few or no logic paths, you get the testing pyramid, with more unit tests and fewer integration tests. The whole thing ends up being quite beautiful. Tests end up being very fast without any extra effort — not even stubbing or mocking. This tells us that things have been decomposed very well — an elegant design.

Gary makes the case that immutable objects can be treated as values, and passed across boundaries. Even process boundaries. This is something I’ve noticed as I’ve been working on my own Uncle Bob style hexagonal framework, but nobody in that camp ever mentioned that — they prefer DTOs or something more like hashes. I’m completely against hashes, because of the “stringly-typed” problem. And I don’t see much advantage in a DTO if I’ve got an immutable object; I’d be basically copying the object to an almost identical object. And I’d be losing any laziness possible for derived values within the original immutable object.

It’s striking to me how Gary’s image of an imperative shell around a functional core, plus Net, Disk, and State outside of the shell mirror’s Uncle Bob’s concentric circles. Uncle Bob has entities in the middle, surrounded by use cases, surrounded by Web, DB, and UI.

Another advantage that Gary shows is that breaking things up this way allows easy concurrency. In his example, he shows using the actor model — either just using threads and queues, or an actor library (or language feature).

After several years of thinking about the architectural issues seen in most large Rails apps, I’m starting to come to an understanding of how to combine all these ideas and come up with an architecture that will work better.

 

Hexagonal Rails Controllers

I’ve had a long love-hate relationship with Rails. I love the MVC framework and how it’s improved our speed of writing web apps. But I’ve never really been completely happy with it. I don’t generally agree with most of its opinions. I prefer models that follow the Data Mapper pattern, not the Active Record pattern. This includes separating the persistence layer from the models’ business logic. I prefer Slim or HAML to ERB. I prefer RSpec to Test::Unit or MiniTest. When Merb hit the scene, I was ready to make the jump, until Merb merged with Rails.

So inspired by PJ Hagerty’s recent article on alternative Ruby web frameworks, I started thinking about how I’d write a replacement for Rails. I’d definitely keep the basic MVC framework. But I’d also want to implement a more hexagonal architecture.

I started sketching out what this would look like, but I ended up starting with a Rails controller and finding the simplest way to make it hexagonal. I really don’t like callbacks, because they make tracing program execution difficult. But I didn’t see any other alternative. I found a simple pub/sub Ruby library called Wisper. It literally has only publish, subscribe, and on methods. (You use on to register single callbacks via blocks, and subscribe to register an object with method names corresponding to the callback names.)

The trick was figuring out how to break the controller into 2 pieces. What finally helped me was to find the single responsibilities of the 2 pieces. The Rails controller would remain in charge of managing the web interface, but would delegate to the other piece to handle any application-specific business logic. I decided to re-watch Uncle Bob Martin’s “Architecture The Lost Years” talk, which was the first time I was introduced to the ideas of Hexagonal Architecture. (He doesn’t name the architecture in the talk, but later calls it Clean Architecture.) He does a decent job of explaining how to break these 2 pieces apart. He used the term “interactor” in that talk, so I decided to go with that. He said that Jacobsen calls it a Control Object in Object Oriented Software Engineering, but that’s too close to Rails’s “controller”.

So here’s an example of what I ended up with:

class OrderController < ApplicationController
  def index
    interactor.on(:display) { |orders| render orders }
    interactor.list
  end

  def show
    interactor.on(:display) { |order| render order }
    interactor.on(:not_found) { |order_id| render status: 404 }
    interactor.get(params[:id])
  end

private

  def interactor
    @interactor ||= OrderInteractor.new
  end
end
require "wisper"
require "order"

class OrderInteractor
  include Wisper.publisher

  def list
    orders = Order.all
    publish(:display, orders)
  end

  def get(id)
    order = Order.find(id)
    publish(:display, order)
  rescue ActiveRecord::RecordNotFound
    publish(:not_found, id)
  end
end

I do have a few problems with this solution though. I’m not a fan of the name “interactor” for the business logic. I thought about calling it OrderOperator, or maybe OrderOperations, because it’s really a collection of operations. Perhaps it would be better to separate each operation into a separate class. Trailblazer does it that way. And for more complicated business logic, I would do that too, using the Method Object pattern. But like a Rails controller, there’s a lot in common among all the operations. I feel like a separate class for each operation for each would create too many coupled classes.

I’m also uncomfortable with the fact that the controller is delegating almost everything to the interactor. I guess this is OK, but it feels like there’s too little left when every line starts with interactor. I suppose extracting things some more would help mitigate this concern I’ll likely write a small gem to perform that extraction. I expect that that will allow a typical controller to be written in only a few lines. And maybe the same for the interactor side.

With the business logic extracted out of the controller, it was really easy for me to write a command-line version of the app. As Uncle Bob says, “the web is not particularly important to your application.”

I’ve put the code for this example on GitHub: https://github.com/boochtek/hexagonal-rails. I’ll likely experiment with it some more over the next few weeks and months.

Ruby Pattern: Parameterized Module Inclusion

I’ve run across a pattern in Ruby lately that I really like. It solves some problems that I’ve struggled with for several years. Let me start with the problems.

Let’s say you want to include an ORM in a model class, and want to tell it what database table to use. Typically, you’d do this:

class User
  include MyORM::Model
  table 'people'
end

But that table call is more like an option to the module inclusion than anything else. So what we’d really like is something like this:

class User
  include MyORM::Model, table: 'people'
end

But that’s not valid Ruby; include doesn’t let you pass anything other than a module.

So when I was learning about Virtus, I noticed that its example of how to include it is a bit different than the standard Ruby idiomatic include:

class User
  include Virtus.model
end

At first glance, it reads like the first example. But on closer inspection and consideration, it’s quite a bit different. Where MyORM::Model is a constant that refers to a module, Virtus.model is a method call. So there’s a method named model in the Virtus module. That method returns another module — which is exactly what’s needed in order to include it into our model class.

The easiest way to implement Virtus.model would be this:

module Virtus
  def model
    ::Virtus::Model
  end
end

module Virtus::Model
  # ...
end

If Virtus.model doesn’t need to take any arguments, that’s perfectly fine. In fact, I’ve started to use this implementation of the pattern for modules that don’t need parameters.

Because Virtus.model is a method, we can also call it with options:

class User
  include Virtus.model(constructor: false, mass_assignment: false)
end

We could even pass a block. But how do we process those options? There are a few different ways. However we do it, we have to be sure to return a module. And we can create modules in a few different ways.

Virtus uses the builder pattern. It takes the parameters passed in and builds a module dynamically. By that, I mean that it calls Module.new and then adds methods to that module. It does this by mixing in other modules, but it could do it by dynamically defining methods as well.

I’ve never seen this pattern in any other language. It’s obviously only possible because we can dynamically create modules.

The use of this idiom seems to be catching on a bit in the Ruby community. I’ve started using it myself, and will be adding it to my Includable::ActiveRecord gem soon.

 

 

Includable ActiveRecord

I created a Ruby gem recently, called includable-activerecord. It’s pretty small, but I thought I might explain why I created it, and discuss its implementation.

Classical Inheritance

When you use ActiveRecord, you normally include it in your model like this:

class User < ActiveRecord::Base
  # ...
end

Your User class is inheriting from the ActiveRecord::Base class. This is class-based inheritance, also called “classical” inheritance. (That’s “classical” as in “class”, not as a synonym for “traditional”.) Class-based inheritance represents an “is-a” relationship. So we’re saying that a user is an ActiveRecord base. Another way to say this is that User is a subclass of ActiveRecord::Base.

There are a few problems with this. First, what is a “base”? The name was chosen because it’s a base class. But just like we don’t give factory classes names like UserFactory (at least not in Ruby), we shouldn’t name base classes Base.

I suppose that we’re trying to say that this is an ActiveRecord model. That sounds fine at first glance — this is our model for users. But what if we also want to say that a user is a person? Ruby doesn’t allow inheriting from multiple classes. Now we have to choose whether to inherit from ActiveRecord::Base or Person. Person makes more sense, because it fills the “is-a” rule better. Class inheritance is intended for a hierarchical “is-a” relationship, such as “a user is a person”, or “a circle is a shape”. But since ActiveRecord::Base is a base class, we have to use it as our base class.

We could work around this problem by subclassing Person from ActiveRecord::Base and then subclassing User from Person. That’s fine if Person is also a model that we store in the database. But if that’s not the case, then we have a problem.

Mixins

Ruby provides another way of implementing inheritance — mixins. We often don’t think of this as an inheritance model, but it really is. When we include a module, that module gets added to the class’s ancestor chain. We can mix in as many modules as we want.

Mixins indicate more of an “acts like” relationship than an “is-a” relationship. It’s for shared behavior between classes that don’t have a hierarchical relationship. For example, when we mix in the Enumerable module, we’re saying that we want our class to act like other classes that include Enumerable. That sounds more like what we want ActiveRecord to be. We want our user model to behave like other ActiveRecord models, in the way that they can persist to a database.

But ActiveRecord doesn’t support that. Almost all the other Ruby ORMs do; as we’ve shown above, this is for good reasons.

Implementation

So I decided to see if I could implement the equivalent of the ActiveRecord::Base class as a module that could be mixed into model classes. I decided to call my mixin module ActiveRecord::Model, because classes that mix it in will behave as ActiveRecord models.

It turns out that ActiveRecord::Base is a pretty complex class. It includes and extends a lot of other modules. Luckily, as of ActiveRecord 4.0, that’s all the code it includes.

The module only defines a single class method, included. This is one of Ruby’s many hook methods. It gets called when the module in question gets included in another module, and receives that other model as its argument. All we need to have this method do is to include everything that ActiveRecord::Base includes, and extend everything that ActiveRecord::Base extends. Ruby provides a method that’s defined on all classes, called included_modules, which we can use to get the list of everything that’s included in ActiveRecord::Base. Unfortunately, there’s no equivalent list of extended_modules. But a quick search on Stack Overflow found an implementation of extended_modules that we could use.

So with a bit of magic (i.e. hooks and meta-programming), we can get the lists of constituent modules from the ActiveRecord::Base class, and include them in our ActiveRecord::Model module.

So with all that, we can now include the includable-activerecord gem and mix it in, with all the advantages that provides:

class User
  include ActiveRecord::Model
  # ...
end

It was exciting to be able to make this work. Since I wrote it as a proof of concept, I haven’t written any tests yet. But it seems to be working just fine. The main thing I really need to look into is making sure that plugins that extend ActiveRecord::Base from their own code will still work. I’m pretty sure this will work out of the box, because the ActiveRecord::Model.included doesn’t run until the model class is loaded, and that happens after those plugins have initialized themselves.

My Thoughts on Python vs. Ruby

I’ve been using Python at work for the past few months.  I learned Python back in the early 2000s, but never used it for any large projects.  I learned Ruby in late 2005, and it quickly became my language of choice for most cases.

So while I still prefer Ruby, and will likely use Ruby more in the future than Python, I wanted to assess the strengths and weaknesses of Python in relation to Ruby.  Perhaps some of the lessons could be applied when writing Ruby, and it could help to decide when to use each.  Also, I’m interested in programming language design, and wanted to document pros and cons in that light.

Where Python Sucks (As Compared to Ruby)

  • Have to explicitly include self in EVERY method declaration.
    • Including @classmethod declarations (although people usually use the name cls instead of self there).
    • Except @staticmethod declarations.
  • Have to use self everywhere to reference an object’s own attributes.
  • Inconsistency of things like the len() function, when everything else is a method.
  • Inconsistency of having some built-in classes with lower-case names.
    • And they’re not whole words, so you have to memorize them: list, dict, str, int.
    • Even more bizarre is dict vs. OrderedDict.
  • I’m not a fan of True and False being uppercase.
    • I’m a bit less concerned about None, for some reason.
    • If they’re going to be uppercase, they might as well be all caps, in my opinion.
  • Inconsistency of camelCase versus snake_case in some modules.
  • Claims to prefer explicit over implicit, yet does not use new to create an object.
    • It is nicely concise, but makes it look too much like a regular function call.
  • The implementation of lambdas is too limited.
    • Makes using map function not very useful.
  • Class methods are less than ideal to implement.
    • Probably better to use functions within a module instead, in many cases.
  • Lack of good functional programming tools makes it harder to manipulate lists.
    • Have to often resort to creating an initial list and adding to it in a for loop.
  • I don’t understand why r’regex_string’ doesn’t just create an actual regular expression object.
  • I miss Ruby’s method-call-or-property-getter syntax.
    • Nice that I can get it by adding @property to method definitions, but that’s a bit messy.
  • I don’t understand why lists don’t have a join() method; it seems backwards to call join on the string used to connect the list elements.
  • I miss unless; seems like with all the keywords, Python would have added that.
  • I really miss ||= to memoize.
    • The distinction between unset variables and variables set to None makes it hard.
    • I also miss +=.
  • I really miss statement modifiers (if or unless at the end of the statement/expression instead of the beginning).
  • I miss being able to assign to a compound statement (x = if True: 1; else: 2).
    • While Python does allow this simple case, it does not allow more complex cases.
  • The way modules, files, and classes work, I either have a lot of classes in one file, or have to come up with more module names in order to split classes into different files.
  • The preference for bare functions within modules over class methods often leads to functions outside of classes, when they’d make more sense more closely associated with the class.
  • I don’t quite understand the necessity for pass.
    • Seems like allowing 0 lines of indented code would suffice instead.
  • I miss implicit return.
    • Explicit return looks a bit nicer, but is less concise, and I’ve gotten out of the habit.
  • Mixins turn out to be more useful than multiple inheritance.
  • I miss string interpolation.
  • I really miss Array#first and Array#last.
  • I really don’t like that 0 is falsey.
    • I could live with empty lists and dicts being false, but not 0.0 and 0.
  • Comparing a string to a regular expression seems harder than it should be.
  • Converting to a Boolean is not as easy a Ruby’s !! syntax.
    • OK, bool(expression) isn’t so bad, I guess.

Where Python Rocks

  • Indentation removes the need for end everywhere.
  • Import statements are nice.
    • Can import everything, or just a few things.
  • Annotations are nice for aspect-oriented modification of methods.
    • Does it make it hard to debug though, like alias_method_chain in Ruby?
    • It’s kind of tricky to write them though, due to doubly-nested function definitions.
  • List comprehensions are more powerful than map.
    • But map can be more concise for the most common cases.
  • Can add arbitrary attributes to any object, class, or module.
  • The object creation syntax is nicely concise.
  • Sometimes the ability to add a bare function within a module is nice.
  • Keyword arguments are nicely done.
    • Ruby’s emulation via hashes without braces is OK, but the corner cases are problematic.
  • Docstrings as part of the language is nice.
  • The new with keyword looks like a halfway-decent replacement for blocks.
    • Seems like a lot more work that using blocks and yield though.
  • Python 3 drops support for octal literals starting with a 0.
    • Still allows it via a 0o prefix.
  • No support for all the crazy Perl-inspired globals.
  • The names list and dictionary are better than array and hash.
    • The Ruby names are more about the implementation, especially hash.
    • I’d much prefer the name map over hash, or even dictionary.
    • The name list is only slightly better than array.
  • List and string slicing is quite nice.
    • I do wish that the syntax was “..” instead of “:” though.
    • Slice assignment is even cooler.
  • I prefer the Python dict syntax over the Ruby hash syntax (“:” vs. “=>”).
    • The Ruby 1.9 symbol hash syntax is an improvement, but not quite as good.
  • Checking for string containment is nice: if substring in string.

Where Ruby Rocks

  • Consistency.
  • Blocks.
  • Excellent functional tools to deal with Enumerable.
  • Meta-programming.
  • Optional parentheses.
  • Modules and classes are also objects.

 

Debugging Pattern – Grenade

I’ve been more interested in programming patterns lately, partly due to the Ruby community’s recent interest — especially the Ruby Rogue podcast’s love affair with “Smalltalk Best Practice Patterns”.

I consider most of the patterns in “Smalltalk Best Practice Patterns” to be patterns “in the small” — things that are typically employed on a single line or method. The Gang Of Four patterns are more medium sized, dealing with methods and classes. The PEAA book covers architectural-scale patterns. I suppose “The Pragmatic Programmer” and similar books could (should!) be considered to be very general patterns, mostly about the practice of programming.

One type of pattern that I have not seen discussed much is debugging patterns. I’m not exactly sure why that is; probably just our general concentration on designing the program code itself. There are definitely testing patterns. But I can’t think of anything that I’ve seen that covers debugging patterns. A quick search on the Internet doesn’t turn up too much.

Anyway, a co-worker (Helena) and I were debugging recently, and were having trouble figuring out where a certain method on a certain object was being called. We came up with a really cool solution. We immediately called it a grenade, because its entire purpose is to blow up (raise an exception) and display a backtrace to show us the caller of the method that we’re looking for.

Here’s our implementation in Ruby:


module Grenade
  def method_under_investigation(*)
    raise "method_under_investigation called"
  end
end

class Xyz
  ...
  def xyz
    object_under_investigation.extend(Grenade)
  end
  ...
end

I’m sure we could wrap that in some more semantic sugar, even going as far as making it look something like this:


class Xyz
  ...
  def xyz
    object_under_investigation.blow_up_when_method_called(:method_under_investigation)
  end
  ...
end

I’m not sure that would really be worth it though, unless we were to add it to some sort of library.

So that’s our contribution to the (small) list of debugging patterns.