Data Transformation

Synopsis

Inserting to the Database

Procedural Example

Enter the Command Pattern

Synopsis

Data transformation is an integral operation of most useful programs. Often times we want to convert data of one type to another or from one format to another. Data transformations typically encompass specific operations; the danger is that when implemented incorrectly our applications become static and inflexible.

In this article I will present my initial thoughts on a flexible solution to data transformation; more specifically I will be using the command design pattern. Examples will be provided in PHP5.

This is also available in MS Word format.

Inserting to the Database

Consider how we might transform data that originates from a web form before we use it within a MySQL query.

This is a simple transformation that has three steps:

Undo magic quotes
Call mysql_real_escape_string()
Wrap the value in single quotes for use in a query

Procedural Example

Procedurally we might program the above diagram like so.

// Check if user is valid
$username = undo_magic_quotes($username);
$username = mysql_real_escape_string($username);
$username = "'" . $username . "'";
$password = undo_magic_quotes($password);
$password = mysql_real_escape_string($password);
$password = "'" . $password . "'";
$sql = "
  SELECT COUNT(*) AS `valid_user` FROM `users`
  WHERE `username`={$username} AND `password`={$password}
";

There are a few problems with this code. First notice how we have to repeat the same steps for different fields; imagine there were ten fields instead of two, that's a lot of typing. The second problem is our code is not flexible; imagine that we had to introduce a transformation into the process.

Now we could clean this up and improve maintainability by introducing a function.

function db_prepper($data){
  $data = undo_magic_quotes($data);
  $data = mysql_real_escape_string($data);
  $data = "'" . $data . "'";
  return $data;
}

$username = db_prepper($username);
$password = db_prepper($password);

I won't lie; that's not terrible. It still remains inflexible however. Currently our code supports MySQL as the database, but what if we want to support another database engine.

We just add a new db_prepper_new_engine() function, right? But then we have to remember which engine the db_prepper() was for. Eventually we have several db_prepper_*() functions floating around out there.

Now what happens when we decide that all data coming into the database should be converted to upper case? We have to remember to modify all of our db_prepper_*() functions to add that transformation. If your application is only supporting a single database engine, then this isn't terrible. But our goal here is flexibility.

Enter the Command Pattern

I'll be honest; I'm slightly new to design patterns. This may not be the command pattern exactly, but I think it's pretty close. Essentially the command pattern wraps up various requests behind a single interface.

Here is a small code example of the command pattern in action.

$command1 = new CommandToDoSomething();
$command2 = new CommandToDoSomethingElse();

$command1->Execute();
$command2->Execute();

I'm concerned with data transformation, so instead of calling my objects Commands I will be calling them DataTransformers. Instead of an Execute() method they will have a Transform() method.

At the base of this hierarchy is the DataTransformer class. It defines a single method Transform() that accepts a piece of data to transform and returns that piece of data with the applied transformation.

From there we can create all sorts of single-task data transformers. It is critical, at least in my opinion, that the actual data transformers be as single-task oriented as possible.

In code, we might invoke transformations like so.

$transformer = new MagicQuoteRemover();
$username = $transformer->Transform($username);
$password = $transformer->Transform($password);
$transformer = new MySQLEscape();
$username = $transformer->Transform($username);
$password = $transformer->Transform($password);
$transformer = new SingleQuoteEncloser();
$username = $transformer->Transform($username);
$password = $transformer->Transform($password);

At first glance this doesn't appear much better than the procedural approach. Let's fix that with macros.

Using Macros

What we really want to do in our client code is this.

$transformer = new MySQLSafeMacro();
$username = $transformer->Transform($username);
$password = $transformer->Transform($password);

Simply put a macro combines a sequence of commands into a single command.

The DataTransformerMacro object extends the DataTransformer object. It adds a Transformers property which is a list of DataTransformers. It also adds a protected AddTransformer() method which derived classes can use to add to the list of transformers. Finally it implements the Transform() method defined in DataTransformer by looping over all of the DataTransformers in its internal list and applying the transformations.

The key point here is that the macros expose the same interface as a DataTransformer. This means there is no difference in creating and invoking a transformer and that macros can even contain macros themselves.

Now that is flexibility.

Adding Undo

One of the key points of the command pattern is that commands should be able to undo themselves. I'm not going to show any code implementing undo, but it's not hard to imagine how it might be accomplished.

First we need to add an empty Undo() method to our DataTransformer base class.

Then we need to define an Undo() method to each of the DataTransformer derived classes that are not macros. Not all data transformations can be undone however, so we rely on the empty method in the DataTransformer class to handle the ones that can't.

Finally we add a final Undo() method to our DataTransformerMacro class. This method will traverse the list of DataTransformers in reverse order and call Undo() on each one.

A Full Code Example

Programming is useless without examples that can be played with, so here is a full, albeit simple, example. Undo functionality is not implemented here.

<?php
  /**
   * Defines base class for all DataTransformer related objects.
   */
  abstract class DataTransformer {
    abstract protected function Transform($data);
  }

  /**
   * Keeps a list of transformations in order to create command macros, i.e.
   * transformations that perform multiple steps on data.
   */
  class DataTransformerMacro extends DataTransformer {
    private $Transformers = Array();

    final protected function AddTransformer(DataTransformer $Transformer){
      $this->Transformers[] = $Transformer;
    }

    final public function Transform($data){
      if(!empty($this->Transformers)){
        foreach($this->Transformers as $Transformer){
          $data = $Transformer->Transform($data);
        }
      }
      return $data;
    }
  }

  /**
   * Special transformer that combines the two!
   */
  class SpecialMacro_DataTransformerMacro extends DataTransformerMacro {
    function __construct(){
      $this->AddTransformer(
        new ToUppercase_DataTransformer()
      );
      $this->AddTransformer(
        new MagicQuoteRemover_DataTransformer()
      );
    }
  }

  /**
   * An uppercase data transformer
   */
  class ToUppercase_DataTransformer extends DataTransformer {
    public function Transform($data){
      return strtoupper($data);
    }
  }

  /**
   * Magic Quote removing data transformer
   */
  class MagicQuoteRemover_DataTransformer extends DataTransformer {
    static private $MagicQuotesOn = null;

    public function Transform($data){
      if(self::$MagicQuotesOn === null){
        self::$MagicQuotesOn = get_magic_quotes_gpc();
      }
      if(self::$MagicQuotesOn){
        $data = stripslashes($data);
      }
      return $data;
    }
  }

  $transformer = new SpecialMacro_DataTransformerMacro();
  echo "That\'s Bob\'s! --> " . $transformer->Transform("That\'s Bob\'s!");
?>

Final Thoughts

Time is precious so I threw this together in a hurry. I will expand on this article as needed to answer any questions that may come up.

rbredlau