The definitive introduction to D8 and Composer

Note: This blog post was written in 2015 to cover Drupal 8.0, and is no longer up to date. Composer Manager is deprecated and should not be used. Here’s an updated version.

We’re making a new site based on Drupal 8. This site needs to display addresses, and some quick research leads us to the Address module.
However, after downloading and trying to enable Address, we get the following message: Address requires the commerceguys/addressing library.

In the old days a requirement like this would be satisfied by manually downloading and extracting the library into sites/all/libraries/addressing. Nowadays that’s no longer possible, PHP libraries must be installed using Composer.

What’s Composer?

Composer is a tool for managing dependencies on the project level, a project being your site or web application.

Let’s say our (non-Drupal) site needs the commerceguys/intl and commerceguys/addressing libraries (“packages” in Composer-speak). We would create a composer.json file, and list the dependencies there:

{
    "require": {
        "commerceguys/intl": "dev-master",
        "commerceguys/addressing": "dev-master"
    }
}

Available packages are listed on Packagist, a central repository anyone can add to. Running composer install will download the dependencies:

Loading composer repositories with package information
Installing dependencies (including require-dev)
  - Installing commerceguys/intl (dev-master d4dca7e)
    Cloning d4dca7e4d8e5d5cee65c0e987e650daa056f3948

  - Installing commerceguys/enum (v1.0)
    Loading from cache

  - Installing doctrine/collections (v1.3.0)
    Loading from cache

  - Installing commerceguys/addressing (dev-master 98e2cae)
    Cloning 98e2caedbd71a838a7b2ececbe7691d480b0f6a5

Writing lock file
Generating autoload file

Notice how Composer has installed two more packages we didn’t ask for.
Those are the dependencies of our dependencies. You can see them listed in the commerceguys/addressing composer.json file.

Once installed, packages can be kept up to date by running composer update.
Composer knows which versions it’s allowed to install/update to by looking at the composer.json file. The dev-master in our examples means that it should always fetch the latest dev release. We could instead use wildcards to allow only releases of a specific branch.

If you’ve ever used Drush Make to download Drupal modules and themes, then all of this sounds familiar. You can think of Composer as the more advanced Drush Make that works for all PHP projects and packages. Compared to Drush Make, Composer has the benefit of being able to recursively resolve dependencies (downloading the dependencies of each dependency) and being able to detect conflicts.
For example, let’s say package A and package B both require Guzzle, the HTTP client. However, package A specifies that it only works with Guzzle5, while package B specifies that it only works with Guzzle6. In this case Composer will abort the install process and warn us.

Composer is clearly an awesome tool, but why do we NEED to use it? Using Drush Make was always optional for site builders. The secret lies in the last line of Composer output: Generating autoload files.
Modern applications such as Drupal 8 consist of many classes, so it would be impractical and costly to manually include each one. Instead, the application includes one special class, called the autoloader which then automatically includes other classes when they are first needed. When Composer runs, it regenerates the autoloader, giving it the locations of the newly downloaded dependencies.

Inside the folder where we ran composer install there’s now a vendor directory containing both our dependencies and the autoloader.

We can commit this folder to git, or even better, run Composer on every deploy, to ensure our dependencies are fresh. People on shared hosting will need to run Composer locally and commit/upload the result, while others will find running Composer on the server more convenient.

Drupal and Composer

Now that we know how Composer works, let’s use it to download the dependencies for Address. Inside the Address module folder there’s a composer.json file listing the required packages.
Can we run composer install inside and call it a day? No.
We would end up with multiple, possibly conflicting vendor directories on the same site, one for core and one for each module that has Composer dependencies. There can be only one. Instead, we need to run Composer at the root of the Drupal install. There’s already a composer.json file there, ready to be edited.

Here’s how that composer.json file looks like by default, trimmed for sanity:

{
  "name": "drupal/drupal",
  "type": "project",
  "require": {
    "composer/installers": "^1.0.21",
    "wikimedia/composer-merge-plugin": "^1.3.0"
  },
  "replace": {
    "drupal/core": "~8.0"
  },
  "minimum-stability": "dev",
  "prefer-stable": true,
  "extra": {
    "merge-plugin": {
      "include": [
        "core/composer.json"
      ],
      "recurse": false,
      "replace": false,
      "merge-extra": false
    }
  }
}

The first requirement (composer/installers) ensures Composer knows where to put the downloaded drupal modules and themes, so that they don’t end up in the vendor directory.
The second requirement (wikimedia/composer-merge-plugin) allows other composer.json files (listed under extra/merge-plugin/include) to be merged when Composer runs. By default it’s used to merge the core requirements from core/composer.json, but can be used to merge module requirements as well.

This gives us two options for getting the Address requirements:

Composer Manager

Composer Manager discovers each module’s composer.json file and adds it to the above mentioned merge list.

cd mysiteroot
drush dl composer_manager
php modules/composer_manager/scripts/init.php
composer drupal-update

And we’re done. The vendor directory now contains our module’s dependencies.
Once we decide to install another module with Composer dependencies, for example Drupal Commerce, we just need to download that module, then run composer drupal-update again.

For more details, check out the Composer Manager documentation.

Sugar-free Composer

The second option is to treat the root composer.json file as a real Drush Make alternative, and add entire modules as requirements.

  "require": {
    "composer/installers": "^1.0.21",
    "wikimedia/composer-merge-plugin": "^1.3.0",
    "drupal/address": "8.1.*@dev"
  },

Thanks to Composer’s recursive dependency resolving, the libraries required by the module will be automatically downloaded as well.

The next thing we need to do is add the Drupal Packagist repository to the file:

    "repositories": [
        {
            "type": "composer",
            "url": "https://packagist.drupal-composer.org"
        }
    ],

This will allow Composer to find the listed Drupal modules.
This is a temporary measure, once Drupal Packagist gets moved to drupal.org, the root composer.json will come with the repository included by default.

We can now run composer update. This downloads the address module to modules/.

The next time we need to add a module or theme, we can take a shortcut:

composer require drupal/address 8.1.*@dev

This will add the line to the ‘require’ section of composer.json, then download just that specific requirement and its dependencies.

The sugar-free method brings a lot more convenience, since core, modules, themes can now be managed using the same powerful mechanism.

For more details, check out the documentation.

Sugar-free, tarball-free Composer

If Composer downloads everything for us, then there’s no need to actually start from a Drupal tarball. We can use the composer create-project command instead which will give us the structure we need:

composer create-project drupal-composer/drupal-project:8.x-dev myproject --stability dev --no-interaction

This creates a myproject directory with a slightly different project layout:

There’s once again a composer.json file and a matching vendor directory, but the actual Drupal install has been moved a level deeper, into the web directory.

The project comes with Drush and Drupal Console. They can always be removed by editing composer.json and running composer update.

Conclusion

Composer is here to stay, and it’s time we got familiar with it.
Due to its power, more and more modules are choosing to separate pieces of their logic into separate packages that can be used by the wider PHP community. They’re also choosing to depend on other people’s libraries in order to reduce their own workload.
There is also no doubt that Composer will replace Drush Make, and make itself the preferred way of assembling sites. Advanced Drupal hosting services like Platform.sh (wink wink) already support it.

In the meantime people who already started making sites without Composer can rely on Composer Manager to get required dependencies without altering their existing workflows.

Storing hierarchical data: Materialized Path

Web applications often need a way to represent and store hierarchies.
A menu with its submenus. A category with its subcategories. A comment and its replies.
Storing the hierarchy, and later reconstructing it from the stored data is just a part of the puzzle. We also need a way to find the parents or children of an item. We need to be able to re-parent an item (move it to another part of the hierarchy). Finally, there is the need to order items in a way that reflects their position in the hierarchy.

There are several ways to do this, each with its own pros and cons:

• Adjacency list
• Nested set
• Closure table (aka bridge table)
• Materialized path (path enumeration)

I won’t compare all of them, but a quick surf through search results and StackOverflow will tell you that closure table and materialized path are potentially the two best choices.

Looking at our storage requirements, materialized path starts to look like a simpler option:
Storing the hierarchy in a materialized path requires only one column in the table.
Storing the hierarchy in a closure table requires an additional table with a large number of rows.
The closure table also won’t work if you need to sort items by hierarchy, and re-parenting items is slow and costly. On the other hand, it’s normalized, which can’t be said for materialized paths.

So, let’s give materialized paths a shot. We’ll then see how our encoding trick makes it even better.

How Devel causes heisenbugs

Here’s what killed my Friday.
The story has been edited to remove pain, suffering, prolonged coffee intake.

Inside a module I have code similar to the commerce cart refresh code.
It loads a line item, clones it, runs pricing rules and other modifications on the created clone. Then it compares the cloned and original line items to determine whether something had changed (requiring the line item in the database to be updated).

That looks something like this:

$line_item = commerce_line_item_load($line_item_id);
// dsm($line_item)
$cloned_line_item = clone $line_item;
// Pretend that as a result of a complex calculation
// or function call, the unit price of the new line 
// item has been changed.
$cloned_line_item->commerce_unit_price[LANGUAGE_NONE][0] = array(
  'amount' => '66600', 
  'currency_code' => 'EUR',
);
// Now let's compare the old and the new amount.
$old_amount = $line_item->commerce_unit_price[LANGUAGE_NONE][0]['amount'];
$new_amount = $cloned_line_item->commerce_unit_price[LANGUAGE_NONE][0]['amount'];
if ($old_amount != $new_amount) {
  dsm('The price has been changed.');
  commerce_line_item_save($cloned_line_item);
}
else {
  dsm('This can never happen.');
}

I am using Devel to do some light debugging. At one point, wanting to see which
line items get processed, I uncomment the dsm($line_item) on top of the script.

Suddenly, the output changes to “This can never happen.”. What happened?
The dsm call had turned the line item variables (such as commerce_unit_price) into references. PHP’s clone is a shallow clone, so it only cloned the line item, not caring about what’s inside.
With both $line_item and $cloned_line_item having the same references for its fields, changing $cloned_line_item->commerce_unit_price also changed $line_item->commerce_unit_price. This means that the line item in the entity controller static cache has been changed as well, so doing another commerce_line_item_load() inside the same request will return a line item with the wrong unit price.
Many subtle bugs followed, followed with suspicious glances at Entity Wrapper (later proved to be innocent).

So, my simple debugging call caused all of the bugs I was seeing from that point on.

There’s an issue in Devel’s issue queue from November 2012 about this:
Krumo side effect: Object vars become references.
Click the “follow” button, and be very careful what you use Devel for.
Note that Devel in D8 uses Kint instead of Krumo, so it might be immune to this.

Detecting the system timezone from PHP

PHP Warning: date(): It is not safe to rely on the system’s timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone ‘UTC’ for now, but please set date.timezone to select your timezone.

We’ve all seen this at least once.
PHP5 requires you to have a timezone specified in your php.ini.
If you don’t, it will issue the warning and versions <5.4 will try to autodetect the system timezone.
This didn't always work, so PHP 5.4 dropped the autodetection code completely and left us on our own. Wonderful.

Normally, this isn’t such a big deal, a script can always run
date_default_timezone_set() to set a new default.
However, I’m currently writing a CLI tool (using Symfony Console) and prompting the user to specify a timezone is much more annoying in this context.
So I wrote some code that tries to autodetect the system timezone, with a UTC fallback:

$timezone = 'UTC';
if (is_link('/etc/localtime')) {
    // Mac OS X (and older Linuxes)    
    // /etc/localtime is a symlink to the 
    // timezone in /usr/share/zoneinfo.
    $filename = readlink('/etc/localtime');
    if (strpos($filename, '/usr/share/zoneinfo/') === 0) {
        $timezone = substr($filename, 20);
    }
} elseif (file_exists('/etc/timezone')) {
    // Ubuntu / Debian.
    $data = file_get_contents('/etc/timezone');
    if ($data) {
        $timezone = $data;
    }
} elseif (file_exists('/etc/sysconfig/clock')) {
    // RHEL / CentOS
    $data = parse_ini_file('/etc/sysconfig/clock');
    if (!empty($data['ZONE'])) {
        $timezone = $data['ZONE'];
    }
}

date_default_timezone_set($timezone);

A further improvement would be to try and make autodetection work on Windows as well.

Learning PHP development with Silex

A small group of students wants to learn PHP and use it to develop a small portal for a university project.
It should show best practices, and have the usual set of features (comments, a bit of ajax, sending an email, login / logout, a small admin panel).
They are already familiar with OOP through previous Java work, have learned the basic PHP syntax, and are now asking you how to proceed. A framework? Which one?

I was asked the same thing back in June, and I wasn’t sure what to answer. The last PHP frameworks I developed with were Zend Framework 1 (ugh) and CodeIgniter. Some research had to be done.

Making the choice

The basic requirements were:

1. Modern PHP code.
   This means PHP 5.3+, namespaces, PSR0, hopefully Composer.
2. Decent number of users.
   More popular frameworks have more documentation, StackExchange answers and other resources. They are also more likely to be useful later in the job market.
3. Minimal and clear.
   We wanted something that is easy to read and understand, approachable to a beginner.

Entity Bundle Plugin

After weeks of work Commerce License is finally up, as well as Commerce File 7.x-2.x to go along with it.

Commerce License provides a framework for selling access to local or remote resources.

In practice, this means that there’s a license entity, usually created during order checkout, that holds information about accessing the purchased resource, and it has a status and an optional expiration date.
This allows selling access to anything from files to node types, or perhaps ZenDesk tickets and accounts on remote sites, all using a common API, while always having a record of the purchased access.

At the heart of that API is the entity bundle plugin, which allows different license types to have different logic.
What is entity bundle plugin? The project page says only this:

This API module allow developers to build an entity type which is attached to strong behaviors.

That doesn’t help much, so let’s dive in. Let’s start by looking at how entities are built on D7.

Let’s stop using the issue queues for providing support

We spend a lot of our time in the Drupal.org issue queues, working on bug fixes and adding new features. It’s an okay tool for organizing development, and we’ve grown used to its quirks by now.
Users also try to use the issue queues to receive support, with more or less luck (typically, the bigger the module is, the less luck they have).
And while it’s great to have the support request category to reclassify confused bug reports or already implemented feature requests, as an actual tool for support the issue queue is terrible and it hurts the community.

Using Features for install profiles: The problem of default configuration

Recently I’ve made several remarks on Twitter about the pain of using the Features module for install profiles. Since they weren’t universally understood, I’ve decided to blog a bit about what is one of my pet topics: the problem of default configuration.

The problem

Requirements:

1. An install profile / distribution needs to be able to export and provide default configuration (content types, fields, variables, etc).
2. The user needs to be able to revert to default configuration when desired.
3. The user needs to be able to “untrack” any piece of configuration, allowing it to be deleted.
4. The user needs to be able to export and version his own export of the configuration, maintaining his changes across distribution upgrades.

The last two requirements are not satisfiable with Features, because Features has no concept of “default configuration” over regular configuration.

Using a distribution as site core via Drush Make

By now most of us have used a Drupal distribution.

Drupal.org takes a distribution’s drush make file, fetches Drupal core, contributed modules, themes, libraries, adds custom modules, and provides an archive that’s ready to use and install.

While distributions provide a great base for a site, we always need additional pieces (modules, libraries) in order to satisfy our use case.
So, how do we use a distribution as a starting point, and add our modules on top?

bojanz.wordpress.com

It’s been two and a half years since I last blogged.

Back then, I used an alpha version of D7 that didn’t even have imagefield working properly. In time, the installation became too annoying to remigrate (there was no upgrade path for pre-releases), and I decided that Twitter was a better medium for what I wanted to say, so I closed the blog.

Fast forward to May 2013, I’ve slowly filled up a queue of technical topics I want to discuss, so I’ve decided to reopen a blog. Also decided I want a hosted solution, but not Drupal Gardens, since I’m already very familiar with Drupal 7. Instead, I am giving WordPress a shot for the first time. It fits nicely with my efforts to get off the island this year, branch into other projects and work on new things. So far I’ve had a lot of fun playing and developing with Silex and AngularJS, as well as contributing to an oauth2 server library. The pull of Python is always strong, too. Fun times ahead, especially considering the approaching Drupal 8 and the many advances it brings.

So, here we go…