Lullabot

A Recipe for Creating CouchDB Views

When we built the Lullabot.com front-end using React, we needed a way to get data from Drupal. Setting up a REST API directly from Drupal was one way of doing this. We found, however, that the existing modules for implementing our own API were rather complex and not so performant. In addition, since the navigation at Lullabot.com was anonymous, we did not need the Drupal bootstrap for user sessions when querying the API. Instead, we decided to use CouchDB, a NoSQL database which stores its data as a flat collection of JSON documents. The best thing about CouchDB compared to other NoSQL offerings available out there is that it provides a REST API out of the box. This makes CouchDB the bee’s knees for this project. In this article, we will share a few recipes for querying and aggregating data with CouchDB.

We will be using curl to access the API in the examples for this article, but you can use your preferred REST client.

Creating a Design Document

Since CouchDB is a NoSQL database, we can't write queries to join and aggregate data in the same way we would when using a relational database such as MySQL. There are no tables and no relationships. There is, instead, a JavaScript view engine to help us create and run queries using the map-reduce paradigm. The map portion of the paradigm takes a list of values and applies the same computation to each value to produce a new refined list. The reduce operation then collapses or combines those values into a single or given number of values. For example, the total sum of all the values. The views which act as the map part of the paradigm, are defined using JavaScript functions that take a single CouchDB document as an argument. These functions determine which documents are made available through the views. 

View functions live in the database’s design documents. Design documents store application code that is run inside the database. The following example declares a view:

{ "views": { "all": { "map": "function(doc) { emit(doc.title, doc) }", } } }

We can save this into a javascript file, upload it to CouchDB, and save it as design document to create the view. For example, using curl:

curl -X PUT @docs_view.js 'http://localhost:5984/lullabot/_design/[your_design_doc]'

In this case we are making a PUT request because a new design document is being created at http://localhost:5984/lullabot/_design/docs. If the request was successful we get this response:

{"ok":true,"id":"_design/docs","rev":"1-485d01ba401d41a9e922780eca73a384"}

You can check on the design document by retrieving it:

curl -X GET 'http://localhost:5984/lullabot_1/_design/[your_design_doc]' { "_id":"_design/docs_1", "_rev":"1-485d01ba401d41a9e922780eca73a384", "views":{"all":{"map":"function(doc) { emit(doc.title, doc) }"}} } The Basic View

To start with, as an example, we will use the following document structure for storing a speaker's conference events:

{ events: [{ title: "DrupalCon Barcelona 2015", subtitle: "Monday 21st-25th September 2015 — Barcelona", role: "speaking", link: "http://lanyrd.com/2015/drupalcon-barcelona/", location: "Barcelona, Spain", start_date: 1442808000, end_date: 1443153600 }, { title: "ZendCon 2015", subtitle: "Monday 19th-22nd October 2015 — Las Vegas", role: "attending", link: "http://lanyrd.com/2015/zendcon/", location: "Las Vegas, United States", start_date: 1445227200, end_date: 1445486400 }], speaker: { name: "Joe Shindelar", headshot: { uri: "bio_image/dsc_0272.jpg", mime: "image/jpeg", }, link: "who-we-are/joe-shindelar" }, type: "speaker" }

To get a list of all the speakers, the map function looks like this:

function(doc) { if (doc.type && doc.type == 'speaker') { emit(doc.speaker.name, doc); } }

When we query the database using a view for the first time, each record in the database is run through the map function. This only happens once when the view is first run so that the database can be indexed and stored in a B-tree (Further reading on performance can be found here). The emit() function outputs a key and the value, which we specify as the first and second parameter respectively. In this case the key is the speaker's name and the value is the entire document. Here we also check if the document being returned is of the type speaker so that only speaker documents are returned.

Each view comes with a neat way to access it via the REST API. The views can be retrieved at a url that looks like this: /[your_database]/_design/[your_design_doc]/_view/[your_view]. The view we just created is accessed at http://localhost:5984/lullabot/_design/docs/_view/speakers.

To query the above view we can send a GET request using curl:

curl http://localhost:5984/lullabot/_design/docs/_view/speakers

This returns a list of speakers with their names as the key:

{ "total_rows": 22, "offset": 0, "rows": [{ "id": "84e77231cae12a572cc6724241004f43", "key": "Angus Mak", "value": { "_id": "84e77231cae12a572cc6724241004f43", "_rev": "6-5f9bee7b6b2c428657fdc8b2a7d5dcea", "events": [...], "username": "makangus", "speaker": {... }, "type": "speaker" } }, { "id": "84e77231cae12a572cc6724241001f47", "key": "Jared Ponchot", "value": { "_id": "84e77231cae12a572cc6724241001f47", "_rev": "6-87ee7de4b615a75dc786a85367c5e445", "events": [...], "username": "jponch", "speaker": {... }, "type": "speaker", } }] } Querying a Document Set

Now that we've nailed down a simple view, we can look at an example of how to query it. Suppose we want to display a list of events a speaker is attending. We would need to query for their particular speaker record. As the map function above returns the speaker's name as the key, we can easily pass the in the query parameter when we are making our request:

curl http://localhost:5984/lullabot/_design/docs/_view/speakers?startkey=['Angus Mak']&&endkey=['Angus Mak']

Here we have used the startkey and endkey parameters. Under the hood, the view is stored in a tree structure, when we specify a startkey, it tells CouchDB the position to start looking. If we have an endkey defined, the lookup will stop when it hits the endkey. In this case the start and end keys are the same, so we return just this record as there is only one match.

Building more complex queries

So a straightforward display of each record is simple enough, let’s complicate matters with a real scenario. For the Upcoming Talks page, we want to display a list of all talks that staff at Lullabot are presenting in chronological order.

The lullabot.com upcoming talks page

The problem is that each event does not have it’s own record but is stored as an array as part of the speaker record.

Here is the map function:

function(doc) { if (typeof doc.type !== 'undefined' && doc.type === 'speakers') { doc.events.forEach(function(event) { emit([event.role, event.session_date, event.title], { event: event, speaker: doc.speaker.name }); }); } }

To solve this problem, we iterate over all the events listed in each speaker record and output a result for each event. The resulting objects contain both the event and speaker objects. The key is a composite one made up of the role, event title, and session date. The fields in the key have been carefully chosen for the following reasons:

  • We want to be able to order the sessions chronologically, so the session_date was included.
  • We would also like to filter by whether or not a speaker is just attending or speaking at an event.

Here is a resulting row from the view with each event displayed as it’s own result row. A key/value pair is produced for each event:

{ "id": "84e77231cae12a572cc6724241004f43", "key": ["attending", 1442808000, "DrupalCon Barcelona 2015"], "value": { "id": "84e77231cae12a572cc6724241004f43", "event": { "title": "DrupalCon Barcelona 2015", "subtitle": "Monday 21st-25th September 2015 — Barcelona", "role": "attending", "link": "http://lanyrd.com/2015/drupalcon-barcelona/", "location": "Barcelona, Spain", "start_date": 1442808000, "end_date": 1443153600 }, "speaker": "Angus Mak" } }

To display just the speaking events, the query looks like this:

curl http://localhost:5984/lullabot/_design/docs/_view/upcoming_talks?startkey=["speaking"]&endkey=["speaking",{},{}]

Since we are only using one part of the key to filter with, we need to also specify the values of the other two parts of the key in the end key. As we are only interested in filtering by role, we can use wildcards {} for the other parts.

As a further example of querying using a compound key, if we want to filter all the speaking events between a date range, the query is as follows:

curl http://localhost:5984/lullabot/_design/docs/_view/upcoming_talks?startkey=["speaking",1442808000]&endkey=["speaking",1444968000,{}]

Here we are filtering between 21st Sept 2015 - 16th Oct 2015. When we are querying using a compound key, the order of each key part is important. We can only filter in the order of each part of the key. For example in this case, we can filter by [role], [role, session date] and [role, session date, event title], but we cannot filter by [role, event title].

Sorting

On the Upcoming Talks page, we would like to display the most recent speaking session first. We can use the descending parameter.

curl http://localhost:5984/lullabot/_design/docs/_view/upcoming_talks?startkey=["speaking",1444968000,{}]&endkey=["speaking",1442808000]&descending=true

Notice that the start and end keys have been flipped compared to the previous query example. This is because when we sort in a descending order we reverse the position in which we begin searching. The example below displaying a list of keys illustrates this.

["speaking",1431359982,"DrupalCon Los Angeles 2015"] ["speaking",1442808000,"DrupalCon Barcelona 2015"] // End position here ["speaking",1443844800,"Open Source North"] ["speaking",1444241700,"HOW Interactive Design Conference: Chicago"] ["speaking",1444449600,"New England Drupalcamp (NEDCAMP)"] ["speaking",1444968000,"DrupalCamp Atlanta 2015"] // Start position here ["speaking",1445227200,"ZendCon 2015"]

When you have a compound key, the fields are sorted in the same order as they are in the key. So in this case, the results are sorted by role, date, and then title.

Using the Reduce Function

Suppose we want to display some statistics on how many sessions Lullabot staff are speaking at in total. We can use the reduce function to help aggregate the data. The reduce function is executed on the documents returned by the map function. Similar to the map function, the reduce function operates once on each document.

With a few modifications, the map function looks like this:

function(doc) { if (typeof doc.type !== 'undefined' && doc.type === 'speakers') { doc.events.forEach(function(event) { emit([event.role, event.title], 1); }); } }


The resulting rows are of the following structure:

{"id": "84e77231cae12a572cc6724241004f43", "key": ["attending","DrupalCon Barcelona 2015"], "value": 1}

In order to count up all the speaking sessions, we can add a reduce function to our view in the design document:

{ "views": { "upcoming_talks": { "map": "function(doc){ //map function operations..}", "reduce": "function(key, values){ //reduce function operations..}" } } }

Here is the reduce function:

function(key, values) { return sum(values); }

The reduce function is called after the map function. The key parameter is an array containing all the keys returned by the map function and the id of the corresponding document. The values parameter contains an array of the values returned by the map function. When the function is run at index time, the values (in this case 1) are all added together using the the sum function to give us the number of sessions Lullabot are speaking at for each event.

The results are grouped using the whole key. In this case, we are adding the values where both parts of the key (role and title) are the same. For example:

["speaking", "DrupalCon Barcelona"] ["speaking", "DrupalCon Barcelona"] ["speaking", "DrupalCamp Atlanta 2015"]

The above contents of the key array will return the values 2 sessions at DrupalCon Barcelona and 1 session at DrupalCamp Atlanta 2015.

To return the number of total sessions overall, we can add the group_level parameter to the query.

curl http://localhost:5984/lullabot/_design/docs/_view/upcoming_talks?startkey=["speaking"]&endkey=["speaking",{}]&group_level=1

This will group the results by only the first part of the key and the reduce function will return the total number of sessions overall.

Conclusion

In this article, we discussed some basic techniques to get the data from couchDB. From our experience with using couchDB in the lullabot.com project, we found that it was a great alternative to building a Drupal REST API for powering the data layer of a decoupled site. The advantages are:

  • If the Drupal database becomes unavailable, it does not affect the front end site.
  • CouchDB provides a REST API out of the box.
  • Increased security as only the content editors have access to the Drupal back end.

When considering if couchDB is a suitable solution for a decoupled site, the advantages need to be balanced with the overhead of managing how the data is exported from Drupal to couchDB, and maintaining a couchDB instance in addition to the back-end Drupal database.

Lullabot's 8th Annual DrupalCon Party

Lullabot’s annual party has become a DrupalCon tradition – fun friendly people hanging out and having a good time. If you’re new to DrupalCon, it’s a great place to meet people. And if you’re an old-timer, it’s a great place to see old friends and make new ones.

Lullabot’s DrupalCon Party 2016
Wednesday, May 11th at the Ace Hotel
600 Carondelet St.
New Orleans, LA 70130
7PM ‘til whenever
(a 15 minute walk from Drupalcon)

We have 31 Lullabots attending Drupalon this year. Nine of them are presenting sessions, so don’t miss those. Also, both Lullabot and Tugboat will be representing at booth 206 in the exhibit hall. We’ll have our famous floppy disk party invites at the booth, so stop by early on Tuesday if you want to fill out your collection. And finally, since it's our 10 year anniversary, if you happen to stop by the booth wearing an old Lullabot tee, you'll also receive a new special edition Lullabot shirt. Hooray for new threads!

The venue for the party is a short 15 minute walk from the Convention Center. So stop by on Wednesday evening, enjoy a drink with us, and say “hello!”.

Lullabot DrupalCon Sessions 2016

This year we have a stellar lineup of sessions by the Lullabot and Drupalize.Me teams which were accepted for DrupalCon North America being held in New Orleans. Take a look at who is presenting and read a short synopsis of what they’ll be talking about.

Coding and Development Altering, Extending, and Enhancing Drupal 8 - Joe Shindelar A large part of Drupal's appeal lies in its flexibility. The fact that a developer can alter, extend, and enhance almost any aspect of Drupal without having to hack core. Historically this versatility has been made possible through the existence of hooks. Specially named PHP functions that are executed at critical points during the fulfillment of a request. And they've served the framework well for years. But times are changing, and Drupal 8 offers a variety of new patterns that all module developers will be required to learn, and understand. Configuration Management for Developers in Drupal 8 - Matthew Tift Is the configuration system your favorite feature of Drupal 8? Are you interested in doing continuous integration? Do you want to easily export all of your Drupal configuration to code? Interested in building a best practice continuous integration and deployment solution? This session, hosted by co-maintainers of the configuration system, will focus on how Drupal 8's configuration management system works, how to integrate it with a continuous integration system, and what developers can do to extend its power through contributed modules and custom code. Come with your questions and learn more about this magical part of Drupal 8. Core Conversations Drupal (admin) as an application: More JavaScript in core? - Marc Drummond In recent months, much debate has revolved around the compelling user experiences increasingly accompanying the runaway growth of JavaScript frameworks. Some argue that Drupal already has many moving parts and should evolve toward more seamless user experiences with existing tools and better processes. Some argue that Drupal should address this trend with additional capabilities for JavaScript in the form of a JavaScript framework. Some argue we should look at using modern PHP and JavaScript technologies that don’t require a JavaScript framework. Others have positions that fall both inside and outside this spectrum! Learning to Let Go (Contrib Burnout) and Module Giveaway - Dave Reid How can someone deeply involved in the Drupal contributed module ecosystem start to step away? How do we handle burnout not just in Drupal core development, but in contrib? I'd like to engage a conversation based the challenges I have encountered and currently face personally/emotionally on my journey from being one of the top contributors to Drupal 7, prolific writer of modules, to someone starting a family and needing to rebalance their personal, work, and Drupal life. With so much focus on getting people involved in Drupal.org, are there technical solutions we can explore to help make active contributors happier? Drupal.org Documentation Is Getting An Overhaul - Joe Shindelar Having high-quality documentation available for Drupal.org is key to gaining wider adoption, growing the community, and the overall success of the Drupal project. I want to share the work related to documentation going on in the community, as well as some of our plans for continued improvement in the future. Front End Debugging, Profiling, & Rocking Out with Browser-Based Developer Tools! - Mike Herchel Browser based developer tools have become an indispensable tool for modern front-end web development. New features and changes are being added at a rapid pace, and keeping up with all of the changes is difficult, but well worth it! In this session, Mike will walk attendees through modern debugging techniques, tips and tricks, front-end profiling, and more! Sizing up responsive images: Make a plan before you Drupal - Marc Drummond Drupal 8 has built-in responsive images support based off of Drupal 7’s contributed Picture and Breakpoint modules. Understanding how to use those modules without first making a plan could easily lead to a cat-tastrophe! Horizons AMPing up Drupal - Karen Stevenson, Matthew Tift, and Marc Drummond In many cases, the mobile web is a slow and frustrating experience. The Accelerated Mobile Pages (AMP) Project which involves Google is an open source initiative that embodies the vision that publishers can create mobile optimized content once and have it load instantly everywhere. When AMP was first introduced last October 2015, many commentators immediately compared it to Facebook's Instant Articles and Apple's News app. One of the biggest differentiators between AMP and other solutions is the fact that AMP is open source. Beyond the Blink: Add Drupal to Your IoT Playground - Amber Himes Matz What does making a light blink have to do with Drupal? Come to this session to find out how you can add Drupal to your Internet of Things data playground. (THERE WILL BE BLINKING LIGHTS.) Site Building Recoupling: Bridging Design and Structured Content - Jeff Eaton For years we’ve talked about separating content and presentation. Structure, reuse, and standardization are the name of the game in a future-friendly, multi-channel world — aesthetics are someone else’s concern … right? UX Web Accessibility 101: Principles, Concepts, and Financial Viability - Helena Zubkow If your website wouldn't work for anyone living in the state of New York, would that be a launch-blocker? Of course! So why are we ignoring the even larger population of people with disabilities?

Photo by: Jeff Turner and used via Creative Commons License

Selling Drupal Modules and Distributions

Matt and Mike talk to Taco Potze, Robert Douglas, Matthew Tift, Greg Dunlap, and Jeff Eaton about selling Drupal modules, and all of the benefits, downsides, and questions that arise from selling GPL and dual-licensed software in the Drupal community.

Why Paid Drupal Modules Fail: Drupal as Art

Recently the Drupal Association announced accepted sessions for DrupalCon New Orleans. While it looks like we can expect another great DrupalCon (this will be my 7th straight North American DrupalCon), one particular session on the program about the sale of Drupal modules caught my attention. Although I have tried to stay focused on preparing my own sessions, I have had conversations with other people in the Drupal community about “paid modules” that have led me to the conclusion that confusion about this topic persists. So here I would like to offer a perspective on why these kinds of plans consistently fail. Specifically, I hope to expand the scope of this frequently discussed issue and suggest why so many paid module initiatives fail: the Drupal community protects its free software with the same vigor that other communities protect artistic freedom.

The GPL Protects Software Freedoms

Before offering my analysis, I should start by acknowledging the obvious reason why paid Drupal modules fail: the General Public License (GPL). The Drupal Association clearly states that “Drupal modules and themes are a derivative work of Drupal. If you distribute them, you must do so under the terms of the GPL version 2 or later.” Consequently, anyone who wishes to sell a Drupal module must provide the source code.

Drupal community members are standing by with their credit cards ready to purchase those modules – and then post the code to Drupal.org so that everyone else can use it. These principled individuals believe that copyleft provides a powerful tool for ensuring software freedom and will exercise their right guaranteed by the GPL that allows the recipient (buyer) “to copy, distribute or modify” the module or theme purchased. The seller “may not impose any further restrictions on the recipients’ exercise of the rights.” The GPL greatly curtails most plans to sell Drupal modules, and these would-be capitalists might have more success selling popsicles during a blizzard.

Yet the GPL does not close the debate on this subject. I have heard many reasons provided to justify revisiting this conversation, and most frequently they come down to money. Let me share some direct quotes used to justify the pursuit of a “Drupal Marketplace,” “paid Drupal apps,” or “paid modules”:

I have read and listened to a great deal of arguments defending these commercial endeavors, and I remain unconvinced that the potential upsides outweigh the considerable drawbacks.

A History of Failure

The words of the American literary critic Fredric Jameson influence my thinking on this topic: “always historicize!” A look at attempts to sell Drupal modules reveals a distinct lack of success, and yet people continue to claim they have found a solution. Consider SubHub, who announced in 2011 to great fanfare the “World’s First Drupal-Powered App Store.” They hoped to offer some of their “apps” at no cost, while other “apps” would require a small recurring charge. This plan failed and SubHub abandoned their initiative in 2013, lamenting the fact that the Drupal community “simply didn’t share the same motivation to make Drupal a highly commercial, competitive alternative to WordPress.” My apologies to anyone who built a Drupal site with integral SubHub functionality.

Also in 2011 – a big year for “apps” in the Drupal community – Phase2 announced the Open App Standard initiative. The title contains the word “open” so surely this plan would find traction with Drupal people, right? While Phase2 found some success with OpenPublic, which uses apps, I don’t see evidence that “apps” ever found traction in the Drupal community, and certainly not adoption with alacrity.

Keep in mind that many people make money selling Drupal services and that the community generally supports such efforts. I work at a company filled with people who make money building Drupal websites. Rather, I think this evidence shows that paid module schemes tend to fail, that others have found Drupal to be “actually a horrible solution to build apps,” and that when people ask the question, “Is a Drupal App Store a good idea?” the community generally responds with a decisive “no” (unless it features ponies). We absolutely want people to succeed in the community, just not by selling modules.

Certainly exceptions exist. For instance, some companies have found success selling Drupal themes. A case could be made that Acquia, a company founded by Drupal’s creator, peddles its own variety of “paid apps.” The Acquia Connector module “enhances the Drupal experience by providing the support and network services to operate a trouble-free Drupal website.” However, the module does little without an Acquia Subscription, which requires regular payments.

Acquia, and other similar services, get around the restrictions of the GPL by taking advantage of something known as the “application service provider (ASP) loophole.” When the Free Software Foundation published GPL version 2 (all Drupal code must be GPL version 2 or later) in 1991 we did not use web services like we do today. Because providing a subscription service (such as Mollom or New Relic) does not involve distributing software, Acquia need not provide any additional code than that which connects the site to the service.

The Drupal community could adopt a stronger copyleft license that closes the ASP loophole, such as the Affero General Public License (AGPL). Just do not expect this to happen anytime soon. Dries Buytaert, the founder and project lead of Drupal, built a business that takes advantage of this loophole and he made his opinion clear a decade ago in a blog post titled, “Long Live the Web Services Loophole.”

Consequently, the focus of the discussion around paid modules often revolves more around the merits and problems of what Richard Stallman calls “Service as a Software Substitute (SaaSS)” than on actually selling Drupal modules that address common challenges. While some companies have found success with the SaaSS model, why do so many others fail? To answer this question, we must look beyond code and licenses to the Drupal community.

Products and Practices

I wrote at length about the Drupal community in previous articles such as “The Cultural Construction of Drupal” and “Better, then Bigger: Cultivating the Drupal Community,” and I do not intend to re-hash those same ideas here. Rather, I would like to examine why “paid module” schemes have flopped.

As I see it, one fundamental explanation for failure has to do with when business people misunderstand Drupal as a product rather than an organic accumulation of practices, or more broadly, when they ignore the differences between machines and humans. I deliberately add the qualifier “business” to describe these people because all efforts to create paid products are intended to generate revenue. Efforts to sell Drupal modules, in the Marxist sense, epitomize efforts to create capital, to accumulate money – a goal that comes off as at odds with a community that values participation.

All modules that come with a fee are intended to generate revenue, but we must not forget that “free” modules (that are also “Free”) have the potential to do the same. Many people, including me, enjoy getting paid to write Drupal code. For the paid module argument to work, its defendant must demonstrate that existing models are somehow lacking. Defending a new commercial approach necessitates, in my view, demonstrating the unsatisfactory nature of a free software project with more than 33,000+ shared modules that has existed for more than 15 years and has a vibrant marketplace of vendors offering Drupal services. Such an argument not only requires a high level of mental gymnastics, it, at least tacitly, represents an affront to years of successful collaboration.

Because some of those recommending that Drupal re-evaluate its commercial ecosystem are contributing members of the community, I open myself up to the critique of constructing inaccurate divisions, pitting Drupal and its community against business people who desire to convert Drupal into a revenue-generating product. But we know too well that many of us in the Drupal community could represent either side. We aim to balance our need to support our families with our desire to defend Drupal and fight for its freedom. Luckily, we can often choose both.

Drupal as Art

Moving away from discussions of licenses and capitalist motives, I would now like to venture beyond the typical boundaries of the “paid module” discussion to explore why people feel so connected to Drupal. I get the sense that many people in the Drupal community do not actually understand the intricacies of the GPL and how it differs from other free software licenses such as the AGPL. I believe that the community is protecting something more ephemeral. Consequently, this part of my argument ventures into much more abstract topics and requires thinking about Drupal differently, less as a “technology” and more as an artistic practice.

I am not the first person to notice Drupal’s artistic side. For example, a DrupalCon session had the title “Zen and the Art of Drupal.” Someone else even created a new word, “Drupl’Art,” to describe “Drupal code that is both beautiful and artistic.” The conception of “Drupal as art” is not new, but I am going to be using it here in order to offer new insights into how the Drupal community works.

More than just a thought experiment, the idea of Drupal as art becomes useful when we position it within another long-running debate, namely the effect of technology on art. For instance, when proponents position technological advances as something that improves the lives of many, critics will sometimes note that those advances also threaten the purity of art – for example, photographs (the “new technology”) were seen as a threat to portraiture (“pure art”). Ironically, Drupal, as I construct it here plays the role of art, even as we cannot deny that most people would label Drupal “technology” well before they would ever call it “art.” And yet when members of the Drupal community react negatively to the suggestion of paid modules, it is not simply because of the GPL, it is because the community is defending its perceived “purity.”

Drupal and art have both been understood as pure expressions, deeply tied to their predecessors, consisting of ever changing practices, and driven by community. Yet it would be idle to suggest that concerns about the threat of technology to art are exactly the same as concerns about the effect of paid modules on Drupal’s ecosystem. Nonetheless, I believe that we can better understand the Drupal community’s typical allergic reaction to “paid modules” by interrogating previous debates about technology and art.

This line of reasoning follows a long history of thinkers who have conflated art and technology that goes at least as far back as Ancient Greece. The Greeks did not enjoy art for aesthetic reasons. The word “technology” refers to a “treatise on a practical art or craft” and its Greek root is techne. For the Greeks, techne also referred to art. Aristotle described techne as both practical knowledge and practical art. And what Drupal enthusiast would not admit that most websites convey practical knowledge?

In his 1954 essay, “The Question Concerning Technology,” the German philosopher Martin Heidegger variously described both art and technology as a “mode of revealing,” a way of “bringing forth.” A painting reveals another person’s point-of-view and a website “reveals” information. As uncomfortable as it may seem to some among us to describe Drupal as art, it helps explain why people feel so connected to Drupal and vigorously defend its purity. The community wants to work together and reveal its solutions rather than hide them behind a web service. Developers treasure not just revealing websites, but revealing (sharing) the code in the modules that enables their functionality.

Another well-regarded German philosopher, Walter Benjamin, provided valuable insights that are useful for our purposes in his 1936 essay, “The Work of Art in the Age of Its Technological Reproducibility.” Therein Benjamin explores when inventions such as photography and film (both being kinds of “mechanical reproduction”) “separated art from its basis in cult, the semblance of its autonomy disappeared forever.” Not only did these products threaten the group, they also threatened the group’s output (its art). He believed that “the unique value of the ‘authentic’ work of art always has its basis in ritual.” Likewise in the Drupal community we value ritual: we discuss issues, post patches, review the work of others, attend camps and cons, celebrate the accomplishments of new members, and create code with unique value – code that lives with other Drupal code on Drupal.org. Hiding code threatens our rituals.

Paid modules or services revoke our access, our autonomy, and our beloved practices. Drupal is something people do, and we cannot learn by doing when we cannot see the code. Products are purchased, while practices are lived. Drupal, like other forms of techne, is communication, and we cannot communicate in the same manners with black boxes as we can in the issue queue.

In addition to affecting what we do, the existence of paid modules could negatively affect perceptions of the Drupal community. Benjamin writes of a “decay of the aura,” which sounds much like what the Drupal community works against. While some may argue, as one Slate writer did, simply that “Drupal hates change,” many in the Drupal community still believe that Drupal exudes a sort of magical aura. We hold the community close to our hearts and defend its name. We do not half-heartedly promote our community, we instead speak of our “unique, diverse Drupal community,” say that “Drupal has made a big difference in my life,” and that “I’m truly proud to be part of this community.” We protect our people, practices, and reputation by following the Drupal Code of Conduct, in the hope that “we preserve the things that got us here.”

Our current system for creating modules on Drupal.org values the work of humans. Benjamin observes, “What matters is the way the orientation and aims of that technology differ from those of ours. Whereas the former made the maximum possible use of human beings, the latter reduces their use to the minimum.” Drupal is not so much the thing that is built but the way it is built by people. In fact, one of Lullabot’s most frequently quoted core values is “be human.”

The Drupal software will never be complete, and we hope the sharing continues indefinitely. In the same spirit, Benjamin believes, “It has always been one of the primary tasks of art to create a demand whose hour of full satisfaction has not yet come.” The Drupal community wants to continue building, to keep moving forward, never content that we have finally arrived at an “end” where our interactions have been replaced by web services.

Fredric Jameson once warned that “machines are indeed machines of reproduction rather than of production.” The Drupal community has produced a great deal of code that powers much of the web. Few among us would want to shift from production to reproduction, from working together to solve problems to purchasing solutions to problems. Repeatedly and continuously solving the sames problems – building the same websites – breeds boredom. We like to create new and shiny things, producing not reproducing. Our rituals, our process, and our freedom to tinker allows us to continuously move forward.

The Contrasting Case of WordPress

Even if these theories accurately describe the Drupal community, they do not fully account for the success of paid WordPress plugins. Do paid plugins make WordPress unpure, inhabited by a community of dispassionate contributors? Absolutely not. Does the availability of paid plugins somehow cheapen the WordPress community or imply that WordPress lacks meaningful rituals? Certainly no. WordPress powers a huge chunk of the web with GPL-licensed code. It would be fruitless to deny the overwhelming success of a project with a vibrant community. Rather, I hope to convey how the WordPress community’s embrace of paid plugins informs my argument that the Drupal community understands its practices as a kind of artistic expression.

Simply suggesting that WordPress and Drupal are different helps about as much as arguing that Drupal and WordPress are words that contain an incommensurable number of letters. We could go a step further, as Dries Buytaert often does and argue not only are they different, but that WordPress and Drupal target different markets, with “WordPress dominating the small business segment” and Drupal geared toward the “larger organizations with more complex requirements” (an idea I dispute). If that were true, one would think the community that caters to “large organizations” would have the customers with the funds to purchase modules rather than get them at no cost. Then again, the reverse argument seems equally defensible, and that small businesses would rather pay for plugins than for those reportedly expensive Drupal developers. None of these avenues feel satisfactory.

One might expect that WordPress and Drupal espouse contrasting ideas about paid add-ons and the GPL. Quite the contrary, the Wordpress.org licensing page provides clear guidance about how plugins should be licensed: “derivatives of WordPress,” including plugins and themes, “inherit the GPL license.” While they admit to “some legal grey area regarding what is considered a derivative work,” they “feel strongly that plugins and themes are derivative work and thus inherit the GPL license.” The community leaders back up their assertions with thoughtful discussion as well as expert analysis from the Software Freedom Law Center. The WordPress licensing page even provides a link to Drupal’s “excellent page on licensing as it applies to themes and modules (their word for plugins).” Thus, WordPress and Drupal provide almost exactly the same guidelines.

Remarkably, certain members of the WordPress community completely ignore the advice on the licensing page. They claim “the GPL FAQ has no legal validity.” Some sellers proudly declare, “got my own licensing terms” and others offer code with terms and conditions that make no mention of the GPL. Some explain the norm thusly: “Premium WordPress plugin and theme developers usually sell you a license key, which allows you access to support and automatic upgrades.” One store believes it takes advantage of GPL loopholes and sells plugins under a default split license ostensibly to “protect authors.” In other words, the WordPress.org plugin directory that lists over 40,000 plugins is not a comprehensive directory of WordPress plugins. If this sounds a bit like the Wild West, then you may be a Drupal developer.

If Drupal modules are about process, then WordPress plugins – at least a portion of them – are products. Members of the WordPress community write at length about the benefits of "premium products." Some stores offer plugins that are “guaranteed to work, always updated, top quality plugins. Beautifully coded, packed with features, and easy to use.” They offer “free trials.” Another store proudly trumpets its “professionally developed and supported options not found on WordPress.org.” Ventures of this nature are justified with such syllogisms as “‘free’ doesn’t make me rich.” Plugins like these are products, plain and simple, and clearly the WordPress community (“happy customers,” as one site put it) willingly pays.

In comparison, Drupal developers get modules from Drupal.org. More than just a “best practice,” this is the norm in the Drupal community. It keeps things simple, and has a practical benefit: acquiring a new module requires typing drush dl module-name.

Incidentally, another kind of software facilitates an analogous workflow: GNU/Linux. For example, to install software on my operating system, Debian, I type apt install package_name. I would never consider downloading software to my computer from some random website, let alone software that I could not inspect. For me, at least, these two processes (drush dl and apt get) feel nearly identical, and I could make a good argument that the Drupal community is more like the Debian community (i.e., that many of the commitments outlined the Debian Social Contract are lived in the Drupal community) than the WordPress community.

Once again, the words of Benjamin feel relevant: “It might be stated as a general formula that the technology of reproduction detaches the reproduced object from the sphere of tradition. By replicating the work many times over, it substitutes a mass existence for a unique existence.” App stores for paid plugins fragment traditions established on WordPress.org. The Drupal community, on the other hand, has decided not to purchase paid Drupal modules, not to depart from its tradition of keeping Drupal as close to 100% free as possible. Whenever this issue arises, the Drupal community votes with its voices and its wallets to favor practices over products, to reveal the modules it creates rather than conceal them behind paywalls, to work with – rather than sell to – each other. The fact that WordPress customers have chosen a different path reveals contrasting, not misplaced, priorities.

While it is difficult to generalize a community with more than a million users (or at least user accounts), Holly Ross, Executive Director of the Drupal Association, believes “the one thing we all have in common is that we care about the fate of Drupal.” It might be fate that someday paid modules will find success in the Drupal community, and that would not necessarily be wrong. It would mean, however that the essence of Drupal has changed. That may even signal the end of Drupal. But history suggests that day will not soon come. Until then, the Drupal community will continue to defend its practices. It will band together to resist paid module schemes, treat its software with a reverence that others reserve for works of art, share code, and encourage others to do the same.

Works Cited

Benjamin, Walter. The Work of Art in the Age of Its Technological Reproducibility, and Other Writings on Media. Edited by Michael W. Jennings, Brigid Doherty, and Thomas Y. Levin. Cambridge, Mass: Belknap Press, 2008.

Heidegger, Martin. The Question Concerning Technology and Other Essays. New York: Harper Perennial, 1977.

Jameson, Fredric. The Political Unconscious: Narrative as a Socially Symbolic Act. Cornell University Press, 1982.

Jameson, Fredric. Postmodernism, or, The Cultural Logic of Late Capitalism. Duke University Press, 1992.

Welcome to Drupal

Drupal 8 is here! And with all the new bells and whistles that the community built into the administrative experience, it’s simpler than ever to get your site up and running. If you’ve worked with Drupal before, you’re probably familiar with terms like “node” and “theme,” and you may have even installed Views to generate blocks of content to drop into regions in your page template. However, if you are new to Drupal, these words may seem more appropriate in a video game than in your website. You’ve come to the right place! Allow me to show you around and introduce you to Drupal.

If you’d like to follow along at home you’ll need to have a working Drupal 8 site up and running, though it's not required to continue though this article. The easiest way to get a Drupal site up and running is to sign up for a service like Pantheon or download Acquia’s Dev Desktop. If you’re more of a do-it-yourself-er, you’ll first need to setup a local environment. There is documentation on Drupal.org for just about any system you might be running. Then you’ll need to install Drupal 8. You can follow the steps in Drupal’s install.txt file, or try this tutorial written by Hui Jing.

The Basics

Before we start pushing buttons and twisting knobs, let’s go over some basic terminology. I listed the terms we'll talk about in a glossary at the bottom of the page for an additional reference.

When you download Drupal, you are downloading a set of files called Drupal core. These files make up everything Drupal needs to function. Inside there are modules, which extend Drupal’s functionality, and themes, which provide the front-end structure, layout and design. Drupal core comes with some modules that are required, others that you can enable if you like, and there are a plethora of modules created by the community on Drupal.org that you can install separately to add functionality to your site. Just like modules, Drupal ships with some themes you can use right out of the box, or you can download and install one from the community. Some themes are designed to be used right away with just a little configuration. Others are known as base themes, which provide some basic structure and are intended to be used as a starting point to create your custom theme. For example, you can choose Bartik or Seven and have a good looking and functional site right away. In fact, Seven is the theme that is used for administrative pages. Classy and Stable are two base themes in Drupal core that you can use to start building out your own theme. Check out Marc Drummond's article "A Tale of Two Base Themes in Drupal Core" to learn more about these awesome tools.

Now that we know the basics, let’s take a look at what Drupal does.

Creating Content

Every website has different requirements, but most commonly the goal is to create content that you can deliver to your visitors. In Drupal we do this by using the administrative tools to create a content type, for example Event. We then add fields to it such as Date and Location. Now we can create pieces of content, commonly referred to as nodes, of the type Event. When we add a new event, Drupal will automatically create a URL to view it and will display it in a nice format. In the simplest form that's all we need! We can create events and give people URLs where they can view them.

If you are playing along at home, this is a good place to try things out for yourself. See if you can complete the following steps:

  1. Create a new content type. Call it whatever you like.
  2. Add two fields to the content type.
  3. Create a new piece of content of this type.
  4. Visit the URL that Drupal provides to view it.

Defining content types and adding the necessary fields to them is the basis for getting content into Drupal, but what if we want to change the way it is presented on the screen? Luckily, Drupal also makes it very easy for us to change the order in which the fields appear, their formatting, their labels and even whether they should be displayed at all.

Customizing the Display

When you are adding fields to a content type you'll notice a tab labeled Manage Display. On this tab you can drag your fields up and down to reorder them, hide or show the labels and many more options depending on the fields you created. Just above the field table you’ll notice a secondary set of tabs with the option of Default selected. These are view modes. The most common way of viewing content is by visiting its url, but there are times we want to display a piece of content in a different format, for example, as a teaser in a list of recent posts. View modes let us change the way a node is displayed. If we don’t choose a specific view mode, the Default view mode is selected. A couple of common view modes are provided out of the box and can be found under Custom Display Settings at the bottom of the Manage Display page. We can also define different view modes for different purposes by going to Structure > Display Modes > View Modes through the toolbar. For the content we’ve created so far, Drupal will use the Default view mode when we visit that node’s URL. We'll talk more about ways to use view modes later. Next, let's talk about how to create custom pages that are not just pieces of content, such as an events list, blog roll, or your homepage.

So far Drupal has been providing URLs for the content we created. Somewhere in our website, however, we’ll likely want pages that show not just a single piece of content but collections of many pieces content. Maybe we want to list all of our upcoming events on the homepage. There are many modules that can help us create and manage custom pages, but installing new modules is slightly out of scope for this article. Luckily, Drupal comes with a module will take us pretty far called Views.

Custom Lists of Content

The Views module lets us define filters and sorting mechanisms to create lists of content. It calls these lists views. The Views module also allows us to display that content in a variety of formats. For example, if we want a list of the 5 most recent blog posts or the next upcoming event, we can create a view to get that content and format it for us. Let’s continue with events as an example.

When we create a new view from the admin interface, we give it basic information about what we want, such as the type of content we are looking for and how to sort it. Next we create different displays of that view. We could create a Block display to show the next upcoming event, and a Page display to show all upcoming events. Views will even link them together for us.

If you are playing along at home, this is a good place to try things out for yourself. See if you can complete the following steps:

  1. Add a new view and limit it to the content type you created earlier.
  2. Enable the Page and Block options, then Save and Edit.
  3. Visit the URL for the page you created to view it.

There are many ways to configure a view, but for now you have the basics. One last piece that we haven’t talked about yet are blocks. Blocks are small chunks of content that don’t deserve their own page, but more likely live in a separate region like a sidebar or footer. Let’s add a block to one of our pages.

Working with Blocks

Opening the blocks UI from the admin toolbar, you’ll see a list of regions with a button to Place Blocks. These regions are defined in the current theme. If you enable a new theme or create one yourself, the regions will likely change. Use the Place Blocks button to choose a block from the available list and put it into a region. If you created a view in the last section, you should see your view block in the list! Each block can be configured to only show on certain pages, to certain users, etc. After adding your block to a region, use the button at the bottom to save your changes, and you should now see your new block in whichever region you added it.

Now you have all the tools necessary to start building your website in Drupal 8. You can create content types, add fields to them, change the way they are displayed, create lists of content with Views and place blocks into your layout. There’s a lot more that Drupal can do, and the best way to learn is by trying. Now that you have a grasp on the basics, explore some of the other fields and settings in the content types, views and blocks. On behalf of the Drupal community, welcome!

Glossary

These are some of the terms I referenced. There is a more complete glossary at https://www.drupal.org/glossary.

Base Theme
A Base theme is a well-written set of CSS and template files that a theme developer can make use of in a new custom theme. Theme developers can make sub themes to override the default base theme. Some of the popular base themes include Zen, Omega and AdaptiveTheme.
Block
The boxes visible in the regions of a Drupal website. Most blocks (e.g. recent forum topics) are generated on-the-fly by various Drupal modules, but they can be created in the “administer blocks” area of a Drupal site.
Drupal Core
The files and modules included with the Drupal project download.
Module
Software (usually PHP and CSS) that extends Drupal features and functionality. Drupal distinguishes between “core” and “contributed” modules.
Content Type
Every node belongs to a single “node type” or “content type”, which defines various default settings for nodes of that type, such as whether the node is published automatically and whether comments are permitted. Common "Content Types" that just about any website would have include: blog post and page. Content types can have different fields and modules can define their own content types. The core Drupal Book and Poll modules are two examples of modules that define content types.
Field
Elements of data that can be attached to a node or other Drupal entities. Fields commonly contain text, image, or terms.
Node
A piece of content in Drupal, typically corresponding to a single page on the site, that has a title, an optional body, and perhaps additional fields. Every node also belongs to a particular content type, and can additionally be classified using the taxonomy system. Examples of nodes are polls, stories, book pages and images.
Region
Defined areas of a page where content can be placed. Basic regions include: Header, Footer, Content, Left sidebar, Right Sidebar. Different themes can define different regions so the options are often different per-site. Content is assigned to regions via blocks. They can be ordered by weight within regions to define the order in which they display.
Theme
A file or collection of files (PHP, INFO, CSS, JPG, GIF, PNG), which together determine the look and feel of a site. A theme contains elements such as the header, icons, block layout, etc. Drupal modules define themeable functions which can be overridden by the theme file. There are additional themes available in the themes section of downloads.
View Mode
A view mode is way to customize how an entity is rendered. Examples are 'Full Page', 'Teaser' or 'JSON'. In Drupal 8 this is located under Structure > Display Modes.
Views
A module which allows site developers a graphical interface for creating lists of various Drupal entities; most notably users and nodes. Views permits selection of specific fields to display, filtration against various attributes, choice of basic layout options (i.e. list, full entities, teasers, etc.), and other more advanced features. Many Drupal sites use Views extensively. In Drupal 8, the Views module is part of Drupal Core.
View
In reference to the Views module above, a view is what is created when the site builder adds or creates a new view from the interface. Each view uses a table in the database as a "base table" to build a list of objects. The view by itself is an abstract wrapper of instruction that are inherited by each view display created within the view.
View Display
In reference to the Views module above, view displays are created inside of a view to display the objects fetched by the view in different manners. For example, a view called "Blog Posts" might have view displays for "Top 5", "Most Recent", or "Posts by Author".

Injecting services in your D8 plugins

Drupal 8's plugin system is one of the most flexible and widely used subsystems in Drupal. If you are a developer, chances that you will write plugins in Drupal 8 are high.

A couple of months ago I made a case in favor of unit tests in a series of articles. Today I have good news for you, your plugins are good candidates for testing! Before you get carried away by overexcitement, it's likely that your plugins depend on other parts of the system, and that complicates things a little bit. In these cases, it is a good idea to inject the services that include the dependencies you need. Dependency injection is an alternative to the static \Drupal::service. If you don't inject your services you will have a hard time writing unit tests for your plugin's code.

There are, at least, two widely spread patterns in Drupal 8 in order to inject services into your objects. The first uses a service to inject services. The second uses a factory method that receives Drupal's container. Both of these involve the dependency injection container, although you can still inject services using other manual techniques.

Injection via services

When you declare a new service you can also specify the new service’s dependencies by using the arguments property in the YAML file. You can even extend other services by using the parent property, having the effect of inheriting all the dependencies from that one. There is thorough documentation about writing services on drupal.org. The following example defines a service that receives the entity manager:

services: plugin.manager.network.processor: class: Drupal\my_modyle\Plugin\MyPluginManager arguments: ['@container.namespaces', '@cache.discovery', '@module_handler', '@entity.manager']

You cannot use this pattern directly to inject services to a plugin, since your plugin cannot be a service. This is because services are one-instance classes, a global object of sorts. Plugins are –by their definition– multiple configurable objects of a given class. However, the plugin manager is a great candidate to be a service. If you declare your plugin manager as a service, and inject other services to it, then you are only one step away from injecting those services into the actual plugin instances. To do so you only need to do something similar to:

class MyPluginManager extends DefaultPluginManager { protected $entityManager; … public function createInstance($plugin_id, array $configuration = array()) { $instance = parent::createInstance($plugin_id, $configuration); $instance->setEntityManager($this->entityManager); return $instance; } }

As you can see (aside from the lack of docblocks for brevity), every time that your plugin manager creates an instance of a plugin it will set the entity manager for that particular instance. in this scenario you only need to write setEntityManager in your plugin. This strategy is a mix of service injection and setter injection.

Factory injection

The other big injection pattern in Drupal 8 involves adding all your dependencies as arguments in the constructor for your class (also known as the constructor injection pattern). This pattern is very straightforward, you just pass in all the dependencies upon object creation. The problem arises for objects that Drupal instantiates for you, meaning that you don't get to do new MyClass($service1, $service2, ...). How do you pass the services to the constructor then? There is a convention in the Drupal community to use a factory method called create. To implement it you need to write create static method that will receive the dependency injection container as the first parameter. The create method should use the container to extract the services from it, and then call the constructor. This looks like:

class MyOtherClass { … public function __construct($service1, $service2, ...) { // Store the services in class properties. } public static function create(ContainerInterface $container) { // new static() means "Call the constructor of the current class". // Check PHP’s late static binding. return new static( $container.get('service1'), $container.get('service2'), … ); } }

There is the remaining question of "who calls the create method with the container as the first parameter?". If Drupal is instantiating that object for you, then the system should know about the _create method pattern_ for that particular object.

The default plugin manager (\Drupal\Core\Plugin\DefaultPluginManager) will use the container factory (\Drupal\Core\Plugin\Factory\ContainerFactory) to create your plugin objects. It is very likely that your new plugin manager extends from the default plugin manager –via code scaffolding or manual creation–. That means that by default if you add a create method to your plugin, it will be ignored. In order to have your new plugin manager use the create pattern, the only thing your plugins need is to implement the ContainerFactoryPluginInterface. In that case your plugin will look like:

class MyPluginBase extends PluginBase implements PluginInspectionInterface, ContainerFactoryPluginInterface { protected $entityManager; … public function __construct(array $configuration, $plugin_id, $plugin_definition, EntityManagerInterface $entity_manager) { parent::__construct($configuration, $plugin_id, $plugin_definition); $this->entityManager = $entity_manager; } public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) { return new static( $configuration, $plugin_id, $plugin_definition, $container->get('entity.manager') ); } }

The plugin manager will detect that the object that it's about to create implements that interface and will call the create method instead of the basic constructor.

The plugin system is so extensible that you may run into corners where those techniques are not enough. Just know that there are other ways to achieve this. Some of those, for instance, involve using a different factory in the plugin manager, but that is out of the scope of this article.

Sometimes the create factory method pattern can be considered a bit more robust, since it does not involve injecting the services to an object that doesn't need them –the plugin manager– only to pass them down to the plugin. Regardless of the approach that you use, you are now in a good position to start writing unit tests for your plugin class.

Drupalcon: A Look Ahead to ‘Nawlins

Matt and Mike talk with Drupalcon organizers Rachel Friesen, Amanda Gonser, and Tina Krauss, alongside NOLA natives Eric and Sabrina Schmidt. We talk session and site submission criteria, and where the next Drupalcon will be! We also talk a lot about 'Nawlins including what and where to eat, where to listen to music, museums, what's within walking distance, and lots more!

How to try out AMP with Drupal

What is AMP, and why should I give it a try?

In “AMPing up Drupal,” Matthew Tift explained what AMP is, and how we worked to create a module, theme and PHP library to use AMP with Drupal. Spoiler alert: AMP stands for Accelerated Mobile Pages, and it’s an open source project developed to make web pages super fast on mobile. It’s particularly geared towards content like articles or informational pages, rather than e-commerce or applications. If you haven’t read Matthew’s article, do so now, as from here on out, I’ll assume you know the basics of AMP.

Lullabot released the Drupal 8 version of the AMP module and theme on February 26, and we are delighted to announce that a Drupal 7 version is now available. These are still beta versions, and we hope to continue improving upon them. We think they’re at a point where it’s worth giving them a try, even if only to test how it works locally or in a feature branch for your site.

I’m going to walk you through how to get started. Setting up your site for AMP takes a few steps, but you get a good deal of control over how your AMP pages look as a result. This walkthrough will explain the process step-by-step, providing a little more context than you might find in the README. For the bold and adventurous, you’re welcome to try this out straight from the Drupal 7 README or the Drupal 8 README

After the walkthrough, I’ll share a few thoughts on why I think you should try out AMP.

Setting up AMP with Drupal

The first key thing to understand is that AMP does not replace your entire site. AMP exists to provide a lightning fast way for site visitors to view articles and other content-focused pages.

At its essence, the AMP module provides an AMP view mode for certain content types, which the site can show when the browser requests an AMP version of a page.

On that AMP view mode, you can select AMP field formatters to take care of converting some of your content so they use the proper AMP tags. The AMP PHP library processes raw HTML, like the content in your body field after it has been filtered and had tokens replaced (if you are making use of the Token module). The library attempts to replace HTML content with AMP HTML components. It filters markup it cannot convert out of the final AMP HTML. This helps ensure the validity of content under editorial control, which may have a wide variety of embedded content.

When visiting pages with AMP-enabled content types, if the URL has ‘amp’ in the query string (e.g. my/favorite/page?amp), Drupal will use the AMP view mode and AMP theme selected on the AMP configuration page (admin/config/content/amp). The AMP Theme project provides an AMP Base theme that takes care of converting some of the larger parts of the page into AMP. For Drupal 8, that’s the html.html.twig, page.html.twig and node.html.twig templates; in Drupal 7, html.tpl.php, page.tpl.php, and node.tpl.php. The theme provides a subtheme aptly named the ExAMPle theme that demonstrates how to customize the appearance of AMP pages with custom styles. You could also create your own custom subtheme with your own styles. Then on the block layout page, you can place a variety of blocks into the regions for the AMP theme you’ve chosen.

That’s the big overview. Let’s dig into the details.

Installing the AMP module, theme and library

If you’ve used Drupal 8, you’re probably already familiar with Composer, a packaging tool for PHP that helps to install dependencies for a project. We’ll use Composer to install a PHP library that converts raw HTML into AMP HTML; Composer will also help to get that library working with Drupal. That’s true both for the Drupal 8 and the Drupal 7 versions of the AMP module.

The workflow we recommend involves using a Drupal module named Composer Manager. A composer.json file within the AMP module declares the dependency on the AMP PHP library. That library in turn has its own dependencies. Other modules that also have their own composer.json files could declare the same library dependencies. So Composer Manager takes care of going through all the modules installed on a site, collecting the info in their composer.json files, and bringing all of that information together so that multiple modules don’t end up installing the same dependencies that might conflict with each other.

However, the AMP module does not explicitly require Composer Manager as a dependency. Alternate workflows can make use of module Composer files without using Composer Manager. For example, Karen Stevenson provides an overview of how to use Composer to manage your entire site installation in her article, Goodbye Drush Make, Hello Composer!

If you are working with Drupal 7, you might find Composer Manager particularly useful. Drupal 8 has a built-in PHP autoloader. This allows you to take advantage of other PHP classes in a project without a require statement that specifies the exact file where that code is located. Composer Manager provides an autoloader for Drupal 7, so that the AMP module can make use of the code found within the AMP PHP library that Composer Manager installs.

Drush makes working with Composer Manager even easier. With Drush installed, several CLI commands, which we will cover below, can help to make use of Composer Manager. In addition, when you enable modules through Drush, Composer Manager can automatically install dependencies.

Whether you are working with a Drupal 7 or a Drupal 8 site in your local development environment, make sure you are using the most recent version of Drush 8, which now works with both Drupal 7 and Drupal 8 sites.

Let’s get ready to set up your Drupal site with AMP. With your local prepped, you can type the following terminal command within the directory where your Drupal site is located:

drush dl amp, amptheme, composer_manager

This will download the AMP module, the AMP theme and the Composer Manager module (if you don’t already have Composer Manager installed).

Next we’re going to enable the first items we’ll need to get started.

drush en composer_manager, amptheme, ampsubtheme_example

Notice we’re not enabling the AMP module yet! We want Composer Manager enabled and working before we do so. We’re also enabling an example subtheme included with AMP Theme: ‘ExAMPle subtheme.’ If you review the code in the subtheme, you can see how to set up custom styles for AMP pages. You can create your own subtheme if you like and install that instead. The default configuration for the AMP module sets the AMP Theme to ‘ExAMPle subtheme.’ So if you just want to try out the AMP module, enabling the example subtheme is a handy way to see how the module and theme work together to provide valid AMP HTML with a custom design.

You should now be able to enable the AMP module using Drush, and when you do so, Composer Manager will take care of downloading the AMP library and its required dependencies. Enabling the module in the UI does not trigger Composer Manager to install dependencies, so Drush is the preferred way to enable the AMP module.

drush en amp

You should get a message about dependencies being downloaded. If that doesn’t happen, then the modules should provide an error message about the AMP library not being available. Here are some things you can try for troubleshooting.

If you are using Drupal 8:

  • Go to the docroot for your site and enter composer drupal-update. If that fails, you may need to apply a patch from https://www.drupal.org/node/2405811 with the following command: curl https://www.drupal.org/files/issues/2664274-19-fix-composer.patch | patch -p1.

If you are using Drupal 7:

  • Go to the docroot for your site and enter drush composer json-rebuild followed by drush composer-manager install. Check /admin/config/system/composer-manager to verify everything is green. If so, try drush en amp again.

  • You may find it useful to go to sites/default/files/composer and enter composer install if the above doesn’t work. Then try drush en amp again.

Once the AMP library is installed, if you later need to update to a newer version, drush composer-manager update should take care of the update. Review the Composer Manager documentation for additional details.

You should now have the AMP module, AMP theme, AMP example subtheme and AMP library all installed and enabled. Next, we need to configure your site to make use of these new tools.

Configuring your site for AMP

Once everything is installed and enabled, you can find the AMP configuration page at /admin/config/content/amp. The first thing you’ll see is the list of your site’s content types. Take a minute to think about which content types should have AMP versions. It might not be all of your content types! News articles, blog posts and basic content-oriented pages are all great candidates for AMP.

View mode information on AMP configuration page in Drupal 8

To enable a particular content type for AMP, click on the “Enable AMP in Custom Display Settings” link. On the next page, open the “Custom Display Settings” fieldset. Check the AMP box, then press Save. This will bring you back to the main AMP config page. You’ll then want to press the corresponding “Configuring AMP view mode” link for that content type. On that page, you’ll be able to select which fields should display and the field formatters that each will use.

Custom display settings, found on the default display mode under the Manage Display tab for a content type in Drupal 8

The first thing you’ll want to do is change the body field to use the AMP Text field formatter. There are also custom AMP field formatters for images, videos (file fields) and iframes (text fields). More field formatters may be added in the future. This should help get you started. Some of the field formatters, like AMP Image, have additional options you might want to review by pressing the gear icon, which appears to the right of where you select that field’s formatter.

In Drupal 8, on the Manage Display tab for a content type, under the AMP view mode you can select AMP field formatters for each field

This combination of AMP view modes and field formatters is the heart of how your content is transformed into an AMP page. The AMP Text field formatters are particularly important in converting freeform text blocks that might contain embedded content. Some items like Twitter embeds will be automatically converted to the equivalent AMP components. Items with markup that is invalid for AMP, and which cannot be converted by the library into AMP markup, will be filtered out of the markup so that the page will validate as AMP.

Selecting an AMP theme

While the AMP module’s view modes and field formatters take care of the main content of the page, the AMP theme changes the markup outside the main content area of the page. The AMP theme also allows you to provide custom styles for your AMP pages. You will likely want to do that so that your AMP pages have branding similar to the non-AMP pages on your site.

On the main AMP config page, you’ll also want to make sure that the setting for the AMP theme is set to the ExAMPle Subtheme or your custom AMP subtheme. The AMP Base theme changes markup necessary for all AMP-based themes. In Drupal 8, AMP Base uses core’s Stable theme as its parent theme in order to keep the markup as lean and class-free as possible.

On the AMP configuration page, you can select the AMP theme

If you want to create a custom AMP subtheme with your own styles, take a look at how the ExAMPle subtheme does so. AMP does not allow custom JS, so the main thing you’ll be interested in is providing some custom styles and any markup changes necessary to support those styles. In Drupal 7, custom styles go in amp-custom-styles.css within the css directory at the root of the custom theme. In Drupal 8, the file where you’ll put your CSS is amp-custom-styles.html.twig, in the amp-css directory, which is inside the templates directory at the root of the custom theme. That’s because the CSS is embedded in the head of the html.html.twig file. In order for Drupal 8's Twig template engine to import code into a Twig template like html.html.twig, that code must be in a Twig template (and not another type of file) located in the templates directory of the theme.

Please note that the custom CSS has a maximum size of 50K! So this shouldn’t be all the styles for your entire site, but just whatever minimal styles are needed for AMP pages. If you want to use Sass to generate that CSS, you could definitely do so, as long as it compiles down to the one CSS file needed by the theme. Make sure to review the AMP Project guidelines for styling pages.

After you have the AMP theme selected, go to that theme’s block layout page. You will likely want to review block placement. You probably only need a few blocks: AMP pages are geared towards loading quickly and providing a great reading experience, so extensive menus for navigating the site may appear out of place (and could be difficult to implement without custom JS).

Select a limited number of blocks for your AMP theme on the block layout

Let’s check one last thing on the main AMP config page. There’s a checkbox near the bottom that enables full processing of the entire page’s markup through the AMP library. I highly recommend checking this box, as that helps to further ensure your page will be valid AMP HTML. If a module adds an attribute to some template somewhere, or if a block has an image that should be converted, this final pass through the AMP library will help to catch that.

Make sure to check the option to run the full page through the AMP library Analytics and ads

If you’re using Google Analytics, the AMP config page lets you enter your analytics ID so that basic Google Analytics tracking can be added to your AMP pages. The AMP specification also provides an AMP Pixel component that can be used for analytics in a way similar to how image tracking pixels work. At present, you can set up one AMP Pixel tracker by entering the domain name and URL path for the tracker. The random number tracking component is there as an example to see how this sort of tracking works. More robust AMP pixel tracking and additional analytics options may be available in the future.

The AMP configuration page offers options to configure ads and analytics for your AMP pages

You can also enter account information for Adsense and Double Click ad accounts on the AMP config page. Once you do so, if you return to the block layout page for your AMP theme, you can place ad blocks for those networks. In Drupal 7, you are currently limited to four ad blocks for each; Drupal 8 does not have a limit on the number of ad blocks. When you place each ad block, you’ll also need to enter the slot ID from your ad network of choice for that particular ad.

View AMP pages

Congratulations! Your site can now display AMP pages! If you are editing an item where its content type has an AMP view mode enabled, you will then have an option to save and view the AMP page. You can also add ‘?amp’ to the end of the URL for the page. If the URL for the page already has a query string, you can add ‘&amp’ to the end of the existing URL. Voilà! You can now see how your AMP page looks and decide if you want to make further changes.

If some of your content appears to be missing, the AMP library validator may have removed it from the final markup. All of your original content remains within your site: it just might not show up on an AMP page. To see what actions the validator has taken, add ‘&warnfix’ to the end of the page’s URL; warning messages will then display regarding any fixes the AMP library made in order for the page to validate as AMP HTML. There is also an option on the main AMP config page to show warning messages on all AMP pages.

You can also debug your AMP pages with in-browser developer tools like Chrome Dev Tools that have a console tab. The AMP JS validator shows error messages in the console tab that can help you determine if you need to address  a problem in order for your page to validate as AMP.

Our goal is for you to have flexibility and control with how your AMP pages display. That does mean a little work to get AMP set up on a Drupal site, but we think that’s time well spent. We want your AMP pages to look just the way you want. AMP pages can be beautiful as well as fast!

Why you should try out AMP

Now that you have AMP all set up, let’s talk about why it’s worth the effort.

The web could definitely use a jump start. Too many sites get bogged down with ads, trackers and other JS. AMP offers one way to help alleviate that pain. By clamping restrictions on sites and focusing strictly on the mobile web, AMP offers impressive results. Valid AMP pages with proper schema data get cached in Google’s CDN, and furthermore can be preloaded before you even get to the page. The end result feels like magic: click a link to an AMP page, and it appears before you in a flash. Some tests have shown AMP pages rendering four times faster than equivalent non-AMP pages, while using ten times less data. Those are pretty exciting results!

Google has rolled out carousels of AMP articles in their search results, and Twitter supports AMP now as well. I’d expect we’ll see more and more places that support AMP. That’s one of the big benefits of AMP being an open source project: anybody can create an implementation to show the AMP versions of pages.

As more sites and services make use of AMP pages, creating AMP versions of your content will be more and more key. Yet despite the opportunities AMP offers, I have heard some concerns about AMP, and I wanted to discuss them.

Isn’t this like m.example.com sites with separate content for mobile? Thankfully, with the solution we’ve worked on for Drupal, you don’t need to create two entirely different sites nor do you need to keep two sets of content in sync. You can simply provide a way for the content in your current site to be transformed into AMP HTML. That gives you the benefits of AMP’s speed without too much hassle. This also doesn’t rely on browser sniffing, like m. sites did. You don’t need need to detect if the URL request comes from a mobile device, which could be a huge pain. The site linking to yours determines whether they should provide a link to the AMP or non-AMP page.

Is AMP too prescriptive in how it tackles site speed? Tim Kadlec asks that question and more in his proposal to create Content Performance Policies as an alternative to AMP. CPP could be a standardized way to say which performance standards your site adheres to, and in theory a browser could enforce adherence to these standards. Tim speculates that AMP could end up being an implementation framework to adhere to a particular CPP standard.

Can’t sites just use good practices and be fast without AMP? I think it’s great to see all the energy and competition around ways to make sites faster. Front-end performance is something I’m passionate about because it’s one more way to make the web available to all, not just those with fast devices and unlimited data. Learning how browsers work can help you to better structure your HTML, CSS and JS, fonts and images in ways that make sites speedy. There’s nothing stopping anybody from using that knowledge to make a very fast site without using AMP, a fact Paul Bakaus from the AMP project acknowledged in “Life After AMP” and in this video about AMP.

That said, knowing performance best practices and putting them in place on media-rich publishing sites is another matter. Digital publishers are under a lot of pressure, and business needs often result in ads and analytics that serve business goals, but can result in slower page loads and sluggish scrolling. Even with solid work on front-end performance optimizations, sometimes there’s only so much you can do to speed up a site.

While in the future Content Performance Policies might encourage publishers to make fast sites on their own, right now publishers can get a super fast mobile site with AMP and see firsthand the results of improving user experience with that speed. AMP forces a site to adhere to some tight limits, putting the most problematic items for performance either off limits or in a controlled sandbox. I’ve been super impressed when I’ve seen real AMP pages in action through Google AMP article carousels. I think if publishers give this a try, they’ll be impressed too. There are AMP components for analytics, so publishers can get some real world data on how well this works.

My bet is that when somebody encounters an AMP page in the wild, they’ll be wowed by the speed, and they’ll be more likely to read the article and share it. If publishers see that engagement in their analytics, that will be real world tangible evidence of the importance of front-end performance on their own sites.

With that sort of data, publishers might decide to be more selective about the JS, trackers and other items that can slow down their main site. Maybe if Content Performance Policies become more than just a proposal, publishers will rework their entire sites to adhere to those emerging standards. Perhaps they set performance budgets for their sites as a whole. Those would all be good things.

No matter the size of your site, consider giving AMP a try. With the Drupal module and theme, as well as the AMP PHP library, there’s a good way to try out AMP with your site. With a little configuration and some CSS, you can have a fast mobile site that matches your branding. You can then test the metrics for visits to AMP pages versus non-AMP pages. You might just find that front-end performance could be a key feature to further pursue. So give it a shot!

Want to learn more about AMP and Drupal? Matthew Tift, Karen Stevenson and I will be presenting a session at DrupalCon New Orleans, “AMPing up Drupal.” We hope to see you there!

And with that, I’ll speed you on your way.

Photo by Anton Pinchuk, used under Creative Commons license.

Connecting Design Research to Your Sketches

When you are able to easily communicate how a design idea is directly inspired by your design research, you'll know that you are on the path towards creating a truly meaningful user experience. But creating meaningful experiences is not easy, and the difficulty arises in the awkward transition from conducting design research to sketching ideas.

The design research phase usually leaves us with many "bits" of data, such as what our audience values, what problems they are trying to solve, organizational goals from our client, or an opportunity to create more business value than a competitor. Sketching is a sacred opportunity in the design process where our brain can more easily make connections between disparate ideas by thinking visually and spatially. If we're going to find meaningful connections between research and our design work, sketching is the time to do it. Great designers know that the true beauty of a successful visual design is born through it's strength in concept, which is best considered separately from other layers of the design process (e.g., decisions on hierarchy, styles).

If we do not successfully connect our research to our sketches, the negative impact will be felt through the entire life of the project and beyond, even into future sales cycles. If we complete our visual designs and can't make connections back to the research, we're reinforcing the lack of value in research itself, creating a reputation for research as a process where we generate a lot of data that we don't use. We create a cycle where research is difficult to sell, because we are not regularly proving its value and ultimately, producing meaningful work is very difficult. At Lullabot, our hope is to break this cycle that is impacting the larger design services community and help research to be a lean, invaluable component to creating great work.

So, how do we learn to better connect the data from our research to the ideas on our sketchpads? Here is the key:

Create a short, focused list of research highlights that are only relevant to the user story you are sketching

Better connecting design research to our sketches begins with reducing our project-wide summary of research highlights (commonly referred to as “brief”) to a more focused list of key things that are only relevant to the user story we're designing for at this moment—our team calls this a user story brief.

The project-wide research brief is very useful for sharing with the larger product team to give insight into the design process (clients, developers, project manager, etc). But even a lean process can produce pretty big pile of research findings. When we begin the process of sketching ideas, we focus on one user story at a time. A user story is a high-level user flow that may involve multiple tasks or pages (e.g. sign up for an event, make a donation, learn how to make sweet potato brownies). If we're working on a site that's more content-driven and we understand the basic information architecture up front, we may sketch for particular page types (e.g. Article, Our Work, Contact). In any case, the general idea is that we're working to simplify that original project-wide research brief to focus on only highlighting research that is relevant to what we're sketching right now.

At Lullabot, we typically have a bucket of "go to" research highlights that we reach for to help form our user story brief before we begin sketching. Here are a few examples of those key components:

Project Vision Statement

A single sentence to focus the entire team on a common, meaningful goal for the project. It gives us a one-liner that serves as a star to sail our ship by. When ideas are being evaluated, disagreements arise, or compromises have to be made, it gives us one fundamental we can all agree on and evaluate against. Below are a few examples:

  • Connect and engage employees wherever they are with timely, relevant content.
  • Connect, inspire, and empower advocates for music making and education.

So, the first step when evaluating ideas for new content or experiences is to answer the questions above...how does this idea connect and engage employees wherever they are with timely, relevant content? Or how does this idea connect, inspire, and empower advocates for music making and education?

Design Principles

Guiding design principles are informed by all areas of the research process. We like to write principles with a simple, memorable, title and a description. We typically leave the description out when preparing for sketching and simply write the principles near our workstation, along with the project vision statement. Here are a select few principles we've written for our projects:

  • Relevance over breadth
    This principle makes us constantly ask “Does this audience type care about that component, that piece of content, that link?” as we’re evaluating ideas and designs.
  • Care for the newcomer
    Look for creative ways to provide tailored experiences for on-boarding employees. These ways may include prioritization decisions within the default designs, but also technical approaches for providing specialized experiences for the on-boarding process.
  • Show and tell stories
    Whenever and wherever possible, tell users about the Spredfast platform and products by telling real world stories, in the words of actual Spredfast customers. Find ways to show the value of something rather simply describing it.
Project Goals

Before sketching, re-read all project goals (e.g. business goals, marketing goals, design goals, technical goals). Are there any you'd like to keep front-of-mind for this user story design? Here are few examples of goals:

  • Increase donations significantly. An increase in donations would be one indication that the site is doing a better job of telling the story of the impact of the NAMM Foundation’s work. A site that truly “inspires and empowers” can play a role in increasing the number and significance of donors.
  • Increased site speed, particularly on mobile. Success: A 25-50% decrease in load time for the home page on mobile (the home page currently loads in 11.91 seconds)
  • Create a design that will appear on the short-list of best-designed entertainment and news web sites. Success: Submit Grammy.com as an entry to multiple awards organizations and receive a press-worthy award.
Persona Needs and Values

What personas are important to keep in mind for this user story? What are their common pain points? What are their key questions when they arrive at this page? What do they value? Below is an example of how we keep our personas text-based (no fake profile pictures) and simple.

Bob Bot the Lullabot:
Bob is a developer at Lullabot. He thinks that working for Lullabot is pretty awesome and loves the idea of showing off his company culture and work to the development community. Bob is fascinated by new delightful experiences on the web and appreciates design ingenuity.Bob Bot likes apps and websites that are intuitive, easy to use, clean & minimal with no distractions from the overall task. Apps and websites used regularly: Github, Reddit, Facebook, Twitter, Yammer, Amazon, Kindle, Readability, Unread, Feedly, New York Times, Google Inbox, Pocket, phpStorm

Visits Lullabot.com to: Review Lullabot.com redesign and share with dev community, read or post articles, listen to podcasts created by fellow bots, review his own bio on the who we are page, and read the new case study of a project he worked on. Main aspects that drew Bob Bot to work for Lullabot: Having great co-workers that are knowledgeable, passionate, talented and fun and having an outstanding reputation in the Drupal/web development community

Presentation Model

"Presentation model" is a fancy term for "here are ways we can present content on the page". For example, if we're redesigning a donation page, it may be clear to us that having a 1) donation form and 2) privacy policy links, etc are definitely "things that needs to be on this page".

Market research takeaways and other relevant research

Are there any notes you have on competitors, inspiration sites, Nielsen Norman reports, etc. that would be helpful to reference when sketching this story? Other ideas include: general design concerns or technical constraints presented by the client, google analytics highlights and key adjectives that represent the brand (Bold, Risky, Elegant).

Pulling together your user story brief

In summary, use the research highlight ideas mentioned above to create a simple brief that is specific to the user story you are sketching. The more simple your brief, the more likely you will be to connect your research to your sketch. Every user story brief will have a different structure because every sketch involves a unique combination of research "bits" that are relevant to only the specific problem you're solving. The key is to keep the user story brief as short and focused as possible. The more simple our list, the more likely we are to truly make strong connections from our research to our sketchpads.

Bring your research highlights into the same physical space as your sketchpad

Andrew Smyk with Adobe writes about a creative way that he creates new connections when sketching:

"One of my favorite techniques is to photocopy some of my sketches, cut them up into the base components, as described in Brad Frost’s Atomic Design, and then begin fleshing out the design concept by grouping and reshuffling components. I find that by physically moving the pieces of paper around, it allows me to identify patterns and think about how the user will interact with the design solution and interpret my messaging."

Let's apply the same idea of components and spatial thinking to our design research data. Chose a select few research highlights and write each highlight on a single notecard. Place those cards near your sketchpad while designing. Notecards are a designer’s best friend! The small space on a notecard helps to focus you on singular ideas. In order to more easily connect our research data to our ideas, we reduce our research data to singular components and bring them into the same physical space as our sketches. Now we can physically move the highlights around and make new connections with our sketched components, allowing you to reap the benefits of spatial thinking and learning. When you present your sketches, you should be able to easily communicate the research that inspired the idea by simply reading each notecard that is sitting next to your sketched component.

Summary
  • Research + sketches = more meaningful work, higher chance of success for the project
  • Before beginning to sketch, reduce your project-wide research findings to only the key highlights that are relevant to the user story or page you are sketching. The more simple the highlights list, the easier it will be to truly connect them to your ideas and communicate those connections to your team.
  • Try writing those research "bits" on individual notecards and placing them next to your sketchpad to reap the benefits of spatial thinking and learning.

Get a decorator for your Drupal home

Design patterns

Software design patterns are general and reusable software design solutions to a defined problem. They were popularized in 1994 by the “Gang of Four” after their book Design Patterns: Elements of Reusable Object-Oriented Software.

These generic solutions are not snippets of code, ready to be dropped in your project, nor a library that can be imported and reused. Instead they are a templated solution to the common software challenges in your project. Design patterns can also be seen as best practises when encountering an identified problem.

With Drupal 8’s leap into modern PHP, design patterns are going to be more and more relevant to us. The change from (mostly) procedural code to (a vast majority of) object oriented code is going to take the Drupal community at large through a journey of adaptation to the new programming paradigm. Quoting the aforementioned Design Patterns: Elements of Reusable Object-Oriented Software:

[…] Yet experienced object-oriented designers do make good designs. Meanwhile new designers are overwhelmed by the options available and tend to fall back on non-object-oriented techniques they've used before. It takes a long time for novices to learn what good object-oriented design is all about. Experienced designers evidently know something inexperienced ones don't. What is it?

Even if you don’t know what they are, you have probably been using design patterns by appealing to common sense. When you learn what they are you’ll be thinking “Oh! So that thing that I have been doing for a while is called an adapter!”. Having a label and knowing the correct definition will help you communicate better.

The decorator

Although there are many design patterns you can learn, today I want to focus in one of my favorites: the decorator.

The decorator pattern allows you to do unobtrusive behavior inheritance. We can have many of the benefits of inheritance and polymorphism without creating a new branch in the class’ inheritance tree. That sounds like a mouth full of words, but this concept is especially interesting in the Drupal galaxy.

In Drupal, when you are writing your code, you cannot possibly know what other modules your code will run with. Imagine that you are developing a feature that enhances the entity.manager service, and you decide to swap core’s entity.manager service by your enhanced version. Now when Drupal uses the manager, it will execute your manager, which extends core’s manager and overrides some methods to add your improvements. The problem arises when there is another module that does the same thing. In that situation either you are replacing that module’s spiced entity manager or that module is replacing your improved alternative. There is no way to get both improvements at the same time.

This is the type of situations where the decorator pattern comes handy. You cannot have this new module inheriting from your manager, because you don’t know if all site builders want both modules enabled at the same time. Besides, there could be an unknown number of modules –even ones that are not written yet– that may create the conflict again and again.

Using the decorator

In order to create our decorator we’ll make use of the interface of the object we want to decorate. In this case it is EntityManagerInterface. The other key component is the object that we are decorating, let’s call it the subject. In this case our subject will be the existing object in the entity.manager service.

Take for instance a decorator that does some logging every time the getStorage() method is invoked, for debugging purposes. To create a decorator you need to create a pristine class that implements the interface, and receives the subject.

class DebuggableEntityManager implements EntityManagerInterface { protected $subject; public function __construct(EntityManagerInterface $subject) { $this->subject = $subject; } }

The key concept for a decorator is that we are going to delegate all method calls to the subject, except the ones we want to override.

class DebuggableEntityManager implements EntityManagerInterface { protected $subject; // ... public function getViewModeOptions($entity_type_id) { return $this->subject->getViewModeOptions($entity_type_id); } // … public function getStorage($entity_type) { // We want to add our custom code here and then call the “parent” method. $this->myDebugMethod($entity_type); // Now we use the subject to get the actual storage. return $this->subject->getStorage($entity_type); } }

As you have probably guessed, the subject can be the default entity manager, that other module’s spiced entity manager, etc. In general you’ll take whatever the entity.manager service holds. You can use any object that implements the EntityManagerInterface.

Another nice feature is that you can decorate an already decorated object. That allows you to have multiple decorators adding different features without changing the inheritance. You can now have a decorator that adds logging to every method the entity manager executes, on top of a decorator that adds extra definitions when calling getFieldDefinitions(), on top of …

I like the coffee making example in the decorator pattern entry in Wikipedia, even if it’s written in Java instead of PHP. It’s a simple example of the use of decorators and it reminds me of delicious coffee.

Benefits and downsides

One of the downsides of using the decorator pattern is that you can only act on public methods. Since you are –intentionally– not extending the class of the decorated object, you don’t have access to any private or protected properties and methods. There are a couple of situations similar to this one where the decorator pattern may not be the best match.

The business logic you want to override is contained in a protected method, and that method is reused in several places. In this case you would end up overriding all the public methods where that protected one is called.

You are overriding a public method that is executed in other public methods. In such scenario you would not want to delegate the execution to the subject, because in that delegation your overridden public method would be missed.

If you don’t find yourself in one of those situations, you’ll discover that the decorator has other additional benefits:

  • It favors the single responsibility principle.
  • It allows you to do the decoration during run-time, whereas subclassing can only be done in compile-time.
  • Since the decorator pattern is one of the commonly known design patterns, you will not have to thoroughly describe your implementation approach during the daily scrum. Instead you can be more precise and just say “I’m going to solve the problem using the decorator pattern. Tada!”.
Write better designs

Design patterns are a great way to solve many complex technical problems. They are a heavily tested and discussed topic with lots of examples and documentation in many programming languages. That does not imply that they are your new golden hammer, but a very solid source of inspiration.

In particular, the decorator pattern allows you to add features to an object at run-time while maintaining the object’s interface, thus making it compatible with the rest of the code without a single change.

Tugboat: A Visual Website QA Tool for Enterprise Teams

Matt and Mike sit down with a number of Lullabots to talk about Tugboat, which allows you to "review every feature or bug fix with automatic builds." In developer-speak, it's an easy-to-use, multi-platform pull-request environment generation tool. Tugboat is one of the ingredients of Lullabot's secret sauce, and it's awesome!

The AMP Module for Drupal 8

The mobile web can be an experience that leaves a lot to be desired for the end user. In January 2016, Google and Lullabot started working together on a Drupal 8 module to support Accelerated Mobile Pages, or AMP Project, an open-source initiative to help make the mobile web faster and more user-friendly. For publishers, AMP helps create content that is optimized for mobile, but also loads instantly, virtually anywhere.

One of the most touted features of Drupal software is its flexibility, so making Drupal produce AMP HTML required a lot of careful consideration with regard to its design approach. In a recent article, Matthew Tift, a senior developer here at Lullabot, discusses why this open-source project is so important to the future of the web, and what to expect from the Drupal 8 AMP module we created. 


The team who worked on the AMP Project for Drupal 8 (a Drupal 7 module is forthcoming) also submitted a proposal to speak about the project at DrupalCon 2016.
 

UX for Brains: Let’s Be Honest, People Suck

I really love psychology, and learning about what makes people tick. I even debated psychology as a possible career, but settled for a minor in order to focus on my design obsession full time. Post graduation, I thought I‘d left that psychologist’s hat behind me. But like many recent grads, I was wrong. Now, as a user experience designer, I bring these foundations into my work constantly. If you're a designer or work in the web, what I'm about to share are some basic ways to leverage psychology in your work.

Understand Psychology To Understand People

Using a psychological perspective to approach design problems isn’t meant to side-step UX practices, but actually enhance them. I think of it as adding dimension to your practices and teams, by allowing you to see out of two eyes, instead of just one.

As a designer in this ever-evolving technical world, it is essential to have a core understanding of the fundamentals of human perception, cognition, and behavior, especially as they apply to how humans use websites. Utilize psychology as part of your own design practice to better tap into your users. There are volumes out there—not in the design section—full of timeless explanations of what motivates people, what their needs are, how they complete tasks, and how they think...or, you know, don’t. And as Donald Norman—the cognitive psychologist who originally coined the term UX—puts it in his must-read design bible, Design for Everyday Things:

The one thing I can predict with certainty is that the principles of human psychology remain the same, which means that the design principles here, based on psychology, on the nature of human cognition, emotion, action, and interaction with the world, will remain unchanged. Donald Norman, The Design of Everyday Things: Revised and Expanded Edition
People Aren't Always The Users We Dream Of

Ok, I admit, I have an underlying reason for pushing you all to learn about people through psychology. There is one observation that I have made in particular that I would hope for more designers to discover for themselves. And no matter how I try to sugarcoat it, or back it up with study findings, everyone thinks I’m a bummer when I share it. Simply, there are some things about human bodies, minds, and habits that kind of suck. Look at some of my examples, then decide for yourself how much credit we should give our users.

Before I jump into the quantitative statistics, let me ease into the fun with a qualitative persona. This is a culmination of users I’ve tested, all rolled up into my grandmother. I pray she never finds this article.

Think of this persona while you work, and consider how she might struggle with an interaction.

Elderly Eleanor is an older woman and mother of eight, who first experienced computers in the DOS era. Eleanor wrote technology off early after making some irreversible mistakes. And the poor dear had no Google, Siri, or user-centric customer support teams to help, so she gave up until her family forcibly gifted technology back into her life. Now, she has a PC and an iPad. She uses them to look at weather predictions (Maine gets quite a bit of snow) and to stay connected with her family through social media. But she doesn’t really understand how it works, and often points her camera way above her head during Skype calls. She marvels when a family member buys something online. But if they try to explain it, she will cut them off to say that it’s beyond her.

I know that Eleanor is not our standard user. But keep her in mind as you design your next experience, as the ultimate usability test. WWED?

The Technology Adoption Cycle shows us how rare our innovators and early adopters are.

While Eleanor’s aversion to technology is not the norm, it’s not too far off; in fact, an aversion to technology is mainstream, as we can see in a standard technology adoption lifecycle. This trend can be easy to overlook when we are the innovators building the technology, but the adventurous innovators and adopters are rare. The rest of humanity is not nearly so fearless, and will wait a long time before trying something new.

And They’re Needy

Not only do people interact with our experiences in less than ideal ways—more on that to follow—but they also have less than ideal expectations. People need a lot, and expect even more. If you need a refresher on the hierarchy of needs, check out Abraham Maslow’s pyramid: the gist is that people seek to meet their most basic needs first, like food and safety. As soon as these are met, instead of feeling satisfaction, now they need the next thing, and the next, in an endless and frustrating adaptation cycle known as the hedonic treadmill.

From Smashing Magazine, this pyramid suggests the hierarchy of design needs, with functionality as our primary focus.

I like this Smashing Magazine spin-off pyramid for a design hierarchy of needs. It follows the same bottom-up process, but with functionality at the bottom and creativity at the top. This means that people need a working product well before they need a beautiful one, so our priorities as a design team should follow this hierarchy. Another interesting observation here is the modest value provided by each of these levels. It’s frustrating, isn’t it? So much of our hard work to launch a working product is taken for granted. 

Take a deep breath before we look at the next chart.

The Kano Model is a theory of customer satisfaction developed in 1984 by Professor Noriaki Kano.

The Kano model was developed in the 80’s, but is still a classic. Note the trends of basic needs (must-haves), performance needs (wants), and latent needs (delighters). First of all, the must-haves are completely below the threshold of satisfaction, so it’s not enough to meet users’ basic needs. As far as they’re concerned, it’s the bare minimum that an app lets them sign up, keeps their information saved but private, doesn’t crash, and on and on and on. Wants can go either way, and some people will appreciate these extra thoughtful features and interactions. Users will delight at thoughtful experiences that meet their higher emotional and intellectual needs, but this can be a tall order. Since these latent needs are the ones users struggle most to articulate. I quote and re-quote Henry Ford on this, the manufacturer of the first automobiles: “If I had asked people what they wanted, they would have said faster horses.”

Humans Miss A LOT Of Goings-On

One of the hardest steps when people interact with something you’ve made is the first one: getting users to discover your hard work. I really like information architect Steve Krug’s way of describing the situation:

We’re thinking “great literature”... the user’s reality is much closer to “billboard going by at 60 miles an hour”. Steve Krug, Don't Make Me Think

So much for that thoughtfully-crafted experience. As you’ll see, humans have tons of mechanisms built in to protect them from actually caring about stuff. Let’s start with our eyes.

Our cone cells produce a small, precise foveal area, while our rod cells fill in the rest with our blurry peripheral vision.

In the center of our eyeballs, our cone receptor cells capture light with quality and focus. The rod cells around the rest of the back of the eye are lazier, and can’t distinguish things like color or shape all that well (but they shine when it comes to night vision). The cones have a lot of heavy lifting to do, but sadly, they only create an area of focused vision no bigger than your thumbnail, known as your foveal area. The rest of the world (and your website) is getting seen in our blurry peripheral vision, but we don’t notice because our eyes—and foveal areas— are constantly moving to fill in the gaps. Not too surprising that a lot of things get overlooked.

The next step requires keeping our attention, which is borderline goldfish status. As I’m sure you know, humans are overwhelmed with a firehose of information. We divide our attention between many distractions, and as a result, have a hard time focusing on anything. It doesn’t take long to lose focus: almost half of users give up if a page takes three or more seconds to load. Our attention spans are just minutes long—how many minutes exactly is debatable because it’s getting shorter all the time. These minutes are broken up by distractions too, so you can really only expect about eight focused seconds of unbroken attention to a task at a time. Eight!

And it’s way easier for a user to get distracted while using your product at home, online, alone (as compared to supervised or incentivized test subjects). Even though it’s fiction, I think this Portlandia bit does a nice job of telling the story. Point being, take that aspect of your usability tests with a grain of salt. For example—this one always cracks me up—think about the last time you went into a retail store, got a cart, filled it up with stuff, and then got bored and left that cart in an aisle and walked out. Hopefully you can’t, because who does that? And yet online cart abandonment is 67 percent!

With all the distractions we are already faced with, users are averse to anything that could be mistaken for a distraction. We have to seriously commit ourselves to getting from point A (our desk chairs) to point B (purchasing those picture frames from Target’s online store), so any pop ups are probably going to get killed immediately. Even that helpful overlay with 15% off brushed silver frames gets the X, because we’d made the snap judgement to terminate it before reading and comprehending any value. While I get that we’re all busy, I would argue that this behavior goes deeper than that. Often designers come up with unique, novel interactions that require learning unique, branded terminology, and I usually end up hating it. Because I don’t want to learn some crazy new mobile gesture if I came to your app to achieve a goal that wasn’t entertainment. Again, A to B.

UCLA psychologists asked 85 undergrads to draw the Apple logo from memory, and only one got it right. From Gizmodo.

Going back to the example: let’s say you’re giving your full focus to the picture frame hunt, because you are going to get that Christmas shopping done early this year, gosh darn it. Even when you think you’re giving something your full attention, your brain is doing you a solid and ignoring most of the thing. People learn as little as they can and instead rely on salient cues to get by. A recent study discussed on Gizmodo leveraged a popular brand to illustrate this concept: when participants were asked to draw the Apple logo from memory, most of them weren’t even close. At least they’re all apples?

This heatmap shows us how users skim past carousels, due to an evolved banner blindness. From ConversionXL.

Heatmaps provide an even more relevant example, showing us how humans skim down the left side of a webpage in an F-shaped pattern, without processing everything. Note how people are getting accustomed to ads and tuning them out. But you’ve probably heard of banner blindness; the exciting twist is that this has mutated so that people ignore anything that even looks like an ad. Like, say, carousels: turns out hero image carousels generate the same blindness, and consequently waste a lot of important space. So please don’t use them!

There is another wrinkle to this content discovery hurdle, which usability consultant Jakob Nielsen sums up succinctly in his Nielsen Norman Group article:

“How users read on the web: they don’t”. Jakob Nielsen

Humans are wired for language, but specifically for speaking it, and not for reading it. Evolution hasn’t had nearly enough time to catch up to the Gutenberg press, and so we skim. Users at most have the time to read around 20% of the words on a webpage during an average visit. And even if we do read something, we will probably forget it, because, spoiler alert, our memories are garbage.

Even More Minefields There are tons of pitfalls that users can struggle with over the different steps of human interactions: discovery, thinking, and acting/decision-making.

The section above outlined the pitfalls of discovery, which is just the first step in human interactions. There are tons of other problems that arise in the steps that follow: thinking, interacting, and decision-making. I will spare you the novel and give you the highlights: people come from different backgrounds, cultures, and places, and often misinterpret the message; they create and reinforce incorrect mental models of the world; they’re irrational, often making emotionally-biased or too-quick snap decisions instead of thoughtful judgments; they make all kinds of mistakes, from motor malfunctions and slips to memory biases and full-on gaps.

Now What?

To be clear, I’m not telling you all of this so you feel bad, and then try to correct your own behavior. I’m telling you so you can accept these limitations, expect them from your users, and adapt as a designer. There is hope. In my next article, I will outline suggestions for enhancing usability and reducing friction around some of these struggles. But I want to touch on the biggest challenge in the meantime, so you can start adapting your design practices: stop designing experiences for us, for the “interactive 1%”. Before, I asked you to think about how much credit you give users. I think often the truth is that we give our users far too much credit. Maybe we assume people will do the right thing, or that they’ll understand if they just take time to read through the page and figure it out. I don’t think this is doing our users any favors. We’ve seen that they can’t be expected to process something the same way that we do, and they won't keep trying for more than a few seconds.

It’s easy to misjudge intricate interaction patterns as common sense because we are extremely familiar with them. Even basic interactions can be difficult to understand for “pedestrians”—this was a term an old design professor used to describe any non-designer. The biggest mistake designers make, in my humble opinion, is leaning too heavily on design trends without checking for visual communication basics. Designers are taking the signifiers out of interactions more and more, in the name of aesthetic value, but at the cost of hurting usability and adding unnecessary confusion. I know flat design is sexy and everything, but make sure your buttons actually look like clickable buttons before—and after—you ship.

Do your due diligence and make sure you aren’t alienating anyone before jumping into design. Do some user research to understand your users’ needs as best you can, prioritize those needs, then start with the basics. Conduct usability tests to understand how users try to meet those needs, making sure your designs serve the user's purpose and fade into the background.

Finally, remember that it’s a mistake to assume that our users’ needs and expectations will be much like our own, and simply design for ourselves. Psychology offers us valuable insights towards what users actually need, and a little effort toward understanding our users can go a long way towards creating more successful designs.

Who's Who in VR

Some of us at Lullabot have been following the Virtual Reality and WebVR scene for a couple of years now. It’s a rich landscape with a lot of interest being thrown into the ring and attracting some high stakes players in the industry. We thought we’d share with you who some of the top hardware companies are in this space and what they’re building to advance the future of VR.

Facebook’s Oculus, and Samsung

Facebook bought Oculus for $2b in March of 2014. Oculus makes a tethered headset called the Rift which uses a Samsung OLED screen, headphones, Xbox and remote controllers, and positional tracking. They also have hand-tracked controllers which will be sold separately.

The Rift has had more than a few versions so far. The Development Kit 1 and 2 (commonly called the DK1 and DK2) were limited in production and sold to developers for building experiences. There were also some prototypes shown off but not released such as the HD prototype, Crystal Cove, and Crescent Bay. The consumer version (shown above) is selling for $599 and will be shipping to consumers in the first and second quarter of this year. That doesn’t include the high-end computer you’ll also need to run this technical marvel.

Samsung partnered with Oculus to produce a mobile headset called the GearVR which uses about five possible Samsung phone models as the screen and processing for apps bought through the Oculus Store which you pay for through Samsung Pay.

The GearVR is the most highly polished VR experience on the market. Samsung’s Oculus store is really locked down the way Apple’s app store is, and you can only use a few different Samsung phones (Note 4, Note 5, S6, S6 Edge and S6 Edge plus) but this helps ensure a superior VR experience from the moment you snap your phone into the lightweight headset. Headsets sell for $100USD and the phones can run you from somewhere around $400 to $800. Still the best experience for under $1000.

Valve's SteamVR, and HTC

Valve is the maker of the SteamVR software platform. HTC partnered with Valve to produce the tethered headset called the HCT Vive which uses an HTC screen, hand controllers, positional tracking, and a front facing camera.

The Vive’s big differentiator is what it calls “Room Scale” VR. There are two laser position sensors which can track an area of up to 15x15 ft (4.5 meter) and capture the user, allowing you to move around within a virtual space. You can see what I mean by watching this video of Fantastic Contraption, which is being built for the Vive.

Again, being a tethered headset means that along with buying the headset you also need a high-end computer to run games and applications at that magical 90fps smoothness.

Google’s Cardboard, and Magic Leap

Google makes the software and a very inexpensive open source mobile headset called Cardboard to achieve the dual rendering on any Android or iOS phone. Just fold your cardboard like a puzzle, insert the lenses, download the software, and insert your phone.

While Google Cardboard is making VR as cheap as possible and therefore accessible to the widest range of consumers, the experience (while amazing) is still sub-par to that of many others. But then if you haven’t experienced the other solutions, you’re not really going to know the difference anyway, so you really have nothing to lose in trying out Cardboard.

Magic Leap is an Augmented Reality company whom Google invests in which supposedly uses retinal projection (no one has yet seen this device) to overlay digital images over the real world.

They have released a few videos, and they’ve hired Neal Stephenson, the author of the infamous Snow Crash as their “Chief Futurist”. Google invested $542m USD in 2014. Total funding is somewhere around $1.4b USD and the post-money valuation on Magic Leap is around $4.5b USD. Needless to say the people who have seen this technology are being wowed, but the secrecy around this may also be partly hype. We just don’t know, but we’re very hopeful.

Google has also reportedly created a department for VR and there are several job listings now that mention VR in them.

Sony’s PlayStation VR

Sony is building VR for their Playstation 4 platform called PlayStation VR which uses a tethered headset, positional tracking, and hand controllers connected to their video game console.

PlayStation VR has been said to be working on 100s of game titles and is of course, mostly focused on the gaming market. They had an earlier prototype you may have heard of, Project Morpheus. The consumer-grade hardware should launch in the first half of 2016 and with many people already owning a PS4, they have a great market and a leg up on those who still need to buy a high-end computer.

These are some of the major players. Others that are worth noting are:

Friendly reminder: this is by no means a comprehensive list, just some of the top players.

It’s also being speculated that Apple is jumping into VR/AR space as well in that it holds several patents related to VR, has hired a human-computer interaction researcher who specializes in 3D interfaces, and hired a former Microsoft employee who worked on the HoloLens. Read some more here for other evidence.

AMPing up Drupal

Today we are proud to announce a new Drupal 8 module that provides support for the Accelerated Mobile Pages (AMP) Project. The AMP Project is a new open source initiative, led by Google, which drastically improves the performance of the mobile web. In January 2016 Lullabot and Google started working together to create the Drupal AMP module. A beta version of AMP module for Drupal 8 is available immediately, and we are starting work on a Drupal 7 version of the module that will be available in mid-March.

In many cases, the mobile web is a slow and frustrating experience. The AMP Project is an open source initiative that embodies the vision that publishers can create mobile optimized content once and have it load instantly everywhere. When AMP was first introduced last October, many commentators immediately compared it to Facebook's Instant Articles and Apple's News app. One of the biggest differentiators between AMP and other solutions is the fact that AMP is open source.

AMP HTML is, essentially, a subset of HTML. And it really makes the web fast. AMP HTML is designed to support smart caching, predictable performance, and modern, beautiful mobile content. Since AMP HTML is built on existing web technologies, publishers continue to host their own content, craft their own user experiences, and flexibly integrate their advertising and business models -- all using familiar tools, which will now include Drupal!

One of the most touted features of Drupal is its flexibility, so making Drupal produce AMP HTML has required a lot of careful consideration of the design approach. To make Drupal output AMP HTML, we have created an AMP module, AMP theme, and a PHP Library.

When the AMP module is installed, AMP can be enabled for any content type. At that point, a new AMP view mode is created for that content type, and AMP content becomes available on URLs such as node/1/amp or node/article-title/amp. We also created special AMP formatters for text, image, and video fields.

The AMP theme is designed to produce the very specific markup that the AMP HTML standard requires. The AMP theme is triggered for any node delivered on an /amp path. As with any Drupal theme, the AMP theme can be extended using a subtheme, allowing publishers as much flexibility as they need to customize how AMP pages are displayed. This also makes it possible to do things like place AMP ad blocks on the AMP page using Drupal's block system.

The PHP Library analyzes HTML entered by users into rich text fields and reports issues that might make the HTML non-compliant with the AMP standard. The library does its best to make corrections to the HTML, where possible, and automatically converts images and iframes into their AMP HTML equivalents. More automatic conversions will be available in the future. The PHP Library is CMS agnostic, designed so that it can be used by both the Drupal 8 and Drupal 7 versions of the Drupal module, as well as by non-Drupal PHP projects.

We have done our best to make this solution as turnkey as possible, but the module, in its current state, is not feature complete. At this point only node pages can be converted to AMP HTML. The initial module supports AMP HTML tags such as amp-ad, amp-pixel, amp-img, amp-video, amp-analytics, and amp-iframe, but we plan to add support for more of the extended components in the near future. For now the module supports Google Analytics, AdSense, and DoubleClick for Publisher ad networks, but additional network support is forthcoming.

While AMP HTML is already being served by some of the biggest publishers and platforms in the world — such as The New York Times, The Economist, The Guardian, BBC, Vox Media, LinkedIn, Pinterest, and many more! — you don't have to be a large publisher to take advantage of AMP. Today, any Drupal 8 site can output AMP HTML using the AMP module. We invite you to try it out and let us know what you think!

Finally, if you are interested in this topic and want to learn more about publishing AMP with Drupal, please leave a comment on our DrupalCon New Orleans proposal.

New and shiny vs Good old software

«It depends». That may be the most used sentence when evaluating the correct technology for a generic challenge. Choosing the right technology is not a trivial problem. When the needs are poorly defined, or not defined at all, it may not even be possible to suggest a good solution. But even if the problem is broken down into simple technical challenges, there are multiple factors that influence the implementation of the solution. Budget constraints, internal knowledge, timelines, resourcing issues, corporate policies, integration with existing systems, technical feasibility, and a long etcetera will influence the final decision.

Leaving these other—extremely valid—considerations aside, let’s focus on the technical part. Often times you have different frameworks, applications, libraries, SAAS, … to provide a solution with different degrees of success. The sheer size of this list of options can complicate the decision. It is common practice to check what everyone else is doing. Doing market research is a good first step, but it should not determine the final decision.

Many times when a new technology arises, there is media hype. All of a sudden many computer science related blogs and news sites are flooded with posts explaining the merits of this groundbreaking new solution. Often times, these communications are very passionate and contagious, and that’s good! Their goal is to get other people to try the new tools. Information flows rapidly and allows people that are solving that particular problem to assess if they want to try a new direction.

This situation can result in what I call the shiny effect. I admit that I have been blinded by it in the past. I have spent large amounts of time learning frameworks that turned out to be great for only a narrow set of use cases. That has made me cautious about how I approach new tools, even though I still get excited about them.

One widely used argument is based on authority. (If all these multi-milion dollar organizations are using this solution for the exact same problem I have, then I should do the same. After all, they would never choose to implement it without careful analysis.) Resorting to the argument from authority will not always lead you to a reliable answer. The market trend will not always be accurate, there are multiple examples of that.

I remember how in a not so distant past, NoSQL databases were postulated by some people as the future, some people went so far as to say that SQL was dead. There was a time when Apache was to be completely usurped by other alternatives. The LAMP stack was supposed to be replaced by the MEAN stack, without leaving any trace. SOAP and RPC were to disappear because of REST, while REST is now irrelevant because of GraphQL. Also, at the present time no one was supposed to be using Basic Auth. It goes without saying that the opposite happened as well: prophecies of emerging solutions being doomed to disappear that were never fulfilled.

None of that happened. All the tools and technologies listed above have found a way to coexist and share the spectrum of solutions. Even when they seemed to overlap at the beginning, the emerging alternatives ended up being only a better solution for a subset of scenarios in most of the cases. That is a very big win for everyone. We now have two different tools that are very good at solving similar problems. Even if you can build a search backend in MySQL, you are probably going to have a better experience if you do it with Solr, for instance.

Disregarding well-proven technological solutions, with proliferous ecosystems, is dangerous. Failing to know new solutions that may prove to be better at solving your problem is just as dangerous. We really need to take the time to understand the problem that the new solution is fixing before jumping into it with both feet. Carey Flichel attributes this, amongst other causes, to boredom and lack of understanding in Why Developers Keep Making Bad Technology Choices.

It’s no surprise then that we want to try something new, even if we’ve adequately solved a problem before. We are natural puzzle solvers, and sometimes you just want to try a new puzzle.

Choosing the right solution often involves checking what everyone else is doing, and then analyzing the problem for yourself while taking all the options into consideration. You must not trust new solutions just because they're new anymore than you trust old solutions just because they're old. Instead, zero in on the problem to be solved and find the perfect solution regardless of the buzz. Keep every technology solution on the table until you understand the nuances of the problem space, and let that be your guiding light. Being aware of all the technologies involved, and knowing what’s the best choice, takes time and a lot of research. Many times this will require a software architect to guide you.

Javascript aggregation in Drupal 7

Javascript aggregation in Drupal 7

What is it? Why should we care?

Javascript aggregation in Drupal is just what it sounds like: it aggregates Javascript files that are added during a page request into a single file. Modules and themes add Javascript using Drupal’s API, and the Javascript aggregation system takes care of aggregating all of that Javascript into one or more files. Drupal does this in order to cut down on the number of HTTP requests needed to load a page. Fewer HTTP requests is generally better for front-end performance.

In this article we’ll take a look at the API for adding Javascript, paying specific attention to the options affecting aggregation in order to make best use of the system. We’ll also take a look at some common pitfalls to look out for and how you can avoid them using Advanced Aggregation (AdvAgg) module. This article will mostly be looking at Drupal 7, however much of what we’ll cover applies equally to Drupal 8, and we’ll take a look specifically at what’s changed in Drupal 8.

If fewer is better, why multiple aggregates?

Before we dig in, I said that fewer HTTP requests is generally better for front-end performance. You may have noticed that Drupal creates more that one Javascript aggregate for each page request. Why not a single aggregate? The reason Drupal does this is to take advantage of browser caching. If you had a single aggregate with all the Javascript on the page, any difference in the aggregate from page to page would cause the browser to download a different aggregate with a lot of the same code. This situation arises when Javascript is conditionally added to the page, as is the case with many modules and themes. Ideally, we would group Javascript that appears on every page apart from conditionally added Javascript. That way the browser would download the every page Javascript aggregate once and subsequently load it from cache on other page requests on your website. Drupal’s Javascript aggregation attempts to give you the ability to do exactly that, striking a balance between making fewer HTTP requests, and leveraging browser caching.

The Basics

Let’s take a step back and briefly go over how to use Javascript aggregation in Drupal 7. You can turn on Javascript aggregation by going to Site Configuration > Development > Performance in your Drupal admin. Check off "Aggregate Javascript files" and Save. That’s it. Drupal will start aggregating module and theme Javascript from there on out.

Using the API

Once you’ve enabled Javascript aggregation, great! You’re on your way to sweet, sweet performance gains. But you’ve got modules and themes to build, and Javascript to go along with them. How do you make use of Javascript aggregation provided by Drupal? What kind of control do you have over the process?

There are a couple API entry points for adding Javascript to the page. Most API entry points have parameters that let you tell Drupal how to aggregate your Javascript. Let’s look at each in turn.

drupal_add_js

This is a loaded function that is used to add a Javascript file, externally hosted Javascript, a setting, or inline Javascript to the page. When it comes to Javascript aggregation, only Javascript files are aggregated by Drupal, so we’re going to focus on adding files. The other types (inline and external) do have important impacts on Javascript aggregation though, which we’ll get into later.

hook_library/drupal_add_library

Drupal also provides a mechanism to register Javascript/CSS "libraries" associated with a module. This is nice because you can specify the options to pass to drupal_add_js in one place. That gives you the flexibility to call drupal_add_library in multiple places, without having to repeat the same options. The other nice thing is that you can specify dependencies for your library and Drupal will take care of resolving those dependencies and putting them in the right order for you.

I like to register every script in my module as a library, and only add scripts using drupal_add_library (or the corresponding #attached method, see below). This way I’m clearly expressing any dependencies and in the case where my script is added in more than one place, the options I give the aggregation system are all in one place in case I need to make a change. It’s also a nice way to safely deprecate scripts. If all the dependencies are spelled out, you can be assured that removing a script won’t break things.

Instructions for using hook_library and drupal_add_library are well documented. However, one important thing to note is the third parameter to drupal_add_library, every_page, which is used to help optimize aggregation. At registration time in hook_library, you typically don’t know if a library will be used on every page request or not. Especially if you’re registering libraries for other module authors to use. This is why drupal_add_library has an every_page parameter. If you’re adding a library unconditionally on every page, be sure to set every_page parameter to TRUE so that Drupal will group your script into a stable aggregate with other scripts that are added to every page. We’ll discuss every_page in more detail below.

#attached

attached is a render array property that lets you add Javascript, CSS, libraries and arbitrary data to the page. #attached is the preferred approach for adding Javascript in Drupal. In fact, drupal_add_js and drupal_add_library are gone in Drupal 8 in favor of #attached. #attached is nice since it is part of a render array and can be easily altered by other modules and themes. It’s also essential for places where render arrays are cached. With caching in play, functions that build out a render array are bypassed in favor of a cached copy. In that scenario, you need to have your Javascript addition included in #attached on the render array, because a call to drupal_add_js/drupal_add_library in your builder function would otherwise be missed.

Drupal developers often fall back on drupal_add_js for lack of a render array in context. It’s common to add Javascript in places like hook_init, preprocess and process functions, where there is no render array available to add your Javascript via #attached. In these scenarios, consider whether your Javascript is more appropriately added via a hook_page_build, or hook_node_view, where there is a render array available to add your script via #attached. Not only will you begin to align yourself with the Drupal 8 way of doing things, but you open yourself up to being able to use the render_cache module, which can gain you some performance. See the documentation for drupal_process_attached for more information.

Controlling aggregation

Let’s take a close look at the parameters affecting Javascript aggregation. Drupal will create an aggregate for each unique combination of the scope, group, and every_page parameters, so it’s important to understand what each of these mean.

scope

The possible values for scope are ‘header’ and ‘footer’. A scope of ‘header’ will output the script in the

tag, a scope of ‘footer’ will output the script just before the closing tag.

group

The ‘group’ option takes any integer as a value, although you should stick to the constants JS_LIBRARY, JS_DEFAULT and JS_THEME to avoid excessive numbers of aggregates and follow convention.

every_page

The ‘every_page’ option expects a boolean. This is a way to tell the aggregation system that your script is guaranteed to be included on every page. This flag is commonly overlooked, but is very important. Any scripts that are added on every page should be included in a stable aggregate group, one that is the same from page to page, such that the browser can make good use of caching. The every_page parameter is used for exactly that purpose. Miss out on using every_page and your script is apt to be lumped into a volatile aggregate that changes from page to page, forcing the browser to download your code anew on each page request.

What can go wrong

There are a few problems to watch out for when it comes to Javascript aggregation. Let’s look at some in turn.

Potential number of aggregates

The astute reader will have noticed that there is a potential for a lot of aggregates to be created. When you combine scope, group and every_page options, the number of aggregates per page can get out of control if you’re not careful. With two values for scope, three for groups and two for every_page, there is potential to climb to twelve aggregates in a page. Twelve seems a little out of control. Sure, we want to leverage browser caching, but this many HTTP requests is concerning. If you’re getting high into the number of aggregates it’s likely that modules and themes adding Javascript are not using the API as best as they could.

Misuse of Groups

As was mentioned, JS_LIBRARY, JS_DEFAULT and JS_THEME are default constants that ship with Drupal to group Javascript. When adding Javascript, modules and themes should stick to these groups. As soon as you deviate, you introduce a new aggregate. Often when people deviate from the default groups, it’s to ensure ordering. Maybe you need your script to be the very first script on the page, so you set your group to JS_LIBRARY - 100. Instead, use the ‘weight’ option to manage ordering. Set your group to JS_LIBRARY and set the weight very low to ensure your first in the JS_LIBRARY aggregate. Another issue I see is themes adding script explicitly under the JS_THEME group. This is logical, after all, it is Javascript added by the theme! However, it’s better to make use of hook_library, declare your dependencies and let the group default to JS_DEFAULT. The order you need is preserved by declaring your dependencies, and you avoid creating a new aggregate unessesarily.

Inline and External Scripts

Extra care should be employed when adding external and inline scripts. drupal_add_js and friends preserve order really well, such that they track the order the function is called for all the script added to the page and respect that. That’s all well and good. However, if an inline or external script is added between file scripts, it can split the aggregate. For example if you add 3 scripts: 2 file scripts and 1 external, all with the same scope, group, and every_page values, you might think that Drupal would aggregate the 2 file scripts and output 2 script tags total. However, if drupal_add_js gets called for the file, then external, then file, you end up getting two aggregates generated with the external script in between for a total of 3 script tags. This is probably not what you want. In this case it’s best to specify a high or low weight value for the external script so it sits at the top or bottom of the aggregate and doesn’t end up splitting it. The same goes for inline JS.

Scripts added in a different order

I alluded to this in the last one, but certain situations can arise where scripts get added to an aggregate in a different order from one request to the next. This can be for any number of reasons, but because Drupal is tracking the call order to drupal_add_js, you end up with a different order for the same scripts in a group and thus a different aggregate. Sadly the same code will sit in two aggregates on the server, with slightly different source order, but otherwise they could have been exactly equal, produced a single aggregate and thus cached by the browser from page to page. The solution in this case is to use weight values to ensure the same order within an aggregate from page to page. It’s not ideal, because you don’t want to have to set weights on every hook_library / drupal_add_js. I’d recommend handling it on a case by case basis.

Best Practices

Clearly, there is a lot that can go wrong, or at least end up taking you to a place that is sub-optimal. Considering all that we’ve covered, I’ve come up with a list of best practices to follow:

Always use the API, never ‘shoehorn’ scripts into the page

A common problem is running into modules and themes that sidestep the API to add their Javascript to the page. Common methods include using drupal_add_html_head to add script to the header, using hook_page_alter and appending script markup to $variables[‘page_bottom’], or otherwise outputting a raw

Use hook_library

hook_library is a great tool. Use it for any and all Javascript you output (inline, external, file). It centralizes all of the options for each script so you don’t have to repeat them, and best of all, you can declare dependencies for your Javascript.

Use every_page when appropriate

The every_page option signals to the aggregation system that your script will be added to all pages on the site. If your script qualifies, make sure your setting every_page = TRUE. This puts your script into a "stable" aggregate that is cached and reused often.

AdvAgg

Advanced CSS/JS Aggregation (AdvAgg) is a contributed module that replaces Drupal’s built in aggregation system with it’s own, making many improvements and adding additional features along the way. The way that you add scripts is the same, but behind the scenes, how the groups are generated and aggregated into their respective aggregates is different. AdvAgg also attempts to overcome many of the scenarios where core aggregation can go wrong that I listed above. I won’t cover all of what AdvAgg has to offer, it has an impressive scope that would warrant it’s own article or more. Instead I’ll touch on how it can solve some of the problems we listed above, as well as some other neat features.

Out of the box the core AdvAgg module supplies a handful of transparent backend improvements. One of those is stampede protection. After a code release with JS/CSS changes, there is a potential that multiple requests for the same page will all trigger the calculation and writing of the same new aggregates, duplicating work. On high traffic sites, this can lead to deadlocks which can be detrimental to performance. AdvAgg implements locking so that only the first process will perform the work of calculating and writing aggregates. Advagg also employs smarter caching strategies so the work of calculating and writing aggregates is done as infrequently as possible, and only when there is a change to the source files. These are nice improvements, but there is great power to behold in AdvAgg’s submodules.

AdvAgg Compress Javascript

AdvAgg comes with two submodules to enhance JS and CSS compression. They each provide a pluggable way to have a compressor library act on the each aggregate before it’s saved to disk. There are a few options for JS compression, JSqueeze is a great option that will work out of the box. However, if you have the flexibility on your server to install the JSMIN C extension, it’s slightly more performant.

AdvAgg Modifier

AdvAgg Modifier is another sub module that ships with AdvAgg, and here is where we get into solving some of the problems we listed earlier. Let’s explore some of the more interesting options that are made available by AdvAgg Modifier:

Optimize Javascript Ordering

There are a couple of options under this fieldset that seek to solve some of the problems laid out above. First up, "Move Javascript added by drupal_add_html_head() into drupal_add_js()" does just what it says, correcting the errant use of the API by module/theme authors. Doing this allows the Javascript in question to participate in Javascript aggregation and potentially removes an HTTP request if it was a file that was being added. Next “Move all external scripts to the top of the execution order” and “Move all inline scripts to the bottom of the execution order” are both an attempt at resolving the issue of splitting the aggregate. This is more or less assuming that external files are more library-esque and inline Javascript is meant to be run after all other Javascript has run. That may or may not be the case depending on what you’re doing and you should definitely use caution. Such as that is, if it works for you, it could be a simple way to avoid splitting the aggregate without having to do manual work of adjusting weights.

Move JS to the footer

As we discussed, you typically add Javascript to one of two scopes, header or footer. A common performance optimization is to move all the Javascript to the footer. AdvAgg gives you a couple choices in how you do that ranging from some to all Javascript moved to the footer. This can easily break your site however, and you definitely need to test and use caution here. The reason of course is that there could be other script that aren’t declaring their dependencies properly, and/or are ‘shoehorning’ their Javascript into the page that are depending on a dependency existing in the header. These will need to be fixed on a case by case basis (submit patches to contrib modules!). If you still have Javascript that really does need to be in the header, AdvAgg extends the options for drupal_add_js and friends to include a boolean scope_lock that when set to TRUE, will prevent AdvAgg from moving the script to the footer. There always seems to be some stubborn ads or analytics provider that insists on presence in the header, and is backed by the business that forces you to make at least one of these exceptions. Another related option is "Put a wrapper around inline JS if it was added in the content section incorrectly". This is an attempt to resolve the problem of inline Javascript in the body depending on things being in the header. AdvAgg will wrap any inline Javascript in a timer that waits on jQuery and Drupal to be defined before running the inline code. It uses a regular expression to search for the scripts, which while neat, feels a little brittle. There is an alternate option to search for the scripts using xpath instead of regex, which could be more reliable. If you try it, do so with care, test first.

Drupal 8

Not a lot is different when it comes to Javascript aggregation in D8, but there have been a few notable changes. First, the aggregation system has been refactored under the hood to be pluggable. You can now cleanly swap out and/or add to the aggregation system, whereas in Drupal 7 it was a difficult and messy affair to do so. Modules like AdvAgg should have a relatively easier time implementing their added functionality. Other improvements to the aggregation system that didn’t quite make the D8 release will now have a clearer path to implementation for the same reason. In terms of its function and the way in which Javascript is grouped for aggregation, everything is as it was in Drupal 7.

The second change centres around adding Javascript to the page. drupal_add_js and drupal_add_library have been removed in Drupal 8 in favor of #attached. The primary motivation is to improve cacheability. With Javascript consistently added using #attached, it’s possible to do better render array caching. The classic example being that if you were using drupal_add_js in your form builder function, your Javascript would be missed if the cached render array was used and the builder function skipped. Caching aside, it’s nice that there is a single consistent way to add Javascript. Also related to this change, is hook_library definitions have gone the way of the dodo. In their place, *.libraries.yml files are now used to define your Javascript and CSS assets in libraries, continuing the Drupal 8 theme of replacing info hooks with YML files.

A third somewhat related change that I’ll point out is that Drupal core no longer automatically adds jQuery, Drupal.js and Drupal.settings (drupalSettings in Drupal 8). jQuery still ships with core, but if you need it, you have to declare it as a dependency in your *.libraries.yml file. There have also been steps taken to reduce Drupal core’s reliance on jQuery. This isn’t directly related to Javascript aggregation per se, but it’s a big change nonetheless that you’ll have to keep in mind if you’re used to writing Javascript in Drupal 7 and below. It’s a welcome improvement with potential for some nice front end performance gains. For more information and specific implementation details, check out the guide for adding Javascript and CSS from a theme and adding Javascript and CSS from a module.

HTTP 2.0

Any article about aggregating Javascript assets these days deserves a disclaimer regarding HTTP 2.0. For the uninitiated, HTTP 2.0 is the new version of the HTTP protocol that while maintaining all the semantics of HTTP 1.1, significantly changes the implementation details. The most noted change in a nutshell is that instead of opening a bunch of TCP connections to download all the resources you need from a single origin, HTTP 2.0 opens a single connection per origin and multiplexes resources on that connection to achieve parallelism. There is a lot of promise here because HTTP 2.0 enables you to minimize the number of TCP connections your browser has to open, while still downloading assets in parallel and individually caching assets. That means that you could be better off not aggregating your assets at all, allowing the browser to maximize caching efficiency without incurring the cost of multiple open TCP connections. In practice, the jury is still out on whether no aggregation at all is a good idea. It turns out that compression doesn’t do as well on a collection of individual small files as it does on aggregated large files. It’s likely that some level of aggregation will continue to be used in practice. However as hosts more widely support HTTP 2.0 and old browsers fall off, this will become a larger area of interest.

Conclusion

Whew, we’ve come to the end at last. My purpose was to go over all of the nuances of Javascript aggregation in Drupal. It’s an important topic when it comes to front end performance, and one that deserves careful consideration when you're trying to eek every last bit of performance out of your site. With Drupal 8 and HTTP 2.0 now a reality, it’ll be interesting to watch as things evolve for the better.

Pages