Feed aggregator

Jeff Geerling's Blog: Testing redirections using Behat and the Behat Drupal Extension

Planet Drupal -

One project I'm working on needed a Behat test added to test whether a particular redirection works properly. Basically, I wanted to test for the following:

  1. An anonymous user accesses a file at a URL like http://www.example.com/pictures/test.jpg
  2. The anonymous user is redirected to the path http://www.example.com/sites/default/files/pictures/test.jpg

Since the site uses Apache, I added the actual redirect to the site's .htaccess file in the docroot, using the following Rewrite configuration:

<IfModule mod_rewrite.c>
  RewriteEngine on

  # Rewrite requests for /profile_images to public files directory location.
  RewriteRule ^ pictures/(.*)$ /sites/default/files/pictures/$1 [L,NC,R=301]
</IfModule>

Testing with curl --head, I could see that the proper headers were set—Location was set to the correct redirected URL, and the response gave a 301. So now I had to add the Behat test.

Modules Unraveled: New FREE Series! Drupal 8: Composer and Configuration Management

Planet Drupal -

I'm releasing a new series today!!!!!

Over the last year, I've given a talk at DrupalCon, DrupalCorn Camp, and Drupal Camp Colorado all about using Composer and Configuration Management in Drupal 8. Those sessions were around 45 minutes, which is much too short to go in depth, and explain everything thoroughly.

This series is the answer to that issue.

There is just over 1hr 15min of content in 26 videos that fall pretty well into seven parts. Here's the outline:

Part 1: Intro
  • Intro
Part 2: Installing Drupal 8 Locally
  • Creating a New Drupal 8 Project Using the Composer Template
  • Setting Up MAMP to Serve Your Site Locally
  • Using xip.io for Local Device Testing
  • Creating a Drush Alias
  • Installing Drupal with Console
  • Configuring settings.php and settings.local.php
  • Committing Your Project to Git
Part 3: Using Composer
  • Installing and Uninstalling Modules with Composer
  • Installing the Dev Version of Modules
  • Updating and Downgrading Modules
  • Skipping Specific Module Versions
  • Specifying Acceptable Version Ranges
  • Enabling Modules with Drush and Deciding What Version Pattern to Use
Part 4: Configuration Management
  • Setting the Config Directory in settings.php
  • Exporting Config Locally
  • Using the Configuration Installer Install Profile
Part 5: Installing Drupal 8 on a Remote Server
  • Installing the Site on a Production Server with Composer
Part 6: Overriding Settings in Code with settings.local.php
  • Setting up settings.local.php
  • Changing the Site Name and Disabling CSS Aggregation in settings.local.php
  • How to Enable Theme Debugging on Development Sites
  • Overriding Module Configuration (Like Google Analytics) in settings.local.php
Part 7: Putting it all Together
  • Configuring a Local Site and Exporting it's Configuration with Git
  • Pulling Changes to a Remote Site (And some gotchas)
  • Using Drush Shell Aliases to Make Development Easier
  • Verifying the Changes Made on Local and Reflected on Live

I'm pretty excited to finally have this series out, and hope you enjoy it! Oh... did I mention that it's ABSOLUTELY FREE?!?! Well, it is! So, check it out now, and let me know what you think.

Tags: Drupal 8ComposerConfiguration ManagementLocal DevelopmentDrushGitShell-Aliasesplanet-drupal

James Oakley: How to secure a Drupal website

Planet Drupal -

OK - I'll hold my hands up. The title of this post is misleading. I'm not going to give you an ABC on how to secure a Drupal site (maybe another day). I'm responding to a post on the Reseller Club blog entitled How to Secure Your Client's Drupal Website.

There is some good advice in that article, but it's mixed in with some bad advice, and in other parts it's just plain confused. In the hope that it helps people, I'm going to try and untangle things.

Blog Category: Drupal Planet

ADCI Solutions: Drupal 7 Performance Tips

Planet Drupal -

Drupal 8 has been released more than one year ago, but Drupal 7 is still widely used: it's totally stable, feature-rich, actively maintained and has tons of available modules for functionality extension. In this article I would love to draw your attention to Drupal 7 performance only. You can find a lot of information on Drupal 7 performance over the Internet since Drupal 7 is available for a long time, but anyway things change and new options appear. Website performance is very important because it can lead to higher conversion rates, higher search ranking and hosting costs reduction. I'll focus on Drupal 7 performance on the server side, but there are other areas where website performance can and should be improved including front-end, database, etc. These options are highlighted here.

 

Pronovix: Graph databases in Drupal: a Neo4j module that integrates with Rules

Planet Drupal -

SQL databases are really fast when you need all the information stored together in a record row, but they are a bad fit when you need to search for relationship patterns that are not already stored together in your database. A significant performance penalty is incurred for every additional table that needs to be joined for a query. That is why SQL databases are notoriously bad at deducting relationships from datasets. Graph databases however are really good at this task.

Kris Vanderwater: Hedron: Or my dissatisfaction with local development.

Planet Drupal -

Hedron: Or my dissatisfaction with local development. Kris Vanderwater 12 January 2017

I like things that work. I think most technicians do, but as a web developer I have a very serious problem. My most effective environment for doing web development is the one that exists on my own personal box. It can also be a rather impractical place to develop because most of my customers (current and historic) are on rather customized server stacks. Typically, the host has customized the environment to their own specifications. It's not uncommon to find additional services like solr or maybe a memcache server in the mix.

Drupal core announcements: Drupal core security release window on Wednesday, January 18, 2017

Planet Drupal -

Start:  2017-01-17 12:00 - 2017-01-19 12:00 UTC Organizers:  stefan.r David_Rothstein Fabianx catch xjm cilefen Event type:  Online meeting (eg. IRC meeting)

The monthly security release window for Drupal 8 and 7 core will take place on Wednesday, January 18.

This does not mean that a Drupal core security release will necessarily take place on that date for any of the Drupal 8 or 7 branches, only that you should watch for one (and be ready to update your Drupal sites in the event that the Drupal security team decides to make a release).

There will be no bug fix or feature release on this date. The next window for a Drupal core patch (bug fix) release for all branches is Wednesday, February 01. The next scheduled minor (feature) release for Drupal 8 will be on Wednesday, April 5.

Drupal 6 is end-of-life and will not receive further security releases.

For more information on Drupal core release windows, see the documentation on release timing and security releases, and the discussion that led to this policy being implemented.

Web Wash: Video: Create Custom Pages using Page Manager and Panels in Drupal 8

Planet Drupal -

Yesterday I presented WebWash’s first webinar on Page Manager and Panels. I had lots of fun doing the presentation and was asked some pretty good questions at the end. In the video I cover the following: What’s new in Drupal 8. Demonstrate how to create a custom page. Show you how to use multiple variants. Demonstrate Panels IPE (in-place editor). Finally, I show you how to use Bootstrap Layouts.

Drupal.org blog: Recognizing more types of contribution in the Drupal.org Marketplace

Planet Drupal -

Within weeks of introducing the contribution credit system on Drupal.org we realized we had created something powerful. Like all open source projects, Drupal has a behind-the-scenes economy of contribution in which individuals, organizations, and end users work together to maintain the software as a public good. That behind-the-scenes economy was brought to the fore when we chose to rank the Drupal Marketplace by issue credits. For the first time, Drupal.org gave businesses a direct financial incentive to contribute code.  

Being good stewards of these incentives is a sobering responsibility, but also a great opportunity. We can use this system to recognize the selfless effort of our community volunteers, to reward the organizations that sponsor their employees' time to give back to the project, and to connect end-users with the organizations that are the biggest contributors.

But as we often say in this community—contribution is more than code. It is the time provided by dedicated volunteers; the talent of community organizers, documentation maintainers, and developers; and the treasure provided by organizations that sponsor Drupal events and fund the operations and infrastructure that maintain the project.

What are we changing?

We’re updating the ranking algorithm for Drupal.org’s Marketplace of service providers and list of all organizations in the Drupal ecosystem. We've expanded on the issue credit system to create a more generic contribution credit system which lets us recognize more types of contribution. Each type of contribution is now weighted to give the organization an overall amount of contribution credit. We've built this system so that we can continuously evolve the incentives it creates by adjusting the weight given to each type of contribution as the project's needs change. To prevent gaming, we will not be publishing the exact weights or total contribution score, but those weights have been reviewed by the Association Board and Community Working Group.

We've carefully chosen a few new types of contribution to factor into the ranking. These were selected because they create incentives to reach specific goals: encouraging organizations to sponsor development of Drupal, gathering more Drupal 8 success stories that can be used to promote Drupal adoption, and recognizing the financial contributions that promote the fiscal health of the Drupal association.

We now calculate the following 4 types of contribution into overall contribution credit:

What about other types of contribution?

Of course, these new factors still don't include all types of contribution. This iteration aims to add measurable factors that reward the behavior of organizations that are good Drupal citizens, and incentivize some of the most important contributions that have a big impact in moving the project forward. But there are other factors we'd like to include in the future! We're keeping track of these additional kinds of contribution, such as sponsoring local user groups, organizing training days, writing documentation, and more, in this issue: #2649100: Improve contribution statistics on user and organization profiles.

There are two factors in particular that we are not yet including that we'd like to address.

The first is project application reviews. These reviews are a critical part of the lifecycle of a new project on Drupal.org, but because we are making the Project Application Revamp a key priority for the first part of 2017, this was not our focus in this initial update. We may revisit this factor as the Project Application Revamp initiative gets underway.

The second is camp organization. We know that there are many individuals and organizations who invest heavily in Drupal Camps, and this has been a critical part of the project's success. However, at this time our data about the individuals and organizations who participate in camp organization is purely self-reported, and therefore too vulnerable to manipulation to include in the algorithm at this time. In the future we hope we can find a responsible way to measure and credit this kind of contribution.

We’ll continue to look for other good factors to add, and do our best to weigh them fairly.

How often will the algorithm change? Who governs these changes?

As this is our first major change to the marketplace ranking system since the launch of issue credits, we may need to make some small adjustments in the first weeks following the launch. However, we know that too frequent changes to the incentive structure will be frustrating for the individuals and organizations who are contributing to the project. Therefore, after the initial tuning we intend to update the marketplace ranking system on a roughly 6 month cycle.

While the primary responsibility to manage the contribution credit system is ours, we have committed to vetting these and future changes with members of the Drupal Association Board and Community Working Group.

Recognizing more types of contribution in the Drupal.org Marketplace

Drupal News -

Within weeks of introducing the contribution credit system on Drupal.org we realized we had created something powerful. Like all open source projects, Drupal has a behind-the-scenes economy of contribution in which individuals, organizations, and end users work together to maintain the software as a public good. That behind-the-scenes economy was brought to the fore when we chose to rank the Drupal Marketplace by issue credits. For the first time, Drupal.org gave businesses a direct financial incentive to contribute code.  

Being good stewards of these incentives is a sobering responsibility, but also a great opportunity. We can use this system to recognize the selfless effort of our community volunteers, to reward the organizations that sponsor their employees' time to give back to the project, and to connect end-users with the organizations that are the biggest contributors.

But as we often say in this community—contribution is more than code. It is the time provided by dedicated volunteers; the talent of community organizers, documentation maintainers, and developers; and the treasure provided by organizations that sponsor Drupal events and fund the operations and infrastructure that maintain the project.

What are we changing?

We’re updating the ranking algorithm for Drupal.org’s Marketplace of service providers and list of all organizations in the Drupal ecosystem. We've expanded on the issue credit system to create a more generic contribution credit system which lets us recognize more types of contribution. Each type of contribution is now weighted to give the organization an overall amount of contribution credit. We've built this system so that we can continuously evolve the incentives it creates by adjusting the weight given to each type of contribution as the project's needs change. To prevent gaming, we will not be publishing the exact weights or total contribution score, but those weights have been reviewed by the Association Board and Community Working Group.

We've carefully chosen a few new types of contribution to factor into the ranking. These were selected because they create incentives to reach specific goals: encouraging organizations to sponsor development of Drupal, gathering more Drupal 8 success stories that can be used to promote Drupal adoption, and recognizing the financial contributions that promote the fiscal health of the Drupal association.

We now calculate the following 4 types of contribution into overall contribution credit:

What about other types of contribution?

Of course, these new factors still don't include all types of contribution. This iteration aims to add measurable factors that reward the behavior of organizations that are good Drupal citizens, and incentivize some of the most important contributions that have a big impact in moving the project forward. But there are other factors we'd like to include in the future! We're keeping track of these additional kinds of contribution, such as sponsoring local user groups, organizing training days, writing documentation, and more, in this issue: #2649100: Improve contribution statistics on user and organization profiles.

There are two factors in particular that we are not yet including that we'd like to address.

The first is project application reviews. These reviews are a critical part of the lifecycle of a new project on Drupal.org, but because we are making the Project Application Revamp a key priority for the first part of 2017, this was not our focus in this initial update. We may revisit this factor as the Project Application Revamp initiative gets underway.

The second is camp organization. We know that there are many individuals and organizations who invest heavily in Drupal Camps, and this has been a critical part of the project's success. However, at this time our data about the individuals and organizations who participate in camp organization is purely self-reported, and therefore too vulnerable to manipulation to include in the algorithm at this time. In the future we hope we can find a responsible way to measure and credit this kind of contribution.

We’ll continue to look for other good factors to add, and do our best to weigh them fairly.

How often will the algorithm change? Who governs these changes?

As this is our first major change to the marketplace ranking system since the launch of issue credits, we may need to make some small adjustments in the first weeks following the launch. However, we know that too frequent changes to the incentive structure will be frustrating for the individuals and organizations who are contributing to the project. Therefore, after the initial tuning we intend to update the marketplace ranking system on a roughly 6 month cycle.

While the primary responsibility to manage the contribution credit system is ours, we have committed to vetting these and future changes with members of the Drupal Association Board and Community Working Group.

Red Route: Talking about How We Work

Planet Drupal -

Following the previous blog post about our software engineering team culture that I wrote with my colleague Andrew Harmel-Law, I spoke about the subject at the January Drupal Show & Tell last night.

I've been meaning to speak at a meetup for a long time, and if I hadn't done it last night, I'd probably be putting it as one of my objectives for the year. The trouble was, I could never think of what to say. But conversations turned to tweets turned to blog posts, and it felt right to talk about this subject, particularly given that one of the themes of the blog post is the importance of communication between people.

I'd been to the Drupal Show & Tell meetup a couple of times before, and it's a friendly group with some familiar faces, so when I saw the call for speakers, it seemed the ideal opportunity for my first venture into public speaking.

As I rode my bike through the snow to the meetup, I was a little worried that the attendance might be a little sparse, and my blocked nose wasn't helping my confidence. After a few anxious moments where we thought there might be more speakers than people in the audience, more people arrived, and we got started, with interesting and thought-provoking talks from Anthony Seale and Nigel Milligan.

Finally, it was my turn, and despite losing my thread once or twice, I think it went fairly well for a first attempt. As I mentioned in the talk, one of the key points is about improving through iteration - I'll be tweaking the talk and delivering a new version of it at one of our internal lightning talks sessions soon.

My slides for the talk are available online, and I've embedded them below. Thanks to Cameron & Wilding for organising the meetup, and to everyone who came along.

Your browser does not support iframes. Please visit https://malcomio.github.io/presentations/how-we-work/#/ to view the presentation.

Tags:  Drupal Presentation development All tags

Drupal Blog: Nominations are now open for the 2017 Aaron Winborn Award

Planet Drupal -

The Drupal Community Working Group is pleased to announce that nominations for the 2017 Aaron Winborn Award are now open. This annual award recognizes an individual who demonstrates personal integrity, kindness, and above-and-beyond commitment to the Drupal community. It will include a scholarship and stipend to attend DrupalCon and recognition in a plenary session at the event.

Nominations are open to not only well-known Drupal contributors, but also people who have made a big impact in their local or regional community. If you know of someone who has made a big difference to any number of people in our community, we want to hear about it.

This award was created in honor of long-time Drupal contributor Aaron Winborn, whose battle with Amyotrophic lateral sclerosis (ALS) (also referred to as Lou Gehrig's Disease) came to an end on March 24, 2015. Based on a suggestion by Hans Riemenschneider, the Community Working Group, with the support of the Drupal Association, launched the Aaron Winborn Award.

Nominations are open until March 1, 2017. A committee consisting of the Community Working Group members and past award winners will select a winner from the submissions. Members of this committee and previous winners are exempt from winning the award.

Previous winners of the award are:

*  2015: Cathy Theys  
*  2016: Gábor Hojtsy  

If you know someone amazing who should benefit from this award please nominate them at https://www.drupal.org/aaron-winborn-award

Nominations are now open for the 2017 Aaron Winborn Award

Drupal News -

The Drupal Community Working Group is pleased to announce that nominations for the 2017 Aaron Winborn Award are now open. This annual award recognizes an individual who demonstrates personal integrity, kindness, and above-and-beyond commitment to the Drupal community. It will include a scholarship and stipend to attend DrupalCon and recognition in a plenary session at the event.

Nominations are open to not only well-known Drupal contributors, but also people who have made a big impact in their local or regional community. If you know of someone who has made a big difference to any number of people in our community, we want to hear about it.

This award was created in honor of long-time Drupal contributor Aaron Winborn, whose battle with Amyotrophic lateral sclerosis (ALS) (also referred to as Lou Gehrig's Disease) came to an end on March 24, 2015. Based on a suggestion by Hans Riemenschneider, the Community Working Group, with the support of the Drupal Association, launched the Aaron Winborn Award.

Nominations are open until March 1, 2017. A committee consisting of the Community Working Group members and past award winners will select a winner from the submissions. Members of this committee and previous winners are exempt from winning the award.

Previous winners of the award are:

*  2015: Cathy Theys  
*  2016: Gábor Hojtsy  

If you know someone amazing who should benefit from this award please nominate them at https://www.drupal.org/aaron-winborn-award

OSTraining: Contribute Your Code on Drupal.org, Part 6: Review Process

Planet Drupal -

Previously we talked about connecting and checking that you are connected to your sandbox project, uploading your project and checking it against Paraview. 

{snippet drupalcontributionlinks}

Now, in Part 6, we're going to look at getting your theme reviewed. This is perhaps the trickiest and slowest part of the whole contribution process, so pay close attention. We're going to help you do everything possible to ensure a speedy and successful application submission. 

Ixis.co.uk - Thoughts: Drupal 8 adoption and what 2017 is likely to bring

Planet Drupal -

It’s now over a year since the release of Drupal 8, the first new version of the open source content management framework in five years. It represented a significant rethinking of the platform when it launched in November 2015 and 2017 is likely to be another significant year for Drupal 8 with further updates and developments expected. As Dries Buytaert, the founder and lead developer of Drupal outlined in a blog last September:

“The only way to stay competitive is to have the best product and to help people adopt it more seamlessly. This means that we have to continue to be able to reinvent ourselves and that we need to make the resulting changes less scary and easier to absorb. We decided that we wanted more frequent releases of Drupal, with new features, API additions, and an easy upgrade path.”

One of the most common questions we get asked is, “How is Drupal 8 doing and what can we expect in the future?". Understandably, this is important to many organisations currently using previous versions and the Drupal community that has a stake in its success. But it’s also important to new users looking to migrate over.

There’s a significant amount of interest because, as with any new update, organisations have to decide whether the framework is right for them. While Drupal 8 is still maturing, adoption rates are now growing fast. This year we migrated the Ixis website to Drupal 8, which you can read more about our experience here.

User growth and high profile successes

2016 saw progress in a variety of key areas including user growth, as well as a number of high-profile successes and this is likely to continue in 2017. As with all new updates, it has taken some time for Drupal 8 to gain traction in terms of the number of users but there has been a clear upward trajectory over the course of the year. There are currently 120,000 Drupal 8 projects, and while Drupal 7 is currently running more than 1 million, this still represents significant growth one year in, especially with adoption rates starting to increase, as outlined on drupal.org.

The wide range of prominent Drupal 8 projects that have launched in the past year have helped to showcase the power of the new platform. These include:

  • NBA.com - Millions of fans around the globe rely on the NBA's Drupal 8 website to livestream games, read stats and standings, and stay up to date on their favourite team.

  • Nasdaq - Drupal 8 is used as the basis for its next generation Investor Relations Website Platform. IR websites are where public companies share their most sensitive and critical news and information with their shareholders, institutional investors, the media and analysts.
The migration question

Although the Drupal 7 user base remains solid and the platform will be supported for a long time, there is an end-of-life in sight for it. This is expected to be in two to three years time but might be even sooner, with some sources predicting an end to Drupal 7 development as soon as October 2017. Either way, Drupal 8 promising a host of further improvements, with Drupal 8.2 already available, many organisations are beginning to look at early migration. The good thing about Drupal 8 is that it comes bundled with a suite of tools to assist with the migration of your content from previous versions, making the task less daunting than it might initially seem. With its ‘continuous innovation’ mission statement a migration to Drupal 8 in 2017 will provide the best possible access to the latest functionality and improvements, helping to unlock the framework’s true potential.

For more information about Drupal 8 contact us on [email protected] or call 01925 320 041.

Agiledrop.com Blog: AGILEDROP: Top Drupal 8 Books

Planet Drupal -

The purpose of a great content is first to be explored and then to be read. Therefore, being listed on a first page on search engines is a key thing. Shortly, you will be able to read how to Search Engine Optimize your Drupal 8 website. But before Drupal SEO Book is released – you can pre-order it here or wait until 15th January – we will look at the best books Drupal 8 has to offer. The first choice is Drupal 8 Development Cookbook by Matt Glaman, which enjoys the best feedback from readers. The book is appropriate for web developers and site builders. It contains the whole chapter for… READ MORE

Introducing Accelerated Mobile Links: Making the Mobile Web App-Quick

Cloudflare Blog -

In 2017, we've predicted that more than half of the traffic to Cloudflare's network will come from mobile devices. Even if they are formatted to be displayed on a small screen, the mobile web is built on traditional web protocols and technologies that were designed for desktop CPUs, network connections, and displays. As a result, browsing the mobile web feels sluggish compared with using native mobile apps.

In October 2015, the team at Google announced Accelerated Mobile Pages (AMP), a new, open technology to make the mobile web as fast as native apps. Since then, a large number of publishers have adopted AMP. Today, 600 million pages across 700,000 different domains are available in the AMP format.

The majority of traffic to this AMP content comes from people running searches on Google.com. If a visitor finds content through some source other than a Google search, even if the content can be served from AMP, it typically won't be. As a result, the mobile web continues to be slower than it needs to be.

Making the Mobile Web App-Quick

Cloudflare's Accelerated Mobile Links helps solve this problem, making content, regardless of how it's discovered, app-quick. Once enabled, Accelerated Mobile Links automatically identifies links on a Cloudflare customer's site to content with an AMP version available. If a link is clicked from a mobile device, the AMP content will be loaded nearly instantly.

To see how it works, try viewing this post from your mobile device and clicking any of these links:

Increasing User Engagement

One of the benefits of Accelerated Mobile Links is that AMP content is loaded in a viewer directly on the site that linked to the content. As a result, when a reader is done consuming the AMP content closing the viewer returns them to the original source of the link. In that way, every Cloudflare customers' site can be more like a native mobile app, with the corresponding increase in user engagement.

For large publishers that want an even more branded experience, Cloudflare will offer the ability to customize the domain of the viewer to match the publisher's domain. This, for the first time, provides a seamless experience where AMP content can be consumed without having to send visitors to a Google owned domain. If you're a large publisher interested in customizing the Accelerated Mobile Links viewer, you can contact Cloudflare's team.

Innovating on AMP

While Google was the initial champion of AMP, the technologies involved are open. We worked closely with the Google team in developing Cloudflare's Accelerated Mobile Links as well as our own AMP cache. Malte Ubl, the technical lead for the AMP Project at Google said of our collaboration:

"Working with Cloudflare on its AMP caching solution was as seamless as open-source development can be. Cloudflare has become a regular contributor on the project and made the code base better for all users of AMP. It is always a big step for a software project to go from supporting specific caches to many, and it is awesome to see Cloudflare’s elegant solution for this."

Cloudflare now powers the only compliant non-Google AMP cache with all the same performance and security benefits as Google.

In the spirit of open source, we're working to help develop updates to the project to address some of publishers' and end users' concerns. Specifically, here are some features we're developing to address concerns that have been expressed about AMP:

  • Easier ways to share AMP content using publisher's original domains
  • Automatically redirecting desktop visitors from the AMP version back to the original version of the content
  • A way for end users who would prefer not to be redirected to the AMP version of content to opt out
  • The ability for publishers to brand the AMP viewer and serve it from their own domain

Cloudflare is committed to the AMP project. Accelerated Mobile Links is the first AMP feature we're releasing, but we'll be doing more over the months to come. As of today, Accelerated Mobile Links is available to all Cloudflare customers for free. You can enable it in your Cloudflare Performance dashboard. Stay tuned for more AMP features that will continue to increase the speed of the mobile web.

Mike Crittenden: Exporting and importing big Drupal databases

Planet Drupal -

Exporting and importing big Drupal databases

Once your site's database dump file gets to be 1GB or more, phrases like "oh, just download and import a DB dump" can't really be taken for granted anymore. So here are some tips for dealing with large databases, especially those of the Drupal variety.

Exporting

Before we can import, we must export. With a big DB, you don't want to just do a regular old mysqldump > outfile.sql and call it a day. Here are some tips.

Find the size before exporting

It can sometimes be useful to see how big the export is going to be before you actually export anything. That way, you can know ahead of time if you need to be doing this or that to reduce the size, or if it won't matter since the whole thing won't be that big anyway.

Here's a query you can run to see the size per DB table:

SELECT TABLE_SCHEMA, TABLE_NAME, DATA_LENGTH / POWER(1024,1) Data_KB, DATA_LENGTH / POWER(1024,2) Data_MB, DATA_LENGTH / POWER(1024,3) Data_GB FROM information_schema.tables WHERE table_schema NOT IN   ('information_schema','performance_schema','mysql') ORDER BY DATA_LENGTH;

And here's another query you can run to see what the total size for the entire DB is: 

SELECT Data_BB / POWER(1024,1) Data_KB, Data_BB / POWER(1024,2) Data_MB, Data_BB / POWER(1024,3) Data_GB FROM (SELECT SUM(data_length)   Data_BB FROM information_schema.tables WHERE table_schema NOT IN   ('information_schema','performance_schema','mysql')); Dump without unnecessary data

For those cases where you need the database structure for all of the tables, but you don't need the data for all of them, here's a technique you can use. This will grab the entire DB structure, but lets you exclude data for any tables that you want. For example, search_index, cache_*, or sessions tables will be good places to cut out some fat.

# First we export the table structure. mysqldump --no-data database_name > /export.sql # Grab table data, excluding tables we don't need. mysqldump --no-create-info   --ignore-table=database_name.table_name1   --ignore-table=database_name.table_name2   database_name >> export.sql

Just replace "table_name1" and "table_name2" with the tables that you want to skip, and you're golden. Also note that you can use the % character as a wildcard, so for example, you could ignore "cache%" for all cache tables.

After you do that, you'll have a single export.sql file that contains the DB structure for all tables and the DB data for all tables except the ones you excluded. Then, you'll probably want to compress it...

Compress all the things

This one may go without saying, but if you're not compressing your database dumps then either they're really tiny, or you're dumber than a dummy. 

drush sql-dump --gzip --result-file=db.sql

Compare that with the regular old:

drush sql-dump --result-file=db.sql

...and you're going to see a huge difference.

Or if you already have the SQL dump that you need to compress, you can compress the file directly using:

gzip -v db.sql

That will output a db.sql.gz file for you.

Importing

Now you have a nice clean compressed DB dump with everything you need and nothing you don't, and you're ready to import. Here are a few ways to ease the pain.

Import a compressed dump directly

Instead of having to decompress the dump before importing, you can do it inline:

gunzip -c db.sql.gz | drush sqlc Exclude data when importing

If you receive a DB dump that has a lot of data you don't need (caches, sessions, search index, etc.), then you can just ignore that stuff when importing it as well. Here's a little one-liner for this:

gunzip -c db.sql.gz | grep -Ev "^INSERT INTO \`(cache_|search_index|sessions)" | drush sqlc

What this is doing is using "grep" as a middleman and saying "skip any lines that are insertion lines for these specific tables we don't care about". You can edit what's in the parenthesis to add/remove tables as needed.

Monitor import progress

There's nothing worse than just sitting and waiting and having no idea how far along the import has made it. Monitoring progress makes a long import seem faster, because there's no wondering. 

If you have the ability to install it (from Homebrew or apt-get or whatever), the "pv" (Pipe Viewer) command is great here:

pv db.sql | drush sqlc

Or if your database is compressed:

pv db.sql.gz | gunzip | drush sqlc

Using "pv" will show you a progress bar and a completion percentage. It's pretty awesome.

If you don't have "pv" then you can settle for the poor man's version:

watch "mysql database_name -Be 'SHOW TABLES' | tail -n2"

That slick little guy will show you the table that is currently importing, and auto-updates as it runs, so you can at least see how far through the table list it has gone.

Tools and Resource

In this post I tried to focus on commands that everyone already has. If this just isn't cutting it for you, then look into these tools which could help even more:

  • SyncDB - a couple Drush commands that split DB dumps into separate files and import them in parallel, drastically speeding things up
  • Drush SQL Sync Pipe - an alternative to "drush sql-sync" that uses pipes where possible to speed things up
mcrittenden Wed, 01/11/2017 - 23:15

Palantir: New Years Resolution: Spend More Time With Family and Friends

Planet Drupal -

New Years Resolution: Spend More Time With Family and Friends brandt Wed, 01/11/2017 - 16:11 Allison Manley Jan 11, 2017

In this five-part series, every Monday in January we’ll explore a New Year’s resolution and how it can apply to your web project.

Stay connected with the latest news on web strategy, design, and development.

Sign up for our newsletter.

Surrounding oneself with a community of friends and family that offer needed support is important to us all. Palantir spent twenty years building our own culture and community right here at the office! But we’ve also been active members in the Drupal community for 12 years:

  • We’ve made contributions to every facet of the Drupal project: Core development, contributed modules, themes, financial assistance, training, documentation, conference organizing, and one Palantiri is a member of the Drupal Board.
  • This means we have a long history of helping organizations level up so they can become Drupal contributors and participants as well.
  • The collaboration in the open source community is one of the reasons Palantiri love Drupal so much.
Upcoming Events

Are you looking to get involved in the Drupal community? Some ideas:

Besides the Drupal and Open Source communities, Palantir works in some specific verticals that have their own rich and robust communities. We’re still finalizing exactly where we’ll be in 2017, but we know for sure you’ll find us at the following conferences so we can connect with friends in those industries and offer them support as needed:

Next week’s resolution: get organized. 

We'd love to help you keep your 2017 resolution.

Let's chat.

Pages

Subscribe to Cruiskeen Consulting LLC aggregator