Drupal News

Appnovation Technologies: Simple Website Approach Using a Headless CMS: Part 1

Planet Drupal -

Simple Website Approach Using a Headless CMS: Part 1 I strongly believe that the path for innovation requires a mix of experimentation, sweat, and failure. Without experimenting with new solutions, new technologies, new tools, we are limiting our ability to improve, arresting our potential to be better, to be faster, and sadly ensuring that we stay rooted in systems, processes and...

Drupal Association blog: Features and the future

Planet Drupal -

Drupal.org has been in existence since 2001. That's a long time for a website to serve an ever changing community! We're doing this work thanks to the support of our members, supporters, and partners. As time goes on needs change, technology evolves, and features are deployed to improve the experiences of site visitors.

As a web professional, you know how delivering small feature requests can have a big impact. To ensure people take notice of the improvements the Engineering Team makes on all of the *Drupal.org sites, we share frequent updates with the community. You can read a monthly what's new on Drupal.org blog, watch for change notifications, and follow on Twitter to know what's on the horizon.

Recently, these improvements were deployed:

  • More maintainers can now grant issue credit

  • Security Advisory nodes are now included in the /news feed

  • Project page screenshots will display in a lightbox

  • DrupalCI.yml Documented

We'll continue to make Drupal.org better every day, with your help. Find out more about what we do and become a member today. Thank you!

 Follow Drupal.org on Twitter: news and updates, infrastructure announcements, commits (and deployments).

Droptica: Working with a team that is spread out over many locations can be comfortable and pleasant, just like working in a team working at a single office. See how we do it at Droptica.

Planet Drupal -

At Droptica, remote work is something ordinary. From the very first day, we had two offices in Wrocław and Gdańsk. Recently, we also opened our third office in Rzeszów. Every day, developers, testers, graphic designers and other specialists work together in each of these locations. In addition, 90% of our clients come from abroad, like in the case of most software houses. Throughout several years, we have developed methods of effective and efficient cooperation in a dispersed team. We are constantly improving our work model, testing new tools and ways of working. In this article, you will learn how our system works today.

Droptica: Models of cooperation of the development team and the client based on the examples used at Droptica

Planet Drupal -

Droptica helps clients from all over the world to complete and implement their projects. Each of these clients has already developed their way of working. Everyone is different. In this article, I have collected the most common ways and systems of cooperation between Droptica and our clients. Why do we work a little differently with every client? We are Agile. We always want to maximise the results of our work, so our development team always adjusts and adapts their way of working to the client’s needs. The elements that are adapted and changed the most often include:

Drupal Association blog: The Values & Principles Committee

Planet Drupal -

In a previous blog post, I wrote that Dries asked for help in continuing the development of the Values & Principles and that a Values & Principles Committee will be formed to facilitate this. Well, we are at the point where we can explain how the committee will be formed, its purpose and how it will achieve its goals.

Purpose

The Values & Principles Committee will exist to focus on specific Values & Principles to see where continual improvements can be made and to propose those to the Project Lead for approval. For example, in meetings at DrupalCon Nashville, Dries expressed a desire to focus on Principle 8: “Every person is welcome; every behavior is not” because it is both critically important, in need of work, and requires more input.

Formation

To learn more about how the Values and Principle Committee will work, please read the charter, which is attached.

We have been giving thought to how we can facilitate a better, more open, process for appointing members to this Committee and we have come up with the following:

We will be posting role descriptions for the volunteer committee roles on jobs.drupal.org. The Values & Principles Committee Member voluntary role description will describe the:

  • Expectations of the role

  • The specific jobs the role will require

  • An indication of the time commitment of the role

  • The attributes that would be expected of a successful candidate

  • How to apply

The Committee Member role will be advertised from today until 3 August 2018 at https://jobs.drupal.org/drupal-community/job/15126 and then I will take the applications to Dries Buytaert and Megan Sanicki to select candidates based on the membership requirements outlined in the charter and role description.

This work matters to them personally and through their roles:  Dries as Drupal Project Lead and Megan as Drupal Association Executive Director. In addition to their different experiences and perspectives, they bring a wealth of experience in enterprise leadership, organizational culture and community building.  They hope to assemble a group that is inspired by this work, diverse, and representative of the values and principles we hope to inspire in the community. After the initial membership is selected, the membership will help recruit new members going forward.

Once the committee is selected, it can then begin work, in consultation with subject matter experts, on updating the values and principles.

My call to action for you is to consider whether volunteering as a Member of this Values & Principles Committee is a way that you can contribute to the Drupal Community and, if so, to submit your application.

I’m really keen to ensure that the Values & Principles Committee has membership from many places, with a diverse background and representing many a wide understanding of what makes Drupal - Drupal. It is even baked into the Values & Principles Committee Charter, attached.

File attachments:  Values Committee Charter.pdf

OPTASY: How to Add Autocomplete to Text Fields in Drupal 8: Defining a Custom Route

Planet Drupal -

How to Add Autocomplete to Text Fields in Drupal 8: Defining a Custom Route radu.simileanu Wed, 07/18/2018 - 10:15

Let's say that it's a WhatsApp-like, a decoupled, Drupal 8-backed, real-time chat platform that you're building. One using Node.js. In this case, implementing field autocomplete functionality becomes a must, doesn't it? But how do you add autocomplete to text fields in Drupal 8?

Needless to add that such otherwise "basic" functionality — implemented on fields such as node reference and user/tags — would instantly:
 

  1. improve the user experience 
  2. increase the level of user interactivity and engagement
     

Users would group around different "channels" and be able to easily add new members. The auto-complete text fields will make the whole “new member coopting” process conveniently easy:

mark.ie: Getting Value (URI) of a Drupal Media (File) Field from within a Twig Template

Planet Drupal -

Getting Value (URI) of a Drupal Media (File) Field from within a Twig Template

To the future or to the past, to a time when thought is free, to the next time when I need to get the value of file field to use as a variable in Drupal 8 with Twig.

markconroy Tue, 07/17/2018 - 17:07

Working my way down through one of Drupal's render arrays of doom to try to get the URI of a file in a media field (in a paragraph type), I came up with this. If you can improve it, feel free to drop a note in the comments:

{% set slide_url = file_url(content.field_p_ei_speaker_slides[0]['#media'].field_m_file_file.entity.uri.value) %}

In steps:

  1. Get the {{ content }} variable
  2. Drill down into the media field (Speaker Slides - pdf, ppt, etc)
  3. Get the first element (0 - it's not a multi-value field in this case)
  4. Load up the #media object
  5. Interrogate the field on the media entity that has the file attached (the File field)
  6. Load this entity (entity here is not presented as an item in the {{ dpm() }} but it's very handy to know
  7. Get the uri.value from here
  8. Wrap it all in a file_url() function

For clarity, here's what I had in PatternLab:

  {# Begin Slides Download #}
  {% if event_slide_download %}
   
     
        {% include '@basic-elements/icons/_svg.twig'
          with {
            svgpath : '@basic-elements/icons/svg/download.svg'
          }
        %}
     

   
  {% endif %}
  {# End Slides Download #}

And here's what I have in the corresponding Drupal paragraph.html.twig tempate:

{% if paragraph.field_p_ei_speaker_slides.value %}
  {% set event_slide_download = true %}
  {% set slide_url = file_url(content.field_p_ei_speaker_slides[0]['#media'].field_m_file_file.entity.uri.value) %}
  {% set event_slide_download_link = slide_url %}
{% endif %}

{% include "@building-blocks/event-section/event-item.twig" %}

So now, my future self, you will know where to find this next time.

July Drupal User Group and JAM @ Advantage Labs, 7/18/18 6-9pm

Twin Cities Drupal Group -

Start:  2018-07-18 18:00 - 21:00 UTC Event type:  User group meeting

Join us Wednesday evening for our monthly user group! We'll have the open JAM and networking starting at 6pm. At 7pm we'll transition over to a show and tell session. Learn something helpful? Build something cool? Come share or bring a question for the group. Jeremiah will kick things off by demonstrating a new module and showing a quick theming tip!

Location
Advantage Labs at FLOCK
2611 1st Avenue South
Minneapolis, MN, 55408

Pizza and beverages sponsored by Advantage Labs!

OpenSense Labs: Approach your Mobile-First Design with This Guide

Planet Drupal -

Approach your Mobile-First Design with This Guide Harshit Tue, 07/17/2018 - 19:21

It’s always a battle when planning a website design. Your end goals, target audience, content, user experience, everything must align. It’s like a cloud of thoughts right inside your team’s head. It’s critical to optimize your website experience for the devices your audience is or will be using. 

Emarketer forecasts, approximately 83% of internet users will use a mobile phone to go online by 2021. 

Since now that we’ve established the fact that mobile is where the future lies, a mobile-first redesign is a way to go. We’ll dive into what a mobile-first approach entails, including tips for creating a seamless experience for visitors from mobile devices. 

In 2018, it is critical to analyze what’s the right way to go about it. 

1. Shoot for the least 

Without compromising the functionalities, it is definitely a challenge to pull everything into such a limited real estate. The difference in the screen space of various devices should tell you the varied approach to design. 

Minimalism begins by removing every element one by one.

After listing down all the elements, prioritizing is key. Sort out the cruciality of every element and how would they hierarchically sit in the interface. This should be in a way that the most important elements are displayed prominently. 

If after this exercise there’s still space for more, carefully add some elements in order of importance without overdoing it. Doing this would make sure every inch is utilised to the fullest. 

Content Repository 

The approach to this should be by drafting a document or a spreadsheet containing all the content elements you want the interface to entail. 

List down the page titles, navigation titles and be open to comments from peers. An example is given below:

An example of a content repository2. Analyse the Important Interaction Points

Comparing smartphones from 2008 to the ones in 2018,  a lot has changed since then. The user experience is no more frustrating. Earlier you had to touch the exact alphabet in the keyboard now the smart devices can read your movement through sensors. Keyboards differ according to the smartphone size, button space and overall sensitivity. And so your mobile design should be approached in the same manner. 

Unfortunately, many websites are still lagging behind when it comes to the user experience which leads to weak engagement, poor sessions and doubled bounce rates. 
Make sure the important elements are not missed. 

This includes: 

  • Make button more clickable
  • Make CTAs more appealing and responsive
  • Choose the right color combination for the user to read on the sunny noon
  • Give hyperlinks plenty of space
  • Keep the tabs in the drop down manner
Good vs Bad UX design The number of mobile phone users in the world is expected to pass the five billion mark by 2019, according to Statista.  3. Optimize Your Images

When it comes to loading size, images make up nearly 64% of an average web page. Outlining the “why”, it is now important to know how you can optimize your images to deliver an optimum and engaging experience to your user, without hogging on those extra kilobytes of data. 

Hacks to Optimize Images
  • Use correct image dimensions for faster loading
  • Use the correct image format. JPEG image would be a lot lighter than the PNG.
  • Compress your image with lossless compression
  • Use Lazy loader to keep the user engaged
4. Content is all that matters

Devising content around mobile is time-consuming as it requires due thinking and goes through numerous iterations along the course of finalizing the content itself. 

Taking the limited space aside for a while, there are more added difficulties when it comes to mobile web development. 

There comes screen rotation, device to device compatibility, text and image overlaps and what not. These factors have everything to make your content look very less appealing. What do you do for cutting these hurdles off? 

Think from an app perspective

Mobile users are accustomed to motion and a modicum of control in their experience. Think about off-canvas navigation, expandable widgets, AJAX calls, or other elements on the screen with which users can interact without refreshing the page. This would help you always maintain the right threshold for user experience. 

Screen and remove all the distracting elements, make the content as precise as possible while catering to actual product or service display. 

Landscape photos and complex graphics don't show well when your screen is reduced by half. Take into account the mobile user with pictures that are comprehensible on handheld screens.

5. Be a User, Before Presenting to the User

Our industry standard for approaching design is devising wireframes first. During wireframing or prototyping, use adaptive breakpoints reference. It streamlines the process of moving to different screen sizes, starting with the least real estate. 

Hover is off the table

Interactive interfaces from 2018 mostly utilize hover effects. UX designers go bonkers over them. But that’s a desktop only thing as we do not have the finger-hover technology in town. You will have to forget hover' existence when going mobile. 

Having trouble configuring your site? Check out services by OpenSense Labs.

6. Opt for Accelerated Mobile Pages

Your users are likely to bounce off your website if it doesn’t load within 3 seconds. With mobile, the standard has been set high (or rather less). Since the internet has been taken for granted, people have become second sensitive these days. 

AMP does bring you speedy pages and is a great way to boost your SEO and stop losing out of potential customers. It’s important to note that Google has gone lengths to identify and promote AMP pages.

If your web pages follow AMP standards, it becomes eligible to be cached by Google and appear in the search results and carousels with the AMP logo, indicating that they offer a fast experience. 

A standard built on top of existing technologies to speed up the loading of web pages on mobile devices.

How to go about it?

Configure AMP in your website

7. Make engagement on Mobile Simple 

Whether it’s deciding between a “hamburger” style menu or a more visible-style on the front end or how to display forms on specific pages, don’t make engagement paths more abstract than they need to be. Engagement should be extremely crisp and effective in nature.


Set the Display Order

Now, after listing down all the elements, prioritizing is key. Sort out the cruciality of every element and how would they hierarchically sit in the interface. This should be in a way that the most important elements are displayed prominently. 

8. Test It Before You Sell It

Even after having the final product in your hands, recommendations and tweaks will keep flowing in. Change is evergreen. With changes being made, you need to be testing them as soon as they are deployed. 

Nothing beats discovering for yourself how usable a website is (or isn’t). Step away from your computer desktops and load up your product on a real phone or tablet. 

Test it in a real devices 

Using testing tools, you can conduct A/B tests. Some tests may be like, an orange CTA vs a Yellow CTA, Button size changes, block layout tweaks and etc. Tests can range from regular functionality checks to user experience tests.

Tap through pages. Is the site easy to navigate? Does it load in a timely fashion? Are the text and graphics easy to read?

It's as clear as the skies that the future of the internet is dependent on mobile experiences. Responsive web design is a must if you run your business online or attend to your customers for any purpose. 

The tips above will help you build what you are looking at while making the least possible amount of mistakes. 

If you need any further recommendations, we are here. Hook with us at [email protected] or tweet us at @OpenSenseLabs and our mobile development team will save you the trouble.

blog banner blog image Mobile first Mobile friendly  User experience Drupal Web Redesign Drupal websites Mobile approach Websites Mobile first approach Webpages Design Ideas Design Ideology Design Blog Type Articles Is it a good read ? On

Proxying traffic to Report URI with Cloudflare Workers

Cloudflare Blog -

The following is a guest post by Scott Helme, a Security Researcher, international speaker, and blogger. He's also the founder of the popular securityheaders.com and report-uri.com, free tools to help people deploy better security.

With the continued growth of Report URI we're seeing a larger and larger variety of sites use the service. With that diversity comes additional requirements that need to be met, some of them simple and some of them less so. Here's a quick look at those challenges and how they can be solved easily with a Cloudflare Worker.

Sending CSP Reports

When a browser sends a CSP report for us to collect at Report URI, we receive the JSON payload sent to us but we also have access to two other pieces of information, the client IP and the User Agent string. We never store, collect or analyse the client IP, we simply don't need or want to, and all we do with the UA string is extract the browser name like Chrome or Firefox. Most site operators are perfectly happy with our approach here and things work just fine. There are however some issues when the site operator simply doesn't want to have us to have this information and some cases have come up where they can't allow us to have access to that information because of restrictions placed on them by a regulator. The other common thing that comes up, which I honestly never anticipated, was simply the perception of the reporting endpoint being a 3rd party address. There are various different ways we can and do tackle these problems.

CNAME

Up until now, if a client didn't want to report to a 3rd party address we would ask them to CNAME their subdomain to us and run a dedicated instance that would ingest reports using their subdomain. We take control of certificate issuance and renewal and the customer doesn't need to do anything further. This is a fairly common approach across many different technical requirements and it's something that has worked well for us. The problem is that it does come with some administrative overheads for both parties. From our side the technical requirements of managing separate infrastructure are an additional burden, we're responsible for a subdomain belonging to someone else and there are more moving parts in the system, increasing complexity. I was curious if there was another way.

HTTP Proxy

One idea that we discussed with a customer a while back, but never deployed, was for them to run a proxy on premise. They could report to their own endpoint under their own control and simply forward reports to their Report URI reporting address. This means they could shield the client IP from us, mask the User Agent string if required and generally do any sanitisation they liked on the payload. The problem with this was that it just seemed like an awful lot of work, I'd much rather have discussed deploying Report URI on premise instead. The client is also still at risk of things like accidentally DDoSing their endpoint, which removes one of the good reasons to use Report URI.

Finding Another Way

For the most part our current model was working really well but there were some customers who had a hard requirement to not send reports directly to us. Our on premise solution also isn't ready for prime time just yet so we needed something that we could offer, without it requiring too much of the overhead mentioned above. That's when I had an idea that might just cut it.

Javascript On A Plane

I was sat on a flight just a few days ago and I never like to waste time. When I'm sat in the car on the way to the airport, sat in the airport or sat on my flight, I'm working. Those are precious hours that can't be wasted and during a recent flight between Manchester and Vienna I was playing around with Cloudflare Workers in their playground. I was tinkering with a worker to add Security Headers to websites, which has since been launched, and whilst inspecting the headers object and looking through the headers that were in the request I saw the User Agent string. "Oh hey, I could remove that if I wanted to" I thought to myself, and then the rapid fire series of events triggered in my brain when you're in the process of realising a great idea. I could remove the UA header... From the request... Then the worker can make any subrequests it likes... Requests to a different origin... THE WORKER CAN RECEIVE A REPORT AND FORWARD IT!!!

I realised that (of course) a Cloudflare Worker could be used to receive reports on a subdomain of your own site and then forward them to your reporting address at Report URI.

Using Cloudflare Workers As A Report Proxy

One of the main benefits of using Report URI is just how simple everything is to do and all of the solutions mentioned at the start of this blog changed that. With a Cloudflare Worker we could keep the absolute simplicity of deploying Report URI but also easily allow you the option to shield your client's IP addresses, or any other information in the payload, from us.


let subdomain = 'scotthelme' addEventListener('fetch', event => { event.respondWith(forwardReport(event.request)) }) async function forwardReport(req) { let newHdrs = new Headers() newHdrs.set('User-Agent', req.headers.get('User-Agent')) const init = { body: req.body, headers: newHdrs, method: 'POST' } let path = new URL(req.url).pathname let address = 'https://' + subdomain + '.report-uri.com' + path let response = await fetch (address, init); return new Response(response.body, { status: response.status, statusText: response.statusText }) }

This simple worker, deployed on your own site, provides a solution to all of the above problems. All you need to do is configure your subdomain in the var on the first line and everything else will be taken care of for you. Deploy this worker onto the subdomain you want to send reports to, follow the same naming convention for the path when sending reports, and everything will Just Work(TM).

The script above is configured for my subdomain, so if I wanted to deploy this on any site, say example.com, I'd choose the subdomain on my site where I wanted to send reports report-uri.example.com and off we go.


https://scotthelme.report-uri.com/r/d/csp/enforce becomes https://report-uri.example.com/r/d/csp/enforce

The reports are now being sent to a subdomain under your own control, the worker will intercept the request and forward it to the destination at Report URI for you. In the process you will shield the client IP as we will only see the source IP as being the Cloudflare Worker and in the example above we are forwarding the UA string for browser identification.

Amazingly Simple

With the worker above we don't need to worry about setting up a CNAME, certificate provisioning, separate infrastructure or anything else that goes with it. You also don't need to worry about setting up and managing a proxy to forward the reports to us and traffic or processing power required to do so. The worker will take care of all of that and what's best is that it will take care of it with minimal overhead, taking only a few minutes to setup and costing only $0.50 for every 1 million reports it processes.

Taking It One Step Further

The great thing about this is that once the worker is setup and processing reports, you can start to do some pretty awesome things beyond just proxying reports, workers are incredibly powerful.

Downsample report volume

If you want to save your quota on Report URI, maybe you're early in the process of deploying CSP and it's quite noisy, no problem. The worker can select a random downsample of reports to forward on so you can still receive reports but not eat your quota quite as quickly. Make the following change to the start of the forwardReport() function to randomly drop 50% of reports.


async function forwardReport(req) { if(Math.floor((Math.random() * 100) + 1) <= 50)="" {="" return="" new="" response("discarded")="" }<="" code="">
Hide the UA string

If you did want to hide the UA string from Report URI and not let us see that either, you simple need to remove the following line of code.


newHdrs.set('User-Agent', req.headers.get('User-Agent'))
Advanced work

The worker can pretty much do anything you like. Maybe there are sections of your site that you don't want to send reports from. You could parse the JSON and check which page triggered the report and discard them. You could do a regex match on the JSON payload to make sure no sensitive tokens or information get sent too. The possibilities are basically endless and what we can say is that if you need to do it, it's easy and cheap enough to do in a Cloudflare Worker.

Pricing

Talking about cheap enough, I thought I'd actually quantify that and quote the Cloudflare pricing for workers.

Starting at $5 per month and covering your first 10 million requests is an amazingly cheap deal. Most websites that report through us wouldn't even come close to sending 10 million reports so you'd probably never pay any more than $5 for your Cloudflare Worker. That's it, $5 per month... By the time you've even thought about creating a CNAME or standing up your own proxy you've probably blown through more than Cloudflare Workers would ever cost you. What's best is that if you already use Cloudflare Workers then you can roll this into your existing usage and it might not even increase the cost if you have some of your initial 10 million requests spare. If you don't use Cloudflare on your site already then you could just as easily grab a new domain name exclusively for reporting, that'd cost just a few dollars, and stand that up behind Cloudflare too. One way or another this is insanely easy and insanely cheap.

Blair Wadman: How to add page templates for content types in Drupal 8

Planet Drupal -

This weeks tutorial will dive into how you can add page templates for a specific content type in Drupal 8. Sometimes you need to create a page template for a specific content type so that you can customise it. Drupal allows you to create content types for node templates out of the box by following the naming convention node--content-type.html.twig. But things aren’t so simple for page templates. Drupal does not automatically detect page templates for content types purely based on the naming convention. Fortunately it just takes a few lines of code and you can create a page template for any content type you choose....

Mark Shropshire: Demystifying Decoupled Drupal with Contenta CMS Presentation at Drupal Camp Asheville

Planet Drupal -

It was a pleasure to present "Demystifying Decoupled Drupal with Contenta CMS" with Bayo Fodeke at Drupal Camp Asheville 2018 on July 14th, 2018. I want to thank the organizers, volunteers, attendees, presenters, and sponsors for making another awesome year in the beautiful North Carolina mountains. This is one great camp that gets better and better each year. Below you will find the video for my talk and slide deck::

"Demystifying Decoupled Drupal with Contenta CMS".

Blog Category: 

Drupal core announcements: Drupal 8.6.0 will be released September 5; alpha begins week of July 16

Planet Drupal -

Drupal 8.6.0-alpha1 will be released the week of July 16

In preparation for the minor release, Drupal 8.6.x will enter the alpha phase the week of July 16, 2018. Core developers should plan to complete changes that are only allowed in minor releases prior to the alpha release. (More information on alpha and beta releases.)

  • Developers and site owners can begin testing the alpha next week.

  • The 8.7.x branch of core has been created, and future feature and API additions will be targeted against that branch instead of 8.6.x. All outstanding issues filed against 8.6.x will be automatically migrated to 8.7.

  • All issues filed against 8.5.x will then be migrated to 8.6.x, and subsequent bug reports should be targeted against the 8.6.x branch.

  • During the alpha phase, core issues will be committed according to the following policy:

    1. Most issues that are allowed for patch releases will be committed to 8.6.x and 8.7.x.

    2. Drupal 8.5.x will receive only critical bugfixes in preparation for its final patch release window on August 1. (Drupal 8.4.x and older versions are not supported anymore and changes are not made to those branches.)

    3. Most issues that are only allowed in minor releases will be committed to 8.7.x only. A few strategic issues may be backported to 8.7.x, but only at committer discretion after the issue is fixed in 8.7.x (so leave them set to 8.7.x unless you are a committer), and only up until the beta deadline.

Drupal 8.6.0-beta1 will be released the week of July 29

Roughly two weeks after the alpha release, the first beta release will be created. All the restrictions of the alpha release apply to beta releases as well. The release of the first beta is a firm deadline for all feature and API additions. Even if an issue is pending in the Reviewed & Tested by the Community (RTBC) queue when the commit freeze for the beta begins, it will be committed to the next minor release only.

The release candidate phase will begin the week of August 13, and we will post further details at that time. See the summarized key dates in the release cycle, allowed changes during the Drupal 8 release cycle, and Drupal 8 backwards compatibility and internal API policy for more information.

Dropsolid: Our Dropsolid CTO featured on the Modern CTO Podcast

Planet Drupal -

16 Jul Our Dropsolid CTO featured on the Modern CTO Podcast Nick Veenhof Drupal Drupal conferenties

Recently, I was invited to go on the Modern CTO podcast as a guest. We talked about developer culture, how to measure efficiency and velocity and, more importantly, how you can make the teams as independent as possible without losing that team and company feeling.

Modern CTO is the place where CTOs hang out. Listen in on our weekly podcast while we hang out with interesting Fortune 500 CTO’s in Aerospace, Artificial Intelligence, Robotics + Many more industries. As of 2018: 72k listeners we are incredibly grateful to each and everyone one of you.

It was a real honour to talk to Joel Beasley and have this back-and-forth conversation about how we transformed Dropsolid into a great place to work, but measurable and technically innovative!

 

 

Some of the topics that we talked about in the podcast were also seen at the presentation I gave at Drupal Developer Days in Lisbon.  Feel free to scroll through the slides to get more context out of the podcast!

 

Drupal Developer Days - One Flew Over The Developers Nest 2018 by Nick Veenhof

OpenSense Labs: Use Elasticsearch to Indexing in Drupal

Planet Drupal -

Use Elasticsearch to Indexing in Drupal Raman Mon, 07/16/2018 - 14:29

Modern applications are expected to be equipped with powerful search engines. Drupal provides a core search module that is capable of doing a basic keyword search by querying the database. When it comes to storing and retrieving data, databases are very efficient and reliable. They can be also used for basic filtering and aggregating of data. However, they are not very efficient when it comes to searching for specific terms and phrases.


Performing inefficient queries on large sets of data can result in a poor performance. Moreover, what if we want to sort the search results according to their relevance, implement advanced searching techniques like autocompletion, full-text, fuzzy search or integrate search with RESTful APIs to build a decoupled application?

This is where dedicated search servers come into the picture. They provide a robust solution to all these problems. There are a few popular open-source search engines to choose from, such as Apache Solr, Elasticsearch, and Sphinx. When to use which one depends on your needs and situation, and is a discussion for another day. In this article, we are going to explore how we can use Elasticsearch for indexing in Drupal.

What is Elasticsearch?

Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time.” – elastic.co 

It is a search server built using Apache Lucene, a Java library, that can be used to implement advanced searching techniques and perform analytics on large sets of data without compromising on performance.

“You Know, for Search”

It is a document-oriented search engine, that is, it stores and queries data in JSON format. It also provides a RESTful interface to interact with the Lucene engine. 

Many popular communities including Github, StackOverflow, and Wikipedia benefit from Elasticsearch due to its speed, distributed architecture, and scalability.

Downloading and Running Elasticsearch server

Before integrating Elasticsearch with Drupal, we need to install it on our machine. Since it needs Java, make sure you have Java 8 or later installed on the system. Also, the Drupal module currently supports the version 5 of Elasticsearch, so download the same.

  • Download the archive from its website and extract it
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.10.tar.gz $ tar -zxvf elasticsearch-5.6.10.tar.gz
  • Execute the “elasticsearch” bash script located inside the bin directory. If you are on Windows, execute the “elasticsearch.bat” batch file
$ elasticsearch-5.6.10/bin/elasticsearch

The search server should start running on port 9200 port of localhost by default. To make sure it has been set up correctly, make a request at http://localhost:9200/ 

$ curl http://localhost:9200

If you receive the following response, you are good to go

{   "name" : "hzBUZA1",   "cluster_name" : "elasticsearch",   "cluster_uuid" : "5RMhDoOHSfyI4a9s78qJtQ",   "version" : {     "number" : "5.6.10",     "build_hash" : "b727a60",     "build_date" : "2018-06-06T15:48:34.860Z",     "build_snapshot" : false,     "lucene_version" : "6.6.1"   },   "tagline" : "You Know, for Search" }

Since Elasticsearch does not do any access control out of the box, you must take care of it while deploying it.

Integrating Elasticsearch with Drupal

Now that we have the search server up and running, we can proceed with integrating it with Drupal. In D8, it can be done in two ways (unless you build your own custom solution, of course).

  1. Using Search API and Elasticsearch Connector
  2. Using Elastic Search module
Method 1: Using Search API and Elasticsearch Connector

We will need the following modules.

However, we also need two PHP libraries for it to work – des-connector and php-lucene. Let us download them using composer as it will take care of the dependencies.

$ composer require 'drupal/elasticsearch_connector:^5.0' $ composer require 'drupal/search_api:^1.8'

Now, enable the modules either using drupal console, drush or by admin UI.

$ drupal module:install elasticsearch_connector search_api

or

$ drush en elasticsearch_connector search_api -y

You can verify that the library has been correctly installed from Status reports available under admin/reports/status.

Viewing the status of the library under Status ReportsConfiguring Elasticsearch Connector

Now, we need to create a cluster (collection of node servers) where all the data will get stored or indexed.

  1. Navigate to Manage → Configuration → Search and metadata → Elasticsearch Connector and click on “Add cluster” button
  2. Fill in the details of the cluster. Give an admin title, enter the server URL, optionally make it the default cluster and make sure to keep the status as Active.Adding an Elasticsearch Cluster
  3. Click on “Save” button to add the cluster
Adding a Search API server

In Drupal, Search API is responsible for providing the interface to a search server. In our case, it is the Elasticsearch. We need to make the Search API server to point to the recently created cluster.

  1. Navigate to Manage → Configuration → Search and metadata → Search API and click on “Add server” button
  2. Give the server a suitable name and description. Select “Elasticsearch” as the backend and optionally adjust the fuzzinessAdding a Search API server
  3. Click on “Save” to add the serverViewing the status of the newly added server
Creating a Search API Index and adding fields to it

Next, we need to create a Search API index. The terminologies used here can be a bit confusing. The Search API index is basically an Elasticsearch Type (and not Elasticsearch index). 

  1. On the same configuration page, click on “Add Index” button
  2. Give an administrative name to the index. Select the entities in the data sources which you need to indexAdding the data sources of the search index
  3. Select the bundles and language to be indexed while configuring the data source, and also select the indexing order.Configuring the added data sources
  4. Next, select the search API server, check enabled. You may want to disable the immediate indexing. Then, click on “Save and add fields”Configuring the search index options
  5. Now, we need to add the fields to be indexed. These fields will become the fields of the documents in our Elasticsearch index. Click on the “Add field” button.
  6. Click on “Add” button next to the field you wish to add. Let’s add the title and click on “Done”Adding the required fields to the index
  7. Now, configure the type of the field. This can vary with your application. If you are implementing a search functionality, you may want to select “Full-text”Customizing the fields of the index
  8. Finally, click on “Save Changes”
Processing of Data

This is an important concept of how a search engine works. We need to perform certain operations on data before indexing it into the search server. For example, consider an implementation of a simple full-text search bar in a view or a decoupled application. 

  1. To implement this, click on the “Processors” tab. Enable the following and arrange them in this order.
    1. Tokenization: Split the text into tokens or words
    2. Lower Casing: Change the case of all the tokens into lower
    3. Removing stopwords: Remove the noise words like ‘is’, ‘the’, ‘was’, etc
    4. Stemming: Chop off or modify the end of words like  ‘–-ing’, ‘–uous’, etc

      Along with these steps, you may enable checks on Content access, publishing status of the entity and enable Result Highlighting
  2. Scroll down to the bottom, arrange the order and enable all the processes from their individual vertical tabs.Arranging the order of Processors
  3. Click on “Save” to save the configuration.

Note that the processes that need to be applied can vary on your application. For example, you shouldn’t remove the stopwords if you want to implement Autocompletion.

Indexing the content items

By default, Drupal cron will do the job of indexing whenever it executes. But for the time being, let’s index the items manually from the “View” tab.

Indexing the content items

Optionally alter the batch size and click on “Index now” button to start indexing.

Wait for the indexing to finish

Now, you can view or browse the created index using the REST interface or a client like Elasticsearch Head or Kibana. 

$ curl http://localhost:9200/elasticsearch_drupal_content_index/_search?pretty=true&q=*:* Creating a view with full-text search

You may create a view with the search index or use the REST interface of Elasticsearch to build a decoupled application.

Example of a full-text search using Drupal viewMethod 2: Using Elastic Search module

As you may notice, there is a lot of terminology mismatch between Search API and Elasticsearch’s core concepts. Hence, we can alternatively use this method.

For this, we will need the Elastic Search module and 3 PHP libraries – elasticsearch, elasticsearch-dsl, and twlib. Let’s download the module using composer.

$ composer require 'drupal/elastic_search:^1.2'

Now, enable it either using drupal console, drush or by admin UI.

$ drupal module:install elastic_search

or

$ drush en elastic_search -y Connecting to Elasticsearch Server

First, we need to connect the module with the search server, similar to the previous method.

  1. Navigate to Configuration → Search and metadata → Elastic Server
  2. Select HTTP protocol, add the elastic search host and port number, and optionally add the Kibana host. You may also add a prefix for indices. Rest of the configurations can be left at defaults.Adding the Elasticsearch server
  3. Click on “Save configurations” to add the server
Generating mappings and configuring them

A mapping is essentially a schema that will define the fields of the documents in an index. All the bundles of entities in Drupal can be mapped into indices.

  1. Click on “Generate mappings”
  2. Select the entity type, let’s say node. Then select its bundles. Optionally allow mapping of its childrenAdding the entity and selecting its bundles to be mapped
  3. Click on “Submit” button. It will automatically add all the fields, you may want to keep only the desired fields and configure them correctly. Their mapping DSL can also be exported.Configuring the fields of a bundle
Generating index and pushing the documents

Now, we can push the indices and the required documents to the search server.

  1. For that, move on to the indices tab, click on “Generate New Elastic Search Indices” and then click on “Push Server Indices and Mappings”. This will create all the indices on the server.
  2. Now index all the nodes using “Push All Documents”. You may also push the nodes for a specific index. Wait for the indexing to finish.Managing the indices using the admin UI
Conclusion

Drupal entities can be indexed into the Elasticsearch documents, which can be used to create an advanced search system using Drupal views or can be used to build a decoupled application using the REST interface of Elasticsearch. 
While Search API provides an abstract approach, the Elastic Search module follows the conventions and principles of the search engine itself to index the documents. Either way, you can relish the flexibility, power, and speed of Elasticsearch to build your desired solution.

blog banner blog image Blog Type Tech Is it a good read ? On

Behind the Screens with Elli Ludwigson

Lullabot -

Elli Ludwigson fills us in on how a DrupalCon sprint day comes together and how you can participate, either as a mentor, sprinter, or planner. And, always put up some flowers to appease the neighbors.

Pages

Subscribe to Cruiskeen Consulting LLC aggregator - Drupal News