The Challenge of Drupal 8 Implementation - The Hardcore Experience

Submitted by vorapoap on Mon, 05/11/2015 - 22:56

Hello everyone, this is the first blog post in the series about my experience at one Thailand's big media firm which is still in the transition process from the business 2.0 that has lasted for more than 3 decades. The process is tough and some of it failed before my time, but now I hope that what I have been doing will give the company the bright future. As the very first pioneer company who implemented Drupal8 in Thailand (or in the world?), I am quite sure that this will be useful and interesting to some people.

Making Decision is the first Challenge

I have been offered the authority to make many major decisions for the company both in business part and technology part. However I tend to started on the technology part first since my background is very strong in such area. My fundamental belief is base on creating a good product, and such good product will create a long-term beneficial relationship and generate a sustainable income. Since the company I worked for is the media company, our goal is to bring customer to our advertiser - so a good product must be able to bring customer to our advertiser. And to create a long-term relationship, a product itself must be sustainable and maintainable.

A loosen framework or non-existing-framework at all is not sustainable in my view. Some thousands line of code in a single file without a single function is an example of non-sustainable product. An piece of code which is alien to other people in the team is also the other example. Lots of people in an executive level doesn't care about this as long as it gives working result and generate income.  But to me, we need to establish a working standard, a working framework to make sure we are here to live not just now but to make sure the foundation is strong for the future. And here, Drupal8 comes in.

Not everyone knows Drupal, and Drupal has less popularity comparing to other CMS. But it offers our team a solid web development framework to create a system with a full-featured multi-lingual content management system, user access and security control and complex interaction for our users, this is why I am totally agree to the article written by John Lock that "Drupal is a great fit for startups". Not counting that, Drupal 8 will be successful like its ancestors. It will be supported by the very strong community creating ton of plugins and add-on modules.

However, a holy lance doesn't come with simplicity. 

Source: Learning Curve for Popular CMS

Drupal developers requires a long learning-curve for starting. Especially Drupal8, the slope magnifies! - less to little documentation, out-dated information given and doesn't reflect the latest API changes. This somewhat discouraged my decision and my team during the beginning. In such timing, we implemented a prototype for our customer data management system in Drupal7. But I am not a type of people who giving things up easily. And now... as of today, we start to have a strong foundation of system implemented purely in Drupal8 which will eventually turn into an eco system for our internal user, our customer and end-users.

System Integration and Migration is the second Challenge

The project I am doing will not be just the eco system for our customers and users, but it will also offers variety of services that must me migrated from the generation 2.0 system and coding. The first key priority is our Search Engine. Up to today, we rely mainly on the FAST ESP (The Norwegian Search Engine) which is now acquired by Microsoft. I doesn't see much light on continuing using this product as we are approaching the future very quickly. We need something more sustainable, more maintainable and more manageable with simplicity in scalability and can provide us a high performance search engine with limited hardware support. I will come to share information in this area when the timing is right.

Our current servers of stand-alone MySQL database also suffers badly from the high traffic. We are now in the stage of replacing them with the cluster running MariaDB Gallera Server. The cluster will provide the long term support for any of our "legacy" database and as a base database for our Drupal8 for a time being. The database migration will be very painful since our trial on exporting data from MyISAM tables alone from one of our key server drained all the resource and crashed the rest of the system in the network. Our team will come to that, meanwhile we have established several layers of caching using APC and MemCache to ensure our system can survive the high load for the time being.

The MariaDB Gallera Cluster will run along side with our cluster of MongoDB database. MongoDB provides the document-type database system which perfectly fits our ambiguous database structure for the future. It will mainly power our expanding business listing and Online Catalog database dealing different types of items(meaning different schema). Some may disagree why we would need 2 types of database system running together. But I have a strong business case for that - SQL database has been here for at least 3 decades(since 1970?), every developer knows about it. And it is a right tool dealing with a kind of transaction and tabular data. Our team is familiar and confident with it and will use it  as a primary database of all source. But when it comes to our front end database which requires instantly schema change without locking up the whole table,  I am pretty sure MongoDB is the answer.

By having groups of cluster servers base on functionalities (not by project). I believe we can maximize the performance of our hardware. 4+3+3+3+extra network storage/load balance will be a minimum formula for our system setup. Retiring machines will be used to increase bandwidth in each group. We will be able to plan which machine will be removed, which MA will discontinue. The money will be spent much wisely with solid reason and timing.

We also have other problematic part on synchronizing our database of over 400 thousands records. The way things are transferring is in old-fashioned of drag-and-drop folder-to-folder transfer. The database entries were also updated in the similar way but in broken XML format. The thing may work well in a small scale, but wound bleeds when we handle more and more data every year. Never-ending day-to-day problem solving is what we are tacking daily and in the same time we must implement the new method of synchronizing using our limited resource. Thanks to our team! Although we are not 100% in completing this part of the work, but I already foresee ... we are going to ..!

...

So...... if you have read till this line, that is all for the introduction. The hardcore story is coming....